Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Authors: Tyler Ingebrand
Adam Thorpe
ufuk topcu
Venue: ICML 2025 poster (2025)
Keywords: Transfer Learning
Learned Basis Functions
Function Encoder
Primary area: theory->domain_adaptation_and_transfer_learning
TL;DR: This paper introduces a novel categorization of inductive transfer based on interpolation and extrapolation, and demonstrates that the function encoder achieves superior performance across four diverse transfer learning benchmarks.

Abstract: A central challenge in transfer learning is designing algorithms that can quickly adapt and generalize to new tasks without retraining. Yet, the conditions of when and how algorithms can effectively transfer to new tasks is poorly characterized. We introduce a geometric characterization of transfer in Hilbert spaces and define three types of inductive transfer: interpolation within the convex hull, extrapolation to the linear span, and extrapolation outside the span. We propose a method grounded in the theory of function encoders to achieve all three types of transfer. Specifically, we introduce a novel training scheme for function encoders using least-squares optimization, prove a universal approximation theorem for function encoders, and provide a comprehensive comparison with existing approaches such as transformers and meta-learning on four diverse benchmarks. Our experiments demonstrate that the function encoder outperforms state-of-the-art methods on four benchmark tasks and on all three types of transfer.

Function Encoders:
A Principled Approach to Transfer Learning in Hilbert Spaces
Tyler Ingebrand 12
Adam J. Thorpe
Ufuk Topeu' 13
Abstract
H
A central challenge in transfer learning is design- ing algorithms that can quickly adapt and gener- alize to new tasks without retraining. Yet, the conditions of when and how algorithms can ef- fectively transfer to new tasks poorly char- acterized. We introduce geometric character- ization of transfer in Hilbert spaces and define three types of inductive transfer: interpolation within the convex hull, extrapolation to the lin- car span, and extrapolation outside the span. We propose method grounded in the theory of func- tion encoders achieve all three types of trans- fer. Specifically, we introduce novel train- ing scheme for function encoders using least- squares optimization, prove universal approxi- mation theorem for function encoders, and pro- vide a comprehensive comparison with exist- ing approaches such as transformers and meta- learning on four diverse benchmarks. Our experi- ments demonstrate that the function encoder out- performs state-of- of- ff--hhe-n art methods four bench- mark tasks and on all three types of transfer.
Type
Type 2
Type 1
Training Functions
Figure The Categorization of Transfer Learning. Black points are functions present in the training set. Purple points indi- cate type transfer, interpolation within the convex hull. Orange points represent type 2 transfer, extrapolation to the linear span, The red point type transfer, extrapolation the Hilbert space.
scenarios or terrain, and computer vision models for med- ical diagnoses should be able to generalize entirely new tasks, such as identifying emerging or rare diseases. While training internet-scale data has shown promise in im- proving generalization performance by increasing data di- versity, it does not avoid the fundamental issue of gaps in the data, does not directly address knowledge transfer, and does not innately enable adaptation to new, evolving tasks. Thus, a crucial goal is to develop models that are capable of transfer runtime without retraining. Addressing these challenges requires incorporating mathematical structure with learning-based representations to quantify and exploit the relatedness of tasks seen during training.
Project page: syler-ingehrand..gin wwnnchhrannggghuuhio/Frrreeeer.
1. Introduction
Learned models must be able to draw upon multiple knowl- edge sources to handle tasks that were not encountered dur- ing training. For example, robots operating in remote, un- structured environments must be able to adapt to unseen
We consider an inductive transfer setting and present a ge- ometric characterization inductive transfer using princi- ples from functional analysis. Inductive transfer involves transferring knowledge to new, unseen tasks while keeping the data distribution the same. For instance, labeling im- ages according to a new, previously unknown class, where only a few examples are provided after training. While prior works have studied transfer learning, gaps remain in identifying when learned models will succeed and when they will fail. We seek characterization of inductive trans-
'Oden Institute for Computational Engincering and Sciences, The University Texas Austin, Austin, TX, USA 'Chandra Department of Electrical and Computer Engineering. The Uni- versity of Texas Austin, Austin, TX, USA 'Deparmment of Aerospace Engineering and Engineering Mechanics, The Univer- sity of Texas Austin, Austin, TX, USA. Correspondence to: Tyler Ingebrand <tyleringebrande @uteaaa.du.xx.............
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
fer, based on Hilbert spaces, which will provide intuition about the difficulty of given transfer learning problem. Formally, we consider a Hilbert space H. of tasks, and the inductive transfer problem is to accurately represent a new, unknown task f E H without retraining, using only a few data examples provided at runtime.
source and target tasks. However, they require task-specific fine-tuning retraining, which may fail when tasks only weakly related or when task relatedness is not explic- itly modeled. In contrast, we compute a task representation from data without retraining the underlying model.
Other approaches use the idea of basis functions, but in dis- tinet ways. Kernel methods fit a function by placing a ba- sis function at every observed data point, and represent the function as linear combination of these basis functions (Hofmann et al., 2008). Kernel methods can also be used for transfer learning (Radhakrishnan et al., 2023). How- ever, kernel methods may scale poorly with data, because the size of the Gram matrix grows with the amount of data, and require prior knowledge, e.g., the user must choose the kernel. In contrast, the function encoder scales well the amount of data because it uses fixed number of basis func- tions, and requires no prior knowledge because the basis functions are learned. Dictionary learning decomposes a data matrix into a set of atoms. Functions are represented as a linear combination of the atoms, often with sparse regu- larization (Tillmann, 2015; Aharon et al., 2006). However, these atoms are only represented at fix input locations, and thus are more akin discretized representation of basis function. In contrast, the function encoder can be evaluated at any point in the input domain.
Specifically, we characterize transfer using three types:
(Type 1) Interpolation within the convex hull. Tasks that can be represented as a convex combination of ob- served source tasks.
(Type 2) Extrapolation to the linear span. Tasks that are in the linear span of source tasks, which may lie far from observed data but share meaningful features.
(Type 3) Extrapolation the Hilbert space. Tasks that are outside the linear span of the source predictors in an infnite-dimensional function space. Type 3 transfer is the most important and challenging form of transfer.
We propose a novel method to achieve transfer across all three types using the theory of function encoders (Inge- brand et al., 2024b). Function encoders learn a set of neural network basis functions to represent elements in a Hilbert space. We generalize the theory of function encoders to the full Hilbert space transfer learning setting. Further, we show that this approach has natural and principled means of extrapolation, a difficult task for existing inductive trans- fer learning approaches.
Internet-scale training and fine-tuning pretrained models is gaining interest for improving generalization performance. For example, the Open-X Embodiment dataset (O (O'Neill et al., 2024) aggregates data from diverse robot platforms to train models that generalize across tasks and robot sys- tems. These approaches seek to leverage sheer data volume and scale to enable transfer instead of leveraging structural insights or feature relationships between tasks. Transform- ers ( (Vaswani et al., 2017) and generative pretrained models perform well on language tasks and time-series prediction, but perform poorly even in simplistic benchmark transfer tasks. There is some preliminary work assessing trans- fer quality for pretrained models (Mehra et al., 2024). Our approach may be useful for such models, e.g. in mixture- of-experts (Jacobs et al., 1991).
1.1. Related Work
Transfer learning and generalization have been key focus of the machine learning community. However, much of the existing literature focuses on statistical shifts (out-of- distribution generalization) rather than structural and task- related perspectives (Zhuang et al., 2021). We employ the definitions from Pan & Yang (2010), which distinguishes transfer learning based on differences between the source and target domains. Our focus is on inductive transfer learning. Several transfer learning approaches focus modeling task relatedness, including via graph-based rep- resentations (Eaton et al., 2008) and manifolds (Ko et al., 2024). We consider a similar geometric view of task relat- edness which depends on the relationship between a target task and the training data.
1.2. Contributions
We summarize our contributions as follows.
A geometric characterization of transfer. We define and characterize three types of inductive transfer, which capture geometric task relationships in Hilbert spaces.
Meta-learning approaches, such as MAML (Finn al., 2017) and related meta-adaptation methods (Nichol et al., 2018), aim to adapt to new tasks by learning parame- ters or representations that can quickly adapt to new data (Hospedales et al., 2022). These approaches are effective in scenarios with a high degree of similarity between the
A method to achieve transfer across all three types. We present principled method for inductive transfer using the theory of function encoders (Ingebrand et al., 2024b). We propose a novel approach to train function encoders us- ing least squares, prove a universal function space approx-
2

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
imation theorem for function encoders, and generalize the function encoder new function spaces.
and aim to transfer dynamics estimates policies to new environment. To make use of Hilhert space theory, we make the following assumption about the tasks.
A comprehensive comparison with the state of the art. We compare state-of-the the-art art approaches for inductive trans- fer and meta-learning with our proposed approach on sev- eral benchmark transfer tasks, including regression, image classification, camera pose estimation, and dynamies mod- cling. We demonstrate that our approach achieves compa- rable performance at type transfer and superior perfor- mance at type 2 and type 3 transfer.
Assumption 1. Consider Hilbert space H of functions from X' to y equipped with the inner product (i-- We assume that the predictors f fs. and fr are ele- ments of H.
Note that Assumption 1 is not restrictive because Hilbert spaces are generally flexible and encompass wide number of problems of interest. For instance, under mild regularity assumptions, the space 2 of square-integrable functions is Hilbert space. We can define Hilbert spaces over many types of functions and tasks, such as probability distribu- tions. See Appendix C for more information.
2. Background
2.1. Inductive Transfer in Hilbert Spaces
We present the following definitions, adapted from Pan & Yang (2010) of domains, tasks, and transfer learning.
2.2. A Geometric View of Transfer
Definition (Domain). A domain D = (X, P(2)) consists of an input space X and a marginal distribution (.)).
We first seek to characterize the various geometric rela- tionships between the target task fr and the source tasks fs.....: fs. via the properties of the Hilbert space. Specif- ically, we define three types of inductive transfer in Hilbert spaces: interpolation within the convex hull of source pre- dictors, extrapolation to the linear span of source predic- tors, and extrapolation to the rest of H.
Definition 2 (Task). A task T = ff. consists of an out- put space y and a predictor f X' > Y, which is not observed, but can be learned from data. Alternatively, the predictor can be a conditional distribution PP(|| X)..
In general, we presume that we do not have access to the predictors directly. Instead, we presume access to a dataset consisting of pairs (xx y) E X x y
We first consider a typical case of generalization for learned models, where the target predictor lies within the convex hull Ch. of the source predictors
fS..
Definition 3 (Dataset). A dataset D = {(x,,, vi) con- sists of points I3 drawn from P(X), and the corresponding labels yi from y.
Cn = - osss, Σo;=
s=
0: = 10==1,0;3 20
Definition& 4 (Transfer Learning, Pan & Yang, 2010, Defi- nition 1). Let Ds and Ts be the source domain and task, and Dr and Tr be the target domain and task. Transfer learning seeks to improve the target predictor fT using the knowledge in Ds and Ts. where Ds + Dr or Ts + Tr-
(1)
Definition 5 (Type 1, Interpolation in the Convex Hull), Given source predictors fs....., the target task pre- dictor fr is in the convex hull C₁ of the source predictors.
Specifically, we focus on an inductive transfer scenario, where the source and target predictors differ, meaning fss + fo or, alternatively, P()ss Xs) * P(Yrr Xr), but the output spaces V/8 = Y'r and domains Ds DDr are the same. For simplicity of notation, we drop the subscripts S and T on X' and )' where appropriate. Additionally, we presume that multiple source datasets Ds Ds., are available during training, which is similar to the so-called multi-task learning scenario (Caruana, 1997). The model thus has access multiple datasets during training. and we seek to transfer the knowledge from the source domain Dg and tasks .s....., Ts, to new target task fr. This scenario is increasingly common in practice. For instance, in few-shot image classification, we have access to a large dataset of diverse images and their corresponding labels, and we seek to transfer to a new class. In robotics, we have access to data collected from multiple environments,
Next, we consider extrapolation beyond the convex hull of the source predictors to the linear span,
{
n Σ
span Ss.. - SEA E H
Cafs, ar E R
(2)
Definition 6 (Type 2, Extrapolation to the Linear Span). Given source predictors fs.....,fs. fs, the target task pre- dictor fr E span(fs,.....f,, J. We presume f & Cn to distinguish extrapolation from interpolation.
Extrapolation to the linear span extends beyond standard generalization performance and tests whether the learned features are meaningful. In other words, extrapolation to the linear span implics that the model has learned some- thing about the structure of the space of predictors.
3

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Lastly, we consider the extreme case where the target task lies outside the span of source predictors. This type of transfer is often the most desirable, since it involves trans- ferring (partial) knowledge to entirely different tasks.
tion f can be computed using the inner product method,
(4)
(f, 9k
Definition 7 (Type 3, Extrapolation to H). The target task predictor Jr E H. but fr &
which leads to stable learning algorithm (Ingebrand et al., 2024b). Computing (4) involves evaluating the inner prod- uct. Many commonly used inner products such as the L inner product (f. g) = fx f()) )dx involve comput- ing integrals over the input space, which is computationally intractable. Instead, we can use Monte Carlo integration to approximate the inner product using data. For exam- ple, can estimate the L inner product using dataset D = {(11, {k)) 1=1 as
Note that this scenario is particularly challenging, since it involves transferring to parts of H that significantly differ from the observed tasks. Additionally, H. typically infi- nite dimensional, while we have finite training tasks, and so there are infinite dimensions which fr may differ from the source tasks. It is not possible, a priori, to determine which of these infinite dimensions are relevant.
3. Function Encoders
(f.9))12= = J (())))
= m Σ (())),)(a)), (5)
A natural representation of a task f H is via lin- combination of basis functions. We employ method for learning finite set of neural network basis functions in Hilbert spaces called the function encoder (Ingebrand et al., 2024b). Function encoders learn a set, (9I,.....9 J. of basis functions parameterized by neural networks to span a Hilbert space of functions, Then, functions in the learned space are represented via a linear combination of the learned basis,
where V is the volume of X' and f(z)) = yr- Since gi is a neural network basis function, we can query it for arbitrary data points I, In practice, V is unknown, so we assume V 1, which scales the inner product by constant value 1/V. This approximation induces weighted inner product if P(X) is not uniform. For discussion, see Appendix D.
The inner product method of computing the coefficients suffers from two main drawbacks. First, as noted in In- gebrand et al. (2024b), the basis functions naturally tend toward orthonormality during training to minimize the loss. However, during training, the inner product method intro- duces errors in the coefficient calculations if this property is not explicitly enforced at each step, e.g., using Gram- Schmidt, which can be computationally expensive. Sec- ond, estimating the coefficients via the inner product con- verges slowly as the basis functions must converge to or- thonormality to minimize loss. Below, we provide a faster and more accurate method for computing the coefficients.
k (xx) = Σ gg 0,)-
(3)
We first provide a brief overview of the theory of function encoders. Then, we present a novel approach to training function encoders using least-squares optimization, which offers several numerical and computational benefits. Ad- ditionally, present a universal approximation theorem for function encoders. Lastly, we remark that we have gen- eralized all function encoder definitions to use only inner products. Thus by appropriately defining the inner prod- uct, we also generalize the algorithm to any function space, e.g., probabiliyy distributions for classification.
Computing Coefficients via Least Squares. We pro- pose modifying the function encoder training procedure to compute the coefficients as the solution to a least-squares optimization problem,
Function encoders consist of two steps: offline training of the basis functions and online inference.
arg min eER
999
(6)
3.1. Offline Training Procedure
The least-squares problem in (6) admits closed-form so- lution. The coefficients for function f are computed as
The basis functions (gi...., } are first trained using a set of datasets {Ds,,... Ds. } as in Definition 3 corre- sponding to the source predictors .s Ss.- For each function fs, and dataset Ds., we first compute the coeffi- cients of the basis functions and then obtain an empirical estimate fs. of fs. via (3).
(01,91)
(91, (H
(f,
(7)
(gk, 91
(gk. gk)
((F.k)HH)
where the inner products are estimated using Monte Carlo integration as in (5). To compare the relative performance
During training, the coefficients e corresponding to a func-
4

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Algorithm Function Encoder Training (LS) given source datasets {Ds.,... Ds..t J. learning rate a Initialize basis {91,.... g with parameters while not converged do for all Ds. (Ds......... Ds do 11)x
of the basis functions. Notably, the coefficient calculations are computationally simple. The inner product approxima- tion is effectively sample mean, which can be computed quickly in parallel. Furthermore, if using the sast-ss method, the Gram matrix in (7) is of size k x k, which we select as a hyperparameter, meaning the matrix inverse can be computed quickly even for large datasets.
9x)
(fse.
d=
(gk,9) 91)
(gk (k. HL
(
3.3. Universal Function Space Approximation Theorem
fst Σラ= c3g9
An important question is which properties a Hilbert space must have to be well represented by learned basis functions. In this section, we provide an existence proof to show that a function encoder represent any separable Hilbert space with arbitrary precision.
end for
L + p= Il.ss - fse 11
Treg f Σ ( (1lg: 1igi - 1)²
0 - a V + Lireg)
end while return (01...., 9
Theorem 1. Let K C IR" be compact. Define the in- ner product (f.9/H == Sx (x)) g(x)dx and the induced norm |I == (f.f) Let H = {f K - If continuous, llf < oo) be a separable Hilbert space. Then, there exist neural network basis functions {ê,, E,.... such that for any e > and for any function f E H. there exists N E N and c RR such that
of the coefficient calculation methods, we distinguish the inner product method (IP) using (4) and the least- squares method (LS) using (7).
The solution in (7) provides a theoretically optimal projec- tion onto the learned basis in the least-squares sense. The key benefit of least squares is that it does not require the basis functions be orthonormal. Instead, we only pre- sume that the basis functions linearly independent, mild assumption (see Lamperski, 2022). In practice, this advantage is significant, and least squares performs better in most cases.
115- Σ Geill < <e|1.|lae
Proof (Sketch). Every separable Hilbert space H. has a countable orthonormal basis {c1, c2,...) (Oden & Demkowicz, 2018, Theorem 6.3. 1), and any function f e H. can be represented in terms of the basis. Let é be neu- ral network approximation of eg. By the universal approx- imation theorem of neural networks (Leshno et al., 1993), such neural network always exists with arbitrary error. Bound the error of each basis function approximation as a decreasing geometric series. Then, the overall error of representing any function as a linear combination of these approximations is finite and arbitrarily small.
Although least squares conceptually simple, the use of least squares during the training process imposes additional computational challenges. Primarily, least squares requires additional regularization to ensure the basis function mag- nitudes remain in an acceptable range. See Appendix A.
Training Algorithm. To train the basis functions, we as-
sume access to a dataset Ds. for each source task fs.- Us- ing this dataset, we compute the coefficients cf for each Js, using either IP or L., depending on the method cho- sen. This yields an approximation for fs, via (3). The loss the mean approximation error over source tasks,
The proof shows that in the limit of infinite basis functions, any member of the Hilbert space is well approximated by function encoder. See Appendix B for the full proof and a discussion.
n
4. Experimental Comparison of Transfer
L
fsr
9
(8)
71
P
We compare our proposed approach against existing trans- fer approaches on challenging transfer scenarios. Specifi- cally, we compare function encoders using the inner prod- uet (IP) method (Ingebrand et al., 2024b), function coders using the least-squares (LS) method (ours), auto encoders, transformers (Vaswani et al., 2017), transformer functional encodings (TFE), an oracle with privileged in- formation, MAML (Finn et al., 2017), and two naive algo- rithms: brute force (BF) and brute force basis (BFB). We
The basis functions are trained via gradient descent to min- imize this loss. See Algorithm for the pscudo-code.
3.2. Online Inference
After training, we may approximate a target task fr given a small dataset Drr We use the dataset to compute the coefficients, and approximate fr as linear combination
5

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Type Transter
Type Transter
Type Transter
104.
10r
12 Doetams Squared
10P
10
10".
10"
10a
101
10P:
to"
10k
105
1o-1
10-1
10
10
10
10"1
10
10-3
10
10"11
12500
250000 37800
50000
12500 250000 373000
50000
12500
25000
37500
50000
12500 25000
50000
Step
Step
FE (LS)
P2 (IP)
AB
Theas
IVE
Uracle
DV
MANL (n-
MAML (Um5)
Figure 2. Empirical Results on the Polynomial Dataset. While many approaches demonstrate moderate type transfer, only the function encoder successfully achieves all three types, as illustrated by its orders of magnitude advantage over other approaches.
Type Transter
Type 2 Transfer
Type Transfer
200
0-
F7rt- 100)
3122-1)
150
-50
40
-100
100
2
30
1150
200
0-
-50
2
Figure 3. Qualitative Analysis of Transfer on the Polynomial Dataset, In this illustrative example, we visualize the function encoder and one baseline, the auto encoder, on each of three types transfer. observe that both approaches achieve reasonable per- formance for type I transfer. For type 2 transfer, the target function is much larger in magnitude than any function in the training set. The auto encoder fails at this function because has only learned to output functions from the training function space. In contrast, the function encoder generalizes to the entire span of the training function space by design. For type 3 transfer, the target function is cubic function. The auto encoder nonetheless outputs a function that is similar to the ones seen during training. When using function encoder with only three basis functions, the basis functions only span the three-dimensional space of quadratic functions, and its approximation is the best quadratic to fit the data. When using 100 basis functions, the basis functions spans the space of quadratics, but additionally have 97 unconstrained dimensions. Due to the use of least squares, the function encoder with 100 basis functions optimally uses these extra 97 dimensions to fit the new function. Therefore, it is able to reasonable approximate this function as well, despite having never seen cubic function during training.
assess the capability of these existing approaches trans- fer to new tasks that have varying geometric relationships to the set of observed source predictors. See Appendix I for more information on the baselines.
current approaches. The models are trained using polyno- mials fax² + br +c a, c 1-333}3 We evaluate type I transfer by sampling unseen functions from the same space. We evaluate type 2 transfer be sampling polynomials with coefficients a, b, e E [-20, 20). We then consider extrapo- lation to a new function class, {f(x) = az az² +bx + cr++d d a, b, c, [-),3],
We consider four benchmark transfer tasks: 1) a polyno- mial regression task to illustrate the proposed categories, 2) a CIFAR image classification task, 3) an inference task on the 7-Scenes dataset, and 4) a dynamics estimation task on MuJoCo data. See Appendix H for more information. We show that the proposed approach, FE (LS), transfers to new tasks, even when the tasks are outside the convex hull of source predictors. The algorithm learns the underlying structure and can transfer knowledge to new domains.
. to evaluate type transfer.
In Figure 2, see that existing approaches achieve moder- ate type 1 transfer, interpolation in the convex hull. How- ever, existing approaches fail to extrapolate to the linear span (type 2) or to other functions in H (type 3), while the function encoder using least squares achieves low T,2 error.
The function encoder's excellent performance the result of least squares, which is the optimal projection of the tar- get task onto the learned basis. For type and type trans- fer, this projection is almost perfect recreation of the tar- get task, since the basis functions are trained to span the source tasks, and the target task lies in the span of the
4.1. An Illustrative Polynomial Regression Task
We first consider transfer on a simple polynomial regres- sion task. This problem is designed to be simple while still demonstrating the types of transfer and failure modes of
6

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Type Transfor
Type 3 3'Tansfee
10075
100%
100%
90%
80%
00%
80%
70%
70%
70%
00%-
6775
00%+
50%
50%
40%
40%
12500
40%
Gradient
Goudinnt Step
Gradient
FE (LSS)
AF
TFF
RPR
MAMI (n=1)
Stumeon Net.
FE (IP)
Ococle
BF
MAML (n=5)
Protu/yyp.
Figure 4. Empirical Results on the CIFAR Dataset. The training curves show the two ad-hoc baselines seem to be performing best, and many algorithms fail to converge on all or some seeds. However, when measuring type 1 transfer, the function encoder performs best, achieving slightly hetter performance than Siamese networks. For type 3 transfer, few-shot classification of unseen classes, the function encoder again performs best, albeit similar to Siamese networks. The key idea is that function encoders are performing comparably to ad-hoc approaches despite being designed for a more general setting.
source tasks. For type? 3 transfer, this projection necessar- ily has error, but the error is minimized given the learned basis. See Figure 3 for a visualization of how this property greatly improves transfer.
1993) and prototypical networks (Snell et al., 2017).
Our results indicate that the function encoder and the two ad-hoc baselines perform best. While many approaches have some convergent seeds, most do not converge con- sistently, indicating the difficulty of learning in this setting. Furthermore, the two ad-hoe basclines perform best on the training set, but demonstrate a significant drop in perfor- mance in a type or type 3 setting. The best algorithm overall the function encoder, although Siamese networks are similar. The key result of this experiment is that the function encoder is achieving comparable performance to ad-hoe algorithms designed explicitly for this setting, de- spite the fact that the function encoder is more general.
4.2. CIFAR Dataset: Classifying Unseen Objects
We evaluate the function encoder and relevant baselines on the CIFAR 100 dataset (Krizhevsky, 2009). Specifically, we evaluate on the few-shot classification problem, where the classifier is given a set of images belonging to a class and a set of counterexamples that belong to other classes, and must determine if a new image belongs to the speci- fied class. The classifier function corresponding to given class is one function from the function space. For exam- ple, the "apple" classifier returns true if an image contains an apple and false otherwise. 90 classes are used during training, and 10 are held out for testing. Type transfer is evaluating& model's performance on unseen images from the training classes. Indeed, these are the training func- tions, but for new inputs. Type 2 transfer not readily testable with this dataset. Conceptually, linear combina- tions of classes would correspond images which belong to both classes. An "apple" classifier plus "green" clas- sifier would correspond to a "green apple" classifier. Type 3 transfer is the model performance on unseen classes. See Figure 19 for a visualization of the data setting.
4.3. 7-Scenes Dataset: Estimating Position from Images
We use the 7-Scenes dataset (Shotton et al., 2013), where the goal is to estimate the position of camera from an image. The example dataset consists of pairs of images and their locations within the scene. As the name suggests, there are seven unique scenes in this dataset, along with various trajectories of images in each scene. Six scenes are used for training, and one is heldout.
Type 1 transfer is again evaluated on the training scenes, but on heldout images. Type 2 transfer is not readily testable, though it conceptually corresponds to a change in units, c.g., changing the position from meters to centime- ters, or changing the origin. Type transfer corresponds to the performance on the unseen scene. See Figure 21 for a visualization of the data setting.
The function encoder uses a probability distribution- based inner product; See Appendix C.2 for more information. To make fair comparison, we also train other algorithms, where applicable, using the same inner produced-based dis- tance function. In Appendix H, we include an experi- ment for other algorithms using a cross-entropy loss func- tion. Furthermore, as is common for image based classifi- cation problems, veighh-regularizzaion is necessary to pre- vent overfitting for some algorithms, including the function encoder. In addition to the standard baselines, we also use two ad-hoc baselines: Siamese networks (Bromley et al.,
The results indicate that while most algorithms achieve strong training performance, all see significant decreases in performance for type 1 and type 2 transfer. The function encoder however achieves the best transfer due to its use of least squares. Effectively, the function learns useful set of features for the training dataset, and then optimally lever- ages these features for transfer, even though these features
7

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Train
Type Transfer
Type 3 Transfer
10a
t2 Ditaaoo Squarred
10-1
30
1or3
10'
30-1
10-1
10-1
12500
25000
37500
50000
12500
25000
37500
50000
0
12500
25000
37300
50000
Gradiest Step
Gradent Step
Gradicnn Step
(LS)
Tress
Oracle
BFB
BF
MAML (i=1)
MAML (n-5)
Figure S. Empirical Results on the 7-Seenes Dataset.
Many approaches converge during training. As expected, all approaches
perform much worse at type transfer, indicating a degree of over-fitting. The function encoder performs best at both type and type transfer, indicating its ability to optimally use the learned features for unseen data.
Teain
Type 1 Transfer
Type 2 Transfer
Type 3 Traansfe
102
102
10
Dietaaze Sqqarred
10P
10P
10-3
30
10a
10-31
12500 25000
37500 Grodicnt Stop
50000
12500 25000 37500 30000
12500
25000
37500
50000
12500 25000 37500 50000
Gradiest Stop
Gradient Stop
Grodicat Stop
(LS)
Tress.
Oraclo
BFB
MAME
MAML (n-5)
Figure 6. Empirical Results on the Ant Dataset. All algorithms demonstrate convergence during training, albeit to various levels. However, many algorithms perform much worse for type transfer. The function encoder performs best, although many approaches. such as the transformer and MAML(n=5), are comparable. Furthermore, the function encoder is clearly best for type transfer. Type 3 transfer tells interesting story. The function encoder has the best stable performance, although approaches such the transformer and the auto encoder are not far behind. At the beginning of training, the auto encoder shows the best overall performance, although it degrades as training continues. This is because training is not optimizing for type 3 transfer, and the best model parameters for the training dataset are not the best model parameters for type 3 transfer. Thus, its performance is unstable.
are likely suboptimal for the unseen images.
achieves good type I transfer, but fails at type 2 and type 3. Interestingly, the auto encoder briefly achieves great type 3 transfer, but its performance degrades as training pro- gresses. This is because minimizing training loss may di- rectly conftict with minimizing type transfer, and type 3 transfer by definition cannot be used for training. The func- tion encoder achieves the overall best transfer, although some algorithms may be comparable for a given type of transfer. Additionally, for simplicity the function encoder did not use either the residuals method (see Appendix E) or neural ODE basis functions, which are minor changes that have been shown lead to great improvements in accuracy for this problem (Ingebrand et al., 2024b;a).
4.4. MuJoCo Dataset: Estimating Unseen Dynamics
We adapt the hidden-parameter MuJoCo Ant dataset from Ingebrand et al. (2024a). This dataset consists of a Mu- JoCo Ant (four-legged robot) walking on a flat surface. The lengths of the robot's limbs, and the control authority, are varied which effectively changes the dynamics func- tion. The goal is predict the next state of the robot given the current state and action, where the example dataset pro- vides data from the current set of hidden parameters.
During training, hidden parameters are sampled from 0.5x to 1x the default robot values. Type transfer consists of unseen paraamers sampled from that same space. Type transfer consists of synthetically generated dynam- ies which correspond to linear combinations of the dynam- ics sampled from the type I parameter space. Type 3 trans- fer consists of hidden parameters sampled from 1.5x to 2x the default values, which corresponds to a much larger robot. See Appendix H.3 for more details on this dataset. See Figure 6 for experimental results.
5. Conclusion
We have introduced a novel categorization of transfer learn- ing based on the geometric interpretation of Hilbert spaces. We made novel improvements to the function encoder al- gorithm, and argued that it is natural solution transfer learning in Hilbert spaces. We empirically validated this argument using four transfer learning datasets.
The results indicate that many approaches achieve some but not all types of transfer this dataset. Meta learning
We leave many open questions for future work. Primar- ily, applying the function encoder algorithm to other set-
8

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
tings requires well-defined inner product. Therefore, de- signing inner products for sequences and graphs is neces- sary and interesting challenge. Furthermore, this work does not address how to efficiently represent a low-dimensional manifold lying in an infinite dimensional space. In such a setting, naively learning basis functions may require an unpractical number of basis functions to reasonable span the manifold, while the manifold itself is low-dimensional. Thus there may be similar algorithms which learn basis only for the manifold.
Egozeue, J., Diaz-Barrero, J., and Pawlowsky-Glahn, V. Hilbert space of probability density functions based on Aitchison geometry. In Acta Mathematica Sinica, vol- ume 22, pp. 1175-1182, 2006.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. In Pro- ceedings of the 34th International Conference on Ma- chine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1 1126-11 1126-1135. 1135. PMLR, 2017.
Hofmann, T., Schölkopf, B., and Smola, J. Kernel meth- ods in machine learning. The Annals of Statistics, 36(3), 2008
Impact Statement
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neu- ral Networks, 2(5):359-366 1989.
Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. Meta-lcarning in neural networks: A survey. IEEE Transactions on Puttern Analysis AND Machine Intelli- gence, 44(9), 2022.
Acknowledgments
This work is supported by DARPA HR0011-24-9-0431,
NSF 2214939, RTX CW2231110, and AFOSR FA9550- 19-1-0005. Any opinions, findings, conclusions or recom- mendations expressed in this material are those of the au- thors and do not necessarily reflect the views of the funding organizations.
Ingebrand, T., Thorpe, A. and Topcu, U. sero-shot trans- fer of neural ODEs. CoRR, Abs/24455089555 2024a.
Ingebrand, T., Zhang, A., and Topcu, U. Zero-shot re- inforcement learning via function encoders. In Pro- ceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 21007-21019, PMLR, 2024b.
References
Aharon, M., Elad, M., and Bruckstein, A. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Pro- cessing, 54(1)), 2006.
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. Adaptive mixtures of local experts. Neural Com- putation, 3(1):79-87, 1991.
Bromley, J., Guyon, I. LeCun, Y., Säckinger, E., and Shah, R. Signature verification using a Siamese time delay neural network. In NIPS, pp. 737-744, 1993.
Ko, S. M.M Lee, S., Jeong, D.-W., Lim, W., and Han, S. Ge- ometrically aligned transfer encoder for inductive trans- fer in regression tasks. In The Twelfth International Con- ference on Learning Representations, 2024.
Caruana, R. Multitask learning. Machine learning. 28:41- 75, 1997.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Cybenko, G. Approximation by superpositions of sig- moidal function. Mathematics of Control, Signals and Systems, 2:303-314, 1989.
Lamperski, A. Neural network independence properties with applications to adaptive control. In 2022 IEEE 61 Conference Decision and Control (CDC), pp. 3365- 3370, 2022.
Eaton, E., desJardins, M., and Lane, T. Modeling trans- fer relationships between learning tasks for improved in- ductive transfer. In Machine Learning and Knowledge Discovery in Databases, pp. 317-332. Springer Berlin Heidelberg, 2008
Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, Mul- tilayer feedforward networks with nonpolynomial ac- tivation function can approximate any function. Neural Networks, 6(6):811-867 1993.
Egozcue, J., Pawlowsky-Glahn, V., Mateu-Figueras, G., and Barcelo- Vidal, C. Isometric logratio transformations for compositional data analysis. In Mathematical Geol- ogy, volume 35, pp. 279-300, 2003.
Mehra, A., Zhang. Y., and Hamm, J. Understanding the transferability of representations via task-relatedness. In

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
The Tiihtt-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024.
Belkhale, S., Park, S., Nair, S., Mirchandani, S., Osa, T., Gupta, T., Harada, T., Matsushima, T., Xiao, T., Kollar, T., Yu, T. Ding, T., Davchev, T., Zhao, T. Z., Armstrong, T., Darrell, T., Chung, T., Jain, V., Vanhoucke, V., Zhan, W., Zhou, W., Burgard, W., Chen, X., Wang, X., Zhu, X., Geng, X., Liu, X., Xu, L.,, Li, X., Lu, Y., Ma, Y. J., Kim, Y., Chebotar, Y., Zhou, Y., Zhu, Y., Wu, Y., Xu, Y., Wang, Y.. Bisk, Y., Cho, Y., Lee, Y., Cui, Y., Cao, Y., Wu, Y., Tang. Y., Zhu, Y., Zhang. Y., Jiang, Y., Li, Y., Li, Y., Iwasawa, Y., Matsuo, Y., Ma, Z., Xu, Z., Cui, Z. J., Zhang, Z., and Lin, Z. Open X-embodiment: Robotic learning datasets and RT-X models. In ICRA, 6892- 6903. IEEE, 2024.
Nichol, A., Achiam, J., and Schulman, J. On first-order meta- learning algorithms. CoRR, Abs//803.02999, 2018.
Oden, J. T. and Demkowicz, L. F. Apply Functional Anal- ysis. CRC Press, third edition edition, 2018.
O' O'Neill, A., Rehman, A., Maddukuri, A., Gupta, A.. Padalkar, A., Lee, A., Pooley, A., Gupta, A., Mandlekar,
A., Jain, A., Tung, A., Bewley, A., Herzog, A., Irpan.
A., Khazatsky, A., Rai, A., Gupta, A., Wang, A., Singh, A. Garg, A., Kembhavi, A., Xie, A., Brohan, A., Raffin, A., Sharma, A., Yavary, A., Jain, A., Balakrishna, A., B., Kim, B., Schölkopf, B., Wulfe, B., Ichter, B., Lu, C., Xu, C., Le, C., Finn, C., Wang, C., Xu, C., Chi, C., Huang, C., Chan, C., Agia, C., Pan, C., Fu, C., Devin, C.. Xu, D., Morton, D., Driess, D., Chen, D., Pathak, D., Shah, D., Büchler, D., Jayaraman, D., Kalashnikov, D., Sadigh, D., Johns, E., Foster, E. P., Liu, F., Ceola, F., Xia, F., Zhao, F., Stulp, F. Zhou, G., Sukhatme, G. S., Salhotra, G., Yan, G., Feng, G., Schiavi, G., Berseth, G., Kahn, G., Wang, G., Su, H., Fang. H., Shi, H., Bao, H., Amor, H. B., Chris- tensen, I., Furuta, H., Walke, H., Fang. H., Ha, H., Mordatch, I., Radosavovic, I., Leal, I., Liang, J., Abou- Chakra, J., Kim, J., Drake, J., Peters, J., Schneider, J., Hsu, J., Bohg, J., Bingham, J., Wu, J., Gao, J., Hu, J., Wu, J., Wu, J., Sun, J., Luo, J., Gu, J., Tan, J., Oh, J., Wu, J.. Lu, J., Yang, J.. Malik, J., Silvério, J., Hejna, J., Booher, J., Tompson, J., Yang, J., Salvador, J., Lim, J., Han, J., Wang, K., Rao, K., Pertsch, K., Hausman, K. Go, K., Gopalakrishnan, K., Goldberg, K., Bymne K. Oslund, K., Kawaharazuka, K., Black, K., Lin, K., Zhang, K., Ehsani, K., Lekkala, K., Ellis, K., Rana, K., Srinivasan, K., Fang, K., Singh, K. P., Zeng, K., Hatch, K. Hsu, K. Itti, L., Chen, L. Y., Pinto, L., Fei-Fei, L., Tan, L., Fan, L. J., Ott, L., Lee, L., Weihs, L., Chen, M., Lepert, M., Memmel, M., Tomizuka, M., Itkina, M., Castro, M. G., Spero, M., Du M., Ahn, M., Yip, M. C., Zhang, M., Ding, M., Heo, M., Srirama, M. K., Sharma, M., Kim, M. J., Kanazawa, N., Hansen, N., Heess, N., Joshi, N. J., Sünderhauf, N., Liu, N., Palo, N. D., Shafi- ullah, N. M. M., Mees, O., Kroemer, O., Bastani, O., Sanketi, R., Miller, P. T. Yin, P., Wohlhart, P., Xu, P., Fagan, P. D., Mitrano, P., Sermanet, P., Abbeel, P., Sundaresan, P.. Chen, Q. Vuong, Q.. Rafailov, R., Tian, R. Doshi, R., Martín-Martín, R., Baijal, R., Scalise, R., Hendrix, R., Lin, R., Qian, R., Zhang, R., Mendonca, R., Shah, R., Hoque R., Julian, R., Bustamante, S., Kir- mani, S., Levine, S., Lin, S., Moore, S., Bahl, S., Dass, S., Sonawani, S. D., Song, S., Xu, S., Haldar, S., Karam- cheti, S., Adebola, S., Guist, S., Nasiriany, S., Schaal, S., Welker, S., Tian, S., Ramamoorthy, S., Dasari, S.,
Pan, S. and Yang, Q. A survey transfer learning. IEEE Transactions Knowledge and Data Engineering, 22 (10):1345-1359, 2010.
Wahid. A.,
Radhakrishnan, A., Ruiz Luyten, M., Prasad, N., and Uh- ler, C. Transfer learning with kernel methods. Nature Communications, 14(1):5570, 2023.
Shotton, J., Glocker, B., Zach, C., Izadi, S., Criminisi, A., and Fitzgibbon, A. Scene coordinate regression forests for camera relocalization in RGB-D images. In Proc. Computer Vision and Pattern Recognition (CVPR). IEEE, June 2013.
Snell, J., Swersky, K., and Zemel, R. S. Prototypical net- works for few-shot learning. In NIPS, pp. 4077-4087, 2017.
Tillmann, A. M. On the computational intractability of ex- act and approximate dictionary learning. IEEE Signal Processing Letters, 22(1), 2015.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Atten- tion is all you need. In NIPS, pp. 5998-5998000 2017.
Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., and He, Q. A comprehensive survey on trans- fer learning. Proceedings of the IEEE, 109(1)43-76, 2021.
10

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
A. Basis Function Regularization
As with all (unregularized) least-squares approaches, the Gram matrix can occasionally be close to singular which makes the matrix inverse numerically unstable during training. Thus, we add small regularization the diagonal of the Gram matrix in the style of Ridge regression, which improves the numerical stability of the inverse. Yet, the regularization causes the eigenvalues to increase, meaning the solution the regularized least-squares problem will be slight under- approximation. Since the basis functions are trained to minimize the error of the approximation, and the approximation is an underestimate, the basis functions constantly grow in magnitude. Thus, we additionally impose an overall regularizer on the norm of the basis functions to be close to to prevent this:
Treg
(11a 1 1)2
Note that we do not need the basis functions to be unit, we only require that they do not grow infinitely in magnitude as training progresses.
B. Universal Function Space Approximation Theorem
Theorem (Restated). Let K C R be compact. Define the inner product (f.9)m := Sx f(x)Tg(z)dd g(z)dz and the induced norm llfl == fi. Let H = {f K + Rm If continuous, ll < ook be a separable Hilbert space. Then, there exists neural network basis functions (01,02,.. ...} such that for any and for any function f E H. there exists N N and R.N such that
llf Σ cêllx < elffllx.
This theorem states that any separable Hilbert space can be arbitrarily well approximated with (a potentially infinite number of) sufficiently large neural network basis functions.
We will build on the universal function approximation theorem for neural networks (Hornik et al., 1989; Cybenko, 1989). We use the results stated in (Leshno et al., 1993):
Theorem (Leshno et al., 1993). Let a be non non-polynomial activation function such that the closure of its discontimuous points is of zero Lebesgue measure. Then for n N, compact C R." f E C(K, [[K R), 8 > there exists neural network
parameters & E N. A BÊx=, B E BC, D E Re such that
sup 1f(z)=((ft)) <8 zeK
where f(x) D D B)).
This theorem states that any continuous, scalar-valued function can be arbitrarily well approximated by sufficiently wide neural network.
Proof. First, we will extend the universal function approximation theorem from continuous, scalar-valued functions to continuous, vector-valued functions. Suppose the functions map to m instead of R, and consider the Euclidean norm. Then there exists a neural network such that [x) f(x)] = (f(z) f(x)) (f(z) - f()) < 8mm for all æ, where the only change to the neural network is that D E R fxm_ This results from the fact that we can treat each output dimension
independently, and each output dimension has arbitrarily small error. Thus the sum of the squared errors is also arbitrarily small.
Since the point-wist error is bounded in Euclidean sense and K has finite measure, then |If flw (f(x))
((e))T(((a)) - dr < Jx 5dd = V& where V is the volume of K. Therefore, the universal approximation theorem for neural networks implies that any function f E H. can be arbitrarily well approximated by a sufficiently wide neural network.

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Next, will leverage the fact that every separable Hilbert space has countable orthonormal basis (Oden & Demkowicz, 2018, Theorem 6.3.1). Let {e1, e2,... be such basis. Then for any f E H.
Ge = lim
No+00
1=1
where 6; = ex). This sequence converges because span e2, e2,... ...) is dense in H (Oden & Demkowicz, 2018, Chapter 6.3). Also, these basis functions are members of H. and are thus continuous.
Let é, be the neural network approximation of with ||é; eal и 2. for some 0. By the universal function approximation theorem (extended to the vector-valued case), such neural network always exists.
Each é; can be decomposed into e; and an error vector d,.
é = e + de.
By definition,
8
11:11x < 20
Let
fN
Σ 1=1
Σ
(i, + dz)
Σ
c
+ Σ
cdj.
1=1
i=1 1=1
Consider lim Ilfn fllm.
lim
Ilfr fllx
lim
Ge +
Σ
c,d,
flln
i=1
N
lim N
G4
fll
Σ
Gdllu
Notice that E addll S Em lci/ [ddelx by the triangle inequality. Also, by the definition of d,, ΣN |ca [dllldll/ < Σ Ice 6 2
Furthermore, for given f of size ||fl| the largest value of |e. is achieved if f = a for some a E IR, i E N, and in this case [c|] = al = ||f]|u. Thus, we can bound |cal $ 11fllu. Therefore,
N
cddllu < fllfll
Σ
2
We can rewrite the limit above as
N cee fl12 +
N
lim 200
fI fllx < lim
N
2
MN gr is geometric series whose limit is 1 as N + xo. Since both components are finite in the limit,
lim IlNx fll lim
Σ qe
f
+ lim ol[fll
6 fl
2
Thus, in the limit of infinite basis functions, the error of the approximation is proportional to the magnitude of the function and arbitrarily small constant To achieve an error of [|
fllx < <|Ifllx with finite number of basis functions
12

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
N, we only need to choose a such that 8 0. Then find sufficiently large N such that I1 flx 5 + (c -) This concludes the proof,
VE > 0, E H, EN EN,CER"
|1/ Σ Geill , < <e|ls|/
Discussion. In the following. we discuss several implications of the proof. First, this proof does not imply that an infinite dimensional function space only needs finite dimensional basis. A infinite dimensional function space unavoidably requires infinite basis functions. However, for any given function in this space, we only need finite number of basis functions achieve arbitrary error. Second, this proof implies that for finite dimensional space, only need finite number of learned basis functions and we can achieve arbitrarily small error for all functions in this function space. Lastly, the requirement that each basis function è has smaller error than the basis function é implies that the size of basis function é will tend to be bigger than the basis function éG-1- other words, increasing basis function precision likely increases neural network width.
C. Choosing an Inner Product
The key design decision for function encoders is the choice of an appropriate and well-defined inner product. Fortunately, many prior works have determined viable inner products for many spaces of interest, such as deterministic function spaces and probability distributions. In general, we define a valid inner product over the output space y, where y is a Hilbert space also, and use the Bochner-Leehesuue integral define the inner product for H = ( X - }}}
(f (f.)x: :0 (())
(10)
Vector addition and scalar multiplication for H are defined pointwise:
(f + ()(x)) = f(x) + g())
(11)
=
(12)
Thus, function encoders can be extended to many function spaces with minimal modifications. Below, we describe three common inner products, along with equivalent forms that improve computational efficiency.
C.1. Euclidean Vectors
The most common output space real vector space y R We use the standard Euclidean vector operations. The resulting inner product for {ʃ: X [m is
(n = / ((x) [(x) g(x)dr.
(13)
C.2. Discrete Probability Distributions
Another common output space is discrete probability distributions over classes. The distributions themselves are un- observed but we instead observe samples from the distributions. We leverage the Hilbert space definitions for discrete probability distributions from Egozcue al. (2003). The D-class probability distribution is defined
DD
[2 L2,
mpl xe > 0. Σ 24 = 1.
(14)
(=1
13

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Note that this implies the probability of any class is always greater than O. This is necessary ensure the space is Hilbert
space. The corresponding vector operations are defined as follows:
2181
D2
DDD
+ y =
(15)
E
Biyi
Ei=1
Ei=1
D
Titl
Or
1
22
D
(16)
1
27
1==1
1
> 1=
where æ and y are in SP. The inner product is defined
sp = Σ log
Tz G(r)
log
Vi G(y)
(17)
where G(z) = (x1 is the geometric mean. Under these definitions, S is a Hilbert space (Egozcue et al., 2003). However, as vector addition requires multiplication and scalar multiplication requires exponentiation, these defini- tions are computationally inefficient.
Instead, we can leverage equivalent definitions that use addition and multiplication instead of multiplication and exponen- tiation, respectively. To do so, we will define equivalent operations in logit space 1 which is simply TR. with modified inner product. We define logit: DD 1D
logit(x)) = (log(r1),..... log()
(18)
and probability :P > S as
e
probabilityy)) =
(19)
Li=1 e0:
These two operations allow us to move from probability space to logit space and vice versa. Note that logit space is D- dimensional while probability space actually (D - -)ddimensional. Effectively, we are working with a quotient space where members of IR." are equivalent if they map to the same probability distribution. Logit space convenient because the vector operations become the standard operations in RD, i.e. they use addition for vector addition and multiplication for scalar multiplication. These operations are significantly cheaper and produce more stable gradients than the equivalent probability space operations. The inner product for logit space 1 is
D
(I, = ( u(v),,
(20)
i=1
where ((x) D El 24 is the mean of æ. Under these definitions, we can work entirely in logit space and only convert to probability distributions when necessary.
Practical Data Considerations practice, we do not observe the probability distribution directly. Instead, we observe one sample point from the distribution. Therefore, we take the maximum likelihood perspective and would assign all probability mass to the observed class and 0 to the others. However, this would violate the requirement that the probability of a given class is non-zero. Therefore, we assign high probability (or positive logit) to the observed class and low probability (or negative logits) to the other classes. The exact value chosen for the high probability (or logit) is hyper- parameter, and affects the model's confidence. For example, choosing 80% probability for the observed class will make the model appear uncertain, while choosing 99.9% probability will make the model outputs appear very certain. Thus, this parameter should be carefully chosen.
Conditional Discrete Probability Distributions Typically, we are concerned with probability distributions conditioned on some input, i.e. assigning class an input. In this case, we simply use the Bochner-Lebesse Lebesgue integral define the inner product for functions mapping to discrete probability distributions. Thus, the function encoder is applicable to conditional, discrete probability distributions that are prevalent in classification problems. Below, we illustrate what this looks like for the CIFAR dataset.
Let the input space X be the set of possible images and the output space y consists of the categories True and Fal se. The classifier for a given object, fobj, determines if image contains the object. other words, the specified object appears
14

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
in the image v., Soby(a True. If not, Jory (2) False. A dataset D {(rr, y;))= consists of pairs of images and True/Falss se labels. Let lpos and kneg be the positive and negative logits described above. Then True pess kney) and False R Knner /pos] In other words, the first dimension of the distribution corresponds to the (log) probability that the object is contained in the image. and the second dimension corresponds to the (log) probability that object is not contained in the image. Thus, the True/Fa a 1 se labels are converted to distributional representations. Lastly, we only need to define the approximate inner product for two classifiers f and h given a dataset of m images and labels:
(. = J h(o)) Ro
1 Σ
M(a)))
In summary, we have defined an inner product for conditional distributions, and can thus directly apply the function encoder algorithm to model conditional distributions.
C.3. Continuous Probability Distributions
Similar to discrete distributions, it is often useful to estimate probability densities in continuous spaces. We leverage the definitions of Egozcue et al. (2006), which is analogous to the discrete distribution case with integration in place of summation. For finite interval (a, b).
A(a, b) = (a,b) + RR > 0 a... log E La,
(21)
The vector operations are
(S ( == f())l())
(22)
Stt ((o)hooddo
(af)(z) = f(z)
(23)
f(oy
Note the similarity to the discrete case. Lastly, the inner iroduct is
(xx log h(x) dr, (f, = I: log G(f) G(h)
[oo
(24)
where G(f)= = exp( 1 (x)) is the geometric mean of the function f. Thus, we can learn probability distributions using a function encoder. Similar to the discrete case, we can improve computational efficiency by using logits. Thus we
define an equivalent logit space I(a, b) == {f : (a.b) b) + R. f E (a, b)}. Vector addition and scalar multiplication for logit space are the traditional point-wise operations. The inner product is defined as
(f. = ff((00 (((())((())((())))) - =(h))dr,
(25)
where again p= = & 1h (x)dr is the mean of f.
The continuous distribution case is mostly analogous to the discrete case, but with few important distinctions. Given d samples from some distribution P(Y = y) defined over a set V, we again wish to use the maximum likelihood perspective. However, while for the discrete case we can iterate all possible classes which were not sampled, in the continuous case there are infinite values in y' not sampled. Therefore, we must also sample "negative" values from the support of the distribution. In practice, this means either defining a uniform sampling function over V, and using these uniformly sampled values as negative examples, or using a dataset of values in y and sampling uniformly from this dataset. The former case makes sense in simpler problems where we can properly define y, for example if y is an interval (0, 1). However, there are many cases where the set y is hard to define, especially in real-world datasets where it is not clear what range of values y can take. In this case, we can uniformly sample from a dataset.
These negative samples are used to define points which are not likely, since they were not sampled. Without them, the empirical distribution is uniform over the positive samples, and therefore the distribution of best fit would be uniform over all of y. The negative samples provide additional information on which samples of y are unlikely. Intuitively, the model
15

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
only has access "positive" examples, which means the simplest solution is uniform model over all of X. The negative samples prevent this behavior.
Analogous to the discrete case, we again assign some large probability mass (positive logits) to the sampled values of ) and some small probability mass (negative logits) to the negative samples.
Lastly, we again need a method to convert from probability densities to logits and vice versa. Analagous to the discrete case, we define logil: A(a, b) - I(a, b) as
logit(f)(v) (f
(26)
and probability: :(aab) A(a,b)
ell
probabilityyh)(()))
(27)
Ja co
D. The Approximate Inner Product and the Distribution of Inputs
The approximate inner product is an important aspect of the function encoder, and Monte-Carlo integration makes many implicit assumptions. For the sake of simplicity, the following will use the I inner product, but holds for a more general Bochner-Lebessue setting too.
Consider the following Mote Carlo approximation.
(f.9) = J f())g(a)dz
(28)
f(x))((a))
(29)
This approximation assumes that the distribution over X' is uniform. If this is not the case, then this approximation will not hold. Suppose the inputs are sampled according to distribution p. Then the correct Monte-Carke approximation with importance sampling is
f(x1)g()))
(f.g)
(30)
p(x.)
Unfortunately, importance sampling is not possible in many settings because we cannot estimate the distribution p. For example, in an image classification setting, we cannot estimate the probability of seeing a given image. In reinforcement learning, the distribution of states extremely difficult to estimate, and furthermore varies depending on the policy.
Instead, we consider the weighted inner product
(f-9); == / f(z)g(z))(()ddz.
(31)
This inner product corresponds specifically to the distribution If we use importance sampling, the approximation is
1 m
((r))((1))p())))
(f.g)p ~ m
(32)
p(z,)
f(z))g(a;).
(33)
Thus, by weighting the inner product by the probability distribution, Put another way, by using (33), we are implicitly redefining the inner to be weighted by the probability distribution. The inner product in (31) effectively assigns more weight to the imput values that are more likely to be sampled. This assumption often reasonable. For example, consider an image classification task. The majority of images appear like white noise, and are not of interest. These images should not affect the inner product calculation for two functions defined on images. The naive Monte Carlo integration automatically reduces the importance of these images in the implicit weighted integral.
longer need estimate the distribution directly.
16

---

fo arg min Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
We may also have the case that the distribution of inputs and the function change simultaneously. For example, this occurs in reinforcement learning where the distribution of states is heavily dependent on the hidden parameters. In this case, still cannot estimate p, and so we must use the approximation in (33). However, by doing we now have a separate inner product defined for each function in the training set. This is true even in the extreme case where the distributions are disjoint, i.e., domain transfer.
Interestingly, even in the presence of distribution shifts, if there exists a set of basis functions which spans all of the training functions, these basis functions are still optimal. Regardless of the inner product, these basis functions would still be able to reproduce the training functions, and therefore minimize the loss. However, if this not the case, for example if the dimensionality of the space is larger than the number of basis functions, this function-specifie inner product is going to affect which solutions are locally optimal. Future work should explore what implications Monte Carlo integration has on performance, and what changes should be made to account for it, if any.
E. The Residuals Method
The residuals method for the function encoder was introduced in Ingebrand al. (2024b). For the sake of completeness, include a small discussion below.
In the residuals method, neural network called the average function is trained to fit the function space. As the average function is single function, it cannot accurately fit the entire function space; Instead, the function it learns is the geometric center of the training functions. Then, the basis functions are trained to minimize the residual error of the average function. See Figure 7. In effect, this allows the function encoder to Iearn an affine subspace of the function space which may not include the zero vector.
Formally, the average function To is solving the following minimization problem:
1 8 71 Σ
llfs. foll2.
(34)
This optimization problem solved via gradient descent on the neural network parameters Then, the basis functions are trained on the residuals. For given function fe, the coefficients are calculated
k
c¹ = arg min
(Je - Fo) - Σ
(35)
cE
H
Lastly, the error of the approximation is evaluated as
4 Fo +
2
L
Σ fst
Σ j=1
(36)
Note the loss in (36) is only used to train the basis functions.
As a byproduct, using the residuals method means the learned function space may not include the zero vector. However, this may sometimes be advantageous. For example, prior work (Ingebrand et al., 2024b) has demonstrated that the residuals method is effective for dynamics prediction in hidden-parameter Markov decision processes. This is because the zero- function, ie. dynamics that return 0 for all transitions, is not typically a valid transition function under any set of hidden parameters.
Another benefit is the inductive bias of the function estimate. In online settings, is often convenient to estimate function from an insufficient amount of data. Using the residuals method is beneficial because its predictions already centered around the average function in the training dataset. For example, consider the problem of estimating the transition dynamics of robot from online data. Initially, there is no example data to compute the coefficients. For the residuals method, setting the coefficients to O will still give a decent estimate of dynamics, since it will be the average function. Furthermore, one could constrain the coefficients to be close to 0 through additional regularization, which would ensure that the function estimate does not deviate too far from the training dataset.
17

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Function Encoder
Function Encoder + Residuals
2D Linear Subspace
2D Affine
Subspace
fo
Figure 7. An Illustration of the Residuals Method. The standard function encoder algorithm Icarns a subspace which most accurately spans the training functions. This necessarily implics that the zero vector representable using coefficients using the residuals method, the average function first shifts the origin of the basis functions to the center of the training data. Effectively, this allows function encoder to instead learn an aftine subspace, or a linear manifold, the Hilbert space. When using residuals method, the zero vector is not necessarily representable, and c = 0 corresponds to the average function in the training set.
0. In contrast, when
F. Architecture
The function encoder algorithm is agnostic to the architecture of the basis functions, so long as they are differentiable. Prior work has shown that using neural ODEs for the basis functions greatly improves the performance of the function encoder on continuous-time dynamics prediction tasks (Ingebrand et al., 2024a). This paper demonstrated that the function encoder inherits the inductive biases of the underlying architecture. We conjecture that this trend will hold for other architectures, such RNNs, GNNs, transformers, etc.
Another consideration is whether to implement the basis functions completely separate neural networks run in parallel, or one neural network with multiple output heads. From computational standpoint, current software is optimized for single neural networks, rather than multiple parallel neural networks, and so one network with multiple output heads is much faster during both training and inference. However, this raises a question of whether one approach performs better than the other. We run ablations to compare the performance of the two approaches. See Figure Note that the total number of parameters is held approximately constant, and so the parallel neural networks are much smaller individually than the single, multi-headed neural network.
The results indicate that the performance is roughly equal for type 1 and type 2 transfer. That is, both methods accurately learn the function space present in the training set. However, parallel basis functions are significantly worse at type 3 transfer. We posit that the reason is basis function diversity. For a multi-headed neural network, a change to one basis function changes all, since the weights are shared. This constantly perturbs every basis function, so a given basis function unlikely learn a simple function, or even to remain stagnant. Consequently, the basis functions are diverse. This diversity improves type transfer because diverse basis functions are less likely to be linearly dependent.
In contrast, parallel neural networks allow individual basis function updates. It is therefore possible for the learning procedure train only few basis functions, while the rest remain unused, if the dimensionality of the space is low. We validate this intuition empirically. See Figure 9 for a visualization of this problem.
While this phenomenon degrades type 3 transfer, it has an interesting side effect that the parallel strategy trains an approx- imately minimal number of basis functions. would therefore be possible to prune the unused basis functions to get a minimal set. This method could be useful in computationally limited settings, where the total number of basis functions and parameters could greatly be reduced after training.
G. Ablations
Number of Basis Functions. We vary the number of basis functions and run FE (LS) on all four datasets. We would expect that using more basis functions improves performance, but there would be diminishing returns. Once the number of basis functions exceeds the dimensionality of the space, performance should stop improving. The experimental results
18

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Train
Type Transfer
Type 2 Transfer
Type 3 Transfer
10a
10a
La Disaanoe Squared
10a
10"
10"
10:
10"3
10-2
10-3
10a
10-3
10-1
10:2
10-1
12500 25000 37500 50000
12500 25000
37500 50000
0
12500 25000 37500 50000
12500 25000 37500 50000 Gradient Step
Gradlent S389
Gradient Step
Gradient Step
Paralie)
Figure 8. Multi-Headed Neural Network Parallel Neural Networks the Polynomial Dataset. For type and type transfer, hoth architecture methods perform roughly equally. However, parallel neural networks perform much worse at type transfer.
Multi-Headed NN
Parallel NNs
3
1.0
0.5
0.0
1-
f(x)
-05
110
1.5
-2
-2.
-2.5
-5.0
-2.5
0.0
2.5
5.0
-5.0
-2.5
0.0
2.5
5.0
X
X
Figure 9. Comparison of Learned Basis Functions on the Polynomial Dataset. Each line indicates one basis function. Left: One multi-headed neural network learns diverse set of basis functions. Right: Parallel neural networks learn small set of diverse basis functions, while the majority lack diversity or interesting structure.
19

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
agree with this hypothesis. See Figures 11, 12, 13, and 14. The figures show the performance the end of training, averaged over the last 50 evaluations. 3 seeds max values over the seeds. We also consider the cost of increasing the number of basis functions with respect to the training time, and find that it is minor. See Figure 10. Therefore, reasonable method to choose & is to simply overestimate the dimensionality of the space. We find that 100 basis functions are enough for many problems.
per quantity of basis functions, and figures show min, median, and
Number of example data points We vary the number of example data points and evaluate the effect on performance. We would expect that using more example data improves performance, but with diminishing returns. The experimental results, shown in Figures 15, 16, 17, and 18, agree with these claims. The results suggest that minimal amount of data is required for the function encoder make accurate predictions, and that the minimal amount depends on the dataset. Similar to the above ablation, the figures show seeds evaluated after training, with their performance averaged over the last 50 evaluation steps.
Phlynomtal
75eenes
20%
201
Relatve
1556
10%
Numbbrr
Drests Flunctions
Number
F'uncttons
Number of Bosts Functions
Number
Basts Prunctions
Figure 10. An Ablation the Number of Basis Functions vs Compute Time. This figures shows percentage increase training time relative to one basis function. The figure shows that the increase in cost is less than 20% for up to 100 basis functions.
20

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Train
Type Transfer
Type Trensfer
Type Transfer
304
(tteer 00kk Seatioat Stepol
101
10:
10
12 Ditance Squaree
10a
10'
1o-1
10-1
107
10'
10-3
10-3
10
30"
50 Number Functions
Stmber Panstiions
Number Functions
Number Functions
Figure II. An Ablation on the Number Basis Functions Polynomial Dataset. This figure shows final performance FE (LS) after training for varying number of basis functions. The performance initially improves the number of basis functions increases, and then levels off.
Type Transfer
Type Transfer
100%
100%
1005
00%
001%
90%
Naatia Coatint
M004
005
80%
00%
00%
30%
30%
50%
20
100
40%
100
Number of Bosts Functions
Number of Bosis Functions
Number of Bosks Functions
Figure 12. An Ablation on the Number of Basis Functions CIFAR Dataset. This figure shows the final performance of FE (LS) after training for varying number of basis functions. The figure shows stcady improvement as the number of basis functions grows, until it levels off around 20 basis functions. This indicates only 20 features necessary for good performance.
Train
Type Transfer
Type: 3 Transfer
12 Bistame Squioeed LAter Stops)
10-1
10%
100
6x 10
3x
20
60
80
100
20 60
100
20
40
100
Funcsious
Number Functions
Fanctions
Figure 13. An Ablation on the Number of Basis Functions 7-Scenes Dataset. This figure shows the final performance of FE (LS) after training for varying number of basis functions. The figure shows steady improvement as the number of basis functions grows, although it begins leveling off around 100 basis functions. This indicates the dimensionality of the space is relatively high.
Train
10a
Type Transfer
Type Transfer
10%
Type 3 Transfer
Shops)
122 Wittane Squard
102
10a.
60
1o!
4 101
10
10
0
75 Number of Basis Punctions
23
50
100
75 Number Baods Panctiota
25
50
160
25
50
75
100
50
75
100
Number of Buots Functions
Number of Busis Functions
Figure 14. An Ablation on the Number of Basis Functions MuJoCo Ant Dataset. (LS) after training for a varying number of basis functions. For type and type transfer, increasing the number of basis functions improves performance until around 50 basis functions, where levels off. Interestingly, the number of basis functions does not greatly affect type 3 transfer performance, suggesting type 3 transfer in this setting is challenging.
This figure shows the final performance of FE
21

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Train
Type Transfer
Type Transfer
Type Transfer
(Aaau 50k Grodinn Simpo)
10"
Squareed
10.2
10-2
102
10-1
10
10"1
10
104
500
500
1000
Ecamppe
Example
Usample
Escemple
Figure 15, An Ablation on the Number of Example Data Points Polynomial Dataset. This figures shows the final performance of FE (LS) after training for varying number of example data points.
100%
Train
100%
Type Transfer
100%
Type Transfer
(Atorr b0lk Grscktat Stopst
00%%
50%
90%
Accutoey
7056
70%
70%
60%
60%
60%
407
50
100
150
200
40%
50
100
150
200
40%
-01
100
150
200
Numbor of Example Data Points
Number of Exampie Data Polnts
Number of Excsuple Date Points
Figure 16. An Ablation on the Number of Example Data Points CIFAR Dataset. This figures shows the final performance of FE (LS) after training for a varying number of example data points.
Train
Type Transfer
Type 3 Transfer
La Dittane Squunded olttr 5MK Gredieett Stope)
102
10
2x 10-1
6 10
0 10
3x
10-4
50
100
150
200
50
100
150
200
50
100
150
20m
Number of Ussemple Data Points
Number Examppe Data Potsts
Number of Usample Data Points
Figure 17. An Ablation the Number of Example Data Points 7-Scenes Dataset. This figures shows the final performance of FE (LS) after training for a varying number of example data points.
Train
Type Transfer
Type Trensfer
3 Transfer
3.4 14x110 16
(Ater S0k Grudiit Steps)
12 Distance Squarcdd
32x 10
10a
3 10
10a
Tu-1
2,8 10
10-2
x 16
102
10-3
244
Number
Data
Number Exampte Points
Number Esample
Number Data Points
Figure 18. An Ablation on the Number Example Data Points MuJoCo Dataset. This figures shows final performance
FE (LS) after training for varying number of example data points.
22

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Positive Examples
Negative Examples
Query Image
Apple
apole
lion
nocket
Axc 88%
Nxc C
bear
beer
bear
cockmeeth
television
bridge
PX 79% PxECi=64
cestle
sannii
bonil
beyyte
PxE ReL Pir 74N
colphin
doishin
dolphin
phin
equarium fish
lamp
kzrd
AxEC 8%% Px1
Figure 19. A Visualization of the CTFAR Dataset. For each class, 200 images are provided as examples, half negative. Positive images (left) images from the class, whereas negative images (center) images from other classes. The model should evaluate whether the query images (right) belong to the class or not. The two images on the right in each column are not evaluated together, and their probabilities unrelated. This figure shows the performance of FE (LS) after 50,000 gradient steps in type 3 transfer. The model gets most of the shown query images correct, although it mistakes skunk for bear.
which are positive
H. Datasets
All figures smooth the plot with moving average of size 40 to reduce noise in the training curves. This is necessary due to some approaches which show inconsistent performance across functions and/or input samples, especially the transformer baseline.
H.1. CIFAR
We use modified version of the CIFAR 100 dataset (Krizhevsky, 2009). Each function is classifier corresponding to a specific class. For example, the "apple" classifier should label images of apples with True and images of anything else with False. 200 examples images are provided to the algorithm for each class. 100 are positive images from the correct class, and 100 are negative images from other classes, selected uniformly at random. The classifier should then classify query images True if they belong the given class and False otherwise. See Figure 19.
Cross Entropy on CIFAR We run the same CIFAR dataset with cross entropy loss functions for all baselines. We report the results in Figure 20. The results are largely unchanged, with FE (LS) and Siamese networks still performing best.
H.2. 7-Scenes
We use modified version of the 7-Scenes dataset (Shotton et al., 2013). We down-sample the images to 40x30 for the sake of compute time. 200 example images and their locations are provided. Then, the model should estimate the location of any new query image. We hold out the "Red Kitchen" scene for type 3 transfer. See Figure 21.
H.3. MuJoCo Ant
We use a modified version of the MuJoCo Ant environment where the lengths of the limbs and the control authority change every episode (Ingebrand et al., 2024a). We use 10,000 episodes of length 1,000 for training. The first 200 transitions are used as example data, and the next 800 are query points. The policy is uniformly random. Type 1 hidden parameters are sampled from 0.5x the default values. Type hidden parameters are sampled from 1.5x to 2x the default values. See
23

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Train
Type Transfer
Type Transter
100%
100%
100%
Aocuces
80%
80%
80%
70%
70%
70%
80%
80%
60%
40%
12500
25000
37500
50000
30%
12500
25000
37500
50000
40%
12500
25000
37500
50000
Gradieut Stey
Grudient Step
Gradieat Step
FE (LS)
AE
TFE
BFB
MAML (n-1)
Stamese Not Protollyp.
(IP)
Trans.
Oracie
(n=3)
Figure 20. Cross Entropy Results CIFAR. The empirical results for each baseline using cross entropy instead of the Hilbert space norm as a loss function. Function encoders, prototypical networks, and Siamese networks cannot use cross entropy, but are included for comparison.
Examples
Queries
Figure 21. A Visualization of the 7-Scenes Dataset. 200 Example images (left) and their locations provided the model. The model then estimates the location of the query images (right). This figure shows the performance of FE (LS) in type 3 transfer.
24

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
Type 1
Type
Figure 22. A Visualization of the Ant Dataset. Type transfer data is generated from robots smaller than the default settings. Type 2 transfer (not shown) linear combinations of the dynamics experienced in type transfer. Type transfer data is generated from robots much larger than the default settings.
Figure 22 for a visualization of how the hidden parameters affect the robot.
Baselines
We implement various baselines to benchmark the function encoder against. Furthermore, we use the Hilbert space distance function as the loss function. For the non-classification datasets, this corresponds to scaled version of mean squared error. For the classification task, we use the discrete distribution-ban appendix. Below, we describe each baseline in detail. For datasets using images, an additional CNN is used to transform the image latent representation. As typical, the CNN is trained as an additional component using the algorithm's standard loss function. Each algorithm is a predictor f X (X' y)m dataset Dj {æ;, fr()))) m to ÿ. For most algorithms, need not be fixed. Where obvious, we omit reference to model parameters. All baselines use approximately one million total parameters.
inner product for the main results, and cross entropy the
which maps from query point and
FE (IP) The function encoder from Ingebrand et al. (2024b) which uses (4) to compute the coefficients of the basis function. This approach does not ensure orthonormality of the basis functions explicitly, but a non-orthonormal basis has high loss.
FE (LS) The modified version of the function encoder introduced in this paper. This approach uses (7) compute the coefficients, and regularizes the basis functions to prevent exponential growth.
AE An auto encoder which has two components, the encoder and the decoder. The encoder is a learned function e X' x + & The encoder is used to generate a latent representation using 2 == 1 Σm e(x;, / (x;)). The decoder is learned function d X R y. The decoder is used estimate query points using fax D2y) d(x, 2). The full formula therefore f(x D5x) = d(x, m m e(x,, (x:))) Both components are neural networks trained on end-to-end loss.
Trans. A encoder-decoder transformer (Vaswani et al., 2017). This transformer has four components: An example encoder Cen, a query encoder eq. a transformer f, and decoder d. The example encoder represents each example point as 25th := (2,, fx:;).. ). The query encoder represents queries as 2 : (z).. The transformer accepts two input sequences, one to each side of the transformer, and outputs a latent representation, zout = (((2(" )m =9). Lastly, the decoder converts the latent representation to the output space, f(x, D,r) = d(zout). The full equation is f(x, Diz) d((((eez (#, [xx)) e, (x))). All components are trained on end-to-end prediction loss. Positional encodings are not used because the data unordered, and empirically they do not improve performance for unordered data.
TFE- Transformer functional encodings. This model uses a decoder-only transformer to output the coefficients of basis functions. This model has four components: an example encoder e, a transformer t, and decoder d, and a set of basis functions g. The encoder represents the example points 2, == e(v,, (a:)). The transformer converts the
25

---

Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces
set of example point representations an output representation, Fout == (((2))=))) representation the coefficients of the basis functions, c = d(zout). Lastly, the estimate is traditional linear com- bination of basis functions, f(z, p ((a)).c. c. The full formula is f(x, Dr) = g(x)" ((x))))) end-to-end loss.
1). The decoder converts the latent
All four components trained
The oracle is learned function fa which is provided with hidden information H. In the Polynomial dataset, is the coefficients of the polynomials. the CIFAR dataset, H is a one-hot encoding of the class. In the 7- Scenes dataset, H a one-hot encoding of the scene (the sequence, specifically). Lastly, in the Ant dataset, H is a normalized version of the hidden environment parameters (e.g. the length of the robot's limbs). Thus the estimate is f(z, H) fo (z, H), and this neural network is trained end-to-end.
Oracle
BFB- A naive brute force implementation with basis functions. BFB is two neural networks, a coefficient calculator b (X x y) + & and basis functions g 0: X' y b a neural network which takes all of the example data as input and outputs the coefficients of the basis functions, c = b(fz,, f(x))= Thus it has the form f(x Drr) = b((22, (m:)) g(x). It is trained end-to-end.
BF A naive brute force implementation. BF is single neural network which takes as input all of the example data and the query point, and outputs the estimated value. Thus it has the form f(x, 1rr) = fa(z, fr. (x.)) It is trained end-to-end.
MAML A meta learning algorithm for few-shot adaptation. This algorithm consists of a single model fo X y The inference procedure consists of using the example dataset Dr to fine-tune Se. and then using this fine-tuned model to estimate the query points. This is known as the inner training step. During training, gradients are back- propagated through the inner training step to update the initial model weights A. This is know as the outer training step. MAML trains the model fa to be easily fine-tuned for a given source target task. An additional hyper parameter, n, the number of internal gradient steps. For more information on this algorithm, see (Finn et al., 2017). MAML additionally required hyper-parameter tuning, especially the internal learning rate. We find that the best internal learning rate for type of transfer is not typically the best for the other types of transfer, and sometimes MAML may even be unstable, seen in the Ant experiments.
The following ad-hoc algorithms are only applicable to classification problems, in this case the CIFAR dataset.
Siamese networks Uses triplet loss to learn latent space which maximizes the distance between different classes. The chosen class for a query image is the label of the most similar image in the example dataset. See Bromley et al. (1993) for more information.
Prototypical networks- Learns latent representation of each image. The prototype of class is the average of the representations of the corresponding images, and the label for a query image is the class corresponding the nearest prototype in the latent space. See Snell et al. (2017) for more information.
Connections between Siamese networks and Hilbert spaces. Siamese networks performed well on the few-shot clas- sification problem. We believe this is because there deep connections between Siamese networks and Hilbert space theory. Siamese networks use a contrastive loss to minimize the distance between inputs belonging to the same category and to maximize the distance otherwise. If you consider the network as a set of basis functions, then the mean output from one category can be interpreted as a mean embedding from kernel literature. Consequently, the difference in these mean embeddings is analogous to the maximum mean discrepancy. Therefore, maximizing the distance between individual embeddings from different classes is similar to maximizing the difference between mean embeddings, and thus Siamese networks are potentially learning basis functions which maximally distinguish between the distributions corresponding to each category.
26


Reviews:
[{"review_id": "JOfkX0IRch", "reviewer": "Reviewer_wRbf", "summary": "The authors define three types of inductive transfer in Hilbert spaces: interpolation within the convex hull, extrapolation to the linear span, and extrapolation outside the span. They propose to learn neural network basis functions, named as function encoders, that can represent any function in this space. Specifically, they employ a least-squares formulation to compute the coefficients of the basis functions. The paper support their method with a universal function space approximation theorem, showing that any function in a separable Hilbert space can be approximated arbitrarily well if enough basis functions are used. They validate their approach through experiments on several benchmarks, including polynomial regression, CIFAR image classification, camera pose estimation, and dynamics estimation on robotic data.\n\n## update after rebuttal\nI appreciate the authors' detailed response and the additional experiments addressing the computational cost of increasing $k$, as well as the discussion on basis function regularization and domain shifts. Based on these clarifications, I will maintain my overall score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. I would suggest some additional experiments on the computational cost of increasing k.\n\nUsing the worst-case strategy, the compute time for a forward pass is linear in k if you compute the basis functions sequentially on a single thread. Beyond the cost for the forward pass, the compute time for the Gram matrix inverse is cubic in k, but is typically insignificant relative to the forward pass. \n\nHowever, in practice, the compute time for a forward pass is approximately constant with respect to the number of basis functions since each can be run simultaneously via GPU parallelization. Therefore, there is little overhead for increasing k. Empirically, the percentage increase in training time of k=100 relative to k=1 is the following:\n\nPolynomial - 14.4 % \n\nCIFAR - 8 %\n\n7 Scenes - 11.5 %\n\nAnt - 17.8 %\n\nIndeed, the overhead is minor even though we increase k by two orders of magnitude. We will include a graph comparing training time for different numbers of basis functions in the appendix. \n\n2. It would be better if there are some experiments regarding a reasonable method for choosing k from the dataset adaptively.\n\nWe agree that exploring adaptive methods to reduce the number of basis functions (either during training or after training) is an interesting area for future research. We anticipate that L1 regularization can be used to impose sparsity during training, or that we could \"constructively\" train a set of basis functions--adding new basis functions periodically until we encounter diminishing returns. However, in most cases the overhead of choosing a large k is sufficiently small such that often the best solution is to simply choose a large value of k at the start. \n\n3. It would be beneficial to explicitly write out the details of Section A Basis Function Regularization, e.g., the form of additional regularizer on the diagonal of Gram matrix.\n\nWe will improve the discussion of regularization in the final version by incorporating the regularization loss shown in Algorithm 1 as part of the main text. \n\n4. ... It would be beneficial to discuss potential extensions to their framework under domain shifts.\n\nThis is an excellent point. In practice, we only require that the input space X is the same for all datasets. However, the distribution between tasks can be different. If the distributions are entirely disjoint, this is a special case of the weighted inner product described in Appendix D. So the algorithm is still applicable, but implicitly the inner product’s definition changes between domains. Note that both image-based experiments in this paper include domain transfer in addition to function space transfer, since the distribution of images is different for different classes/scenes. Implicitly, our use of least squares calibrates the function estimate to new domains. We will improve the discussion in Appendix D to highlight these details.\nFurthermore, analyzing the interplay between domain shifts, inner products, and least squares is an interesting direction for future work. \n\n5. Given that Section G suggests that the optimal k varies across datasets, what is the impact of larger k on computational cost across all tasks? Is there a reasonable method to balance empirical performance with computational efficiency?\n\nPlease see our response to question 1 above for the computational cost of choosing k and the scaling across tasks. However, increasing k does not reduce prediction accuracy. Therefore, a good naive strategy is to simply choose large k, which maximizes performance while only mildly increasing compute time. As mentioned in our response to question 2 above, we anticipate that we could \"constructively\" train a set of basis functions if computational efficiency is critical.\n\n\n6. How does learning neural network–based basis functions provide advantages over fixed basis methods... It would be helpful to discuss the limitations in fixed basis methods and explain how a learned representation can adapt to more intricate data structures in challenging transfer scenarios.\n\nWe anticipate that kernel methods and dictionary learning are amenable to type 2 & 3 transfer due to the use of least squares. However, the key advantage of function encoders over kernel methods is that the Gram matrix inverse in kernel methods scales cubically with the amount of data, whereas the scaling is cubic with the hyper-parameter k for function encoders.  Additionally, we note that function encoders avoid the issue of pre-specifying a kernel, and therefore may learn basis functions which are specialized to the given problem. On the other hand, dictionary learning is effectively learning a discretized version of basis functions evaluated at a fixed mesh of input locations, and so cannot be queried at new points, which makes it intractable for domains like robotics where we can’t measure the same state twice. In contrast, the function encoder is queryable at any new point due to the use of neural networks.", "claims_and_evidence": "The paper’s claims are well supported by both theoretical analysis and experimental results. I did not encounter any obviously problematic or unclear statements in the claims.", "methods_and_evaluation_criteria": "Yes. The authors evaluate their approach on a wide range of datasets, from simulated polynomial regression to real-world tasks like CIFAR image classification, camera pose estimation on the 7-Scenes dataset, and dynamics estimation with MuJoCo. This diversity in experiments demonstrates the versatility of the method across different transfer learning scenarios, especially in cases requiring extrapolation. Some issues are noted in the question section.", "theoretical_claims": "I checked the detailed proof provided for Theorem 1, which establishes a universal approximation guarantee for the function encoder framework. Overall, the proof is correct in its high-level idea and is built upon classical universal approximation results.", "experimental_designs_or_analyses": "I checked the Section 4 in the paper. Overall, the experimental designs are sound and cover a diverse set of tasks. However, addressing the points above could further solidify the claims and help clarify the practical limitations of the proposed approach.", "relation_to_broader_scientific_literature": "The paper’s contributions are closely connected to kernel methods. The proposed approach shares similar spirit to that in kernel learning, where functions are represented in a RKHS, but here the authors replace fixed kernels with learned neural network basis functions. This provides a more flexible representation that can adapt to complex data structures.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper presents a geometric perspective for transfer learning, distinguishing between different types of transfer. Overall, the manuscript is well-structured, making it easy to follow the progression from theoretical motivation to algorithmic details and experimental validation. The experimental comparisons are comprehensiv, covering a broad spectrum of tasks, enhancing method's credibility.\nSee Questions part for weakness.", "other_comments_or_suggestions": "1. I would suggest some additional experiments on the computational cost of increasing $k$. It would be better if there are some experiments regarding a reasonable method for choosing $k$ from the dataset adaptively.\n2. It would be beneficial to explicitly write out the details of Section A Basis Function Regularization, e.g., the form of additional regularizer on the diagonal of Gram matrix.", "questions_for_authors": "1. The method assumes that the multiple source and target domains are identical, i.e., $D_{S_1}=\\cdots=D_{S_n}=D_T$, which is a strong assumption in practice. Given that many real-world scenarios involve some degree of domain shift or weak task relatedness, what specific aspects of the FE framework could be adapted to handle cases when there exists some $j$ such that $D_{S_j}\\ne D_T$? It would be beneficial to discuss potential extensions to their framework under domain shifts.\n\n2. Given that Section G suggests that the optimal $k$ varies across datasets, what is the impact of larger $k$ on computational cost across all tasks? Is there a reasonable method to balance empirical performance with computational efficiency?\n\n3. How does learning neural network–based basis functions provide advantages over fixed basis methods (e.g., kernel method or dictionary atoms) in capturing complex nonlinearities and enabling robust extrapolation, particularly for Type 2 and Type 3 transfer? It would be helpful to discuss the limitations in fixed basis methods and explain how a learned representation can adapt to more intricate data structures in challenging transfer scenarios.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F05P3oalux", "reviewer": "Reviewer_qZy4", "summary": "This paper studies inductive transfer learning where the source tasks and the target task share the same domain $(\\mathcal{X}, \\mathbb{P}(\\mathcal{X}))$ and output space $\\mathcal{Y}$, but have different predictors $f:\\mathcal{X}\\to\\mathcal{Y}$ that lie in some Hilbert space. It characterizes the difficulty of transfer by the geometry of the source and target predictors. Drawing inspirations from prior works on function encoders, it proposes to learn a set of basis functions that approximately span the Hilbert space of predictors, so as to adapt to future tasks without retraining the model. To train function encoders, it applies least squares which is more efficient than the previously used inner product method. Numerical experiments on polynomial regression, image binary classification, position estimation from image, and robot state prediction demonstrate the effectiveness of the algorithm.\n\n## Update after Rebuttal\nI thank the authors for the detailed response. I still think that the work [1] diminishes the novelty of the methodology of this work. For the geometric view of transfer, the authors explained with the example of classifying horses and classifying lung cancer, but I believe that one can tell the two tasks are highly dissimilar without the Hilbert space framework. The geometric categorization, while intuitive, does not seem to provide any fundamentally deeper understanding of the problem, and does not really give insights into the development of the methodology. Therefore, given the limited novelty of the theoretical framework and the methodology, I choose to maintain my score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Originality of the proposed approach. The least squares method for function encoder training seems to already exist in the literature. See \"Essential References Not Discussed\" for more details.\n\nWe thank the reviewer for pointing out the referenced work in [1], and we have added the reference to our final manuscript. To clarify the originality of our contribution in relation to [1], the development of the referenced work occurred concurrently with our manuscript. Indeed, the referenced work [1] uses the least-squares formulation and implementation from this work, despite appearing first. Leaving out [1] as a reference was an oversight and has been corrected.\n\nIn any case, while [1] addresses operator learning and PDE modeling, this paper is concerned with function space approximation. We introduce a geometric characterization of transfer, we  define the necessary regularization for the LS approach to converge, we generalize the inner product definition, we provide a universal function space approximation theorem, and we compare against SOTA baselines such as meta learning and transformers on various benchmarks, including a classification task that function encoders have not previously been able to handle. Thus while both approaches use LS, these two papers are distinct.\n\n2. Significance of Theorem 1. While the theorem is correct, I feel that it is mathematically trivial given the universal approximation theorem of neural networks...\n\nWe agree that, in general, universal approximation theorems are fundamentally existence proofs. However, they play a foundational role in justifying neural networks as universal approximators of continuous functions, and thus have significant practical value. Nevertheless, we agree that universal approximation theorems may have limited use in determining the quantitative computational requirements or error bounds, which vary from problem to problem and depend on the training data. \n\n3. More importantly, it offers minimal insights into the transfer learning problem. Indeed, the basis functions are learned using source tasks, and whether the basis functions can well approximate the target task should depend on the similarity and the diversity of the tasks. This important problem is not covered by Theorem 1.\n\nFrom our geometric categorization of transfer, we know that the difficulty of approximating a new task depends on its relation to the source tasks, e.g. whether it lies in the span of the source tasks and if not, the distance from the span of the source task as measured by the Hilbert space norm. On the other hand, since the function encoder trains basis functions, its approximation error is proportional to the distance between the span of the basis and the target task. Therefore, the approximation error of a function encoder on a new task aligns with the expected difficulty of that target task according to the geometric categorization of transfer. UAT is useful because it guarantees the existence of a function encoder which spans the source tasks. So assuming that the training procedure converges, we expect the function encoder’s approximation error to be conceptually equivalent to the difficulty of transfer suggested by the geometric view. We have added a paragraph highlighting this connection at the end of section 3. \n\n4. I do not see how the geometric view can give us more insights into the difficulty of the transfer learning problem, or how it can used to facilitate transfer learning in specific scenarios.\n\nThis is a great question. The geometric view of transfer provides a framework for categorizing transfer problems. Consider classifying images by attributes, such as identifying horses or stripes. If you want to detect zebras (striped horses), this likely involves type 1 transfer, where existing methods typically perform well. However, then classifying lung cancer from X-rays isn’t a linear combination of previous attributes, indicating type 3 transfer, making it significantly harder despite the apparent similarity between tasks. Thus, the geometric view helps assess the difficulty of transfer problems and suggests suitable approaches; for instance, our results indicate transformers aren’t effective for type 3 transfer.\n\n5. Do the authors have a principled way of choosing an appropriate k?\n\nPlease see our response to reviewer wRbf for questions 1 and 2 for additional details. The overhead for choosing a larger k is relatively small. \n\nFurthermore, the ablations show that k=100 is enough for all of the problems in this paper; This aligns with prior work. If k is chosen to be less than the dimensionality of the space, then the algorithm will likely learn the k principle basis functions, since doing so would minimize the loss. For example, see the ablation on k for the CIFAR dataset, where most of the prediction accuracy is due to 20 basis functions, and using additional basis functions provides only diminishing returns.", "claims_and_evidence": "The claims made in the paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method is conceptually simple and intuitive. It is tested on four different datasets covering different applications.", "theoretical_claims": "I have checked that Theorem 1 and its proof are correct.", "experimental_designs_or_analyses": "The experiment procedures and results in the paper are clearly described, and the results look convincing.", "relation_to_broader_scientific_literature": "This paper proposes a general approach for transfer learning and demonstrates its effectiveness for a variety of tasks. Transfer learning is still a widely open problem in machine learning, and is important for various applications including image classification and robotics.", "essential_references_not_discussed": "A key methodological contribution of this paper is to replace the inner product method in the function encoder training procedure by the least squares method. However, it seems that the least squares method is not new, as it already exists in the paper [1] which proposes the same method for the same purpose of overcoming the orthogonality assumption. (See, e.g., Figure 2 in that paper.)\n\n**Reference**\n\n[1] Ingebrand et al. (2024). Basis-to-basis operator learning using function encoders. https://doi.org/10.1016/j.cma.2024.117646", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is well written, and the ideas are clearly explained.\n\n2. The proposed method is conceptually simple and intuitive.\n\n\n**Weaknesses:**\n\n1. Originality of the proposed approach. The least squares method for function encoder training seems to already exist in the literature. See \"Essential References Not Discussed\" for more details.\n\n2. Significance of Theorem 1. While the theorem is correct, I feel that it is mathematically trivial given the universal approximation theorem of neural networks. In particular, the result is qualitative; it only states that any predictor in the Hilbert space can be approximated using sufficiently many basis functions. More importantly, it offers minimal insights into the transfer learning problem. Indeed, the basis functions are learned using source tasks, and whether the basis functions can well approximate the target task should depend on the similarity and the diversity of the tasks. This important problem is not covered by Theorem 1.\n\n3. Implications of the geometric view of transfer. The paper divides the transfer learning problem into three types, according to whether the target predictor lies in the convex hull or the linear span of the source predictors. The geometric characterization is intuitive, and the numerical experiments confirm the three types of problems have different difficulty levels. However, beyond these points, I do not see how the geometric view can give us more insights into the difficulty of the transfer learning problem, or how it can used to facilitate transfer learning in specific scenarios.", "other_comments_or_suggestions": "The notations for domain ($\\mathcal{D}$) and dataset ($D$) use the same letter \"D\", which can be a bit confusing.", "questions_for_authors": "1. Ablation studies in the paper show that the choice of the number $k$ of basis functions can have a significant impact on the performance of the algorithm. However, in practice, the appropriate $k$ is unknown beforehand. While the experiments suggest that $k$ can be chosen as the dimensionality of the space, it can be too large and make the algorithm computationally expensive. Do the authors have a principled way of choosing an appropriate $k$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ur6YGwZkKg", "reviewer": "Reviewer_PQ3a", "summary": "This work provides a geometric characterization of transfer learning in the form of 3 types of inductive transfers presented in a Hilbert space. A type 1 transfer is when the predictor of the target task is the Convex Hull of the source predictors, type 2 is when the target is the linear span of the source predictors, and type 3 is when the target sits outside the linear span.\n\nThe paper uses the theory of function encoders to propose a method that achieves transfer learning across all 3 types. This is done by providing a closed-form least-squares solution to fit the target predictor (i.e. L2 projection onto the linear span for source predictors).\n\n A detailed study  of the proposed method against other transfer learning benchmarks is presented, showcasing how the method can match and outperform SOTA in various settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. I think the paper can benefit from additional experiments in other domains, e.g. text, structured prediction.\n\nFunction encoders are applicable in many domains. However, there are additional theoretical questions on how to define an appropriate inner product and geometric characterization of task relatedness for text prediction and structured representations that would need to be addressed first, and are thus beyond the scope of the current work. \n\n2. The main theorem in the paper is not very strong IMO, as it mainly applies in the limit of infinite basis functions.\n\nPlease see our response to reviewer qZy4, question 2. Universal approximation theorems are existence proofs, meaning they provide justification that the proposed approach could achieve arbitrary accuracy given enough resources. Practically, Theorem 1 guarantees the existence of a function encoder which spans the source tasks, which is necessary for the types of transfer we characterize.\n \n3. Use of H for both the Hilbert space and Convex Haul (even with different fonts) can be confusing, especially since predictors live in both of them.\n\nThank you for this comment, we will change the notation for convex hull in the final version.\n\n4. When comparing ad-hoc methods (Siamese and prototypical networks) we see comparable performance to the proposed method in multiple settings. Do you have any insights into how these ad-hoc methods achieve the type 2 or 3 transfer in the cases they can?\n\nSiamese networks use a contrastive loss to minimize the distance between inputs belonging to the same category and to maximize the distance otherwise. We believe there are deep theoretical connections between this approach and maximum mean discrepancy (MMD) from kernel methods, where you could view the mean output across a category as a mean embedding, and the distance between embeddings as the MMD. Therefore, maximizing the distance between individual embeddings is somehow similar to maximizing the difference between mean embeddings, and thus Siamese networks are effectively learning basis functions which maximally distinguish between the distributions corresponding to each category. Prototypical networks do something similar, but instead work with the mean embedding directly, using the closest mean embedding to a given input as the correct category. Therefore we believe these approaches have unrecognized theoretical connections to Hilbert space theory, which potentially explains their performance.", "claims_and_evidence": "The claims in the paper are mostly supported by theoretical and empirical analysis.", "methods_and_evaluation_criteria": "IMO, L2 loss and accuracy metrics are reasonably chosen based on the setting.", "theoretical_claims": "The proof for the theoretical claims (Theorem 1) is provided in the appendix. Insights into the design of the inner product for other spaces (e.g. probably spaces) are also provided in the appendix.", "experimental_designs_or_analyses": "The experimental domains include toy models, small-ish image tasks (CIFAR 100, 7-Scenes) and the MuJoCo Ant (dynamic control). I think the paper can benefit from additional experiments in other domains, e.g. text, structured prediction.", "relation_to_broader_scientific_literature": "The paper compares and contrasts with prior work in great detail in section 1.1. Specifically there are comparisons with common approaches in transfer learning, meta-learning and kernel methods, in terms of sample and computational complexity and how they differ in transfer efficiency.", "essential_references_not_discussed": "Most relevant and related work is referenced in the paper.", "other_strengths_and_weaknesses": "The main premises of this work are interesting and insightful. The use of L2 projection onto the span of source predictors is novel to this work (to the best of my knowledge), and experiments suggest it outperforms the inner product method (Ingebrand et al., 2024b).\n\nThe main theorem in the paper is not very strong IMO, as it mainly applies in the limit of infinite basis functions.", "other_comments_or_suggestions": "Use of H for both the Hilbert space and Convex Haul (even with different fonts) can be confusing, especially since predictors live in both of them.", "questions_for_authors": "When comparing ad-hoc methods (Siamese and prototypical networks) we see comparable performance to the proposed method in multiple settings. Do you have any insights into how these ad-hoc methods achieve the type 2 or 3 transfer in the cases they can?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a geometric framework for transfer learning in Hilbert spaces, identifying three transfer types: interpolation, extrapolation within span, and beyond. The authors propose a function encoder trained via least-squares, backed by a universal approximation theorem. Experiments show it outperforms transformers and meta-learning across diverse benchmarks and transfer scenarios. The reviewers appreciate the simplicity and clarity of the proposed method. They also pointed out a few limitations in the theory and experiment. The authors should revise the paper according to the reviews and rebuttals.
Decision: Accept (poster)