Learning Functional Transduction
Authors: Mathieu Chalvidal
Thomas Serre
Rufin VanRullen
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Meta-learning
Neural Operators
Kernel methods
In-context learning
TL;DR: We introduce a meta-learned regression program that can approximate any target function of general kernel Banach spaces from any finite collection of input/output examples in a feedforward way.

Abstract: Research in statistical learning has polarized into two general approaches to perform regression analysis: Transductive methods construct estimates directly based on exemplar data using generic relational principles which might suffer from the curse of dimensionality. Conversely, inductive methods can potentially fit highly complex functions at the cost of compute-intensive solution searches. In this work, we leverage the theory of vector-valued Reproducing Kernel Banach Spaces (RKBS) to propose a hybrid approach: We show that transductive regression systems can be meta-learned with gradient descent to form efficient _in-context_ neural approximators of function defined over both finite and infinite-dimensional spaces (operator regression). Once trained, our _Transducer_ can almost instantaneously capture new functional relationships and produce original image estimates, given a few pairs of input and output examples. We demonstrate the benefit of our meta-learned transductive approach to model physical systems influenced by varying external factors with little data at a fraction of the usual deep learning training costs for partial differential equations and climate modeling applications.

Learning Functional Transduction
Mathieu Chalvidal
Thomas Serre
Capital Fund Management Paris, France
Camey Institute for Brain Science Brown University, U.S.
mathieu. whw.eeaallaaaiaaaaaaaaaaaaa com
Shomas_serreeuee edu
Rufin VanRullen
Centre de Recherche Cerveau & Cognition CNRS, Universite de Toulouse, France rufin. vanrullen@cnss......
Abstract
Research in statistical learning has polarized into two general approaches to perform regression analysis: Transductive methods construct estimates directly based on exemplar data using generic relational principles which might suffer from the curse dimensionality. Conversely, inductive methods can potentially fit highly complex functions the cost of compute-intensive solution searches. In this work, we leverage the theory of vector-valued Reproducing Kernel Banach Spaces (RKBS) propose hybrid approach: We show that transductive regression systems can be meta-learned with gradient descent to form efficient in-context neural approximators of function defined over both finite and (operator regression). Once trained, our Transducer can almost instantaneously capture new functional relationships and produce original image estimates, given a few pairs of input and output examples. We demonstrate the benefit of our meta- learned transductive approach model physical systems influenced by varying external factors with little data at fraction of the usual deep learning training costs for partial differential equations and climate modeling applications.
spaces
Introduction
Transduction vs. induction In statistical learning, transductive inference (Vapnik, 2006) refers to the process of reasoning directly from observed (training) cases new (testing) cases and contrasts with inductive inference, which amounts to extracting general rules from observed training cases to produce estimates. The former principle powers some the most successful regression algorithms benefiting from straightforward construction properties, from k-Nearest Neighbors (Cover and Hart, 1967) Support Vector Machines (Boser al., 1992) or Gaussian Processes (Williams and Rasmussen, 1995). In contrast, deep learning research has mostly endeavored to find inductive solutions, relying on stochastic gradient descent faithfully encode functional relationships described by large datasets into the weights of neural network. Although ubiquitous, inductive neural learning with gradient descent compute-intenn intensive, necessitates large amounts of data, and poorly generalizes outside of the training distribution (Jin et al., 2020) such that a slight modification of the problem might require retraining and cause "catastrophic" forgetting of the previous solution (McCloskey and Cohen, 1989). This may be particularly problematic for real-world applications where data has heterogeneous sources, or only a few examples of the target function are available.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

V3
V8
Oa
01
03
space x
(1²) (,(x, t) C((O, 1J2, obtained with the Transducer model To but conditioned, each row, by different dataset (Do, )iga during feedfoward computation. Each underlying operator O, corresponds to different (defined in Sec. 5.1) with spatially varying advection, diffusion, and reaction parameters unseen during training. and functions (uy)sss correspond initial conditions. While usual neural regression approaches learn single target function (one row), model learns approximate instantaneously infinity of them.
Figure 1: Batches of functional images To (Do) (v;) )
paarctionddiffssioo-raattiiioiii equation
Meta-lcarning functional regression In this work, we meta-learn a regression program in the form of a neural network able to approximate an infinity of functions defined on finite or infinite- dimensional spaces through transductive formulation of the solution based on the representer theorem. Namely, our model is meta-trained to take input any dataset Do of pairs (v,, O(v)))s of some target function O together with a query element o' and produces directly an estimate of the image O(v'). After meta-training, our network is able perform regression of unseen operators O' from varying dataset sizes in a single feedforward pass, such that our model can be interpreted as performing in-context functional learning. order to build such a model, we leverage the theory of Reproducing Kernel Banach Spaces (RKBS) (Micchelli and Pontil, 2004; Zhang, 2013; Lin et al., 2022) and interpret the Transformer's (Vaswani et al., 2017) attention mechanism as parametric rector-valuee reproducing kernel. While kernel regression might be plagued by the "curse of dimensionality" (Bellman, 1966; Aggarwal et al., 2001), we show that our meta-learning approach can escape this pitfall, allowing, for instance. to perform instantancous regressions spaces of operators from a few example points, by building solutions to regression problem instances directly from the general reproducing kernel associated with such spaces.
Contributions o We introduce the Transducer, a novel meta-learning approach leveraging reproduc- ing kernel theory and deep learning methods to perform instantaneous regression of an infinity of functions reproducing kernel spaces.
Our model learns an implicit regression program able identify, in single feedforward pass, elements of specific functional spaces from any corresponding collection of output pairs describing the target function. Such ultra-fast regression program, which bypasses the need for gradient-based training, is also general and can be applied to functions either defined on finite dimensional spaces (scalar-valued function spaces) or infinite dimensional spaces (function-valued operator spaces).
In particular, demonstrate the flexibility and efficiency of our framework for fitting cunction-valusee operators in two PDEs and one climate modeling problem. We show that transductive approach allows for better generalization properties of neural operator regression, better precision when relevant data is available, and can be combined with iterative regression schemes that holding potential to improve neural operators applicability.
expensive for previous inductive approaches, thus
To the best of our knowledge, our proposal is the first to marry vector-valued RKBS theory with deep meta-learning and might also shed new light on the in-context learning abilities observed in deep attentional architectures.

---

2 Problem formulation
Let V and U be two (finite or anfiite-dimensionall Banach spaces, respectively referred as the input and output space, and let B a Banach space of functions from V/ to U. We also note L(44, B) (resp. L(4)) the set of bounded linear operators from U to B (resp. to itself). We consider the meta-learning problem of creating a function T able to approximate any functional element O in the space B from any finite collection of example pairs Do = {(,,, u:) v; E V. u, = prominent approach in statistical learning is empirical risk minimization which consists in predefining a class B = B of computable functions from V to U and subsequently selecting model O as minimizer (provided its existence) of a risk function L: B D TR:
A
T(Do) E argmin ((Ó,Do)
õeB
For instance, the procedure consisting in performing gradient-basee optimization of objective (I) over a parametric class B of neural networks defines implicitly such function T. Fundamentallyy this technique works by induction: It captures the statistical regularities of a single map O into the parameters of the neural network Ô such that Do is discarded for inference. Recent examples of gradient-based optimization of neural networks for operator regression (i.e when V and 2/ are infiiteediimessional) are DeepOnet (Lu et al., 2019) or Fourier Neural Operator (FNO) (Li et al., 2020a). As previously discussed, for every regression problem instance, evaluating T with these approaches requires a heavy training procedure. Instead, we show in this work that for specific spaces B, we can meta-learn parametric map To that transductively approximates (in a certain functional sense) any target function O B given a corresponding dataset Do such that:
Yo E V, (()))(b) (( ...t),.......... Un O( ~ O(v)
(2)
3 Vector-valued Reproducing Kernel Banach Space regression
In order to build To. we leverage the structure of Reproducing Kernel Banach Spaces (RKBS) of functions B and combine it with the universal approximation abilities of deep networks. As will see in the experimental section, RKBS are very general spaces occurring in wide range of machine learning applications. We start by recalling some elements of the theory vector-valued RKBS developed in Zhang (2013). Namely, we will consider throughout uniform Banach spaces S (such condition guarantees the unicity of a compatible semi-inner product (...>s 8 S IR, i.e. Vs S, (,, = || and allows to build a bijective and isometric dual space S*).
Theorem -Vector-valued RKBS (Zhang, 2013)). A Aavuude reproducing kernel Banach space B of functions from V to U is Banach space such that for all e V. the point evalutation 5e : B + l defined as (O) = O(v) is continuous. In this case, there exists a unique function K. K:Vr V - L(4) such that for all (v, u) E V U:
- K(v, B O (O(v), uru = (0, K(oo,)(u))e Vo'E,, Vo'EV v, [K(e, L')l & 110011((..0) ow (L4
Informally, theorem (1) states that RKBS are spaces sufficiently regular such that the image of any element O at a given point v can be expressed in terms of a unique function K. The latter is hence called the reproducing kernel of B and our goal is to leverage such unicity build the map Te. Let D be the set of all datasets Do previously defined. The following original theorem gives the existence of solution to our meta- problem and relates it to the reproducing kernel.
Theorem 2 (RKBS representer map). Let B be a U-valued RKBS from V to U, for any dataset Do D. (()) is lower cmni-contiuuus, coercive and bounded below, then there exists function T D B such that T(Do) ) is minimizer of equation (1). If L. is of the form L(., Do) L (8 with É Lt + IR, then the dual T(Do)* in span(K((ss (u)),, n. u E Uf. Furthermore, for any Do. L(., Do) is strictly-convex, then T is unique.
While theorem (2) provides conditions for the existence of solutions to each regression problem defined by (1), the usual method consisting in solving instance-specific minimization problems
3

---

derived from representer theorems characterizations is generally intractable in RKBS for sev- eral reasons (non-convexity and infinite dimeniioanim of the problem w.r.t variable u, non- additivity of the underlying semi-inner product). Instead, we propose to define image solutions T(D)) Eign Ko(1), vi (ū,) where Kg and (0,) are respectively the learned approximation of the U-valued reproducing kernel K. and a set of functions in // resulting from a sequence of deep transformations of image examples (u:) that we define below.
Transformers attention as a reproducing kernel o We first need to build K. Several pieces of work have proposed constructions of K. in the context of a non-symmetric and nonpositive semi-defiiite real-valued kernel (Zhang et al., 2009; Georgiev et al., 2014; Lin et al., 2019; Xu and Ye, 2019). In particular, the exponential key-query function in the popular Transformer model (Vaswani et al.. 2017) has been interpreted as real-valued reproducing kernel Kg V V - I in Wright and Gonzalez (2021). We extend below this interpretation to more general vector-valued RKBS:
Proposition (Dot-product attention as U-valeed reproducing kernel). Let (pi)jeJ a finite sequence of strictly positive integers. let (A)) JaJ be applications from V x V to I. let Vg be linear applications from L(U, RP)) and We a linear application from L( [1 RP,, application Ke
IR,, ,2), the (multi-heade J6J
V V - L(U) defined by
Ne(v, v(),,')(u)) = We (L...A (V/l,,,)))VVVvv........))))")) vo((w)...
(4)
is the reproducing kernel of an U-valued RKBS. In particular, if = V= R2, for p N+ and A'g exp (x (") (K2'))/ v') with (Qg, Ko)js, applications from )d), Ke corresponds to the dot- product attention mechanism of Vaswani et al. (2017).
Note that in (4), the usual softmax normalization the dot-product attention is included the linear operations Ag through a. We show in the next section how such kernel construction can be leveraged to build the map Te and that several variations of the kernel construction are possible, depending on the target space B and applications. Contrary usual kernel methods, our model jointly builds the full reproducing kernel approximation Ke and the instance instance-specific parametrization (ū.)æ by integrating the solutions iteratively over several residual kernel transformations. We refer to system as a Transducer, both as a tribute to the Transformer computation mechanism from which it is inspired and by analogy with signal conversion devices.
4 The Transducer
Model definition o We define To as the sum of L residual kernel transformations (KoJeez eez whose expression can be written:
VVEV, To(Do)(U) Eoo(,, ()(
Σ Σ rg(ef, o²) (u!))
(5)
1 DEL
where (vf, u u.iignte and (v") refer to sequences of representations starting respectively with (o) (ol,ut))s ut)ien = Do. v² = and defined by the following recursive relation:
v ++ = =)(()) Det1 F6(D")
(6)
u uit1 at 2i (0)
e+1 oo ((()) where of (
where (Fg. GolEE correspond to (optional) parametric non-linear residual transformations applied in parallel representations (of, of) sinnn while (re))s are intermediate kernel transformations of the form x : V x V - L(U) such as the one defined in equation (4). Breaking down kernel estimation through this sequential construction allows for iteratively refining the reproducing kernel estimate and approximating on on-the-fly the set of solutions (ū,) We particularly investigate the importance of depth L in the experimental section. Note that equations (5) and (6) allow to handle both varying dataset sizes and efficient parallel inference by building the sequences Jegg) with (æf)isn, in batches and simply masking the unwanted cross-relational features during the kernel operations. All the operations are parallelizable and implemented on GPU-accelerated tensor manipulation libraries such that each regression with To orders of magnitude faster than gradient- based regression methods.

---

Discretization o In the case of infinite-dinne dimensional functional input and output spaces V and U, we can accommodate, for numerical computation purposes, different types of function representations previously proposed for neural operator regression and allowing for evaluation at an arbitrary point of their domain. For instance, output functions u can defined as linear combination of learned or hardcoded finite set of functions, as in Lu et al. (2019) and Bhattacharya et al. (2020). We focus instead on a different approach inspired by Fourier Neural Operators (Li et al., 2020a), by applying our model on the . first modes of a fast Fourier transform of functions (v, us)isnn and transform back its output, allowing us to work with discrete finite function representations.
Meta- Deet-training In order to train To to approximate solution for all problems of the form (1), we jointly learn the kernel operations (r2g) Jest as well as transformations (FF))ss) Let us assume that C is of the form L(O', Do) E, L(O'(u,) (u,),O(e)) that datasets Do are sampled according to a probability distribution D over the set of possible example sets with finite cardinality and that a random variable I select the indices of each test set Dier {(0,, u.) (m), uj) E Do.j E I) such
(b))
that the train set is Digien = Do' Doo..
Our meta-learning objective is defined as:
((0)- ED
Σ E(To(DD (og), JET
which can be tackled with gradient- based optimization w.r. parameters 0 provided L is differentiable (see S.I for details). In order estimate gradients of (7), we gather meta-dataset of M operators example sets (Do.,)mes M and form, at each training step, a Monte-Carlo estimator over a batch of k datasets from this meta-dataset with random train/test splits (I&). For each dataset in the batch, in order to form outputs To (D
) (v,) defined by equation (5), we initialize the model sequence in (6) by concatenating Dig with Doury {(v,, 0z)) v; e Di and obtain each infered output Te (Drr") as L Duman Kg(U,, (v,, (r,) Since each regression consists in single feedforward pass, estimating gradienfs of the meta-patt pertapparaettess with respect L for each batch consists in single backward pass achieved through automatic differentiation.
Numerical experiments
In this section, we show empirically that our meta-optimized model is able to approximate any element O of diverse function spaces B such as operators defined on scalar and vector- vector-valued valued function spaces derived from parametric physical systems or regression problems in Euclidean spaces. In all experiments, we use the Adam optimizer (Kingma and Ba, 2014) to train for fixed number of steps with an initial learning rate gradually halved along training. All the computation carried on a single Nvidia Titan Xp GPU with 12GB memory. Further details can be found in S.I.
5.1 Regression of Advection-Diffusion Reaction PDEs
3iter
sla; 0) 0 a(x,0)
12-1
1eitec
05
10
110-2
NcRU Bees
10:1
7NO
10
Dataset size
Training shpps
Figure 2: Left: RMSEs (and 95% C.I) on unseen operators as a function of the dataset size. The grey area corresponds to dataset cardinalities seen during the Transducer meta-training. For comparison, we train baselines from scratch with the corresponding number of examples. Middle: Training losses of Transducers with different depths. Applying several times the kernel improves performance. Untied weights yield the best performance. Right: (Up) examples of evolution of s(x, f) for different ADR equations and (bottom) spatial MSEs of intermediate representations (u") colored by iteration f. The decreasing error, consistent with the MSE reduction of deeper models, suggests that network depth allows for progressively refining function estimates.
5

---

First, we examine the problem of regressing operators O associating functions from V C
C([o, R) to their solutions u = ( C(10, 1].B) with respect to advection-diffusion reaction equations defined on the domain f = [0. x [0,4] with Dirichlet boundary conditions s(0, 1) (1,
We consider the space B of operators (...).k.) specifically defined by
v(x) = s(x, 0), =xx) s(x, t) and follows an equation depending on unknown random continu-
ous spatially-varying diffusion ((x), advection ((r), and a scalar reaction term &s ~ U[0 0.1J:
(z, V (6(( s(,,
(x) t)
(s(x,t))²
(8)
diffusion
advection
reaction
Eq. (8) is generic with components arising in many physical systems of interest, leading to various forms of solutions s(x,t) (We show examples for three different operators in figure 2.) Several methods exist for modeling such PDEs, but they require knowledge of the underlying parameters (ò, v, k) and often impose constraints on the evaluation point as well as expensive time-marching schemes recover solutions. Here instead, assume no priori knowledge of the solution and directly regress each operator O behavior from the example set Do.
METHOD
RMSE
TIME (S)
GFLOPS
FNO
2.96e 4
1.72c²
1.68c²
DEEPONET
2.02c-2
7.85cl
1.54e2
FNO-MAML
1.4e 11
2.10c
1.6e 1
TRANSDUCER
2.39e
3.10e
1.06e
Table 1: RMSE and compute costs of regression over 50 unseen datasets with 50 examples. Note that DeepONet and FNO are optimized from scratch while the Transducer and FNO-MAMI. have been pre-trained. GFL.OPs represent the total number of floating point operations for regression.
Baselines and evaluation o We meta-trained our model to regress 500 different operators ((1...k11) with t = 1 fixed and varying number of examples n E [20, 100] with images evalu- ated at 100 equally spaced points (Irx ( on the domain [o, 1] and meta-tested
30
set of 500 operators with new parameters d, v,, & and initial states u. Although not directly equiva- lent to existing approaches, we compared our method with standard regression methods as well as inductive neural operator approxima- tors. We applied standard finite-dimensional regression methods, K K-Nearest-Neighhors (Fix and Hodges, 1989). Decision Trees (Quinlan, 1986) and Ridge regression with radial basis kernel (Hastic et al., 2009) to each discretized problems (1O(v))zza) = u, (xx) j.)) as well as two neural-based operators to each dataset instance: DeepONet (Lu et al., 2021) and FNO (Li et al., 2020a). Finally, we also tried to meta-learn a parametrization of FNO that could adapt in 100 gradient steps following MAML (Finn et al., 2017) using the same meta-dataset. For all but this approach, an explicit optimization problem solved before inference order to fit the target operator. On the other hand, after meta-training of the Transducer, which takes only a few minutes to converge, each regression is solved in a single feedforward pass of the network, which is orders of magnitude faster and can be readily applied to new problems (Table 1).
30
20
tme
0.0
0.5
0.1
0.2
0.3
0.4
0.5
C.
Figure 3: Example of Transducer regression extrapola- tion and RMSES on OOD tasks with n = 100 examples. Color code corresponds to different correlation lengths used to generate the random functions ((x)) and (xr) Much of the result remains below 1% error despite never being trained on such operators.
Results We first verified that our model approximates well unseen operators from the test set (Table 1). We noted that our model learns a non-trivial kernel since the estimation produced with NeaNests Neighbors remains poor even after le² examples. Moreover, since our model can perform inference for varying input dataset sizes, we examined the Transducer accuracy when varying the number of examples and found that it learns a converging regression program (Figure 2) which consistently outperforms other spstance-specffic regression approaches with the exception of FNO when enough data is available (> 60). We also found that deeper Transducer models with more layers increase kernel approximation accuracy, with untied weights yielding the best performance (figure 2.)

---

Extrapolation to OOD tasks We further tested the Transducer ability to regress different operators than those seen during meta-training, Specifically, we varied the correlation length (C.L) of the Gaussian processes used to generate functions ((z) and vxx)) and specified different target time transfers well these new regression problems and that regression performance degrades gracefully as the target operators behave further away from the training set (figure 3), while inductive solutions do not generalize.
* We showed that the kernel meta-optimized for solution
5.2 Outliers detection on 2D Burgers' equation
We further show that our regression method can fit operators of vector-valued functions by examining the problem of predicting 2D vector fields defined as solution of two-dimensional Burgers' equation with periodic spatial boundary condition on the domain Ω = [0. 1] x 10]:
Ōis(ū T, -Dw ( s V ((,
(9)
diffiusion
Here, we condition our model with operators of the form, v( s a 1). u( 8(Ø, (') such that
our model can regress the evolution of the vector field F starting at any time, with arbitrary temporal increment t' & 10 seconds and varying diffusion coefficient v [0.1, 0.5]. We show in figure (4) and table (2) that our model is able to fit new instances of this problem with unseen parameters v.
ba
0.25
5/5
00
Figure Illustrative example of initial 0). target (t 10) and Transducer estimation of the vector field s(Ø 1) discretized resolution 64 x over the domain [0, 1J for the Burgers' equation experiment.
panel represents absolute error to ground truth.
Fast and differentiable regression Since the fitting operation is orders of magnitude faster than other operator regression approaches well fully differentiable, it allows for quickly executing expensive schemes requiring multiple regressions. This can have several applications, from bootstrapping or producing confidence intervals by varying the example Dinuin or performing inverse problems using Monte-Carlo Markov Chain in the dataset space. We showcase an example of this potential with outlier detection experiment: We use the Transducer identify outliers of dataset of Burgers' equation with coefficient v1 artificially contaminated with elements from another dataset 1/2 > V1 5% level. We identify outliers by estimating RMSES over 5000 different regressions using random 50 % splits with outliers potentially present in both training and testing sets. This technique takes only a few seconds to estimate while outliers are clearly identified data points with significantly higher RMSE than the dataset average (figure 5). As a comparison, performing Spectral Clustering (Yu and Shi, 2003) on the FFT of elements (u:) yields very poor precision (table 2)
t 5s
10s
RMSE sets)
2.2e
5.9e 3
RMSE
0.02
Outliers (Pre./Rec.)
100% /100%
100%/1000
/100%
S.C. (Pre./Rec.)
6%/85%
7%/85%
Table 2 & Figure 5: Left: Meta-test regression and outlier detection results at two target times. RMSEs on Burgers' equations averaged over 200 different parameter conditions E [0.1. 0.5] each with 100 train examples. Precision/Recall in outlier detection of the Transducer each element the contaminated dataset over the 5000 regressions. Outliers are clearly identified.
Spectral clustering- Right: RMSE distributions of

---

100e
1050
1000
1.0
270
24
Lingiude
Lorgbuan
-40
1100
1050
-30
1000
(880) J
Lattude
850
500
500
-60
150
-60
100
50 reggguudeeeeee (+W)
30
100
Longltude
63
800
000
Lor giait (-10)
Figure Up- Illustrative examples of 720 720 temperature (left) and pressure (right) fields of the ERA5 dataset. Bottom Estimated pressure field from conditioning the Transducer with 15 days data dating week before the target date. Insets show recovered details of the estimation (blue) compared with ground truth (red).
5.3 Climate modeling with seasonal adaptation
One advantage of our approach is the ability to select the data that is most relevant with respect to certain prediction task and subsequently adapt the model response. For instance, robust and precise prediction of climate variables is difficult because models need to account for seasonal variability and adapt to drifting parameters. Even with a globally large amount of available data, the underlying operator of interest might change over time or be affected by unobserved phenomena. Hence, in order to fully exploit the potential of data-driven methods, being able to capture such variations might greatly fluctuating and drifting data distributions. In order to illustrate the
help prediction performance applicability and scalability of deep transductive learning, we considered the problem of predicting the Earth's surface air pressure solely from the Earth's 's surface air temperature at high resolution. Data taken from the ERA5 reanalysis (Hersbach et al., 2020) publicly made available by the ECMWE, which consists of hourly high-resolution estimates of multiple atmospheric variables from 1979 to the current day. We model pressure estimate on a 720 x 720 grid, resulting in a spatial resolution of 0.25 x 0.5% allowing to capture small features such local dynamics and geographic relief.
Similar to (Pathak et al., 2022), we modify VIT backbone to incorporate a kernel transduction layer before ev- ery patch attention and compare model to an unmodified ViT baseline with matching number of parame- ters. We additionally compare with a fully transductive Nearest Neighbors approach. In Figure 6 and Table 3, we present results obtained training Transducer with data from 2010 to 2014 and testing it on data from 2016 to 2019, We trained our model by predicting 5 random days sampled from random 20-day windows and present two test configurations: We either condition the Transducer with window centered at the previous year's same date (P. or with 15 days window lagging by week (P.W) (see SI for details). Both cases outperform transductive and inductive baselines with fast inference time, confirming that our solution can scale to large problems and be combined with other deep learning modules.
METHOD
LWMSE (HPA)
TIME (S)
NEAREST-NEIGHRORS
67.326
5.915
VIT TRANSDUCER (P.Y)
32.826
0.053
25.293
0.192
TRANSDUCER (P.W)
22.718
0.192
Table 3: Latitude- weighted mean-square error (in hectopascals) and inference time for the earth surface pressure prediction task.
5.4 Finite-dimensional case: MNIST-like datasets classification
We finally confirm the generality of our approach in the case of finiteddimensional spaces 2/ and by studying the meta-learning problem presented Kirsch et al. (2022) which consists in re- gressing classification functions from the 784-dimensional space of MNIST-like images (LeCun

---

METHOD
MNIST
FASHIONMNIST
KMNIST
MAML (FINN ET AL., 2017)
53.71%
48.44%
36.33%
VSML (KIRSCH ET AL., 2021)
79.04%
68.49%
54.69%
GPICL (KIRSCH ET AL., 2022)
73.70 %
62.24%
53.39%
TRANSDUCER
81.83%
69.85%
60.64%
Figure 7: Comparison of meta-test accuracies of MNIST-like datasets classification tasks. Meta-learning models trained transformations of MNIST and meta-tested original MNIST, FashionMNIST and KMNIST
and Cortes, 2010) 10-dimensional space of one-hot class encoding (i.e functions considered O [0, 1J784 - [0, 1j We meta-train 2-layer Transducer to classify consistently pixel-permuted and class-permuted versions of MNIST. We then meta-test the Transducer to classify the unpermuted MNIST dataset and how the regression map transfer to Fashion-MNIST (Xiao et al., 2017) and KMNIST (Clanuwat et al., 2018). We show in Table 7 that the Transducer outperforms previous meta-learning approaches on both the original MNIST classification task as well as transfer better on Fashion MNIST and K-MNIST classfication tasks.
6
Related work
Transductive Machine learning o Principles of transductive statistical estimation have been formally described in Gammerman et al. (1998); Vapnik (1999). Algorithms relying on relational structures between data points such as K-nearest neighbors (Cover and Hart, 1967) and kernel methods (Nadaraya, 1964; Watson, 1964) build estimates by weighing examples with respect to certain metric space. Further, the "kernel trick" allows to embed possibly (Ferraty and Vieu, 2006) into finite Gram matrix representations that are also well-suited for multi- task regression (Evgeniou et al., 2005; Caponnetto et al., 2008). Distinctively, Gaussian processes regression (Williams and Rasmussen, 1995) combines transduction with Bayesian modeling to estimate a posterior distribution over possible functions. These techniques might suffer from the so-called "curse of dimensionality":: with growing dimensionality, the density of exemplar point diminishes, which increases estimators' variance. More recent work combining deep learning with transductive inference has shown promising results even in nigh-dimensional spaces for few-shot learning (Snell et al., 2017; Sung et al., 2018) or sequence modeling (Jaitly et al., 2015), but the vast majority of neural networks still remain purely inductive.
features
Neural operator learning The universal approximation abilities of neural networks have been generalized to infinite-dimension parametrization can approximate well infiniteddimensional operators. More recent work using neural networks to perform operator regression has shown strong results (Lu et al.. 2019), especially when mixed with tools from functional analysis and physics (Raissi et al., 2017: Li et al., 2020a; Gupta al., 2021; Li al., 2020b; Nelsen and Stuart, 2021; Wang et al., 2021: Roberts al., 202 and constitutes booming research direction in particular for physical applications (Goswami et al., 2022; Pathak et al., 2022; Vinuesa and Brunton, 2022; Wen et al., 2022; Pickering al., 2022). Recently, the Transformer's s attentional computation has been interpreted as Petrov-Gale Galoo-n projection (Cao, 2021) or through Reproducing Kernel Hilbert Space theory (Kissas et al., 2022) for building such neural operators, but these perspectives apply attention to fit single target operator.
function spaces: Chen and Chen (1995) showed that finite neural
Meta-learning and in-context learning Promising work towards more general and adaptable machines has consisted in automatically "learning to learn" or meta-learning programs (Schmidhuher al., 1997; Vilalta and Drissi, 2002), by either explicitly treating gradient descent as an optimizable object (Finn et al., 2017), modeling an optimizer as a black-box autoregressive model (Ravi and Larochelle, 2017) informing sequential strategies via memorization (Santoro et al., 2016; Ortega et al., 2019) More recently, converging findings in various domains from reinforcement learning (Mishra al., 2018; Laskin et al., 2022), natural language processing (Brown al., 2020; Xie et al., 2021; Olsson et al., 2022) and functional regression (Garg et al., 2022) have established the ability of set-based attentional computation in the Transformer (Vaswani et al., 2017) for in-context learning by flexibly extracting functional relationships and performing dynamic association such as linguistic analogy or few-shot behavioral imitation. We show that the theory of RKBS can help interpret such property and extends it to manction-valuc operators regression.

---

7 Discussion
We proposed novel transductive model combining kernel methods and neural networks that is capable of performing regression over entire function spaces. We based our model on the theory of vector vector-valued Reproducing Kernel Banach Spaces and showcased several instances where it learns regression program able, in a single feedforward pass, to reach performance levels that match or outperform previous instance-specific neural operators or meta-learning systems. Our approach holds potential to create programs flexibly specified by data and able model entire families of complex physical systems, with particular applications in functional hypothesis testing, dataset curation or fast ensemble learning. However, limitation is that model relics meta-training, which requires collecting a sufficiently diverse meta-datasct to explore the kernel space. In future work, we plan to investigate methods such synthetic augmentation to reduce the costs of meta-training.
8
Acknowledgements
This work was funded by ANR-3IA Artificial and Natural Intelligence Toulouse Institute (ANR- 19-P13A0004) ONR (N00014-19. 1-2029), NSF (IIS-1912280 and EAR-1925481), DARPA (D19AC00015) and NIH/NINDS (R21 NS 112743). Additional support provided by the Carney Institute for Brain Science and the Center for Computation and Visualization (CCV) and the NIH Office of the Director grant S100D025181.
10

---

References
Aggarwal, C. C., Hinneburg, A., and Keim, D. A. (2001). On the surprising behavior distance metrics in high dimensional space. In International conference on database theory, pages 420-434. Springer.
Bellman, R. (1966). Dynamic programming. Science, 153(3731):34-37.
Bhattacharya, K., Hosseini, B., Kovachki, N. B., and Stuart, A. M. (2020). Model reduction and neural networks for parametric pdes.
Boser, B. E., Guyon, M., and Vapnik, (1992). A training algorithm for optimal margin classifiers. In Proceedings of the fifth annual workshop on Computational learning theory, pages 144-152.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners, Advances in neural information processing systems, pages 1877-1901.
Cao, S. (2021). Choose a transformer: Fourier or galerkin. Advances in Neural Information Processing Systems, 34:24924-24940.
Caponnetto, A., Micchelli, C. A.. Pontil, M., and Ying. Y. (2008). Universal multi-task kernels. The Journal of Machine Learning Research, 9:1615-1646.
Chen, T. and Chen, H. (1995). Universal approximation nonlinear operators by neural networks with arbitrary activation functions and its application dynamical systems. IEEE Transactions on Neural Networks, 6(4):911-91
Clanuwat, T., -ober---izzzn M., Kitamoto, A., Lamb, A., Yamamoto, K., and Ha, D. (2018). Deep learning for classical japanese literature.
Cover, T. and Hart, P. (1967). Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1:221-27.
Evgeniou, T., Micchelli, C. A., Pontil, M., and Shawe-Taylor, J. (2005). Learning multiple tasks with kernel methods. Journal of machine learning research, 6(4).
Ferraty, F. and Vieu, P. (2006). Nonparametric functional data analysis: theory and practice, volume 76. Springer.
Finn, C., Abbeel, P., and Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR.
Fix, and Hodges, L. (1989). Discriminatory analysis. nonparametric discrimination: Consistency properties. International Statistical Review Revue Internationale de Statistique, 57(3):238-247,
Gammerman, A., Vovk, V., and Vapnik, V. (1998). Learning by transduction. In Proceedings of the Fourteenth conference on Uncertainty artificial intelligence, pages 148-155.
Garg, S., Tsipras, D., Liang, P., and Valiant, G. (2022). What can transformers learn in-context? a case study of simple function classes.
Georgiev, P., Sánchez- González, L.,, and Pardalos, P. (2014). Construction of pairs of reproducing kernel banach spaces.
Goswami, S., Bora, A., Yu, Y., and Karniadakis, G. E. (2022). Physics-informed neural operators. arXiv preprini arXiv:2207.0574. 05748.
Gupta, G., Xiao, X., and Bogdan, (2021). Multiwavelet-based operator learning for differential equations. Advances in Neural Information Processing Systems, 0024048-24066
Hastie, T., Tibshirani, R., and Friedman, J. (2009). The elements of statistical learnin. Cited on, 33.

---

Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Horányi, A., Muñoz-Sabater, J., Nicolas, J.,
Peubey, C., Radu, R., Schepers, D., Simmons, A., Soci, C., Abdalla, S., Abellan, X., Balsano, G., Bechtold, P., Biavati, G., Bidlot, J., Bonavita, M., De Chiara, G., Dahlgren, P., Dee, D., Diamantakis, M., Dragani, R., Flemming, J., Forbes, R., Fuentes, M., Geer, A., Haimberger, I.., Healy, S., Hogan, R. J., Hólm, E., Janisková, M., Keeley, S., Laloyaux, P., Lopez, P., Lupu, C., Radnoti, G., de Rosnay, P., Rozum, I., Vamborg, F., Villaume, S., and Thépaut, J.-N. (2020). The era5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146(730):199-- 2049.
Jaitly, N., Sussillo, D., Le, Q. V., Vinyals, O., Sutskever, I., and Bengio, S. (2015). A neural transducer. arXiv preprint arXiv: 1eeees1111.050 1.04868.
Jin, P., Lu, L., Tang. Y., and Karniadakis, G. E. (2020). Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness. Neural Networks, 130:85-99.
Kingma, D. P. and Ba, J. (2014), Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980
Kirsch, L., Harrison, J., Sohl-Dickstein, J., and Metz, L. (2022). General purpose in-context learning by meta-learning transformers. In Sixth Workshop Meta Meta-Learning the Conference on Neural Information Processing Systems.
General-purpose
Kirsch, L., Schmidhuber, J., and Al (2021). Meta learning backpropagation and improving it. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang. P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages 44132-14134. Curran Associates, Inc.
Kissas, G., Seidman, J. H., Guilhoto, L. F., Preciado, V. M., Pappas, G. J., and Perdikaris, P. (2022). Learning operators with coupled attention. Journal of Machine Learning Research, 23(215):
Laskin, M., Wang, L., Oh, J., Parisotto, E., Spencer, S., Steigerwald, R., Strouse, D., Hansen, S., Filos, A., Brooks, E., Gazeau, M., Sahni, H., Singh, S., and Mnih, V. (2022). In-context reinforcement learning with algorithm distillation.
LeCun, Y. and Cortes, C. (2010). MNIST handwritten digit database.
Li, Z., Kovachki. N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. (2020a). Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08885.
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Stuart, A., Bhattacharya, K., and Anandkumar, A. (2020b). Multipole graph neural operator for parametric partial differential equations. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages
Curran Associates, Inc.
Lin, R., Zhang, H., and Zhang, (2019). On reproducing kernel banach spaces: Generic definitions and unified framework of constructions.
Lin, R. R., Zhang, H. 7.,, and Zhang, J. (2022). On reproducing kernel banach spaces: Generic definitions and unified framework of constructions. Acta Mathematica Sinica, English Series, 3899):1459-1483
Lu, I.., Jin, P., and Karniadakis, E. (2019). Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint 190.v:1910.03192
Lu, L.,, Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. E. (2021). Learning nonlinear operators via deeponet based the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218-229.
McCloskey, M. and Cohen, J. (1989). Catastrophic interference in connectionist networks: The sequential learning problem, volume 24 of Psychology of Learning and Motivation, pages 109-165. Academic Press.
12

---

Micchelli, C. A. and Pontil, M. (2004). A function representation for learning in banach spaces. In Learning Theory: 17th Annual Conference on Learning Theory, COLT 2004, Banff, Canada, July 1-4, 2004. Proceedings 17, pages 255-269. Springer.
Mishra, N., Rohaninejad, M., Chen, X., and Abbeel, P. (2018). A simple neural attentive meta-learner. In International Conference on Learning Representations.
Nadaraya, E. A. (1964). On estimating regression. Theory of Probability & Its Applications, 91):1411-14..
Nelsen, N. H. and Stuart, A. M. (2021). The random feature model for input-output maps between banach spaces. SIAM Journal on Scientific Computing. 43(5):A3212-A322
Olsson, C.,, Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. (2022). In-context learning and induction heads. arXiv preprint wrXiv:2209.18955
Ortega, P. A., Wang. J. X., Rowland, M., Genewein, Kurth-Nelson, Z., Pascanu, R., Heess, N., Veness, J., Pritzel, A., Sprechmann, P., et al. (2019), Meta-learning of sequential strategies. arXiv
preprint XXXv::900.03000
Pathak, J., Subramanian, S., Harrington, P., Raja, S., Chattopadhyay, A., Mardani, M., Kurth, T., Hall, D., Li, Z., Azizzadenesheli, K., Hassanzadeh, P., Kashinath, K., and Anandkumar, A. (2022). Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators.
Pickering, E., Guth, S., Karniadakis, G. E., and Sapsis, T. P. (2022). Discovering and forecasting extreme events via active learning in neural operators. Nature Computational Science, 2(12):823-
833.
Quinlan, J. R. (1986). Induction of decision trees. Machine learning. 1(1):81-106.
Raissi, M., Perdikaris, P., and Karniadakis, G. E. (2017). Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint 17X1v:1711.10561
Ravi, S. and Larochelle, H. (2017). Optimization as a model for few-shot learning. ICLR.
Roberts, N. C., Khodak, M., Dao, T., Li, L.., Re, C., and Talwalkar, A. (2021). Rethinking neural operations for diverse tasks. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems.
Santoro, A., Bartunov, S., Botvinick, M. M., Wierstra, D., and Lillicrap, T. P. (2016). One-shot learning with
neural networks. CoRR, abs/1605.06065.
Schmidhuber, J., Zhao, J., and Wiering, M. (1997). Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-improvement. Machine Learning, 28.
Snell, J., Swersky, K., and Zemel, R. (2017). Prototypical networks for few-shot learning. Advances in neural information processing systems, 30.
Sung. F., Yang. Y., Zhang, L., Xiang, T., Torr, P. H., and Hospedales, T. M. (2018). Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Vapnik, V. (1999). The nature of statistical learning theory. Springer science & business media.
Vapnik, V. (2006). Estimation of dependences based on empirical data. Springer Science & Business Media.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
13

---

Vilalta, R. and Drissi, Y. (2002). A perspective view and survey of meta-learning. Artif. Intell. Rev., 18(2):77-95.
Vinuesa, R. and Brunton, S. L. (2022). Enhancing computational fluid dynamics with machine learning. Nature Computational Science, 2(6):358-366.
Wang, S., Wang, H., and Perdikaris, P. (2021). Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science Advances, 7(40):eabi8605.
Watson, G. S. (1964). Smooth regression analysis. Sankhyā: The Indian Journal of Statistics, Series A. pages 359-372.
Wen, G., Li, Z., Azizzadenesheli, K., Anandkumar, A., and Benson, S. M. (2022). U-fno an enhanced fourier neural operator- based deep-learning model for multiphase flow. Advances in Water Resources, 163:104180.
Williams,
Rasmussen, C. (1995). Gaussian processes for regression. Advances neural information processing systems, 8.
Wright, M. A. and Gonzalez, J. (2021). Transformers deep fiinitedimmesiioal non-mercer binary kernel machines. ArXiv, abs/2106.01506.
Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.
Xie, S. M.. Raghunathan, A., Liang, P., and Ma, T. (2021). An explanation in-context learning as implicit bayesian inference.
Xu, Y. and Ye, Q. (2019). volume 258. American Mathematical Society.
Yu and Shi (2003). Multiclass spectral clustering. In Proceedings Ninth IEEE International Confer- ence on Computer Vision, pages 313-319 vol.1.
Zhang, H. (2013). Vector-valued reproducing kernel banach spaces with applications to multi-task learning, Journal of Complexity, 1952)11
Zhang, H., Xu, Y., and Zhang, J. (2009). Reproducing kernel banach spaces for machine learning. Journal of Machine Learning Research, 10(12).
14


Reviews:
[{"review_id": "qs0J8vVtj1", "reviewer": "Reviewer_Sczv", "summary": "**SUMMARY AFTER REBUTTAL**: as described below, most of my comments were addressed and the authors made a significant job in including new results. I have increase my score during the rebuttal phase and I strongly vote for acceptance.\n\n---\n\nNeural operators are neural networks that are trained to approximate a function-to-function mapping. This paper proposes an algorithm to perform few-shot learning of these models, where at inference time the network is given a set of examples of the operator, and an input function as the test point.\n\nThe specific algorithm they propose is built on reproducing kernel Banach spaces, where for a given prediction, the kernel and the dictionary on which it is evaluated is computed by a recursive embedding of the few-shot examples. This is inspired by the transformer architecture. When examples are continuous, they are discretized with an FFT.\n\nThey analyze a series of classical benchmarks including modeling PDEs, showing the method to perform well.", "questions": "- I think rephrasing the entire \"transductive\" discussion in terms of few-shot learning could significantly improve the paper. At the least, objectively incorrect sentences should be amended.\n\n- Discussing better training and test complexity is important.\n\n- I think the paper can improve readability by providing a practical example of (5)-(6) in the more familiar case of finite-dimensional input-output spaces.\n\n- There has been a limited amount of works that have explored using neural networks to perform Bayesian posterior inference on-the-fly on a large family of distributions (e.g., Prior-Data Fitted Networks (PFNs)). I would be curious to see a discussion on the connection with this work. On the related work part, there is also some relevant literature on recursive (recurrent) kernel evaluation which is not mentioned.\n\n- There is a small typo on page 3: point evalutation.\n\n- Will the code be released?", "limitations": "The authors are clearly discussing the limitations of their few-shot setup. However, they are not discussing the computational complexity of the method, which I think would be the biggest limitation in scaling this up to large setups. A few sentences on this would be appreciated.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "As far as I know, the idea of performing few-shot learning of neural operators is novel. These networks are used extensively in physical modelling, so the paper can have a sizable impact there.\n\nThe paper is also relatively clear (with a few exceptions, see below), especially if one considers the complexity of some underlying ideas. The connection between their method and the transformer is also interesting, but from what I understand this is extended from Wright and Gonzalez (2021).\n\nThe experiments are varied and cover a wide range of use cases.", "weaknesses": "1) I have found the initial discussion on the difference between transduction and inference a bit misleading. According to Vapnik, 2006 (which they cite), almost anything which is used today in ML / deep learning is inductive. Transductive methods are only those that can make predictions on a given set of test points but *cannot* operate outside those. For example, SVM is inductive in general, except for some variants such as the transductive SVMs discussed in Collobert et al., 2006. In fact, what they are calling \"transductive\" is what is typically called \"instance-based\" in ML (kNN, SVM). Their setup is a standard few-shot setup extended to operators. This also leads to some strange sentences, such as \"inductive neural learning with gradient descent is compute-intensive\" referred to neural networks; SVM training is also notoriously intensive (and it can also be done via gradient descent).\n\n2) There are some \"standard\" methods for performing few-shot learning in the literature (e.g., MAML, prototype networks, ...). Many of these methods start from a standard neural network and adapt it to a few-shot scenario. Architectures for performing operator learning are known, and it is not clear from reading the paper why the few-shot learning methods are not immediately extensible to this setup (e.g., why can't we do MAML on a standard neural operator?), and why we need instead to resort to a more complex formulation in terms of kernels. To clarify: I think the algorithm shown here is interesting, but it's a bit hard to motivate it by reading the paper itself.\n\n3) The computational complexity of the method is not discussed, in particular the need to execute multiple FFTs for the few-shot examples (Section \"Discretization\"). This also connects to recent literature on performing FFTs in an efficient way on GPUs (e.g., FlashConv). Also for testing, if I understand correctly the authors are mostly comparing their method (which is only a forward pass) with a full training of its competitors, which is a bit unfair. No baselines for few-shot learning are tested. For example, they state that their setup is \"Similar to (Pathak et al., 2022)\", but they do not compare.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and for this thoughtful review. We are happy that you seem enthusiastic about our approach and we try to answer specifically to your remarks and questions below:\n\n  > Regarding “the initial discussion on the difference between transduction and inference”.\n - Indeed, we definitely agree with you that current deep learning techniques rely, in a vast majority, on inductive learning principles, which constitutes a starting motivation of our work. We also definitely agree that \"instance-based learning\" is another valid denomination regarding how decisions rules are formed in our approach. However, we disagree on your definition of transductivity (_\"Transductive methods are only those that can make predictions on a given set of test points but cannot operate outside those\"_) See for instance the seminal _Learning by Transduction_ [1]. (Section 1: \"This is a problem of transduction, in the sense that we are interested in the classification of a particular example rather than in a general rule for classifying future examples\" and section 6 \"\"Transduction\" is inference from particular to particular; for the problem of pattern recognition, it means that, given the classifications Yi, i = 1, . .. , l,of the l points x1, ... , Xl in the training set, we are only trying to guess the classifications of the k points xl+1, ... , Xl+k in the test set.\" Remark 5 of the same  reference also directly agree with your denomination of \"instance-based learning\" which we will mention in the introduction. Finally, we reworked the expression \"inductive neural learning [...] is compute-intensive\" to emphasize instead that iterative optimization procedures for tackling high-dimensional problems are compute-intensive and bottlenecked bye their sequential nature.\n\n> Regarding data regimes and comparison “with standard methods\".\n - We would like to point out that our method is not restricted to the \"few-shot\" data regime but can be applied to problem instances with more than thousands exemples (for instance on section 5.3, we test out model with up to 1000 exemples pairs.). However, as you noted, our model needs an original meta-optimization procedure that is unusual in the recent neural operator litterature. To your suggestion, we are working on complementing section 5.1 with a comparison with a MAML-like adaptation procedure applied to FNO/DeepONet and will communicate the results as soon as possible. Note however, that such adaptation technique will be much slower than our feedforward adaptation process.\n\n> Regarding the computational complexity of the method, scaling and the cost of FFTs.\n -  We are actually performing a single FFT/IFFT transformation as a pre/post-processing operations, in line with recent litterature on Fourier operators [2]. This largely mitigates the cost of such operation which allowed us to scale our system to vert high-resolution climatic prediction (720x720 images). Note that in this experiment, we are not interested in obtaining state-of-the-art climatic variable prediction as in Pathak et al., 2022, but we are showing that our transductive approach can augment popular inductive models in the context of operator regression problems.\n\n > Regarding improving readability of equations 5-6 with a finite-dimensional example. \n - To your suggestion, we are working on integrating a cartoon of the kernel computation to the exemples depicted in figure 1 to improve readability.\n\n> Regarding related work on Bayesian posterior inference and recursive kernel iterations. \n - Thank you for these interesting suggestions, we will definitely mention this relevant litterature to our discussion. Regarding recursive kernel evaluation, are there specific pointers that you could share for us to adapt our discussion?\n\n> There is a small typo on page 3: point evalutation.\"\n - Thank you for this catch.\n\n> Will the code be released? \n - Yes we plan to release a public repository with the code to replicate the experiments as well as pre-run notebooks to help the interested reader familiarize with our model.\n\nWe thank you again for your interest and this valuable feedback which definitely helps strenghten our work.\n\n[1] Learning by transduction A. Gammerman, V. Vovk, V. Vapnik. UAI'98: Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence\n[2] Transform Once: Efficient Operator Learning in Frequency Domain. Michael Poli, Stefano Massaroli, Federico Berto, Jinykoo Park, Tri Dao, Christopher Ré, Stefano Ermon. NeurIPS 2022\n"}, {"review_id": "TeZSah2eNb", "reviewer": "Reviewer_f2BS", "summary": "This paper proposes a method for transductive learning based on reproducing kernel banach spaces (RKBS). The resulting model is capable of learning *in-context* in the sense that given a new instance of a learning problem or a new dataset $\\mathcal{D}$, it can infer the resulting functional relationship at inference time. The authors show that transformer attention layers can be considered reproducing kernels for this model. They provide a meta training objective that allows them to learn the kernel operations $\\kappa^\\ell$ and nonlinearities $F^\\ell$ which generalize well over the distribution $\\mathfrak{D}$ of possible datasets. Once trained, the model can generate functions for new datasets. \n\nThe authors test the proposed ideas to learn operators $\\mathcal{O}$ for PDEs including advection reaction diffusion equation and Burgers' equation as well as climate modeling. Lastly, they test on a MNIST like task where there are pixel permuted and class permuted versions of the dataset. In all of these settings they identify benefits to their approach over several existing benchmarks.", "questions": "On the MNIST experiment, how well would a pure supervised algorithm (perhaps with a provided context signal to indicate permutation) trained on all instances of the data perform? I am wondering if the meta-learning objective outperforms standard supervised learning because it sees a larger amount of total data. ", "limitations": "The authors addressed their need to work with finite dimensional outputs for the PDE modeling (Fourier coefficients) and also acknowledged the computational and statistical requirements to optimize the metal learning objective. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper provides a nice framework and perspective to think about transductive learning through reproducing kernels. To the best of my knowledge, this is a novel idea.\n\nThe paper also provides several experiments that showcase the benefits of the approach, especially in PDE modeling.", "weaknesses": "As one of the primary motivations of the model is a meta-learning algorithm capable of in-context learning, I think that an experiment involving in-context learning of language patterns would greatly strengthen the paper. Most of the experiments at this point are for PDEs but the proposal is much broader. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and the interesting suggestions that you shared. We are happy about your positive assessment and we reply specifically to your comments below.\n\n > Regarding adding \"an experiment involving in-context learning of language patterns\" \n - This is an intriguing suggestion. We agree that an experiment involving natural language could definitely strenghten the interpretation of in-context learning in attentional layers as a form of reproducing kernel fitting since ICL has been predominently demonstrated  in this domain. However, analyzing natural language in terms of functional reproducing banach space of is non trivial and we chose to select more straightforward exemple tasks to illustrate our presentation. We plan on developing further this idea in a follow-up work.\n\n> Regarding the MNIST experiment.\n - We want to emphasize that all baselines are meta-learning systems that receive the same training curriculum as our model. Hence, the difference of performance can not be attributed to training data volume. Moreover, note that providing a context signal to indicate permutation will be counter-productive at test time, since it would provide a shortcut to circumvent meta-learning of a genuine regression program, hence, preventing transfer learning on other datasets (FashionMNIST and KMNIST). On the other hand, training a supervised baseline on a large quantity of class and pixel permutation with no permutation signal is likely to provide no learning at all. We agree however with you, that quantifying how our meta-learned kernel solution converge as a function of data volume is an important future direction for our approach.\n\nWe thank you again for your time and valuable feedback. "}, {"review_id": "PByd3nyd4p", "reviewer": "Reviewer_w6wT", "summary": "This paper proposes a new deep learning approach for the problem of meta-learning. Inspired by the theory of reproducing kernel Banach space, the proposed method jointly trains a deep transformation as a representation, and a parametrized kernel function $K(vi, \\cdot)$ of the problem instances, at the meta-training stage. Experimental results validate the effectiveness of the proposed method. ", "questions": "It would be better if the authors could give a thorough discussion about their method’s superiority, as there are many meta-learning approaches that are in a similar style, that is, first training a meta model in the dataset containing sufficient information of the task space, then apply some task-specific procedures in the target task. It would be better if the authors could clearly state several specific points, at which their method is better than most of the others in the literature. ", "limitations": "No. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The motivation is strong, as the problem of meta-learning is a rather important problem in the community. The link between meta-learning and the theory of RKBS discussed in this paper may bring some new ideas.", "weaknesses": "My main concern comes from the fact that the proposed method also needs a meta-training procedure. At this point, the comparison in some experiments (for example, section 5.1, if there is no mis-understanding) seems somewhat unfair. Also, it would be better if there is some more comparison between the proposed method and some other meta-learning approaches. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your time and interest in reviewing our work. We would like to bring some answers and clarifications in relation to your comments, and hope they might decide you to increase your score. It seems that your concern is about not properly evaluating our approach against other meta-learning approaches, so we try to emphasize this aspect in the specific replies below.\n \n > Regarding _“[…] the proposed method jointly trains a deep transformation as a representation, and a kernel function […] of the problem instances”_. \n - We want to clarify that no vectorial representation is carried over from the meta-training procedure. Rather, the meta-optimization fit the kernel function $K: V \\times V \\mapsto U$ which is expressed as an iterative application of the intermediate parametric function $k_{\\theta}^{l}$ (equation 5). This kernel function is then directly applied to each new dataset along with query points to produce new output estimates.\n\n > Regarding _“the comparison in some experiments […] seems somewhat unfair.”_\n - We break down our reply in different arguments: First, we dedicated a whole experiment on a task previously used to benchmark meta-learning models in [1], namely MNIST-like datasets classification and compared against several state-of-the-art meta-learning systems, which directly provides evidence that the Transducer meta-learned adaptation transfer better to new tasks when meta-trained on the **same task distribution**. Second, in section 5.3, we show that our formalism can enhance the performance a regular supervised method on a large natural dataset, even though, it is again trained with the **exact same amount of data**. Third, in section 5.1 we emphasize that our original meta-learning approach can find better predictive solutions than previous neural operator approaches even when neural operators have access to large regimes of data (~1000 training trajectories). This is itself an interesting result as meta-learning systems have been very scarcely applied to the problem operator regression. To our knowledge, there is little literature on applying MAML or RNN-adaptation to neural operator. However, to convince you more of the interest of our approach, we are working on complementing section 5.1 with a comparison with a MAML-like adaptation procedure applied to FNO/DeepONet and will communicate the results as soon as possible. (Note however, that such adaptation technique is much slower than our feedforward adaptation process, see \"Speed\" below)\n\n > Regarding _“It would be better if the authors could give a thorough discussion about their method’s superiority […]”_\n - In the experimental section, we tried to provide different arguments showcasing the benefits of our method in several domains, against different class of models and along different features. We synthesize the main aspects below:\n   - **Accuracy**: First, our original transductive system can fit solution of operator regression problems at the level of accuracy of previous (gradient-based) neural operator regression approaches. (Section 5.1). Conversely, we show that our system is also a better meta-learner than several state-of-the-art meta-learning approaches on a task where they have been extensively benchmarked. (Section 5.4). Finally, we show that our philosophy can boost purely inductive systems in section 5.3.\n   - **Speed**: The adaptation mechanism of our model is parallel by nature and bypass the need for sequential adaptation such as in gradient-or-RNN-based adaptation. This in turn translate to significant gain in computation time, showcased in section 5.2 (outliers detection), where 5000 operator regression instances can be performed in under a few seconds while other meta-learning systems would need a much greater time because of their sequential adaptation process (I.e iterations of gradient descent for instance). This can potentially unlock new operator regression applications requiring fast sampling of multiple fitted models (such as bagging, conformal prediction or Monte-Carlo sampling in the operator space…). We will add a specific remark regarding this point in section 5.2.\n   - **Theoretical soundness and interpretability**: As you noted, our model builds on clear theoretical results regarding the existence and analytical form of the solutions to the considered regression problems. This is in contrast to few-shot gradient-based learning, which performs only a few gradient step (while paradoxically gradient descent is an asymptotic process by nature) and offer, in the general setting, weak guarantees regarding the quality of the found solutions. Furthermore, since our model takes explicitly as inputs the whole dataset, it offers a direct possibility, contrary to other approaches, to apply sensitivity analysis of the model with respect to each exemple for learning each task regression instance, which can be leverage to directly analyse to interpret model decisions. \n   - **Originality**: The original use of the theory of RKBS which generalise that of RKHS allows to encompass both operator (infinite-dimensional) and regular finite-dimensional regression problems in a single meta-regression framework and is, as such, worthy of exploration and analysis, along gradient-based or RNN-based adaptive systems, as it might open new discussion and connections with existing questions such as the interpretation in terms of kernel of in-context learning in attentional models.\n\nWe thank you again for your time and remain committed to answer any additional clarification that you might need and engage with you further during the discussion period.\n"}]

Author rebuttal:
We would like to thank again all three reviewers for their interest and precious feedback. We are happy that this work has been positively regarded by reviewers Sczv and f2BS while reviewer w6wT seemed less confident. We answered directly to specific remarks and questions in each review thread (see below) while integrating common suggestions to our text which we believe are strenghtening our proposal. 

As a synthesis, all three reviewers recognised that our work constituted an original proposal with a _"strong motivation"_ and _"novel ideas"_ that can potentially impact research at the intersection of neural operators and meta-learning. Reviewer Sczv noted a _"clear"_ presentation of ideas and reviewer f2BS liked "a nice framework and perspective"_. A central concern of reviewer w6wT was _"to give a thorough discussion about [our] method's superiority" against other meta-learning approaches. However this is specifically the aim of section 5.4 which compares our system against existing meta-learning approaches, while we demonstrate several original results in the other experimental sections: Namelly, In-context learning in infinite-dimensional spaces in section 5.1, outliers detection in section 5.2 and scaling/inductive model boosting in section 5.3. We are working on complementing our results to reviewer w6wT suggestion, but we note simultaneously that reviewer Sczv underscored that our _"experiments are varied ans cover a wide range of use case"_ and that reviewer f2BS remarked that we _"identify benefits"_ in all of them.

We will be happy to discuss further these points during the discussion period and remain commited to answer any additional question that may arise after our rebuttal.

The authors

Meta-review: The paper presents a quite interesting framework for meta-learning that uses neural networks to learn a function in an RKBS. The idea is that the neural operator learned in a RKBS will be mapping new training points to a new model. The proposed approach makes a clear connection to transformers. The paper presents several compelling applications of the proposed methodology.

The reviewers have asked a number of relevant questions, and I would like to encourage the authors to take them into account to prepare the final version of the manuscript (e.g. relation to other MetaL, computational complexity, etc). 

As an additional comment, I would like to encourage the authors to make section 3 less steep and more self contained: for a reader who has not read Zhang (2013), Theorem 1 and Theorem 2 are fairly abstract, and it is difficult to understand what the different mathematical objects are mean to represent in the context of learning without an explanation which would provide some intuition about which kind of object will be learned in which space etc, and a motivation for the structure imposed on these objects. This kind of explanation could probably given with a single additional paragraph before Theorem 1. For the reader not familiar with Zhang, 2013, the relevance of the condition in (3) of Theorem 2 are difficult to digest the way they are presented.
Decision: Accept (spotlight)