SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we show empirically that our meta-optimized model is able to approximate any element  $\mathcal{O}$  of diverse function spaces  $\mathcal{B}$  such as operators defined on scalar and vector-valued function spaces derived from parametric physical systems or regression problems in Euclidean spaces. In all experiments, we use the Adam optimizer (Kingma and Ba, 2014) to train for a fixed number of steps with an initial learning rate gradually halved along training. All the computation is carried on a single Nvidia Titan Xp GPU with 12GB memory. Further details can be found in S.I.
5.1 Regression of Advection-Diffusion Reaction PDEs
Figure 2: Left: RMSEs (and  $95\%$  C.I) on unseen operators as a function of the dataset size. The grey area corresponds to dataset cardinalities seen during the Transducer meta-training. For comparison, we train baselines from scratch with the corresponding number of examples. Middle: Training losses of Transducers with different depths. Applying several times the kernel improves performance. Untied weights yield the best performance. Right:  $(Up)$  3 examples of the evolution of  $s(x,t)$  for different ADR equations and (bottom) spatial MSEs of intermediate representations  $\left(\boldsymbol{u}^{\ell}\right)$  colored by iteration  $\ell$ . The decreasing error, consistent with the MSE reduction of deeper models, suggests that network depth allows for progressively refining function estimates.
First, we examine the problem of regressing operators  $\mathcal{O}$  associating functions  $\pmb{v}$  from  $\nu \subset C([0,1],\mathbb{R})$  to their solutions  $\pmb{u} = \mathcal{O}(\pmb{v}) \subset C([0,1],\mathbb{R})$  with respect to advection-diffusion-reaction equations defined on the domain  $\Omega = [0,1] \times [0,t]$  with Dirichlet boundary conditions  $\pmb{s}(0,t) = \pmb{s}(1,t) = 0$ . We consider the space  $\mathcal{B}$  of operators  $\mathcal{O}_{(\delta,\nu,k,t)}$  specifically defined by  $\pmb{v}(x) = \pmb{s}(x,0)$ ,  $\pmb{u}(x) = \pmb{s}(x,t)$  and  $\pmb{s}$  follows an equation depending on unknown random continuous spatially-varying diffusion  $\delta(x)$ , advection  $\nu(x)$ , and a scalar reaction term  $k \sim \mathcal{U}[0,0.1]$ :
$$
\partial_ {t} \boldsymbol {s} (x, t) = \underbrace {\nabla \cdot (\boldsymbol {\delta} (x) \nabla_ {x} \boldsymbol {s} (x , t))} _ {\text {d i f f u s i o n}} + \underbrace {\boldsymbol {\nu} (x) \nabla_ {x} \boldsymbol {s} (x , t)} _ {\text {a d v e c t i o n}} + \underbrace {\boldsymbol {k} \cdot (\boldsymbol {s} (x , t)) ^ {2}} _ {\text {r e a c t i o n}} \tag {8}
$$
Eq. (8) is generic with components arising in many physical systems of interest, leading to various forms of solutions  $s(x,t)$ . (We show examples for three different operators in figure 2.) Several methods exist for modeling such PDEs, but they require knowledge of the underlying parameters  $(\delta, \nu, k)$  and often impose constraints on the evaluation point as well as expensive time-marching schemes to recover solutions. Here instead, we assume no a priori knowledge of the solution and directly regress each operator  $\mathcal{O}$  behavior from the example set  $\mathcal{D}_{\mathcal{O}}$ .
METHOD	RMSE	TIME (s)	GFLOPs
FNO	2.96e-4	1.72e2	1.68e2
DEEPONET	2.02e-2	7.85e1	1.54e2
FNO-MAML	1.4e-1	2.10e0	1.6e-1
TRANSDUCER	2.39e-4	3.10e-3	1.06e-1
Table 1: RMSE and compute costs of regression over 50 unseen datasets with  $n = 50$  examples. Note that DeepONet and FNO are optimized from scratch while the Transducer and FNO-MAML have been pre-trained. GFLOPs represent the total number of floating point operations for regression.
Baselines and evaluation  $\diamond$  We meta-trained our model to regress 500 different operators  $\mathcal{O}_{(\delta, \nu, k, 1)}$  with  $t = 1$  fixed and varying number of examples  $n \in [20, 100]$  with images evaluated at 100 equally spaced points  $(x_k)_{k \in [0, 100]}$  on the domain  $[0, 1]$  and meta-tested on a set of 500 operators with new parameters  $\delta, \nu, k$  and initial states  $\pmb{v}$ . Although not directly equivalent to existing approaches, we compared our method with standard regression methods as well as inductive neural operator approximators. We applied standard finite-dimensional regression methods,  $K$ -Nearest-Neighbors (Fix and Hodges, 1989), Decision Trees (Quinlan, 1986) and Ridge regression with radial basis kernel (Hastie et al., 2009) to each discretized problems  $\left(\{\mathcal{O}(\pmb{v}_j)(x_k) = \pmb{u}_j(x_k)\}_{j,k}\right)$  as well as two neural-based operators to each dataset instance: DeepONet (Lu et al., 2021) and FNO
Figure 3: Example of Transducer regression extrapolation and RMSEs on OOD tasks with  $n = 100$  examples. Color code corresponds to different correlation lengths used to generate the random functions  $\delta(x)$  and  $\nu(x)$ . Much of the result remains below  $1\%$  error despite never being trained on such operators.
(Li et al., 2020a). Finally, we also tried to meta-learn a parametrization of FNO that could adapt in 100 gradient steps following MAML (Finn et al., 2017) using the same meta-dataset. For all but this approach, an explicit optimization problem is solved before inference in order to fit the target operator. On the other hand, after meta-training of the Transducer, which takes only a few minutes to converge, each regression is solved in a single feedforward pass of the network, which is orders of magnitude faster and can be readily applied to new problems (Table 1).
Results  $\diamond$  We first verified that our model approximates well unseen operators from the test set (Table 1). We noted that our model learns a non-trivial kernel since the estimation produced with  $\ell_2$ -Nearest Neighbors remains poor even after  $1e^3$  examples. Moreover, since our model can perform inference for varying input dataset sizes, we examined the Transducer accuracy when varying the number of examples and found that it learns a converging regression program (Figure 2) which consistently outperforms other instance-specific regression approaches with the exception of FNO when enough data is available ( $>60$ ). We also found that deeper Transducer models with more layers increase kernel approximation accuracy, with untied weights yielding the best performance (figure 2.)
Extrapolation to OOD tasks  $\diamond$  We further tested the Transducer ability to regress different operators than those seen during meta-training. Specifically, we varied the correlation length (C.L) of the Gaussian processes used to generate functions  $\delta(x)$  and  $\nu(x)$  and specified a different target time  $t' \neq 1$ . We showed that the kernel meta-optimized for a solution at  $t = 1$  transfers well to these new regression problems and that regression performance degrades gracefully as the target operators behave further away from the training set (figure 3), while inductive solutions do not generalize.
5.2 Outliers detection on 2D Burgers' equation
We further show that our regression method can fit operators of vector-valued functions by examining the problem of predicting 2D vector fields defined as a solution of a two-dimensional Burgers' equation with periodic spatial boundary condition on the domain  $\Omega = [0,1]^2\times [0,10]$ :
$$
\partial_ {t} \boldsymbol {s} (\vec {\boldsymbol {v}}, t) = \underbrace {\boldsymbol {\nu} \Delta_ {\boldsymbol {v}} \cdot \boldsymbol {s} (\vec {\boldsymbol {v}} , t)} _ {\text {d i f f u s i o n}} - \underbrace {\boldsymbol {s} (\vec {\boldsymbol {v}} , t) \nabla_ {\boldsymbol {x}} \boldsymbol {s} (\vec {\boldsymbol {v}} , t)} _ {\text {a d v e c t i o n}} \tag {9}
$$
Here, we condition our model with operators of the form,  $\pmb{v}(\vec{x}) = s(\vec{x},t)$ ,  $\pmb{u}(\vec{x}) = s(\vec{x},t')$  such that our model can regress the evolution of the vector field  $\vec{v}$  starting at any time, with arbitrary temporal increment  $t' - t \leqslant 10$  seconds and varying diffusion coefficient  $\nu \in [0.1,0.5]$ . We show in figure (4) and table (2) that our model is able to fit new instances of this problem with unseen parameters  $\nu$ .
Figure 4: Illustrative example of initial  $(t = 0)$ , target  $(t = 10)$  and Transducer estimation of the vector field  $s(\vec{x}, t)$  discretized at resolution  $64 \times 64$  over the domain  $[0, 1]^2$  for the Burgers' equation experiment. The last panel represents absolute error to ground truth.
Fast and differentiable regression  $\diamond$  Since the fitting operation is orders of magnitude faster than other operator regression approaches as well as fully differentiable, it allows for quickly executing expensive schemes requiring multiple regressions. This can have several applications, from bootstrapping or producing confidence intervals by varying the example set  $\mathcal{D}_{\mathcal{O}}^{train}$ , or performing inverse problems using Monte-Carlo Markov Chain in the dataset space. We showcase an example of this potential with an outlier detection experiment: We use the Transducer to identify outliers of a dataset of Burgers' equation with coefficient  $\nu_{1}$  artificially contaminated with elements from another dataset  $\nu_{2} > \nu_{1}$  at  $5\%$  level. We identify outliers by estimating RMSEs over 5000 different regressions using random  $50\%$  splits with outliers potentially present in both training and testing sets. This technique takes only a few seconds to estimate while outliers are clearly identified as data points with significantly higher RMSE than the dataset average (figure 5). As a comparison, performing Spectral Clustering (Yu and Shi, 2003) on the FFT of elements  $(u_{i})$  yields very poor precision (table 2)
t = 5s	t = 10s
RMSE (test sets)	2.2e-3	5.9e-3
Outliers (Pre./Rec.)	100%/100%	100%/100%
S.C. (Pre./Rec.)	6%/85%	7%/85%
Table 2 & Figure 5: Left: Meta-test regression and outlier detection results at two target times. RMSEs on Burgers' equations averaged over 200 different parameter conditions  $\nu \in [0.1, 0.5]$  each with 100 train examples. Precision/Recall in outlier detection of the Transducer versus Spectral clustering. Right: RMSE distributions of each element in the contaminated dataset over the 5000 regressions. Outliers are clearly identified.
Figure 6: Up - Illustrative examples of  $720 \times 720$  temperature (left) and pressure (right) fields of the ERA5 dataset. Bottom - Estimated pressure field from conditioning the Transducer with 15 days data dating 1 week before the target date. Insets show recovered details of the estimation (blue) compared with ground truth (red).
5.3 Climate modeling with seasonal adaptation
One advantage of our approach is the ability to select the data that is most relevant with respect to a certain prediction task and subsequently adapt the model response. For instance, robust and precise prediction of climate variables is difficult because models need to account for seasonal variability and adapt to drifting parameters. Even with a globally large amount of available data, the underlying operator of interest might change over time or be affected by unobserved phenomena. Hence, in order to fully exploit the potential of data-driven methods, being able to capture such variations might greatly help prediction performance on fluctuating and drifting data distributions. In order to illustrate the applicability and scalability of deep transductive learning, we considered the problem of predicting the Earth's surface air pressure solely from the Earth's surface air temperature at a high resolution. Data is taken from the ERA5 reanalysis (Hersbach et al., 2020) publicly made available by the ECMWF, which consists of hourly high-resolution estimates of multiple atmospheric variables from 1979 to the current day. We model pressure estimate on a  $720 \times 720$  grid, resulting in a spatial resolution of  $0.25^{\circ} \times 0.5^{\circ}$ , allowing us to capture small features such as local dynamics and geographic relief.
Similar to (Pathak et al., 2022), we modify a ViT backbone to incorporate a kernel transduction layer before every patch attention and compare our model to an unmodified ViT baseline with a matching number of parameters. We additionally compare with a fully transductive Nearest Neighbors approach. In Figure 6 and Table 3, we present results obtained on training a
METHOD	LWMSE (HPA)	TIME (s)
NEAREST-NEIGHBORS	67.326	5.915
VIT	32.826	0.053
TRANSDUCER - (P.Y)	25.293	0.192
TRANSDUCER - (P.W)	22.718	0.192
Table 3: Latitude-weighted mean-square error (in hectopascals) and inference time for the earth surface pressure prediction task.
Transducer with data from 2010 to 2014 and testing it on data from 2016 to 2019. We trained our model by predicting 5 random days sampled from random 20-day windows and present two test configurations: We either condition the Transducer with a window centered at the previous year's same date (P.Y) or with a 15 days window lagging by a week (P.W) (see SI for details). Both cases outperform transductive and inductive baselines with fast inference time, confirming that our solution can scale to large problems and be combined with other deep learning modules.
5.4 Finite-dimensional case: MNIST-like datasets classification
We finally confirm the generality of our approach in the case of finite-dimensional spaces  $\mathcal{U}$  and  $\mathcal{V}$  by studying the meta-learning problem presented in Kirsch et al. (2022) which consists in regressing classification functions from the 784-dimensional space of MNIST-like images (LeCun
METHOD	MNIST	FASHIONMNIST	KMNIST
MAML (FINN ET AL., 2017)	53.71%	48.44%	36.33%
VSML (KIRSCH ET AL., 2021)	79.04%	68.49%	54.69%
GPICL (KIRSCH ET AL., 2022)	73.70 %	62.24%	53.39%
TRANSDUCER	81.83%	69.85%	60.64%
Figure 7: Comparison of meta-test accuracies of MNIST-like datasets classification tasks. Meta-learning models are trained on transformations of MNIST and are meta-tested on original MNIST, FashionMNIST and KMNIST
and Cortes, 2010) to a 10-dimensional space of one-hot class encoding (i.e. functions considered are  $\mathcal{O}:[0,1]^{784}\mapsto [0,1]^{10}$ ). We meta-train a 2-layer Transducer to classify consistently pixel-permuted and class-permuted versions of MNIST. We then meta-test the Transducer to classify the unpermuted MNIST dataset and how the regression map transfer to Fashion-MNIST (Xiao et al., 2017) and KMNIST (Clanuwat et al., 2018). We show in Table 7 that the Transducer outperforms previous meta-learning approaches on both the original MNIST classification task as well as transfer better on Fashion MNIST and K-MNIST classification tasks.
Transductive Machine learning  $\diamond$  Principles of transductive statistical estimation have been formally described in Gammerman et al. (1998); Vapnik (1999). Algorithms relying on relational structures between data points such as  $K$ -nearest neighbors (Cover and Hart, 1967) and kernel methods (Nadaraya, 1964; Watson, 1964) build estimates by weighing examples with respect to a certain metric space. Further, the "kernel trick" allows to embed possibly infinite-dimensional features (Ferraty and Vieu, 2006) into finite Gram matrix representations that are also well-suited for multi-task regression (Evgeniou et al., 2005; Caponnetto et al., 2008). Distinctively, Gaussian processes regression (Williams and Rasmussen, 1995) combines transduction with Bayesian modeling to estimate a posterior distribution over possible functions. These techniques might suffer from the so-called "curse of dimensionality": with growing dimensionality, the density of exemplar point diminishes, which increases estimators' variance. More recent work combining deep learning with transductive inference has shown promising results even in high-dimensional spaces for few-shot learning (Snell et al., 2017; Sung et al., 2018) or sequence modeling (Jaitly et al., 2015), but the vast majority of neural networks still remain purely inductive.
Neural operator learning  $\diamond$  The universal approximation abilities of neural networks have been generalized to infinite-dimensional function spaces: Chen and Chen (1995) showed that finite neural parametrization can approximate well infinite-dimensional operators. More recent work using neural networks to perform operator regression has shown strong results (Lu et al., 2019), especially when mixed with tools from functional analysis and physics (Raissi et al., 2017; Li et al., 2020a; Gupta et al., 2021; Li et al., 2020b; Nelsen and Stuart, 2021; Wang et al., 2021; Roberts et al., 2021) and constitutes a booming research direction in particular for physical applications (Goswami et al., 2022; Pathak et al., 2022; Vinuesa and Brunton, 2022; Wen et al., 2022; Pickering et al., 2022). Recently, the Transformer's attentional computation has been interpreted as a Petrov-Galerkin projection (Cao, 2021) or through Reproducing Kernel Hilbert Space theory (Kissas et al., 2022) for building such neural operators, but these perspectives apply attention to fit a single target operator.
Meta-learning and in-context learning  $\diamond$  Promising work towards more general and adaptable machines has consisted in automatically "learning to learn" or meta-learning programs (Schmidhuber et al., 1997; Vilalta and Drissi, 2002), by either explicitly treating gradient descent as an estimizable object (Finn et al., 2017), modeling an optimizer as a black-box autoregressive model (Ravi and Larochelle, 2017) or informing sequential strategies via memorization (Santoro et al., 2016; Ortega et al., 2019) More recently, converging findings in various domains from reinforcement learning (Mishra et al., 2018; Laskin et al., 2022), natural language processing (Brown et al., 2020; Xie et al., 2021; Olsson et al., 2022) and functional regression (Garg et al., 2022) have established the ability of set-based attentional computation in the Transformer (Vaswani et al., 2017) for in-context learning by flexibly extracting functional relationships and performing dynamic association such as linguistic analogy or few-shot behavioral imitation. We show that the theory of RKBS can help interpret such property and extends it to function-valued operators regression.
We proposed a novel transductive model combining kernel methods and neural networks that is capable of performing regression over entire function spaces. We based our model on the theory of vector-valued Reproducing Kernel Banach Spaces and showcased several instances where it learns a regression program able, in a single feedforward pass, to reach performance levels that match or outperform previous instance-specific neural operators or meta-learning systems. Our approach holds potential to create programs flexibly specified by data and able to model entire families of complex physical systems, with particular applications in functional hypothesis testing, dataset curation or fast ensemble learning. However, one limitation is that our model relies on meta-training, which requires collecting a sufficiently diverse meta-dataset to explore the kernel space. In future work, we plan to investigate methods such as synthetic augmentation to reduce the costs of meta-training.
This work was funded by ANR-3IA Artificial and Natural Intelligence Toulouse Institute (ANR-19-PI3A0004) ONR (N00014-19-1-2029), NSF (IIS-1912280 and EAR-1925481), DARPA (D19AC00015) and NIH/NINDS (R21 NS 112743). Additional support provided by the Carney Institute for Brain Science and the Center for Computation and Visualization (CCV) and the NIH Office of the Director grant S10OD025181.
Vilalta, R. and Drissi, Y. (2002). A perspective view and survey of meta-learning. Artif. Intell. Rev., 18(2):77-95.
Vinuesa, R. and Brunton, S. L. (2022). Enhancing computational fluid dynamics with machine learning. Nature Computational Science, 2(6):358-366.
Wang, S., Wang, H., and Perdikaris, P. (2021). Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science Advances, 7(40):eabi8605.
Watson, G. S. (1964). Smooth regression analysis. *Sankhya: The Indian Journal of Statistics*, Series A, pages 359-372.
Wen, G., Li, Z., Azizzadenesheli, K., Anandkumar, A., and Benson, S. M. (2022). U-fno—an enhanced fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources, 163:104180.
Williams, C. and Rasmussen, C. (1995). Gaussian processes for regression. Advances in neural information processing systems, 8.
Wright, M. A. and Gonzalez, J. (2021). Transformers are deep infinite-dimensional non-mercer binary kernel machines. ArXiv, abs/2106.01506.
Xiao, H., Rasul, K., and Vollgraf, R. (2017). Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.
Xie, S. M., Raghunathan, A., Liang, P., and Ma, T. (2021). An explanation of in-context learning as implicit bayesian inference.
Xu, Y. and Ye, Q. (2019). volume 258. American Mathematical Society.
Yu and Shi (2003). Multiclass spectral clustering. In Proceedings Ninth IEEE International Conference on Computer Vision, pages 313-319 vol.1.
Zhang, H. (2013). Vector-valued reproducing kernel banach spaces with applications to multi-task learning. Journal of Complexity, 29(2):195-215.
Zhang, H., Xu, Y., and Zhang, J. (2009). Reproducing kernel banach spaces for machine learning. Journal of Machine Learning Research, 10(12).