SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mathieu Chalvidal
Capital Fund Management Paris, France
mathieu.chalvidal@gmail.com
Thomas Serre
Carney Institute for Brain Science Brown University, U.S.
thomas_serre@brown.edu
Rufin VanRullen  
Centre de Recherche Cerveau & Cognition CNRS, Universite de Toulouse, France  
rufin.vanrullen@cnrs.fr
Research in statistical learning has polarized into two general approaches to perform regression analysis: Transductive methods construct estimates directly based on exemplar data using generic relational principles which might suffer from the curse of dimensionality. Conversely, inductive methods can potentially fit highly complex functions at the cost of compute-intensive solution searches. In this work, we leverage the theory of vector-valued Reproducing Kernel Banach Spaces (RKBS) to propose a hybrid approach: We show that transductive regression systems can be meta-learned with gradient descent to form efficient in-context neural approximators of function defined over both finite and infinite-dimensional spaces (operator regression). Once trained, our Transducer can almost instantaneously capture new functional relationships and produce original image estimates, given a few pairs of input and output examples. We demonstrate the benefit of our meta-learned transductive approach to model physical systems influenced by varying external factors with little data at a fraction of the usual deep learning training costs for partial differential equations and climate modeling applications.
Transduction vs. induction In statistical learning, transductive inference (Vapnik, 2006) refers to the process of reasoning directly from observed (training) cases to new (testing) cases and contrasts with inductive inference, which amounts to extracting general rules from observed training cases to produce estimates. The former principle powers some of the most successful regression algorithms benefiting from straightforward construction properties, from  $k$ -Nearest Neighbors (Cover and Hart, 1967) to Support Vector Machines (Boser et al., 1992) or Gaussian Processes (Williams and Rasmussen, 1995). In contrast, deep learning research has mostly endeavored to find inductive solutions, relying on stochastic gradient descent to faithfully encode functional relationships described by large datasets into the weights of a neural network. Although ubiquitous, inductive neural learning with gradient descent is compute-intensive, necessitates large amounts of data, and poorly generalizes outside of the training distribution (Jin et al., 2020) such that a slight modification of the problem might require retraining and cause "catastrophic" forgetting of the previous solution (McCloskey and Cohen, 1989). This may be particularly problematic for real-world applications where data has heterogeneous sources, or only a few examples of the target function are available.
Figure 1: Batches of functional images  $\mathcal{T}_{\theta}(\mathcal{D}_{\mathcal{O}_i})(\boldsymbol{v}_j) \approx \mathcal{O}_i(\boldsymbol{v}_j) = \boldsymbol{u}_j(x,t) \in C([0,1]^2,\mathbb{R})$  obtained with the same Transducer model  $\mathcal{T}_{\theta}$  but conditioned, at each row, by a different dataset  $(\mathcal{D}_{\mathcal{O}_i})_{i \leqslant 3}$  during feedforward computation. Each underlying operator  $\mathcal{O}_i$  corresponds to a different advection-diffusion-reaction equation (defined in Sec. 5.1) with spatially varying advection, diffusion, and reaction parameters unseen during training, and functions  $(\boldsymbol{v}_j)_{j \leqslant 9}$  correspond to initial conditions. While usual neural regression approaches learn a single target function (one row), our model learns to approximate instantaneously an infinity of them.
Meta-learning functional regression In this work, we meta-learn a regression program in the form of a neural network able to approximate an infinity of functions defined on finite or infinite-dimensional spaces through a transductive formulation of the solution based on the representer theorem. Namely, our model is meta-trained to take as input any dataset  $\mathcal{D}_{\mathcal{O}}$  of pairs  $(\pmb{v}_i,\mathcal{O}(\pmb{v}_i))_{i\leqslant I}$  of some target function  $\mathcal{O}$  together with a query element  $\pmb{v^{\prime}}$  and produces directly an estimate of the image  $\mathcal{O}(\pmb{v}^{\prime})$ . After meta-training, our network is able to perform regression of unseen operators  $\mathcal{O}'$  from varying dataset sizes in a single feedforward pass, such that our model can be interpreted as performing in-context functional learning. In order to build such a model, we leverage the theory of Reproducing Kernel Banach Spaces (RKBS) (Micchelli and Pontil, 2004; Zhang, 2013; Lin et al., 2022) and interpret the Transformer's (Vaswani et al., 2017) attention mechanism as a parametric vector-valued reproducing kernel. While kernel regression might be plagued by the "curse of dimensionality" (Bellman, 1966; Aggarwal et al., 2001), we show that our meta-learning approach can escape this pitfall, allowing, for instance, to perform instantaneous regressions over spaces of operators from a few example points, by building solutions to regression problem instances directly from the general reproducing kernel associated with such spaces.
Contributions  $\diamond$  We introduce the Transducer, a novel meta-learning approach leveraging reproducing kernel theory and deep learning methods to perform instantaneous regression of an infinity of functions in reproducing kernel spaces.
- Our model learns an implicit regression program able to identify, in a single feedforward pass, elements of specific functional spaces from any corresponding collection of input-output pairs describing the target function. Such ultra-fast regression program, which bypasses the need for gradient-based training, is also general and can be applied to functions either defined on finite dimensional spaces (scalar-valued function spaces) or infinite dimensional spaces (function-valued operator spaces).
- In particular, we demonstrate the flexibility and efficiency of our framework for fitting function-valued operators in two PDEs and one climate modeling problem. We show that our transductive approach allows for better generalization properties of neural operator regression, better precision when relevant data is available, and can be combined with iterative regression schemes that are too expensive for previous inductive approaches, thus holding potential to improve neural operators applicability.
- To the best of our knowledge, our proposal is the first to marry vector-valued RKBS theory with deep meta-learning and might also shed new light on the in-context learning abilities observed in deep attentional architectures.
Let  $\mathcal{V}$  and  $\mathcal{U}$  be two (finite or infinite-dimensional) Banach spaces, respectively referred to as the input and output space, and let  $\mathcal{B}$  a Banach space of functions from  $\mathcal{V}$  to  $\mathcal{U}$ . We also note  $L(\mathcal{U},\mathcal{B})$  (resp.  $L(\mathcal{U})$ ) the set of bounded linear operators from  $\mathcal{U}$  to  $\mathcal{B}$  (resp. to itself). We consider the meta-learning problem of creating a function  $\mathcal{T}$  able to approximate any functional element  $\mathcal{O}$  in the space  $\mathcal{B}$  from any finite collection of example pairs  $\mathcal{D}_{\mathcal{O}} = \{(v_i,u_i)\mid v_i\in \mathcal{V},u_i = \mathcal{O}(v_i)\}_{i\leqslant n}$ . A prominent approach in statistical learning is empirical risk minimization which consists in predefining a class  $\tilde{\mathcal{B}}\subset \mathcal{B}$  of computable functions from  $\mathcal{V}$  to  $\mathcal{U}$  and subsequently selecting a model  $\tilde{\mathcal{O}}$  as a minimizer (provided its existence) of a risk function  $\mathcal{L}: \mathcal{B}\times \mathcal{D}\mapsto \mathbb{R}$ :
$$
\mathcal {T} \left(\mathcal {D} _ {\mathcal {O}}\right) \in \underset {\tilde {\mathcal {O}} \in \tilde {\mathcal {B}}} {\operatorname {a r g m i n}} \mathcal {L} \left(\tilde {\mathcal {O}}, \mathcal {D} _ {\mathcal {O}}\right) \tag {1}
$$
For instance, the procedure consisting in performing gradient-based optimization of objective (1) over a parametric class  $\tilde{\mathcal{B}}$  of neural networks defines implicitly such a function  $\mathcal{T}$ . Fundamentally, this technique works by induction: It captures the statistical regularities of a single map  $\mathcal{O}$  into the parameters of the neural network  $\tilde{\mathcal{O}}$  such that  $\mathcal{D}_{\mathcal{O}}$  is discarded for inference. Recent examples of gradient-based optimization of neural networks for operator regression (i.e when  $\mathcal{V}$  and  $\mathcal{U}$  are infinite-dimensional) are DeepOnet (Lu et al., 2019) or Fourier Neural Operator (FNO) (Li et al., 2020a). As previously discussed, for every regression problem instance, evaluating  $\mathcal{T}$  with these approaches requires a heavy training procedure. Instead, we show in this work that for specific spaces  $\mathcal{B}$ , we can meta-learn a parametric map  $\mathcal{T}_{\theta}$  that transductively approximates (in a certain functional sense) any target function  $\mathcal{O} \in \mathcal{B}$  given a corresponding dataset  $\mathcal{D}_{\mathcal{O}}$  such that:
$$
\forall \boldsymbol {v} \in \mathcal {V}, \quad \mathcal {T} (\mathcal {D} _ {\mathcal {O}}) (\boldsymbol {v}) = \mathcal {T} _ {\boldsymbol {\theta}} (\boldsymbol {v} _ {1}, \mathcal {O} (\boldsymbol {v} _ {1}), \dots , \boldsymbol {v} _ {n}, \mathcal {O} (\boldsymbol {v} _ {n}), \boldsymbol {v}) \approx \mathcal {O} (\boldsymbol {v}) \tag {2}
$$
3 Vector-valued Reproducing Kernel Banach Space regression
In order to build  $\mathcal{T}_{\theta}$ , we leverage the structure of Reproducing Kernel Banach Spaces (RKBS) of functions  $\mathcal{B}$  and combine it with the universal approximation abilities of deep networks. As we will see in the experimental section, RKBS are very general spaces occurring in a wide range of machine learning applications. We start by recalling some elements of the theory of vector-valued RKBS developed in Zhang (2013). Namely, we will consider throughout uniform Banach spaces  $\mathcal{S}$  (such condition guarantees the unicity of a compatible semi-inner product  $\langle ., . \rangle_{\mathcal{S}}: \mathcal{S} \times \mathcal{S} \mapsto \mathbb{R}$ , i.e.  $\forall s \in \mathcal{S}, \langle s, s \rangle_{\mathcal{S}} = ||s||_{\mathcal{S}}^2$  and allows to build a bijective and isometric dual space  $\mathcal{S}^*)$ ).
Theorem 1 (Vector-valued RKBS (Zhang, 2013)). A  $\mathcal{U}$ -valued reproducing kernel Banach space  $\mathcal{B}$  of functions from  $\mathcal{V}$  to  $\mathcal{U}$  is a Banach space such that for all  $\pmb{v} \in \mathcal{V}$ , the point evaluation  $\delta_{\pmb{v}}: \mathcal{B} \mapsto \mathcal{U}$  defined as  $\delta_{\pmb{v}}(\mathcal{O}) = \mathcal{O}(\pmb{v})$  is continuous. In this case, there exists a unique function  $\mathcal{K}: \mathcal{V} \times \mathcal{V} \mapsto L(\mathcal{U})$  such that for all  $(\pmb{v}, \pmb{u}) \in \mathcal{V} \times \mathcal{U}$ :
$$
\left\{ \begin{array}{l} \boldsymbol {v} ^ {\prime} \mapsto \mathcal {K} (\boldsymbol {v}, \boldsymbol {v} ^ {\prime}) (\boldsymbol {u}) \in \mathcal {B} \\ \forall \mathcal {O} \in \mathcal {B}, \langle \mathcal {O} (\boldsymbol {v}), \boldsymbol {u} \rangle_ {\mathcal {U}} = \langle \mathcal {O}, \mathcal {K} (\boldsymbol {v},.) (\boldsymbol {u}) \rangle_ {\mathcal {B}} \\ \forall \boldsymbol {v} ^ {\prime} \in \mathcal {V}, \| \mathcal {K} (\boldsymbol {v}, \boldsymbol {v} ^ {\prime}) \| _ {L (\mathcal {U})} \leqslant \| \delta_ {\boldsymbol {v}} \| _ {L (\mathcal {B}, \mathcal {U})} \| \delta_ {\boldsymbol {v} ^ {\prime}} \| _ {L (\mathcal {B}, \mathcal {U})} \end{array} \right. \tag {3}
$$
Informally, theorem (1) states that RKBS are spaces sufficiently regular such that the image of any element  $\mathcal{O}$  at a given point  $\pmb{v}$  can be expressed in terms of a unique function  $\kappa$ . The latter is hence called the reproducing kernel of  $\mathcal{B}$  and our goal is to leverage such unicity to build the map  $\mathcal{T}_{\theta}$ . Let  $\mathcal{D}$  be the set of all datasets  $\mathcal{D}_{\mathcal{O}}$  previously defined. The following original theorem gives the existence of a solution to our meta-learning problem and relates it to the reproducing kernel.
Theorem 2 (RKBS representative map). Let  $\mathcal{B}$  be a  $\mathcal{U}$ -valued RKBS from  $\mathcal{V}$  to  $\mathcal{U}$ , if for any dataset  $\mathcal{D}_{\mathcal{O}} \in \mathcal{D}$ ,  $\mathcal{L}(.,\mathcal{D}_{\mathcal{O}})$  is lower semi-continuous, coercive and bounded below, then there exists a function  $\mathcal{T}: \mathcal{D} \mapsto \mathcal{B}$  such that  $\mathcal{T}(\mathcal{D}_{\mathcal{O}})$  is a minimizer of equation (1). If  $\mathcal{L}$  is of the form  $\mathcal{L}(.,\mathcal{D}_{\mathcal{O}}) = \tilde{\mathcal{L}} \circ \{\delta_{v_i}\}_{i \leqslant n}$  with  $\tilde{\mathcal{L}}: \mathcal{U}^n \mapsto \mathbb{R}$ , then the dual  $\mathcal{T}(\mathcal{D}_{\mathcal{O}})^*$  is in  $\overline{\text{span}}\{\mathcal{K}(v_i,.)(\boldsymbol{u})^*, i \leqslant n, \boldsymbol{u} \in \mathcal{U}\}$ . Furthermore, if for any  $\mathcal{D}_{\mathcal{O}}$ ,  $\mathcal{L}(.,\mathcal{D}_{\mathcal{O}})$  is strictly-convex, then  $\mathcal{T}$  is unique.
While theorem (2) provides conditions for the existence of solutions to each regression problem defined by (1), the usual method consisting in solving instance-specific minimization problems
derived from representer theorems characterizations is generally intractable in RKBS for several reasons (non-convexity and infinite-dimensionality of the problem w.r.t to variable  $\pmb{u}$ , non-additivity of the underlying semi-inner product). Instead, we propose to define image solutions  $\mathcal{T}(\mathcal{D}_{\mathcal{O}}) = \sum_{i\leqslant n}\mathcal{K}_{\pmb{\theta}}(\pmb{v}_i,.)(\tilde{\pmb{u}}_i)$  where  $\kappa_{\theta}$  and  $(\tilde{u}_i)$  are respectively the learned approximation of the  $\mathcal{U}$ -valued reproducing kernel  $\kappa$  and a set of functions in  $\mathcal{U}$  resulting from a sequence of deep transformations of image examples  $(\pmb{u}_i)$  that we define below.
Transformers attention as a reproducing kernel  $\diamond$  We first need to build  $\kappa$ . Several pieces of work have proposed constructions of  $\kappa$  in the context of a non-symmetric and nonpositive semi-definite real-valued kernel (Zhang et al., 2009; Georgiev et al., 2014; Lin et al., 2019; Xu and Ye, 2019). In particular, the exponential key-query function in the popular Transformer model (Vaswani et al., 2017) has been interpreted as a real-valued reproducing kernel  $\kappa_{\theta}: \mathcal{V} \times \mathcal{V} \mapsto \mathbb{R}$  in Wright and Gonzalez (2021). We extend below this interpretation to more general vector-valued RKBS:
Proposition 1 (Dot-product attention as  $\mathcal{U}$ -valued reproducing kernel). Let  $(p_j)_{j\leqslant J}$  a finite sequence of strictly positive integers, let  $(A_{\theta}^{j})_{j\leqslant J}$  be applications from  $\mathcal{V}\times \mathcal{V}$  to  $\mathbb{R}$ , let  $V_{\theta}^{j}$  be linear applications from  $L(\mathcal{U},\mathbb{R}^{p_j})$  and  $W_{\theta}$  a linear application from  $L(\prod_{j\leqslant J}\mathbb{R}^{p_j},\mathcal{U})$ , the (multi-head) application  $\kappa_{\theta}:\mathcal{V}\times \mathcal{V}\mapsto L(\mathcal{U})$  defined by
$$
\boldsymbol {\kappa} _ {\boldsymbol {\theta}} \left(\boldsymbol {v}, \boldsymbol {v} ^ {\prime}\right) (\boldsymbol {u}) \triangleq W _ {\boldsymbol {\theta}} \left(\left[ \dots , A _ {\boldsymbol {\theta}} ^ {j} \left(\boldsymbol {v}, \boldsymbol {v} ^ {\prime}\right) \cdot V _ {\boldsymbol {\theta}} ^ {j} (\boldsymbol {u}), \dots \right] _ {j \leqslant J}\right) \tag {4}
$$
is the reproducing kernel of an  $\mathcal{U}$ -valued RKBS. In particular, if  $\mathcal{U} = \mathcal{V} = \mathbb{R}^p$ , for  $p \in \mathbb{N}^+$  and  $A_{\boldsymbol{\theta}}^{j} = \exp \left(\frac{1}{\tau}(Q_{\boldsymbol{\theta}}^{j}\boldsymbol{v})^{T}(K_{\boldsymbol{\theta}}^{j}\boldsymbol{v}^{\prime})\right) / \sigma(\boldsymbol{v},\boldsymbol{v}^{\prime})$  with  $(Q_{\boldsymbol{\theta}}^{j},K_{\boldsymbol{\theta}}^{j})_{j \leqslant J}$  applications from  $L(\mathcal{V},\mathbb{R}^d)$ ,  $\kappa_{\boldsymbol{\theta}}$  corresponds to the dot-product attention mechanism of Vaswani et al. (2017).
Note that in (4), the usual softmax normalization of the dot-product attention is included in the linear operations  $A_{\theta}^{j}$  through  $\sigma$ . We show in the next section how such kernel construction can be leveraged to build the map  $\mathcal{T}_{\theta}$  and that several variations of the kernel construction are possible, depending on the target space  $\mathcal{B}$  and applications. Contrary to usual kernel methods, our model jointly builds the full reproducing kernel approximation  $\kappa_{\theta}$  and the instance-specific parametrization  $(\tilde{\boldsymbol{u}}_i)_{i\leqslant 1}$  by integrating the solutions iteratively over several residual kernel transformations. We refer to our system as a Transducer, both as a tribute to the Transformer computation mechanism from which it is inspired and by analogy with signal conversion devices.
4 The Transducer
Model definition  $\diamond$  We define  $\mathcal{T}_{\theta}$  as the sum of  $L$  residual kernel transformations  $\{\kappa_{\theta}^{\ell}\}_{\ell \leqslant L}$  whose expression can be written:
$$
\forall \boldsymbol {v} \in \mathcal {V}, \quad \mathcal {T} _ {\boldsymbol {\theta}} (\mathcal {D} _ {\mathcal {O}}) (\boldsymbol {v}) = \sum_ {i \leqslant I} \mathcal {K} _ {\boldsymbol {\theta}} (\boldsymbol {v} _ {i}, \boldsymbol {v}) (\tilde {\boldsymbol {u}} _ {i}) = \sum_ {i \leqslant I} \sum_ {\ell \leqslant L} \boldsymbol {\kappa} _ {\boldsymbol {\theta}} ^ {\ell} (\boldsymbol {v} _ {i} ^ {\ell}, \boldsymbol {v} ^ {\ell}) (\boldsymbol {u} _ {i} ^ {\ell}) \tag {5}
$$
where  $(\pmb{v}_i^\ell, \pmb{u}_i^\ell)_{i \leqslant n, l \leqslant L}$  and  $(\pmb{v}^\ell)_{l \leqslant L}$  refer to sequences of representations starting respectively with  $(\pmb{v}_i^1, \pmb{u}_i^1)_{i \leqslant n} = \mathcal{D}_{\mathcal{O}}, \pmb{v}^1 = \pmb{v}$  and defined by the following recursive relation:
$$
\left\{ \begin{array}{l} \boldsymbol {v} _ {i} ^ {\ell + 1} = F _ {\boldsymbol {\theta}} ^ {\ell} \left(\boldsymbol {v} _ {i} ^ {\ell}\right), \quad \boldsymbol {v} ^ {\ell + 1} = F _ {\boldsymbol {\theta}} ^ {\ell} \left(\boldsymbol {v} ^ {\ell}\right) \\ \boldsymbol {u} _ {i} ^ {\ell + 1} = \tilde {\boldsymbol {u}} _ {i} ^ {\ell} + \sum_ {j} \kappa_ {\boldsymbol {\theta}} ^ {\ell} \left(\boldsymbol {v} _ {j} ^ {\ell + 1}, \boldsymbol {v} _ {i} ^ {\ell + 1}\right) \left(\tilde {\boldsymbol {u}} _ {j} ^ {\ell}\right) \text {w h e r e} \tilde {\boldsymbol {u}} _ {i} ^ {\ell} = G _ {\boldsymbol {\theta}} ^ {\ell} \left(\boldsymbol {u} _ {i} ^ {\ell}\right) \end{array} \right. \tag {6}
$$
where  $(F_{\theta}^{\ell}, G_{\theta}^{\ell})_{\ell \leqslant L}$  correspond to (optional) parametric non-linear residual transformations applied in parallel to representations  $(\pmb{v}_i^\ell, \pmb{v}_i^\ell)_{i \leqslant n}$  while  $(\kappa_{\theta}^{\ell})_{\ell \leqslant L}$  are intermediate kernel transformations of the form  $\kappa: \mathcal{V} \times \mathcal{V} \mapsto \mathcal{L}(\mathcal{U})$  such as the one defined in equation (4). Breaking down kernel estimation through this sequential construction allows for iteratively refining the reproducing kernel estimate and approximating on-the-fly the set of solutions  $(\tilde{\pmb{u}}_i)_{i \leqslant I}$ . We particularly investigate the importance of depth  $L$  in the experimental section. Note that equations (5) and (6) allow to handle both varying dataset sizes and efficient parallel inference by building the sequences  $(\pmb{v}^{\ell})_{\ell \leqslant L}$  with  $(\pmb{v}_i^\ell)_{i \leqslant n, \ell \leqslant L}$  in batches and simply masking the unwanted cross-relational features during the kernel operations. All the operations are parallelizable and implemented on GPU-accelerated tensor manipulation libraries such that each regression with  $\mathcal{T}_{\theta}$  is orders of magnitude faster than gradient-based regression methods.
Discretization In the case of infinite-dimensional functional input and output spaces  $\mathcal{V}$  and  $\mathcal{U}$ , we can accommodate, for numerical computation purposes, different types of function representations previously proposed for neural operator regression and allowing for evaluation at an arbitrary point of their domain. For instance, output functions  $\pmb{u}$  can be defined as a linear combination of learned or hardcoded finite set of functions, as in Lu et al. (2019) and Bhattacharya et al. (2020). We focus instead on a different approach inspired by Fourier Neural Operators (Li et al., 2020a), by applying our model on the  $M$  first modes of a fast Fourier transform of functions  $(\pmb{v}_i,\pmb{u}_i)_{i\leqslant n}$ , and transform back its output, allowing us to work with discrete finite function representations.
Meta-training  $\diamond$  In order to train  $\mathcal{T}_{\theta}$  to approximate a solution for all problems of the form (1), we jointly learn the kernel operations  $(\kappa_{\theta}^{\ell})_{\ell \leqslant L}$  as well as transformations  $(F_{\theta}^{\ell})_{\ell \leqslant L}$ . Let us assume that  $\mathcal{L}$  is of the form  $\mathcal{L}(\mathcal{O}',\mathcal{D}_{\mathcal{O}}) = \sum_{j}\mathcal{L}(\mathcal{O}'(\pmb{v}_j),\mathcal{O}(\pmb{v}_j))$ , that datasets  $\mathcal{D}_{\mathcal{O}}$  are sampled according to a probability distribution  $\mathfrak{D}$  over the set of possible example sets with finite cardinality and that a random variable  $\mathfrak{T}$  selects the indices of each test set  $\mathcal{D}_{\mathcal{O}}^{test} = \{(v_i,\pmb{u}_i)\mid (v_j,\pmb{u}_j)\in \mathcal{D}_{\mathcal{O}}, j\in \mathfrak{T}\}$  such that the train set is  $\mathcal{D}_{\mathcal{O}}^{train} = \mathcal{D}_{\mathcal{O}}\backslash \mathcal{D}_{\mathcal{O}}^{test}$ . Our meta-learning objective is defined as:
$$
\mathcal {J} (\boldsymbol {\theta}) = \mathbb {E} _ {\mathfrak {D}, \mathfrak {T}} \left[ \sum_ {j \in \mathfrak {T}} \tilde {\mathcal {L}} \left(\mathcal {T} _ {\boldsymbol {\theta}} \left(\mathcal {D} _ {\mathcal {O}} ^ {\text {t r a i n}}\right) \left(\boldsymbol {v} _ {j}\right), \mathcal {O} \left(\boldsymbol {v} _ {j}\right)\right) \right] \tag {7}
$$
which can be tackled with gradient-based optimization w.r.t parameters  $\pmb{\theta}$  provided  $\mathcal{L}$  is differentiable (see S.I for details). In order to estimate gradients of (7), we gather a meta-dataset of  $M$  operators example sets  $(\mathcal{D}_{\mathcal{O}_m})_{m\leqslant M}$  and form, at each training step, a Monte-Carlo estimator over a batch of  $k$  datasets from this meta-dataset with random train/test splits  $(\mathfrak{T}_k)$ . For each dataset in the batch, in order to form outputs  $\mathcal{T}_{\pmb{\theta}}(\mathcal{D}_{\mathcal{O}}^{\text{train}})(\pmb{v}_j)$  defined by equation (5), we initialize the model sequence in (6) by concatenating  $\mathcal{D}_{\mathcal{O}}^{\text{train}}$  with  $\mathcal{D}_{\mathcal{O}}^{\text{query}} = \{(v_i,0_{\mathcal{U}}) \mid v_i \in \mathcal{D}_{\mathcal{O}}^{\text{test}}\}$  and obtain each inferred output  $\mathcal{T}_{\pmb{\theta}}(\mathcal{D}_{\mathcal{O}}^{\text{train}})(\pmb{v}_j)$  as  $\sum_{\pmb{v}_i \in \mathcal{D}_{\mathcal{O}}^{\text{train}}} \mathcal{K}_{\pmb{\theta}}(\pmb{v}_i, \pmb{v}_j)(\tilde{\pmb{u}}_i)$ . Since each regression consists in a single feedforward pass, estimating gradients of the meta-parameters  $\pmb{\theta}$  with respect to  $\mathcal{L}$  for each batch consists in a single backward pass achieved through automatic differentiation.