SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tara Akhound-Sadegh *
School of Computer Science, McGill University,
Mila - Quebec Artificial Intelligence Institute,
Laurence Perreault-Levasseur
Université de Montréal, Montreal, Quebec, Canada
Ciela Institute, Montreal, Quebec, Canada
Mila - Quebec Artificial Intelligence Institute, Montreal, Quebec, Canada
Trottier Space Institute, Montreal, Quebec, Canada
CCA, Flatiron Institute, New York, USA
Perimeter Institute, Waterloo, Ontario, Canada
Microsoft Research AI4Science,
Max Welling
University of Amsterdam,
Siamak Ravanbakhsh
Symmetries have been leveraged to improve the generalization of neural networks through different mechanisms from data augmentation to equivariant architectures. However, despite their potential, their integration into neural solvers for partial differential equations (PDEs) remains largely unexplored. We explore the integration of PDE symmetries, known as Lie point symmetries, in a major family of neural solvers known as physics-informed neural networks (PINNs). We propose a loss function that informs the network about Lie point symmetries in the same way that PINN models try to enforce the underlying PDE through a loss function. Intuitively, our symmetry loss ensures that the infinitesimal generators of the Lie group conserve the PDE solutions. Effectively, this means that once the network learns a solution, it also learns the neighbouring solutions generated by Lie point symmetries. Empirical evaluations indicate that the inductive bias introduced by the Lie point symmetries of the PDEs greatly boosts the sample efficiency of PINNs.
In recent years, deep learning has accelerated data-driven approaches to science and engineering. A prominent example is the role of deep learning in solving partial differential equations (PDEs), which are ubiquitous in many scientific disciplines.
*Corresponding author: tara.akhound-sadegh@mila.quebec
$^{\dagger}$ Contribution was done when at Microsoft Research AI4Science, Amsterdam.
This is mainly driven by the fact that traditional handcrafted numerical solvers can be prohibitively expensive, and thus, learning to solve PDEs has the potential to significantly impact various areas of science, ranging from quantum chemistry to biology to climate science to cosmology, [Wang et al., 2020a, Kochkov et al., 2021, Kashinath et al., 2021, Gupta and Brandstetter, 2022, Nguyen et al., 2023, Bi et al., 2022]. Like any other machine learning problem, learning to solve PDEs can benefit from inductive biases to boost sample efficiency and generalization capabilities. As such, the PDE itself and its respective Lie point symmetries, i.e. joint symmetries of coordinate and field transformations, are two natural inductive biases for constructing efficient neural PDE solvers.
For parametric PDEs, Physics-Informed Neural Networks (PINNs) have emerged as an efficient new learning paradigm. PINNs constrain the solution network to satisfy the underlying PDE via the PINN loss terms [Raissi et al., 2019], which comprise
a residual loss, as well as loss terms for initial and boundary points. As such, PINNs can be seen as a data-free learning approach, utilizing available physics information to guide neural network training. Lie point symmetries are uniquely special in that it is possible to characterize the full set of permissible transformations for each PDE, and thus naturally offer additional physics information which piggy-backs the data-free learning objective. Lie point symmetries have been introduced to deep learning by Brandstetter et al. [2022a] for Lie point symmetry data augmentation which places a firm mathematical footing on data augmentation pipelines for neural PDE solvers. Further, Lie point symmetries showed promising results when used in extending the framework to self-supervised learning for neural PDE solvers [Mialon et al., 2023]. In this work, for the first time we integrate symmetries into PINNs and demonstrate the effectiveness of this symmetry regularization on generalization capabilities.
Lie point symmetries of a PDE, by definition, map a solution to a solution, preserving the PDE. Lie point symmetry transformations act on both independent (e.g., time and space), and dependent (PDE solution) fields and, by extension, on all partial derivatives. To enforce a Lie point symmetry, we first need to prolong a symmetry transformation to find how it transforms the partial derivatives of the respective PDE. We show how to do this using automatic differentiation, which enables us to transform the entire PDE for each infinitesimal generator of each Lie point symmetry. "Imposing a symmetry" means that by transforming the PDE via a symmetry transformation, the PDE is required to be preserved. In practical terms, Lie point symmetries should be orthogonal to the PDE gradient. Each Lie Point symmetry results in one such orthogonality constraint, which we enforce using a penalty term within the PINN framework. We can enforce these on the same points where the PINN loss is imposed. Importantly, our Lie Point symmetry regularization is complementary to the PINN loss: while the PDE loss regularizes the neural network to satisfy the PDE at select points, the Lie point symmetry loss regularizes the neural network such that infinitesimal changes in different directions continue to satisfy the underlying PDE.
2.1 Partial Differential Equations
Partial differential equations (PDEs) are used to mathematically describe the dynamics of various physical systems. In a PDE, the evolution of a function that involves several variables is described in terms of local updates expressed by partial derivatives. Since temporal PDEs are especially prevalent
Figure 1: All the solutions of the differential equation of a harmonic oscillator can be reached via symmetry transformations of a single solution. Its Lie-point symmetry group is the special linear group,  $SL(3)$ , corresponding to eight one-parameter subgroups transforming a given solution, identified using a different colour in this figure. While, in general, Lie-point symmetries form more than one orbit – that is, we cannot reach all solutions using such transformations – they can significantly narrow down the solution space.
in physical sciences, we consider PDEs of the following general form:
$$
\Delta = \mathbf {u} _ {t} + \mathcal {D} _ {\mathbf {x}} [ \mathbf {u} ] = 0, \quad t \in [ 0, T ], \mathbf {x} \in \Omega \tag {1}
$$
$$
\mathbf {u} (0, \mathbf {x}) = f (\mathbf {x}), \quad \mathbf {x} \in \Omega
$$
$$
\mathbf {u} (t, \mathbf {x}) = g (t, \mathbf {x}), \quad \mathbf {x} \in \partial \Omega , t \in [ 0, T ]
$$
where  $\mathbf{u}(t,\mathbf{x})\in \mathbb{R}^{D_u}$  is the solution to the PDE,  $t$  denotes the time, and  $\mathbf{x}$  is a vector of possibly multiple spatial coordinates.  $\Omega \subset \mathbb{R}^D$  is the domain and  $\mathcal{D}_{\mathbf{x}}[.]$  is a non-linear differential operator.  $f(\mathbf{x})$  is known as the initial condition function and  $g(t,\mathbf{x})$  describes the boundary conditions. In the following, we sometimes explicitly separate the time from other independent variables and sometimes for convenience group them together.
Example 1 (Heat Equation). The one-dimensional heat equation describes the heat conduction in a one-dimensional rod, with viscosity coefficient  $\nu$ . Its differential operator is given by  $\mathcal{D}_x[u] = -\nu u_{xx}$ .
Using Neural Networks for Solving PDEs. For many systems, obtaining an analytical solution to the PDE is impossible; hence, numerical methods are traditionally used to obtain approximate solutions. Numerical solvers, such as finite element methods (FEM) or finite difference methods (FDM) rely on discretizing the space  $T \times \Omega$  [Quarteroni, 2009]. The topology of the space has to be taken into account when constructing the mesh, and the resolution of the discretization will affect the accuracy of the predicted solutions. Additionally, these solvers are often computationally expensive, especially for complex dynamics, and each time the initial or boundary conditions of the PDE change
the solver must be rerun. These considerations and constraints make designing numerical solvers difficult, and scientists often need to handcraft a specific solver for an application [Quarteroni, 2009].
Given the recent successes of neural networks, especially in dealing with large datasets, using deep learning to solve PDEs has become a promising direction. The idea is to learn a function or, more generally, a functional that bypasses the numerical computation and produces the solution to the PDE in a single shot or by progression through time. Broadly, there are two approaches to solving PDEs with neural networks: neural operator methods which approximate the solution operator that maps between solutions of the underlying PDEs, and direct methods which learn the underlying solution function. The most prominent representative of the latter are PINNs.
PINNs. In contrast to neural operator methods [Lu et al., 2021, Li et al., 2021a] where the main idea is to generalize neural networks to obtain mappings between function space, PINNs directly learn the solution function of the underlying PDE. Consequently instead of relying on large training sets,
PINNs operate as a surrogate model for the PDE solution, trained directly with the PDE equation itself. The simplicity of the PINN idea has made it the subject of many follow-up improvements; see Krishnapriyan et al. [2021], Wang et al. [2020b, 2022, 2023]
In PINNs, the PDE solution  $u(t,\mathbf{x})$  of Eq. (1) is a neural network  $u_{\theta}(t,\mathbf{x})$  with parameters  $\theta$ . The loss function is then compromised of two parts:
$$
\mathcal {L} (\theta) = \mathcal {L} _ {\mathrm {P D E}} + \mathcal {L} _ {\text {d a t a - f i t}} \tag {2}
$$
The first term is the physics-informed objective, ensuring that the function learned by the neural network satisfies the PDE Eq. (1). Let  $r(t,\mathbf{x})$  denote the residual error in agreement with the PDE equation:
$$
r _ {\theta} (t, \mathbf {x}) = \frac {\partial}{\partial t} u _ {\theta} (t, \mathbf {x}) + \mathcal {D} _ {\mathbf {x}} [ u _ {\theta} (t, \mathbf {x}) ]
$$
where the derivatives of the solution network  $u_{\theta}$  are calculated using automatic differentiation [Raissi et al., 2019]. This penalty is then imposed on a finite set of points  $(t, \mathbf{x})_{1:N_r}$  which are sampled from inside the domain  $[0, T] \times \Omega$  to obtain the PDE loss:
$$
\mathcal {L} _ {\mathrm {P D E}} = \frac {1}{N _ {r}} \sum_ {i = 1} ^ {N _ {r}} \| r _ {\theta} \left(t _ {i}, \mathbf {x} _ {i}\right) \| _ {2} ^ {2} \tag {3}
$$
The second term in Eq. (2), is a supervised loss which ensures that the function learned by the neural network satisfies the initial and boundary conditions of the problem – that is
$$
\mathcal {L} _ {\text {d a t a - f i t}} = \frac {1}{N _ {s}} \sum_ {i = 1} ^ {N _ {s}} \| u _ {\theta} \left(0, \mathbf {x} _ {i} ^ {0}\right) - f \left(x _ {i} ^ {0}\right) \| _ {2} ^ {2} + \frac {1}{N _ {b}} \sum_ {i = 1} ^ {N _ {b}} \| u _ {\theta} \left(t _ {i} ^ {b}, \mathbf {x} _ {i} ^ {b}\right) - g \left(t _ {i} ^ {b}, x _ {i} ^ {b}\right) \| _ {2} ^ {2} \tag {4}
$$
where  $(\mathbf{x}^0)_{1:N_s}\in \Omega$  are  $N_{s}$  samples at which the initial condition function  $f$  is sampled.  $(t^{b},\mathbf{x}^{b})_{1:N_{b}}$  are  $N_{b}$  points sampled on the boundary (from  $[0,T]\times \partial \Omega)$ .
There have also been multiple models that combine the operator learning approach with the physics-informed loss. For example, in Li et al. [2021b], the NO approach is combined with the PINN loss. Another example is Wang et al. [2021], which we describe further in Section 3.1.
2.2 Symmetries
Groups. Symmetries are essentially transformations of the object that leave an aspect of it invariant: for example, the nature of an object does not change if we translate or rotate it. In mathematics, symmetries of an object are described by abstract objects known as groups. More concretely, a group is a set  $G$  and a group operation  $\cdot : G \times G \to G$  satisfying: associativity, the existence of an identity element, and the presence of an inverse element for every element in the set.
Lie Groups. Groups that are also differentiable manifolds are known as Lie groups, and are important in the study of continuous symmetries. In Lie groups, in addition to satisfying the three properties, the group operation  $\cdot$  and its inverse are smooth maps. Each Lie group has an associated Lie algebra which is its tangent space, as a vector space, at identity. Intuitively, Lie algebras describe the smooth transformations of the Lie groups in the limit of infinitesimal transformations. The elements of the Lie algebra are vectors describing the direction of the infinitesimal symmetry transformation.
One Parameter Subgroups. The Lie group,  $\mathcal{G}$ , can be multi-dimensional and complex. When the  $n$ -dimensional group  $\mathcal{G}$  is simply connected, it is often represented in terms of a series of  $n$  one-parameter transformations,  $g = g_{1}(\epsilon_{1})g_{2}(\epsilon_{2})\ldots g_{n}(\epsilon_{n})$ , where  $g_{i}:\mathbb{R}\to \mathcal{G}$ ,  $i = 1,\dots ,n$  and such that  $g_{i}(\epsilon)g_{i}(\delta) = g_{i}(\epsilon +\delta)$ . The  $\epsilon$ 's are the real parameters of the transformation, and each  $g_{i}$  is a continuous group homomorphism (i.e., a smooth, group-structured map) from this parameter to the symmetry group.
2.3 Symmetries of Partial Differential Equations
In this section, we will provide the mathematical background of symmetries of differential equations. We will mostly follow the exposition presented in Olver [1986] and refer the reader to this original text for a more in-depth treatment of this topic.
In the context of differential equations, symmetries are transformations that map a solution of the PDE to another solution. For example, in Fig. 1, we can see how the solutions of a simple harmonic oscillator can be obtained via symmetry transformations of a given solution, where the symmetry group is  $SL(3)$ . Consider the PDE  $\Delta$ , involving  $p$  independent variables  $\mathbf{x} = (x_{1},\ldots ,x_{p})\in X$  and  $q$  dependent variables  $\mathbf{u} = (u_{1},\dots,u_{q})\in U$ . The solutions to the PDE will be of the form  $u_{i} = f_{i}(x_{1},\ldots ,x_{p})$  for  $i = 1,\ldots ,q$ , for  $\mathbf{x}\in \Omega$ , where  $\Omega \subset X$  is the domain of the function  $f$ . The symmetry group  $\mathcal{G}$ , of  $\Delta$ , is the local group of transformations on an open subset of the space of dependent and independent variables,  $M\subset X\times U$ , transforming solutions of  $\Delta$  to other solutions.
Prolongations. To formalize this abstract definition of PDE symmetries, Lie proposed viewing  $\Delta$  as a concrete geometric object and introduced the concept of prolongation [Olver, 1986]. The idea is to prolong the space of independent and dependent variables,  $X\times U$ , to a space that also represents the partial derivatives involved in the PDE. More concretely, we have the following definition:
Definition 1. The  $n$ -th order prolongation (or  $n$ -th order jet space) of  $X \times U$  is denoted as  $X \times U^{(n)} = X \times U_1 \times \dots \times U_n$ , whose coordinates represent the independent and dependent variables as well as all the partial derivatives of the dependent variables up to order  $n$ .
Equivalently we have the notion of prolongation of  $\mathbf{u}$  as  $\mathbf{u}^{(n)} = (\mathbf{u}_{\mathbf{x}},\mathbf{u}_{\mathbf{x}\mathbf{x}},\dots ,\mathbf{u}_{n\mathbf{x}})$  where  $\mathbf{u}_{i\mathbf{x}}$  is all the unique  $i^{\mathrm{th}}$  derivatives of  $u$ , for  $i = 1,\ldots ,n$ . For example, if  $\mathbf{x} = (x,y)$ , then  $\mathbf{u}_{\mathbf{x}\mathbf{x}} = (\partial_{xx}\mathbf{u},\partial_{xy}\mathbf{u},\partial_{yy}\mathbf{u})$ .
Formally, the Lie algebra is a vector space equipped with a binary operation known as the Lie bracket.
Using this notion of prolongation, we can represent a PDE as an algebraic equation,  $\Delta (\mathbf{x},\mathbf{u}^{(n)}) = 0$  where  $\Delta$  is the map that determines the PDE, i.e.,  $\Delta :X\times U^{(n)}\to \mathbb{R}$ . In other words, the PDE tells us where the map  $\Delta$  vanishes on  $X\times U^{(n)}$ .
For example, for the heat equation described in 1,  $\Delta$  is given by:
$$
\Delta ((x, t), \mathbf {u} ^ {(2)}) = u _ {t} - \nu u _ {x x}, \tag {5}
$$
The graph of all prolonged solutions is the set  $S_{\Delta} \subset X \times U^{(n)}$ , and is defined as  $S_{\Delta} = \{(\mathbf{x}, \mathbf{u}^{(n)}): \Delta(\mathbf{x}, \mathbf{u}^{(n)}) = 0\}$ . In this new notation, we can say that  $\mathbf{u}(\mathbf{x})$  is a solution of the PDE if  $\Gamma_u^{(n)} = \{(\mathbf{x}, \operatorname{pr}^{(n)}\mathbf{u}(\mathbf{x}))\} \subset S_{\Delta}$ . where  $\operatorname{pr}^{(n)}\mathbf{u}(\mathbf{x}): X \to U^n$ , is a vector-valued function whose entries represent all derivatives of  $\mathbf{u}$  wrt  $\mathbf{x}$  up to order  $n$ .
Prolongations of the Infinitesimal Generators. Let  $\mathbf{v}$  be the vector field on the subspace  $M\subset X\times U$  with corresponding one-parameter subgroup  $\exp (\epsilon \mathbf{v})$ . In other words, the vector field  $\mathbf{v}$  is the infinitesimal generator of the one-parameter subgroup. Intuitively, this vector field describes the infinitesimal transformations of the group to the independent and dependent variables, and we can write it as:
$$
\mathbf {v} = \sum_ {i = 1} ^ {p} \xi_ {i} (\mathbf {x}, \mathbf {u}) \frac {\partial}{\partial x ^ {i}} + \sum_ {\alpha = 1} ^ {q} \phi_ {\alpha} (\mathbf {x}, \mathbf {u}) \frac {\partial}{\partial u ^ {\alpha}}, \tag {6}
$$
where  $\xi^i (\mathbf{x},\mathbf{u})$  and  $\phi_{\alpha}(\mathbf{x},\mathbf{u})$  are coordinate-dependent coefficients. To study how symmetries transform a solution to another solution, we need to know how they transform the partial derivatives and, therefore, the jet space  $X\times U^{(n)}$ .
A symmetry transformation of the independent  $(\mathbf{x})$  and dependent  $(\mathbf{u})$  variables will also induce transformations in the partial derivatives  $\mathbf{u}_{\mathbf{x}}, \mathbf{u}_{\mathbf{x}\mathbf{x}}, \ldots$ . The prolongation of the infinitesimal generator,  $\operatorname{pr}^{(n)}\mathbf{v}$  is a generalization of the generator  $\mathbf{v}$  which describes these induced transformations in these partial derivatives.
This prolongation will be defined on the jet-space  $X \times U^{(n)}$  and it is given by:
$$
\Pr^ {(n)} \mathbf {v} = \sum_ {i = 1} ^ {p} \xi_ {i} (\mathbf {x}, \mathbf {u}) \frac {\partial}{\partial x ^ {i}} + \sum_ {\alpha = 1} ^ {q} \sum_ {J} \phi_ {\alpha} ^ {(J)} (\mathbf {x}, \mathbf {u}) \frac {\partial}{\partial u _ {J} ^ {\alpha}}, \tag {7}
$$
where we have used the notation  $J = (i_1, \dots, i_k)$  for the multi-indices, with  $0 \leq i_k \leq p$  and  $0 \leq k \leq n$  and  $\mathbf{u}_J^\alpha = \frac{\partial^k \mathbf{u}^\alpha}{\partial x^{i_1} \cdots \partial x^{i_k}}$ . Calculating  $\phi_\alpha^{(J)}$ , the coefficients of  $\partial_{\mathbf{u}_j^\alpha}$ , can be done using the prolongation formula, which involves the total derivative operator  $D$  (see Olver [1986] for a derivation):
$$
\phi_ {\alpha} ^ {(J)} = D _ {J} Q _ {\alpha} + \sum_ {i = 1} ^ {p} \xi_ {i} \frac {\partial \mathbf {u} _ {J} ^ {\alpha}}{\partial x ^ {i}} \quad \text {w h e r e} \quad Q _ {\alpha} = \phi_ {\alpha} - \sum_ {i = 1} ^ {p} \xi_ {i} \frac {\partial \mathbf {u} ^ {\alpha}}{\partial x ^ {i}}. \tag {8}
$$
The upshot is that we can mechanically calculate the prolonged vector field using partial derivatives of  $\mathbf{u}$ , which are, in turn, produced by automatic differentiation. In practice, prolonged vector fields are implemented as vector-valued functions (or functionals) of  $\mathbf{x}$  and  $\mathbf{u}$ . Therefore, the implementation of this mechanical process is generic and can be applied to any PDE. See the Appendix A for examples.
Lie Point Symmetries of PDEs. We can now define the prolongation of the action of  $\mathcal{G}$ :
Definition 2. For symmetry group  $\mathcal{G}$  acting on  $M$ , the prolongation of action of  $\mathcal{G}$  on the open subset  $M \subset X \times U$  is the induced action on  $M^{(n)} = M \times U^{(n)}$  which transforms derivatives of a solution,  $\mathbf{u} = f(\mathbf{x})$ , into corresponding derivatives of another solution,  $\mathbf{u}' = f'(\mathbf{x}')$ . We can write this as:
$$
\operatorname {p r} ^ {(n)} g \cdot (\mathbf {x}, \mathbf {u} ^ {(n)}) = (g \cdot \mathbf {x}, \operatorname {p r} ^ {(n)} g \cdot \mathbf {u} ^ {(n)}).
$$
Using the definition above, we can provide a new criterion for  $\mathcal{G}$  being a symmetry group of  $\Delta$ , under a mild assumption on the PDE equation.
This assumption is that  $\Delta$  is of maximal rank. We refer the readers to Olver [1986] for the definition of this condition. However, we note that this assumption does not pose a restriction since for any PDE not satisfying this condition, it is possible to find an equivalent PDE which does.
Theorem 2.1.  $\mathcal{G}$  is the symmetry group of the  $n$ -th order PDE  $\Delta (\mathbf{x},\mathbf{u}^n)$ , if  $\mathcal{G}$  acts on  $M$ , and its prolongation leaves the solution set  $\mathcal{S}_{\Delta}$  invariant:  $\mathrm{pr}^{(n)}g\cdot (\mathbf{x},\mathbf{u}^{(n)})\in \mathcal{S}_{\Delta}$ ,  $\forall g\in \mathcal{G}$ .
Finally, we can express the symmetry condition in terms of the infinitesimal generators  $\mathbf{v}$  of  $\mathcal{G}$ :
Theorem 2.2 (Infinitesimal Criterion).  $\mathcal{G}$  is a symmetry group of the PDE  $\Delta (\mathbf{x},\mathbf{u}^n)$  if for every infinitesimal generator  $\mathbf{v}$  of  $\mathcal{G}$ , we have that
$$
\operatorname {p r} ^ {(n)} \mathbf {v} [ \Delta ] = 0 \quad \text {w h e n} \quad \Delta = 0.
$$
Example 2 (A Symmetry of the Heat Equation). As an illustrative example, we can consider the heat Eq. (5) and will show that the following vector field generates a symmetry group for this PDE:  $\mathbf{v} = 2\nu t\partial_x - xu\partial_u$ . We need to find the first prolongation  $\phi^{(t)}$  and the second prolongation  $\phi^{(xx)}$  where  $\phi = -xu$ . Using the prolongation formula given in Eq. (7), we get:
$$
\phi^ {(t)} = - x u _ {t} - 2 \nu u _ {x} \quad a n d \quad \phi^ {(x x)} = - 2 u _ {x} - x u _ {x x}
$$
Now:  $\operatorname{pr}^{(2)}\mathbf{v}[\Delta] = \phi^{(t)} - \nu \phi^{(xx)} = xu_t + 2\nu u_x - \nu \bigl(2\nu u_x - x\nu u_{xx}\bigr) = x(u_t - \nu u_{xx})$
Clearly  $\mathrm{pr}^{(2)}\mathbf{v}[\Delta] = 0$  when  $\Delta = 0$ , hence  $\mathbf{v}$  is a symmetry of the heat equation.
3.1 Solving PDEs with Different Initial/Boundary Conditions with PINNs
Wang et al. [2021] combines the approach introduced in Lu et al. [2021] with the PINN loss to solve PDEs with different initial or boundary conditions without requiring retraining of the model as the original PINN model does. We will also use this proposed framework to examine the effect of enforcing the symmetry condition of the PDE on the model.
Recall that we want to model the operator  $\mathcal{O}:\mathcal{A}\to \mathcal{U}$ , where  $\mathcal{A}$  is the space of initial condition functions and  $\mathcal{U}$  is the space of PDE solutions. Our model consists of two neural networks:  $e_{\theta_1}$  embeds the initial condition function, and  $g_{\theta_2}$  embeds the independent variables,  $[t,\mathbf{x}]\in \mathbb{R}^p$ .
In particular, to embed the initial condition function  $f(\mathbf{x}) = \mathbf{u}(0,\mathbf{x})\in \mathbb{R}^p$ , it is sampled at fixed points  $\{\mathbf{x}_1,\dots ,\mathbf{x}_n\}$ , and the concatenated values are fed to  $f_{\theta_1}$
The final prediction is the inner product of these embedding vectors:
$$
\mathcal {O} _ {\theta} (f) (\mathbf {x}, t) = e _ {\theta_ {1}} ^ {\top} \left(\mathbf {u} \left(0, \mathbf {x} _ {1}\right), \dots , \mathbf {u} \left(0, \mathbf {x} _ {n}\right)\right) g _ {\theta_ {2}} (\mathbf {x}, t), \tag {9}
$$