SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Daolang Huang
Manuel Haussmann<sup>1</sup>
Ulpu Remes
ST John
Grégoire Clarté³
Kevin Sebastian Luck4,6
Samuel Kaski<sup>1,5</sup>
Luigi Acerbi<sup>3</sup>
<sup>1</sup>Department of Computer Science, Aalto University, Finland
$^{2}$ Department of Mathematics and Statistics, University of Helsinki
$^{3}$ Department of Computer Science, University of Helsinki
$^{4}$ Department of Electrical Engineering and Automation (EEA), Aalto University, Finland
$^{5}$ Department of Computer Science, University of Manchester
$^{6}$ Department of Computer Science, Vrije Universiteit Amsterdam, The Netherlands
{daolang.huang, manuel.haussmann, ti.john, samuel.kaski}@aalto.fi
k.s.luck@vu.nl
{ulpu.remes, gregoire.clarte, luigi.acerbi}@helsinki.fi
Conditional Neural Processes (CNPs) are a class of metalearning models popular for combining the runtime efficiency of amortized inference with reliable uncertainty quantification. Many relevant machine learning tasks, such as in spatiotemporal modeling, Bayesian Optimization and continuous control, inherently contain equivariances – for example to translation – which the model can exploit for maximal performance. However, prior attempts to include equivariances in CNPs do not scale effectively beyond two input dimensions. In this work, we propose Relational Conditional Neural Processes (RCNPs), an effective approach to incorporate equivariances into any neural process model. Our proposed method extends the applicability and impact of equivariant neural processes to higher dimensions. We empirically demonstrate the competitive performance of RCNPs on a large array of tasks naturally containing equivariances.
Conditional Neural Processes (CNPs; [10]) have emerged as a powerful family of metalearning models, offering the flexibility of deep learning along with well-calibrated uncertainty estimates and a tractable training objective. CNPs can naturally handle irregular and missing data, making them suitable for a wide range of applications. Various advancements, such as attentive (ACNP; [20]) and Gaussian (GNP; [30]) variants, have further broadened the applicability of CNPs. In principle, CNPs can be trained on other general-purpose stochastic processes, such as Gaussian Processes (GPs; [34]), and be used as an amortized, drop-in replacement for those, with minimal computational cost at runtime.
However, despite their numerous advantages, CNPs face substantial challenges when attempting to model equivariances, such as translation equivariance, which are essential for problems involving spatio-temporal components or for emulating widely used GP kernels in tasks such as Bayesian Optimization (BayesOpt; [12]). In the context of CNPs, kernel properties like stationarity and isotropy would correspond to, respectively, translational equivariance and equivariance to rigid transformations. Lacking such equivariances, CNPs struggle to scale effectively and emulate (equivariant) GPs even in moderate higher-dimensional input spaces (i.e., above two). Follow-up work has introduced Convolutional CNPs (ConvCNP; [15]), which leverage a convolutional deep sets construction to
Figure 1: Equivariance in 1D regression. Left: Predictions for a CNP (a) and RCNP (b) in an interpolation (INT) task, trained for 20 epochs to emulate a GP (c) with Matérn-  $\frac{5}{2}$  kernel and noiseless observations. The CNP underfits the context data (black dots), while the RCNP leverages translation equivariance to learn faster and yield better predictions. Right: The CNP (d) fails to predict in an out-of-input-distribution (OOID) task, where the input context is outside the training range (note the shifted  $x$  axis); whereas the RCNP (e) generalizes by means of translational equivariance.
induce translational-equivariant embeddings. However, the requirement of defining an input grid and performing convolutions severely limits the applicability of ConvCNPs and variants thereof (ConvGNP [30]; FullConvGNP [2]) to one- or two-dimensional equivariant inputs; both because higher-dimensional implementations of convolutions are poorly supported by most deep learning libraries, and for the prohibitive cost of performing convolutions in three or more dimensions. Thus, the problem of efficiently scaling equivariances in CNPs above two input dimensions remains open.
In this paper, we introduce Relational Conditional Neural Processes (RCNPs), a novel approach that offers a simple yet powerful technique for including a large class of equivariances into any neural process model. By leveraging the existing equivariances of a problem, RCNPs can achieve improved sample efficiency, predictive performance, and generalization (see Figure 1). The basic idea in RCNPs is to enforce equivariances via a relational encoding that only stores appropriately chosen relative information of the data. By stripping away absolute information, equivariance is automatically satisfied. Surpassing the complex approach of previous methods (e.g., the ConvCNP family for translational equivariance), RCNPs provide a practical solution that scales to higher dimensions, while maintaining strong performance and extending to other equivariances. The cost to pay is increased computational complexity in terms of context size (size of the dataset we are conditioning on at runtime); though often not a bottleneck for the typical metalearning small-context setting of CNPs. Our proposed method works for equivariances that can be expressed relationally via comparison between pairs of points (e.g., their difference or distance); in this paper, we focus on translational equivariance and equivariance to rigid transformations.
Contributions. In summary, our contributions in this work are:
- We introduce a simple and effective way - relational encoding - to encode exact equivariances directly into CNPs, in a way that easily scales to higher input dimensions.
- We propose two variants of relational encoding: one that works more generally ('Full'); and one which is simpler and more computationally efficient ('Simple'), and is best suited for implementing translation equivariance.
- We provide theoretical foundations and proofs to support our approach.
- We empirically demonstrate the competitive performance of RCNPs on a variety of tasks that naturally contain different equivariances, highlighting their practicality and effectiveness.
Outline of the paper. The remainder of this paper is organized as follows. In Section 2, we review the foundational work of CNPs and their variants. This is followed by the introduction of our proposed relational encoding approach to equivariances at the basis of our RCNP models (Section 3). We then provide in Section 4 theoretical proof that relational encoding achieves equivariance without losing essential information; followed in Section 5 by a thorough empirical validation of our claims in various tasks requiring equivariances, demonstrating the generalization capabilities and predictive performance of RCNPs. We discuss other related work in Section 6, and the limitations of our approach, including its computational complexity, in Section 7. We conclude in Section 8 with an overview of the current work and future directions.
Our code is available at https://github.com/acerbilab/relational-neural-processes.
2 Background: the Conditional Neural Process family
In this section, we review the Conditional Neural Process (CNP) family of stochastic processes and the key concept of equivariance at the basis of this work. Following [30], we present these notions within the framework of prediction maps [8]. We denote with  $\mathbf{x} \in \mathcal{X} \subseteq \mathbb{R}^{d_x}$  input vectors and  $\mathbf{y} \in \mathcal{V} \subseteq \mathbb{R}^{d_y}$  output vectors, with  $d_x, d_y \geq 1$  their dimensionality. If  $f(\mathbf{z})$  is a function that takes as input elements of a set  $\mathbf{Z}$ , we denote with  $f(\mathbf{Z})$  the set  $\{f(\mathbf{z})\}_{\mathbf{z} \in \mathbf{Z}}$ .
Prediction maps. A prediction map  $\pi$  is a function that maps (1) a context set  $(\mathbf{X},\mathbf{Y})$  comprising input/output pairs  $\{(\mathbf{x}_1,\mathbf{y}_1),\ldots ,(\mathbf{x}_N,\mathbf{y}_N)\}$  and (2) a collection of target inputs  $\mathbf{X}^{\star} = (\mathbf{x}_1^{\star},\dots,\mathbf{x}_M^{\star})$  to a distribution over the corresponding target outputs  $\mathbf{Y}^{\star} = (\mathbf{y}_1^{\star},\dots,\mathbf{y}_M^{\star})$ :
$$
\pi \left(\mathbf {Y} ^ {\star} \mid (\mathbf {X}, \mathbf {Y}), \mathbf {X} ^ {\star}\right) = p \left(\mathbf {Y} ^ {\star} \mid \mathbf {r}\right), \tag {1}
$$
where  $\mathbf{r} = r((\mathbf{X},\mathbf{Y}),\mathbf{X}^{\star})$  is the representation vector that parameterizes the distribution over  $\mathbf{Y}^{\star}$  via the representation function  $r$ . Bayesian posteriors are prediction maps, a well-known example being the Gaussian Process (GP) posterior:
$$
\pi \left(\mathbf {Y} ^ {\star} \right| (\mathbf {X}, \mathbf {Y}), \mathbf {X} ^ {\star}) = \mathcal {N} \left(\mathbf {Y} ^ {\star} | \mathbf {m}, \mathbf {K}\right), \tag {2}
$$
where the prediction map takes the form of a multivariate normal with representation vector  $\mathbf{r} = (\mathbf{m},\mathbf{K})$ . The mean  $\mathbf{m} = m_{\mathrm{post}}((\mathbf{X},\mathbf{Y}),\mathbf{X}^{\star})$  and covariance matrix  $\mathbf{K} = k_{\mathrm{post}}(\mathbf{X},\mathbf{X}^{\star})$  of the multivariate normal are determined by the conventional GP posterior predictive expressions [34].
Equivalence. A prediction map  $\pi$  with representation function  $r$  is  $\mathcal{T}$ -equivariant with respect to a group  $\mathcal{T}$  of transformations<sup>1</sup> of the input space,  $\tau : \mathcal{X} \to \mathcal{X}$ , if and only if for all  $\tau \in \mathcal{T}$ :
$$
r \left(\left(\mathbf {X}, \mathbf {Y}\right), \mathbf {X} ^ {\star}\right) = r \left(\left(\tau \mathbf {X}, \mathbf {Y}\right), \tau \mathbf {X} ^ {\star}\right), \tag {3}
$$
where  $\tau \mathbf{x} \equiv \tau(\mathbf{x})$  and  $\tau \mathbf{X}$  is the set obtained by applying  $\tau$  to all elements of  $\mathbf{X}$ . Eq. 3 defines equivariance of a prediction map based on its representation function, and can be shown to be equivalent to the common definition of an equivariant map; see Appendix A. Intuitively, equivariance means that if the data (the context inputs) are transformed in a certain way, the predictions (the target inputs) transform correspondingly. Common groups of transformations include translations, rotations, reflections – all examples of rigid transformations. In kernel methods and specifically in GPs, equivariances are incorporated in the prior kernel function  $k(\mathbf{x},\mathbf{x}^{\star})$ . For example, translational equivariance corresponds to stationarity  $k_{\mathrm{sta}} = k(\mathbf{x} - \mathbf{x}^{\star})$ , and equivariance to all rigid transformations corresponds to isotropy,  $k_{\mathrm{iso}} = k(||\mathbf{x} - \mathbf{x}^{\star}||_2)$ , where  $||\cdot||_2$  denotes the Euclidean norm of a vector. A crucial question we address in this work is how to implement equivariances in other prediction maps, and specifically in the CNP family.
Conditional Neural Processes. A CNP [10] uses an encoder $^2$ $f_{e}$  to produce an embedding of the context set,  $\mathbf{e} = f_{e}(\mathbf{X},\mathbf{Y})$ . The encoder uses a DeepSet architecture [48] to ensure invariance with respect to permutation of the order of data points, a key property of stochastic processes. We denote with  $\mathbf{r}_m = (\mathbf{e},\mathbf{x}_m^\star)$  the local representation of the  $m$ -th point of the target set  $\mathbf{X}^{\star}$ , for  $1\leq m\leq M$ . CNPs yield a prediction map with representation  $\mathbf{r} = (\mathbf{r}_1,\dots ,\mathbf{r}_M)$ :
$$
\pi \left(\mathbf {Y} ^ {\star} \mid (\mathbf {X}, \mathbf {Y}), \mathbf {X} ^ {\star}\right) = p \left(\mathbf {Y} ^ {\star} \mid \mathbf {r}\right) = \prod_ {m = 1} ^ {M} q \left(\mathbf {y} _ {m} ^ {\star} \mid \lambda \left(\mathbf {r} _ {m}\right)\right), \tag {4}
$$
where  $q(\cdot |\lambda)$  belongs to a family of distributions parameterized by  $\lambda$ , and  $\lambda = f_d(\mathbf{r}_m)$  is decoded in parallel for each  $\mathbf{r}_m$ . In the standard CNP, the decoder network  $f_{d}$  is a multi-layer perceptron. A common choice for CNPs is a Gaussian likelihood,  $q(\mathbf{y}_m^\star |\boldsymbol {\lambda}) = \mathcal{N}(\mathbf{y}_m^\star |\boldsymbol {\mu}(\mathbf{r}_m),\boldsymbol {\Sigma}(\mathbf{r}_m))$ , where  $\pmb{\mu}$  and  $\pmb{\Sigma}$  represent the predictive mean and covariance of each output, independently for each target (a mean field approach). Given the closed-form likelihood, CNPs are easily trainable via maximum-likelihood optimization of parameters of encoder and decoder networks, by sampling batches of context and target sets from the training data.
<sup>1</sup>A group of transformations is a family of composable, invertible functions with identity  $\tau_{\mathrm{id}}\mathbf{x} = \mathbf{x}$ .
2We use purple to highlight parametrized functions (neural networks) whose parameters will be learned.
Gaussian Neural Processes. Notably, standard CNPs do not model dependencies between distinct target outputs  $\mathbf{y}_m^{\star}$  and  $\mathbf{y}_{m'}^{\star}$ , for  $m \neq m'$ . Gaussian Neural Processes (GNPs [30]) are a variant of CNPs that remedy this limitation, by assuming a joint multivariate normal structure over the outputs for the target set,  $\pi(\mathbf{Y}^{\star} | (\mathbf{X}, \mathbf{Y}), \mathbf{X}^{\star}) = \mathcal{N}(\mathbf{Y} | \boldsymbol{\mu}, \boldsymbol{\Sigma})$ . For ease of presentation, we consider now scalar outputs  $(d_y = 1)$ , but the model generalizes to the multi-output case. GNPs parameterize the mean as  $\mu_m = f_\mu(\mathbf{r}_m)$  and covariance matrix  $\Sigma_{m,m'} = k(f_\Sigma(\mathbf{r}_m), f_\Sigma(\mathbf{r}_{m'})f_v(\mathbf{r}_m)f_v(\mathbf{r}_{m'})$ , for target points  $\mathbf{x}_m^{\star}, \mathbf{x}_{m'}^{\star}$ , where  $f_\mu, f_\Sigma$  and  $f_v$  are neural networks with outputs, respectively, in  $\mathbb{R}$ ,  $\mathbb{R}^{d_\Sigma}$ , and  $\mathbb{R}^+$ ,  $k(\cdot, \cdot)$  is a positive-definite kernel function, and  $d_\Sigma \in \mathbb{N}^+$  denotes the dimensionality of the space in which the covariance kernel is evaluated. Standard GNP models use the linear covariance (where  $f_v = 1$  and  $k$  is the linear kernel) or the kvv covariance (where  $k$  is the exponentiated quadratic kernel with unit lengthscale), as described in [30].
Convolutional Conditional Neural Processes. The Convolutional CNP family includes the ConvCNP [15], ConvGNP [30], and FullConvGNP [2]. These CNP models are built to implement translational equivariance via a ConvDeepSet architecture [15]. For example, the ConvCNP is a prediction map  $p(\mathbf{Y}^{\star} | (\mathbf{X}, \mathbf{Y}), \mathbf{X}^{\star}) = \prod_{m=1}^{M} q(\mathbf{y}_m^{\star} | \Phi_{\mathbf{X}, \mathbf{Y}}(\mathbf{x}_m^{\star}))$ , where  $\Phi_{\mathbf{X}, \mathbf{Y}}(\cdot)$  is a ConvDeepSet. The construction of ConvDeepSets involves, among other steps, gridding of the data if not already on the grid and application of  $d_x$ -dimensional convolutional neural networks ( $2d_x$  for FullConvGNP). Due to the limited scaling and availability of convolutional operators above two dimensions, ConvCNPs do not scale in practice for  $d_x > 2$  translationally-equivariant input dimensions.
Other Neural Processes. The neural process family includes several other members, such as latent NPs (LNP; [11]) which model dependencies in the predictions via a latent variable – however, LNPs lack a tractable training objective, which impairs their practical performance. Attentive (C)NPs (A(C)NPs; [20]) implement an attention mechanism instead of the simpler DeepSet architecture. Transformer NPs [31] combine a transformer-based architecture with a causal mask to construct an autoregressive likelihood. Finally, Autoregressive CNPs (AR-CNPs [3]) provide a novel technique to deploy existing CNP models via autoregressive sampling without architectural changes.
3 Relational Conditional Neural Processes
We introduce now our Relational Conditional Neural Processes (RCNPs), an effective solution for embedding equivariances into any CNP model. Through relational encoding, we encode selected relative information and discard absolute information, inducing the desired equivariance.
Relational encoding. In RCNPs, the (full) relational encoding of a target point  $\mathbf{x}_m^\star \in \mathbf{X}^\star$  with respect to the context set  $(\mathbf{X},\mathbf{Y})$  is defined as:
$$
\rho_ {\text {f u l l}} \left(\mathbf {x} _ {m} ^ {\star}, (\mathbf {X}, \mathbf {Y})\right) = \bigoplus_ {n, n ^ {\prime} = 1} ^ {N} f _ {r} \left(g \left(\mathbf {x} _ {n}, \mathbf {x} _ {m} ^ {\star}\right), \mathbf {R} _ {n n ^ {\prime}}\right), \quad \mathbf {R} _ {n n ^ {\prime}} \equiv \left(g \left(\mathbf {x} _ {n}, \mathbf {x} _ {n ^ {\prime}}\right), \mathbf {y} _ {n}, \mathbf {y} _ {n ^ {\prime}}\right), \tag {5}
$$
where  $g: \mathcal{X} \times \mathcal{X} \to \mathbb{R}^{d_{\mathrm{comp}}}$  is a chosen comparison function that specifies how a pair  $\mathbf{x}, \mathbf{x}'$  should be compared;  $\mathbf{R}$  is the relational matrix, comparing all pairs of the context set;  $f_r: \mathbb{R}^{d_{\mathrm{comp}}} \times \mathbb{R}^{d_{\mathrm{comp}} + 2d_y} \to \mathbb{R}^{d_{\mathrm{rel}}}$  is the relational encoder, a neural network that maps a comparison vector and element of the relational matrix into a high-dimensional space  $\mathbb{R}^{d_{\mathrm{rel}}}$ ;  $\bigoplus$  is a commutative aggregation operation (sum in this work) ensuring permutation invariance of the context set [48]. From Eq. 5, a point  $\mathbf{x}^\star$  is encoded based on how it compares to the entire context set.
Intuitively, the comparison function  $g(\cdot, \cdot)$  should be chosen to remove all information that does not matter to impose the desired equivariance. For example, if we want to encode translational equivariance, the comparison function should be the difference of the inputs,  $g_{\mathrm{diff}}(\mathbf{x}_n, \mathbf{x}_m^{\star}) = \mathbf{x}_m^{\star} - \mathbf{x}_n$  (with  $d_{\mathrm{comp}} = d_{\mathrm{x}}$ ). Similarly, isotropy (invariance to rigid transformations, i.e. rotations, translations, and reflections) can be encoded via the Euclidean distance  $g_{\mathrm{dist}}(\mathbf{x}_n, \mathbf{x}_m^{\star}) = ||\mathbf{x}_m^{\star} - \mathbf{x}_n||_2$  (with  $d_{\mathrm{comp}} = 1$ ). We will prove these statements formally in Section 4.
3We use green to highlight the selected comparison function that encodes a specific set of equivariances.
Full RCNP. The full-context RCNP, or FullRCNP, is a prediction map with representation  $\mathbf{r} = (\pmb{\rho}_1, \dots, \pmb{\rho}_M)$ , with  $\pmb{\rho}_m = \rho_{\mathrm{full}}(\mathbf{x}_m^\star, (\mathbf{X}, \mathbf{Y}))$  the relational encoding defined in Eq. 5:
$$
\pi \left(\mathbf {Y} ^ {\star} | (\mathbf {X}, \mathbf {Y}), \mathbf {X} ^ {\star}\right) = p \left(\mathbf {Y} ^ {\star} \mid \mathbf {r}\right) = \prod_ {m = 1} ^ {M} q \left(\mathbf {y} _ {m} ^ {\star} \mid \boldsymbol {\lambda} \left(\boldsymbol {\rho} _ {m}\right)\right), \tag {6}
$$
where  $q(\cdot |\lambda)$  belongs to a family of distributions parameterized by  $\lambda$ , where  $\lambda = f_d(\pmb{\rho}_m)$  is decoded from the relational encoding  $\pmb{\rho}_m$  of the  $m$ -th target. As usual, we often choose a Gaussian likelihood, whose mean and covariance (variance, for scalar outputs) are produced by the decoder network.
Note how Eq. 6 (FullRCNP) is nearly identical to Eq. 4 (CNP), the difference being that we replaced the representation  $\mathbf{r}_m = (\mathbf{e},\mathbf{x}_m^\star)$  with the relational encoding  $\pmb{\rho}_{m}$  from Eq. 5. Unlike CNPs, in RCNPs there is no separate encoding of the context set alone. The RCNP construction generalizes easily to other members of the CNP family by plug-in replacement of  $\mathbf{r}_m$  with  $\pmb{\rho}_{m}$ . For example, a relational GNP (RGNP) describes a multivariate normal prediction map whose mean is parameterized as  $\mu_{m} = f_{\mu}(\pmb{\rho}_{m})$  and whose covariance matrix is given by  $\Sigma_{m,m^{\prime}} = k\big(f_{\Sigma}(\pmb{\rho}_{m}),f_{\Sigma}(\pmb{\rho}_{m^{\prime}})\big)f_{v}(\pmb{\rho}_{m})f_{v}(\pmb{\rho}_{m^{\prime}})$ .
Simple RCNP. The full relational encoding in Eq. 5 is cumbersome as it asks to build and aggregate over a full relational matrix. Instead, we can consider the simple or 'diagonal' relational encoding:
$$
\rho_ {\mathrm {d i a g}} \left(\mathbf {x} _ {m} ^ {\star}, (\mathbf {X}, \mathbf {Y})\right) = \bigoplus_ {n = 1} ^ {N} f _ {r} \left(g \left(\mathbf {x} _ {n}, \mathbf {x} _ {m} ^ {\star}\right), g \left(\mathbf {x} _ {n}, \mathbf {x} _ {n}\right), \mathbf {y} _ {n}\right). \tag {7}
$$
Eq. 7 is functionally equivalent to Eq. 5 restricted to the diagonal  $n = n'$ , and further simplifies in the common case  $g(\mathbf{x}_n, \mathbf{x}_n) = \mathbf{0}$ , whereby the argument of the aggregation becomes  $f_r(g(\mathbf{x}_n, \mathbf{x}_m^\star), \mathbf{y}_n)$ .
We obtain the simple RCNP model (from now on, just RCNP) by using the diagonal relational encoding  $\rho_{\mathrm{diag}}$  instead of the full one,  $\rho_{\mathrm{full}}$ . Otherwise, the simple RCNP model follows Eq. 6. We will prove, both in theory and empirically, that the simple RCNP is best for encoding translational equivariance. Like the FullRCNP, the RCNP easily extends to other members of the CNP family.
In this paper, we consider the FullRCNP, FullRGNP, RCNP and RGNP models for translations and rigid transformations, leaving examination of other RCNP variants and equivariances to future work.
4 RCNPs are equivariant and context-preserving prediction maps
In this section, we demonstrate that RCNPs are  $\mathcal{T}$ -equivariant prediction maps, where  $\mathcal{T}$  is a transformation group of interest (e.g., translations), for an appropriately chosen comparison function  $g: \mathcal{X} \times \mathcal{X} \to \mathbb{R}^{d_{\mathrm{comp}}}$ . Then, we formalize the statement that RCNPs strip away only enough information to achieve equivariance, but no more. We prove this by showing that the RCNP representation preserves information in the context set. Full proofs are given in Appendix A.
4.1 RCNPs are equivariant
Definition 4.1. Let  $g$  be a comparison function and  $\mathcal{T}$  a group of transformations  $\tau : \mathcal{X} \to \mathcal{X}$ . We say that  $g$  is  $\mathcal{T}$ -invariant if and only if  $g(\mathbf{x}, \mathbf{x}') = g(\tau \mathbf{x}, \tau \mathbf{x}')$  for any  $\mathbf{x}, \mathbf{x}' \in \mathcal{X}$  and  $\tau \in \mathcal{T}$ .
Definition 4.2. Given a comparison function  $g$ , we define the comparison sets:
$$
g \left(\left(\mathbf {X}, \mathbf {Y}\right), \left(\mathbf {X}, \mathbf {Y}\right)\right) = \left\{\left(g \left(\mathbf {x} _ {n}, \mathbf {x} _ {n ^ {\prime}}\right), \mathbf {y} _ {n}, \mathbf {y} _ {n ^ {\prime}}\right) \right\} _ {1 \leq n, n ^ {\prime} \leq N},
$$
$$
g \left(\left(\mathbf {X}, \mathbf {Y}\right), \mathbf {X} ^ {\star}\right) = \left\{\left(g \left(\mathbf {x} _ {n}, \mathbf {x} _ {m} ^ {\star}\right), \mathbf {y} _ {n}\right) \right\} _ {1 \leq n \leq N, 1 \leq m \leq M}, \tag {8}
$$
$$
g \left(\mathbf {X} ^ {\star}, \mathbf {X} ^ {\star}\right) = \{g \left(\mathbf {x} _ {m} ^ {\star}, \mathbf {x} _ {m ^ {\prime}} ^ {\star}\right) \} _ {1 \leq m, m ^ {\prime} \leq M}.
$$
If  $g$  is not symmetric, we can also denote  $g(\mathbf{X}^{\star},(\mathbf{X},\mathbf{Y})) = \{(g(\mathbf{x}_m^{\star},\mathbf{x}_n),\mathbf{y}_n)\}_{1\leq n\leq N,1\leq m\leq M}$
Definition 4.3. A prediction map  $\pi$  and its representation function  $r$  are relational with respect to a comparison function  $g$  if and only if  $r$  can be written solely through set comparisons:
$$
r ((\mathbf {X}, \mathbf {Y}), \mathbf {X} ^ {\star}) = r \big (g ((\mathbf {X}, \mathbf {Y}), (\mathbf {X}, \mathbf {Y})), g ((\mathbf {X}, \mathbf {Y}), \mathbf {X} ^ {\star}), g (\mathbf {X} ^ {\star}, (\mathbf {X}, \mathbf {Y})), g (\mathbf {X} ^ {\star}, \mathbf {X} ^ {\star}) \big). \tag {9}
$$