SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Wessel Bruinsma. Neural processes. https://github.com/wesselb/neuralprocesses, 2023. Accessed: October 2023.
[2] Wessel P Bruinsma, James Requeima, Andrew YK Foong, Jonathan Gordon, and Richard E Turner. The Gaussian neural process. In 3rd Symposium on Advances in Approximate Bayesian Inference, 2021.
[3] Wessel P Bruinsma, Stratis Markou, James Requeima, Andrew YK Foong, Tom R Andersson, Anna Vaughan, Anthony Buonomo, J Scott Hosking, and Richard E Turner. Autoregressive conditional neural processes. In International Conference on Learning Representations, 2023.
[4] Benjamin Chidester, Tianming Zhou, Minh N Do, and Jian Ma. Rotation equivariant and invariant neural networks for microscopy image analysis. Bioinformatics, 35(14):i530-i537, 2019.
[5] Taco Cohen and Max Welling. Group equivariant convolutional networks. In International Conference on Machine Learning, pages 2990-2999, 2016.
[6] Brian Drawert, Andreas Hellander, Ben Bales, Debjani Banerjee, Giovanni Bellesia, Bernie J Daigle, Jr., Geoffrey Douglas, Mengyuan Gu, Anand Gupta, Stefan Hellander, Chris Horuk, Dibyendu Nath, Aviral Takkar, Sheng Wu, Per Lötstedt, Chandra Krintz, and Linda R Petzold. Stochastic simulation service: Bridging the gap between the computational expert and the biologist. PLOS Computational Biology, 12(12):e1005220, 2016.
[7] Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3165-3176, 2020.
[8] Andrew YK Foong, Wessel P Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard E Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. In Advances in Neural Information Processing Systems, volume 33, pages 8284-8295, 2020.
[9] Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew G Wilson. GPyTorch: Blackbox matrix-matrix Gaussian process inference with GPU acceleration. In Advances in Neural Information Processing Systems, volume 31, pages 7576-7586, 2018.
[10] Marta Garnelo, Dan Rosenbaum, Chris J Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo J Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pages 1704–1713, 2018.
[11] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Ali Eslami, and Yee Whye Teh. Neural processes. In ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018.
[12] Roman Garnett. Bayesian Optimization. Cambridge University Press, 2023.
[13] Robert A Gatenby and Edward T Gawlinski. A reaction-diffusion model of cancer invasion. Cancer Research, 56(24):5745-5753, 1996.
[14] Marc G Genton. Classes of kernels for machine learning: A statistics perspective. Journal of Machine Learning Research, 2(Dec):299-312, 2001.
[33] Chris Olah, Nick Cammarata, Chelsea Voss, Ludwig Schubert, and Gabriel Goh. Naturally occurring equivariance in neural networks. Distill, 5(12):e00024-004, 2020.
[34] Carl Edward Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[35] Fergus Simpson, Ian Davies, Vidhi Lalchand, Alessandro Vullo, Nicolas Durrande, and Carl Edward Rasmussen. Kernel identification through transformers. In Advances in Neural Information Processing Systems, volume 34, pages 10483-10495, 2021.
[36] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, volume 30, pages 4077-4087, 2017.
[37] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1199-1208, 2018.
[38] Abiy Tasissa and Rongjie Lai. Exact reconstruction of Euclidean distance geometry problem using low-rank matrix completion. IEEE Transactions on Information Theory, 65(5):3124-3144, 2019.
[39] Warren S Torgerson. Multidimensional scaling: I. Theory and method. Psychometrika, 17(4): 401-419, 1952.
[40] Aimo Torn and Antanas Zilinskas. Global optimization, volume 350. Springer, 1989.
[41] Clément Vignac, Andreas Loukas, and Pascal Frossard. Building powerful and equivariant graph neural networks with structural message-passing. In Advances in Neural Information Processing Systems, volume 33, pages 14143-14155, 2020.
[42] Vito Volterra. Fluctuations in the abundance of a species considered mathematically. Nature, 118(2972):558-560, 1926.
[43] Edward Wagstaff, Fabian B Fuchs, Martin Engelcke, Ingmar Posner, and Michael A Osborne. On the limitations of representing functions on sets. In International Conference on Machine Learning, pages 6487-6494, 2019.
[44] Edward Wagstaff, Fabian B Fuchs, Martin Engelcke, Michael A Osborne, and Ingmar Posner. Universal approximation of functions on sets. Journal of Machine Learning Research, 23(151): 1-56, 2022.
[45] Naoki Wakamiya, Kenji Leibnitz, and Masayuki Murata. A self-organizing architecture for scalable, adaptive, and robust networking. In Nazim Agoulmine, editor, Autonomic Network Management Principles: From Concepts to Applications, pages 119–140. Academic Press, 2011.
[46] Qi Wang and Herke Van Hoof. Model-based meta reinforcement learning using graph structured surrogate models and amortized policy search. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23055-23077, 2022.
[47] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In International Conference on Artificial Intelligence and Statistics, pages 370-378, 2016.
[48] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J Smola. Deep sets. In Advances in Neural Information Processing Systems, volume 30, pages 3391-3401, 2017.
In this Appendix, we include our methodology, mathematical proofs, implementation details for reproducibility, additional results and extended explanations omitted from the main text.
A Theoretical proofs 15
A.1 Definition of equivariance for prediction maps 15
A.2 Proof that RCNPs are equivariant 16
A.3 Proof that RCNPs are context-preserving 16
B Methods 18
B.1 Training procedure, error bars and statistical significance 18
B.2 Experimental details and reproducibility 19
B.3 Details on Figure 1 19
C Computational cost analysis 19
C.1 Inference time analysis 19
C.2 Overall training time 20
D Details of synthetic regression experiments 20
D.1 Models 21
D.2 Data 21
D.3 Full results 22
E Details of Bayesian optimization experiments 23
E.1 Models 23
E.2 Data 24
E.3 Bayesian optimization 25
E.4 Computation time 26
F Details of Lotka-Volterra model experiments 26
F.1 Models 26
F.2 Data 28
F.3 Full results 28
F.4 Computation time 29
G Details of Reaction-Diffusion model experiments 29
G.1 Models 29
G.2 Data 29
G.3 Additional results 31
G.4 Computation time 31
H Additional experiments 31
H.1 Autoregressive CNPs 31
H.1.1 Computation time 32
H.2 Rotation equivariance 32
H.2.1 Computation time 33
H.3 Image datasets 34
H.3.1 Setup 34
H.3.2 Experiment 1: Centered images (no translation) 34
H.3.3 Experiment 2: Translated images 35
H.3.4 Computation time 38
A Theoretical proofs
In this section, we provide extended proofs for statements and theorems in the main text. First, we show that our definition of equivariance for the prediction map is equivalent to the common definition of equivalent maps (Section A.1). Then, we provide full proofs that RCNPs are equivariant (Section A.2) and context-preserving (Section A.3) prediction maps, as presented in the theorems in Section 4 of the main text.
A.1 Definition of equivariance for prediction maps
We show that the definition of equivariance of a prediction map, which is introduced in the main text based on the invariance of the representation function (Eq. 3 in the main text), is equivalent to the common definition of equivariance for a generic mapping.
First, we need to extend the notion of group action of a group  $\mathcal{T}$  into  $\mathbb{R}^d$ , for  $d \in \mathbb{N}$ . We define two additional actions,  $(i)$  for  $\tau \in \mathcal{T}$  and  $f$  a function from  $\mathbb{R}^d$  into  $\mathbb{R}^p$ , and  $(ii)$  for a couple  $(\mathbf{x}, \mathbf{y}) \in \mathbb{R}^d \times \mathbb{R}^p$ . Following [15]:
$$
\tau (\mathbf {x}, \mathbf {y}) = (\tau \mathbf {x}, \mathbf {y}), \qquad \tau f (\mathbf {x}) = f (\tau^ {- 1} \mathbf {x}).
$$
These two definitions can be extended for  $n$ -uplets  $\mathbf{X}$  and  $\mathbf{Y}$ :
$$
\tau (\mathbf {X}, \mathbf {Y}) = (\tau \mathbf {X}, \mathbf {Y}), \qquad \tau f (\mathbf {X}) = f (\tau^ {- 1} \mathbf {X}).
$$
In the literature, equivariance is primarily defined for a map  $F:\mathbf{Z}\in \mathcal{X}\times \mathbb{R}^d\mapsto F_{\mathbf{Z}}\in \mathcal{F}(\mathcal{X},\mathbb{R}^d)$  by:
$$
\tau F _ {\mathbf {Z}} = F _ {\tau \mathbf {Z}}. \tag {S1}
$$
Conversely, in the main text (Section 2), we defined equivariance for a prediction map through the representation function: a prediction map  $\pi$  with representation function  $r$  is  $\mathcal{T}$ -equivariant if and only if for all  $\tau \in \mathcal{T}$
$$
r \left(\left(\mathbf {X}, \mathbf {Y}\right), \mathbf {X} ^ {*}\right) = r \left(\left(\tau \mathbf {X}, \mathbf {Y}\right), \tau \mathbf {X} ^ {*}\right). \tag {S2}
$$
Eqs. S1 and S2 differ in that they deal with distinct objects. The former corresponds to the formal definition of an equivariant map, while the latter uses the representation function, which is simpler to work with for the scope of our paper.
To show the equivalence of the two definitions (Eqs. S1 and S2), we recall that a prediction map  $\pi$  is defined through its representation  $r$ :
$$
P _ {\mathbf {Z}} \left(\mathbf {X} ^ {*}\right) = \pi (\cdot \mid \mathbf {Z}, \mathbf {X} ^ {*}) = p (\cdot \mid r \left(\left(\mathbf {X}, \mathbf {Y}\right), \mathbf {X} ^ {*}\right)),
$$
where  $\mathbf{Z} \equiv (\mathbf{X}, \mathbf{Y})$ . Then, starting from Eq. S1 applied to the prediction map,
$$
\begin{array}{l} P _ {\tau (\mathbf {X}, \mathbf {Y})} = \tau P _ {(\mathbf {X}, \mathbf {Y})} \Longleftrightarrow \forall \mathbf {X} ^ {*}, p \big (\cdot \mid r ((\tau \mathbf {X}, \mathbf {Y}), \mathbf {X} ^ {*}) \big) = p \big (\cdot \mid r ((\mathbf {X}, \mathbf {Y}), \tau^ {- 1} \mathbf {X} ^ {*}) \big) \\ \Longleftrightarrow \forall \mathbf {X} ^ {*}, r ((\tau \mathbf {X}, \mathbf {Y}), \mathbf {X} ^ {*}) = r ((\mathbf {X}, \mathbf {Y}), \tau^ {- 1} \mathbf {X} ^ {*}). \\ \end{array}
$$
Noting that  $\tau^{-1}r((\mathbf{X},\mathbf{Y}),\mathbf{X}^{*}) = r((\mathbf{X},\mathbf{Y}),\tau \mathbf{X}^{*})$ , by applying  $\tau^{-1}$  to each side of the last line of the equation above, and swapping sides, we get:
$$
P _ {\tau (\mathbf {X}, \mathbf {Y})} = \tau P _ {(\mathbf {X}, \mathbf {Y})} \Longleftrightarrow \forall \mathbf {X} ^ {*}, r ((\mathbf {X}, \mathbf {Y}), \mathbf {X} ^ {*}) = r ((\tau \mathbf {X}, \mathbf {Y}), \tau \mathbf {X} ^ {*}),
$$
that is, Eqs. S1 and S2 are equivalent.
A.2 Proof that RCNPs are equivariant
We recall from Eq. 3 in the main text that a prediction map  $\pi$  with representation function  $r$  is  $\mathcal{T}$ -equivariant with respect to a group  $\mathcal{T}$  of transformations,  $\tau : \mathcal{X} \to \mathcal{X}$ , if and only if for all  $\tau \in \mathcal{T}$ :
$$
r \left(\left(\mathbf {X}, \mathbf {Y}\right), \mathbf {X} ^ {\star}\right) = r \left(\left(\tau \mathbf {X}, \mathbf {Y}\right), \tau \mathbf {X} ^ {\star}\right),
$$
where  $\tau \mathbf{x} \equiv \tau(\mathbf{x})$  and  $\tau \mathbf{X}$  is the set obtained by applying  $\tau$  to all elements of  $\mathbf{X}$ .
Also recall that a prediction map  $\pi$  and its representation function  $r$  are relational with respect to a comparison function  $g: \mathcal{X} \times \mathcal{X} \to \mathbb{R}^{d_{\mathrm{comp}}}$  if and only if  $r$  can be written solely through the set comparisons defined in Eq. 8 of the main text:
$$
r ((\mathbf {X}, \mathbf {Y}), \mathbf {X} ^ {\star}) = r \big (g ((\mathbf {X}, \mathbf {Y}), (\mathbf {X}, \mathbf {Y})) , g ((\mathbf {X}, \mathbf {Y}), \mathbf {X} ^ {\star}), g (\mathbf {X} ^ {\star}, (\mathbf {X}, \mathbf {Y})) , g (\mathbf {X} ^ {\star}, \mathbf {X} ^ {\star}) \big).
$$
We can now prove the following Lemma from the main text.
Lemma (Lemma 4.4). Let  $\pi$  be a prediction map,  $\mathcal{T}$  a transformation group, and  $g$  a comparison function. If  $\pi$  is relational with respect to  $g$  and  $g$  is  $\mathcal{T}$ -invariant, then  $\pi$  is  $\mathcal{T}$ -equivariant.
Proof. From Eq. 8 in the main text, if  $g$  is  $\mathcal{T}$ -invariant, then for a target set  $\mathbf{X}^{\star}$  and  $\tau \in \mathcal{T}$ :
$$
g \left(\left(\mathbf {X}, \mathbf {Y}\right), \mathbf {X} ^ {\star}\right) = \left\{\left(g \left(\mathbf {x} _ {n}, \mathbf {x} _ {m} ^ {\star}\right), \mathbf {y} _ {n}\right) \right\} = \left\{\left(g \left(\tau \mathbf {x} _ {n}, \tau \mathbf {x} _ {m} ^ {\star}\right), \mathbf {y} _ {n}\right) \right\} = g \left(\left(\tau \mathbf {X}, \mathbf {Y}\right), \tau \mathbf {X} ^ {\star}\right),
$$
for all  $1 \leq n \leq N$  and  $1 \leq m \leq M$ . A similar equality holds for all the comparison sets in the definition of a relational prediction map (see above). Since  $\pi$  is relational, and using the equality above, we can write its representation function  $r$  as:
$$
\begin{array}{l} r \left(\left(\mathbf {X}, \mathbf {Y}\right), \mathbf {X} ^ {\star}\right) = r \left(g \left(\left(\mathbf {X}, \mathbf {Y}\right), \left(\mathbf {X}, \mathbf {Y}\right)\right), g \left(\left(\mathbf {X}, \mathbf {Y}\right), \mathbf {X} ^ {\star}\right), g \left(\mathbf {X} ^ {\star}, \left(\mathbf {X}, \mathbf {Y}\right)\right), g \left(\mathbf {X} ^ {\star}, \mathbf {X} ^ {\star}\right)\right)\left. \right. \\ = r \big (g ((\tau \mathbf {X}, \mathbf {Y}), (\tau \mathbf {X}, \mathbf {Y})), g ((\tau \mathbf {X}, \mathbf {Y}), \tau \mathbf {X} ^ {\star}), g (\tau \mathbf {X} ^ {\star}, (\tau \mathbf {X}, \mathbf {Y})), g (\tau \mathbf {X} ^ {\star}, \tau \mathbf {X} ^ {\star}) \big) \\ = r \left(\left(\tau \mathbf {X}, \mathbf {Y}\right), \tau \mathbf {X} ^ {\star}\right). \\ \end{array}
$$
Thus,  $\pi$  is  $\mathcal{T}$ -equivariant.
From the Lemma, we can prove the first main result of the paper, that RCNPs are equivariant prediction maps for a transformation group  $\mathcal{T}$  provided a suitable comparison function  $g$ .
Proposition (Proposition 4.5). Let  $g$  be the comparison function used in a RCNP, and  $\mathcal{T}$  a group of transformations. If  $g$  is  $\mathcal{T}$ -invariant, the RCNP is  $\mathcal{T}$ -equivariant.
Proof. In all RCNP models, elements of the context and target sets are always processed via a relational encoding. By definition (Eqs. 5 and 7 in the main text), the relational encoding is written solely as a function of  $g((\mathbf{X}, \mathbf{Y}), (\mathbf{X}, \mathbf{Y}))$  and  $g((\mathbf{X}, \mathbf{Y}), \mathbf{X}^{\star})$ , so the same holds for the representation function  $r$ . Thus, any RCNP is a relational prediction map with respect to  $g$ , and it follows that it is also a  $\mathcal{T}$ -equivariant prediction map according to Lemma 4.4.
A.3 Proof that RCNPs are context-preserving
We prove here that RCNPs preserve information in the context set. We will do so by providing a construction for a function that reconstructs the context set given the relationally encoded vector  $\rho_{m}$  (up to transformations). As a disclaimer, the proofs in this section should be taken as 'existence proofs' but the provided construction is not practical. There are likely better context-preserving encodings, as shown by the empirical performance of RCNPs with moderately sized networks. Future theoretical work should provide stronger bounds on the size of the relational encoding, as recently established for DeepSets [43, 44].
For convenience, we recall Eq. 5 of the main text for the relational encoding provided a comparison function  $g$ :
$$
\boldsymbol {\rho} _ {m} = \rho_ {\mathrm {f u l l}} (\mathbf {x} _ {m} ^ {\star}, (\mathbf {X}, \mathbf {Y})) = \bigoplus_ {n, n ^ {\prime} = 1} ^ {N} f _ {r} \left(g (\mathbf {x} _ {n}, \mathbf {x} _ {m} ^ {\star}), \mathbf {R} _ {n n ^ {\prime}}\right), \qquad \mathbf {R} _ {n n ^ {\prime}} \equiv \left(g (\mathbf {x} _ {n}, \mathbf {x} _ {n ^ {\prime}}), \mathbf {y} _ {n}, \mathbf {y} _ {n ^ {\prime}}\right),
$$
where  $\mathbf{R} \equiv g((\mathbf{X}, \mathbf{Y}), g(\mathbf{X}, \mathbf{Y}))$  is the relational matrix. First, we show that from the relational encoding  $\rho_{\mathrm{full}}$  we can reconstruct the matrix  $\mathbf{R}$ , modulo permutations of rows or columns as defined below.
Definition A.1. Let  $\mathbf{R},\mathbf{R}^{\prime}\in \mathbb{R}^{N\times N}$ . The two matrices are equal modulo permutations, denoted  $\mathbf{R}\cong \mathbf{R}^{\prime}$ , if there is a permutation  $\sigma \in \operatorname {Sym}(N)$  such that  $\mathbf{R}_{nn^{\prime}} = \mathbf{R}_{\sigma (n)\sigma (n^{\prime})}^{\prime}$ , for all  $1\leq n,n^{\prime}\leq N$ .
In other words,  $\mathbf{R} \cong \mathbf{R}'$  if  $\mathbf{R}'$  is equal to  $\mathbf{R}$  after an appropriate permutation of the indices of both rows and columns. Now we proceed with the reconstruction proof.
Lemma A.2. Let  $g: \mathcal{X} \times \mathcal{X} \to \mathbb{R}^{d_{\mathrm{comp}}}$  be a comparison function,  $(\mathbf{X}, \mathbf{Y})$  a context set,  $\mathbf{X}^{\star}$  a target set,  $\mathbf{R}$  the relational matrix and  $\rho_{\mathrm{full}}$  the relational encoding as per Eq. 5 in the main text. Then there is a reconstruction function  $h$ , for  $\mathbf{r} = (\pmb{\rho}_1, \dots, \pmb{\rho}_M)$ , such that  $h(\mathbf{r}) = \mathbf{R}'$  where  $\mathbf{R}' \cong \mathbf{R}$ .
Proof. For this proof, it is sufficient to show that there is a function  $h$  applied to a local representation of any context point,  $\pmb{\rho}_{m} = \rho_{\mathrm{full}}(\mathbf{x}_{m}^{\star}, (\mathbf{X}, \mathbf{Y}))$ , such that  $h(\pmb{\rho}_{m}) = \mathbf{R}^{\prime}$ , with  $\mathbf{R}^{\prime} \cong \mathbf{R}$ . In this proof, we also consider that all numbers in a computer are represented up to numerical precision, so with  $\mathbb{R}$  we denote the set of real numbers physically representable in a chosen floating point representation. As a final assumption, without loss of generality, we take  $\mathbf{y}_n \neq \mathbf{y}_{n'}$  for  $n \neq n'$ . If there are non-unique elements  $\mathbf{y}_n$  in the context set, in practice we can always disambiguate them by adding a small jitter.
Let  $f_{r}:\mathcal{R}\to \mathbb{R}^{d_{\mathrm{rel}}}$  be the relational encoder of Eq. 5 of the main text, a neural network parametrized by  $\theta_r$  , where  $\mathcal{R} = \mathbb{R}^{d_{\mathrm{comp}}}\times \mathbb{R}^{d_{\mathrm{comp}} + 2d_y}$  . Here, we also include in  $\theta_r$  hyperparameters defining the network architecture, such as the number of layers and number of hidden units per layer. We pick  $d_{\mathrm{rel}}$  large enough such that we can build a one-hot encoding of the elements of  $\mathcal{R}$  , i.e., there is an injective mapping from  $\mathcal{R}$  to  $\{0,1\}^{d_{\mathrm{rel}}}$  such that one and only one element of the output vector is 1. This injective mapping exists, since  $\mathcal{R}$  is discrete due to finite numerical precision. We select  $\theta_r$  such that  $f_{r}$  approximates the one-hot encoding up to arbitrary precision, which is achievable thanks to universal approximation theorems for neural networks [29]. Thus,  $\rho_{m} = \bigoplus_{n,n^{\prime}}f_{r}(g(\mathbf{x}_{n},\mathbf{x}_{m}^{\star}),\mathbf{R}_{nn^{\prime}})$  is a (astronomically large) vector with ones at the elements that denote  $(g(\mathbf{x}_n,\mathbf{x}_m^\star),\mathbf{R}_{nn^{\prime}})$  for  $1\leq n,n^{\prime}\leq N$
Finally, we can build a reconstruction function  $h$  that reads out  $\rho_{m}$ , the sum of one-hot encoded vectors, mapping each 'one', based on its location, back to the corresponding input  $(g(\mathbf{x}_n,\mathbf{x}_m^\star),\mathbf{R}_{nn'})$ . We have then all the information to build a matrix  $\mathbf{R}'$  indexed by  $\mathbf{y}$ , that is
$$
\mathbf {R} _ {\mathbf {y y} ^ {\prime}} ^ {\prime} = \mathbf {R} _ {n n ^ {\prime}}, \qquad \text {w i t h} \mathbf {y} = \mathbf {y} _ {n}, \mathbf {y} ^ {\prime} = \mathbf {y} _ {n ^ {\prime}}, \quad \text {f o r s o m e} \mathbf {y}, \mathbf {y} ^ {\prime} \in \mathbf {Y}, 1 \leq n, n ^ {\prime} \leq N.
$$
Since we assumed  $\mathbf{y}_n\neq \mathbf{y}_{n'}$  for  $n\neq n'$ ,  $\mathbf{R}'$  is unique up to permutations of the rows and columns, and by construction  $\mathbf{R}'\cong \mathbf{R}$ .
We now define the matrix
$$
\mathbf {Q} _ {m n n ^ {\prime}} = \left(g \left(\mathbf {x} _ {n}, \mathbf {x} _ {m} ^ {\star}\right), g \left(\mathbf {x} _ {n}, \mathbf {x} _ {n ^ {\prime}}\right), \mathbf {y} _ {n}, \mathbf {y} _ {n ^ {\prime}}\right) = \left(g \left(\mathbf {x} _ {n}, \mathbf {x} _ {m} ^ {\star}\right), \mathbf {R} _ {n n ^ {\prime}}\right),
$$
for  $1 \leq m \leq M$  and  $1 \leq n, n' \leq N$ . We recall that a comparison function  $g$  is context-preserving with respect to a transformation group  $\mathcal{T}$  if for any context set  $(\mathbf{X}, \mathbf{Y})$  and target set  $\mathbf{X}^{\star}$ , for all  $1 \leq m \leq M$ , there is a submatrix  $\mathbf{Q}' \subseteq \mathbf{Q}_{m:}$ , a reconstruction function  $\gamma$ , and a transformation  $\tau \in \mathcal{T}$  such that  $\gamma(\mathbf{Q}') = (\tau \mathbf{X}, \mathbf{Y})$ . We showed in Section 4.2 of the main text that the distance comparison function  $g_{\mathrm{dist}}$  is context-preserving with respect to the group of rigid transformations; and that the difference comparison function  $g_{\mathrm{diff}}$  is context-preserving with respect to the translation group. Similarly, a family of functions  $h_\theta((\mathbf{X}, \mathbf{Y}), \mathbf{X}^\star) \to \mathbb{R}^{d_{\mathrm{rep}}}$  is context-preserving under  $\mathcal{T}$  if there exists  $\theta \in \Theta$ ,  $d_{\mathrm{rep}} \in \mathbb{N}$ , a reconstruction function  $\gamma$ , and a transformation  $\tau \in \mathcal{T}$  such that  $\gamma(h_\theta((\mathbf{X}, \mathbf{Y}), \mathbf{X}^\star)) = (\tau \mathbf{X}, \mathbf{Y})$ .
With the construction and definitions above, we can now show that RCNPs are in principle context-preserving. In particular, we will apply the definition of context-preserving functions  $h$  to the class of representation functions  $r$  of the RCNP models introduced in the paper, for which  $d_{\mathrm{rep}} = Md_{\mathrm{rel}}$  since  $\mathbf{r} = (\pmb{\rho}_1, \dots, \pmb{\rho}_M)$ . The proof applies both to the full RCNP and the simple (diagonal) RCNP, for an appropriate choice of the comparison function  $g$ .
Proposition (Proposition 4.8). Let  $\mathcal{T}$  be a transformation group and  $g$  the comparison function used in a FullRCNP. If  $g$  is context-preserving with respect to  $\mathcal{T}$ , then the representation function  $r$  of the FullRCNP is context-preserving with respect to  $\mathcal{T}$ .
Proof. Given Lemma A.2, for any given  $m$ , we can reconstruct  $\mathbf{Q}_{m:} = (\dots, \mathbf{R})$  (modulo permutations) from the relational encoding. Since  $g$  is context-preserving with respect to  $\mathcal{T}$ , we can
reconstruct  $(\mathbf{X},\mathbf{Y})$  from  $\mathbf{R}$  modulo a transformation  $\tau \in \mathcal{T}$ . Thus,  $\rho_{\mathrm{full}}$  is context-preserving with respect to  $\mathcal{T}$  and so is  $r$ .
Proposition (Proposition 4.9). Let  $\mathcal{T}$  be the translation group and  $g_{\mathrm{diff}}$  the difference comparison function. The representation of the simple RCNP model with  $g_{\mathrm{diff}}$  is context-preserving with respect to  $\mathcal{T}$ .