SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The second term regarding the Perron-Frobenius operators comes from the bound in Proposition 4.6. We try to reduce the generalization error by reducing the magnitude of the norm of the Perron-Frobenius operators.
5.1 Representative theorem
We first show a representative theorem to guarantee that a solution of the minimization problem (1) is represented only with given samples.
Proposition 5.1 Let  $h: \mathcal{A}^n \times \mathcal{A}^n \to \mathbb{R}_+$  be an error function, let  $g_1$  be an  $\mathbb{R}_+$ -valued function on the space of bounded linear operators on  $\tilde{\mathcal{M}}_1$ , and let  $g_2: \mathbb{R}_+ \to \mathbb{R}_+$  satisfy  $g_2(a) \leq g_2(b)$  for  $a \leq b$ . Assume the following minimization problem has a solution:
$$
\min  _ {(f _ {j} \in \mathcal {M} _ {j}) _ {j}} h (f _ {L} \circ \dots \circ f _ {1} (x _ {1}), \ldots , f _ {L} \circ \dots \circ f _ {1} (x _ {n})) + g _ {1} (P _ {f _ {L - 1}} \dots P _ {f _ {L}} | _ {\tilde {\mathcal {V}} (\mathbf {x})}) + g _ {2} (\| f _ {L} \| _ {\mathcal {M} _ {L}}).
$$
Then, there exists a solution admitting a representation of the form  $f_{j} = \sum_{i=1}^{n} \phi_{j}(x_{i}^{j-1}) c_{i,j}$  for some  $c_{1,j}, \ldots, c_{n,j} \in \mathcal{A}$  and for  $j = 1, \ldots, L$ . Here,  $x_{i}^{j} = f_{j} \circ \dots \circ f_{1}(x_{i})$  for  $j = 1, \ldots, L$  and  $x_{i}^{0} = x_{i}$ .
Remark 5.2 An advantage of deep RKHM compared to deep vvRKHS is that we can make use of the structure of matrices. For example, the product of two diagonal matrices is calculated by the elementwise product of diagonal elements. Thus, when  $\mathcal{A}_1 = \dots = \mathcal{A}_L = \text{Block}((1, \ldots, 1), d)$ , interactions among elements in an input are induced only by the kernels, not by the product  $k_j(x, x_i^{j-1}) \cdot c_{i,j}$ . That is, the form of interactions does not depend on the learning parameter  $c_{i,j}$ . On the other hand, if we set  $\mathcal{A}_j = \text{Block}(\mathbf{m}_j, d)$  with  $\mathbf{m}_j = (m_{1,j}, \ldots, m_{M_j,j}) \neq (1, \ldots, 1)$ , then at the  $j$ th layer, we get interactions among elements in the same block through the product  $k_j(x, x_i^{j-1}) \cdot c_{i,j}$ . In this case, the form of interactions is learned through  $c_{i,j}$ . For example, the part  $M_1 \leq \dots \leq M_l$  (the encoder) in Example 3.1 tries to gradually reduce the dependency among elements in the output of each layer and describe the input with a small number of variables. The part  $M_l \geq \dots \geq M_L$  (the decoder) tries to increase the dependency.
5.2 Computing the norm of the Perron-Frobenius operator
We discuss the practical computation and the role of the factor  $\| P_{f_{L - 1}} \cdots P_{f_1} |_{\tilde{\mathcal{V}}(\mathbf{x})} \|_{\mathrm{op}}$  in Eq. (1). Let  $G_j \in \mathcal{A}^{n \times n}$  be the Gram matrix whose  $(i, l)$ -entry is  $k_j(x_i^{j-1}, x_l^{j-1}) \in \mathcal{A}$ .
Proposition 5.3 For  $j = 1, L$ , let  $[\phi_j(x_1^{j-1}), \ldots, \phi_j(x_n^{j-1})]R_j = Q_j$  be the QR decomposition of  $[\phi_j(x_1^{j-1}), \ldots, \phi_j(x_n^{j-1})]$ . Then, we have  $\| P_{f_{L-1}} \cdots P_{f_1}|_{\tilde{\mathcal{V}}(\mathbf{x})} \|_{\mathrm{op}} = \| R_L^* G_L R_1 \|_{\mathrm{op}}$ .
The computational cost of  $\| R_L^* G_L R_1\|_{\mathrm{op}}$  is expensive if  $n$  is large since computing  $R_{1}$  and  $R_{L}$  is expensive. Thus, we consider upper bounding  $\| R_L^* G_L R_1\|_{\mathrm{op}}$  by a computationally efficient value.
Proposition 5.4 Assume  $G_{L}$  is invertible. Then, we have
$$
\left\| P _ {f _ {L - 1}} \dots P _ {f _ {1}} \right| _ {\tilde {\mathcal {V}} (\mathbf {x})} \| _ {\mathrm {o p}} \leq \left\| G _ {L} ^ {- 1} \right\| _ {\mathrm {o p}} ^ {1 / 2} \left\| G _ {L} \right\| _ {\mathrm {o p}} \left\| G _ {1} ^ {- 1} \right\| _ {\mathrm {o p}} ^ {1 / 2}. \tag {2}
$$
Since  $G_{1}$  is independent of  $f_{1},\ldots ,f_{L}$ , to make the value  $\| P_{f_{L - 1}}\dots P_{f_1}|_{\tilde{\mathcal{V}} (\mathbf{x})}\|_{\mathrm{op}}$  small, we try to make the norm of  $G_{L}$  and  $G_{L}^{-1}$  small according to Proposition 5.4. For example, instead of the second term regarding the Perron-Frobenius operators in Eq. (1), we can consider the following term with  $\eta >0$ :
$$
\lambda_ {1} \left(\left\| (\eta I + G _ {L}) ^ {- 1} \right\| _ {\mathrm {o p}} + \left\| G _ {L} \right\| _ {\mathrm {o p}}\right).
$$
Note that the term depends on the training samples  $x_{1}, \ldots, x_{n}$ . The situation is different from the third term in Eq. (1) since the third term does not depend on the training samples before applying the representer theorem. If we try to minimize a value depending on the training samples, the model seems to be more specific for the training samples, and it may cause overfitting. Thus, the connection between the minimization of the second term in Eq. (1) and generalization cannot be explained by the classical argument about generalization and regularization. However, as we will see in Subsection 6.2, it is related to benign overfitting and has a good effect on generalization.
Remark 5.5 The inequality (2) implies that as  $\{\phi_L(x_1^{L - 1}),\ldots ,\phi_L(x_n^{L - 1})\}$  becomes nearly linearly dependent, the Rademacher complexity becomes large. By the term  $\| (\eta I + G_L)^{-1}\|$  in Eq. (1), the function  $f_{L - 1}\circ \dots \circ f_1$  is learned so that it separates  $x_{1},\ldots ,x_{n}$  well.
Remark 5.6 To evaluate  $f_{1} \circ \dots \circ f_{L}(x_{i})$  for  $i = 1, \ldots, n$ , we have to construct a Gram matrix  $G_{j} \in \mathcal{A}_{j}^{n \times n}$ , and compute the product of the Gram matrix and a vector  $c_{j} \in \mathcal{A}_{j}^{n}$  for  $j = 1, \ldots, L$ . The computational cost of the construction of the Gram matrices does not depend on  $d$ . The cost of computing  $G_{j}c_{j}$  is  $O(n^{2}\tilde{d}_{j})$ , where  $\tilde{d}_j$  is the number of nonzero elements in the matrices in  $\mathcal{A}_j$ . Note that if  $\mathcal{A}_j$  is the  $C^*$ -algebra of block diagonal matrices, then we have  $\tilde{d}_j \ll d^2$ . Regarding the cost with respect to  $n$ , if the positive definite kernel is separable, we can use the random Fourier features [37] to replace the factor  $n^2$  with  $mn$  for a small integer  $m \ll n$ .
6 Connection and Comparison with Existing Studies
The proposed deep RKHM is deeply related to existing studies by virtue of  $C^*$ -algebra and the Perron-Frobenius operators. We discuss the connection below.
6.1 Connection with CNN
The proposed deep RKHM has a duality with CNNs. Let  $\mathcal{A}_0 = \dots = \mathcal{A}_L = \text{Circ}(d)$ , the  $C^*$ -algebra of  $d$  by  $d$  circulant matrices. For  $j = 1, \ldots, L$ , let  $a_j \in \mathcal{A}_j$ . Let  $k_j$  be an  $\mathcal{A}_j$ -valued positive definite kernel defined as  $k_j(x,y) = \tilde{k}_j(a_jx, a_jy)$ , where  $\tilde{k}_j$  is an  $\mathcal{A}_j$ -valued function. The  $\mathcal{A}_j$ -valued positive definite kernel makes the output of each layer become a circulant matrix, which enables us to apply the convolution as the product of the output and a parameter. Then,  $f_j$  is represented as  $f_j(x) = \sum_{i=1}^{n} \tilde{k}_j(a_jx, a_jx_i^{j-1})c_{i,j}$  for some  $c_{i,j} \in \mathcal{A}_j$ . Thus, at the  $j$ th layer, the input  $x$  is multiplied by  $a_j$  and is transformed nonlinearly by  $\sigma_j(x) = \sum_{i=1}^{n} k_j(x, a_jx_i^{j-1})c_{i,j}$ . Since the product of two circulant matrices corresponds to the convolution,  $a_j$  corresponds to a filter. In
addition,  $\sigma_{j}$  corresponds to the activation function at the  $j$ th layer. Thus, the deep RKHM with the above setting corresponds to a CNN. The difference between the deep RKHM and the CNN is the parameters that we learn. Whereas for the deep RKHM, we learn the coefficients  $c_{1,j},\ldots ,c_{n,j}$ , for the CNN, we learn the parameter  $a_{j}$ . In other words, whereas for the deep RKHM, we learn the activation function  $\sigma_{j}$ , for the CNN, we learn the filter  $a_{j}$ . It seems reasonable to interpret this difference as a consequence of solving the problem in the primal or in the dual. In the primal, the number of the learning parameters depends on the dimension of the data (or the filter for convolution), while in the dual, it depends on the size of the data.
Remark 6.1 The connection between CNNs and shallow RKHMs has already been studied [27]. However, the existing study does not provide the connection between the filter and activation function. The above investigation shows a more clear layer-wise connection of deep RKHMs with CNNs.
6.2 Connection with benign overfitting
Benign overfitting is a phenomenon that the model fits any amount of data yet generalizes well [20, 21]. For kernel regression, Mallinar et al. [22] showed that if the eigenvalues of the integral operator associated with the kernel function over the data distribution decay slower than any powerlaw decay, than the model exhibits benign overfitting. The Gram matrix is obtained by replacing the integral with the sum over the finite samples. The inequality (2) suggests that the generalization error becomes smaller as the smallest and the largest eigenvalues of the Gram matrix get closer, which means the eigenvalue decay is slower. Combining the observation in Remark 5.5, we can interpret that as the right-hand side of the inequality (2) becomes smaller, the outputs of noisy training data at the  $L - 1$ th layer tend to be more separated from the other outputs. In other words, for the random variable  $x$  following the data distribution,  $f_{L-1} \circ \dots \circ f_1$  is learned so that the distribution of  $f_{L-1} \circ \dots \circ f_1(x)$  generates the integral operator with more separated eigenvalues, which appreciates benign overfitting. We will also observe this phenomenon numerically in Section 7 and Appendix C.3.2. Since the generalization bound for deep vvRKHSs [5] is described by the Lipschitz constants of the feature maps and the norm of  $f_j$  for  $j = 1, \ldots, L$ , this type of theoretical interpretation regarding benign overfitting is not available for the existing bound for deep vvRKHSs.
Remark 6.2 The above arguments about benign overfitting are valid only for deep RKHMs, i.e., the case of  $L \geq 2$ . If  $L = 1$  (shallow RKHM), then the Gram matrix  $G_{L} = [k(x_{i},x_{j})]_{i,j}$  is fixed and determined only by the training data and the kernel. On the other hand, if  $L \geq 2$  (deep RKHM), then  $G_{L} = [k_{L}(f_{L - 1}\circ \dots \circ f_{1}(x_{i}),f_{L - 1}\circ \dots \circ f_{1}(x_{j}))]_{i,j}$  depends also on  $f_{1},\ldots ,f_{L - 1}$ . As a result, by adding the term using  $G_{L}$  to the loss function, we can learn proper  $f_{1},\ldots ,f_{L - 1}$  so that they make the right-hand side of Eq. (2) small, and the whole network overfits benignly. As  $L$  becomes large, the function  $f_{L - 1}\circ \dots \circ f_1$  changes more flexibly to attain a smaller value of the term. This is an advantage of considering a large  $L$ .
6.3 Connection with neural tangent kernel
Neural tangent kernel has been investigated to understand neural networks using the theory of kernel methods [6, 7]. Generalizing neural networks to  $C^*$ -algebra, which is called the  $C^*$ -algebra network, is also investigated [32, 38]. We define a neural tangent kernel for the  $C^*$ -algebra network and develop a theory for combining neural networks and deep RKHMs as an analogy of the existing studies. Consider the  $C^*$ -algebra network  $f: \mathcal{A}^{N_0} \to \mathcal{A}$  over  $\mathcal{A}$  with  $\mathcal{A}$ -valued weight matrices  $W_j \in \mathcal{A}^{N_j \times N_{j-1}}$  and element-wise activation functions  $\sigma_j: f(x) = W_L \sigma_{L-1}(W_{L-1} \cdots \sigma_1(W_1 x) \cdots)$ . The  $(i,j)$ -entry of  $f(x)$  is  $f_i(\mathbf{x}_j) = \mathbf{W}_{L,i} \sigma_{L-1}(W_{L-1} \cdots \sigma_1(W_1 \mathbf{x}_j) \cdots)$ , where  $\mathbf{x}_j$  is the  $j$ th column of  $x$  regarded as  $x \in \mathbb{C}^{dN_0 \times d}$  and  $\mathbf{W}_{L,i}$  is the  $i$ th row of  $W_L$  regarded as  $W_L \in \mathbb{C}^{d \times dN_L-1}$ . Thus, the  $(i,j)$ -entry of  $f(x)$  corresponds to the output of the network  $f_i(\mathbf{x}_j)$ . We can consider the neural tangent kernel for each  $f_i$ . Chen and Xu [7] showed that the RKHS associated with the neural tangent kernel restricted to the sphere is the same set as that associated with the Laplacian kernel. Therefore, the  $i$ th row of  $f(x)$  is described by the shallow RKHS associated with the neural tangent kernel  $k_i^{\mathrm{NT}}$ . Let  $k^{\mathrm{NN}}$  be the  $A$ -valued positive definite kernel whose  $(i,j)$ -entry is  $k_i^{\mathrm{NT}}$  for  $i = j$  and 0 for  $i \neq j$ . Then, for any function  $g \in \mathcal{M}_{k^{\mathrm{NN}}}$ , the elements in the  $i$ th row of  $g$  are in the RKHS associated with  $k_i^{\mathrm{NT}}$ , i.e., associated with the Laplacian kernel. Thus,  $f$  is described by this shallow RKHM.
Remark 6.3 We can combine the deep RKHM and existing neural networks by replacing some  $f_{j}$  in our model with an existing neural network. The above observation enables us to apply our results in Section 4 to the combined network.
6.4 Comparison to bounds for classical neural networks
Existing bounds for classical neural networks typically depend on the product or sum of matrix  $(p,q)$  norm of all the weight matrices  $W_{j}$  [14, 15, 16, 17]. Typical bound is  $O(\sqrt{1 / n}\prod_{j = 1}^{L}\| W_{j}\|_{\mathrm{HS}})$ . Note that the Hilbert-Schmidt norm is the matrix (2,2) norm. Unlike the operator norm, the matrix  $(p,q)$  norm tends to be large as the width of the layers becomes large. On the other hand, the dependency of our bound on the width of the layer is not affected by the number of layers in the case where the kernels are separable. Indeed, assume we set  $k_{1} = \tilde{k}_{1}a_{1}$  and  $k_{L} = \tilde{k}_{L}a_{L}$  for some complex-valued kernels  $\tilde{k}_1$  and  $\tilde{k}_2$ ,  $a_1\in \mathcal{A}_1$ , and  $a_{L}\in \mathcal{A}_{L}$ . Then by Proposition 5.3, the factor  $\alpha (L)\coloneqq \| P_{f_{L - 1}}\dots P_{f_1}|_{\tilde{\mathcal{V}} (\mathbf{x})}\|_{\mathrm{op}}$  is written as  $\| \tilde{R}_L^*\tilde{G}_L\tilde{R}_1\otimes a_L^2 a_1\|_{\mathrm{op}} = \| \tilde{R}_L^*\tilde{G}_L\tilde{R}_1\|_{\mathrm{op}}\| a_L^2 a_1\|_{\mathcal{A}}$  for some  $\tilde{R}_L,\tilde{G}_L,\tilde{R}_1\in \mathbb{C}^{n\times n}$ . Thus, it is independent of  $d$ . The only part depending on  $d$  is  $\operatorname {tr}k_{1}(x_{i},x_{i})$  which results in the bound  $O(\alpha (L)\sqrt{d / n})$ . Note that the width of the  $j$ th layer corresponds to the number of nonzero elements in a matrix in  $\mathcal{A}_j$ . We also discuss in Appendix B the connection of our bound with the bound by Koopman operators, the adjoints of the Perron-Frobenius operators [18].
7 Numerical Results
We numerically confirm our theory and the validity of the proposed deep RKHM.
Comparison to vvRKHS We compared the generalization property of the deep RKHM to the deep vvRKHS with the same positive definite kernel. For  $d = 10$  and  $n = 10$ , we set  $x_{i} = (az_{i})^{2} + \epsilon_{i}$  as input samples, where  $a \in \mathbb{R}^{100 \times 10}$  and  $z_{i} \in \mathbb{R}^{10}$  are randomly generated by  $\mathcal{N}(0,0.1)$ , the normal distribution of mean 0 and standard deviation 0.1,  $(\cdot)^{2}$  is the elementwise product, and  $\epsilon_{i}$  is the random noise drawn from  $\mathcal{N}(0,1e - 3)$ . We reshaped  $x_{i}$  to a 10 by 10 matrix. We set  $L = 3$  and  $k_{j} = \tilde{k}I$  for  $j = 1,2,3$ , where  $\tilde{k}$  is the Laplacian kernel. For RKHMs, we set  $\mathcal{A}_{1} = \text{Block}((1,\dots,1),d)$ ,  $\mathcal{A}_{2} = \text{Block}((2,\dots,2),d)$ , and  $\mathcal{A}_{3} = \mathbb{C}^{d \times d}$ . This is the autoencoder mentioned in Example 3.1. For vvRKHSs, we set the corresponding Hilbert spaces with the Hilbert-Schmidt inner product. We set the loss function as  $1/n\|\sum_{i=1}^{n}|f(x_{i})-x_{i}|_{\mathcal{A}}^{2}\|_{\mathcal{A}}$  for the deep RKHM and as  $1/n\sum_{i=1}^{n}\|f(x_{i})-x_{i}\|_{\mathrm{HS}}^{2}$  for the deep vvRKHS. Here,  $f = f_{3} \circ f_{2} \circ f_{1}$ . We did not add any terms to the loss function to see how the loss function with the operator norm affects the generalization performance. We computed the same value  $\|E[|f(x)-x|_{\mathcal{A}}^{2}]\|_{\mathcal{A}}-1/n\|\sum_{i=1}^{n}|f(x_{i})-x|_{\mathcal{A}}^{2}\|_{\mathcal{A}}$  for both RKHM and vvRKHS. Figure 2 (a) shows the results. We can see that the deep RKHM generalizes better than the deep vvRKHS, only with the loss function and without any additional terms.
Observation about benign overfitting We analyzed the overfitting numerically. For  $d = 10$  and  $n = 1000$ , we randomly sampled  $d$  by  $d$  diagonal matrices  $x_{1}, \ldots, x_{n} \in \mathcal{A}_{0}$  from the normal distribution  $\mathcal{N}(0, 0.1)$ . We set  $y_{i} = x_{i}^{2} + \epsilon_{i}$  for  $i = 1, \ldots, n$ , where  $\epsilon_{i}$  is a noise drawn from the normal distribution  $\mathcal{N}(0, 0.001)$ . The magnitude of the noise is  $10\%$  of  $x_{i}^{2}$ . In addition, we set  $L = 2$ ,  $\mathcal{A}_{1} = \mathbb{C}^{d \times d}$ ,  $\mathcal{A}_{2} = \text{Block}((1, \ldots, 1), d)$ , and  $k_{j}$  as the same kernel as the above experiment. The additional term to the loss function is set as  $\lambda_{1}(\| (\eta I + G_{L})^{-1} \|_{\mathrm{op}} + \| G_{L} \|_{\mathrm{op}}) + \lambda_{2} \| f_{L} \|_{\mathcal{M}_{L}}^{2}$ , where  $\eta = 0.01$  and  $\lambda_{2} = 0.01$  according to Subsection 5.2. We computed the generalization error for the cases of  $\lambda_{1} = 0$  and  $\lambda_{1} = 10^{2}$ . Figure 2 (b) shows the result. We can see that the generalization error saturates without the additional term motivated by the Perron-Frobenius operator. On the other hand, with the additional term, the generalization error becomes small, which is the effect of benign overfitting.
Comparison to CNN We compared the deep RKHM to a CNN on the classification task with MNIST [39]. We set  $d = 28$  and  $n = 20$ . We constructed a deep RKHM combined with a neural network with 2 dense layers. For the deep RKHM, we set  $L = 2$ ,  $A_0 = \mathbb{C}^{d \times d}$ ,  $A_1 = \text{Block}((7,7,7,7),d)$ , and  $A_2 = \text{Block}((4,\dots,4),d)$ . Then, two dense layers are added. See Subsection 6.3 about combining the deep RKHM with neural networks. Regarding the additional term to the loss function, we set the same term with the previous experiment with  $\lambda_2 = 0.001$  and
(b)
Figure 2: (a) The box plot of the generalization error of the deep RKHM and vvRKHS at the point that the training error reaches 0.05. (b) Behavior of the generalization error during the learning process with and without the additional term regarding the Perron-Frobenius operators. (c) Test accuracy of the classification task with MNIST for a deep RKHM and a CNN.
set  $\lambda_{1} = 1$  or  $\lambda_{1} = 0$ . To compare the deep RKHM to CNNs, we also constructed a network by replacing the deep RKHM with a CNN. The CNN is composed of 2 layers with  $7 \times 7$  and  $4 \times 4$  filters. Figure 2 (c) shows the test accuracy of these networks. We can see that the deep RKHM outperforms the CNN. In addition, we can see that the test accuracy is higher if we add the term regarding the Perron-Frobenius operators. We discuss the memory consumption and the computational cost of each of the deep RKHM and the CNN in Appendix C.3, and we empirically show that the deep RKHM outperforms the CNN that has the same size of learning parameters as the deep RKHM. We also show additional results about benign overfitting in Appendix C.3.
8 Conclusion and Limitations
In this paper, we proposed deep RKHM and analyzed it through  $C^*$ -algebra and the Perron-Frobenius operators. We derived a generalization bound, whose dependency on the output dimension is alleviated by the operator norm, and which is related to benign overfitting. We showed a representative theorem about the proposed deep RKHM, and connections with existing studies such as CNNs and neural tangent kernel. Our theoretical analysis shows that  $C^*$ -algebra and Perron-Frobenius operators are effective tools for analyzing deep kernel methods. The main contributions of this paper are our theoretical results with  $C^*$ -algebra and the Perron-Frobenius operators. More practical investigations are required for further progress. For example, although we numerically showed the validity of our method for the case where the number of samples is limited (the last experiment in Section 7), more experimental results for the case where the number of samples is large are useful for further analysis. Also, although we can apply random Fourier features (Remark 5.6) to reduce the computational costs, studying more efficient methods specific to deep RKHM remains to be investigated in future work. As for the theoretical topic, we assumed the well-definedness of the Perron-Frobenius operators. Though separable kernels with invertible matrices, which are typical examples of kernels, satisfy the assumption, generalization of our analysis to other kernels should also be studied in future work.
Hachem Kadri is partially supported by grant ANR-19-CE23-0011 from the French National Research Agency. Masahiro Ikeda is partially supported by grant JPMJCR1913 from JST CREST.
[1] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. In Proceedings of Advances in Neural Information Processing Systems (NIPS) 22, 2009.
[2] Sebastian W. Ober, Carl E. Rasmussen, and Mark van der Wilk. The promises and pitfalls of deep kernel learning. In Proceedings of the 37th Conference on Uncertainty in Artificial Intelligence (UAI), 2021.
[40] Andreas Maurer. A vector-contraction inequality for rademacher complexities. In Proceedings of the 27th International Conference on Algorithmic Learning Theory (ALT), 2016.
We provide the proofs of the theorems, propositions, and lemmas in the main paper.
Proof Assume  $\sum_{i=1}^{n} \phi_1(x_i) c_i = \sum_{i=1}^{n} \phi_1(x_i) d_i$  for  $n \in \mathbb{N}$ ,  $c_i, d_i \in \mathcal{A}$ . Since  $\{\phi_1(x) \mid x \in \mathcal{X}\}$  is  $\mathcal{A}$ -linearly independent, we have  $c_i = d_i$  for  $i = 1, \ldots, n$ . Thus,  $P_f \sum_{i=1}^{n} \phi_1(x_i) c_i = \sum_{i=1}^{n} \phi_1(f(x_i)) c_i = \sum_{i=1}^{n} \phi_1(f(x_i)) d_i = P_f \sum_{i=1}^{n} \phi_1(x_i) d_i$ .
Lemma 2.8 Let  $k_{1} = \tilde{k}a$ , i.e.,  $k$  is separable, for an invertible operator  $a$  and a complex-valued kernel  $\tilde{k}$ . Assume  $\{\tilde{\phi}(x) \mid x \in \mathcal{X}\}$  is linearly independent (e.g.,  $\tilde{k}$  is Gaussian or Laplacian), where  $\tilde{\phi}$  is the feature map associated with  $\tilde{k}$ . Then,  $\{\phi_1(x) \mid x \in \mathcal{X}\}$  is  $\mathcal{A}$ -linearly independent.
Proof If  $\sum_{i=1}^{n} \phi_1(x_i) c_i = 0$ , we have
$$
0 = \left\langle \sum_ {i = 1} ^ {n} \phi_ {1} (x _ {i}) c _ {i}, \sum_ {i = 1} ^ {n} \phi_ {1} (x _ {i}) c _ {i} \right\rangle_ {\mathcal {M} _ {k}} = c ^ {*} G c = (G ^ {1 / 2} c) ^ {*} (G ^ {1 / 2} c),
$$
where  $G$  is the  $\mathcal{A}^{n\times n}$ -valued Gram matrix whose  $(i,j)$ -entry is  $k(x_{i},x_{j})\in \mathcal{A}$  and  $c = (c_1,\ldots ,c_n)$ . Thus, we obtain  $G^{1 / 2}c = 0$ . Let  $\tilde{G}$  be the standard Gram matrix whose  $(i,j)$ -entry is  $\tilde{k} (x_i,x_j)$ . Since  $G = \tilde{G}\otimes a$  and  $\tilde{G}$  is invertible, the inverse of  $G$  is  $\tilde{G}^{-1}\otimes a^{-1}$ . Thus,  $G^{1 / 2}$  is invertible and we have  $c = 0$ .
Proof For  $(x,y),(x^{\prime},y^{\prime})\in \mathcal{A}\times \mathbb{R}^{d\times d}$ , we have
$$
\begin{array}{l} \left\| \left(f (x) - y\right) p \right\| ^ {2} - \left\| \left(f \left(x ^ {\prime}\right) - y ^ {\prime}\right) p \right\| ^ {2} \\ = \| f (x) p \| ^ {2} - 2 \langle f (x) p, y p \rangle + \| y p \| ^ {2} - \| f \left(x ^ {\prime}\right) p \| ^ {2} + 2 \langle f \left(x ^ {\prime}\right) p, y ^ {\prime} p \rangle - \| y ^ {\prime} p \| ^ {2} \leq 2 (C + E) ^ {2}. \\ \end{array}
$$