Title: Deep learning with kernels through RKHM and the Perron-Frobenius operator
Venue: NeurIPS 2023
Accepted: yes
Keywords: kernel method. generalization bound. C*-algebra. Perron-Frobenius operator and Koopman operator.
Abstract: Reproducing kernel Hilbert $C^*$-module (RKHM) is a generalization of reproducing kernel Hilbert space (RKHS) by means of $C^*$-algebra, and the Perron-Frobenius operator is a linear operator related to the composition of functions. Combining these two concepts, we present deep RKHM, a deep learning framework for kernel methods. We derive a new Rademacher generalization bound in this setting and provide a theoretical interpretation of benign overfitting by means of Perron-Frobenius operators. By virtue of $C^*$-algebra, the dependency of the bound on output dimension is milder than existing bounds. We show that $C^*$-algebra is a suitable tool for deep learning with kernels, enabling us to take advantage of the product structure of operators and to provide a clear connection with convolutional neural networks. Our theoretical analysis provides a new lens through which one can design and analyze deep kernel methods.

Deep Learning with Kernels through RKHM and the Perron-Frobenius Operator

Abstract

Reproducing kernel Hilbert C ∗ -module (RKHM) is a generalization of reproducing kernel Hilbert space (RKHS) by means of C ∗ -algebra, and the Perron-Frobenius operator is a linear operator related to the composition of functions. Combining these two concepts, we present deep RKHM, a deep learning framework for kernel methods. We derive a new Rademacher generalization bound in this setting and provide a theoretical interpretation of benign overfitting by means of PerronFrobenius operators. By virtue of C ∗ -algebra, the dependency of the bound on output dimension is milder than existing bounds. We show that C ∗ -algebra is a suitable tool for deep learning with kernels, enabling us to take advantage of the product structure of operators and to provide a clear connection with convolutional neural networks. Our theoretical analysis provides a new lens through which one can design and analyze deep kernel methods.

1 Introduction

Kernel methods and deep neural networks are two major topics in machine learning. Originally, they had been investigated independently. However, their interactions have been researched recently. One important perspective is deep kernel learning [1, 2, 3]. In this framework, we construct a function with the composition of functions in RKHSs, which is learned by given training data. Representer theorems were shown for deep kernel methods, which guarantee the representation of solutions of minimization problems only with given training data [4, 5]. We can combine the flexibility of deep neural networks with the representation power and solid theoretical understanding of kernel methods. Other important perspectives are neural tangent kernel [6, 7] and convolutional kernel [8], which enable us to understand neural networks using the theory of kernel methods. In addition, Bietti et al. [9] proposed a regularization of deep neural network through a kernel perspective.

The generalization properties of kernel methods and deep neural networks have been investigated. One typical technique for bounding generalization errors is to use the Rademacher complexity [10, 11]. For kernel methods, generalization bounds based on the Rademacher complexity can be derived by the reproducing property. Bounds for deep kernel methods and vector-valued RKHSs (vvRKHSs) were also derived [5, 12, 13]. Table 1 shows existing Rademacher generalization bounds for kernel methods. Generalization bounds for deep neural networks have also been actively studied [14, 15, 16, 17, 18, 19]. Recently, analyzing the generalization property using the concept of benign overfitting

Masahiro Ikeda

has emerged [20, 21]. Unlike the classical interpretation of overfitting, which is called catastrophic overfitting, it explains the phenomenon that the model fits both training and test data. For kernel regression, it has been shown that the type of overfitting can be described by an integral operator associated with the kernel [22].

In this paper, we propose deep RKHM to make the deep kernel methods more powerful. RKHM is a generalization of RKHS by means of C ∗ -algebra [23, 24, 25], where C ∗ -algebra is a generalization of the space of complex numbers, regarded as space of operators. We focus on the C ∗ -algebra of matrices in this paper. Applications of RKHMs to kernel methods have been investigated recently [26, 27]. We generalize the concept of deep kernel learning to RKHM, which constructs a map from an operator to an operator as the composition of functions in RKHMs. The product structure of operators induces interactions among elements of matrices, which enables us to capture relations between data components. Then, we derive a generalization bound of the proposed deep RKHM. By virtue of C ∗ -algebras, we can use the operator norm, which alleviates the dependency of the generalization bound on the output dimension.

We also use Perron-Frobenius operators, which are linear operators describing the composition of functions and have been applied to analyzing dynamical systems [28, 29, 30, 31], to derive the bound. The compositions in the deep RKHM are effectively analyzed by the Perron-Frobenius operators.

C ∗ -algebra and Perron-Frobenius operator are powerful tools that provide connections of the proposed deep RKHM with existing studies. Since the norm of the Perron-Frobenius operator is described by the Gram matrix associated with the kernel, our bound shows a connection of the deep RKHM with benign overfitting. In addition, the product structure of a C ∗ -algebra enables us to provide a connection between the deep RKHMs and convolutional neural networks (CNNs).

Our main contributions are as follows.

We propose deep RKHM, which is a generalization of deep kernel method by means of C ∗ -algebra. We can make use of the products in C ∗ -algebra to induce interactions among data components. We also show a representer theorem to guarantee the representation of solutions only with given data.

We derive a generalization bound for deep RKHM. The dependency of the bound on the output dimension is milder than existing bounds by virtue of C ∗ -algebras. In addition, the PerronFrobenius operators provide a connection of our bound with benign overfitting.

We show connections of our study with existing studies such as CNNs and neural tangent kernel. We emphasize that our theoretical analysis using C ∗ -algebra and Perron-Frobenius operators gives a new and powerful lens through which one can design and analyze kernel methods.

2 Preliminaries

2.1 C ∗ -algebra and reproducing kernel C ∗ -module

C ∗ -algebra, which is denoted by A in the following, is a Banach space equipped with a product and an involution satisfying the C ∗ identity (condition 3 below).

Definition 2.1 ( C ∗ -algebra) A set A is called a C ∗ -algebra if it satisfies the following conditions:

A is an algebra over C and equipped with a bijection ( · ) ∗ : A → A that satisfies the following conditions for α, β ∈ C and a, b ∈ A :

A is a normed space endowed with ∥ · ∥ A , and for a, b ∈ A , ∥ ab ∥ A ≤ ∥ a ∥ A ∥ b ∥ A holds. In addition, A is complete with respect to ∥ · ∥ A .

For a ∈ A , the C ∗ identity ∥ a ∗ a ∥ A = ∥ a ∥ 2 A holds.

Example 2.2 A typical example of C ∗ -algebras is the C ∗ -algebra of d by d circulant matrices. Another example is the C ∗ -algebra of d by d block matrices with M blocks and their block sizes are m = ( m 1 , . . . , m M ) . We denote them by Circ ( d ) and Block ( m , d ) , respectively. See [27, 32] for more details about these examples.

We now define RKHM. Let X be a non-empty set for data.

Definition 2.3 ( A -valued positive definite kernel) An A -valued map k : X × X → A is called a positive definite kernel if it satisfies the following conditions:

k ( x, y ) = k ( y, x ) ∗ for x, y ∈ X ,

∑ n i,j =1 c ∗ i k ( x i , x j ) c j is positive semi-definite for n ∈ N , c i ∈ A , x i ∈ X .

Let ϕ : X → A X be the feature map associated with k , defined as ϕ ( x ) = k ( · , x ) for x ∈ X and let M k, 0 = { ∑ n i =1 ϕ ( x i ) c i | n ∈ N , c i ∈ A , x i ∈ X ( i = 1 , . . . , n ) } . We can define an A -valued map ⟨· , ·⟩ M k : M k, 0 ×M k, 0 →A as

which enjoys the reproducing property ⟨ ϕ ( x ) , f ⟩ M k = f ( x ) for f ∈ M k, 0 and x ∈ X . The reproducing kernel Hilbert A -module (RKHM) M k associated with k is defined as the completion of M k, 0 . See, for example, the references [33, 34, 26] for more details about C ∗ -algebra and RKHM.

2.2 Perron-Frobenius operator on RKHM

We introduce Perron-Frobenius operator on RKHM [35]. Let X 1 and X 2 be nonempty sets and let k 1 and k 2 be A -valued positive definite kernels. Let M 1 and M 2 be RKHMs associated with k 1 and k 2 , respectively. Let ϕ 1 and ϕ 2 be the feature maps of M 1 and M 2 , respectively. We begin with the standard notion of linearity in RKHMs.

Definition 2.4 ( A -linear) A linear map A : M 1 → M 2 is called A -linear if for any a ∈ A and w ∈ M 1 , we have A ( wa ) = ( Aw ) a .

Definition 2.5 (Perron-Frobenius operator) Let f : X 1 →X 2 be a map. The Perron-Frobenius operator with respect to f is an A -linear map P f : M 1 →M 2 satisfying

Note that a Perron-Frobenius operator is not always well-defined since ac = bc for a, b ∈ A and nonzero c ∈ A does not always mean a = b .

Definition 2.6 ( A -linearly independent) The set { ϕ 1 ( x ) | x ∈ X} is called A -linearly independent if it satisfies the following condition: For any n ∈ N , x 1 , . . . , x n ∈ X , and c 1 , . . . , c n ∈ A , ' ∑ n i =1 ϕ 1 ( x i ) c i = 0 ' is equivalent to ' c i = 0 for all i = 1 , . . . , n '.

Lemma 2.7 If { ϕ 1 ( x ) | x ∈ X} is A -linearly independent, then P f is well-defined.

The following lemma gives a sufficient condition for A -linearly independence.

Lemma 2.8 Let k 1 = ˜ ka , i.e., k is separable, for an invertible operator a and a complex-valued kernel ˜ k . Assume { ˜ ϕ ( x ) | x ∈ X} is linearly independent (e.g. ˜ k is Gaussian or Laplacian), where ˜ ϕ is the feature map associated with ˜ k . Then, { ϕ 1 ( x ) | x ∈ X} is A -linearly independent.

Note that separable kernels are widely used in existing literature of vvRKHS (see e.g. [36]). Lemma 2.8 guarantees the validity of the analysis with Perron-Frobenius operators, at least in the separable case. This provides a condition for "good kernels" by means of the well-definedness of the Perron-Frobenius operator.

Input

Output

f1EM1

f2EM2

f3EM

f4EM4

Ao

A1

A2

A3

A4

Notation We denote the Euclidean inner product and norm by ⟨· , ·⟩ and ∥ · ∥ without the subscript. For a ∈ A , let | a | A be the unique element in A satisfying | a | 2 A = a ∗ a . If a is a matrix, ∥ a ∥ HS is the Hilbert-Schmidt norm of a . The operator norm of a linear operator A on an RKHM is denoted by ∥ A ∥ op . All the technical proofs are documented in the supplementary.

3 Deep RKHM

We now construct an L -layer deep RKHM. Let A = C d × d be the C ∗ -algebra of d by d matrices. Let A 0 , . . . , A L be C ∗ -subalgebras of A and for j = 1 , . . . , L and k j : A j -1 × A j -1 → A j be an A j -valued positive definite kernel for the j th layer. For j = 1 , . . . , L , we denote by M j the RKHM over A j associated with k j . In addition, we denote by ˜ M j the RKHM over A associated with k j . Note that M j is a subspace of ˜ M j and for u, v ∈ M j , we have ⟨ u, v ⟩ ˜ M j = ⟨ u, v ⟩ M j . We set the function space corresponding to each layer as F L = { f ∈ M L | f ( x ) ∈ R d × d for any x ∈ A L -1 , ∥ f ∥ M L ≤ B L } and F j = { f ∈ M j | f ( x ) ∈ R d × d for any x ∈ A j -1 , ∥ P f ∥ op ≤ B j } for j = 1 , . . . , L -1 . Here, for f ∈ M j with j = 1 , . . . , L -1 , P f is the Perron-Frobenius operator with respect to f from ˜ M j to ˜ M j +1 . We assume the well-definedness of these operators. Then, we set the class of deep RKHM as

Figure 1 schematically shows the structure of the deep RKHM.

Example 3.1 We can set A j = Block (( m 1 ,j , . . . , m M j ,j ) , d ) , the C ∗ -algebra of block diagonal matrices. By setting M 1 ≤ · · · ≤ M l and M l ≥ · · · ≥ M L for l < L , the number of nonzero elements in A j decreases during 1 ≤ j ≤ l and increases during l ≤ j ≤ L . This construction is regarded as an autoencoder, where the 1 ∼ l th layer corresponds to the encoder and the l +1 ∼ L th layer corresponds to the decoder.

Advantage over existing deep kernel methods Note that deep kernel methods with RKHSs and vvRKHSs have been proposed [1, 4, 2]. Autoencoders using RKHSs and vvRKHSs were also proposed [3, 5]. We have at least three advantages of deep RKHM over deep vvRKHSs or RKHSs: 1) useful structures of matrices stated in Remark 5.2, 2) availability of the operator norm in the generalization bound stated in Section 4, 3) connection with CNNs stated in Subsection 6.1.

4 Generalization Bound

We derive a generalization error for deep RKHM. To derive the bound, we bound the Rademacher complexity, which is one of the typical methods on deriving generalization bounds [11]. Let Ω be a probability space equipped with a probability measure P . We denote the integral ∫ Ω s ( ω )d P ( ω ) of a measurable function s on Ω by E[ s ] . Let x 1 , . . . , x n and y 1 , . . . , y n be input and output samples from the distributions of A 0 and A L -valued random variables x and y , respectively. Let σ i,j ( i = 1 , . . . , d, j = 1 , . . . , n ) be i.i.d. Rademacher variables. Let σ j = ( σ 1 ,j , . . . , σ d,j ) . For an R d -valued function class F and x = ( x 1 , . . . , x n ) , the empirical Rademacher complexity ˆ R n ( x , F ) is defined by ˆ R n ( x , F ) := E[sup f ∈F ∑ n i =1 ⟨ σ i , f ( x i ) ⟩ ] /n .

4.1 Bound for shallow RKHMs

We use the operator norm to derive a bound, whose dependency on output dimension is milder than existing bounds. Availability of the operator norm is one of the advantages of considering

C ∗ -algebras (RKHMs) instead of vectors (vvRKHSs). Note that although we can also use the HilbertSchmidt norm for matrices, it tends to be large as the dimension d becomes large. On the other hand, the operator norm is defined independently of the dimension d . Indeed, the Hilbert-Schmidt norm of a matrix a ∈ C d × d is calculated as ( ∑ d i =1 s 2 i ) 1 / 2 , where s i is the i th singular value of a . The operator norm of a is the largest singular value of a .

To see the derivation of the bound, we first focus on the case of L = 1 , i.e., the network is shallow. Let E > 0 . For a space F of A -valued functions on A 0 , let G ( F ) = { ( x, y ) ↦→ f ( x ) -y | f ∈ F , ∥ y ∥ A ≤ E } . The following theorem shows a bound for RKHMs with the operator norm.

Theorem 4.1 Assume there exists D > 0 such that ∥ k 1 ( x, x ) ∥ A ≤ D for any x ∈ A 0 . Let ˜ K = 4 √ 2( √ DB 1 + E ) B 1 and ˜ M = 6( √ DB 1 + E ) 2 . Let δ ∈ (0 , 1) . Then, for any g ∈ G ( F 1 ) , where F 1 is defined in Section 3, with probability at least 1 -δ , we have

Theorem 4.1 is derived by the following lemmas. We first fix a vector p ∈ R d and consider the operator-valued loss function acting on p . We first show a relation between the generalization error and the Rademacher complexity of vector-valued functions. Then, we bound the Rademacher complexity. Since the bound does not depend on p , we can finally remove the dependency on p .

Lemma 4.2 Let F be a function class of R d × d -valued functions on A 0 bounded by C (i.e., ∥ f ( x ) ∥ A ≤ C for any x ∈ A 0 ). Let ˜ G ( F , p ) = { ( x, y ) ↦→∥ ( f ( x ) -y ) p ∥ 2 | f ∈ F , ∥ y ∥ A ≤ E } and M = 2( C + E ) 2 . Let p ∈ R d satisfy ∥ p ∥ = 1 and let δ ∈ (0 , 1) . Then, for any g ∈ ˜ G ( F , p ) , with probability at least 1 -δ , we have

Lemma 4.3 With the same notations in Lemma 4.2, let K = 2 √ 2( C + E ) . Then, we have ˆ R n ( x , G ( F , p )) ≤ K ˆ R n ( x , F p ) , where F p = { x ↦→ f ( x ) p | f ∈ F} .

Lemma 4.4 Let p ∈ R d satisfy ∥ p ∥ = 1 . For F 1 defined in Section 3, we have

4.2 Bound for deep RKHMs

We now generalize Theorem 4.1 to the deep setting ( L ≥ 2 ) using the Perron-Frobenius operators.

Theorem 4.5 Assume there exists D > 0 such that ∥ k L ( x, x ) ∥ A ≤ D for any x ∈ A 0 . Let ˜ K = 4 √ 2( √ DB L + E ) B 1 · · · B L and ˜ M = 6( √ DB L + E ) 2 . Let δ ∈ (0 , 1) . Then, for any g ∈ G ( F deep L ) , with probability at least 1 -δ , we have

We use the following proposition and Lemmas 4.2 and 4.3 to show Theorem 4.5. The key idea of the proof is that by the reproducing property and the definition of the Perron-Frobenius operator, we get f L ◦ · · · ◦ f 1 ( x ) = ⟨ ϕ L ( f L -1 ◦ · · · ◦ f 1 ( x )) , f L ⟩ ˜ M L = 〈 P f L -1 · · · P f 1 ϕ ( x ) , f L 〉 ˜ M L .

Proposition 4.6 Let p ∈ R d satisfy ∥ p ∥ = 1 . Then, we have

Here, ˜ V ( x ) is the submodule of ˜ M 1 generated by ϕ 1 ( x 1 ) , . . . ϕ 1 ( x n ) .

Corollary 4.7 Let p ∈ R d satisfy ∥ p ∥ = 1 . Then, we have

Comparison to deep vvRKHS We can also regard C d × d as the Hilbert space equipped with the Hilbert-Schmidt inner product, i.e., we can flatten matrices and get d 2 -dimensional Hilbert space. In this case, the corresponding operator-valued kernel is the multiplication operator of k ( x, y ) , which we denote by M k ( x,y ) . Then, we can apply existing results for vvRKHSs [5, 12], which involve the term ( ∑ n i =1 tr( M k ( x i ,x i ) )) 1 / 2 . It is calculated as

Thus, using the existing approaches, we have the factor ( d ∑ n i =1 tr k ( x i , x i )) 1 / 2 . On the other hand, we have the smaller factor ( ∑ n i =1 tr k ( x i , x i )) 1 / 2 in Theorems 4.1 and 4.5. Using the operator norm, we can reduce the dependency on the dimension d .

5 Learning Deep RKHMs

We focus on the practical learning problem. To learn deep RKHMs, we consider the following minimization problem based on the generalization bound derived in Section 4:

The second term regarding the Perron-Frobenius operators comes from the bound in Proposition 4.6. We try to reduce the generalization error by reducing the magnitude of the norm of the PerronFrobenius operators.

5.1 Representer theorem

We first show a representer theorem to guarantee that a solution of the minimization problem (1) is represented only with given samples.

Proposition 5.1 Let h : A n ×A n → R + be an error function, let g 1 be an R + -valued function on the space of bounded linear operators on ˜ M 1 , and let g 2 : R + → R + satisfy g 2 ( a ) ≤ g 2 ( b ) for a ≤ b . Assume the following minimization problem has a solution:

Then, there exists a solution admitting a representation of the form f j = ∑ n i =1 ϕ j ( x j -1 i ) c i,j for some c 1 ,j , . . . , c n,j ∈ A and for j = 1 , . . . , L . Here, x j i = f j ◦ · · · ◦ f 1 ( x i ) for j = 1 , . . . , L and x 0 i = x i .

̸

Remark 5.2 An advantage of deep RKHM compared to deep vvRKHS is that we can make use of the structure of matrices. For example, the product of two diagonal matrices is calculated by the elementwise product of diagonal elements. Thus, when A 1 = · · · = A L = Block ((1 , . . . , 1) , d ) , interactions among elements in an input are induced only by the kernels, not by the product k j ( x, x j -1 i ) · c i,j . That is, the form of interactions does not depend on the learning parameter c i,j . On the other hand, if we set A j = Block ( m j , d ) with m j = ( m 1 ,j , . . . , m M j ,j ) = (1 , . . . , 1) , then at the j th layer, we get interactions among elements in the same block through the product k j ( x, x j -1 i ) · c i,j . In this case, the form of interactions is learned through c i,j . For example, the part M 1 ≤ · · · ≤ M l (the encoder) in Example 3.1 tries to gradually reduce the dependency among elements in the output of each layer and describe the input with a small number of variables. The part M l ≥ · · · ≥ M L (the decoder) tries to increase the dependency.

5.2 Computing the norm of the Perron-Frobenius operator

We discuss the practical computation and the role of the factor ∥ P f L -1 · · · P f 1 | ˜ V ( x ) ∥ op in Eq. (1). Let G j ∈ A n × n be the Gram matrix whose ( i, l ) -entry is k j ( x j -1 i , x j -1 l ) ∈ A .

Proposition 5.3 For j = 1 , L , let [ ϕ j ( x j -1 1 ) , . . . , ϕ j ( x j -1 n )] R j = Q j be the QR decomposition of [ ϕ j ( x j -1 1 ) , . . . , ϕ j ( x j -1 n )] . Then, we have ∥ P f L -1 · · · P f 1 | ˜ V ( x ) ∥ op = ∥ R ∗ L G L R 1 ∥ op .

The computational cost of ∥ R ∗ L G L R 1 ∥ op is expensive if n is large since computing R 1 and R L is expensive. Thus, we consider upper bounding ∥ R ∗ L G L R 1 ∥ op by a computationally efficient value.

Proposition 5.4 Assume G L is invertible. Then, we have

Since G 1 is independent of f 1 , . . . , f L , to make the value ∥ P f L -1 · · · P f 1 | ˜ V ( x ) ∥ op small, we try to make the norm of G L and G -1 L small according to Proposition 5.4. For example, instead of the second term regarding the Perron-Frobenius operators in Eq. (1), we can consider the following term with η > 0 :

Note that the term depends on the training samples x 1 , . . . , x n . The situation is different from the third term in Eq. (1) since the third term does not depend on the training samples before applying the representer theorem. If we try to minimize a value depending on the training samples, the model seems to be more specific for the training samples, and it may cause overfitting. Thus, the connection between the minimization of the second term in Eq. (1) and generalization cannot be explained by the classical argument about generalization and regularization. However, as we will see in Subsection 6.2, it is related to benign overfitting and has a good effect on generalization.

Remark 5.5 The inequality (2) implies that as { ϕ L ( x L -1 1 ) , . . . , ϕ L ( x L -1 n ) } becomes nearly linearly dependent, the Rademacher complexity becomes large. By the term ∥ ( ηI + G L ) -1 ∥ in Eq. (1) , the function f L -1 ◦ · · · ◦ f 1 is learned so that it separates x 1 , . . . , x n well.

Remark 5.6 To evaluate f 1 ◦ · · · ◦ f L ( x i ) for i = 1 , . . . , n , we have to construct a Gram matrix G j ∈ A n × n j , and compute the product of the Gram matrix and a vector c j ∈ A n j for j = 1 , . . . , L . The computational cost of the construction of the Gram matrices does not depend on d . The cost of computing G j c j is O ( n 2 ˜ d j ) , where ˜ d j is the number of nonzero elements in the matrices in A j . Note that if A j is the C ∗ -algebra of block diagonal matrices, then we have ˜ d j ≪ d 2 . Regarding the cost with respect to n , if the positive definite kernel is separable, we can use the random Fourier features [37] to replace the factor n 2 with mn for a small integer m ≪ n .

6 Connection and Comparison with Existing Studies

The proposed deep RKHM is deeply related to existing studies by virtue of C ∗ -algebra and the Perron-Frobanius operators. We discuss the connection below.

6.1 Connection with CNN

The proposed deep RKHM has a duality with CNNs. Let A 0 = · · · = A L = Circ ( d ) , the C ∗ -algebra of d by d circulant matrices. For j = 1 , . . . , L , let a j ∈ A j . Let k j be an A j -valued positive definite kernel defined as k j ( x, y ) = ˜ k j ( a j x, a j y ) , where ˜ k j is an A j -valued function. The A j -valued positive definite kernel makes the output of each layer become a circulant matrix, which enables us to apply the convolution as the product of the output and a parameter. Then, f j is represented as f j ( x ) = ∑ n i =1 ˜ k j ( a j x, a j x j -1 i ) c i,j for some c i,j ∈ A j . Thus, at the j th layer, the input x is multiplied by a j and is transformed nonlinearly by σ j ( x ) = ∑ n i =1 k j ( x, a j x j -1 i ) c i,j . Since the product of two circulant matrices corresponds to the convolution, a j corresponds to a filter. In

addition, σ j corresponds to the activation function at the j th layer. Thus, the deep RKHM with the above setting corresponds to a CNN. The difference between the deep RKHM and the CNN is the parameters that we learn. Whereas for the deep RKHM, we learn the coefficients c 1 ,j , . . . , c n,j , for the CNN, we learn the parameter a j . In other words, whereas for the deep RKHM, we learn the activation function σ j , for the CNN, we learn the filter a j . It seems reasonable to interpret this difference as a consequence of solving the problem in the primal or in the dual. In the primal, the number of the learning parameters depends on the dimension of the data (or the filter for convolution), while in the dual, it depends on the size of the data.

Remark 6.1 The connection between CNNs and shallow RKHMs has already been studied [27]. However, the existing study does not provide the connection between the filter and activation function. The above investigation shows a more clear layer-wise connection of deep RKHMs with CNNs.

6.2 Connection with benign overfitting

Benign overfitting is a phenomenon that the model fits any amount of data yet generalizes well [20, 21]. For kernel regression, Mallinar et al. [22] showed that if the eigenvalues of the integral operator associated with the kernel function over the data distribution decay slower than any powerlaw decay, than the model exhibits benign overfitting. The Gram matrix is obtained by replacing the integral with the sum over the finite samples. The inequality (2) suggests that the generalization error becomes smaller as the smallest and the largest eigenvalues of the Gram matrix get closer, which means the eigenvalue decay is slower. Combining the observation in Remark 5.5, we can interpret that as the right-hand side of the inequality (2) becomes smaller, the outputs of noisy training data at the L -1 th layer tend to be more separated from the other outputs. In other words, for the random variable x following the data distribution, f L -1 ◦ · · · ◦ f 1 is learned so that the distribution of f L -1 ◦ · · · ◦ f 1 ( x ) generates the integral operator with more separated eigenvalues, which appreciates benign overfitting. We will also observe this phenomenon numerically in Section 7 and Appendix C.3.2. Since the generalization bound for deep vvRKHSs [5] is described by the Lipschitz constants of the feature maps and the norm of f j for j = 1 , . . . , L , this type of theoretical interpretation regarding benign overfitting is not available for the existing bound for deep vvRKHSs.

Remark 6.2 The above arguments about benign overfitting are valid only for deep RKHMs, i.e., the case of L ≥ 2 . If L = 1 (shallow RKHM), then the Gram matrix G L = [ k ( x i , x j )] i,j is fixed and determined only by the training data and the kernel. On the other hand, if L ≥ 2 (deep RKHM), then G L = [ k L ( f L -1 ◦· · · ◦ f 1 ( x i ) , f L -1 ◦· · · ◦ f 1 ( x j ))] i,j depends also on f 1 , . . . , f L -1 . As a result, by adding the term using G L to the loss function, we can learn proper f 1 , . . . , f L -1 so that they make the right-hand side of Eq. (2) small, and the whole network overfits benignly. As L becomes large, the function f L -1 ◦ · · · ◦ f 1 changes more flexibly to attain a smaller value of the term. This is an advantage of considering a large L .

6.3 Connection with neural tangent kernel

̸

Neural tangent kernel has been investigated to understand neural networks using the theory of kernel methods [6, 7]. Generalizing neural networks to C ∗ -algebra, which is called the C ∗ -algebra network, is also investigated [32, 38]. We define a neural tangent kernel for the C ∗ -algebra network and develop a theory for combining neural networks and deep RKHMs as an analogy of the existing studies. Consider the C ∗ -algebra network f : A N 0 → A over A with A -valued weight matrices W j ∈ A N j × N j -1 and element-wise activation functions σ j : f ( x ) = W L σ L -1 ( W L -1 · · · σ 1 ( W 1 x ) · · · ) . The ( i, j ) -entry of f ( x ) is f i ( x j ) = W L,i σ L -1 ( W L -1 · · · σ 1 ( W 1 x j ) · · · ) , where x j is the j th column of x regarded as x ∈ C dN 0 × d and W L,i is the i th row of W L regarded as W L ∈ C d × dN L -1 . Thus, the ( i, j ) -entry of f ( x ) corresponds to the output of the network f i ( x j ) . We can consider the neural tangent kernel for each f i . Chen and Xu [7] showed that the RKHS associated with the neural tangent kernel restricted to the sphere is the same set as that associated with the Laplacian kernel. Therefore, the i th row of f ( x ) is described by the shallow RKHS associated with the neural tangent kernel k NT i . Let k NN be the A -valued positive definite kernel whose ( i, j ) -entry is k NT i for i = j and 0 for i = j . Then, for any function g ∈ M k NN , the elements in the i th row of g are in the RKHS associated with k NT i , i.e., associated with the Laplacian kernel. Thus, f is described by this shallow RKHM.

Remark 6.3 We can combine the deep RKHM and existing neural networks by replacing some f j in our model with an existing neural network. The above observation enables us to apply our results in Section 4 to the combined network.

6.4 Comparison to bounds for classical neural networks

Existing bounds for classical neural networks typically depend on the product or sum of matrix ( p, q ) norm of all the weight matrices W j [14, 15, 16, 17]. Typical bound is O ( √ 1 /n ∏ L j =1 ∥ W j ∥ HS ) . Note that the Hilbert-Schmidt norm is the matrix (2 , 2) norm. Unlike the operator norm, the matrix ( p, q ) norm tends to be large as the width of the layers becomes large. On the other hand, the dependency of our bound on the width of the layer is not affected by the number of layers in the case where the kernels are separable. Indeed, assume we set k 1 = ˜ k 1 a 1 and k L = ˜ k L a L for some complex-valued kernels ˜ k 1 and ˜ k 2 , a 1 ∈ A 1 , and a L ∈ A L . Then by Proposition 5.3, the factor α ( L ) := ∥ P f L -1 · · · P f 1 | ˜ V ( x ) ∥ op is written as ∥ ˜ R ∗ L ˜ G L ˜ R 1 ⊗ a 2 L a 1 ∥ op = ∥ ˜ R ∗ L ˜ G L ˜ R 1 ∥ op ∥ a 2 L a 1 ∥ A for some ˜ R L , ˜ G L , ˜ R 1 ∈ C n × n . Thus, it is independent of d . The only part depending on d is tr k 1 ( x i , x i ) , which results in the bound O ( α ( L ) √ d/n ) . Note that the width of the j th layer corresponds to the number of nonzero elements in a matrix in A j . We also discuss in Appendix B the connection of our bound with the bound by Koopman operators, the adjoints of the Perron-Frobenius operators [18].

7 Numerical Results

We numerically confirm our theory and the validity of the proposed deep RKHM.

Comparison to vvRKHS We compared the generalization property of the deep RKHM to the deep vvRKHS with the same positive definite kernel. For d = 10 and n = 10 , we set x i = ( az i ) 2 + ϵ i as input samples, where a ∈ R 100 × 10 and z i ∈ R 10 are randomly generated by N (0 , 0 . 1) , the normal distribution of mean 0 and standard deviation 0.1, ( · ) 2 is the elementwise product, and ϵ i is the random noise drawn from N (0 , 1 e -3) . We reshaped x i to a 10 by 10 matrix. We set L = 3 and k j = ˜ kI for j = 1 , 2 , 3 , where ˜ k is the Laplacian kernel. For RKHMs, we set A 1 = Block ((1 , . . . , 1) , d ) , A 2 = Block ((2 , . . . , 2) , d ) , and A 3 = C d × d . This is the autoencoder mentioned in Example 3.1. For vvRKHSs, we set the corresponding Hilbert spaces with the Hilbert-Schmidt inner product. We set the loss function as 1 /n ∥ ∑ n i =1 | f ( x i ) -x i | 2 A ∥ A for the deep RKHM and as 1 /n ∑ n i =1 ∥ f ( x i ) -x i ∥ 2 HS for the deep vvRKHS. Here, f = f 3 ◦ f 2 ◦ f 1 . We did not add any terms to the loss function to see how the loss function with the operator norm affects the generalization performance. We computed the same value ∥ E[ | f ( x ) -x | 2 A ] ∥ A -1 /n ∥ ∑ n i =1 | f ( x i ) -x i | 2 A ∥ A for both RKHM and vvRKHS. Figure 2 (a) shows the results. We can see that the deep RKHM generalizes better than the deep vvRKHS, only with the loss function and without any additional terms.

Observation about benign overfitting We analyzed the overfitting numerically. For d = 10 and n = 1000 , we randomly sampled d by d diagonal matrices x 1 , . . . , x n ∈ A 0 from the normal distribution N (0 , 0 . 1) . We set y i = x 2 i + ϵ i for i = 1 , . . . , n , where ϵ i is a noise drawn from the normal distribution N (0 , 0 . 001) . The magnitude of the noise is 10 %of x 2 i . In addition, we set L = 2 , A 1 = C d × d , A 2 = Block ((1 , . . . , 1) , d ) , and k j as the same kernel as the above experiment. The additional term to the loss function is set as λ 1 ( ∥ ( ηI + G L ) -1 ∥ op + ∥ G L ∥ op ) + λ 2 ∥ f L ∥ 2 M L , where η = 0 . 01 and λ 2 = 0 . 01 according to Subsection 5.2. We computed the generalization error for the cases of λ 1 = 0 and λ 1 = 10 2 . Figure 2 (b) shows the result. We can see that the generalization error saturates without the additional term motivated by the Perron-Frobenius operator. On the other hand, with the additional term, the generalization error becomes small, which is the effect of benign overfitting.

Comparison to CNN We compared the deep RKHM to a CNN on the classification task with MNIST [39]. We set d = 28 and n = 20 . We constructed a deep RKHM combined with a neural network with 2 dense layers. For the deep RKHM, we set L = 2 , A 0 = C d × d , A 1 = Block ((7 , 7 , 7 , 7) , d ) , and A 2 = Block ((4 , . . . , 4) , d ) . Then, two dense layers are added. See Subsection 6.3 about combining the deep RKHM with neural networks. Regarding the additional term to the loss function, we set the same term with the previous experiment with λ 2 = 0 . 001 and

(a)

(b)

(c)

Error

0.00020

Error

x1e-6

20

0.6

Test Accuracy

Generalization

0.00015

Generalization

10

0.5

0.00010

5

0.00005

入1=102

0.4

RKHM,入1=0

2

入1=0

CNN

0.00000

0.3

vvRKHS

200

400

600

0

100

200

300

400

Epochs

Epochs

set λ 1 = 1 or λ 1 = 0 . To compare the deep RKHM to CNNs, we also constructed a network by replacing the deep RKHM with a CNN. The CNN is composed of 2 layers with 7 × 7 and 4 × 4 filters. Figure 2 (c) shows the test accuracy of these networks. We can see that the deep RKHM outperforms the CNN. In addition, we can see that the test accuracy is higher if we add the term regarding the Perron-Frobenius operators. We discuss the memory consumption and the computational cost of each of the deep RKHM and the CNN in Appendix C.3, and we empirically show that the deep RKHM outperforms the CNN that has the same size of learning parameters as the deep RKHM. We also show additional results about benign overfitting in Appendix C.3.

8 Conclusion and Limitations

In this paper, we proposed deep RKHM and analyzed it through C ∗ -algebra and the Perron-Frobenius operators. We derived a generalization bound, whose dependency on the output dimension is alleviated by the operator norm, and which is related to benign overfitting. We showed a representer theorem about the proposed deep RKHM, and connections with existing studies such as CNNs and neural tangent kernel. Our theoretical analysis shows that C ∗ -algebra and Perron-Frobenius operators are effective tools for analyzing deep kernel methods. The main contributions of this paper are our theoretical results with C ∗ -algebra and the Perron-Frobenius operators. More practical investigations are required for further progress. For example, although we numerically showed the validity of our method for the case where the number of samples is limited (the last experiment in Section 7), more experimental results for the case where the number of samples is large are useful for further analysis. Also, although we can apply random Fourier features (Remark 5.6) to reduce the computational costs, studying more efficient methods specific to deep RKHM remains to be investigated in future work. As for the theoretical topic, we assumed the well-definedness of the Perron-Frobenius operators. Though separable kernels with invertible matrices, which are typical examples of kernels, satisfy the assumption, generalization of our analysis to other kernels should also be studied in future work.

References

Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. In Proceedings of Advances in Neural Information Processing Systems (NIPS) 22 , 2009.

Sebastian W. Ober, Carl E. Rasmussen, and Mark van der Wilk. The promises and pitfalls of deep kernel learning. In Proceedings of the 37th Conference on Uncertainty in Artificial Intelligence (UAI) , 2021.

Behnam Gholami and Abolfazl Hajisami. Kernel auto-encoder for semi-supervised hashing. In Proceedings of 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) , 2016.

Bastian Bohn, Michael Griebel, and Christian Rieger. A representer theorem for deep kernel learning. Journal of Machine Learning Research , 20(64):1-32, 2019.

Pierre Laforgue, Stéphan Clémençon, and Florence d'Alche Buc. Autoencoding any data through kernel autoencoders. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS) , 2019.

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In In proceedings of Advances in Neural Information Processing Systems (NeurIPS) 31 , 2018.

Lin Chen and Sheng Xu. Deep neural tangent kernel and laplace kernel have the same RKHS. In Proceedings of the 9th International Conference on Learning Representations (ICLR) , 2021.

Julien Mairal, Piotr Koniusz, Zaid Harchaoui, and Cordelia Schmid. Convolutional kernel networks. In Proceedings of the Advances in Neural Information Processing Systems (NIPS) 27 , 2014.

Alberto Bietti, Grégoire Mialon, Dexiong Chen, and Julien Mairal. A kernel perspective for regularizing deep neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML) , 2019.

Peter L. Bartlett and Shahar Mendelson. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3:463-482, 2002.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning . MIT press, 2018.

Vikas Sindhwani, Ha Quang Minh, and Aurélie C. Lozano. Scalable matrix-valued kernel learning for high-dimensional nonlinear multivariate regression and granger causality. In Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence (UAI) , 2013.

Riikka Huusari and Hachem Kadri. Entangled kernels - beyond separability. Journal of Machine Learning Research , 22(24):1-40, 2021.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Proceedings of the 2015 Conference on Learning Theory (COLT) , 2015.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Proceedings of the 2018 Conference On Learning Theory (COLT) , 2018.

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. In Proceedings of Advances in Neural Information Processing Systems (NIPS) 31 , 2017.

Haotian Ju, Dongyue Li, and Hongyang R Zhang. Robust fine-tuning of deep neural networks with Hessian-based generalization guarantees. In Proceedings of the 39th International Conference on Machine Learning (ICML) , 2022.

Yuka Hashimoto, Sho Sonoda, Isao Ishikawa, Atsushi Nitanda, and Taiji Suzuki. Koopmanbased bound for generalization: New aspect of neural networks regarding nonlinear noise filtering. arXiv: 2302.05825, 2023.

Taiji Suzuki, Hiroshi Abe, and Tomoaki Nishimura. Compression based bound for noncompressed network: unified generalization error analysis of large compressible deep neural network. In Proceedings of the 8th International Conference on Learning Representations (ICLR) , 2020.

Mikhail Belkin, Alexander Rakhlin, and Alexandre B. Tsybakov. Does data interpolation contradict statistical optimality? In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS) , 2019.

Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Neil Rohit Mallinar, James B Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward a refined taxonomy of overfitting. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) 37 , 2022.

Jaeseong Heo. Reproducing kernel Hilbert C ∗ -modules and kernels associated with cocycles. Journal of Mathematical Physics , 49(10):103507, 2008.

Shigeru Itoh. Reproducing kernels in modules over C ∗ -algebras and their applications. Journal of Mathematics in Nature Science , pages 1-20, 1990.

Mohammad S. Moslehian. Vector-valued reproducing kernel Hilbert C ∗ -modules. Complex Analysis and Operator Theory , 16(1):Paper No. 2, 2022.

Yuka Hashimoto, Isao Ishikawa, Masahiro Ikeda, Fuyuta Komura, Takeshi Katsura, and Yoshinobu Kawahara. Reproducing kernel Hilbert C ∗ -module and kernel mean embeddings. Journal of Machine Learning Research , 22(267):1-56, 2021.

Yuka Hashimoto, Masahiro Ikeda, and Hachem Kadri. Learning in RKHM: a C ∗ -algebraic twist for kernel machines. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS) , 2023.

Yoshinobu Kawahara. Dynamic mode decomposition with reproducing kernels for Koopman spectral analysis. In Advances in Neural Information Processing Systems (NIPS) 29 , pages 911-919, 2016.

Isao Ishikawa, Keisuke Fujii, Masahiro Ikeda, Yuka Hashimoto, and Yoshinobu Kawahara. Metric on nonlinear dynamical systems with Perron-Frobenius operators. In Advances in Neural Information Processing Systems (NeurIPS) 31 , pages 2856-2866, 2018.

Dimitrios Giannakis and Suddhasattwa Das. Extraction and prediction of coherent patterns in incompressible flows through space-time Koopman analysis. Physica D: Nonlinear Phenomena , 402:132211, 2020.

Yuka Hashimoto, Isao Ishikawa, Masahiro Ikeda, Yoichi Matsuo, and Yoshinobu Kawahara. Krylov subspace method for nonlinear dynamical systems with random noise. Journal of Machine Learning Research , 21(172):1-29, 2020.

Ryuichiro Hataya and Yuka Hashimoto. Noncommutative C ∗ -algebra net: Learning neural networks with powerful product structure in C ∗ -algebra. arXiv: 2302.01191, 2023.

E. Christopher Lance. Hilbert C ∗ -modules - a Toolkit for Operator Algebraists . London Mathematical Society Lecture Note Series, vol. 210. Cambridge University Press, 1995.

Gerard J. Murphy. C ∗ -Algebras and Operator Theory . Academic Press, 1990.

Yuka Hashimoto, Isao Ishikawa, Masahiro Ikeda, Fuyuta Komura, Takeshi Katsura, and Yoshinobu Kawahara. Analysis via orthonormal systems in reproducing kernel hilbert c ∗ -modules and applications. arXiv: 2003.00738, 2020.

Mauricio A Álvarez, Lorenzo Rosasco, Neil D Lawrence, et al. Kernels for vector-valued functions: A review. Foundations and Trends® in Machine Learning , 4(3):195-266, 2012.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Proceedings of the Advances in Neural Information Processing Systems 20 (NIPS) , 2007.

Yuka Hashimoto, Zhao Wang, and Tomoko Matsui. C ∗ -algebra net: a new approach generalizing neural network parameters to C ∗ -algebra. In Proceedings of the 39th International Conference on Machine Learning (ICML) , 2022.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Andreas Maurer. A vector-contraction inequality for rademacher complexities. In Proceedings of the 27th International Conference on Algorithmic Learning Theory (ALT) , 2016.

A Proofs

We provide the proofs of the theorems, propositions, and lemmas in the main paper.

Lemma 2.7 If { ϕ 1 ( x ) | x ∈ X} is A -linearly independent, then P f is well-defined.

Proof Assume ∑ n i =1 ϕ 1 ( x i ) c i = ∑ n i =1 ϕ 1 ( x i ) d i for n ∈ N , c i , d i ∈ A . Since { ϕ 1 ( x ) | x ∈ X} is A -linearly independent, we have c i = d i for i = 1 , . . . , n . Thus, P f ∑ n i =1 ϕ 1 ( x i ) c i = ∑ n i =1 ϕ 1 ( f ( x i )) c i = ∑ n i =1 ϕ 1 ( f ( x i )) d i = P f ∑ n i =1 ϕ 1 ( x i ) d i . □

Lemma 2.8 Let k 1 = ˜ ka , i.e., k is separable, for an invertible operator a and a complex-valued kernel ˜ k . Assume { ˜ ϕ ( x ) | x ∈ X} is linearly independent (e.g. ˜ k is Gaussian or Laplacian), where ˜ ϕ is the feature map associated with ˜ k . Then, { ϕ 1 ( x ) | x ∈ X} is A -linearly independent.

Proof If ∑ n i =1 ϕ 1 ( x i ) c i = 0 , we have

where G is the A n × n -valued Gram matrix whose ( i, j ) -entry is k ( x i , x j ) ∈ A and c = ( c 1 , . . . , c n ) . Thus, we obtain G 1 / 2 c = 0 . Let ˜ G be the standard Gram matrix whose ( i, j ) -entry is ˜ k ( x i , x j ) . Since G = ˜ G ⊗ a and ˜ G is invertible, the inverse of G is ˜ G -1 ⊗ a -1 . Thus, G 1 / 2 is invertible and we have c = 0 . □

Lemma 4.2 Let F be a function class of R d × d -valued functions on A 0 bounded by C (i.e., ∥ f ( x ) ∥ A ≤ C for any x ∈ A 0 ). Let ˜ G ( F , p ) = { ( x, y ) ↦→∥ ( f ( x ) -y ) p ∥ 2 | f ∈ F , ∥ y ∥ A ≤ E } and M = 2( C + E ) 2 . Let p ∈ R d satisfy ∥ p ∥ = 1 and let δ ∈ (0 , 1) . Then, for any g ∈ ˜ G ( F , p ) , with probability at least 1 -δ , we have

Proof For ( x, y ) , ( x ′ , y ′ ) ∈ A × R d × d , we have

In addition, we have

Since ( x, y ) ↦→∥ g ( x, y ) p ∥ 2 is a real-valued map, by Theorem 3.3 by Mohri et al. [11], we have

The inequality ∥ ( ∑ n i =1 | g ( x i , y i ) | 2 A ) 1 / 2 p ∥ 2 ≤ ∥ ∑ n i =1 | g ( x i , y i ) | 2 A ∥ A completes the proof. □

Lemma 4.3 With the same notations in Lemma 4.2, let K = 2 √ 2( C + E ) . Then, we have ˆ R n ( x , ˜ G ( F , p )) ≤ K ˆ R n ( x , F p ) , where F p = { x ↦→ f ( x ) p | f ∈ F} .

Proof Let h i ( z ) = ∥ z -y i p ∥ 2 for z ∈ { f ( x ) p | f ∈ F} ⊆ R d and i = 1 , . . . , n . The Lipschitz constant of h i is calculated as follows: We have

Thus, we have | h i ( z ) -h i ( z ′ ) | ≤ 2( C + E ) ∥ z -z ′ ∥ . By Corollary 4 by Maurer [40], the statement is proved. □

Lemma 4.4 Let p ∈ R d satisfy ∥ p ∥ = 1 . For F 1 defined in Section 3, we have

Proof We have

where for p ∈ C d and f, g ∈ ˜ M 1 , ( f, g ) ˜ M 1 ,p is the semi-inner product defined by ( f, g ) ˜ M 1 ,p = 〈 p, ⟨ f, g ⟩ ˜ M 1 p 〉 and | f | 2 ˜ M 1 = ⟨ f, f ⟩ ˜ M 1 . In addition, the inequality (3) is by the Cauchy-Schwartz inequality and the inequality (5) is by the Jensen's inequality. Note that the Cauchy-Schwartz inequality is still valid for semi-inner products. The inequality (4) is derived by the inequality

Theorem 4.1 Assume there exists D > 0 such that ∥ k 1 ( x, x ) ∥ A ≤ D for any x ∈ A 0 . Let ˜ K = 4 √ 2( √ DB 1 + E ) B 1 and ˜ M = 6( √ DB 1 + E ) 2 . Let δ ∈ (0 , 1) . Then, for any g ∈ G ( F 1 ) , where F 1 is defined in Section 3, with probability at least 1 -δ , we have

Proof For f ∈ M 1 and x ∈ A 0 , we have

Thus, we set C as √ DB 1 and apply Lemmas 4.2, 4.3, and 4.4. Then, for p ∈ R d satisfy ∥ p ∥ = 1 , with probability at least 1 -δ , we have

Therefore, we obtain

which completes the proof.

Proposition 4.6 We have

Here, ˜ V ( x ) is the submodule of ˜ M 1 generated by ϕ 1 ( x 1 ) , . . . ϕ 1 ( x n ) .

The following lemma by Lance [33, Proposition 1.2] is used in proving Proposition 4.6. Here, for a, b ∈ A , a ≤ b means b -a is Hermitian positive definite.

Lemma A.1 Let M and N be Hilbert A -modules and let A be an A -linear operator from M to N . Then, we have | Aw | 2 N ≤ ∥ A ∥ 2 op | w | 2 M .

Proof

where the inequality (6) is by the Cauchy-Schwartz inequality and the inequality (7) is by Lemma A.1. □

Proposition 5.1 Let h : A n ×A n → R + be an error function, let g 1 be an R + -valued function on the space of bounded linear operators on ˜ M 1 , and let g 2 : R + → R + satisfy g 2 ( a ) ≤ g 2 ( b ) for a ≤ b . Assume the following minimization problem has a solution:

Then, there exists a solution admitting a representation of the form f j = ∑ n i =1 ϕ j ( x j -1 i ) c i,j for some c 1 ,j , . . . , c n,j ∈ A and for j = 1 , . . . , L . Here, x j i = f j ◦ · · · ◦ f 1 ( x i ) for j = 1 , . . . , L and x 0 i = x i .

Proof Let V j ( x ) be the submodule of M j generated by ϕ j ( x j -1 1 ) , . . . , ϕ j ( x j -1 n ) . Let f j = f ∥ j + f ⊥ j , where f ∥ j ∈ V j ( x ) and f ⊥ j ∈ V j ( x ) ⊥ . Then, for j = 1 , . . . , L -1 , we have

In addition, we have

where ˜ V j ( x ) is the submodule of ˜ M j generated by ϕ j ( x j -1 1 ) , . . . ϕ j ( x j -1 n ) . In the same manner as Eq. (8), we obtain

Thus, we have P f j | ˜ V j ( x ) = P f ∥ j | ˜ V j ( x ) . Furthermore, we have

where the last inequality is derived from the fact that 〈 f ∥ L , f ∥ L 〉 M L + 〈 f ⊥ L , f ⊥ L 〉 M L -〈 f ⊥ L , f ⊥ L 〉 M L is positive and Theorem 2.2.5 (3) by Murphy [34]. As a result, the statement is proved. □

Proof The result is derived by the identities

Proposition 5.4 Assume G L is invertible. Then, we have

Proof Since G L is invertible, by Proposition 5.3, ∥ R ∗ L G L R 1 ∥ op is bounded as

B Connection with Generalization Bound with Koopman Operators

Hashimoto et al. [18] derived a generalization bound for the classical neural network composed of linear transformations and activation functions using Koopman operators. The Koopman operator is the adjoint of the Perron-Frobenius operator. In their analysis, they set the final transformation f L in an RKHS and observed that if the transformation of each layer is injective, the noise is separated through the linear transformations and cut off by f L . Since the final transformation f L in our case is in an RKHM, the same interpretation about noise is valid for deep RKHM, too. Indeed, if f L ◦· · · ◦ f 1 is not injective, then G L is not invertible, which results in the right-hand side of the inequality (2) going to infinity. The bound derived by Hashimoto et al. also goes to infinity if the network is not injective.

C Experimental Details and Additional Results

We provide details for the experiments in Section 7. All the experiments are executed with Python 3.9 and TensorFlow 2.6 on Intel(R) Core(TM) i9 CPU and NVIDIA Quadro RTX 5000 GPU with CUDA 11.7.

C.1 Comparison with vvRKHS

We set k j ( x, y ) = ˜ k ( x, y ) I for j = 1 , . . . , 3 with ˜ k ( x, y ) = e -c ∑ d i,j =1 | x i,j -y i,j | and c = 0 . 001 for positive definite kernels. For the optimizer, we used SGD. The learning rate is set as 1 e -4 both for the deep RKHM and deep vvRKHS. The initial value of c i,j is set as a i,j + ϵ i,j , where a i,j ∈ A j is the block matrix all of whose elements are 0 . 1 and ϵ i,j is randomly drawn from N (0 , 0 . 05) . The result in Figure 2 is obtained by 5 independent runs.

C.2 Observation about benign overfitting

We set the same positive definite kernel as Subsection C.1 for j = 1 , 2 . For the optimizer, we used SGD. The learning rate is set as 3 × 1 e -4 . The initial value of c i,j is the same as Subsection C.1. The result in Figure 2 is obtained by 3 independent runs.

C.3 Comparison to CNN

We set k j ( x, y ) = ˜ k ( xa j , ya j ) xy ∗ for j = 1 , 2 , where a 1 and a 2 are block matrices whose block sizes are 2 and 4 , for positive definite kernels. All the nonzero elements of a j are set as 1 . We set a 1 and a 2 to induce interactions in the block elements (see Remark 5.2). For the optimizer, we used Adam with learning rate 1 e -3 for both the deep RKHM and the CNN. The initial value of c i,j is set as ϵ i,j , where ϵ i,j is randomly drawn from N (0 , 0 . 1) .

We combined the deep RKHM and CNN with 2 dense layers. Their activation functions are sigmoid and softmax, respectively. For the CNN layers, we also used the sigmoid activation functions. The loss function is set as the categorical cross-entropy for the CNN. The result in Figure 2 is obtained by 5 independent runs.

C.3.1 Memory consumption and computational cost

Memory consumption We used a CNN with (7 × 7) and (4 × 4) -filters. On the other hand, for the deep RKHM, we learned the coefficients c i,j in Proposition 5.1 for j = 1 , 2 and i = 1 , . . . , n . That is, we learned the following coefficients:

n (= 20) block diagonal matrices each of whom has four (7 × 7) -blocks (for the first layer)

n block diagonal matrices each of whom has seven (4 × 4) -blocks (for the second layer)

Thus, the size of the parameters we have to learn for the deep RKHM is larger than the CNN. Since the memory consumption depends on the size of learning parameters, the memory consumption is larger for the deep RKHM than for the CNN.

Computational cost In each iteration, we compute f ( x i ) for i = 1 , . . . , n and the derivative of f with respect to the learning parameters. Here, f = f 1 ◦ f 2 is the network. For the deep RKHM, we compute the product of a Gram matrix (composed of n × n block diagonal matrices) and a vector (composed of n block diagonal matrices) for computing f j ( x i ) for all i = 1 , . . . , n . Thus, the computational cost for computing f ( x i ) for all i = 1 , . . . , n is O ( n 2 d ( m 1 + m 2 )) , where m 1 = 7 and m 2 = 4 are the sizes of the block diagonal matrices. For the CNN, the computational cost for computing f ( x i ) for all i = 1 , . . . , n is O ( nd 2 ( l 1 + l 2 )) , where l 1 = 7 × 7 and l 2 = 4 × 4 are the number of elements in the filters. Since we set n = 20 and d = 28 , the computational cost of the deep RKHM for computing f ( x i ) for i = 1 , . . . , n is smaller than that of the CNN. However, since the size of the learning parameters of the deep RKHM is large, the computational cost for computing the derivative of f ( x i ) is larger than that of the CNN.

Additional results To compare the deep RKHM to a CNN with the same size of learning parameters (the same memory consumption), we conducted the same experiment as the last experiment in the main text, excepting for the structure of the CNN. We constructed a CNN with (28 × 7 · 20) and (28 × 4 · 20) -filters (The size of learning parameters is the same as the deep RKHM) and replaced the CNN used in the experiment in the main text. Figure 3 shows the result. The result is similar to that in Figure 2 (c), and the deep RKHM also outperforms the CNN with the same learning parameter size. Since the size of the learning parameter is the same in this case, the computational cost for one iteration for learning the deep RKHM is the same or smaller than that for learning the CNN.

C.3.2 Additional results for benign ovefitting

We also observed benign overfitting for the deep RKHM. We compared the train and test losses for the deep RKHM with λ 1 = 1 to those with λ 1 = 0 . The results are illustrated in Figure 4. If we set λ 1 = 0 in Eq. (1), i.e., do not minimize the term in Eq. (2), then whereas the training loss becomes small as the learning process proceeds, the test loss becomes large after sufficient iterations. On the other hand, if we set λ 1 = 1 , i.e., minimize the term in Eq. (2), then the training loss becomes

0.6

Test Accuracy

0.5

0.4

RKHM,入1=0

CNN

0.3

0

100

200

300

400

Epochs

入 = 1

入1= 1

10-1

0.3

Loss

入1=0

DSS

入1=0

Los

10-2

Train

Test

10-3

0.2

10-4

0

100

200

300

400

0

100

200

300

400

Epochs

Epochs

smaller than the case of λ 1 = 0 , and the test loss does not become large even the learning process proceeds. The result implies that the term regarding the Perron-Frobenius operators in Eq. (1) causes overfitting, but it is benign overfitting.

Peer reviews:
comments: SUMMARY:
The authors combines RKHM and Perron-Frobenious Operator to deep RKHM, a deep learning framework for kernel methods. By virtue of $C^*$-algebra, they manage to get a better bound on Rademacher generalization error and provide a clear connection with CNNs. Their theoretical analysis provides a new lens for deep kernel theory.

STRENGTHS:
1. Clear writing
2. Very solid results
3. Novel tools
4. Their work can be inspiring.
5. Experiments support their theories

WEAKNESSES:
As the authors say, more efficient methods specific to deep RKHM remains to be investigated in future work. It remains a problem to scale up to at least ImageNet to be useful.

QUESTIONS:
Any theoretical understanding of why deep RKHMs might be better than nondeep ones?

LIMITATIONS:
The authors very adequately addressed the limitations. Very impressive.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a generalization of RKHS for $C^*$ algebra valued kernels, called RKHM; they build networks by composing sequentially elements taken from a collection of RKHMs, one RKHM per layer. They prove generalization bounds for those networks. These networks output matrices at each layer.\n\nSTRENGTHS:\nI believe that a strength of the paper is that the generalization bound obtained in this paper for RKHM is better than the known ones for vvRKHS. It is unclear to me if RKHMs generalize vvRKHSs: maybe considering RKHM in the commutative $C^*$ of diagonal matrices is a way to represent a vvRKHS as an RKHM. If it is the case then the result of this paper is a better generalization bound for vvRKHS.\n\nWEAKNESSES:\nI feel that the paper is sometimes difficult to read. For example the name '$\\mathcal{A}$-valued positive definite kernel' does not refer to the space $\\mathcal{X}$ that characterizes the domain of the kernel; when defining deep RKHM knowing this information could help make explicit what the domain of $k_j$ as being $A_{j-1}\\times A_{j-1}$.\nTwo remarks in this direction are on some notations: \n- line 83:  shouldn't the content of the brace in the definition of $M_{k,0}$ be $\\sum_{i=1}^{n} \\phi(x_i) c_i \\vert n\\in \\mathbb{N}, (c_i\\in A, i\\leq n), (x_i\\in \\mathcal{X}, i\\leq n) $\n\n- equation line 195, maybe the notation: $(f_j\\in \\mathcal{M}_j)_j$, could recall that the optimization is over all the RKHMs that define the networks.\n\nQUESTIONS:\nIn 'Connection with neural tangent kernel', is the aim of this paragraph to define a neural tangent kernel for deep RKHM? \n\nIn 'Comparison to CNN', how long does the training take and what is the memory consumption of RKHM and CNN?\n\nLIMITATIONS:\nIn Conclusion and Limitations, I feel that the statement 'connections with existing studies such as CNNs and neural tangent kernel' is a bit of a strong statement as the authors explain in section 6.1 that CNN and RKHM do not really relate to one another and it is unclear to me how deep the connection to neural tangent kernel is.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper proposes deep Reproducing kernel Hilbert $\mathcal{C}^*$-module (RKHM), a deep learning framework for kernel methods， which generalizes RKHS by means of $\mathcal{C}^*$-algebra. In this setting, a map as the composition of functions in RKHMs is constructed. Theoretically, the authors develop a new Rademacher generalization bound of deep RKHM using Perron-Frobenious norm. The dependency of the bound on the output dimension is milder than existing bounds. Moreover, they show a representer theorem to guarantee that the solution of a given practical minimization problem is represented only with given data. In addition, they show connections of deep RKHM with existing studies such as CNNs, benign overfitting and neural tangent kernel. Furthermore, this paper presents a series of numerical experiments to support their theory and show the practical performance of deep RKHM.

STRENGTHS:
This paper provides a new approach to analyze and understand deep kernel methods. In particular, the authors derives a generalization bound for deep RKHMs, while existing work focuses on shallow RKHMs. This bound also relaxes the dependence on the output dimension by using Perron-Frobenious norm. It is also interesting how this bound relates to benign overfitting.

Moreover, this paper provides some experiments to support their theoretical findings.
 
The paper is technically sound and the contents are very relevant to the community of NeurIPS.

WEAKNESSES:
The paper is well-organized and well-written.

QUESTIONS:
Can you explain how the generalization bounds in Theorem 4.1 and Theorem 4.5 depend on the input dimension?

LIMITATIONS:
The authors have clearly addressed the limitations of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper establishes properties on the composition of functions belonging to a RKHM, a generaliztion of RKHSs. The authors compute the Rademacher complexity of this function class and establish a representer theorem.\n\nSTRENGTHS:\n- The objects studied in the paper are well introduced. The writing is generally clear.\n- The paper adds another way formalizing the composition of linear models.\n\nWEAKNESSES:\n- The derivations of both the results of the paper are standard : the computation of Rademacher complexity using Cauchy-Schwartz + Jensen is essentially the same for all linear models. It is reproduced here with more terminology and higher dimensions but the steps are the same. The representer theorem which relies simply on the fact that the orthogonal component doesn't affect the objective is the same standard proof for kernels.\n- It is unclear what is gained from this additional abstraction. The paragraph on the connection to CNNs is unconvincing: CNNs learn the filters, whereas (composition) of linear models have fixed filters but learn the coefficients of the terms in the sum. It is difficult to argue that this much abstraction is necessary to derive this observation.\n\n    *The additional abstraction leads to doubts over existence of the objects*:  the existence of a \"well defined\" Perron-Frobenius operator is only established for a very specific case in Lemma 2.8, which appears to be artificial as it is a regular complex valued kernel multiplied by a matrix to create a multi-output kernel. \n\n- The term benign overfitting does not seem appropriate as it is discussed in section 6.2 of the paper :  The term benign overfitting refers to the phenomenon observed that *complex function classes* can fit training data noise without loosing performance on the test set. In section 6.2 the discussion is on the *complexity of the function class*. The authors say that *regularization*  in eq(2) with the operator norm of the Perron-Frobenius operator composition decreases the function class complexity and therefore leads to better generalization, which is normal and expected. This goes completely against the unusual empirical observations that led to the study of benign overfitting - which is that *complex function classes generalize without regularization*. How can benign overfitting be discussed by analyzing a **uniform** generalization bound ?\n\n- The general motivation of this work can be further developed (i.e slightly longer first paragraph) : Why should we formalize compositions of linear models ? The representer theorem is good to have but we have no convexity, so why use a composition of linear models instead of simply learning the feature embeddings as well if convexity is already lost ? Why are \"deep\" kernels worthy of study ?\n\nQUESTIONS:\n- What exactly is being done when writing \"Thus\" in the proof of Lemma 2.7 ? What does well defined mean? Are you showing existence of such a map ? It is unclear.\n- How is Corollary 4.7 derived ? Can you give more details. In particular, how do you control the operator norm of the Perron-Frobenius operator without assuming that the intermediate $\\phi_i$ are bounded ?\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}