Duality for Neural Networks through Reproducing Kernel
Banach Spaces
Len Spek

July 29, 2023

1

Mathematics of Imaging & AI @ University of Twente

2Mathematics of Imaging & AI @ University of Twente

3

Basic Neural Network
Perceptron or shallow neural network with activation function σ : R → Rm

1 X
aj σ(vTj x + bj )f(x) =mj=14

Basic Neural Network
Perceptron or shallow neural network with activation function σ : R → Rm

1 X
aj σ(vTj x + bj )f(x) =m
j=1

Universal approximation theorem: When m → ∞, we can approximate anycontinuous function.
Zf(x) := Aπ =aσ(vT x + b)dπ(w)Ωπ probability distribution of weights w = (a, v, b) ∈ Ω.

4

Basic Neural Network
Perceptron or shallow neural network with activation function σ : R → Rm

1 X
aj σ(vTj x + bj )f(x) =mj=1

Universal approximation theorem: When m → ∞, we can approximate any
continuous function.
Zf(x) := Aπ =aσ(vT x + b)dπ(w)Ωπ probability distribution of weights w = (a, v, b) ∈ Ω.
The functions f form a vector space. What norm?

4

Function Spaces for Neural Networks
Weinan E. and collaborators1 introduced the Barron space.
Zf(x) = Aπ =σ(vT x + b)dπ(w)Ωwith the normZ∥f∥ = inf

f=Aπ5|a|(1 + ∥v∥1 + |b|)dπ(w)Ω

1E, Ma, and Wu, “A priori estimates of the population risk for two-layer neural networks”.

2

Parhi and Nowak, “Banach Space Representer Theorems for Neural Networks and Ridge Splines”.

Function Spaces for Neural Networks
Weinan E. and collaborators1 introduced the Barron space.
Zf(x) = Aπ =σ(vT x + b)dπ(w)Ωwith the normZ∥f∥ = inf

f=Aπ|a|(1 + ∥v∥1 + |b|)dπ(w)ΩFor ReLu activation functions, Parhi and Nowak2 , deﬁne a normed space using theRadon transform based on ridge splines∥f∥ = cd ∥∂t2 Λd−1 R∥

5

1E, Ma, and Wu, “A priori estimates of the population risk for two-layer neural networks”.

2

Parhi and Nowak, “Banach Space Representer Theorems for Neural Networks and Ridge Splines”.

Challenges▶ Connect different function spaces to reproducing kernel framework.

6Challenges▶ Connect different function spaces to reproducing kernel framework.
▶ Lack of an inner product, so no Hilbert space structure.

6

Challenges▶ Connect different function spaces to reproducing kernel framework.
▶ Lack of an inner product, so no Hilbert space structure.
▶ Explore the dual structure of such function spaces? Does it help us understandthe relation between data and weights

6

Reproducing Kernel Hilbert Spaces (RKHS)
Deﬁnition
Hilbert space H of functions f : X → R is an RKHS if|f(x)| ≤ Cx ∥f∥Hfor all f ∈ H.

7

Reproducing Kernel Hilbert Spaces (RKHS)
Deﬁnition
Hilbert space H of functions f : X → R is an RKHS if|f(x)| ≤ Cx ∥f∥Hfor all f ∈ H.
By Riesz representation theorem, there exist a symmetric kernel K : X × X → Rf(x) = ⟨K(x, ·), f⟩

7

Reproducing Kernel Hilbert Spaces (RKHS)
Theorem
A Hilbert space H of functions on X satisﬁes the RKHS property if and only if thereexists a Hilbert space Ψ and a map ψ : X 7→ Ψ such thatH = Ψ/N (A)∥f∥H = inf ∥ν∥Ψ(1)f=Aν

where A maps features in Ψ to functions on X and is deﬁned as(Aν)(x) = ⟨ψ(x), ν⟩
for all x ∈ X and ν ∈ Ψ.

8(2)Reproducing Kernel Banach Spaces (RKBS)
Deﬁnition
A Banach space B of functions f : X → R is an RKBS when|f(x)| ≤ Cx ∥f∥B
for all f ∈ B
For example: the space of continuous functions over X with max norm

9

Reproducing Kernel Banach Spaces (RKBS)
Theorem
A Banach space B of functions on X satisﬁes the RKBS property if and only if thereexists a Banach space Ψ and a map ψ : X 7→ Ψ∗ such thatB = Ψ/N (A)∥f∥B = inf ∥ν∥Ψ(3)f=Aν

where the linear transformation A maps elements of the Banach space Ψ to functions
on X and is deﬁned as(Aν)(x) := ⟨ψ(x), ν⟩(4)for all x ∈ X and ν ∈ Ψ.aa

10

Bartolucci et al., “Understanding neural networks with reproducing kernel Banach spaces”.

A class of integral RKBS
Let µ ∈ M(Ω) a Radon measure and φ ∈ C0 (X × Ω)
Zf(x) := Aµ =φ(x, w)dµ(w)ΩThen we deﬁne the variational space F(X, Ω)3 as
F(X, Ω) := {f : X → R|∃µ ∈ M(Ω) s.t. f = Aµ}∥f∥ := inf ∥µ∥M(Ω) = inf |µ|(Ω)
f=Aµ

11

f=Aµ

3Bach, “Breaking the Curse of Dimensionality with Convex Neural Networks”.

4

Bartolucci et al., “Understanding neural networks with reproducing kernel Banach spaces”.

A class of integral RKBS
Let µ ∈ M(Ω) a Radon measure and φ ∈ C0 (X × Ω)Z
f(x) := Aµ =φ(x, w)dµ(w)ΩThen we deﬁne the variational space F(X, Ω)3 asF(X, Ω) := {f : X → R|∃µ ∈ M(Ω) s.t. f = Aµ}∥f∥ := inf ∥µ∥M(Ω) = inf |µ|(Ω)
f=Aµ

f=Aµ

Bartolucci and collaborators4 showed that this RKBS admits a Representer Theorem
and that the Radon regularisation is an instance of such an RKBS.

113Bach, “Breaking the Curse of Dimensionality with Convex Neural Networks”.

4

Bartolucci et al., “Understanding neural networks with reproducing kernel Banach spaces”.

Barron Spaces and RKBS
We showed that Barron spaces also have an integral RKBS structure, where theBarron norm is equal to the variational norm.
If σ is 1-homogeneous, take Ω = Sd+1 and w = (v, b)
φ(x, w) = σ(vT x + b)
If σ grows sublinearly, take Ω = Rd+1 and w = (v, b)
φ(x, w) =12σ(vT + b)
1 + ∥v∥1 + |b|

Reproducing Kernel Banach Spaces (RKBS)
Where is the kernel in RKBS?

13

Reproducing Kernel Banach Spaces (RKBS)
Where is the kernel in RKBS?
Challenge: No inner product =⇒ No Riesz representation theorem

13Reproducing Kernel Banach Spaces (RKBS)
Where is the kernel in RKBS?
Challenge: No inner product =⇒ No Riesz representation theorem
Adjoint RKBS can be used to deﬁne a reproducing kernel. However, we lose
symmetry of the kernel!

13Adjoint RKBS
Deﬁnition
If the dual space B is as a space of functions on a set Ω and if there exists a function
K : X × Ω → R, such that K(x, ·) ∈ B ∗ for all x ∈ X andf(x) = ⟨K(x, ·), f⟩
for all x ∈ X and f ∈ B, then we call K a reproducing kernel for B.
If B ∗ is also an RKBS on Ω and it holds that K(·, w) ∈ B for all w ∈ Ω andg(w) = ⟨g, K(·, w)⟩for all w ∈ Ω and g ∈ B ∗ , then we call B ∗ an adjoint RKBS of B.
Then K ∗ (w, x) := K(x, w) is a reproducing kernel of B ∗ .aa14

Lin, H. Z. Zhang, and J. Zhang, “On Reproducing Kernel Banach Spaces”.

Adjoint Neural Networks Spaces
We deﬁne a new space G(X, Ω) of ’Adjoint Neural Networks’.
Let ρ ∈ M(X) a Radon measure and φ ∈ C0 (X × Ω)Z∗g(w) := A ρ =φ(x, w)dρ(x)XDeﬁne the norm of g:G(X, Ω) := {g ∈ C0 (Ω)|∃ρ ∈ M(X) s.t. g = A∗ ρ}∥g∥G(X,Ω) := sup |g(w)|w∈Ω

RKBS as point evaluation is bounded.

15

Duality diagram

16

Main Theorem: Data-Weight DualityTheorem
F(X, Ω) is the dual space of G(Ω, X) with the pairingZ⟨f, g⟩ : = ⟨ρ, f⟩ = ⟨µ, g⟩ = ⟨ρ × µ, φ⟩ =φ(x, w)d(ρ × µ)(x, w)

X×Ω

where f = Aµ, g = A∗ ρ.
Furthermore, F(X, Ω) and G(Ω, X) form an adjoint pair of RKBS with reproducing
kernel φ.aa17Spek et al., Duality for Neural Networks through Reproducing Kernel Banach Spaces.

Proof Sketch - Pairing
First show that the duality pairing is well-deﬁned using Fubini
ZZ ZZ
φ(x, w)d(ρ × µ)(x, w) =φ(x, w)dµ(w)dρ(x) =f(x)dρ(x) = ⟨ρ, f⟩
X×ΩZX

φ(x, w)d(ρ × µ)(x, w) =X×ΩZ
φ(x, w)dρ(x)dµ(w) =ΩXHence independent of choice of µ and ρ18X

ΩZ Zg(w)dµ(w) = ⟨µ, g⟩ΩProof Sketch - Pairing
First show that the duality pairing is well-deﬁned using Fubini
ZZ ZZ
φ(x, w)d(ρ × µ)(x, w) =φ(x, w)dµ(w)dρ(x) =f(x)dρ(x) = ⟨ρ, f⟩
X×ΩX

Zφ(x, w)d(ρ × µ)(x, w) =X×ΩXΩZ Z

Z
φ(x, w)dρ(x)dµ(w) =ΩXHence independent of choice of µ and ρ|⟨f, g⟩| = |⟨µ, g⟩| ≤ ∥µ∥M(Ω) ∥g∥C0 (Ω)
Taking the iniﬁmum over µ s.t. f = Aµ|⟨f, g⟩| ≤ ∥f∥F (X,Ω) ∥g∥G(Ω,X)
18g(w)dµ(w) = ⟨µ, g⟩ΩProof Sketch - Duality
As F(X, Ω) is a quotient spaceF(X, Ω) := M(Ω)/N (A)Its dual is given by the annihilator of N (A), i.e. all g ∈ C0 (Ω) s.t.
⟨µ, g⟩ = 0for all µ s.t. Aµ = 0.

19

Proof Sketch - Duality
As F(X, Ω) is a quotient spaceF(X, Ω) := M(Ω)/N (A)Its dual is given by the annihilator of N (A), i.e. all g ∈ C0 (Ω) s.t.
⟨µ, g⟩ = 0
for all µ s.t. Aµ = 0. This turns out to be exactly the space G(Ω, X) as⟨µ, g⟩ = ⟨ρ, Aµ⟩ = 0for some ρ ∈ M(X) s.t. g = A∗ ρ.

19

Proof Sketch - Reproducing Kernel
To show that φ is indeed the Reproducing Kernelf(x) = ⟨f, φ(x, ·)⟩20

and g(w) = ⟨φ(x, ·), g⟩

Proof Sketch - Reproducing Kernel
To show that φ is indeed the Reproducing Kernelf(x) = ⟨f, φ(x, ·)⟩We use thatZ
φ(x, ·) =and g(w) = ⟨φ(x, ·), g⟩φ(x′ , ·)dδx (x′ ) = A∗ δx ∈ G(Ω, X)

X

And by the duality pairing⟨f, φ(x, ·)⟩ = ⟨f, δx ⟩ = f(x)20

Looking Forward
Using this dual framework, we have derived the dual problem and shown strong
duality.

21

Looking Forward
Using this dual framework, we have derived the dual problem and shown strong
duality. Estimation error - Approximation error Duality

21

Looking Forward
Using this dual framework, we have derived the dual problem and shown strong
duality. Estimation error - Approximation error Duality
Leveraging duality in optimisation: Use in experimental design or architecture search.
Example: Bregman iteration for neural networks and optimality conditions. The
duality can be used to determine a source condition which lives in the dual space.

21

Looking Forward
Using this dual framework, we have derived the dual problem and shown strong
duality. Estimation error - Approximation error Duality
Leveraging duality in optimisation: Use in experimental design or architecture search.
Example: Bregman iteration for neural networks and optimality conditions. The
duality can be used to determine a source condition which lives in the dual space.
Further goals: Expanding RKBS to deep networks and exploring the role of depth.

21Thank you for your attention

Analysis of Dynamics ofNeural Fields and Neural Networks

Len Spek, Tjeerd Jan Heeringa, FelixSchwenninger, Christoph Brune.
"Duality for neural networks throughreproducing kernel Banach spaces."
arXiv preprint arXiv:2211.05020 (2023).

Len Spek

22

Dual formulation of ERM
Primal problem: Given a target y : X → R and a data distribution ν ∈ M(X)infµ∈M(Ω)

212 ∥Aµ − y∥L2 (ν) + |µ|(Ω)

Dual problem:sup −J∗ (−ρ) − R∗ (A∗ ρ)ρ∈M(X)J∗ (ρ) =(R1 dρ
X 2 dν (x) + y(x)dρ(x)∞(0 ∥g∥C0 (Ω) ≤ 1R∗ (g) =∞ otherwise23ρ≪ν
otherwise(5)