# Mathematical Foundations of Deep Learning

Sourangshu Ghosh

► To cite this version:

Sourangshu Ghosh. Mathematical Foundations of Deep Learning. 2025. hal-04928560v5

HAL Id: hal-04928560

https://hal.science/hal-04928560v5

Preprint submitted on 29 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Mathematical Foundations of Deep Learning

Sourangshu Ghosh*

*sourangshug@iisc.ac.in, Department of Civil Engineering, Indian Institute of Science Bangalore*
Abstract

Deep learning, as a complex computational paradigm, combines function approximation, optimization, and statistical learning under a formally formulated mathematical setting. This book develops systematically the theory of deep learning in terms of functional analysis, measure theory, and variational calculus and thereby forms a mathematically complete account of deep learning frameworks.

We start with a strict problem formulation by establishing the risk functional as a measurable function space mapping, studying its properties through Fréchet differentiability and convex functional minimization. Deep neural network complexity is studied through VC-dimension theory and Rademacher complexity, defining generalization bounds and hypothesis class constraints. The universal approximation capabilities of neural networks are sharpened by convolution operators, the Stone-Weierstrass theorem, and Sobolev embeddings, with quantifiable bounds on expressivity obtained via Fourier analysis and compactness arguments by the Rellich-Kondrachov theorem. The depth-width trade-offs in expressivity are examined via capacity measures, spectral representations of activation functions, and energy-based functional approximations.

The mathematical framework of training dynamics is established through carefully examining gradient flow, stationary points, and Hessian eigenspectrum properties of loss landscapes. The Neural Tangent Kernel (NTK) regime is abstracted as an asymptotic linearization of deep learning dynamics with exact spectral decomposition techniques offering theoretical explanations of generalization. PAC-Bayesian methods, spectral regularization, and information-theoretic constraints are used to prove generalization bounds, explaining the stability of deep networks under probabilistic risk models.

The work is extended to state-of-the-art deep learning models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, generative adversarial networks (GANs), and variational autoencoders (VAEs) with strong functional analysis of representational capabilities. Optimal transport theory in deep learning is found with the application of Wasserstein distances, Sinkhorn regularization, and Kantorovich duality linking generative modeling with embeddings of probability space. Theoretical formulations of game-theoretic deep learning architectures are examined, establishing variational inequalities, equilibrium constraints, and evolutionary stability conditions in adversarial learning paradigms.

Reinforcement learning is formalized by stochastic control theory, Bellman operators, and dynamic programming principles, with precise derivations of policy optimization methods. We present a rigorous treatment of optimization methods, including stochastic gradient descent (SGD), adaptive moment estimation (Adam), and Hessian-based second-order methods, with emphasis on spectral regularization and convergence guarantees. The information-theoretic constraints in deep learning generalization are further examined via rate-distortion theory, entropy-based priors, and variational inference methods.

Metric learning, adversarial robustness, and Bayesian deep learning are mathematically formalized, with clear derivations of Mahalanobis distances, Gaussian mixture models, extreme value theory, and Bayesian nonparametric priors. Few-shot and zero-shot learning paradigms are analyzed through meta-learning frameworks, Model-Agnostic Meta-Learning (MAML), and Bayesian hierarchical inference. The mathematical framework of neural network architecture search (NAS) is constructed through evolutionary algorithms, reinforcement learning-based policy optimization, and differential operator constraints.
Theoretical contributions in kernel regression, deep Kolmogorov approaches, and neural approxima-
tions of differential operators are rigorously discussed, relating deep learning models to functional
approximation in infinite-dimensional Hilbert spaces. The mathematical concepts behind causal
inference in deep learning are expressed through structural causal models (SCMs), counterfactual
reasoning, domain adaptation, and invariant risk minimization. Deep learning models are discussed
using the framework of variational functionals, tensor calculus, and high-dimensional probability
theory.

This book offers a mathematically complete, carefully stated, and scientifically sound synthesis
of deep learning theory, linking mathematical fundamentals to the latest developments in neural
network science. Through its integration of functional analysis, information theory, stochastic pro-
cesses, and optimization into a unified theoretical structure, this research is a seminal guide for
scholars who aim to advance the mathematical foundations of deep learning.
# Contents

<table><thead><tr><th></th><th></th><th></th></tr></thead><tbody><tr><td>1</td><td>Mathematical Foundations</td><td>1</td></tr><tr><td>1.1</td><td>Problem Definition: Risk Functional as a Mapping Between Spaces</td><td>1</td></tr><tr><td>1.1.1</td><td>Measurable Function Spaces</td><td>1</td></tr><tr><td>1.1.1.1</td><td>Literature Review of Measurable Function Spaces</td><td>2</td></tr><tr><td>1.1.1.2</td><td>Analysis of Measurable Function Spaces</td><td>3</td></tr><tr><td>1.1.2</td><td>Risk as a Functional</td><td>4</td></tr><tr><td>1.1.2.1</td><td>Literature Review of Risk as a Functional</td><td>4</td></tr><tr><td>1.1.2.2</td><td>Analysis of Risk as a Functional</td><td>4</td></tr><tr><td>1.2</td><td>Approximation Spaces for Neural Networks</td><td>6</td></tr><tr><td>1.2.1</td><td>VC-dimension theory for discrete hypotheses</td><td>7</td></tr><tr><td>1.2.1.1</td><td>Literature Review of VC-dimension theory for discrete hypotheses</td><td>7</td></tr><tr><td>1.2.1.2</td><td>Analysis of VC-dimension theory for discrete hypotheses</td><td>7</td></tr><tr><td>1.2.1.3</td><td>Python Code to generate Figures 1.2 and 1.3 illustrating VC-dimension theory for discrete hypotheses</td><td>12</td></tr><tr><td>1.2.2</td><td>Rademacher complexity for continuous spaces</td><td>14</td></tr><tr><td>1.2.2.1</td><td>Literature Review of Rademacher complexity for continuous spaces</td><td>14</td></tr><tr><td>1.2.2.2</td><td>Analysis of Rademacher complexity for continuous spaces</td><td>14</td></tr><tr><td>1.2.2.3</td><td>Python Code to generate Figures 1.4 illustrating Rademacher Complexity vs. 1/√n Bound</td><td>17</td></tr><tr><td>1.2.2.4</td><td>Python Code to generate Figures 1.5 illustrating Rademacher Complexity: Linear vs Quadratic Function Classes</td><td>18</td></tr><tr><td>1.2.3</td><td>Sobolev Embeddings</td><td>19</td></tr><tr><td>1.2.3.1</td><td>Literature Review of Sobolev Embeddings</td><td>19</td></tr><tr><td>1.2.3.2</td><td>Analysis of Sobolev Embeddings</td><td>20</td></tr><tr><td>1.2.3.3</td><td>Python Code to generate Figures 1.6, 1.7, 1.8, and 1.9 illustrating Sobolev Embeddings</td><td>25</td></tr><tr><td>1.2.4</td><td>Rellich-Kondrachov Compactness Theorem</td><td>27</td></tr><tr><td>1.2.4.1</td><td>Literature Review of Rellich-Kondrachov Compactness Theorem</td><td>27</td></tr><tr><td>1.2.4.2</td><td>Analysis of Rellich-Kondrachov Compactness Theorem</td><td>27</td></tr><tr><td>1.2.4.3</td><td>Python Code to generate Figures 1.10 and 1.11 illustrating Rellich-Kondrachov Compactness Theorem</td><td>29</td></tr><tr><td>1.2.5</td><td>Fréchet-Kolmogorov compactness criterion</td><td>30</td></tr><tr><td>1.2.5.1</td><td>Literature Review of Fréchet-Kolmogorov compactness criterion</td><td>30</td></tr><tr><td>1.2.5.2</td><td>Analysis of Fréchet-Kolmogorov compactness criterion</td><td>32</td></tr><tr><td>1.2.5.3</td><td>Python Code to generate Figure 1.12</td><td>33</td></tr><tr><td>1.2.6</td><td>Sobolev-Poincaré inequality</td><td>34</td></tr><tr><td>1.2.6.1</td><td>Literature Review of Sobolev-Poincaré inequality</td><td>34</td></tr><tr><td>1.2.6.2</td><td>Analysis of Sobolev-Poincaré inequality</td><td>35</td></tr><tr><td>1.2.6.3</td><td>Python Code to generate Figures 1.13 and 1.14</td><td>38</td></tr></tbody></table>
## 2 Universal Approximation Theorem: Refined Proof 40

2.1 Literature Review of Universal Approximation Theorem 41

2.2 Approximation Using Convolution Operators 41

2.2.1 Python Code to generate Figures 2.1 and 2.2 44

2.2.2 Stone-Weierstrass Application 45

2.2.2.1 Literature Review of Stone-Weierstrass Application 45

2.2.2.2 Analysis of Stone-Weierstrass Application 46

2.2.2.3 Python Code to generate Figures 2.3, 2.4 and 2.5 49

2.3 Depth vs. Width: Capacity Analysis 51

2.3.1 Bounding the Expressive Power 51

2.3.1.1 Literature Review of Kolmogorov-Arnold Superposition Theorem 51

2.3.1.2 Analysis of Kolmogorov-Arnold Superposition Theorem 53

2.3.1.3 Python Code to generate Figure 2.6 54

2.3.2 Fourier Analysis of Expressivity 56

2.3.2.1 Literature Review of Fourier Analysis of Expressivity 56

2.3.2.2 Analysis of Fourier Analysis of Expressivity 56

2.3.2.3 Python Code to generate Figures 2.7 and 2.8 62

2.3.2.4 Python Code to generate Figures 2.9 and 2.10 63

2.3.3 Fourier transforms of various activation functions 64

2.3.3.1 Fourier Transform of the Sigmoid Function 65

2.3.3.2 Fourier Transform of the Hyperbolic Tangent Function 66

2.3.3.3 Fourier Transform of the ReLU Function 66

2.3.3.4 Fourier Transform of the Leaky ReLU Function 67

2.3.3.5 Fourier Transform of the Sinusoidal Activation Function 67

2.4 The Connection Between Different Mathematics Problems and Deep Learning 68

2.4.1 Basel Problem and Deep Learning 68

2.4.1.1 The Basel Problem, Fourier Series, and Function Approximation in Deep Learning 69

2.4.1.2 The Role of the Basel Problem in Regularization and Weight Decay 70

2.4.1.3 Spectral Bias in Deep Learning and the Basel Problem 70

## 3 Training Dynamics and NTK Linearization 72

3.1 Python Code to generate Figure 3.1 illustrating the Training Dynamics vs NTK Linearization 73

3.2 Literature Review of Training Dynamics and NTK Linearization 75

3.3 Gradient Flow and Stationary Points 75

3.3.1 Literature Review of Gradient Flow and Stationary Points 75

3.3.2 Analysis of Gradient Flow and Stationary Points 76

3.3.3 Hessian Structure 78

3.3.4 NTK Linearization 81

3.3.5 Python Code to generate Figure 3.2 illustrating the Gradient Flow and Stationary Points 83

3.3.6 Python Code to generate Figures 3.3 and 3.4 illustrating the Hessian Structure 84

3.3.7 Python Code to generate Figure 3.5 illustrating the Final Fit & Training Dynamics (NN vs NTK) 86

3.4 NTK Regime 88

3.4.1 Literature Review of NTK Regime 88

3.4.2 Analysis of NTK Regime 89

3.4.3 Python Code to generate Figure 3.6 illustrating the Final Predictions in NTK Regime & Training Dynamics (NN vs NTK) 90
4 Generalization Bounds: PAC-Bayes and Spectral Analysis 93

4.1 PAC-Bayes Formalism 93

4.1.1 Literature Review of PAC-Bayes Formalism 93

4.1.2 Analysis of PAC-Bayes Formalism 94

4.1.2.1 Python Code to generate Figures 4.1, 4.2, and 4.3 illustrating PAC-Bayes Bound vs. Sample Size 97

4.1.3 KL divergence 98

4.1.3.1 Python Code to generate Figures 4.4, 4.5 illustrating KL Divergence in Bernoulli Random Variables and Figures 4.6, 4.7 illustrating KL Divergence in Gaussian Random Variables 104

4.1.4 Rényi divergence 105

4.1.4.1 Python Code to generate Figures 4.8 and 4.9 illustrating Rényi divergence 108

4.1.5 Wasserstein distance 109

4.1.5.1 Python Code to generate Figures 4.10, 4.11, and 4.12 illustrating Wasserstein distance 113

4.2 Spectral Regularization 114

4.2.1 Literature Review of Spectral Regularization 114

4.2.2 Analysis of Spectral Regularization 115

4.2.3 Python Code to generate Figures 4.13 and 4.14 illustrating Spectral Regularization 118

4.2.4 Python Code to generate Figure 4.15 illustrating Singular Value Spectrum of First Layer (MNIST) 118

5 Game-theoretic formulations of Deep Neural Networks 121

5.1 Literature Review of Game-theoretic formulations of Deep Neural Networks 122

5.2 Analysis of Game-theoretic formulations of Deep Neural Networks 124

5.2.1 Python Code to generate Figures 5.1, 5.2, and 5.3 illustrating Game-theoretic formulations of Deep Neural Networks 127

5.2.2 Min-Max (Saddle) dynamics: $L(x, y) = x * y$ 128

5.2.3 Potential/Cooperative game (gradient descent on shared Phi) 130

5.2.4 Replicator dynamics (Rock-Paper-Scissors) — cyclic behavior 132

5.2.5 Python Code to generate Figures 5.4, 5.5 and 5.6 illustrating Min-Max (Saddle) dynamics, Potential/Cooperative game (gradient descent on shared Phi), and Replicator dynamics (Rock-Paper-Scissors) — cyclic behavior 134

5.3 Game-theoretic formulations of deep neural networks (DNNs) through Evolutionary Game Dynamics 138

5.3.1 Python Code to generate Figures 5.7, 5.8 and 5.9 illustrating Game-theoretic formulations of deep neural networks (DNNs) through Evolutionary Game Dynamics 141

5.4 Analysis of Deep neural networks (DNNs) through Variational Inequalities 143

5.4.1 Monotone/potential operator (gradient of a convex potential) 144

5.4.2 Saddle/non-monotone bilinear operator (models min-max behavior) 146

5.4.3 Skew (rotation) 149

5.4.4 Skew + Dissipation 151

5.4.5 Python Code to generate Figures 5.10, 5.11 illustrating Monotone/potential operator and Figure 5.12 illustrating Saddle Bilinear F (min-max) and Figure 5.13 illustrating Skew (rotation) F: R z and Figure 5.14 illustrating Skew + Dissipation F: R + alpha I 153

5.5 Optimal control in reinforcement learning (RL)-based deep neural network (DNN) training 157

5.5.1 Backpropagation as Pontryagin's Maximum Principle (PMP) 159
<table><tr><td>5.5.2</td><td>The Method of Successive Approximation (MSA)</td><td>161</td></tr><tr><td>5.5.3</td><td>Differentiable Optimization Layers</td><td>163</td></tr><tr><td>5.5.4</td><td>Neural Ordinary Differential Equations (Neural ODEs)</td><td>165</td></tr><tr><td>5.5.5</td><td>Stochastic Optimal Control for RL and Robust Training</td><td>167</td></tr><tr><td>5.5.6</td><td>Stable and Efficient Network Design</td><td>169</td></tr><tr><td>5.5.7</td><td>Optimal Control of the Training Process Itself</td><td>172</td></tr><tr><td>5.5.8</td><td>Connections to Reinforcement Learning</td><td>174</td></tr><tr><td>5.6</td><td>Differential Game Theory and Training Dynamics</td><td>177</td></tr></table>

6 Optimal transport theory in Deep Neural Networks 179

<table><tr><td>6.1</td><td>Literature Review of Optimal transport theory in Deep Neural Networks</td><td>180</td></tr><tr><td>6.2</td><td>Analysis of Optimal transport theory in Deep Neural Networks</td><td>182</td></tr><tr><td>6.3</td><td>Jensen-Shannon divergence</td><td>184</td></tr><tr><td>6.4</td><td>Matching Latent and Data Distributions in Probabilistic Autoencoders</td><td>185</td></tr><tr><td>6.4.1</td><td>Probabilistic Autoencoders</td><td>187</td></tr><tr><td>6.4.2</td><td>2-Wasserstein distance</td><td>188</td></tr><tr><td>6.5</td><td>Optimal Transport (OT)-based priors in Bayesian Deep Learning</td><td>190</td></tr><tr><td>6.6</td><td>Application of Optimal Transport in learning energy-based models</td><td>191</td></tr><tr><td>6.7</td><td>Sinkhorn-Knopp Algorithm</td><td>193</td></tr><tr><td>6.7.1</td><td>Sinkhorn Distance</td><td>194</td></tr><tr><td>6.7.2</td><td>Wasserstein GANs</td><td>196</td></tr><tr><td>6.8</td><td>Kantorovich Duality</td><td>197</td></tr><tr><td>6.9</td><td>Entropy Regularization</td><td>198</td></tr><tr><td>6.9.1</td><td>Classical Monge-Kantorovich formulation</td><td>200</td></tr><tr><td>6.9.2</td><td>Fenchel-Rockafellar duality theorem</td><td>202</td></tr><tr><td>6.10</td><td>Optimal transport theory Based Training Dynamics and Sinkhorn Divergences</td><td>203</td></tr><tr><td>6.11</td><td>Optimal transport theory in Neural Architecture and Gradient Flows</td><td>204</td></tr><tr><td>6.12</td><td>Geometric Regularization and Barycenters</td><td>205</td></tr></table>

7 Categorical Foundations of Deep Learning 207

<table><tr><td>7.1</td><td>Introduction</td><td>209</td></tr><tr><td>7.1.1</td><td>Motivation: Why Category Theory for Deep Learning?</td><td>209</td></tr><tr><td>7.1.2</td><td>Limits of Traditional Mathematical Formalism in Modern Machine Learning</td><td>209</td></tr><tr><td>7.1.3</td><td>Overview of the Chapter and Learning Goals</td><td>210</td></tr><tr><td>7.2</td><td>Category Theory Primer for Machine Learners</td><td>211</td></tr><tr><td>7.2.1</td><td>Basic Definitions: Categories, Functors, Natural Transformations, Adjunctions, Monoidal categories</td><td>212</td></tr><tr><td>7.2.1.1</td><td>Categories</td><td>212</td></tr><tr><td>7.2.1.2</td><td>Functors</td><td>213</td></tr><tr><td>7.2.1.3</td><td>Natural Transformations</td><td>215</td></tr><tr><td>7.2.1.4</td><td>Adjunctions</td><td>216</td></tr><tr><td>7.2.1.5</td><td>Monoidal categories</td><td>217</td></tr><tr><td>7.2.2</td><td>Use of String diagrams and Abstraction and Generalization in Machine Learning</td><td>218</td></tr><tr><td>7.2.2.1</td><td>String diagrams as Graphical Calculus for Neural Networks</td><td>218</td></tr><tr><td>7.2.2.2</td><td>Abstraction and Generalization in Machine Learning</td><td>220</td></tr><tr><td>7.2.3</td><td>Examples from Familiar Settings</td><td>221</td></tr><tr><td>7.2.3.1</td><td>Category of Sets (Set)</td><td>221</td></tr><tr><td>7.2.3.2</td><td>Category of Vector Spaces (Vect<sub>K</sub>)</td><td>222</td></tr><tr><td>7.2.3.3</td><td>Category of Neural Networks (Informal Example)</td><td>224</td></tr><tr><td>7.2.4</td><td>Diagrams as Proofs and Reasoning Tools</td><td>225</td></tr><tr><td>7.2.4.1</td><td>Commutative Diagrams</td><td>226</td></tr><tr><td>7.2.4.2</td><td>Universal Properties via Diagrams</td><td>227</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>7.2.4.3 Applications in ML</td>
      <td>229</td>
    </tr>
    <tr>
      <td>7.2.4.3.1 Backpropagation as a Functorial Diagram</td>
      <td>229</td>
    </tr>
    <tr>
      <td>7.2.4.3.2 Attention Mechanisms</td>
      <td>229</td>
    </tr>
    <tr>
      <td>7.2.5 Summary</td>
      <td>230</td>
    </tr>
    <tr>
      <td>7.3 Neural Networks as Composable Morphisms</td>
      <td>231</td>
    </tr>
    <tr>
      <td>7.3.1 Layers as Morphisms; Networks as Compositions</td>
      <td>231</td>
    </tr>
    <tr>
      <td>7.3.1.1 Neural Networks as Composable Functions</td>
      <td>232</td>
    </tr>
    <tr>
      <td>7.3.1.2 Category-Theoretic Interpretation</td>
      <td>233</td>
    </tr>
    <tr>
      <td>7.3.2 The Category of Parameterized Functions</td>
      <td>234</td>
    </tr>
    <tr>
      <td>7.3.2.1 Definition of the Category Para</td>
      <td>235</td>
    </tr>
    <tr>
      <td>7.3.2.2 Functoriality of Neural Networks</td>
      <td>236</td>
    </tr>
    <tr>
      <td>7.3.2.3 Monoidal Structure (for Parallel Composition)</td>
      <td>237</td>
    </tr>
    <tr>
      <td>7.3.3 Role of Associativity and Identity in Sequential Models</td>
      <td>238</td>
    </tr>
    <tr>
      <td>7.3.3.1 Associativity of Composition</td>
      <td>239</td>
    </tr>
    <tr>
      <td>7.3.3.2 Identity Morphisms and Skip Connections</td>
      <td>240</td>
    </tr>
    <tr>
      <td>7.3.3.3 Universality of Sequential Models</td>
      <td>241</td>
    </tr>
    <tr>
      <td>7.3.4 Conclusion</td>
      <td>241</td>
    </tr>
    <tr>
      <td><b>8 Open Set Learning</b></td>
      <td><b>243</b></td>
    </tr>
    <tr>
      <td>8.1 Literature Review of Deep Neural Network-based Open Set Learning</td>
      <td>246</td>
    </tr>
    <tr>
      <td>8.2 Literature Review of Traditional Machine Learning Open Set Learning</td>
      <td>261</td>
    </tr>
    <tr>
      <td>8.2.1 Foundational Theoretical Frameworks of Open Set Recognition</td>
      <td>261</td>
    </tr>
    <tr>
      <td>8.2.2 Sparse and Hyperplane-Based Models for OSR</td>
      <td>261</td>
    </tr>
    <tr>
      <td>8.2.3 Support Vector and Nearest Neighbor Approaches</td>
      <td>262</td>
    </tr>
    <tr>
      <td>8.2.4 Domain Transfer and Zero-shot Learning Integration</td>
      <td>262</td>
    </tr>
    <tr>
      <td>8.2.5 Application-Centric Advances: Face Recognition and Text Classification</td>
      <td>262</td>
    </tr>
    <tr>
      <td>8.2.6 Ensemble and Fusion-Based Enhancements</td>
      <td>263</td>
    </tr>
    <tr>
      <td>8.3 Mahalanobis Distance</td>
      <td>263</td>
    </tr>
    <tr>
      <td>8.4 Literature Review of Bayesian Formulation in Open Set Learning</td>
      <td>265</td>
    </tr>
    <tr>
      <td>8.4.1 Literature Review of Bayesian Neural Networks (BNNs) for OSL</td>
      <td>265</td>
    </tr>
    <tr>
      <td>8.4.2 Literature Review of Dirichlet-Based Uncertainty for OSL</td>
      <td>265</td>
    </tr>
    <tr>
      <td>8.4.3 Literature Review of Gaussian Processes (GPs) for OSL</td>
      <td>266</td>
    </tr>
    <tr>
      <td>8.4.4 Literature Review of Variational Autoencoders (VAEs) and Bayesian Generative Models</td>
      <td>266</td>
    </tr>
    <tr>
      <td>8.4.5 Critical Synthesis</td>
      <td>267</td>
    </tr>
    <tr>
      <td>8.5 Analysis of Bayesian Formulation in Open Set Learning</td>
      <td>267</td>
    </tr>
    <tr>
      <td>8.6 Gaussian Mixture Model (GMM)</td>
      <td>269</td>
    </tr>
    <tr>
      <td>8.7 Dirichlet Process Gaussian Mixture Model (DP-GMM)</td>
      <td>270</td>
    </tr>
    <tr>
      <td>8.8 Conjugate Normal-Inverse-Wishart (NIW) distribution</td>
      <td>272</td>
    </tr>
    <tr>
      <td>8.9 Extreme Value Theory (EVT) Models</td>
      <td>273</td>
    </tr>
    <tr>
      <td>8.10 Bayesian Neural Networks (BNNs)</td>
      <td>275</td>
    </tr>
    <tr>
      <td>8.11 Support Vector Models</td>
      <td>277</td>
    </tr>
    <tr>
      <td>8.12 Support Vector Data Description</td>
      <td>279</td>
    </tr>
    <tr>
      <td><b>9 Zero-Shot Learning</b></td>
      <td><b>281</b></td>
    </tr>
    <tr>
      <td>9.1 Literature Review of Zero-Shot Learning</td>
      <td>282</td>
    </tr>
    <tr>
      <td>9.2 Analysis of Zero-Shot Learning</td>
      <td>283</td>
    </tr>
    <tr>
      <td>9.3 Energy-based models for Zero-Shot Learning</td>
      <td>285</td>
    </tr>
    <tr>
      <td>9.3.1 Various Loss function used in Energy-based models for Zero-Shot Learning</td>
      <td>287</td>
    </tr>
    <tr>
      <td>9.3.2 Generative constraints in Energy-based models for Zero-Shot Learning</td>
      <td>288</td>
    </tr>
    <tr>
      <td>9.3.3 Use of Inference in Energy-based models for Zero-Shot Learning</td>
      <td>289</td>
    </tr>
    <tr>
      <td>9.4 Meta-learning approaches for Zero-Shot Learning</td>
      <td>290</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>9.4.1</td>
      <td>Model-Agnostic Meta-Learning (MAML)</td>
      <td>291</td>
    </tr>
    <tr>
      <td>9.4.2</td>
      <td>Meta-embedding strategy</td>
      <td>293</td>
    </tr>
    <tr>
      <td>9.4.3</td>
      <td>Metric-based meta-learning</td>
      <td>294</td>
    </tr>
    <tr>
      <td>9.4.4</td>
      <td>Graph-based meta-learning</td>
      <td>295</td>
    </tr>
    <tr>
      <td>9.4.5</td>
      <td>Bayesian meta-learning</td>
      <td>297</td>
    </tr>
    <tr>
      <td><b>10</b></td>
      <td><b>Neural Network Basics</b></td>
      <td><b>299</b></td>
    </tr>
    <tr>
      <td>10.1</td>
      <td>Literature Review of Neural Network Basics</td>
      <td>300</td>
    </tr>
    <tr>
      <td>10.2</td>
      <td>Perceptrons and Artificial Neurons</td>
      <td>300</td>
    </tr>
    <tr>
      <td>10.3</td>
      <td>Feedforward Neural Networks</td>
      <td>303</td>
    </tr>
    <tr>
      <td>10.4</td>
      <td>Activation Functions</td>
      <td>304</td>
    </tr>
    <tr>
      <td>10.5</td>
      <td>Loss Functions</td>
      <td>307</td>
    </tr>
    <tr>
      <td>10.5.1</td>
      <td>Loss Functions for Regression Tasks</td>
      <td>307</td>
    </tr>
    <tr>
      <td>10.5.1.1</td>
      <td>Mean Squared Error (MSE/L2 Loss)</td>
      <td>307</td>
    </tr>
    <tr>
      <td>10.5.1.2</td>
      <td>Mean Absolute Error (MAE/L1 Loss)</td>
      <td>309</td>
    </tr>
    <tr>
      <td>10.5.1.3</td>
      <td>Huber Loss (Smooth Mean Absolute Error)</td>
      <td>311</td>
    </tr>
    <tr>
      <td>10.5.1.4</td>
      <td>Python Code to generate Figure 10.1 illustrating Mean Squared Error (MSE/L2) Loss Function</td>
      <td>313</td>
    </tr>
    <tr>
      <td>10.5.1.5</td>
      <td>Python Code to generate Figure 10.2 illustrating Mean Absolute Error (MAE/L1) Loss Function</td>
      <td>314</td>
    </tr>
    <tr>
      <td>10.5.1.6</td>
      <td>Python Code to generate Figure 10.3 illustrating Huber Loss Function</td>
      <td>314</td>
    </tr>
    <tr>
      <td>10.5.1.7</td>
      <td>Python Code to generate Figure 10.4 comparing the Loss Functions: Huber, MSE, MAE</td>
      <td>315</td>
    </tr>
    <tr>
      <td>10.5.2</td>
      <td>Loss Functions for Classification Tasks</td>
      <td>315</td>
    </tr>
    <tr>
      <td>10.5.2.1</td>
      <td>Binary Cross-Entropy (Log Loss)</td>
      <td>316</td>
    </tr>
    <tr>
      <td>10.5.2.2</td>
      <td>Categorical Cross-Entropy</td>
      <td>319</td>
    </tr>
    <tr>
      <td>10.5.2.3</td>
      <td>Sparse Categorical Cross-Entropy</td>
      <td>321</td>
    </tr>
    <tr>
      <td>10.5.2.4</td>
      <td>Kullback-Leibler Divergence (KL Divergence)</td>
      <td>324</td>
    </tr>
    <tr>
      <td>10.5.2.5</td>
      <td>Python Code to generate Figure 10.5 illustrating Binary Cross-Entropy (BCE) Loss Function</td>
      <td>326</td>
    </tr>
    <tr>
      <td>10.5.2.6</td>
      <td>Python Code to generate Figure 10.6 illustrating Binary Cross-Entropy Loss Surface</td>
      <td>327</td>
    </tr>
    <tr>
      <td>10.5.2.7</td>
      <td>Python Code to generate Figure 10.7 illustrating Categorical Cross-Entropy Loss</td>
      <td>327</td>
    </tr>
    <tr>
      <td>10.5.2.8</td>
      <td>Python Code to generate Figure 10.8 illustrating Categorical Cross-Entropy Loss</td>
      <td>328</td>
    </tr>
    <tr>
      <td>10.5.2.9</td>
      <td>Python Code to generate Figure 10.9 illustrating Sparse Categorical Cross-Entropy Loss</td>
      <td>329</td>
    </tr>
    <tr>
      <td>10.5.2.10</td>
      <td>Python Code to generate Figure 10.10 illustrating Surface Plot of Sparse Categorical Cross-Entropy Loss</td>
      <td>330</td>
    </tr>
    <tr>
      <td>10.5.2.11</td>
      <td>Python Code to generate Figure 10.11 illustrating KL Divergence</td>
      <td>331</td>
    </tr>
    <tr>
      <td>10.5.2.12</td>
      <td>Python Code to generate Figure 10.12 illustrating 2D Surface Plot of KL Divergence</td>
      <td>332</td>
    </tr>
    <tr>
      <td>10.5.3</td>
      <td>Advanced and Specialized Loss Functions</td>
      <td>332</td>
    </tr>
    <tr>
      <td>10.5.3.1</td>
      <td>Loss Functions for Generative Adversarial Networks (GANs)</td>
      <td>332</td>
    </tr>
    <tr>
      <td>10.5.3.1.1</td>
      <td>Wasserstein Loss (Earth Mover's Distance)</td>
      <td>332</td>
    </tr>
    <tr>
      <td>10.5.3.1.2</td>
      <td>Least Squares GAN (LSGAN) Loss</td>
      <td>335</td>
    </tr>
    <tr>
      <td>10.5.3.1.3</td>
      <td>Adversarial Loss (Standard GAN Loss)</td>
      <td>337</td>
    </tr>
    <tr>
      <td>10.5.3.2</td>
      <td>Loss Functions for Siamese and Metric Learning</td>
      <td>339</td>
    </tr>
    <tr>
      <td>10.5.3.2.1</td>
      <td>Contrastive Loss</td>
      <td>339</td>
    </tr>
    <tr>
      <td>10.5.3.2.2</td>
      <td>Triplet Loss</td>
      <td>341</td>
    </tr>
    <tr>
      <td>10.5.3.2.3</td>
      <td>Center Loss</td>
      <td>344</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>10.5.3.3</td>
      <td>Loss Functions for Style Transfer and Super-Resolution</td>
      <td>347</td>
    </tr>
    <tr>
      <td>10.5.3.3.1</td>
      <td>Perceptual Loss</td>
      <td>347</td>
    </tr>
    <tr>
      <td>10.5.3.3.2</td>
      <td>Style Loss</td>
      <td>348</td>
    </tr>
    <tr>
      <td>10.5.3.3.3</td>
      <td>Total Variation (TV) Loss</td>
      <td>350</td>
    </tr>
    <tr>
      <td>10.5.3.4</td>
      <td>Loss Functions for Uncertainty Estimation and Bayesian Deep Learning</td>
      <td>353</td>
    </tr>
    <tr>
      <td>10.5.3.4.1</td>
      <td>Evidence Lower Bound (ELBO) Loss</td>
      <td>353</td>
    </tr>
    <tr>
      <td>10.5.3.4.2</td>
      <td>Negative Log-Likelihood (NLL) Loss</td>
      <td>354</td>
    </tr>
    <tr>
      <td>10.5.3.5</td>
      <td>Loss Functions for Domain Adaptation</td>
      <td>356</td>
    </tr>
    <tr>
      <td>10.5.3.5.1</td>
      <td>Domain Adversarial Loss</td>
      <td>356</td>
    </tr>
    <tr>
      <td>10.5.3.5.2</td>
      <td>Maximum Mean Discrepancy (MMD) Loss</td>
      <td>357</td>
    </tr>
    <tr>
      <td>10.5.3.6</td>
      <td>Loss Functions for Object Detection and Segmentation</td>
      <td>359</td>
    </tr>
    <tr>
      <td>10.5.3.6.1</td>
      <td>IoU Loss (Intersection over Union) and its variants (GIoU, DIoU, CIoU)</td>
      <td>359</td>
    </tr>
    <tr>
      <td>10.5.3.6.2</td>
      <td>Dice Loss</td>
      <td>362</td>
    </tr>
    <tr>
      <td>10.5.3.6.3</td>
      <td>Focal Loss</td>
      <td>364</td>
    </tr>
    <tr>
      <td>10.5.3.7</td>
      <td>Loss Functions for Knowledge Distillation</td>
      <td>366</td>
    </tr>
    <tr>
      <td>10.5.3.8</td>
      <td>Loss Functions for Reinforcement Learning</td>
      <td>366</td>
    </tr>
    <tr>
      <td>10.5.3.8.1</td>
      <td>Hinge Loss</td>
      <td>366</td>
    </tr>
    <tr>
      <td>10.5.3.8.2</td>
      <td>Value Loss</td>
      <td>368</td>
    </tr>
    <tr>
      <td>10.5.3.8.3</td>
      <td>Policy Loss</td>
      <td>370</td>
    </tr>
    <tr>
      <td>10.5.3.8.4</td>
      <td>Entropy Regularization</td>
      <td>373</td>
    </tr>
    <tr>
      <td>10.5.3.9</td>
      <td>Loss Functions for Sparse and Structured Outputs</td>
      <td>376</td>
    </tr>
    <tr>
      <td>10.5.3.9.1</td>
      <td>Kullback-Leibler (KL) Divergence Loss (as a Sparsity Constraint)</td>
      <td>376</td>
    </tr>
    <tr>
      <td>10.5.3.9.2</td>
      <td>Structured Support Vector Machine (SSVM) Loss</td>
      <td>378</td>
    </tr>
    <tr>
      <td>10.5.3.9.3</td>
      <td>Conditional Random Field (CRF) Loss</td>
      <td>380</td>
    </tr>
    <tr>
      <td>10.5.3.9.4</td>
      <td>Connectionist Temporal Classification (CTC) Loss</td>
      <td>384</td>
    </tr>
    <tr>
      <td>10.5.3.9.5</td>
      <td>Maximum Margin Markov Networks Loss</td>
      <td>388</td>
    </tr>
    <tr>
      <td>10.5.3.9.6</td>
      <td>Listwise Ranking Losses</td>
      <td>392</td>
    </tr>
    <tr>
      <td>10.5.3.9.7</td>
      <td>Pairwise Ranking Losses</td>
      <td>393</td>
    </tr>
    <tr>
      <td>10.5.3.9.8</td>
      <td>Contrastive Divergence (CD) Loss</td>
      <td>397</td>
    </tr>
    <tr>
      <td>10.5.3.9.9</td>
      <td>Persistent Contrastive Divergence (PCD) Loss</td>
      <td>399</td>
    </tr>
    <tr>
      <td>10.5.3.9.10</td>
      <td>Hausdorff Distance Loss</td>
      <td>402</td>
    </tr>
    <tr>
      <td>10.5.3.10</td>
      <td>Python Code to generate Figure 10.13 illustrating 1D Wasserstein Distance of two distributions having parameters: Normal(μ = -1, σ = 1) and Normal(μ = 1.5, σ = 1)</td>
      <td>404</td>
    </tr>
    <tr>
      <td>10.5.3.11</td>
      <td>Python Code to generate Figure 10.14 illustrating Multidimensional Wasserstein Distance of two distributions having parameters, Distribution A: N(μ = [0, 0], Σ = I), Distribution B: N(μ = [3, 3], Σ = I)</td>
      <td>406</td>
    </tr>
    <tr>
      <td>10.5.3.12</td>
      <td>Python Code to generate Figure 10.15 comparing Equal CDF, WGAN Critic, GAN Discriminator</td>
      <td>407</td>
    </tr>
    <tr>
      <td>10.5.3.13</td>
      <td>Python Code to generate Figure 10.16 illustrating Least Squares GAN (LSGAN) Loss Functions</td>
      <td>408</td>
    </tr>
    <tr>
      <td>10.5.3.14</td>
      <td>Python Code to generate Figure 10.17 illustrating Standard GAN (Adversarial) Loss Functions</td>
      <td>409</td>
    </tr>
    <tr>
      <td>10.5.3.15</td>
      <td>Python Code to generate Figure 10.18 illustrating Contrastive Loss Functions</td>
      <td>409</td>
    </tr>
    <tr>
      <td>10.5.3.16</td>
      <td>Python Code to generate Figure 10.19 illustrating Contrastive Loss Landscapes</td>
      <td>410</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>10.5.3.17 Python Code to generate Figure 10.20 illustrating 3D Contrastive Loss Landscapes</td>
      <td>411</td>
    </tr>
    <tr>
      <td>10.5.3.18 Python Code to generate Figure 10.21 illustrating Triplet Loss Function</td>
      <td>412</td>
    </tr>
    <tr>
      <td>10.5.3.19 Python Code to generate Figure 10.22 illustrating Triplet Loss Landscape</td>
      <td>413</td>
    </tr>
    <tr>
      <td>10.5.3.20 Python Code to generate Figure 10.23 illustrating Triplet Loss Contour Plot</td>
      <td>413</td>
    </tr>
    <tr>
      <td>10.5.3.21 Python Code to generate Figure 10.24 illustrating Center Loss Function</td>
      <td>414</td>
    </tr>
    <tr>
      <td>10.5.3.22 Python Code to generate Figure 10.25 illustrating Center Loss Visualization: Features vs Class Centers</td>
      <td>414</td>
    </tr>
    <tr>
      <td>10.5.3.23 Python Code to generate Figure 10.26 illustrating 3D Center Loss Surface</td>
      <td>415</td>
    </tr>
    <tr>
      <td>10.5.3.24 Python Code to generate Figure 10.27 illustrating Perceptual Loss Function</td>
      <td>416</td>
    </tr>
    <tr>
      <td>10.5.3.25 Python Code to generate Figure 10.28 illustrating Pixel-wise MSE vs Perceptual Loss (Nonlinear Features)</td>
      <td>417</td>
    </tr>
    <tr>
      <td>10.5.3.26 Python Code to generate Figure 10.29 illustrating Style Loss Function in Neural Style Transfer</td>
      <td>417</td>
    </tr>
    <tr>
      <td>10.5.3.27 Python Code to generate Figure 10.30 illustrating Style Loss vs Scaling of Generated Features</td>
      <td>418</td>
    </tr>
    <tr>
      <td>10.5.3.28 Python Code to generate Figure 10.31 illustrating Total Variation (TV) Loss</td>
      <td>419</td>
    </tr>
    <tr>
      <td>10.5.3.29 Python Code to generate Figure 10.32 illustrating Total Variation Loss vs Noise Level</td>
      <td>420</td>
    </tr>
    <tr>
      <td>10.5.3.30 Python Code to generate Figure 10.33 illustrating Evidence Lower Bound (ELBO) Loss during Training</td>
      <td>421</td>
    </tr>
    <tr>
      <td>10.5.3.31 Python Code to generate Figure 10.34 illustrating Negative Log-Likelihood (NLL) Loss</td>
      <td>421</td>
    </tr>
    <tr>
      <td>10.5.3.32 Python Code to generate Figure 10.35 illustrating Domain Adversarial Loss during Training</td>
      <td>422</td>
    </tr>
    <tr>
      <td>10.5.3.33 Python Code to generate Figure 10.36 illustrating Maximum Mean Discrepancy (MMD) Loss during Training</td>
      <td>422</td>
    </tr>
    <tr>
      <td>10.5.3.34 Python Code to generate Figure 10.37 illustrating IoU Loss and Variants (GIoU Loss, DIoU Loss, CIoU Loss)</td>
      <td>423</td>
    </tr>
    <tr>
      <td>10.5.3.35 Python Code to generate Figure 10.38 illustrating Dice Loss during Training</td>
      <td>425</td>
    </tr>
    <tr>
      <td>10.5.3.36 Python Code to generate Figure 10.39 illustrating Focal Loss vs Predicted Probability</td>
      <td>426</td>
    </tr>
    <tr>
      <td>10.5.3.37 Python Code to generate Figure 10.40 illustrating Focal Loss vs Standard Binary Cross-Entropy</td>
      <td>426</td>
    </tr>
    <tr>
      <td>10.5.3.38 Python Code to generate Figure 10.41 illustrating Hinge Loss vs Prediction Score</td>
      <td>427</td>
    </tr>
    <tr>
      <td>10.5.3.39 Python Code to generate Figure 10.42 illustrating Comparison of the Standard Hinge Loss and the Squared Hinge Loss</td>
      <td>428</td>
    </tr>
    <tr>
      <td>10.5.3.40 Python Code to generate Figure 10.43 illustrating Value Loss (MSE) for Different Targets</td>
      <td>429</td>
    </tr>
    <tr>
      <td>10.5.3.41 Python Code to generate Figure 10.44 illustrating Value Loss (MSE) vs Huber Loss</td>
      <td>429</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>10.5.3.42 Python Code to generate Figure 10.45 illustrating Policy Loss vs Action Probability</td>
      <td>430</td>
    </tr>
    <tr>
      <td>10.5.3.43 Python Code to generate Figure 10.46 illustrating PPO Clipped Surrogate Objective</td>
      <td>431</td>
    </tr>
    <tr>
      <td>10.5.3.44 Python Code to generate Figure 10.47 illustrating PPO Loss Comparison</td>
      <td>431</td>
    </tr>
    <tr>
      <td>10.5.3.45 Python Code to generate Figure 10.48 illustrating PPO Clipped Surrogate Loss with Entropy Regularization</td>
      <td>432</td>
    </tr>
    <tr>
      <td>10.5.3.46 Python Code to generate Figure 10.49 comparing PPO Unclipped, PPO Clipped, PPO + Entropy</td>
      <td>433</td>
    </tr>
    <tr>
      <td>10.5.3.47 Python Code to generate Figure 10.50 illustrating Soft Actor-Critic (SAC) Policy Loss with Entropy Regularization</td>
      <td>434</td>
    </tr>
    <tr>
      <td>10.5.3.48 Python Code to generate Figure 10.51 illustrating KL Divergence Loss as Sparsity Constraint</td>
      <td>435</td>
    </tr>
    <tr>
      <td>10.5.3.49 Python Code to generate Figure 10.52 illustrating Total KL Sparsity Loss for 10 Hidden Units (ρ = 0.05)</td>
      <td>435</td>
    </tr>
    <tr>
      <td>10.5.3.50 Python Code to generate Figure 10.53 illustrating Total KL Sparsity Loss for 10 Hidden Units with Random Activations (ρ = 0.05)</td>
      <td>436</td>
    </tr>
    <tr>
      <td>10.5.3.51 Python Code to generate Figure 10.54 illustrating Structured SVM (SSVM) Hinge Loss</td>
      <td>437</td>
    </tr>
    <tr>
      <td>10.5.3.52 Python Code to generate Figure 10.55 illustrating Structured SVM Loss Surface (Δ = 1.0)</td>
      <td>438</td>
    </tr>
    <tr>
      <td>10.5.3.53 Python Code to generate Figure 10.56 illustrating Structured SVM Loss Contour (Δ = 1.0)</td>
      <td>438</td>
    </tr>
    <tr>
      <td>10.5.3.54 Python Code to generate Figure 10.57 illustrating Simplified CRF Loss vs Correct Label Score</td>
      <td>439</td>
    </tr>
    <tr>
      <td>10.5.3.55 Python Code to generate Figure 10.58 illustrating CRF Loss Surface (Single Timestep, 2 Labels)</td>
      <td>440</td>
    </tr>
    <tr>
      <td>10.5.3.56 Python Code to generate Figure 10.59 illustrating CRF Loss Contour (Single Timestep, 2 Labels)</td>
      <td>440</td>
    </tr>
    <tr>
      <td>10.5.3.57 Python Code to generate Figure 10.60 illustrating CTC Loss vs Probability of Correct Sequence</td>
      <td>441</td>
    </tr>
    <tr>
      <td>10.5.3.58 Python Code to generate Figure 10.61 illustrating Approximate CTC Loss vs Per-Timestep Probability for Different Sequence Lengths</td>
      <td>442</td>
    </tr>
    <tr>
      <td>10.5.3.59 Python Code to generate Figure 10.62 illustrating CTC Loss Surface vs Sequence Length and Per-Timestep Probability</td>
      <td>442</td>
    </tr>
    <tr>
      <td>10.5.3.60 Python Code to generate Figure 10.63 illustrating CTC Loss Contour vs Sequence Length and Per-Timestep Probability</td>
      <td>443</td>
    </tr>
    <tr>
      <td>10.5.3.61 Python Code to generate Figure 10.64 illustrating Maximum Markov Networks (M³N) Loss</td>
      <td>444</td>
    </tr>
    <tr>
      <td>10.5.3.62 Python Code to generate Figure 10.65 illustrating M³N Loss Surface (Δ = 1.0)</td>
      <td>445</td>
    </tr>
    <tr>
      <td>10.5.3.63 Python Code to generate Figure 10.66 illustrating Contour Plot of M³N Loss (Δ = 1.0)</td>
      <td>445</td>
    </tr>
    <tr>
      <td>10.5.3.64 Python Code to generate Figure 10.67 illustrating Listwise Ranking Losses (ListNet vs ListMLE)</td>
      <td>446</td>
    </tr>
    <tr>
      <td>10.5.3.65 Python Code to generate Figure 10.68 illustrating Listwise Ranking Losses (3D Surfaces)</td>
      <td>448</td>
    </tr>
    <tr>
      <td>10.5.3.66 Python Code to generate Figure 10.69 illustrating Listwise Ranking Loss Contours</td>
      <td>449</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>10.5.3.67</td>
      <td>Python Code to generate Figure 10.70 illustrating Pairwise Ranking Losses</td>
      <td>451</td>
    </tr>
    <tr>
      <td>10.5.3.68</td>
      <td>Python Code to generate Figure 10.71 illustrating Pairwise Ranking Loss Landscapes</td>
      <td>451</td>
    </tr>
    <tr>
      <td>10.5.3.69</td>
      <td>Python Code to generate Figure 10.72 illustrating 3D Surfaces of Pairwise Ranking Losses</td>
      <td>453</td>
    </tr>
    <tr>
      <td>10.5.3.70</td>
      <td>Python Code to generate Figure 10.73 illustrating Contrastive Divergence (CD) Loss vs Weight</td>
      <td>454</td>
    </tr>
    <tr>
      <td>10.5.3.71</td>
      <td>Python Code to generate Figure 10.74 illustrating Contrastive Divergence (CD) Loss Landscape</td>
      <td>455</td>
    </tr>
    <tr>
      <td>10.5.3.72</td>
      <td>Python Code to generate Figure 10.75 illustrating Contrastive Divergence (CD) Loss Landscape (3D)</td>
      <td>456</td>
    </tr>
    <tr>
      <td>10.5.3.73</td>
      <td>Python Code to generate Figure 10.76 illustrating Persistent Contrastive Divergence (PCD) Loss Landscape</td>
      <td>457</td>
    </tr>
    <tr>
      <td>10.5.3.74</td>
      <td>Python Code to generate Figure 10.77 illustrating Persistent Contrastive Divergence (PCD) Loss Landscape (3D)</td>
      <td>458</td>
    </tr>
    <tr>
      <td>10.5.3.75</td>
      <td>Python Code to generate Figure 10.78 illustrating Hausdorff Distance Loss Visualization</td>
      <td>460</td>
    </tr>
    <tr>
      <td>10.5.3.76</td>
      <td>Python Code to generate Figure 10.79 illustrating Hausdorff Distance Heatmap</td>
      <td>461</td>
    </tr>
    <tr>
      <td><b>11 Few-Shot Learning</b></td>
      <td></td>
      <td><b>462</b></td>
    </tr>
    <tr>
      <td>11.1</td>
      <td>Meta-learning formulation in Few Shot Learning</td>
      <td>463</td>
    </tr>
    <tr>
      <td>11.2</td>
      <td>Bayesian Methods in Few Shot Learning</td>
      <td>465</td>
    </tr>
    <tr>
      <td>11.3</td>
      <td>Prototypical networks in Few Shot Learning</td>
      <td>467</td>
    </tr>
    <tr>
      <td>11.4</td>
      <td>Model-Agnostic Meta-Learning (MAML) in Few Shot Learning</td>
      <td>469</td>
    </tr>
    <tr>
      <td>11.5</td>
      <td>Metric-based learning in Few Shot Learning</td>
      <td>470</td>
    </tr>
    <tr>
      <td>11.6</td>
      <td>Bayesian methods in Few Shot Learning</td>
      <td>472</td>
    </tr>
    <tr>
      <td><b>12 Metric Learning</b></td>
      <td></td>
      <td><b>474</b></td>
    </tr>
    <tr>
      <td>12.1</td>
      <td>Large Margin Nearest Neighbors (LMNN) approach to Metric Learning</td>
      <td>475</td>
    </tr>
    <tr>
      <td>12.2</td>
      <td>Information-theoretic metric learning (ITML) framework approach to Metric Learning</td>
      <td>476</td>
    </tr>
    <tr>
      <td>12.3</td>
      <td>Deep Metric Learning</td>
      <td>477</td>
    </tr>
    <tr>
      <td>12.4</td>
      <td>Normalized temperature-scaled cross-entropy loss (NT-Xent) approach to Metric Learning</td>
      <td>478</td>
    </tr>
    <tr>
      <td><b>13 Adversial Learning</b></td>
      <td></td>
      <td><b>480</b></td>
    </tr>
    <tr>
      <td>13.1</td>
      <td>Fast Gradient Sign Method (FGSM) in Adversial Learning</td>
      <td>481</td>
    </tr>
    <tr>
      <td>13.2</td>
      <td>Projected Gradient Descent (PGD) method in Adversial Learning</td>
      <td>483</td>
    </tr>
    <tr>
      <td>13.3</td>
      <td>Generative approach in Adversarial Learning</td>
      <td>484</td>
    </tr>
    <tr>
      <td>13.4</td>
      <td>Interpreting Generative Adversarial Networks within the framework of Energy-Based models</td>
      <td>485</td>
    </tr>
    <tr>
      <td><b>14 Casual Inference in Deep Neural Networks</b></td>
      <td></td>
      <td><b>487</b></td>
    </tr>
    <tr>
      <td>14.1</td>
      <td>Structural Causal Model (SCM)</td>
      <td>488</td>
    </tr>
    <tr>
      <td>14.2</td>
      <td>Counterfactual reasoning in causal inference for deep neural networks</td>
      <td>490</td>
    </tr>
    <tr>
      <td>14.3</td>
      <td>Domain adaptation in causal inference within deep neural networks</td>
      <td>491</td>
    </tr>
    <tr>
      <td>14.4</td>
      <td>Invariant Risk Minimization (IRM) in causal inference for deep neural networks</td>
      <td>492</td>
    </tr>
    <tr>
      <td>14.5</td>
      <td>Empirical Risk Minimization (ERM) in causal inference for deep neural networks</td>
      <td>494</td>
    </tr>
  </tbody>
</table>
15 Network Architecture Search (NAS) in Deep Neural Networks 496

15.1 Evolutionary Algorithms in Network Architecture Search 497

15.2 Reinforcement Learning in Network Architecture Search 500

15.3 Policy gradient methods in Network Architecture Search 502

15.4 Neural tangent kernels (NTKs) in Network Architecture Search 504

16 Learning paradigms 507

16.1 Unsupervised learning 507

16.1.1 Literature Review of Unsupervised Learning 507

16.1.2 Recent Literature Review of Unsupervised Learning 510

16.1.3 Mathematical Analysis of Unsupervised Learning 513

16.1.4 Information Bottleneck (IB) method 514

16.1.4.1 Literature Review of Information Bottleneck (IB) method 514

16.1.4.2 Recent Literature Review of Information Bottleneck (IB) method 517

16.1.4.3 Mathematical Analysis of Information Bottleneck (IB) method 519

16.1.5 Restricted Boltzmann machines (RBMs) 521

16.1.5.1 Literature Review of Restricted Boltzmann machines (RBMs) 521

16.1.5.2 Recent Literature Review of Restricted Boltzmann machines (RBMs) 523

16.1.5.3 Mathematical Analysis of Restricted Boltzmann machines (RBMs) 525

16.1.6 Deep belief networks (DBNs) 527

16.1.6.1 Literature Review of Deep belief networks (DBNs) 527

16.1.6.2 Recent Literature Review of Deep belief networks (DBNs) 530

16.1.6.3 Mathematical Analysis of Deep belief networks (DBNs) 532

16.1.7 t-distributed stochastic neighbor embedding (t-SNE) 534

16.1.7.1 Literature Review of t-distributed stochastic neighbor embedding 534

16.1.7.2 Recent Literature Review of t-distributed stochastic neighbor embedding 537

16.1.7.3 Mathematical Analysis of t-distributed stochastic neighbor embedding 539

16.1.8 Locally Linear Embedding (LLE) 542

16.1.8.1 Literature Review of Locally Linear Embedding (LLE) 542

16.1.8.2 Recent Literature Review of Locally Linear Embedding (LLE) 544

16.1.8.3 Mathematical Analysis of Locally Linear Embedding (LLE) 546

16.1.8.4 Python Code to generate Figure 16.5 illustrating the Locally Linear Embedding (LLE) applied to a Swiss roll dataset 547

16.1.9 Independent Component Analysis (ICA) 548

16.1.9.1 Literature Review of Independent Component Analysis (ICA) 548

16.1.9.2 Recent Literature Review of Independent Component Analysis (ICA) 550

16.1.9.3 Mathematical Analysis of Independent Component Analysis (ICA) 552

16.1.9.4 Python Code to generate Figure 16.6 illustrating the Independent Component Analysis (ICA) of mixed signals 554

16.2 Supervised Learning 555

16.2.1 Literature Review of Supervised Learning 555

16.2.2 Recent Literature Review of Supervised Learning 558

16.2.3 Mathematical Analysis of Supervised Learning 560

16.2.4 Python Code to generate Figure 16.7 illustrating the Supervised Learning in Regression 562

16.2.5 Bias-Variance Tradeoff of Supervised Learning 563

16.2.5.1 Literature Review of Bias-Variance Tradeoff of Supervised Learning 563

16.2.5.2 Recent Literature Review of Bias-Variance Tradeoff of Supervised Learning 565

16.2.5.3 Mathematical Analysis of Bias-Variance Tradeoff of Supervised Learning 567

16.2.5.3 Mathematical Analysis of Bias-Variance Tradeoff of Supervised Learning 567
<table>
  <tbody>
    <tr>
      <td>16.2.5.4</td>
      <td>Python Code to generate Figure 16.8 illustrating the Bias-Variance Tradeoff of Supervised Learning</td>
      <td>569</td>
    </tr>
    <tr>
      <td>16.2.6</td>
      <td>Support vector machine</td>
      <td>570</td>
    </tr>
    <tr>
      <td>16.2.6.1</td>
      <td>Literature Review of Support vector machine</td>
      <td>570</td>
    </tr>
    <tr>
      <td>16.2.6.2</td>
      <td>Recent Literature Review of Support vector machine</td>
      <td>571</td>
    </tr>
    <tr>
      <td>16.2.6.3</td>
      <td>Mathematical Analysis of Support vector machine</td>
      <td>574</td>
    </tr>
    <tr>
      <td>16.2.6.4</td>
      <td>Python Code to generate Figure 16.9 illustrating the Support Vector Machine (SVM) with linear kernel</td>
      <td>577</td>
    </tr>
    <tr>
      <td>16.2.7</td>
      <td>Linear regression</td>
      <td>578</td>
    </tr>
    <tr>
      <td>16.2.7.1</td>
      <td>Literature Review of Linear regression</td>
      <td>578</td>
    </tr>
    <tr>
      <td>16.2.7.2</td>
      <td>Recent Literature Review of Linear regression</td>
      <td>579</td>
    </tr>
    <tr>
      <td>16.2.7.3</td>
      <td>Mathematical Analysis of Linear regression</td>
      <td>581</td>
    </tr>
    <tr>
      <td>16.2.8</td>
      <td>Logistic regression</td>
      <td>585</td>
    </tr>
    <tr>
      <td>16.2.8.1</td>
      <td>Literature Review of Logistic regression</td>
      <td>585</td>
    </tr>
    <tr>
      <td>16.2.8.2</td>
      <td>Recent Literature Review of Logistic regression</td>
      <td>586</td>
    </tr>
    <tr>
      <td>16.2.8.3</td>
      <td>Mathematical Analysis of Logistic regression</td>
      <td>588</td>
    </tr>
    <tr>
      <td>16.2.9</td>
      <td>Linear discriminant analysis</td>
      <td>590</td>
    </tr>
    <tr>
      <td>16.2.9.1</td>
      <td>Literature Review of Linear discriminant analysis</td>
      <td>590</td>
    </tr>
    <tr>
      <td>16.2.9.2</td>
      <td>Recent Literature Review of Linear discriminant analysis</td>
      <td>591</td>
    </tr>
    <tr>
      <td>16.2.9.3</td>
      <td>Mathematical Analysis of Linear discriminant analysis</td>
      <td>593</td>
    </tr>
    <tr>
      <td>16.2.9.4</td>
      <td>Multiclass Linear discriminant analysis</td>
      <td>595</td>
    </tr>
    <tr>
      <td>16.2.9.5</td>
      <td>Python Code to generate Figure 16.18 illustrating the Eigenvalue Perturbation Analysis of Linear Discriminant Analysis (LDA)</td>
      <td>608</td>
    </tr>
    <tr>
      <td>16.2.9.6</td>
      <td>Python Code to generate Figure 16.19 illustrating the Spectral decomposition of the discriminant space in Linear Discriminant Analysis (LDA)</td>
      <td>610</td>
    </tr>
    <tr>
      <td>16.2.9.7</td>
      <td>Python Code to generate Figure 16.20 illustrating the Asymptotic convergence of the decision boundary in Linear Discriminant Analysis (LDA)</td>
      <td>611</td>
    </tr>
    <tr>
      <td>16.2.9.8</td>
      <td>Python Code to generate Figure 16.21 illustrating the Exact asymptotic bounds for the misclassification probability in Linear Discriminant Analysis (LDA)</td>
      <td>612</td>
    </tr>
    <tr>
      <td>16.2.9.9</td>
      <td>Python Code to generate Figure 16.22 illustrating the Condition number of the Fisher matrix in Linear Discriminant Analysis (LDA)</td>
      <td>613</td>
    </tr>
    <tr>
      <td>16.2.9.10</td>
      <td>Python Code to generate Figure 16.23 illustrating the Random Matrix Theory (RMT) perspective on Linear Discriminant Analysis (LDA)</td>
      <td>615</td>
    </tr>
    <tr>
      <td>16.2.9.11</td>
      <td>Python Code to generate Figure 16.24 illustrating the Exact rate of eigenvalue concentration in Linear Discriminant Analysis (LDA)</td>
      <td>616</td>
    </tr>
    <tr>
      <td>16.2.9.12</td>
      <td>Python Code to generate Figure 16.25 illustrating the Weyl bounds on the eigenvalues of the Fisher matrix in Linear Discriminant Analysis (LDA)</td>
      <td>617</td>
    </tr>
    <tr>
      <td>16.2.9.13</td>
      <td>Python Code to generate Figure 16.26 illustrating the Ky Fan norm bound of the eigenvalues of the Linear Discriminant Analysis (LDA)</td>
      <td>619</td>
    </tr>
    <tr>
      <td>16.2.9.14</td>
      <td>Python Code to generate Figure 16.27 illustrating the Bauer-Fike theorem applied to eigenvalues of the Fisher matrix in LDA</td>
      <td>620</td>
    </tr>
    <tr>
      <td>16.2.9.15</td>
      <td>Python Code to generate Figure 16.28 illustrating the Generalization error of Linear Discriminant Analysis (LDA) using PAC-Bayesian bounds</td>
      <td>622</td>
    </tr>
    <tr>
      <td>16.2.10</td>
      <td>Naïve Bayes classifier</td>
      <td>623</td>
    </tr>
    <tr>
      <td>16.2.10.1</td>
      <td>Literature Review of Naïve Bayes classifier</td>
      <td>623</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>16.2.10.2 Recent Literature Review of Naïve Bayes classifier</td>
      <td>625</td>
    </tr>
    <tr>
      <td>16.2.10.3 Mathematical Analysis of Naïve Bayes classifier</td>
      <td>628</td>
    </tr>
    <tr>
      <td>16.2.10.4 Python Code to generate Figure 16.29</td>
      <td>630</td>
    </tr>
    <tr>
      <td>16.2.11 Decision tree learning</td>
      <td>631</td>
    </tr>
    <tr>
      <td>16.2.11.1 Literature Review of Decision tree learning</td>
      <td>631</td>
    </tr>
    <tr>
      <td>16.2.11.2 Recent Literature Review of Decision tree learning</td>
      <td>632</td>
    </tr>
    <tr>
      <td>16.2.11.3 Mathematical Analysis of Decision tree learning</td>
      <td>635</td>
    </tr>
    <tr>
      <td>16.2.12 k-nearest neighbors algorithm</td>
      <td>637</td>
    </tr>
    <tr>
      <td>16.2.12.1 Literature Review of k-nearest neighbors (KNN) algorithm</td>
      <td>637</td>
    </tr>
    <tr>
      <td>16.2.12.2 Recent Literature Review of k-nearest neighbors (KNN) algorithm</td>
      <td>638</td>
    </tr>
    <tr>
      <td>16.2.12.3 Mathematical Analysis of k-nearest neighbors (KNN) algorithm</td>
      <td>639</td>
    </tr>
    <tr>
      <td>16.2.13 Similarity learning</td>
      <td>642</td>
    </tr>
    <tr>
      <td>16.2.13.1 Literature Review of Similarity learning</td>
      <td>642</td>
    </tr>
    <tr>
      <td>16.2.13.2 Recent Literature Review of Similarity learning</td>
      <td>643</td>
    </tr>
    <tr>
      <td>16.2.13.3 Mathematical Analysis of Similarity learning</td>
      <td>645</td>
    </tr>
    <tr>
      <td>16.3 Self learning</td>
      <td>647</td>
    </tr>
    <tr>
      <td>16.3.1 Literature Review of Self Learning</td>
      <td>647</td>
    </tr>
    <tr>
      <td>16.3.2 Recent Literature Review of Self Learning</td>
      <td>649</td>
    </tr>
    <tr>
      <td>16.3.3 Mathematical Analysis of Self Learning</td>
      <td>651</td>
    </tr>
    <tr>
      <td>16.3.4 Python Code to generate Figure 16.34</td>
      <td>653</td>
    </tr>
    <tr>
      <td>16.4 Reinforcement learning</td>
      <td>654</td>
    </tr>
    <tr>
      <td>16.4.1 Literature Review of Reinforcement learning</td>
      <td>654</td>
    </tr>
    <tr>
      <td>16.4.2 Recent Literature Review of Reinforcement learning</td>
      <td>656</td>
    </tr>
    <tr>
      <td>16.4.3 Mathematical Analysis of Reinforcement learning</td>
      <td>658</td>
    </tr>
    <tr>
      <td>16.4.4 Python Code to generate Figure 16.36</td>
      <td>661</td>
    </tr>
    <tr>
      <td>16.4.5 Policy Gradient Method</td>
      <td>662</td>
    </tr>
    <tr>
      <td>16.4.5.1 Literature Review of Policy Gradient Method</td>
      <td>662</td>
    </tr>
    <tr>
      <td>16.4.5.2 Recent Literature Review of Policy Gradient Method</td>
      <td>664</td>
    </tr>
    <tr>
      <td>16.4.5.3 Mathematical Analysis of Policy Gradient Method</td>
      <td>666</td>
    </tr>
    <tr>
      <td>16.4.5.4 Python Code to generate Figure 16.37</td>
      <td>668</td>
    </tr>
    <tr>
      <td>16.4.6 Deep Reinforcement Learning</td>
      <td>669</td>
    </tr>
    <tr>
      <td>16.4.6.1 Literature Review of Deep Reinforcement Learning</td>
      <td>669</td>
    </tr>
    <tr>
      <td>16.4.6.2 Recent Literature Review of Deep Reinforcement Learning</td>
      <td>671</td>
    </tr>
    <tr>
      <td>16.4.6.3 Mathematical Analysis of Deep Reinforcement Learning</td>
      <td>673</td>
    </tr>
    <tr>
      <td>16.4.6.3.1 Python Code to generate Figure 16.38</td>
      <td>676</td>
    </tr>
    <tr>
      <td>16.4.6.4 Advantage Actor-Critic (A2C)</td>
      <td>677</td>
    </tr>
    <tr>
      <td>16.4.6.4.1 Literature Review of Advantage Actor-Critic (A2C)</td>
      <td>677</td>
    </tr>
    <tr>
      <td>16.4.6.4.2 Recent Literature Review of Advantage Actor-Critic (A2C)</td>
      <td>679</td>
    </tr>
    <tr>
      <td>16.4.6.4.3 Mathematical Analysis of Advantage Actor-Critic (A2C)</td>
      <td>681</td>
    </tr>
    <tr>
      <td>16.4.6.4.4 Python Code to generate Figure 16.39</td>
      <td>683</td>
    </tr>
    <tr>
      <td>16.4.6.5 Deep Deterministic Policy Gradient</td>
      <td>684</td>
    </tr>
    <tr>
      <td>16.4.6.5.1 Literature Review of Deep Deterministic Policy Gradient</td>
      <td>684</td>
    </tr>
    <tr>
      <td>16.4.6.5.2 Recent Literature Review of Deep Deterministic Policy Gradient</td>
      <td>687</td>
    </tr>
    <tr>
      <td>16.4.6.5.3 Mathematical Analysis of Deep Deterministic Policy Gradient</td>
      <td>688</td>
    </tr>
    <tr>
      <td>16.4.6.5.4 Python Code to generate Figure 16.40</td>
      <td>690</td>
    </tr>
    <tr>
      <td>16.4.6.6 Proximal Policy Optimization</td>
      <td>692</td>
    </tr>
    <tr>
      <td>16.4.6.6.1 Literature Review of Proximal Policy Optimization</td>
      <td>692</td>
    </tr>
    <tr>
      <td>16.4.6.6.2 Recent Literature Review of Proximal Policy Optimization</td>
      <td>694</td>
    </tr>
    <tr>
      <td>16.4.6.6.3 Mathematical Analysis of Proximal Policy Optimization</td>
      <td>695</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>16.4.6.6.4</td>
      <td>Python Code to generate Figure 16.41</td>
      <td>697</td>
    </tr>
    <tr>
      <td>16.4.6.7</td>
      <td>Soft Actor-Critic (SAC) algorithm</td>
      <td>699</td>
    </tr>
    <tr>
      <td>16.4.6.7.1</td>
      <td>Literature Review of Soft Actor-Critic (SAC) algorithm</td>
      <td>699</td>
    </tr>
    <tr>
      <td>16.4.6.7.2</td>
      <td>Recent Literature Review of Soft Actor-Critic (SAC) algorithm</td>
      <td>701</td>
    </tr>
    <tr>
      <td>16.4.6.7.3</td>
      <td>Mathematical Analysis of Soft Actor-Critic (SAC) algorithm</td>
      <td>703</td>
    </tr>
    <tr>
      <td>16.4.6.7.4</td>
      <td>Python Code to generate Figure 16.42</td>
      <td>705</td>
    </tr>
    <tr>
      <td>16.5</td>
      <td>Neuroevolution</td>
      <td>707</td>
    </tr>
    <tr>
      <td>16.5.1</td>
      <td>Python Code to generate Figure 16.43</td>
      <td>709</td>
    </tr>
    <tr>
      <td>16.5.2</td>
      <td>Neuro-Genetic Evolution</td>
      <td>711</td>
    </tr>
    <tr>
      <td>16.5.2.1</td>
      <td>Literature Review of Neuro-Genetic Evolution</td>
      <td>711</td>
    </tr>
    <tr>
      <td>16.5.2.2</td>
      <td>Recent Literature Review of Neuro-Genetic Evolution</td>
      <td>713</td>
    </tr>
    <tr>
      <td>16.5.2.3</td>
      <td>Mathematical Analysis of Neuro-Genetic Evolution</td>
      <td>715</td>
    </tr>
    <tr>
      <td>16.5.2.4</td>
      <td>Python Code to generate Figure 16.44</td>
      <td>716</td>
    </tr>
    <tr>
      <td>16.5.3</td>
      <td>Cellular Encoding (CE)</td>
      <td>718</td>
    </tr>
    <tr>
      <td>16.5.3.1</td>
      <td>Literature Review of Cellular Encoding (CE)</td>
      <td>718</td>
    </tr>
    <tr>
      <td>16.5.3.2</td>
      <td>Recent Literature Review of Cellular Encoding (CE)</td>
      <td>720</td>
    </tr>
    <tr>
      <td>16.5.3.3</td>
      <td>Mathematical Analysis of Cellular Encoding (CE)</td>
      <td>722</td>
    </tr>
    <tr>
      <td>16.5.3.4</td>
      <td>Python Code to generate Figures 16.45 and 16.46</td>
      <td>725</td>
    </tr>
    <tr>
      <td>16.5.4</td>
      <td>GeNeralized Acquisition of Recurrent Links (GNARL)</td>
      <td>727</td>
    </tr>
    <tr>
      <td>16.5.4.1</td>
      <td>Literature Review of GeNeralized Acquisition of Recurrent Links (GNARL)</td>
      <td>727</td>
    </tr>
    <tr>
      <td>16.5.4.2</td>
      <td>Mathematical Analysis of GeNeralized Acquisition of Recurrent Links (GNARL)</td>
      <td>729</td>
    </tr>
    <tr>
      <td>16.5.4.3</td>
      <td>Python Code to generate Figure 16.47</td>
      <td>732</td>
    </tr>
    <tr>
      <td>16.5.5</td>
      <td>Neuroevolution of augmenting topologies (NEAT)</td>
      <td>734</td>
    </tr>
    <tr>
      <td>16.5.5.1</td>
      <td>Python Code to generate Figure 16.48</td>
      <td>736</td>
    </tr>
    <tr>
      <td>16.5.6</td>
      <td>Hypercube-based NeuroEvolution of Augmenting Topologies (HyperNEAT)</td>
      <td>738</td>
    </tr>
    <tr>
      <td>16.5.6.1</td>
      <td>Python Code to generate Figure 16.49</td>
      <td>740</td>
    </tr>
    <tr>
      <td>16.5.7</td>
      <td>Evolvable Substrate Hypercube-based NeuroEvolution of Augmenting Topologies (ES-HyperNEAT)</td>
      <td>742</td>
    </tr>
    <tr>
      <td>16.5.7.1</td>
      <td>Python Code to generate Figure 16.50</td>
      <td>744</td>
    </tr>
    <tr>
      <td>16.5.8</td>
      <td>Evolutionary Acquisition of Neural Topologies (EANT/EANT2)</td>
      <td>746</td>
    </tr>
    <tr>
      <td>16.5.8.1</td>
      <td>Python Code to generate Figure 16.51</td>
      <td>748</td>
    </tr>
    <tr>
      <td>16.5.9</td>
      <td>Interactively Constrained Neuro-Evolution (ICONE)</td>
      <td>751</td>
    </tr>
    <tr>
      <td>16.5.9.1</td>
      <td>Python Code to generate Figure 16.52</td>
      <td>752</td>
    </tr>
    <tr>
      <td>16.5.10</td>
      <td>Deus Ex Neural Network (DXNN)</td>
      <td>755</td>
    </tr>
    <tr>
      <td>16.5.10.1</td>
      <td>Python Code to generate Figure 16.53</td>
      <td>757</td>
    </tr>
    <tr>
      <td>16.5.11</td>
      <td>Spectrum-diverse Unified Neuroevolution Architecture (SUNA)</td>
      <td>760</td>
    </tr>
    <tr>
      <td>16.5.11.1</td>
      <td>Python Code to generate Figure 16.54</td>
      <td>762</td>
    </tr>
    <tr>
      <td>17</td>
      <td>Training Neural Networks</td>
      <td>767</td>
    </tr>
    <tr>
      <td>17.1</td>
      <td>Literature Review of Training Neural Networks</td>
      <td>768</td>
    </tr>
    <tr>
      <td>17.2</td>
      <td>Backpropagation Algorithm</td>
      <td>769</td>
    </tr>
    <tr>
      <td>17.2.0.1</td>
      <td>Python Code to generate Figure 17.1</td>
      <td>771</td>
    </tr>
    <tr>
      <td>17.3</td>
      <td>Gradient Descent Variants</td>
      <td>772</td>
    </tr>
    <tr>
      <td>17.3.1</td>
      <td>SGD (Stochastic Gradient Descent) Optimizer</td>
      <td>774</td>
    </tr>
    <tr>
      <td>17.3.1.1</td>
      <td>Literature Review of SGD (Stochastic Gradient Descent) Optimizer</td>
      <td>774</td>
    </tr>
    <tr>
      <td>17.3.1.2</td>
      <td>Analysis of SGD (Stochastic Gradient Descent) Optimizer</td>
      <td>774</td>
    </tr>
    <tr>
      <td>17.3.1.3</td>
      <td>Python Code to generate Figure 17.2</td>
      <td>776</td>
    </tr>
    <tr>
      <td>17.3.2</td>
      <td>Nesterov Accelerated Gradient Descent (NAG)</td>
      <td>777</td>
    </tr>
    <tr>
      <td>17.3.2.1</td>
      <td>Literature Review of Nesterov Accelerated Gradient Descent (NAG)</td>
      <td>777</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>17.3.2.2</td>
      <td>Analysis of Nesterov Accelerated Gradient Descent (NAG)</td>
      <td>779</td>
    </tr>
    <tr>
      <td>17.3.2.3</td>
      <td>Python Code to generate Figure 17.3</td>
      <td>782</td>
    </tr>
    <tr>
      <td>17.3.3</td>
      <td>Adam (Adaptive Moment Estimation) Optimizer</td>
      <td>783</td>
    </tr>
    <tr>
      <td>17.3.3.1</td>
      <td>Literature Review of Adam (Adaptive Moment Estimation) Optimizer</td>
      <td>783</td>
    </tr>
    <tr>
      <td>17.3.3.2</td>
      <td>Analysis of Adam (Adaptive Moment Estimation) Optimizer</td>
      <td>783</td>
    </tr>
    <tr>
      <td>17.3.3.3</td>
      <td>Python Code to generate Figure 17.4</td>
      <td>787</td>
    </tr>
    <tr>
      <td>17.3.4</td>
      <td>RMSProp (Root Mean Squared Propagation) Optimizer</td>
      <td>789</td>
    </tr>
    <tr>
      <td>17.3.4.1</td>
      <td>Literature Review of RMSProp (Root Mean Squared Propagation) Optimizer</td>
      <td>789</td>
    </tr>
    <tr>
      <td>17.3.4.2</td>
      <td>Analysis of RMSProp (Root Mean Squared Propagation) Optimizer</td>
      <td>789</td>
    </tr>
    <tr>
      <td>17.3.4.3</td>
      <td>Python Code to generate Figure 17.5</td>
      <td>795</td>
    </tr>
    <tr>
      <td>17.4</td>
      <td>Overfitting and Regularization Techniques</td>
      <td>796</td>
    </tr>
    <tr>
      <td>17.4.1</td>
      <td>Literature Review of Overfitting and Regularization Techniques</td>
      <td>796</td>
    </tr>
    <tr>
      <td>17.4.2</td>
      <td>Analysis of Overfitting and Regularization Techniques</td>
      <td>797</td>
    </tr>
    <tr>
      <td>17.4.3</td>
      <td>Dropout</td>
      <td>799</td>
    </tr>
    <tr>
      <td>17.4.3.1</td>
      <td>Literature Review of Dropout</td>
      <td>799</td>
    </tr>
    <tr>
      <td>17.4.3.2</td>
      <td>Analysis of Dropout</td>
      <td>800</td>
    </tr>
    <tr>
      <td>17.4.3.3</td>
      <td>Python Code to generate Figure 17.6</td>
      <td>802</td>
    </tr>
    <tr>
      <td>17.4.4</td>
      <td>L1/L2 Regularization and Overfitting</td>
      <td>804</td>
    </tr>
    <tr>
      <td>17.4.4.1</td>
      <td>Literature Review of L1 (Lasso) Regularization</td>
      <td>804</td>
    </tr>
    <tr>
      <td>17.4.4.2</td>
      <td>Literature Review of L2 (Ridge Regression) Regularization</td>
      <td>804</td>
    </tr>
    <tr>
      <td>17.4.4.3</td>
      <td>Analysis of L1/L2 Regularization and Overfitting</td>
      <td>805</td>
    </tr>
    <tr>
      <td>17.4.5</td>
      <td>Elastic Net Regularization</td>
      <td>807</td>
    </tr>
    <tr>
      <td>17.4.5.1</td>
      <td>Literature Review of Elastic Net Regularization</td>
      <td>807</td>
    </tr>
    <tr>
      <td>17.4.5.2</td>
      <td>Analysis of Elastic Net Regularization</td>
      <td>808</td>
    </tr>
    <tr>
      <td>17.4.6</td>
      <td>Early Stopping</td>
      <td>809</td>
    </tr>
    <tr>
      <td>17.4.6.1</td>
      <td>Literature Review of Early Stopping</td>
      <td>809</td>
    </tr>
    <tr>
      <td>17.4.6.2</td>
      <td>Analysis of Early Stopping</td>
      <td>810</td>
    </tr>
    <tr>
      <td>17.4.7</td>
      <td>Data Augmentation</td>
      <td>811</td>
    </tr>
    <tr>
      <td>17.4.7.1</td>
      <td>Literature Review of Data Augmentation</td>
      <td>811</td>
    </tr>
    <tr>
      <td>17.4.7.2</td>
      <td>Analysis of Data Augmentation</td>
      <td>812</td>
    </tr>
    <tr>
      <td>17.4.8</td>
      <td>Cross-Validation</td>
      <td>814</td>
    </tr>
    <tr>
      <td>17.4.8.1</td>
      <td>Literature Review of Cross-Validation</td>
      <td>814</td>
    </tr>
    <tr>
      <td>17.4.8.2</td>
      <td>Analysis of Cross-Validation</td>
      <td>815</td>
    </tr>
    <tr>
      <td>17.4.9</td>
      <td>Pruning</td>
      <td>817</td>
    </tr>
    <tr>
      <td>17.4.9.1</td>
      <td>Literature Review of Pruning</td>
      <td>817</td>
    </tr>
    <tr>
      <td>17.4.9.2</td>
      <td>Analysis of Pruning</td>
      <td>817</td>
    </tr>
    <tr>
      <td>17.4.10</td>
      <td>Ensemble Methods</td>
      <td>819</td>
    </tr>
    <tr>
      <td>17.4.10.1</td>
      <td>Literature Review of Ensemble Methods</td>
      <td>819</td>
    </tr>
    <tr>
      <td>17.4.10.2</td>
      <td>Analysis of Ensemble Methods</td>
      <td>820</td>
    </tr>
    <tr>
      <td>17.4.11</td>
      <td>Noise Injection</td>
      <td>822</td>
    </tr>
    <tr>
      <td>17.4.11.1</td>
      <td>Literature Review of Noise Injection</td>
      <td>822</td>
    </tr>
    <tr>
      <td>17.4.11.2</td>
      <td>Analysis of Noise Injection</td>
      <td>823</td>
    </tr>
    <tr>
      <td>17.4.12</td>
      <td>Batch Normalization</td>
      <td>825</td>
    </tr>
    <tr>
      <td>17.4.12.1</td>
      <td>Literature Review of Batch Normalization</td>
      <td>825</td>
    </tr>
    <tr>
      <td>17.4.12.2</td>
      <td>Analysis of Batch Normalization</td>
      <td>825</td>
    </tr>
    <tr>
      <td>17.4.13</td>
      <td>Weight Decay</td>
      <td>827</td>
    </tr>
    <tr>
      <td>17.4.13.1</td>
      <td>Literature Review of Weight Decay</td>
      <td>827</td>
    </tr>
    <tr>
      <td>17.4.13.2</td>
      <td>Analysis of Weight Decay</td>
      <td>828</td>
    </tr>
    <tr>
      <td>17.4.14</td>
      <td>Max Norm Constraints</td>
      <td>830</td>
    </tr>
    <tr>
      <td>17.4.14.1</td>
      <td>Literature Review of Max Norm Constraints</td>
      <td>830</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>17.4.14.2 Analysis of Max Norm Constraints</td>
      <td>830</td>
    </tr>
    <tr>
      <td>17.4.15 Transfer Learning</td>
      <td>832</td>
    </tr>
    <tr>
      <td>17.4.15.1 Literature Review of Transfer Learning</td>
      <td>832</td>
    </tr>
    <tr>
      <td>17.4.15.2 Analysis of Transfer Learning</td>
      <td>833</td>
    </tr>
    <tr>
      <td>17.5 Hyperparameter Tuning</td>
      <td>834</td>
    </tr>
    <tr>
      <td>17.5.1 Literature Review of Hyperparameter Tuning</td>
      <td>834</td>
    </tr>
    <tr>
      <td>17.5.2 Analysis of Hyperparameter Tuning</td>
      <td>835</td>
    </tr>
    <tr>
      <td>17.5.3 Grid Search</td>
      <td>836</td>
    </tr>
    <tr>
      <td>17.5.3.1 Literature Review of Grid Search</td>
      <td>836</td>
    </tr>
    <tr>
      <td>17.5.3.2 Analysis of Grid Search</td>
      <td>837</td>
    </tr>
    <tr>
      <td>17.5.4 Random Search</td>
      <td>839</td>
    </tr>
    <tr>
      <td>17.5.4.1 Literature Review of Random Search</td>
      <td>839</td>
    </tr>
    <tr>
      <td>17.5.4.2 Analysis of Random Search</td>
      <td>840</td>
    </tr>
    <tr>
      <td>17.5.5 Bayesian Optimization</td>
      <td>842</td>
    </tr>
    <tr>
      <td>17.5.5.1 Literature Review of Bayesian Optimization</td>
      <td>842</td>
    </tr>
    <tr>
      <td>17.5.5.2 Analysis of Bayesian Optimization</td>
      <td>843</td>
    </tr>
    <tr>
      <td>17.5.6 Genetic Algorithms</td>
      <td>845</td>
    </tr>
    <tr>
      <td>17.5.6.1 Literature Review of Genetic Algorithms</td>
      <td>845</td>
    </tr>
    <tr>
      <td>17.5.6.2 Analysis of Genetic Algorithms</td>
      <td>845</td>
    </tr>
    <tr>
      <td>17.5.7 Hyperband</td>
      <td>847</td>
    </tr>
    <tr>
      <td>17.5.7.1 Literature Review of Hyperband</td>
      <td>847</td>
    </tr>
    <tr>
      <td>17.5.7.2 Analysis of Hyperband</td>
      <td>848</td>
    </tr>
    <tr>
      <td>17.5.8 Gradient-Based Optimization</td>
      <td>850</td>
    </tr>
    <tr>
      <td>17.5.8.1 Literature Review of Gradient-Based Optimization</td>
      <td>850</td>
    </tr>
    <tr>
      <td>17.5.8.2 Analysis of Gradient-Based Optimization</td>
      <td>851</td>
    </tr>
    <tr>
      <td>17.5.9 Population-Based Training (PBT)</td>
      <td>852</td>
    </tr>
    <tr>
      <td>17.5.9.1 Literature Review of Population-Based Training (PBT)</td>
      <td>852</td>
    </tr>
    <tr>
      <td>17.5.9.2 Analysis of Population-Based Training (PBT)</td>
      <td>853</td>
    </tr>
    <tr>
      <td>17.5.10 Optuna</td>
      <td>855</td>
    </tr>
    <tr>
      <td>17.5.10.1 Literature Review of Optuna</td>
      <td>855</td>
    </tr>
    <tr>
      <td>17.5.10.2 Analysis of Optuna</td>
      <td>856</td>
    </tr>
    <tr>
      <td>17.5.11 Successive Halving</td>
      <td>857</td>
    </tr>
    <tr>
      <td>17.5.11.1 Literature Review of Successive Halving</td>
      <td>857</td>
    </tr>
    <tr>
      <td>17.5.11.2 Analysis of Successive Halving</td>
      <td>858</td>
    </tr>
    <tr>
      <td>17.5.12 Reinforcement Learning (RL)</td>
      <td>859</td>
    </tr>
    <tr>
      <td>17.5.12.1 Literature Review of Reinforcement Learning (RL)</td>
      <td>859</td>
    </tr>
    <tr>
      <td>17.5.12.2 Analysis of Reinforcement Learning (RL)</td>
      <td>860</td>
    </tr>
    <tr>
      <td>17.5.13 Meta-Learning</td>
      <td>862</td>
    </tr>
    <tr>
      <td>17.5.13.1 Literature Review of Meta-Learning</td>
      <td>862</td>
    </tr>
    <tr>
      <td>17.5.13.2 Analysis of Meta-Learning</td>
      <td>863</td>
    </tr>
    <tr>
      <td><strong>18 Convolution Neural Networks</strong></td>
      <td><strong>866</strong></td>
    </tr>
    <tr>
      <td>18.1 Literature Review of Convolution Neural Networks</td>
      <td>867</td>
    </tr>
    <tr>
      <td>18.2 Key Concepts</td>
      <td>867</td>
    </tr>
    <tr>
      <td>18.3 Applications in Image Processing</td>
      <td>869</td>
    </tr>
    <tr>
      <td>18.3.1 Image Classification</td>
      <td>869</td>
    </tr>
    <tr>
      <td>18.3.1.1 Literature Review of Image Classification</td>
      <td>870</td>
    </tr>
    <tr>
      <td>18.3.1.2 Analysis of Image Classification</td>
      <td>870</td>
    </tr>
    <tr>
      <td>18.3.2 Object Detection</td>
      <td>872</td>
    </tr>
    <tr>
      <td>18.3.2.1 Literature Review of Object Detection</td>
      <td>872</td>
    </tr>
    <tr>
      <td>18.3.2.2 Analysis of Object Detection</td>
      <td>875</td>
    </tr>
    <tr>
      <td>18.4 Real-World Applications</td>
      <td>877</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>18.4.1 Medical Imaging</td>
      <td>877</td>
    </tr>
    <tr>
      <td>18.4.1.1 Literature Review of Medical Imaging</td>
      <td>877</td>
    </tr>
    <tr>
      <td>18.4.1.2 Analysis of Medical Imaging</td>
      <td>878</td>
    </tr>
    <tr>
      <td>18.4.2 Autonomous Vehicles</td>
      <td>880</td>
    </tr>
    <tr>
      <td>18.4.2.1 Literature Review of Autonomous Vehicles</td>
      <td>880</td>
    </tr>
    <tr>
      <td>18.4.2.2 Analysis of Autonomous Vehicles</td>
      <td>881</td>
    </tr>
    <tr>
      <td>18.5 Popular CNN Architectures</td>
      <td>883</td>
    </tr>
    <tr>
      <td>18.5.1 Literature Review of Popular CNN Architectures</td>
      <td>883</td>
    </tr>
    <tr>
      <td>18.5.2 AlexNet</td>
      <td>884</td>
    </tr>
    <tr>
      <td>18.5.3 ResNet</td>
      <td>886</td>
    </tr>
    <tr>
      <td>18.5.4 VGG</td>
      <td>887</td>
    </tr>
    <tr>
      <td><b>19 Recurrent Neural Networks (RNNs)</b></td>
      <td><b>890</b></td>
    </tr>
    <tr>
      <td>19.1 Literature Review of Recurrent Neural Networks (RNNs)</td>
      <td>891</td>
    </tr>
    <tr>
      <td>19.2 Key Concepts</td>
      <td>892</td>
    </tr>
    <tr>
      <td>19.3 Sequence Modeling and Long Short-Term Memory (LSTM) and GRUs</td>
      <td>894</td>
    </tr>
    <tr>
      <td>19.3.1 Literature Review of Sequence Modeling and Long Short-Term Memory (LSTM) and GRUs</td>
      <td>894</td>
    </tr>
    <tr>
      <td>19.3.2 Analysis of Sequence Modeling and Long Short-Term Memory (LSTM) and GRUs</td>
      <td>895</td>
    </tr>
    <tr>
      <td>19.4 Applications in Natural Language Processing</td>
      <td>897</td>
    </tr>
    <tr>
      <td>19.4.1 Literature Review of Applications in Natural Language Processing</td>
      <td>897</td>
    </tr>
    <tr>
      <td>19.4.2 Analysis of Applications in Natural Language Processing</td>
      <td>898</td>
    </tr>
    <tr>
      <td>19.5 Deep Learning and the Collatz Conjecture</td>
      <td>901</td>
    </tr>
    <tr>
      <td>19.5.1 Literature Review of Deep Learning and the Collatz Conjecture</td>
      <td>901</td>
    </tr>
    <tr>
      <td>19.5.2 Analysis of Deep Learning and the Collatz Conjecture</td>
      <td>902</td>
    </tr>
    <tr>
      <td>19.6 Mertens Function and the Collatz Conjecture</td>
      <td>903</td>
    </tr>
    <tr>
      <td>19.6.1 Literature Review of Bounds of Mertens Function</td>
      <td>904</td>
    </tr>
    <tr>
      <td>19.6.2 Deep Learning Approaches</td>
      <td>906</td>
    </tr>
    <tr>
      <td>19.6.3 Experiment 1: Sequence Prediction of the Möbius Function</td>
      <td>907</td>
    </tr>
    <tr>
      <td>19.6.4 Experiment 2: Estimating Growth Bounds on the Mertens function Using Regression Models</td>
      <td>908</td>
    </tr>
    <tr>
      <td>19.6.5 Experiment 3: Neural Network Approximation of the Zeta Function Zeros</td>
      <td>910</td>
    </tr>
    <tr>
      <td>19.6.6 Experiment 4: Graph Neural Networks for Prime Factorization Trees</td>
      <td>910</td>
    </tr>
    <tr>
      <td>19.6.7 Experiment 5: Autoencoders for Dimensionality Reduction of Number-Theoretic Functions</td>
      <td>912</td>
    </tr>
    <tr>
      <td><b>20 Advanced Architectures</b></td>
      <td><b>913</b></td>
    </tr>
    <tr>
      <td>20.1 Transformers and Attention Mechanisms</td>
      <td>914</td>
    </tr>
    <tr>
      <td>20.1.1 Literature Review of Transformers and Attention Mechanisms</td>
      <td>914</td>
    </tr>
    <tr>
      <td>20.1.2 Analysis of Transformers and Attention Mechanisms</td>
      <td>915</td>
    </tr>
    <tr>
      <td>20.2 Generative Adversarial Networks (GANs)</td>
      <td>916</td>
    </tr>
    <tr>
      <td>20.2.1 Literature Review of Generative Adversarial Networks (GANs)</td>
      <td>916</td>
    </tr>
    <tr>
      <td>20.2.2 Analysis of Generative Adversarial Networks (GANs)</td>
      <td>917</td>
    </tr>
    <tr>
      <td>20.3 Autoencoders and Variational Autoencoders</td>
      <td>919</td>
    </tr>
    <tr>
      <td>20.3.1 Literature Review of Autoencoders and Variational Autoencoders</td>
      <td>919</td>
    </tr>
    <tr>
      <td>20.3.2 Analysis of Autoencoders and Variational Autoencoders</td>
      <td>919</td>
    </tr>
    <tr>
      <td>20.4 Graph neural networks (GNNs)</td>
      <td>921</td>
    </tr>
    <tr>
      <td>20.4.1 Literature Review of Graph neural networks (GNNs)</td>
      <td>921</td>
    </tr>
    <tr>
      <td>20.4.2 Analysis of Graph neural networks (GNNs)</td>
      <td>922</td>
    </tr>
    <tr>
      <td>20.5 Physics Informed Neural Networks (PINNs)</td>
      <td>924</td>
    </tr>
    <tr>
      <td>20.5.1 Literature Review of Physics Informed Neural Networks (PINNs)</td>
      <td>924</td>
    </tr>
  </tbody>
</table>
<table><tr><td>20.5.2</td><td>Analysis of Physics Informed Neural Networks (PINNs)</td><td>925</td></tr><tr><td>20.6</td><td>Implementation of the Deep Galerkin Methods (DGM) using the Physics-Informed Neural Networks (PINNs)</td><td>928</td></tr></table>

## 21 Deep Kolmogorov Methods 931

<table><tr><td>21.1</td><td>Literature Review of Deep Kolmogorov Methods</td><td>932</td></tr><tr><td>21.2</td><td>The Kolmogorov Backward Equation and Its Functional Formulation</td><td>933</td></tr><tr><td>21.3</td><td>The Feynman-Kac Representation and Its Justification</td><td>933</td></tr><tr><td>21.4</td><td>Deep Kolmogorov Method: Neural Network Approximation</td><td>935</td></tr></table>

## 22 Reinforcement Learning 940

<table><tr><td>22.1</td><td>Literature Review of Reinforcement Learning</td><td>941</td></tr><tr><td>22.2</td><td>Key Concepts</td><td>942</td></tr><tr><td>22.3</td><td>Deep Q-Learning</td><td>944</td></tr><tr><td>22.3.1</td><td>Literature Review of Deep Q-Learning</td><td>944</td></tr><tr><td>22.3.2</td><td>Analysis of Deep Q-Learning</td><td>945</td></tr><tr><td>22.3.3</td><td>Analysis of Double Q-Learning</td><td>947</td></tr><tr><td>22.3.4</td><td>Analysis of Dueling Q-Learning</td><td>948</td></tr><tr><td>22.3.5</td><td>Analysis of Prioritized Experience Replay</td><td>950</td></tr><tr><td>22.3.6</td><td>Analysis of Rainbow Deep Q-Network</td><td>951</td></tr><tr><td>22.4</td><td>Tabular Q-Learning</td><td>954</td></tr><tr><td>22.5</td><td>SARSA (State-Action-Reward-State-Action)</td><td>955</td></tr><tr><td>22.6</td><td>Expected SARSA (State-Action-Reward-State-Action)</td><td>957</td></tr><tr><td>22.7</td><td>Applications in Games and Robotics</td><td>958</td></tr><tr><td>22.7.1</td><td>Literature Review of Applications in Games and Robotics</td><td>958</td></tr><tr><td>22.7.2</td><td>Analysis of Applications in Games and Robotics</td><td>959</td></tr></table>

## 23 Federated Learning 962

<table><tr><td>23.1</td><td>Literature Review of Federated Learning</td><td>964</td></tr><tr><td>23.2</td><td>Recent Literature Review of Federated Learning</td><td>965</td></tr><tr><td>23.3</td><td>Formal Definition of Federated Learning</td><td>966</td></tr><tr><td>23.4</td><td>Federated Learning and Distributed Optimization Framework</td><td>967</td></tr><tr><td>23.4.1</td><td>Clients</td><td>967</td></tr><tr><td>23.4.2</td><td>Server</td><td>968</td></tr><tr><td>23.4.3</td><td>Local Updates</td><td>969</td></tr><tr><td>23.4.4</td><td>Model Aggregation</td><td>970</td></tr><tr><td>23.5</td><td>Detailed Steps in a Communication Round</td><td>974</td></tr><tr><td>23.5.1</td><td>Challenges in Communication Rounds</td><td>974</td></tr><tr><td>23.5.1.1</td><td>Heterogeneous Client Participation</td><td>975</td></tr><tr><td>23.5.1.2</td><td>Communication Bottlenecks</td><td>978</td></tr><tr><td>23.5.1.3</td><td>Non-IID Data</td><td>980</td></tr><tr><td>23.5.1.4</td><td>Python Code to generate Figures 23.2, 23.3, and 23.4 illustrating Heterogeneous Client Participation in federated learning</td><td>983</td></tr><tr><td>23.5.1.5</td><td>Python Code to generate Figures 23.5 and 23.6 illustrating Communication Bottlenecks and Cumulative communication over rounds in federated learning</td><td>984</td></tr><tr><td>23.5.1.6</td><td>Python Code to generate Figure 23.7 illustrating Client drift in federated learning, measured as L2 distance from the global model</td><td>985</td></tr><tr><td>23.5.1.7</td><td>Python Code to generate Figure 23.8 illustrating Comparative client drift under different heterogeneity scenarios in federated learning</td><td>986</td></tr><tr><td>23.5.1.8</td><td>Python Code to generate Figure 23.9 illustrating Effect of Client Drift on Global Model Accuracy in Federated Learning</td><td>987</td></tr></table>
<table><tbody><tr><td>23.5.2</td><td>Advanced Techniques for Communication Rounds</td><td>988</td></tr><tr><td>23.5.2.1</td><td>Adaptive Client Selection</td><td>989</td></tr><tr><td>23.5.2.2</td><td>Local Step Adaptation</td><td>991</td></tr><tr><td>23.5.2.3</td><td>Secure Aggregation</td><td>993</td></tr><tr><td>23.5.2.4</td><td>Python Code to generate Figures 23.10 and 23.13 illustrating Adaptive Client Selection in federated learning</td><td>995</td></tr><tr><td>23.5.2.5</td><td>Python Code to generate Figures 23.12 and 23.13 illustrating Local Step Adaptation in federated learning</td><td>997</td></tr><tr><td>23.5.2.6</td><td>Python Code to generate Figures 23.14, 23.15, and 23.16 illustrating Secure Aggregation in federated learning</td><td>998</td></tr><tr><td>23.6</td><td>Theoretical Foundations</td><td>999</td></tr><tr><td>23.6.1</td><td>Smoothness and Convexity</td><td>999</td></tr><tr><td>23.6.2</td><td>Bounded Variance</td><td>1002</td></tr><tr><td>23.6.2.1</td><td>Python Code to generate Figure 23.17 illustrating the stochastic gradient bounded variance in federated learning</td><td>1004</td></tr><tr><td>23.6.3</td><td>Heterogeneity</td><td>1006</td></tr><tr><td>23.6.3.1</td><td>Python Code to generate Figures 23.18 and 23.19 illustrating the System heterogeneity in federated learning</td><td>1008</td></tr><tr><td>23.7</td><td>Convergence Analysis</td><td>1009</td></tr><tr><td>23.7.1</td><td>Convergence Rate</td><td>1009</td></tr><tr><td>23.7.1.1</td><td>Python Code to generate Figure 23.21 illustrating the Convergence rate in federated learning under different conditions</td><td>1011</td></tr><tr><td>23.7.1.2</td><td>Python Code to generate Figure 23.21 illustrating the Convergence rate in federated learning (log-log scale)</td><td>1012</td></tr><tr><td>23.7.2</td><td>Communication Complexity</td><td>1013</td></tr><tr><td>23.8</td><td>Advanced Techniques</td><td>1014</td></tr><tr><td>23.8.1</td><td>Adaptive Optimization</td><td>1014</td></tr><tr><td>23.8.2</td><td>Differential Privacy</td><td>1016</td></tr><tr><td>23.8.3</td><td>Sparse Updates</td><td>1018</td></tr><tr><td>23.9</td><td>Statistical Learning Perspective</td><td>1020</td></tr><tr><td>23.10</td><td>Open Problems and Future Directions</td><td>1021</td></tr><tr><td>23.11</td><td>Conclusion</td><td>1021</td></tr></tbody></table>

## 24 Diffusion Models and Score-Based Generative Models 1022

<table><tbody><tr><td>24.1</td><td>Literature Review of Diffusion Models and Score-Based Generative Models</td><td>1023</td></tr><tr><td>24.2</td><td>Analysis of Diffusion Models and Score-Based Generative Models</td><td>1024</td></tr><tr><td>24.3</td><td>Key conceptual components of diffusion models and score-based generative models</td><td>1026</td></tr><tr><td>24.3.1</td><td>Forward Diffusion Process</td><td>1026</td></tr><tr><td>24.3.2</td><td>Reverse Diffusion Process</td><td>1028</td></tr><tr><td>24.3.3</td><td>Probability Flow ODE</td><td>1030</td></tr><tr><td>24.3.4</td><td>Training Objective</td><td>1033</td></tr><tr><td>24.3.5</td><td>Sampling</td><td>1035</td></tr><tr><td>24.3.6</td><td>Score-Based Generative Models</td><td>1037</td></tr><tr><td>24.3.7</td><td>Langevin Dynamics for Sampling</td><td>1040</td></tr><tr><td>24.3.8</td><td>Connection to Diffusion Models</td><td>1043</td></tr><tr><td>24.3.9</td><td>Likelihood Computation</td><td>1045</td></tr><tr><td>24.3.10</td><td>Conclusion</td><td>1048</td></tr><tr><td>24.4</td><td>Stable Diffusion</td><td>1049</td></tr><tr><td>24.4.1</td><td>Literature Review of Stable Diffusion</td><td>1049</td></tr><tr><td>24.4.2</td><td>Analysis of Stable Diffusion</td><td>1050</td></tr><tr><td>24.4.3</td><td>Latent Variable Model and Diffusion Process</td><td>1052</td></tr><tr><td>24.4.4</td><td>Reverse Diffusion and Denoising</td><td>1053</td></tr></tbody></table>
<table><tr><td>24.4.5 Training Objective</td><td>1054</td></tr><tr><td>24.4.6 Architecture: U-Net with Cross-Attention</td><td>1055</td></tr><tr><td>24.4.7 Latent Space and Autoencoder</td><td>1056</td></tr><tr><td>24.4.8 Sampling Process</td><td>1057</td></tr><tr><td>24.4.9 Classifier-Free Guidance</td><td>1058</td></tr></table>

## 25 Kernel Regression 1059

<table><tr><td>25.1 Literature Review</td><td>1060</td></tr><tr><td>25.2 Analysis of Kernel regression</td><td>1060</td></tr><tr><td>25.3 Nadaraya-Watson kernel estimator</td><td>1061</td></tr><tr><td>25.3.1 Literature Review</td><td>1061</td></tr><tr><td>25.3.2 Analysis of Nadaraya-Watson kernel estimator</td><td>1062</td></tr><tr><td>25.4 Priestley-Chao kernel estimator</td><td>1065</td></tr><tr><td>25.4.1 Literature Review</td><td>1065</td></tr><tr><td>25.4.2 Analysis of Priestley-Chao kernel estimator</td><td>1065</td></tr><tr><td>25.5 Gasser-Müller kernel estimator</td><td>1067</td></tr><tr><td>25.5.1 Literature Review</td><td>1067</td></tr><tr><td>25.5.2 Analysis of Gasser-Müller kernel estimator</td><td>1068</td></tr><tr><td>25.6 Parzen-Rosenblatt method</td><td>1070</td></tr><tr><td>25.6.1 Literature Review</td><td>1070</td></tr><tr><td>25.6.2 Analysis of Parzen-Rosenblatt method</td><td>1070</td></tr></table>

## 26 Natural Language Processing (NLP) 1073

<table><tr><td>26.1 Literature Review</td><td>1074</td></tr><tr><td>26.2 Text Classification</td><td>1075</td></tr><tr><td>26.2.1 Literature Review of Text classification</td><td>1075</td></tr><tr><td>26.2.2 Analysis of Text classification</td><td>1075</td></tr><tr><td>26.3 Machine Translation</td><td>1077</td></tr><tr><td>26.3.1 Literature Review of Machine Translation</td><td>1077</td></tr><tr><td>26.3.2 Analysis of Machine Translation</td><td>1078</td></tr><tr><td>26.4 Chatbots and Conversational AI</td><td>1080</td></tr><tr><td>26.4.1 Literature Review of Chatbots and Conversational AI</td><td>1081</td></tr><tr><td>26.4.2 Analysis of Chatbots and Conversational AI</td><td>1081</td></tr><tr><td>26.5 Representation Learning and Optimization</td><td>1083</td></tr><tr><td>26.6 Structured Prediction and Decoding</td><td>1084</td></tr></table>

## 27 Deep Learning Frameworks 1086

<table><tr><td>27.1 TensorFlow</td><td>1087</td></tr><tr><td>27.1.1 Literature Review of TensorFlow</td><td>1087</td></tr><tr><td>27.1.2 Analysis of TensorFlow</td><td>1087</td></tr><tr><td>27.2 PyTorch</td><td>1090</td></tr><tr><td>27.2.1 Literature Review of PyTorch</td><td>1090</td></tr><tr><td>27.2.2 Analysis of PyTorch</td><td>1090</td></tr><tr><td>27.3 JAX</td><td>1092</td></tr><tr><td>27.3.1 Literature Review of JAX</td><td>1092</td></tr><tr><td>27.3.2 Analysis of JAX</td><td>1093</td></tr></table>

## 28 Appendix 1096

<table><tr><td>28.1 Linear Algebra Essentials</td><td>1096</td></tr><tr><td>28.1.1 Matrices and Vector Spaces</td><td>1096</td></tr><tr><td>28.1.2 Vector Spaces and Linear Transformations</td><td>1097</td></tr><tr><td>28.1.3 Eigenvalues and Eigenvectors</td><td>1097</td></tr></table>
<table><tr><td>28.1.4 Singular Value Decomposition (SVD)</td><td>1097</td></tr><tr><td>28.2 Probability and Statistics</td><td>1098</td></tr><tr><td>28.2.1 Probability Distributions</td><td>1098</td></tr><tr><td>28.2.2 Bayes' Theorem</td><td>1099</td></tr><tr><td>28.2.3 Statistical Measures</td><td>1099</td></tr><tr><td>28.3 Optimization Techniques</td><td>1102</td></tr><tr><td>28.3.1 Gradient Descent (GD)</td><td>1102</td></tr><tr><td>28.3.2 Stochastic Gradient Descent (SGD)</td><td>1103</td></tr><tr><td>28.3.3 Second-Order Methods</td><td>1103</td></tr><tr><td>28.4 Matrix Calculus</td><td>1104</td></tr><tr><td>28.4.1 Matrix Differentiation</td><td>1104</td></tr><tr><td>28.4.2 Tensor Differentiation</td><td>1105</td></tr><tr><td>28.5 Information Theory</td><td>1107</td></tr><tr><td>28.5.1 Entropy: The Fundamental Measure of Uncertainty</td><td>1107</td></tr><tr><td>28.5.2 Source Coding Theorem: Fundamental Limits of Compression</td><td>1110</td></tr><tr><td>28.5.3 Noisy Channel Coding Theorem: Fundamental Limits of Communication</td><td>1112</td></tr><tr><td>28.5.4 Rate-Distortion Theory: Lossy Data Compression</td><td>1114</td></tr><tr><td>28.5.5 Applications of Information Theory</td><td>1116</td></tr><tr><td>28.5.6 Conclusion: Information Theory as a Universal Mathematical Principle</td><td>1122</td></tr><tr><td>29 Acknowledgments</td><td>1123</td></tr></table>
List of Tables

<table><tr><td>8.1 Summary of Contributions to Open Set Learning (OSL)</td><td>247</td></tr><tr><td>8.2 Summary of Additional Contributions to Open Set Learning (OSL)</td><td>248</td></tr><tr><td>8.3 Summary of Additional Contributions to Open Set Learning (OSL)</td><td>250</td></tr><tr><td>8.4 Summary of Contributions to Open Set Learning (OSL)</td><td>252</td></tr><tr><td>8.5 Summary of Key Contributions to Open Set Recognition (OSR)</td><td>253</td></tr><tr><td>8.6 Summary of Key Contributions to Open Set Recognition (OSR)</td><td>255</td></tr><tr><td>8.7 Summary of Key Contributions to Open Set Recognition (OSR)</td><td>257</td></tr><tr><td>8.8 Summary of Key Contributions in Open Set Recognition and Related Tasks</td><td>259</td></tr><tr><td>8.9 Summary of Key Contributions in Open Set Learning Literature</td><td>260</td></tr><tr><td>16.1 Summary of Contributions to the Unsupervised Learning</td><td>510</td></tr><tr><td>16.2 Summary of Recent Contributions in Unsupervised Learning</td><td>512</td></tr><tr><td>16.4 Summary of Recent Contributions in Information Bottleneck Research</td><td>518</td></tr><tr><td>16.5 Summary of Contributions on Restricted Boltzmann Machines (RBMs)</td><td>523</td></tr><tr><td>16.7 Summary of Contributions on Deep Belief Networks (DBNs)</td><td>529</td></tr><tr><td>16.8 Summary of Contributions</td><td>531</td></tr><tr><td>16.9 Summary of Contributions in t-SNE Literature Review</td><td>536</td></tr><tr><td>16.10 Summary of Recent Contributions in t-SNE Literature Review</td><td>538</td></tr><tr><td>16.11 Summary of Contributions in Locally Linear Embedding (LLE)</td><td>543</td></tr><tr><td>16.12 Summary of Recent Contributions in Locally Linear Embedding (LLE)</td><td>545</td></tr><tr><td>16.13 Summary of Contributions in Independent Component Analysis (ICA) Literature</td><td>549</td></tr><tr><td>16.14 Summary of Recent Contributions in Independent Component Analysis (ICA) Literature</td><td>552</td></tr><tr><td>16.15 Summary of Contributions in Supervised Learning</td><td>557</td></tr><tr><td>16.16 Summary of Recent Contributions in Supervised Learning</td><td>559</td></tr><tr><td>16.17 Summary of Contributions to the Bias-Variance Tradeoff in Supervised Learning</td><td>564</td></tr><tr><td>16.19 Summary of Contributions in Support Vector Machines</td><td>571</td></tr><tr><td>16.20 Summary of Recent Contributions in Support Vector Machines</td><td>573</td></tr><tr><td>16.21 Summary of Contributions to Linear Regression</td><td>579</td></tr><tr><td>16.22 Summary of Recent Contributions on Linear Regression</td><td>581</td></tr><tr><td>16.23 Summary of Contributions in Logistic Regression</td><td>586</td></tr><tr><td>16.24 Summary of Recent Contributions in Logistic Regression</td><td>588</td></tr><tr><td>16.25 Summary of Contributions to Linear Discriminant Analysis (LDA)</td><td>591</td></tr><tr><td>16.26 Summary of Recent Contributions to Linear Discriminant Analysis (LDA)</td><td>592</td></tr><tr><td>16.27 Summary of Contributions to Naïve Bayes Classifier</td><td>624</td></tr><tr><td>16.28 Summary of Recent Contributions on Naïve Bayes Classifier</td><td>626</td></tr><tr><td>16.29 Summary of Contributions to Decision tree learning</td><td>632</td></tr><tr><td>16.30 Summary of Recent Contributions in Decision Tree Learning</td><td>634</td></tr><tr><td>16.31 Summary of Contributions to k-nearest neighbors algorithm</td><td>638</td></tr><tr><td>16.32 Summary of Recent Contributions to k-nearest neighbors algorithm</td><td>639</td></tr><tr><td>16.33 Summary of Contributions in Similarity Training</td><td>643</td></tr><tr><td>16.34 Summary of Recent Contributions in Similarity Training</td><td>644</td></tr></table>
<table><tr><td>16.35Summary of Contributions in Self-Learning</td><td>649</td></tr><tr><td>16.36Summary of Recent Contributions in Self-Learning</td><td>650</td></tr><tr><td>16.37Summary of Key Contributions in Reinforcement Learning</td><td>655</td></tr><tr><td>16.38Summary of Key Contributions in Recent Reinforcement Learning</td><td>657</td></tr><tr><td>16.39Summary of Key Contributions in Policy Gradient Methods</td><td>663</td></tr><tr><td>16.41Summary of Contributions in Deep Reinforcement Learning</td><td>670</td></tr><tr><td>16.42Summary of Recent Contributions in Deep Reinforcement Learning Research</td><td>672</td></tr><tr><td>16.43Summary of Contributions in A2C Research</td><td>678</td></tr><tr><td>16.44Summary of Recent Contributions in A2C Research</td><td>680</td></tr><tr><td>16.45Summary of Contributions in Deep Deterministic Policy Gradient (DDPG) Research</td><td>686</td></tr><tr><td>16.51Summary of Contributions in Neuro-Genetic Evolution Research</td><td>712</td></tr><tr><td>16.52Summary of Recent Contributions in Neuro-Genetic Evolution Research</td><td>714</td></tr><tr><td>16.53Summary of Contributions in Cellular Encoding Research</td><td>719</td></tr><tr><td>16.54Summary of Recent Contributions in Cellular Encoding Research</td><td>721</td></tr><tr><td>16.55Summary of Contributions to GNARL</td><td>728</td></tr></table>
List of Figures

<table>
  <tbody>
    <tr>
      <td>1.1 Measurable Function Spaces</td>
      <td>2</td>
    </tr>
    <tr>
      <td>1.2 VC growth function m<sub>H</sub>(n) (Linear scale)</td>
      <td>9</td>
    </tr>
    <tr>
      <td>1.3 VC growth function m<sub>H</sub>(n) (Log scale)</td>
      <td>10</td>
    </tr>
    <tr>
      <td>1.4 Rademacher Complexity vs. 1/√n Bound</td>
      <td>15</td>
    </tr>
    <tr>
      <td>1.5 Rademacher Complexity: Linear vs Quadratic Function Classes</td>
      <td>16</td>
    </tr>
    <tr>
      <td>1.6 Sobolev conjugate q* = np/(n-kp) (k = 1)</td>
      <td>21</td>
    </tr>
    <tr>
      <td>1.7 Sobolev conjugate q* = np/(n-kp) (k = 2)</td>
      <td>21</td>
    </tr>
    <tr>
      <td>1.8 Hölder exponent for embedding W<sup>1</sup>(Ω) ↪→ C<sup>0,α</sup></td>
      <td>22</td>
    </tr>
    <tr>
      <td>1.9 Hölder exponent for embedding W<sup>2</sup>(Ω) ↪→ C<sup>0,α</sup></td>
      <td>23</td>
    </tr>
    <tr>
      <td>1.10 Rellich-Kondrachov Compactness Theorem Illustration</td>
      <td>28</td>
    </tr>
    <tr>
      <td>1.11 L<sup>2</sup> norm of u<sub>n</sub> converging to 0 (Strong convergence in L<sup>2</sup>)</td>
      <td>29</td>
    </tr>
    <tr>
      <td>1.12 Fréchet-Kolmogorov Criterion: Translation Equicontinuity</td>
      <td>32</td>
    </tr>
    <tr>
      <td>1.13 Test Functions in H<sup>1</sup><sub>0</sub>(0,1)</td>
      <td>37</td>
    </tr>
    <tr>
      <td>1.14 Sobolev-Poincaré Inequality Ratios: ||u||<sub>L<sup>2</sup></sub>/||u'||<sub>L<sup>2</sup></sub></td>
      <td>37</td>
    </tr>
    <tr>
      <td>2.1 Universal Approximation Theorem Illustration</td>
      <td>42</td>
    </tr>
    <tr>
      <td>2.2 Approximation Error vs Hidden Layer Size</td>
      <td>43</td>
    </tr>
    <tr>
      <td>2.3 Stone-Weierstrass (Weierstrass) demo: Bernstein polynomial approximations</td>
      <td>47</td>
    </tr>
    <tr>
      <td>2.4 Zoom near the kink (x = 0.3) — convergence is uniform</td>
      <td>47</td>
    </tr>
    <tr>
      <td>2.5 Sup-norm error ||f - B<sub>n</sub>(f)||<sub>∞</sub> (log-log scale)</td>
      <td>48</td>
    </tr>
    <tr>
      <td>2.6 Kolmogorov-Arnold Superposition Theorem (Toy Example)</td>
      <td>54</td>
    </tr>
    <tr>
      <td>2.7 Target vs NN approximation (real space)</td>
      <td>57</td>
    </tr>
    <tr>
      <td>2.8 Fourier Spectrum: Expressivity Analysis</td>
      <td>58</td>
    </tr>
    <tr>
      <td>2.9 Target vs Approximations (Real Space)</td>
      <td>59</td>
    </tr>
    <tr>
      <td>2.10 Fourier Spectrum: Expressivity of Architectures</td>
      <td>60</td>
    </tr>
    <tr>
      <td>2.11 Sigmoid Activation Function: σ(x) = 1/(1+e<sup>-x</sup>)</td>
      <td>65</td>
    </tr>
    <tr>
      <td>2.12 Hyperbolic Tangent Activation Function: σ(x) = (e<sup>x</sup>-e<sup>-x</sup>) / (e<sup>x</sup>+e<sup>-x</sup>)</td>
      <td>66</td>
    </tr>
    <tr>
      <td>2.13 ReLU Activation Function: σ(x) = max(0, x)</td>
      <td>67</td>
    </tr>
    <tr>
      <td>2.14 Leaky ReLU Activation Function</td>
      <td>68</td>
    </tr>
    <tr>
      <td>2.15 Sinusoidal Activation Function σ(x) = sin(x)</td>
      <td>69</td>
    </tr>
    <tr>
      <td>3.1 Training Dynamics vs NTK Linearization</td>
      <td>72</td>
    </tr>
    <tr>
      <td>3.2 Gradient Flow and Stationary Points</td>
      <td>77</td>
    </tr>
    <tr>
      <td>3.3 Hessian Structure (NTK Regime) &amp; NTK Matrix</td>
      <td>79</td>
    </tr>
    <tr>
      <td>3.4 Hessian Spectrum (NTK Regime)</td>
      <td>80</td>
    </tr>
    <tr>
      <td>3.5 Final Fit &amp; Training Dynamics (NN vs NTK)</td>
      <td>82</td>
    </tr>
    <tr>
      <td>3.6 Final Predictions in NTK Regime &amp; Training Dynamics (NN vs NTK)</td>
      <td>89</td>
    </tr>
    <tr>
      <td>4.1 PAC-Bayes Bound vs. Sample Size (KL(Q||P) = 2.0)</td>
      <td>95</td>
    </tr>
    <tr>
      <td>4.2 PAC-Bayes Bound vs. Sample Size (KL(Q||P) = 0.5)</td>
      <td>96</td>
    </tr>
    <tr>
      <td>4.3 PAC-Bayes Bound vs. Sample Size (KL(Q||P) = 5.0)</td>
      <td>97</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>4.4</td>
      <td>KL Divergence between Bernoulli(p) and Bernoulli(q)</td>
      <td>99</td>
    </tr>
    <tr>
      <td>4.5</td>
      <td>KL Divergence as function of q for fixed p</td>
      <td>100</td>
    </tr>
    <tr>
      <td>4.6</td>
      <td>Gaussian KL Divergence (equal variances)</td>
      <td>101</td>
    </tr>
    <tr>
      <td>4.7</td>
      <td>Gaussian KL Divergence vs σ<sub>p</sub> (μ<sub>q</sub> = μ<sub>p</sub> = 0)</td>
      <td>102</td>
    </tr>
    <tr>
      <td>4.8</td>
      <td>Rényi Divergence between Gaussians vs. α</td>
      <td>106</td>
    </tr>
    <tr>
      <td>4.9</td>
      <td>Rényi Divergence Heatmap (α = 2.0, σ<sub>p</sub> = σ<sub>q</sub> = 1)</td>
      <td>107</td>
    </tr>
    <tr>
      <td>4.10</td>
      <td>Wasserstein Distance vs Mean Difference (σ<sub>p</sub> = σ<sub>q</sub> = 1)</td>
      <td>110</td>
    </tr>
    <tr>
      <td>4.11</td>
      <td>Wasserstein Distance vs Variance (μ<sub>p</sub> = μ<sub>q</sub> = 0, σ<sub>p</sub> = σ<sub>q</sub> = 1)</td>
      <td>111</td>
    </tr>
    <tr>
      <td>4.12</td>
      <td>Wasserstein Distance Heatmap (σ<sub>p</sub> = 1.0, σ<sub>q</sub> = 1.5)</td>
      <td>112</td>
    </tr>
    <tr>
      <td>4.13</td>
      <td>Effect of Spectral Regularization on Singular Values</td>
      <td>116</td>
    </tr>
    <tr>
      <td>4.14</td>
      <td>Original Weight Matrix &amp; After Spectral Regularization</td>
      <td>116</td>
    </tr>
    <tr>
      <td>4.15</td>
      <td>Singular Value Spectrum of First Layer (MNIST)</td>
      <td>117</td>
    </tr>
    <tr>
      <td>5.1</td>
      <td>Min-Max Game Formulation (Adversarial Training)</td>
      <td>124</td>
    </tr>
    <tr>
      <td>5.2</td>
      <td>Regularization as Game Formulation</td>
      <td>125</td>
    </tr>
    <tr>
      <td>5.3</td>
      <td>Multi-Agent Game: Layers as Players</td>
      <td>126</td>
    </tr>
    <tr>
      <td>5.4</td>
      <td>Min-Max (Saddle) dynamics: L(x, y) = x * y</td>
      <td>129</td>
    </tr>
    <tr>
      <td>5.5</td>
      <td>Potential / Cooperative game (gradient descent on shared Phi)</td>
      <td>131</td>
    </tr>
    <tr>
      <td>5.6</td>
      <td>Replicator dynamics (Rock-Paper-Scissors) — cyclic behavior</td>
      <td>133</td>
    </tr>
    <tr>
      <td>5.7</td>
      <td>Replicator Dynamics as Gradient Descent</td>
      <td>139</td>
    </tr>
    <tr>
      <td>5.8</td>
      <td>Min-Max Dynamics (Adversarial Training)</td>
      <td>140</td>
    </tr>
    <tr>
      <td>5.9</td>
      <td>Cooperative Game (Shared Potential)</td>
      <td>141</td>
    </tr>
    <tr>
      <td>5.10</td>
      <td>Monotone/Potential F=grad Phi</td>
      <td>145</td>
    </tr>
    <tr>
      <td>5.11</td>
      <td>Monotone case: Approx VI gap vs iteration</td>
      <td>145</td>
    </tr>
    <tr>
      <td>5.12</td>
      <td>Saddle Bilinear F (min-max)</td>
      <td>147</td>
    </tr>
    <tr>
      <td>5.13</td>
      <td>Skew (rotation) F: R z</td>
      <td>150</td>
    </tr>
    <tr>
      <td>5.14</td>
      <td>Skew + Dissipation F: R + alpha I</td>
      <td>151</td>
    </tr>
    <tr>
      <td>10.1</td>
      <td>Mean Squared Error (MSE/L2) Loss Function</td>
      <td>308</td>
    </tr>
    <tr>
      <td>10.2</td>
      <td>Mean Absolute Error (MAE/L1) Loss Function</td>
      <td>310</td>
    </tr>
    <tr>
      <td>10.3</td>
      <td>Huber Loss Function</td>
      <td>311</td>
    </tr>
    <tr>
      <td>10.4</td>
      <td>Comparison of Loss Functions: Huber, MSE, MAE</td>
      <td>313</td>
    </tr>
    <tr>
      <td>10.5</td>
      <td>Binary Cross-Entropy (BCE) Loss Function</td>
      <td>317</td>
    </tr>
    <tr>
      <td>10.6</td>
      <td>Binary Cross-Entropy Loss Surface</td>
      <td>318</td>
    </tr>
    <tr>
      <td>10.7</td>
      <td>Categorical Cross-Entropy Loss</td>
      <td>320</td>
    </tr>
    <tr>
      <td>10.8</td>
      <td>Categorical Cross-Entropy Loss</td>
      <td>321</td>
    </tr>
    <tr>
      <td>10.9</td>
      <td>Sparse Categorical Cross-Entropy Loss</td>
      <td>323</td>
    </tr>
    <tr>
      <td>10.10</td>
      <td>Surface Plot of Sparse Categorical Cross-Entropy Loss</td>
      <td>323</td>
    </tr>
    <tr>
      <td>10.11</td>
      <td>Plot of KL Divergence</td>
      <td>325</td>
    </tr>
    <tr>
      <td>10.12</td>
      <td>2D Surface Plot of KL Divergence</td>
      <td>326</td>
    </tr>
    <tr>
      <td>10.13</td>
      <td>1D Wasserstein Distance of two distributions having parameters: Normal(μ = -1, σ = 1) and Normal(μ = 1.5, σ = 1)</td>
      <td>333</td>
    </tr>
    <tr>
      <td>10.14</td>
      <td>Multidimensional Wasserstein Distance of two distributions having parameters, Distribution A: N(μ = [0,0], Σ = I), Distribution B: N(μ = [3,3], Σ = I)</td>
      <td>334</td>
    </tr>
    <tr>
      <td>10.15</td>
      <td>Comparison: Equal CDF, WGAN Critic, GAN Discriminator</td>
      <td>335</td>
    </tr>
    <tr>
      <td>10.16</td>
      <td>Least Squares GAN (LSGAN) Loss Functions</td>
      <td>337</td>
    </tr>
    <tr>
      <td>10.17</td>
      <td>Standard GAN (Adversarial) Loss Functions</td>
      <td>339</td>
    </tr>
    <tr>
      <td>10.18</td>
      <td>Contrastive Loss Functions</td>
      <td>340</td>
    </tr>
    <tr>
      <td>10.19</td>
      <td>Contrastive Loss Landscapes</td>
      <td>341</td>
    </tr>
    <tr>
      <td>10.20</td>
      <td>3D Contrastive Loss Landscapes</td>
      <td>342</td>
    </tr>
    <tr>
      <td>10.21</td>
      <td>Triplet Loss Function</td>
      <td>342</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>10.22Triplet Loss Landscape</td>
      <td>343</td>
    </tr>
    <tr>
      <td>10.23Triplet Loss Contour Plot</td>
      <td>344</td>
    </tr>
    <tr>
      <td>10.24Center Loss Function</td>
      <td>345</td>
    </tr>
    <tr>
      <td>10.25Center Loss Visualization: Features vs Class Centers</td>
      <td>346</td>
    </tr>
    <tr>
      <td>10.263D Center Loss Surface</td>
      <td>347</td>
    </tr>
    <tr>
      <td>10.27Perceptual Loss Function</td>
      <td>348</td>
    </tr>
    <tr>
      <td>10.28Pixel-wise MSE vs Perceptual Loss (Nonlinear Features)</td>
      <td>349</td>
    </tr>
    <tr>
      <td>10.29Style Loss Function in Neural Style Transfer</td>
      <td>350</td>
    </tr>
    <tr>
      <td>10.30Style Loss vs Scaling of Generated Features</td>
      <td>351</td>
    </tr>
    <tr>
      <td>10.31Total Variation (TV) Loss</td>
      <td>351</td>
    </tr>
    <tr>
      <td>10.32Total Variation Loss vs Noise Level</td>
      <td>352</td>
    </tr>
    <tr>
      <td>10.33Evidence Lower Bound (ELBO) Loss during Training</td>
      <td>354</td>
    </tr>
    <tr>
      <td>10.34Negative Log-Likelihood (NLL) Loss during Training</td>
      <td>355</td>
    </tr>
    <tr>
      <td>10.35Domain Adversarial Loss during Training</td>
      <td>356</td>
    </tr>
    <tr>
      <td>10.36Maximum Mean Discrepancy (MMD) Loss during Training</td>
      <td>358</td>
    </tr>
    <tr>
      <td>10.37IoU Loss and Variants (GIoU Loss, DIoU Loss, CIoU Loss)</td>
      <td>362</td>
    </tr>
    <tr>
      <td>10.38Dice Loss during Training</td>
      <td>363</td>
    </tr>
    <tr>
      <td>10.39Focal Loss vs Predicted Probability</td>
      <td>365</td>
    </tr>
    <tr>
      <td>10.40Focal Loss vs Standard Binary Cross-Entropy</td>
      <td>365</td>
    </tr>
    <tr>
      <td>10.41Hinge Loss vs Prediction Score</td>
      <td>366</td>
    </tr>
    <tr>
      <td>10.42Comparison of the Standard Hinge Loss and the Squared Hinge Loss</td>
      <td>367</td>
    </tr>
    <tr>
      <td>10.43Value Loss (MSE) for Different Targets</td>
      <td>369</td>
    </tr>
    <tr>
      <td>10.44Value Loss (MSE) vs Huber Loss</td>
      <td>370</td>
    </tr>
    <tr>
      <td>10.45Policy Loss vs Action Probability</td>
      <td>371</td>
    </tr>
    <tr>
      <td>10.46PPO Clipped Surrogate Objective</td>
      <td>372</td>
    </tr>
    <tr>
      <td>10.47PPO Loss Comparison</td>
      <td>373</td>
    </tr>
    <tr>
      <td>10.48PPO Clipped Surrogate Loss with Entropy Regularization</td>
      <td>374</td>
    </tr>
    <tr>
      <td>10.49Comparison: PPO Unclipped, PPO Clipped, PPO + Entropy</td>
      <td>375</td>
    </tr>
    <tr>
      <td>10.50Soft Actor-Critic (SAC) Policy Loss with Entropy Regularization</td>
      <td>375</td>
    </tr>
    <tr>
      <td>10.51KL Divergence Loss as Sparsity Constraint</td>
      <td>377</td>
    </tr>
    <tr>
      <td>10.52Total KL Sparsity Loss for 10 Hidden Units (ρ = 0.05)</td>
      <td>378</td>
    </tr>
    <tr>
      <td>10.53Total KL Sparsity Loss for 10 Hidden Units with Random Activations (ρ = 0.05)</td>
      <td>379</td>
    </tr>
    <tr>
      <td>10.54Structured SVM (SSVM) Hinge Loss</td>
      <td>380</td>
    </tr>
    <tr>
      <td>10.55Structured SVM Loss Surface (Δ = 1.0)</td>
      <td>381</td>
    </tr>
    <tr>
      <td>10.56Structured SVM Loss Contour (Δ = 1.0)</td>
      <td>382</td>
    </tr>
    <tr>
      <td>10.57Simplified CRF Loss vs Correct Label Score</td>
      <td>383</td>
    </tr>
    <tr>
      <td>10.58CRF Loss Surface (Single Timestep, 2 Labels)</td>
      <td>384</td>
    </tr>
    <tr>
      <td>10.59CRF Loss Contour (Single Timestep, 2 Labels)</td>
      <td>385</td>
    </tr>
    <tr>
      <td>10.60CTC Loss vs Probability of Correct Sequence</td>
      <td>386</td>
    </tr>
    <tr>
      <td>10.61Approximate CTC Loss vs Per-Timestep Probability for Different Sequence Lengths</td>
      <td>387</td>
    </tr>
    <tr>
      <td>10.62CTC Loss Surface vs Sequence Length and Per-Timestep Probability</td>
      <td>388</td>
    </tr>
    <tr>
      <td>10.63CTC Loss Contour vs Sequence Length and Per-Timestep Probability</td>
      <td>389</td>
    </tr>
    <tr>
      <td>10.64Maximum Margin Markov Networks (M<sup>3</sup>N) Loss</td>
      <td>389</td>
    </tr>
    <tr>
      <td>10.65M<sup>3</sup>N Loss Surface (Δ = 1.0)</td>
      <td>390</td>
    </tr>
    <tr>
      <td>10.66Contour Plot of M<sup>3</sup>N Loss (Δ = 1.0)</td>
      <td>391</td>
    </tr>
    <tr>
      <td>10.67Listwise Ranking Losses (ListNet vs ListMLE)</td>
      <td>393</td>
    </tr>
    <tr>
      <td>10.68Listwise Ranking Losses (3D Surfaces)</td>
      <td>394</td>
    </tr>
    <tr>
      <td>10.69Listwise Ranking Loss Contours</td>
      <td>394</td>
    </tr>
    <tr>
      <td>10.70Pairwise Ranking Losses</td>
      <td>395</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>10.71</td>
      <td>Pairwise Ranking Loss Landscapes</td>
      <td>396</td>
    </tr>
    <tr>
      <td>10.72</td>
      <td>3D Surfaces of Pairwise Ranking Losses</td>
      <td>396</td>
    </tr>
    <tr>
      <td>10.73</td>
      <td>Contrastive Divergence (CD) Loss vs Weight</td>
      <td>397</td>
    </tr>
    <tr>
      <td>10.74</td>
      <td>Contrastive Divergence (CD) Loss Landscape</td>
      <td>399</td>
    </tr>
    <tr>
      <td>10.75</td>
      <td>Contrastive Divergence (CD) Loss Landscape (3D)</td>
      <td>400</td>
    </tr>
    <tr>
      <td>10.76</td>
      <td>Persistent Contrastive Divergence (PCD) Loss Landscape</td>
      <td>401</td>
    </tr>
    <tr>
      <td>10.77</td>
      <td>Persistent Contrastive Divergence (PCD) Loss Landscape (3D)</td>
      <td>402</td>
    </tr>
    <tr>
      <td>10.78</td>
      <td>Hausdorff Distance Loss Visualization</td>
      <td>404</td>
    </tr>
    <tr>
      <td>10.79</td>
      <td>Hausdorff Distance Heatmap</td>
      <td>405</td>
    </tr>
    <tr>
      <td>15.1</td>
      <td>Evolution of Architectures in NAS using Evolutionary Algorithm</td>
      <td>498</td>
    </tr>
    <tr>
      <td>15.2</td>
      <td>Evolutionary Algorithm in Neural Architecture Search</td>
      <td>499</td>
    </tr>
    <tr>
      <td>15.3</td>
      <td>Reinforcement Learning in Neural Architecture Search (NAS)</td>
      <td>500</td>
    </tr>
    <tr>
      <td>15.4</td>
      <td>Policy Gradient Methods in Neural Architecture Search (NAS)</td>
      <td>502</td>
    </tr>
    <tr>
      <td>15.5</td>
      <td>NTKs, Eigenvalue Spectra, and Condition Numbers for NAS Architectures</td>
      <td>504</td>
    </tr>
    <tr>
      <td>15.6</td>
      <td>CNN NTKs and Eigenvalue Spectra for NAS Architectures</td>
      <td>505</td>
    </tr>
    <tr>
      <td>16.1</td>
      <td>Diagram of a restricted Boltzmann machine</td>
      <td>527</td>
    </tr>
    <tr>
      <td>16.2</td>
      <td>Diagram of a Deep belief network</td>
      <td>533</td>
    </tr>
    <tr>
      <td>16.3</td>
      <td>T-SNE representation of word embeddings derived from 19th-century literary texts</td>
      <td>540</td>
    </tr>
    <tr>
      <td>16.4</td>
      <td>T-SNE embeddings of MNIST dataset</td>
      <td>541</td>
    </tr>
    <tr>
      <td>16.5</td>
      <td>Locally Linear Embedding (LLE) applied to a Swiss roll dataset. Left: original 3D Swiss roll. Right: 2D LLE embedding preserving local structure</td>
      <td>546</td>
    </tr>
    <tr>
      <td>16.6</td>
      <td>Independent Component Analysis (ICA) of mixed signals. The top panel shows mixed signals, the middle panel shows true sources, and the bottom panel shows ICA recovered signals</td>
      <td>553</td>
    </tr>
    <tr>
      <td>16.7</td>
      <td>Illustration of Supervised Learning in regression. Blue points represent training data, the dashed green line is the true underlying function, and the red line is the model learned from data</td>
      <td>561</td>
    </tr>
    <tr>
      <td>16.8</td>
      <td>Bias-Variance Tradeoff in Supervised Learning. Bias<sup>2</sup> decreases with model complexity, variance increases, training error decreases, and test error initially decreases then increases due to overfitting</td>
      <td>568</td>
    </tr>
    <tr>
      <td>16.9</td>
      <td>Support Vector Machine (SVM) with linear kernel. The solid line represents the decision boundary, dashed lines show the margins, and circled points are the support vectors</td>
      <td>575</td>
    </tr>
    <tr>
      <td>16.10</td>
      <td>Illustration of the Support Vector Machine</td>
      <td>576</td>
    </tr>
    <tr>
      <td>16.11</td>
      <td>Framework of the Linear Regression</td>
      <td>582</td>
    </tr>
    <tr>
      <td>16.12</td>
      <td>Simple Linear Regression</td>
      <td>583</td>
    </tr>
    <tr>
      <td>16.13</td>
      <td>Cubic Polynomial Regression</td>
      <td>584</td>
    </tr>
    <tr>
      <td>16.14</td>
      <td>Example of a logistic regression curve fitted to data</td>
      <td>588</td>
    </tr>
    <tr>
      <td>16.15</td>
      <td>Linear discriminant analysis on a two dimensional space with two classes</td>
      <td>594</td>
    </tr>
    <tr>
      <td>16.16</td>
      <td>Visualisation for one-versus-all LDA axes for 4 classes in 3d</td>
      <td>597</td>
    </tr>
    <tr>
      <td>16.17</td>
      <td>Projections along linear discriminant axes for 4 classes</td>
      <td>598</td>
    </tr>
    <tr>
      <td>16.18</td>
      <td>Eigenvalue Perturbation Analysis of Linear Discriminant Analysis (LDA). The dashed lines represent the original eigenvalues, while the solid curves show perturbed eigenvalues as noise strength increases. This illustrates the stability of discriminant directions under perturbations</td>
      <td>599</td>
    </tr>
  </tbody>
</table>
16.19 Spectral decomposition of the discriminant space in Linear Discriminant Analysis (LDA). Left: original data projected onto the first two raw features. Right: data projected onto the discriminant space obtained from eigen-decomposition of the LDA problem. The eigenvalue ratios indicate the discriminative strength of each axis. 600

16.20 Asymptotic convergence of the decision boundary in Linear Discriminant Analysis (LDA). The black solid line is the true Bayes-optimal boundary. The blue dashed line shows the estimated LDA boundary for different sample sizes. As the number of training samples increases, the LDA boundary converges to the true boundary. 601

16.21 Exact asymptotic bounds for the misclassification probability in Linear Discriminant Analysis (LDA). The black dashed line represents the Bayes error (optimal bound), the red dashed line shows the asymptotic upper bound of order $O(1/\sqrt{n})$, and the blue curve indicates the empirical misclassification probability. As the sample size increases, the empirical curve converges tightly between the bounds. 602

16.22 Condition number of the Fisher matrix $F = S_W^{-1}S_B$ in Linear Discriminant Analysis (LDA). The x-axis shows the number of samples per class (log scale), while the y-axis shows the condition number (log scale). A high condition number indicates numerical instability, while larger sample sizes improve stability. 603

16.23 Random Matrix Theory (RMT) perspective on Linear Discriminant Analysis (LDA). The blue histogram shows the empirical eigenvalue distribution of the Fisher matrix $F = S_W^{-1}S_B$, while the red curve represents the Marchenko-Pastur law predicted by RMT. Outlier eigenvalues outside the bulk correspond to informative class-separating directions, while the bulk follows the universal random-matrix spectrum. 604

16.24 Exact rate of eigenvalue concentration in Linear Discriminant Analysis (LDA). The blue curve shows the empirical maximum deviation of eigenvalues from their mean across multiple simulations. The red dashed line represents the theoretical rate $O(\sqrt{\log p/n})$. As $n$ increases, the eigenvalues concentrate tightly around their expectation in agreement with Random Matrix Theory. 605

16.25 Weyl bounds on the eigenvalues of the Fisher matrix $F = S_W^{-1}S_B$ in Linear Discriminant Analysis (LDA). Blue points denote the original eigenvalues, red points denote the perturbed eigenvalues under a random perturbation of $S_B$, and gray dashed intervals show the Weyl bounds $[\lambda_i(F) - \|E\|_2, \lambda_i(F) + \|E\|_2]$. The perturbed eigenvalues lie within the predicted Weyl intervals. 606

16.26 Ky Fan norm bound of eigenvalues of the Linear Discriminant Analysis (LDA). The curve shows the cumulative sum of the top-$k$ eigenvalues, with the dashed red line indicating the trace bound (sum of all eigenvalues). 607

16.27 Bauer-Fike theorem applied to eigenvalues of the Fisher matrix $F = S_W^{-1}S_B$ in LDA. Blue points denote the original eigenvalues, red crosses denote perturbed eigenvalues, and gray shaded regions indicate the Bauer-Fike bounds $[\lambda_i - \kappa(V)\|E\|_2, \lambda_i + \kappa(V)\|E\|_2]$. All perturbed eigenvalues lie within the predicted intervals. 608

16.28 Generalization error of Linear Discriminant Analysis (LDA) using PAC-Bayesian bounds. The blue curve shows the empirical misclassification error, while the red dashed curve represents the PAC-Bayesian upper bound. As the sample size increases, the bound tightens, demonstrating improved generalization guarantees. 609
16.29 Decision boundary of a Gaussian Naïve Bayes classifier. The contour regions represent predicted classes, while the scatter points show the training data. Naïve Bayes assumes feature independence and Gaussian distributions for features. 629

16.30 Framework of the Decision Tree Learning. 636

16.31 Framework of the k-nearest neighbors (KNN) algorithm. 640

16.32 Autoencoder architecture. 646

16.33 Predictor architecture. 647

16.34 Decision boundary of a Self-Learning (Self-Training) classifier. The classifier is initially trained on labeled data, predicts labels on unlabeled data, and iteratively updates itself using confident predictions. The contour regions represent predicted classes, and scatter points show the original data points. 652

16.35 Framework of the Reinforcement Learning. 659

16.36 Q-Learning policy in a 4x4 grid world. Each cell shows the optimal action learned by the agent (↑, →, ↓, ←) to reach the goal. The agent learns the optimal path using reinforcement learning. 660

16.37 Policy Gradient Method: The policy network (green) outputs a distribution over actions given the state. The environment (gray) produces trajectories and rewards, which are stored (orange). The policy gradient update (red) adjusts the policy parameters using the gradient $\nabla_{\theta}J(\theta) = E_{\pi_{\theta}}[\nabla_{\theta}\log \pi_{\theta}(a|s)R]$. 667

16.38 General pipeline of Deep Reinforcement Learning (DRL): The agent, represented by a deep neural network (blue), interacts with the environment (gray) by producing actions. The environment generates experiences, which are stored in the replay buffer (orange). Mini-batches from the buffer are used to compute predictions and targets for the loss function (red). The optimizer (violet) updates network weights through gradient descent. 675

16.39 Architecture of the Advantage Actor-Critic (A2C) algorithm. The actor network (green) outputs actions based on states. The environment (gray) generates trajectories stored in a buffer (orange). The critic (blue) estimates state values, which together with rewards are used in advantage estimation (violet). The loss function (red) updates both actor and critic networks synchronously. 682

16.40 Architecture of the Deep Deterministic Policy Gradient (DDPG) algorithm. The actor network (green) outputs continuous actions. The environment (gray) produces transitions, which are stored in a replay buffer (orange). The critic (blue) and target networks (violet) evaluate Q-values and compute target values. The loss function (red) updates both actor and critic networks via policy gradients and temporal-difference learning. 689

16.41 Architecture of the Proximal Policy Optimization (PPO) algorithm: The actor (green) outputs actions, interacting with the environment (gray). Rollouts are stored in a trajectory buffer (orange). The critic (blue) estimates value functions, and the advantage estimator (violet) computes advantages $A_t$. The clipped surrogate objective (red) combines policy ratios and advantages to update the actor stably. 697
16.52 Visualization of an Interactively Constrained Neuro-Evolution (ICONE) genome: Neurons are colored by type (input, hidden, output). Frozen neurons (under user constraint) are shaded gray. Arrows represent weighted connections (blue for positive, red for negative), with width proportional to weight magnitude. Disabled connections appear faded, and constrained connections are marked with an “X” 751

16.53 Visualization of a Deus Ex Neural Network (DXNN) genome. Neurons are grouped into cores (dashed rectangles), colored by type (input, hidden, output). Arrows represent weighted connections (blue for positive, red for negative), with line thickness proportional to weight magnitude. Disabled connections appear faded. Each neuron label indicates its identifier and activation function. 755

16.54 Visualization of the Spectrum-diverse Unified Neuroevolution Architecture (SUNA) genome: Node colors represent neuron types, node sizes correspond to neuron speeds, and edge widthscolors denote weight magnitude and sign. Disabled connections are shown with faded edges, while control neurons are highlighted with dashed borders 761

17.1 Visualization of the Backpropagation Algorithm: forward activations (black arrows) and backward error propagation (red dashed arrows) 770

17.2 Trajectory of the Stochastic Gradient Descent (SGD) optimizer on $f(x, y) = x^2 + y^2$ 776

17.3 Trajectory of Nesterov Accelerated Gradient (NAG) on $f(x, y) = x^2 + y^2$ 780

17.4 Trajectory of the Adam Optimizer on $f(x, y) = x^2 + y^2$ 785

17.5 Trajectory of the RMSProp Optimizer on $f(x, y) = x^2 + y^2$ 791

17.6 Comparison of decision boundaries with and without Dropout Regularization on a two-moons dataset 801

23.1 Federated Learning in Deep Learning 963

23.2 Heterogeneous client participation in federated learning over rounds. Each row corresponds to a client and each column to a communication round; a dark cell indicates the client participated in that round 975

23.3 Number of clients participating in each federated learning round. The line shows how client participation varies across rounds under heterogeneous availability 976

23.4 Total participation frequency per client, sorted from most to least active. This highlights heterogeneity in client involvement 977

23.5 Total and average client communication per round in federated learning. Dashed line shows average communication per client, while the solid line shows total communication, highlighting potential bottlenecks due to client heterogeneity 978

23.6 Cumulative communication over rounds in federated learning. The slope shows aggregate communication growth, indicating long-term bandwidth demands and bottlenecks 979

23.7 Client drift in federated learning, measured as L2 distance from the global model. Each colored line represents an individual client, and the black bold line shows the average drift across all clients. Client drift arises due to heterogeneity in local updates and data distribution 980
23.8 Comparative client drift under different heterogeneity scenarios in federated learning. Each transparent line represents an individual client, and the bold lines show the average drift for each scenario: green: IID, orange: Mild Non-IID, red: Highly Non-IID. Greater drift occurs for highly non-iid data due to divergence in local updates 981

23.9 Impact of client drift on global model accuracy in federated learning. The green line (IID) shows smooth convergence with high accuracy. The orange line (Mild Non-IID) shows moderate fluctuations and slower convergence. The red line (Highly Non-IID) shows significant instability and limited convergence due to severe client drift 982

23.10 Heatmap of adaptive client selection across rounds in federated learning. Rows represent clients, columns represent rounds, and dark cells indicate selection. More reliable clients are selected more frequently 989

23.11 Comparison of client selection frequency in federated learning. Random selection chooses clients uniformly, while adaptive selection favors clients with higher reliability 990

23.12 Heatmap of adaptive local step sizes in federated learning. Rows represent clients, columns represent rounds, and color indicates the number of local steps 991

23.13 Comparison of average local step sizes per round under fixed vs adaptive strategies in federated learning. Adaptive steps vary over rounds, while fixed steps remain constant 992

23.14 Raw client updates before encryption in federated learning. These values are sensitive and must be protected 993

23.15 Masked (encrypted) client updates as seen by the server. Individual updates are hidden while still allowing aggregation 994

23.16 Secure aggregation result: the aggregated model update recovered at the server is identical to the true aggregate, while individual updates remain private 995

23.17 Illustration of stochastic gradient bounded variance in federated learning. The black line represents the true gradient norm trajectory, thin lines represent client stochastic gradients, the dashed blue line is the mean stochastic gradient, and the shaded region indicates the bounded variance region ($\pm\sigma$) 1003

23.18 Data heterogeneity in federated learning: each client has a different class distribution, leading to non-IID local datasets 1006

23.19 System heterogeneity in federated learning: clients participate unevenly across rounds due to resource constraints and connectivity 1007

23.20 Convergence rate in federated learning under different conditions. IID data converges faster, non-IID data converges slower, and FedAvg with more local steps speeds up convergence initially but introduces oscillations due to data heterogeneity 1009

23.21 Convergence rate in federated learning (log-log scale). Reference slopes $O(1/t)$ and $O(1/\sqrt{t})$ are shown to illustrate theoretical convergence bounds. IID follows $O(1/t)$, while non-IID slows down closer to $O(1/\sqrt{t})$ 1010
# 1 Mathematical Foundations

Deep learning is an algorithmic framework for addressing high-dimensional function approximation tasks. Fundamentally, it is based on the synergy of:

* **Functional Approximation:** Modeling complex, non-linear functions with neural networks. Functional approximation is one of the building blocks of deep learning, and it lies at the core of how deep learning models, especially neural networks, are able to tackle hard problems. In deep learning, functional approximation means the capability of neural networks to approximate complex, high-dimensional, and non-linear functions that tend to be hard or impossible to model with standard mathematical methods.

* **Optimization Theory:** Efficiently solving non-convex optimization problems. Optimization theory is a key area of deep learning, as training deep neural networks is really just minimizing the best set of parameters (weights and biases) of a given objective, commonly referred to as the loss function. The objective will usually be some measure of difference between the network outputs and the actual values. Optimization methods control the process of training and decide the extent to which a neural network may learn from data.

* **Statistical Learning Theory:** Generalization behavior on unseen data. Statistical Learning Theory (SLT) gives the mathematical basis for the generalization behavior of machine learning methods, including deep learning methods. It provides important insights into how models generalize from training data to novel data, and this is very important in making sure that deep learning models are not just correct on the training set but also work well on new, unseen data. SLT assists in solving basic issues like overfitting, bias-variance tradeoff, and generalization error.

The problem can be formalized as:

$$ \text{Find } f_{\theta}: X \to Y, \text{ such that } \mathbb{E}_{x,y \sim P}[\ell(f_{\theta}(x), y)] \text{ is minimized,} \quad (1.1) $$

where $X$ is the input space, $Y$ is the output space, $P$ is the distribution of data, $\ell(\cdot, \cdot)$ is a loss function, $\theta$ are neural network parameters. This problem entails the integration of various fields, each of which is examined in systematic detail below.

## 1.1 Problem Definition: Risk Functional as a Mapping Between Spaces

### 1.1.1 Measurable Function Spaces

A measurable space is the basic building block of measure theory, notated as $(\mathcal{X}, \Sigma)$, in which $\mathcal{X}$ is an arbitrary but fixed non-empty set called the sample space or underlying set and $\Sigma$ is a $\sigma$-algebra, that is, a particular class of subsets of $\mathcal{X}$ capturing the measurability notion. The $\sigma$-algebra $\Sigma \subseteq 2^{\mathcal{X}}$, the power set of $\mathcal{X}$, holds three axioms, each guaranteeing an essential property of closure with respect to set operations. To start with, $\Sigma$ is closed under complementation such that for every
set $A \in \Sigma$, the complement $A^c = \mathcal{X} \setminus A$ is also in $\Sigma$. This ensures the possibility of establishing
the "non-occurrence" of measurable events in a mathematically coherent manner. Second, $\Sigma$ is
closed under countable unions: for any countable family $\{A_n\}_{n=1}^\infty \subseteq \Sigma$, the union $\bigcup_{n=1}^\infty A_n$ is also
in $\Sigma$, allowing measurable sets to be constructed out of countably infinite operations. De Morgan's
laws subsequently suggest countable intersection closure, since $\bigcap_{n=1}^\infty A_n = (\bigcup_{n=1}^\infty A_n^c)^c$, so that the
structure supports conjunctions of countable sets of events. Lastly, the presence of the empty set
$\emptyset \in \Sigma$ is an axiom which allows for a null reference point, so that the $\sigma$-algebra is not empty and
can be used to express the "impossibility" of some outcomes.

<figure><img src="image_2.png" /><figcaption>Figure 1.1: Measurable Function Spaces</figcaption></figure>

1.1.1.1 Literature Review of Measurable Function Spaces

Rao et. al. (2024) [1] investigated approximation theory within Lebesgue measurable function spaces, providing an analysis of operator convergence. They also established a theoretical framework for function approximation in Lebesgue spaces and provided a rigorous study of symmetric properties in function spaces. Mukhopadhyay and Ray (2025) [2] provided a comprehensive introduction to measurable function spaces, with a focus on Lp-spaces and their completeness properties. They also established the fundamental role of Lp-spaces in measure theory and discussed the relationship between continuous function spaces and measurable functions. Szoldra (2024) [3] examined measurable function spaces in quantum mechanics, exploring the role of measurable observables in ergodic theory. They connected functional analysis and measure theory to quantum state evolution and provided a mathematical foundation for quantum machine learning in function spaces. Lee
(2025) [10] investigated metric space theory and functional analysis in the context of measurable function spaces in AI models. He formalized how function spaces can model self-referential structures in AI and provided a bridge between measure theory and generative models. Song et. al. (2025) [4] discussed measurable function spaces in the context of urban renewal and performance evaluation. They developed a rigorous evaluation metric using measurable function spaces and explored function space properties in applied data science and urban analytics. Mennaoui et. al. (2025) [5] explored measurable function spaces in the theory of evolution equations, a key concept in functional analysis. They established a rigorous study of measurable operator functions and provided new insights into function spaces and their role in solving differential equations. Pedroza (2024) [6] investigated domain stability in machine learning models using function spaces. He established a formal mathematical relationship between function smoothness and domain adaptation and uses topological and measurable function spaces to analyze stability conditions in learning models. Cerreia-Vioglio and Ok (2024) [7] developed a new integration theory for measurable set-valued functions. They introduced a generalization of integration over Banach-valued functions and established weak compactness properties in measurable function spaces. Averin (2024) [8] applied measurable function spaces to gravitational entropy theory. He provided a rigorous proof of entropy bounds using function space formalism and connected measure theory with relativistic field equations. Potter (2025) [9] investigated measurable function spaces in the context of Fourier analysis and crystallographic structures. He established new results on Fourier transforms of measurable functions and introduced a novel framework for studying function spaces in invariant shift operators.

1.1.1.2 Analysis of Measurable Function Spaces

Measurable spaces are not merely abstract structures but are the backbone of measure theory, probability, and integration. For example, the Borel $\sigma$-algebra $\mathcal{B}(\mathbb{R})$ on the real numbers $\mathbb{R}$ is the smallest $\sigma$-algebra containing all open intervals $(a, b)$ for $a, b \in \mathbb{R}$. This $\sigma$-algebra is pivotal in defining Lebesgue measure, where measurable sets generalize the classical notion of intervals to include sets that are neither open nor closed. Moreover, the construction of a $\sigma$-algebra generated by a collection of subsets $C \subseteq 2^{\mathcal{X}}$, denoted $\sigma(C)$, provides a minimal framework that includes $C$ and satisfies all $\sigma$-algebra properties, enabling the systematic extension of measurability to more complex scenarios. For instance, starting with intervals in $\mathbb{R}$, one can build the Borel $\sigma$-algebra, a critical tool in modern analysis.

The structure of a measurable space allows the definition of a measure $\mu$, a function $\mu : \Sigma \to [0, \infty]$ that assigns a non-negative value to each set in $\Sigma$, adhering to two key axioms: $\mu(\emptyset) = 0$ and countable additivity, which states that for any disjoint collection $\{A_n\}_{n=1}^{\infty} \subseteq \Sigma$, the measure of their union satisfies $\mu(\bigcup_{n=1}^{\infty} A_n) = \sum_{n=1}^{\infty} \mu(A_n)$. This property is indispensable in extending intuitive notions of length, area, and volume to arbitrary measurable sets, paving the way for the Lebesgue integral. A function $f : \mathcal{X} \to \mathbb{R}$ is then termed $\Sigma$-measurable if for every Borel set $B \in \mathcal{B}(\mathbb{R})$, the preimage $f^{-1}(B)$ belongs to $\Sigma$. This definition ensures that the function is compatible with the $\sigma$-algebra, a necessity for defining integrals and expectation in probability theory.

In summary, measurable spaces represent a highly versatile and mathematically rigorous framework, underpinning vast areas of analysis and probability. Their theoretical depth lies in their ability to systematically handle infinite operations while maintaining closure, consistency, and flexibility for defining measures, measurable functions, and integrals. As such, the rigorous study of measurable spaces is indispensable for advancing modern mathematical theory, providing a bridge between abstract set theory and applications in real-world phenomena.
Let $(\mathcal{X}, \Sigma_X, \mu_X)$ and $(\mathcal{Y}, \Sigma_Y, \mu_Y)$ be measurable spaces. The true risk functional is defined as:

$$ \mathcal{R}(f) = \int_{\mathcal{X} \times \mathcal{Y}} \ell(f(x), y) dP(x, y), \qquad (1.2) $$

where:

* $f$ belongs to a hypothesis space $\mathcal{F} \subseteq L^p(\mathcal{X}, \mu_X)$.

* $P(x, y)$ is a Borel probability measure over $\mathcal{X} \times \mathcal{Y}$, satisfying $\int_{\mathcal{X} \times \mathcal{Y}} 1 dP = 1$.

## 1.1.2 Risk as a Functional

### 1.1.2.1 Literature Review of Risk as a Functional

Wang et. al. (2025) [11] developed a mathematical risk model based on functional variational calculus and introduced a loss functional regularization framework that minimizes adversarial risk in deep learning models. They also proposed a game-theoretic interpretation of functional risk in security settings. Duim and Mesquita (2025) [12] extended the inverse reinforcement learning (IRL) framework by defining risk as a functional over probability spaces and used Bayesian functional priors to model risk-sensitive behavior. They also introduced an iterative regularized risk functional minimization approach. Khayat et. al. (2025) [13] established functional Sobolev norms to quantify risk in adversarial settings and introduced a functional risk decomposition technique using deep neural architectures. They also provided an in-depth theoretical framework for risk estimation in adversarially perturbed networks. Agrawal (2025) [14] developed a variational framework for risk as a loss functional and used adaptive weighting of loss functions to enhance generalization in deep learning. He also provided rigorous convergence analysis of risk functional minimization. Hailemichael and Ayalew (2025) [15] used control barrier function (CBF) theory to develop risk-aware deep learning models and modeled risk as a functional on dynamical systems, optimizing stability and robustness. They also introduced a risk-minimizing constrained optimization formulation. Nguyen et. al. (2025) [16] developed a functional metric learning approach for risk-sensitive deep models and used convex optimization techniques to derive functional risk bounds. They also established semi-supervised loss functions for risk-regularized learning. Luo et. al. (2025) [17] introduced a geometric interpretation of risk functionals in deep learning models and used integral transform techniques to approximate risk in real-world vision systems. They also developed a functional approach to adversarial robustness.

### 1.1.2.2 Analysis of Risk as a Functional

The functional $\mathcal{R}: \mathcal{F} \to \mathbb{R}_+$ is Fréchet-differentiable if:

$$ \forall f, g \in \mathcal{F}, \quad \mathcal{R}(f + \epsilon g) = \mathcal{R}(f) + \epsilon \langle \nabla \mathcal{R}(f), g \rangle + o(\epsilon), \qquad (1.3) $$

where $\langle \cdot, \cdot \rangle$ denotes the inner product in $L^2(\mathcal{X})$. In the field of risk management and decision theory, the concept of a **risk functional** is a mathematical formalization that captures how risk is quantified for a given outcome or state. A risk functional, denoted as $\mathcal{R}$, acts as a map that takes elements from a given space $X$ (which represents the possible outcomes or states) and returns a real-valued risk measure. This risk measure, $\mathcal{R}(x)$, expresses the degree of risk or the adverse outcome associated with a particular element $x \in X$. The space $X$ may vary depending on the context—this could be a space of random variables, trajectories, or more complex function spaces. The risk functional maps $x$ to $\mathbb{R}$, i.e., $\mathcal{R}: X \to \mathbb{R}$, where each $\mathcal{R}(x)$ reflects the risk associated with the specific realization $x$. One of the most foundational forms of risk functionals is based on the **expectation** of a loss function $L(x)$, where $x \in X$ represents a random variable or state, and $L(x)$
quantifies the loss associated with that state. The risk functional can be expressed as an expected loss, written mathematically as:

$$ \mathcal{R}(x) = \mathbb{E}[L(x)] = \int_{X} L(x)p(x) dx \qquad (1.4) $$

where $p(x)$ is the probability density function of the outcome $x$, and the integration is taken over the entire space $X$. In this setup, $L(x)$ can be any function that measures the severity or unfavorable nature of the outcome $x$. In a financial context, $L(x)$ could represent the loss function for a portfolio, and $p(x)$ would be the probability density function of the portfolio's returns. In many cases, a specific form of $L(x)$ is used, such as $L(x) = (x - \mu)^2$, where $\mu$ is the target or expected value. This choice results in the risk functional representing the **variance** of the outcome $x$, expressed as:

$$ \mathcal{R}(x) = \int_{X} (x - \mu)^2 p(x) dx \qquad (1.5) $$

This formulation captures the variability or dispersion of outcomes around a mean value, a common risk measure in applications like portfolio optimization or risk management. Additionally, another widely used class of risk functionals arises from **quantile-based risk measures**, such as **Value-at-Risk (VaR)**, which focuses on the extreme tail behavior of the loss distribution. The VaR at a confidence level $\alpha \in [0, 1]$ is defined as the smallest value $l$ such that the probability of $L(x)$ exceeding $l$ is no greater than $1 - \alpha$, i.e.,

$$ P(L(x) \le l) \ge \alpha \qquad (1.6) $$

This defines a threshold beyond which the worst outcomes are expected to occur with probability $1 - \alpha$. Value-at-Risk provides a measure of the worst-case loss under normal circumstances, but it does not provide information about the severity of losses exceeding this threshold. To address this limitation, the **Conditional Value-at-Risk (CVaR)** is introduced, which measures the expected loss given that the loss exceeds the VaR threshold. Mathematically, CVaR at the level $\alpha$ is given by:

$$ \text{CVaR}_{\alpha}(x) = \mathbb{E}[L(x) | L(x) \ge \text{VaR}_{\alpha}(x)] \qquad (1.7) $$

This conditional expectation provides a more detailed assessment of the potential extreme losses beyond the VaR threshold. The CVaR is a more comprehensive measure, capturing the tail risk and providing valuable information about the magnitude of extreme adverse events. In cases where the space $X$ represents **trajectories** or paths, such as in the context of continuous-time processes or dynamical systems, the risk functional is often formulated in terms of integrals over time. For example, consider $x(t)$ as a trajectory in the function space $C([0, T], \mathbb{R}^n)$, the space of continuous functions on the interval $[0, T]$. The risk functional in this case might quantify the total deviation of the trajectory from a reference or target trajectory over time. A typical example could be the total squared deviation, written as:

$$ \mathcal{R}(x) = \int_{0}^{T} \|x(t) - \bar{x}(t)\|^2 dt \qquad (1.8) $$

where $\bar{x}(t)$ represents a reference trajectory and $\| \cdot \|$ is a norm, such as the Euclidean norm. This risk functional quantifies the total deviation (or **energy**) of the trajectory from the target path over the entire time interval, and is used in various applications such as control theory and optimal trajectory planning. A common choice for the norm $\|x(t)\|$ might be $\|x(t)\|^2 = \sum_{i=1}^{n} x_i^2(t)$, where $x_i(t)$ are the components of the trajectory $x(t)$ in $\mathbb{R}^n$. In some cases, the space $X$ of possible outcomes may not be a finite-dimensional vector space, but instead a **Banach space** or a **Hilbert space**, particularly when $x$ represents a more complex object such as a function or a trajectory. For example, the space $C([0, T], \mathbb{R}^n)$ is a Banach space, and the risk functional may involve the
evaluation of integrals over this function space. In such settings, the risk functional can take the form:

$$ \mathcal{R}(x) = \int_{0}^{T} \|x(t)\|_{p}^{p} dt \qquad (1.9) $$

where $\| \cdot \|_p$ is the $p$-norm, and $p \ge 1$. For $p = 2$, this risk functional represents the total energy of the trajectory, but other norms can be used to emphasize different types of risks. For instance, the $L^\infty$-norm would focus on the maximum deviation of the trajectory from the target path. The concept of **convexity** plays a significant role in the theory of risk functionals. Convexity ensures that the risk associated with a **convex combination** of two states $x_1$ and $x_2$ is less than or equal to the weighted average of the risks of the individual states. Mathematically, for $\lambda \in [0, 1]$, convexity demands that:

$$ \mathcal{R}(\lambda x_1 + (1 - \lambda)x_2) \le \lambda \mathcal{R}(x_1) + (1 - \lambda) \mathcal{R}(x_2) \qquad (1.10) $$

This property reflects the diversification effect in risk management, where mixing several states or outcomes generally leads to a reduction in overall risk. Convex risk functionals are particularly important in portfolio theory, where they allow for risk minimization through diversification. For example, if $\mathcal{R}(x)$ represents the variance of a portfolio's returns, then the convexity property ensures that combining different assets will result in a portfolio with lower overall risk than the risk of any individual asset. **Monotonicity** is another important property for risk functionals, ensuring that the risk increases as the outcome becomes more adverse. If $x_1$ is worse than $x_2$ according to some partial order, we have:

$$ \mathcal{R}(x_1) \ge \mathcal{R}(x_2) \qquad (1.11) $$

Monotonicity ensures that the risk functional behaves in a way that aligns with intuitive notions of risk: worse outcomes are associated with higher risk. In financial contexts, this is reflected in the fact that **losses** increase the associated risk measure. Finally, in some applications, the risk functional is derived from perturbation analysis to study how small changes in parameters affect the overall risk. Consider $x(\epsilon)$ as a perturbed trajectory, where $\epsilon$ is a small parameter, and the Fréchet derivative of the risk functional with respect to $\epsilon$ is given by:

$$ \frac{d}{d\epsilon} \mathcal{R}(x(\epsilon)) \Big|_{\epsilon=0} \qquad (1.12) $$

This derivative quantifies the sensitivity of the risk to perturbations in the system and is crucial in the analysis of stability and robustness. Such analyses are essential in areas like **stochastic control** and **optimization**, where it is important to understand how small changes in the model's parameters can influence the risk profile.

Thus, the risk functional is a powerful tool for quantifying and managing uncertainty, and its formulation can be adapted to various settings, from random variables and stochastic processes to continuous trajectories and dynamic systems. The risk functional provides a rigorous mathematical framework for assessing and minimizing risk in complex systems, and its flexibility makes it applicable across a wide range of domains.

## 1.2 Approximation Spaces for Neural Networks

The neural network hypothesis space $\mathcal{F}_\theta$ is parameterized as:

$$ \mathcal{F}_{\theta} = \{ f_{\theta} : \mathcal{X} \to \mathbb{R} \mid f_{\theta}(x) = \sum_{j=1}^{n} c_j \sigma(a_j \cdot x + b_j), \theta = (c, a, b) \}. \qquad (1.13) $$

To analyze its capacity, we rely on:
* VC-dimension theory for discrete hypotheses.

* Rademacher complexity for continuous spaces:

$$ \mathcal{R}_N(\mathcal{F}) = \mathbb{E}_\epsilon \left[ \sup_{f \in \mathcal{F}} \frac{1}{N} \sum_{i=1}^N \epsilon_i f(x_i) \right], \qquad (1.14) $$

where $\epsilon_i$ are i.i.d. Rademacher random variables.

### 1.2.1 VC-dimension theory for discrete hypotheses

The VC-dimension (Vapnik-Chervonenkis dimension) is a fundamental concept in statistical learning theory that quantifies the capacity of a hypothesis class to fit a range of labelings of a set of data points. The VC-dimension is particularly useful in understanding the generalization ability of a classifier. The theory is important in machine learning, especially when assessing overfitting and the risk of model complexity.

#### 1.2.1.1 Literature Review of VC-dimension theory for discrete hypotheses

There are several articles that explore the VC-dimension theory for discrete hypotheses very rigorously. N. Bousquet and S. Thomassé (2015) [18] explored in their paper the VC-dimension in the context of graph theory, connecting it to structural properties such as the Erdős-Pósa property. Yıldız and Alpaydin (2009) [19] in their article computed the VC-dimension for decision tree hypothesis spaces, considering both discrete and continuous features. Zhang et. al. (2012) [20] introduced a discretized VC-dimension to bridge real-valued and discrete hypothesis spaces, offering new theoretical tools for complexity analysis. Riondato and Zdonik (2011) [21] adapted VC-dimension theory to database systems, analyzing SQL query selectivity using a theoretical lens. Riggle and Sonderegger (2010) [22] investigated the VC-dimension in linguistic models, focusing on grammar hypothesis spaces. Anderson (2023) [23] provided a comprehensive review of VC-dimension in fuzzy systems, particularly in logic frameworks involving discrete structures. Fox et. al. (2021) [24] proved key conjectures for systems with bounded VC-dimension, offering insights into combinatorial implications. Johnson (2021) [25] discusses binary representations and VC-dimensions, with implications for discrete hypothesis modeling. Janzing (2018) [26] in his paper focuses on hypothesis classes with low VC-dimension in causal inference frameworks. Ghosh (2024) [907] derived the expressions for fundamental differential operators—namely the gradient, divergence, and curl—as well as the vector gradient of a vector field within curvilinear coordinate systems. This work significantly enhances clarity on the application of the del operator in non-Cartesian contexts, laying a robust foundation for advanced analytical and computational treatments of vector calculus in complex geometries. Hüllermeier and Tehrani (2012) [27] in their paper explored the theoretical VC-dimension of Choquet integrals, applied to discrete machine learning models. The book titled “Foundations of Machine Learning” [28] by Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar offers a very good foundational discussion on VC-dimension in the context of statistical learning. Another book titled “Learning Theory: An Approximation Theory Viewpoint” by Felipe Cucker and Ding-Xuan Zhou [29] discusses the role of VC-dimension in approximation theory. Yet another book titled “Understanding Machine Learning: From Theory to Algorithms” by Shai Shalev-Shwartz and Shai Ben-David [30] contains detailed chapters on hypothesis spaces and VC-dimension.

#### 1.2.1.2 Analysis of VC-dimension theory for discrete hypotheses

For discrete hypotheses, the VC-dimension theory applies to a class of hypotheses that map a set of input points to binary output labels (typically 0 or 1). The VC-dimension for a hypothesis class refers to the largest set of data points that can be shattered by that class, where "shattering"
means that the hypothesis class can realize all possible labelings of these points.

We shall now discuss the **Formal Mathematical Framework**. Let $X$ be a finite or infinite set called the **instance space**, which represents the input space. Consider a hypothesis class $H$, where each hypothesis $h \in H$ is a function $h: X \to \{0,1\}$. The function $h$ classifies each element of $X$ into one of two classes: 0 or 1. Given a subset

$$S = \{x_1, x_2, \dots, x_k\} \subseteq X \qquad (1.15)$$

we say that $H$ **shatters** $S$ if for every possible labeling $\vec{y} = (y_1, y_2, \dots, y_k) \in \{0,1\}^k$, there exists some $h \in H$ such that for all $i \in \{1, 2, \dots, k\}$, we have:

$$h(x_i) = y_i \qquad (1.16)$$

In other words, a hypothesis class $H$ shatters $S$ if it can produce every possible binary labeling on the set $S$. The **VC-dimension** $\text{VC}(H)$ is defined as the size of the largest set $S$ that can be shattered by $H$:

$$\text{VC}(H) = \sup\{k \mid \exists S \subseteq X, |S| = k, S \text{ is shattered by } H\} \qquad (1.17)$$

If no set of points can be shattered, then the VC-dimension is 0. Some Properties of the VC-Dimension are

1. **Shattering Implies Non-empty Hypothesis Class:** If a set $S$ is shattered by $H$, then $H$ is non-empty. This follows directly from the fact that for each labeling $\vec{y} \in \{0,1\}^k$, there exists some $h \in H$ that produces the corresponding labeling. Therefore, $H$ must contain at least one hypothesis.

2. **Upper Bound on Shattering:** Given a hypothesis class $H$, if there exists a set $S \subseteq X$ of size $k$ such that $H$ can shatter $S$, then any set $S' \subseteq X$ of size greater than $k$ cannot be shattered. This gives us the crucial result that:

$$\text{VC}(H) \geq k \quad \text{if } H \text{ can shatter a set of size } k \qquad (1.18)$$

3. **Implication for Generalization** A central result in the theory of **statistical learning** is the connection between VC-dimension and the **generalization error**. Specifically, the **VC-dimension** bounds the ability of a hypothesis class to generalize to unseen data. The higher the VC-dimension, the more complex the hypothesis class, and the more likely it is to **overfit** the training data, leading to poor generalization.

We shall now discuss the VC-Dimension and Generalization Bounds (VC Theorem). The **VC-dimension theorem** (often referred to as **Hoeffding's bound** or the **generalization bound**) provides a probabilistic guarantee on the relationship between the training error and the true error. Specifically, it gives an upper bound on the probability that the generalization error exceeds the empirical error (training error) by more than $\epsilon$.

The **Vapnik-Chervonenkis (VC) dimension** is a fundamental measure of the capacity of a hypothesis class $\mathcal{H}$, and it plays a crucial role in understanding the generalization ability of machine learning models, including neural networks. The VC-dimension of a hypothesis class is defined as the largest number $m$ such that there exists a set of $m$ points that can be **shattered** by $\mathcal{H}$. Formally, a set

$$S = \{x_1, x_2, \dots, x_m\} \qquad (1.19)$$

is shattered by $\mathcal{H}$ if for every possible binary labeling $\{y_1, y_2, \dots, y_m\} \in \{0,1\}^m$, there exists a hypothesis $h \in \mathcal{H}$ such that

$$h(x_i) = y_i \quad \text{for all } i \qquad (1.20)$$
The VC-dimension, denoted as $VC(\mathcal{H})$, is the supremum of all such $m$ that can be shattered by $\mathcal{H}$. For a neural network with $W$ total parameters (weights and biases), the capacity of the hypothesis class induced by the network architecture depends on the activation function. If the activation function is piecewise linear (such as ReLU), then an upper bound for the VC-dimension is given by

$$VC(\mathcal{H}) = O(W \log W). \qquad (1.21)$$

This result is derived from combinatorial arguments involving the number of hyperplane arrangements that a neural network can realize in a given input space. If the activation function is nonlinear

<figure><img src="image_3.png" /><figcaption>Figure 1.2: VC growth function $m_H(n)$ (Linear scale)</figcaption></figure>

(such as sigmoid or tanh), the capacity can increase but remains constrained by the number of parameters. The intuition behind this bound lies in the observation that each neuron introduces a decision boundary in the input space, and the number of such boundaries grows approximately as $W \log W$ rather than $W$ due to dependencies among neurons. When considering **discrete hypotheses**, the hypothesis space $\mathcal{H}$ is restricted to a finite number of functions. This occurs in scenarios where the weights and biases are **quantized** to a finite set of values, or where the activation functions themselves take on a finite number of distinct outputs. If there are $K$ possible distinct values for each parameter and there are $W$ parameters, then the total number of possible functions that the network can realize is at most

$$|\mathcal{H}| \leq K^W. \qquad (1.22)$$

From the fundamental relationship between the VC-dimension and the number of hypotheses in a finite hypothesis space, we obtain the bound

$$VC(\mathcal{H}) \leq \log_2 |\mathcal{H}|. \qquad (1.23)$$
Applying this result to a neural network with quantized weights and biases, we obtain

$$VC(\mathcal{H}) \leq W \log_2 K. \qquad (1.24)$$

Thus, **discretization significantly reduces the VC-dimension** compared to continuous-valued networks, where the number of hypotheses is effectively infinite due to the continuous range of parameters. For binary networks, where each weight is restricted to two possible values (e.g., ±1), the bound simplifies further to

$$VC(\mathcal{H}) \leq W. \qquad (1.25)$$

Since VC-dimension directly affects the generalization error, its role in neural network theory is formalized through the following **uniform convergence bound**. If $h$ is chosen from $\mathcal{H}$ based on an i.i.d. training set of size $N$, then with probability at least $1 - \delta$, the true error $\ell(h)$ and the empirical error $\hat{\ell}(h)$ satisfy the bound

$$\sup_{h \in \mathcal{H}} |\ell(h) - \hat{\ell}(h)| \leq \sqrt{\frac{VC(\mathcal{H}) \log \frac{N}{VC(\mathcal{H})} + \log \frac{1}{\delta}}{N}}. \qquad (1.26)$$

A crucial consequence of this inequality is that for a hypothesis class with a **larger VC-dimension**, a larger number of training samples is required to achieve the same generalization error. Since discrete neural networks have a **lower VC-dimension** than their continuous counterparts, they often generalize better given the same training set size, provided the hypothesis class is sufficiently expressive for the problem at hand. The **shattering coefficient**, also known as the **growth function**, plays an essential role in understanding the VC-dimension of neural networks. The shattering coefficient $S_{\mathcal{H}}(m)$ is defined as the maximum number of dichotomies that $\mathcal{H}$ can realize

<figure><img src="image_4.png" /><figcaption>Figure 1.3: **VC growth function $m_H(n)$ (Log scale)**</figcaption></figure>
on any set of $m$ points. If $S_H(m) = 2^m$, then $\mathcal{H}$ can fully shatter an $m$-point dataset. For neural networks with continuous parameters, the growth function satisfies

$$S_H(m) \le O(m^{VC(\mathcal{H})}). \qquad (1.27)$$

For discrete neural networks, where $|\mathcal{H}|$ is finite, we obtain the bound

$$S_H(m) \le |\mathcal{H}|, \qquad (1.28)$$

which ensures that the number of realizable functions is significantly smaller than in the continuous case, thereby **reducing overfitting potential**. The interplay between VC-dimension and **struc- tural risk minimization (SRM)** further highlights the importance of capacity control in neural networks. Given a hierarchy of hypothesis classes $\mathcal{H}_1 \subset \mathcal{H}_2 \subset \dots$, with increasing VC-dimension, the optimal generalization performance is achieved by choosing the class $\mathcal{H}_k$ that minimizes the trade-off between empirical risk and capacity, as given by

$$R(h) \le \hat{R}(h) + \mathcal{O}\left(\sqrt{\frac{VC(\mathcal{H}_k) \log N}{N}}\right). \qquad (1.29)$$

For discrete networks, the practical implication of this bound is that models with **limited pre- cision weights** or **constrained architectures** tend to generalize better than overparameterized networks with unnecessarily high VC-dimension. From an optimization standpoint, discrete neu- ral networks have a **smaller number of local minima** in the loss landscape compared to their continuous counterparts. This is because the number of unique parameter configurations is finite, leading to a more structured and potentially more **tractable optimization problem**. However, the trade-off is that discrete optimization is often **NP-hard**, requiring specialized techniques such as simulated annealing, evolutionary algorithms, or integer programming methods. Thus, VC- dimension theory provides profound insights into the **expressive power, generalization ability**, and **optimization complexity** of neural networks. Discrete neural networks exhibit a **reduced VC-dimension** compared to their continuous counterparts, leading to **potentially better gen- eralization**, provided that the model retains sufficient expressivity for the given problem. The **trade-off between expressivity and generalization** is fundamental to designing efficient neu- ral network architectures that perform well in practice.

Let $\mathcal{D}$ be the distribution from which the training data is drawn, and let $\hat{e}rr(h)$ and $err(h)$ represent the **empirical error** and **true error** of a hypothesis $h \in H$, respectively:

$$e\hat{r}r(h) = \frac{1}{n} \sum_{i=1}^{n} \mathbf{1}_{\{h(x_i) \neq y_i\}} \qquad (1.30)$$

$$err(h) = \mathbb{P}_{(x,y) \sim \mathcal{D}} (h(x) \neq y) \qquad (1.31)$$

where $\{(x_1, y_1), \dots, (x_n, y_n)\}$ are i.i.d. (independent and identically distributed) samples from the distribution $\mathcal{D}$. For a hypothesis class $H$ with **VC-dimension** $d = VC(H)$, with probability at least $1 - \delta$, the following holds for all $h \in H$:

$$|e\hat{r}r(h) - err(h)| \le \epsilon \qquad (1.32)$$

where $\epsilon$ is bounded by:

$$\epsilon \le \sqrt{\frac{8}{n} \left( d \log \left( \frac{2n}{d} \right) + \log \left( \frac{4}{\delta} \right) \right)} \qquad (1.33)$$

This result shows that the generalization error (the difference between the true and empirical error) is small with high probability, provided the sample size $n$ is large enough and the VC-dimension
d is not too large. The sample complexity n required to guarantee that the generalization error is within $\epsilon$ with high probability $1 - \delta$ is given by:

$$n \geq \frac{C}{\epsilon^2} \left( d \log \left( \frac{1}{\epsilon} \right) + \log \left( \frac{1}{\delta} \right) \right) \quad (1.34)$$

where C is a constant depending on the distribution. This bound emphasizes the importance of VC-dimension in controlling the complexity of the hypothesis class. A larger VC-dimension requires a larger sample size to avoid overfitting and ensure reliable generalization. Some Detailed Examples are:

1. **Example 1: Linear Classifiers in $\mathbb{R}^2$**: Consider the hypothesis class $H$ consisting of linear classifiers in $\mathbb{R}^2$. These classifiers are hyperplanes in two dimensions, defined by:

$$h(x) = \operatorname{sign}(w^T x + b) \quad (1.35)$$

where $w \in \mathbb{R}^2$ is the weight vector and $b \in \mathbb{R}$ is the bias term. The **VC-dimension** of linear classifiers in $\mathbb{R}^2$ is 3. This can be rigorously shown by noting that for any set of 3 points in $\mathbb{R}^2$, the hypothesis class $H$ can shatter these points. In fact, any possible binary labeling of the 3 points can be achieved by some linear classifier. However, for 4 points in $\mathbb{R}^2$, it is impossible to shatter all possible binary labelings (e.g., the four vertices of a convex quadrilateral), meaning the VC-dimension is 3.

2. **Example 2: Polynomial Classifiers of Degree $d$**: Consider a polynomial hypothesis class in $\mathbb{R}^n$ of degree $d$. The hypothesis class $H$ consists of polynomials of the form:

$$h(x) = \sum_{i_1, i_2, \dots, i_n} \alpha_{i_1, i_2, \dots, i_n} x_1^{i_1} x_2^{i_2} \dots x_n^{i_n} \quad (1.36)$$

where the $\alpha_{i_1, i_2, \dots, i_n}$ are coefficients and $x = (x_1, x_2, \dots, x_n)$. The **VC-dimension** of polynomial classifiers of degree $d$ in $\mathbb{R}^n$ grows as $O(n^d)$, implying that the complexity of the hypothesis class increases rapidly with both the degree $d$ and the dimension $n$ of the input space.

Neural networks, depending on their architecture, can have very high VC-dimensions. In particular, the **VC-dimension** of a neural network with $L$ layers, each containing $N$ neurons, is typically $O(N^L)$, indicating that the VC-dimension grows exponentially with both the number of neurons and the number of layers. This result provides insight into the complexity of neural networks and their capacity to overfit data when the training sample size is insufficient.

The **VC-dimension** of a hypothesis class is a powerful tool in statistical learning theory. It quantifies the complexity of the hypothesis class by measuring its capacity to shatter sets of points, and it is directly tied to the model's ability to generalize. The **VC-dimension theorem** provides rigorous bounds on the generalization error and sample complexity, giving us essential insights into the trade-off between model complexity and generalization. The theory extends to more complex hypothesis classes such as linear classifiers, polynomial classifiers, and neural networks, where it serves as a critical tool for controlling overfitting and ensuring reliable performance on unseen data.

### 1.2.1.3 Python Code to generate Figures 1.2 and 1.3 illustrating VC-dimension theory for discrete hypotheses

The Python code below produces the Figures 1.2 and 1.3 illustrating VC-dimension theory for discrete hypotheses.
```python
"""
2 vc_plot_separate.py
3
4 Plot illustrating the VC-dimension theorem (Sauer-Shelah lemma),
5 with separate figures for linear and log scale plots.
"""
6
7 import numpy as np
8
9 import matplotlib.pyplot as plt
10
11 import math
12
13 def sauer_bound(n, d):
14     """Compute Sauer bound: sum_{i=0}^d C(n,i)."""
15     s = 0
16     for i in range(0, min(n, d) + 1):
17         s += math.comb(n, i)
18     return s
19
20 def growth_function(n_array, d):
21     """Return arrays: m_H(n), 2^n, and Sauer bound for VC-dim = d."""
22     two twoPow = np.array([2**n for n in n_array], dtype= np.float64)
23     sauer = np.array([sauer_bound(n, d) for n in n_array], ndtype = np.
24                       float64)
25     return np minimum(twoPow, sauer), twoPow, sauer
26
27 def plot_vc bounds separate(n_max=30, d_values=(0,1,2,3,5,10)):
28     n = np arange(0, n_max+1)
29
30 # --- Linear scale plot ---
31 plt .figure(figure size=(7,5))
32 for d in d_values:
33     mH, twoPow, sauer = growth_function(n, d)
34     plt .plot(n, mH, marker='o', style='--', label='mH(n), d={d
35 }')
36     plt .plot(n, sauer, style='--', alpha=0.6,
37               label='Sauer bound (d={d})' if d==d_values[0] else
38               None)
39     plt .plot(n, 2**n, color='k', style=':', label='2^n (max)')
40     plt .title('VC growth function $m_H(n)$ (linear scale)')
41     plt .xlabel('n (number of points)')
42     plt .ylabel('number of dichotomies')
43     plt .ylim(bottom=0)
44     plt .legend(loc='upper left', font size='small')
45     plt .grid(True, style=':', alpha=0.6)
46     plt .show()
47
48 # --- Log scale plot ---
49 plt .figure(figure size=(7,5))
50 for d in d_values:
51     mH, twoPow, sauer = growth_function(n, d)
52     plt .plot(n, mH, marker='o', style='--', label='mH(n), d={d
53 }')
54     plt .plot(n, 2**n, color='k', style=':', label='2**n, mH(n)')
55
```
50

51

52

53

54

55

56

57

58

59

```matlab
$2^n$ (max)')
    plt.title('VC growth function $m_H(n)$ (log scale)')
    plt.xlabel('n (number of points)')
    plt.ylabel('number of dichotomies (log scale)')
    plt.yscale('log')
    plt.legend(loc='upper left', fontsize='small')
    plt.grid(True, which='both', linewidth=':', alpha=0.6)
    plt.show()

if __name__ == '__main__':
    plot_vc bounds_separate(n_max=30, d_values=(0,1,2,3,5,10))
```

## 1.2.2 Rademacher complexity for continuous spaces

### 1.2.2.1 Literature Review of Rademacher complexity for continuous spaces

Truong (2022) [31] in his article explored how Rademacher complexity impacts generalization error in deep learning, particularly with IID and Markov datasets. Gneco and Sanguineti (2008) [32] developed approximation error bounds in Reproducing Kernel Hilbert Spaces (RKHS) and functional approximation settings. Astashkin (2010) [33] discusses applications of Rademacher functions in symmetric function spaces and their mathematical structure. Ying and Campbell (2010) [34] applies Rademacher complexity to kernel-based learning problems and support vector machines. Zhu et al. (2009) [35] examined Rademacher complexity in cognitive models and neural representation learning. Astashkin et al. (2020) [36] investigated how the Rademacher system behaves in function spaces and its role in functional analysis. Sachs et al. (2023) [37] introduced a refined approach to Rademacher complexity tailored to specific machine learning algorithms. Ma and Wang (2020) [38] investigated Rademacher complexity bounds in deep residual networks. Bartlett and Mendelson (2002) [39] wrote a foundational paper on complexity measures, providing fundamental theoretical insights into generalization bounds. Dzahini and Wild (2024) [40] in their paper extended Rademacher-based complexity to stochastic optimization methods. McDonald and Shalizi (2011) [41] showed using sequential Rademacher complexities for I.I.D process how to control the generalization error of time series models wherein past values of the outcome are used to predict future values.

### 1.2.2.2 Analysis of Rademacher complexity for continuous spaces

Let $(\mathcal{X}, \Sigma, \mathcal{D})$ represent a probability space where $\mathcal{X}$ is a measurable space, $\Sigma$ is a sigma-algebra, and $\mathcal{D}$ is a probability measure. The function class $\mathcal{F} \subset L^\infty(\mathcal{X}, \mathbb{R})$ satisfies:

$$ \sup_{f \in \mathcal{F}} \|f\|_{\infty} < \infty, \qquad (1.37) $$

where $\|f\|_{\infty} = \ess \sup_{x \in \mathcal{X}} |f(x)|$ denotes the essential supremum. For rigor, $\mathcal{F}$ is assumed measurable in the sense that for every $\epsilon > 0$, there exists a countable subset $\mathcal{F}_{\epsilon} \subseteq \mathcal{F}$ such that:

$$ \sup_{f \in \mathcal{F}} \inf_{g \in \mathcal{F}_{\epsilon}} \|f - g\|_{\infty} \le \epsilon. \qquad (1.38) $$

Given $S = \{x_1, x_2, \dots, x_n\} \sim \mathcal{D}^n$, the empirical measure $\mathbb{P}_n$ is:

$$ \mathbb{P}_n(A) = \frac{1}{n} \sum_{i=1}^{n} \mathbb{1}_{\{x_i \in A\}}, \quad \forall A \in \Sigma. \qquad (1.39) $$

The integral under $\mathbb{P}_n$ for $f \in \mathcal{F}$ approximates the population integral under $\mathcal{D}$:

$$ \mathbb{P}_n[f] = \frac{1}{n} \sum_{i=1}^{n} f(x_i), \quad \mathcal{D}[f] = \int_{\mathcal{X}} f(x) d\mathcal{D}(x). \qquad (1.40) $$
Let $\sigma = (\sigma_1, \dots, \sigma_n)$ be independent Rademacher random variables:

$$ \mathbb{P}(\sigma_i = +1) = \mathbb{P}(\sigma_i = -1) = \frac{1}{2}, \quad i = 1, \dots, n. \qquad (1.41) $$

These variables are defined on a probability space $(\Omega, \mathcal{A}, \mathbb{P})$ independent of the sample $S$. The

<figure><img src="image_5.png" /><figcaption>Figure 1.4: Rademacher Complexity vs. $\frac{1}{\sqrt{n}}$ Bound</figcaption></figure>

Duality and Symmetrization of Empirical Rademacher Complexity is also very important. The empirical Rademacher complexity of $\mathcal{F}$ with respect to $S$ is:

$$ \hat{\mathfrak{R}}_S(\mathcal{F}) = \mathbb{E}_{\sigma} \left[ \sup_{f \in \mathcal{F}} \frac{1}{n} \sum_{i=1}^{n} \sigma_i f(x_i) \right], \qquad (1.42) $$

where $\mathbb{E}_{\sigma}$ denotes expectation over $\sigma$. The supremum can be interpreted as a functional dual norm in $L^{\infty}(\mathcal{X}, \mathbb{R})$, where $\mathcal{F}$ is the unit ball. Using the symmetrization technique, the Rademacher complexity relates to the deviation of $\mathbb{P}_n[f]$ from $\mathcal{D}[f]$:

$$ \mathbb{E}_S \sup_{f \in \mathcal{F}} |\mathbb{P}_n[f] - \mathcal{D}[f]| \le 2\mathfrak{R}_n(\mathcal{F}), \qquad (1.43) $$

where:

$$ \mathfrak{R}_n(\mathcal{F}) = \mathbb{E}_S \left[ \hat{\mathfrak{R}}_S(\mathcal{F}) \right]. \qquad (1.44) $$

This is derived by first symmetrizing the sample and then invoking Jensen's inequality and the independence of $\sigma$. There are some Complexity Bounds that use Covering Numbers and Entropy that need to be discussed. In Metric Entropy, we let $\| \cdot \|_{\infty}$ be the metric on $\mathcal{F}$. The covering number $N(\epsilon, \mathcal{F}, \| \cdot \|_{\infty})$ satisfies:

$$ N(\epsilon, \mathcal{F}, \| \cdot \|_{\infty}) = \inf\{m \in \mathbb{N} : \exists\{f_1, \dots, f_m\} \subseteq \mathcal{F}, \forall f \in \mathcal{F}, \exists i, \|f - f_i\|_{\infty} \le \epsilon\}. \qquad (1.45) $$
Regarding the Dudley's Entropy Integral, For a bounded function class $\mathcal{F}$ (compact under $\|\cdot\|_{\infty}$):

$$ \mathfrak{R}_n(\mathcal{F}) \le \inf_{\alpha>0} \left( 4\alpha + \frac{12}{\sqrt{n}} \int_{\alpha}^{\infty} \sqrt{\log N(\epsilon, \mathcal{F}, \|\cdot\|_{\infty})} d\epsilon \right). \quad (1.46) $$

There is also a Relation to Talagrand's Concentration Inequality. Talagrand's inequality provides tail bounds for the supremum of empirical processes:

$$ \mathbb{P} \left( \sup_{f \in \mathcal{F}} |\mathbb{P}_n[f] - \mathcal{D}[f]| > \epsilon \right) \le 2 \exp \left( - \frac{n\epsilon^2}{2\|f\|_{\infty}^2} \right), \quad (1.47) $$

reinforcing the link between $\mathfrak{R}_n(\mathcal{F})$ and generalization. There are some Applications in Continuous

<figure><img src="image_6.png" /><figcaption>Figure 1.5: Rademacher Complexity: Linear vs Quadratic Function Classes</figcaption></figure>

Function Classes. One example is the RKHS with Gaussian Kernel. For $\mathcal{F}$ as the unit ball of an RKHS with kernel $k(x, x')$, the covering number satisfies:

$$ \log N(\epsilon, \mathcal{F}, \|\cdot\|_{\infty}) \sim O\left(\frac{1}{\epsilon^2}\right), \quad (1.48) $$

yielding:

$$ \mathfrak{R}_n(\mathcal{F}) \sim O\left(\frac{1}{\sqrt{n}}\right). \quad (1.49) $$

For $\mathcal{F} \subseteq H^s(\mathbb{R}^d)$, the covering number depends on the smoothness $s$ and dimension $d$:

$$ \mathfrak{R}_n(\mathcal{F}) \sim O\left(\frac{1}{n^{s/d}}\right). \quad (1.50) $$

Rademacher complexity is deeply embedded in modern empirical process theory. Its intricate relationship with measure-theoretic tools, symmetrization, and concentration inequalities provides a robust theoretical foundation for understanding generalization in high-dimensional spaces.
### 1.2.2.3 Python Code to generate Figures 1.4 illustrating Rademacher Complexity vs. $\frac{1}{\sqrt{n}}$ Bound

The Python code below produces the Figures 1.4 illustrating Rademacher Complexity vs. $\frac{1}{\sqrt{n}}$ Bound.

```python
"""
rademachercontinuous_comparison.py

Approximate and plot the Rademacher complexity for a simple continuous
space
with hypothesis class H = { h_w(x) = w * x, w ∈ [-1, 1] },
and compare against the theoretical O(1/sqrt(n)) decay.
"""

import numpy as np
import matplotlib.pyplot as plt

def rademacher_complexity(n, num_tries=500, numsigma=200):
    """
    Estimate emprical Rademacher complexity for sample size n.

   Args:
        n : int
            Number of sample points from [0,1].
        num_tries : int
            Number of Monte Carlo repetitions for averaging over S.
        numsigma : int
            Number of sigma (Rademacher vectors) draws per trial.

Returns:
    float : estimated Rademacher complexity

"""
complexities = []
for _ in range(num_tries):
    # Step 1: sample data points from [0,1]
    X = np.randomrand(n)

    # Step 2: compute emprical Rademacher complexity
    trial vals = []
    for _ in range(numsigma):
        sigma = np.random.choice([-1, 1], size=n)
        # Sup over w in [-1,1]: reduces to absolute average of
        sigma_i * x_i
        val = abs(np.sum sigma * X) / n
        trial vals.append(val)
    complexities.append(np.mean(trial vals))

return np.mean(complexities)

def plot rademacher complexity(maxn=100, step=5):
    ns = range(5, maxn+1, step)
    values = [rademacher complexity(n) for n in ns]

    # Theoretical O(1/sqrt(n)) curve (scaled to match emprical at first
    # point)
    c = values[0] * npsqrt(ns[0]) # scaling constant
```
```python
sigma = np.random.choice([-1, 1], size=n)

# coefficients for a, b, c
coef_a = np.sum sigma * (X**2)) / n
coef_b = np.sum sigma * X) / n
coef_c = np.sum sigma) / n

# supremum over a,b,c in [-1,1]: sum of absolute
coefficients
val = abs(coef_a) + abs(coef_b) + abs(coef_c)
trial vals.append(val)
complexities.append(np.mean(trial vals))
return np.mean(complexities)


def plot comparison(max_n=100, step=5):
    ns = range(5, max_n+1, step)

    # Compute emπrical complexities
    values linear = [rademacher complexity linear(n) for n in ns]
    values quadratic = [rademacher complexity quadratic(n) for n in ns]

    # Theoretical O(1/sqrt(n)) comparison (scaled)
    c lin = values linear[0] * np sqrt(ns[0])
    c quad = values quadratic[0] * np sqrt(ns[0])
    theo lin = [c lin / np sqrt(n) for n in ns]
    theo quad = [c quad / np sqrt(n) for n in ns]

    # Plot results
    plt figure(figsize=(9,6))
    plt plot(ns, values linear, 'o-b', label='Linear (empirical)')
    plt plot(ns, theo lin, '--b', label='Linear O(1/sqrt(n))')

    plt plot(ns, values quadratic, 'o-r', label='Quadratic (empirical)')
    plt plot(ns, theo quad, '--r', label='Quadratic O(1/sqrt(n))')

    plt title('Rademacher Complexity: Linear vs Quadratic Function
Classes')
    plt xlabel('Sample size n')
    plt ylabel('Rademacher complexity')
    plt grid(True,olinestyle=':')
    plt legend()
    plt show()

if __name__ == "__main__":
    plot comparison(max_n=100, step=5)
```

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

### 1.2.3 Sobolev Embeddings

#### 1.2.3.1 Literature Review of Sobolev Embeddings

Abderachid and Kenza (2024) [42] in their paper investigated fractional Sobolev spaces defined using Riemann-Liouville derivatives and studies their embedding properties. It establishes new continuous embeddings between these fractional spaces and classical Sobolev spaces, providing applications to PDEs. Giang et al. (2024) [43] introduced weighted Sobolev spaces and derived new Pólya-Szegő
type inequalities. These inequalities play a key role in establishing compact embedding results in function spaces equipped with weight functions. Ruiz and Fragkiadaki (2024) [44] provided a novel approach using Haar functions to revisit fractional Sobolev embedding theorems and demonstrated the algebra properties of fractional Sobolev spaces, which are essential in nonlinear analysis. Bilalov et al. (2025) [45] analyzed compact Sobolev embeddings in Banach function spaces, extending the classical Poincaré and Friedrichs inequalities to this setting and provided applications to function spaces used in modern PDE theory. Cheng and Shao (2025) [46] developed the weighted Sobolev compact embedding theorem for function spaces with unbounded radial potentials and used this result to prove the existence of ground state solutions for fractional Schrödinger-Poisson equations. Wei and Zhang (2025) [47] established a new embedding theorem tailored to variational problems arising in Schrödinger-Poisson equations and used Hardy-Sobolev embeddings to study the zero-mass case, an important case in quantum mechanics. Zhang and Qi (2025) [48] examined the compactness of Sobolev embeddings in the presence of small perturbations in quasilinear elliptic equations and proved multiple solution existence results using variational methods. Xiao and Yue (2025) [49] established a Sobolev embedding theorem for fractional Laplacian function spaces and applied the embedding results to image processing, particularly edge detection. Pesce and Portaro (2025) [50] studied intrinsic Hölder spaces and their connection to fractional Sobolev embeddings and established new embedding results for function spaces relevant to ultraparabolic operators.

### 1.2.3.2 Analysis of Sobolev Embeddings

The Sobolev embedding theorem states that:

$$W^{k,p}(\mathcal{X}) \hookrightarrow C^m(\mathcal{X}), \qquad (1.51)$$

if $k - \frac{d}{p} > m$, ensuring $f_{\theta} \in C^{\infty}(\mathcal{X})$ for smooth activations $\sigma$. For a function $u \in L^p(\Omega)$, its weak derivative $D^{\alpha}u$ satisfies:

$$\int_{\Omega} u(x) D^{\alpha} \phi(x) dx = (-1)^{|\alpha|} \int_{\Omega} v(x) \phi(x) dx \quad \forall \phi \in C_c^{\infty}(\Omega), \qquad (1.52)$$

where $v \in L^p(\Omega)$ is the weak derivative. This definition extends the classical notion of differentiation to functions that may not be pointwise differentiable. The Sobolev norm encapsulates both function values and their derivatives:

$$\|u\|_{W^{k,p}(\Omega)} = \left( \sum_{|\alpha| \le k} \|D^{\alpha}u\|_{L^p(\Omega)}^p \right)^{1/p} . \qquad (1.53)$$

Key properties:

* **Semi-norm Dominance:** The $W^{k,p}$-norm is controlled by the seminorm $|u|_{W^{k,p}}$, ensuring sensitivity to high-order derivatives.

* **Poincaré Inequality:** For $\Omega$ bounded, $u - u_{\Omega}$ satisfies:

$$\|u - u_{\Omega}\|_{L^p} \le C \|Du\|_{L^p}. \qquad (1.54)$$

Sobolev spaces $W^{k,p}(\Omega)$ embed into $L^q(\Omega)$ or $C^m(\bar{\Omega})$, depending on $k, p, q$, and $n$. These embeddings govern the smoothness and integrability of $u$ and its derivatives. There are several Advanced Theorems on Sobolev Embeddings. They are as follows:

1. **Sobolev Embedding Theorem:** Let $\Omega \subset \mathbb{R}^n$ be a bounded domain with Lipschitz boundary. Then:
<figure><img src="image_7.png" /><figcaption>Figure 1.6: Sobolev conjugate $q^* = np/(n-kp)$ ($k=1$)</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 1.7: Sobolev conjugate $q^* = np/(n-kp)$ ($k=2$)</figcaption></figure>

* If $k > n/p$, $W^{k,p}(\Omega) \hookrightarrow C^{m,\alpha}(\bar{\Omega})$ with $m = \lfloor k - n/p \rfloor$ and $\alpha = k - n/p - m$.

* If $k = n/p$, $W^{k,p}(\Omega) \hookrightarrow L^q(\Omega)$ for $q < \infty$.

* If $k < n/p$, $W^{k,p}(\Omega) \hookrightarrow L^q(\Omega)$ where $\frac{1}{q} = \frac{1}{p} - \frac{k}{n}$.

2. **Rellich-Kondrachov Compactness Theorem:** The embedding $W^{k,p}(\Omega) \hookrightarrow L^q(\Omega)$ is compact for $q < \frac{np}{n-kp}$. Compactness follows from:
(a) **Equicontinuity:** $W^{k,p}$-boundedness ensures uniform control over oscillations.

(b) **Rellich's Selection Principle:** Strong convergence follows from uniform estimates and tightness.

The Proof of Sobolev Embedding starts with the Scaling Analysis. Define $u_\lambda(x) = u(\lambda x)$. Then:

$$ \|u_\lambda\|_{L^p(\Omega)} = \lambda^{-n/p} \|u\|_{L^p(\lambda^{-1}\Omega)}. \qquad (1.55) $$

For derivatives:

$$ \|D^\alpha u_\lambda\|_{L^p(\Omega)} = \lambda^{|\alpha|-n/p} \|D^\alpha u\|_{L^p(\lambda^{-1}\Omega)}. \qquad (1.56) $$

The scaling relation $\lambda^{k-n/p}$ aligns with the Sobolev embedding condition $k > n/p$. Sobolev norms in $\mathbb{R}^n$ are equivalent to decay rates of Fourier coefficients:

$$ \|u\|_{W^{k,p}} \sim \left( \int_{\mathbb{R}^n} |\xi|^{2k} |\hat{u}(\xi)|^2 d\xi \right)^{1/2} . \qquad (1.57) $$

For $k > n/p$, Fourier decay implies uniform bounds, ensuring $u \in C^{m,\alpha}$. Interpolation spaces

<figure><img src="image_9.png" /><figcaption>Figure 1.8: Hölder exponent for embedding $W^1(\Omega) \hookrightarrow C^{0,\alpha}$</figcaption></figure>

bridge $L^p$ and $W^{k,p}$, providing finer embeddings. Duality: Sobolev embeddings are equivalent to boundedness of adjoint operators in $L^q$. For $-\Delta u = f$, $u \in W^{2,p}(\Omega)$ ensures

$$ u \in C^{0,\alpha}(\bar{\Omega}) \qquad (1.58) $$

if $p > n/2$. Sobolev spaces govern variational problems in geometry, e.g., minimal surfaces and harmonic maps. On $\Omega$ with fractal boundaries, trace theorems refine Sobolev embeddings.

The Sobolev Embedding Theorem provides a continuous embedding of the space $W^{k,p}(\Omega)$ into a Hölder space $C^{m,\alpha}(\bar{\Omega})$ when the regularity index exceeds the spatial dimension, specifically when $k - m > n/p$. The Hölder exponent $\alpha$ is not arbitrary but is given by the precise fractional part of this index. The fundamental equation is

$$ \alpha = k - m - \frac{n}{p} \qquad (1.59) $$
<figure><img src="image_10.png" /><figcaption>Figure 1.9: Hölder exponent for embedding $W^2(\Omega) \hookrightarrow C^{0,\alpha}$</figcaption></figure>

but this holds strictly only if this value is less than 1. The general formulation is

$$ \alpha = \min \left( 1, k - m - \frac{n}{p} \right) \qquad (1.60) $$

provided $k - m - \frac{n}{p} > 0$. This relation is derived from scaling invariance properties of the underlying Sobolev norms. Consider the dilation $u_{\lambda}(x) = u(\lambda x)$. The Sobolev norm scales as

$$ \|D^m u_{\lambda}\|_{L^p} = \lambda^{m-n/p} \|D^m u\|_{L^p} \qquad (1.61) $$

while the Hölder seminorm scales as

$$ [u_{\lambda}]_{C^{m,\alpha}} = \lambda^{m+\alpha} [u]_{C^{m,\alpha}} \qquad (1.62) $$

For an embedding $W^{k,p} \hookrightarrow C^{m,\alpha}$ to be scale-invariant, the scaling exponents must be consistent across the inequality

$$ [u]_{C^{m,\alpha}} \leq C \|u\|_{W^{k,p}} \qquad (1.63) $$

which forces the condition

$$ m + \alpha = k - \frac{n}{p} \qquad (1.64) $$

leading directly to the identification

$$ \alpha = k - \frac{n}{p} - m \qquad (1.65) $$

The optimality of this exponent is rigorously demonstrated by examining borderline functions that saturate the Sobolev inequalities. A canonical example is the function $u(x) = |x|^{\beta}$ defined on a bounded domain containing the origin. Its membership in $W^{k,p}$ is determined by the integrability of its derivatives. The $L^p$ norm of the $k$-th derivative involves integrating

$$ |D^k u(x)|^p \approx |x|^{(\beta-k)p} \qquad (1.66) $$
In $n$ dimensions, using spherical coordinates, this integral is finite near zero if and only if

$$ \int_0^1 r^{(\beta-k)p} r^{n-1} dr = \int_0^1 r^{(\beta-k)p+n-1} dr < \infty \quad (1.67) $$

which requires $(\beta - k)p + n > 0$, or equivalently, $\beta > k - \frac{n}{p}$. Conversely, the Hölder continuity of $u(x)$ is determined by the ratio $\frac{|u(x)-u(y)|}{|x-y|^{\alpha}}$. For $u(x) = |x|^{\beta}$, this seminorm is finite if and only if $\beta \ge \alpha$. The limiting case $\beta = k - n/p$ shows that a function can be in $W^{k,p}$ yet fail to be Hölder continuous for any exponent greater than $\beta$. This is formalized by considering the function

$$ u(x) = |x|^{\beta} |\log|x||^{\gamma} \quad (1.68) $$

where a detailed calculation shows that for $\beta = k - n/p$, the function belongs to $W^{k,p}$ only if $\gamma p < -1$, but its Hölder seminorm $[u]_{C^{0,\beta}}$ is infinite, proving that the embedding

$$ W^{k,p} \not\subset C^{0,\beta} \quad (1.69) $$

for the critical exponent $\beta$, but holds for any $\alpha < \beta$.

The quantitative control offered by the Hölder exponent is expressed through the embedding inequality $\|u\|_{C^{m,\alpha}} \le C\|u\|_{W^{k,p}}$, where the norm on the left is defined as

$$ \|u\|_{C^{m,\alpha}} = \max \sup_{|\gamma| \le m} |D^\gamma u(x)| + \max \sup_{|\gamma|=m} \frac{|D^\gamma u(x) - D^\gamma u(y)|}{|x-y|^\alpha} \quad (1.70) $$

The constant $C$ depends on $k, p, n, \dots$, and the domain $\Omega$. The mathematical rigor of the exponent $\alpha$ is further highlighted in the Gagliardo-Nirenberg interpolation inequalities, which provide estimates of the form

$$ \|D^j u\|_{L^r} \le C \|D^k u\|_{L^p}^{\theta} \|u\|_{L^q}^{1-\theta} \quad (1.71) $$

where the parameters satisfy

$$ \frac{1}{r} = \frac{j}{n} + \theta \left( \frac{1}{p} - \frac{k}{n} \right) + (1 - \theta) \frac{1}{q} \quad (1.72) $$

and

$$ \frac{j}{k} \le \theta \le 1 \quad (1.73) $$

Setting $j = 0, q = \infty$, and seeking continuity implies $\frac{1}{r} > 0$, which recovers the condition $k > n/p$. The Hölder exponent emerges when estimating the modulus of continuity directly from the $W^{k,p}$-norm via the integral representation of a function through its derivatives, such as using the Fourier transform or convolution with a mollifier, leading to pointwise estimates like

$$ |u(x) - u(y)| \le C|x - y|^{1-n/p} \|Du\|_{L^p} \quad (1.74) $$

for $p > n$, where

$$ \alpha = 1 - n/p \quad (1.75) $$

For higher derivatives, the general formula

$$ \alpha = k - m - n/p \quad (1.76) $$

consistently appears, governing the precise degree of classical regularity attainable from weak solutions. This makes the Hölder exponent a fundamental bridge between the measurable, integral-based world of Sobolev spaces and the pointwise, geometric world of classical analysis.
### 1.2.3.3 Python Code to generate Figures 1.6, 1.7, 1.8, and 1.9 illustrating Sobolev Embeddings

The Python code below produces the Figures 1.6, 1.7, 1.8, and 1.9 illustrating Sobolev Embeddings.

```python
"""
sobolev visualisation.py

Two separate figures illustrating key parts of the Sobolev Embedding
Theorem:

- Figure A: Sobolev conjugate q* = n p / (n - k p) when k p < n.
  Shades the admissible region q <= q*(p).
- Figure B: Holder exponent alpha = k - n/p for the continuous embedding
  when k p > n.

"""

import numpy as np
import matplotlib.pyplot as plt

def q_star(p, n, k):
    """
    Sobolev conjugate q* = n p / (n - k p) for kp < n.
    Return np.inf where kp >= n.
    """
    denom = n - k * p
    q = np FullLike(p, np inf, dtype = float)
    mask = denom > 0
    q[mask] = (n * p[mask]) / denom[mask]
    return q

def holder alpha(p, n, k):
    """
    Holder exponent alpha = k - n/p (valid when k - n/p in (0,1],
    i.e., when kp > n but small enough that fractional part is in (0,1])
    """
    return k - n / p

def plot sobolev conjugate(p_range, dims=(1,2,3), ks=(1,2), qcap=1e4):
    """
    Produce Figure A: q* vs p for different n and k.
    p_range: array of p values to evaluate
    dims: tuple/list of space dimensions n to plot
    ks: list of k values (smoothness order) to plot
    qcap: finite cap for plotting where q* -> inf (for visualization)
    """
    for k in ks:
        plt figure (figsize=(8,5))
    for n in dims:
        p = p_range
        q = q_star(p, n, k)
        # For plotting, cap extremely large q values (visualization
        only)
        q plot = np minimum(q, qcap)
        plt plot(p, q plot, label = f'n={n}, k={k}')
```
```python
# Shade region q <= q*(p)
plt.fill_between(p, 0, q_plot, alpha=0.12)

# mark the critical p0 = n/k where q* -> inf
pcrit = n / k
plt.axvline(pcrit, linewidth='--', alpha=0.6, color='gray')
plt.text(pcrit*1.01, plt.ylim()[1]*0.6, f'p=n/k={pcrit:.2f}',
         rotation=90,
         alpha=0.7, va='center', fontsize=9)

plt.ylim(bottom=0)
plt.yscale('linear')  # linear is fine; if qcap large, consider
                     log scale

plt.xlabel('p')

plt.ylabel('q* (Sobolev conjugate, capped for visualization)')

plt.title(f'Sobolev conjugate $q^*:=n p/(n-k p)$ ($k = \{k\}$)')

# plt.legend()

plt.grid(True, linewidth=':', alpha=0.6)

plt.tight_layout()

plt.show()


def plot_hold exponent(p_range, dims=(1,2,3), ks=(1,2)):
    """
    Produce Figure B: Holder exponent alpha = k - n/p for p > n/k.
    """
    for k in ks:
        plt.figure(figsize=(8,5))
        for n in dims:
            p = p_range
            alpha = holder_alpha(p, n, k)
            # Only plot region where kp > n (i.e., p > n/k)
            mask = p > (n / k)
            if mask.any():
                plt plot(p[mask], alpha[mask], label=f'n={n}, k={k}')
                # Shade where alpha > 0 (continuous embedding)
                plt.fill_between(p[mask], 0, alpha[mask], alpha=0.12)
                # annotate typical small-p region
                p crit = n / k
                plt axvline(p crit, linewidth='--', alpha=0.5, color='gray')
                plt text(p crit*1.01, 0.02, f'p=n/k={pcrit:.2f}',
                         rotation=90,
                         alpha=0.7, va='bottom', fontsize=9)
            axhline(0, color='k', linewidth=0.6, alpha=0.6)
            axlabel('p')
            axlabel(r'$\alpha = k - \frac{n}{p}$ (Holder exponent)')
            title(f'Holder exponent for embedding $W^{\alpha}(\Omega)$ \
                   hookrightarrow $C^{\{0,\alpha\}}$ ($k = \{k\}$)')
            # plt legend()
            plt grid(True, linewidth=':', alpha=0.6)
            plt.tight_layout()
            plt show()

if __name__ == "__main__":
```
```python
# parameter grid for p (avoid p at critical point exactly where
 denom=0)
p vals = np. linspace (1.01, 50, 1000)

# Figure A: q* vs p for k=1 and k=2 (separate figures)
plot_sobolev_conjugate(p vals, dims=(1,2,3), ks=(1,2), qcap=5e2)

# Figure B: Holder exponent alpha vs p for p > n/k
plot_holdexponent(p vals, dims=(1,2,3), ks=(1,2))
```

### 1.2.4 Rellich-Kondrachov Compactness Theorem

The **Rellich-Kondrachov Compactness Theorem** is one of the most fundamental and deep results in the theory of Sobolev spaces, particularly in the study of functional analysis and the theory of partial differential equations. The theorem asserts the compactness of certain Sobolev embeddings under appropriate conditions on the domain and the function spaces involved. This result is of immense significance in mathematical analysis because it provides a rigorous justification for the fact that bounded sequences in Sobolev spaces, under certain conditions, have strongly convergent subsequences in lower-order normed spaces. In essence, the theorem states that while weak convergence in Sobolev spaces is relatively straightforward due to the Banach-Alaoglu theorem, strong convergence is not always guaranteed. However, under the assumptions of the Rellich-Kondrachov theorem, strong convergence in $L^q(\Omega)$ can indeed be obtained from boundedness in $W^{1,p}(\Omega)$. The compactness property ensured by this theorem is much stronger than mere boundedness or weak convergence and plays a crucial role in proving the existence of solutions to variational problems by ensuring that minimizing sequences possess convergent subsequences in an appropriate function space. The theorem can also be viewed as a generalization of the classical **Arzelà–Ascoli theorem**, extending compactness results to function spaces that involve derivatives.

#### 1.2.4.1 Literature Review of Rellich-Kondrachov Compactness Theorem

Lassoued (2026) [51] examined function spaces on the torus and their lack of compactness, highlighting cases where the classical Rellich-Kondrachov result fails. He extended compact embedding results to function spaces with periodic structures. He also discussed trace theorems and regular function spaces in this new context. Chen et al. (2024) [52] extended the Rellich-Kondrachov theorem to Hörmander vector fields, a class of differential operators that appear in hypoelliptic PDEs. They established a degenerate compact embedding theorem, generalizing previous results in the field. They also provided applications to geometric inequalities, highlighting the role of compact embeddings in PDE theory. Adams and Fournier (2003) [53] in their book provided a complete proof of the Rellich-Kondrachov theorem, along with a discussion of compact embeddings. They also covered function space theory, embedding theorems, and applications in PDEs. Brezis (2010) [55] wrote a highly recommended resource for understanding Sobolev spaces and their compactness properties. The book included applications to variational methods and weak solutions of PDEs. Evans (2022) [56] in his classic PDE textbook includes a discussion of compact Sobolev embeddings, their implications for weak convergence, and applications in variational methods. Maz'ya (2011) [57] provided a detailed treatment of Sobolev space theory, including compact embedding theorems in various settings.

#### 1.2.4.2 Analysis of Rellich-Kondrachov Compactness Theorem

To rigorously state the theorem, we consider a bounded open domain $\Omega \subset \mathbb{R}^n$ with a **Lipschitz boundary**. For $1 \le p < n$, the theorem asserts that the embedding

$$ W^{1,p}(\Omega) \hookrightarrow L^q(\Omega) \qquad (1.77) $$