import numpy as np
import matplotlib.pyplot as plt
from sklearn.decomposition import FastICA

# Seed for reproducibility
np.random.seed(0)

# Generate sample signals
n Samples = 1000
time = np.linspace(0, 8, n samples)

s1 = np.sin(2 * time) # Signal 1 : sinusoidal
s2 = np.sign(np.sin(3 * time)) # Signal 2 : square signal
s3 = np.random哦. normal (size=n samples) # Signal 3 : Gaussian noise

S = np.c_[s1, s2, s3]

# 混合各信号
A = np. array( [[1, 1, 1], [0.5, 2, 1.0], [1.5, 1.0, 2.0]]) # Mixing matrix
X = S. dot(A.T)

# Perform ICA
ica = FastICA(n_components=3)
S_ = ica. fittransform(X) # Reconstruct signals
A_ = ica. mixing_ # Estimated mixing matrix

# Plot results
fig. size=(10, 8));
models = [X, S, S_]
names = ['Mixed signals (observed)', 'True Sources', 'ICA Recovered Signals']

for i, (model, name) in enumerate (zip (models, names), 1):
    fig. subp lot(3, 1, i)
    for sig in model.T:
        fig. plot (time, sig)
    fig. title (name)
    fig. tight layout()

fig. show();

16.2 Supervised Learning

16.2.1 Literature Review of Supervised Learning

Supervised learning has been extensively studied through rigorous theoretical and empirical ad-
vancements, forming the backbone of modern machine learning. One of the foundational contribu-
tions in this domain is Vapnik (1995) [143], which introduced statistical learning theory, establishing
key concepts such as the Vapnik-Chervonenkis (VC) dimension, which quantifies the capacity of
a hypothesis class to shatter a given set of data points. The book also presents the principle of structural risk minimization (SRM), which provides a theoretical framework for balancing model complexity and empirical error, thereby preventing overfitting. A critical outcome of this work was the formulation of support vector machines (SVMs), which use kernel functions to transform data into higher-dimensional spaces, enabling the discovery of nonlinear decision boundaries while maintaining strong generalization guarantees. The kernel trick, introduced as a mechanism to implicitly compute inner products in high-dimensional feature spaces, remains one of the most powerful techniques in modern machine learning. Building on statistical foundations, Bishop (2006) [124] provided an extensive treatment of probabilistic models in supervised learning, with a strong emphasis on Bayesian inference. The text rigorously derives Bayesian linear regression, Gaussian processes, and probabilistic neural networks, offering a structured mathematical perspective on uncertainty quantification. The discussion of mixture models and the Expectation-Maximization (EM) algorithm in this work has had profound implications for learning in both supervised and unsupervised settings.

The field of ensemble learning has also seen significant advancements through rigorous mathematical formulations. Breiman (2001) [754] introduced the Random Forest algorithm, an ensemble method based on decision trees, and demonstrated how bootstrap aggregation (bagging) reduces variance in predictive models. The paper provided a theoretical justification for the effectiveness of decorrelating individual decision trees using random feature selection, showing that this mechanism enhances generalization. Moreover, Breiman formally defined the out-of-bag (OOB) error estimation method, an internal validation technique that provides an unbiased estimate of the model's performance without requiring a separate validation set. Around the same time, Friedman, Hastie, and Tibshirani (2000) [1056] offered a rigorous statistical interpretation of boosting algorithms, particularly AdaBoost. Their work demonstrated that boosting can be understood as a stagewise optimization process that minimizes an exponential loss function, providing a function-space perspective on gradient boosting. By linking boosting to numerical optimization techniques, they laid the foundation for gradient boosting machines (GBMs), which remain one of the most effective supervised learning techniques today. Further theoretical contributions to boosting were made by Schapire (1990) [1058], who formally proved that weak classifiers, which perform only marginally better than random guessing, can be transformed into arbitrarily strong classifiers through the process of boosting. The paper established a precise mathematical framework for analyzing how iterative reweighting of training samples reduces error bounds, reinforcing the statistical robustness of boosting methods.

Deep learning, a subset of supervised learning, has also benefited from rigorous theoretical advancements. LeCun et al. (1998) [1055] introduced convolutional neural networks (CNNs), demonstrating their effectiveness in document recognition, particularly for handwritten digit classification in the MNIST dataset. The authors provided a mathematically rigorous derivation of the backpropagation algorithm and the gradient-based optimization techniques used in training deep networks. One of their major contributions was the concept of weight sharing, which significantly reduces the number of learnable parameters in CNNs by enforcing translational invariance in feature detection. This architecture laid the groundwork for modern deep learning-based vision systems, which have since expanded to complex image recognition tasks. More recently, Srivastava et al. (2014) [141] introduced dropout as a regularization technique for deep neural networks. The authors provided a probabilistic interpretation of dropout as an approximation to model averaging over an exponentially large ensemble of subnetworks, rigorously deriving its impact on reducing overfitting. The paper also presented empirical results demonstrating that dropout significantly improves generalization performance across a wide range of supervised learning tasks.

One of the earliest contributions to supervised learning, which laid the foundation for neural net-
works, was made by Rosenblatt (1958) [1057] through the introduction of the perceptron algorithm. This work rigorously proved the perceptron convergence theorem, which guarantees that if a dataset is linearly separable, the perceptron learning rule will converge to a separating hyperplane in a finite number of iterations. Although the perceptron was later shown to be limited in expressive power, particularly in its inability to solve non-linearly separable problems such as the XOR problem, it inspired the development of more advanced architectures, including multi-layer perceptrons (MLPs) and deep neural networks. Hastie, Tibshirani, and Friedman (2009) [139] provided one of the most mathematically rigorous treatments of supervised learning, covering a broad range of techniques including linear models, kernel methods, decision trees, ensemble learning, and deep learning. Their work emphasized the mathematical foundations of machine learning, particularly the bias-variance tradeoff, regularization methods such as ridge regression and Lasso, and kernelized learning methods. The authors also provided in-depth theoretical analyses of the convergence properties of various supervised learning algorithms, making their text a fundamental resource for statistical learning theory.

The optimization of deep learning models has been another area of rigorous mathematical research, with Kingma and Ba (2014) [176] introducing the Adam optimization algorithm. Their work provided a detailed mathematical derivation of Adam's update rules, which rely on the computation of exponentially decaying moving averages of past gradients and squared gradients. The authors rigorously demonstrated how Adam effectively combines the benefits of Adagrad, which adapts learning rates based on the historical sum of squared gradients, and RMSProp, which normalizes updates using an exponentially weighted average of past squared gradients. The method has since become the default optimization algorithm for training deep neural networks, particularly due to its robustness in handling sparse gradients and noisy objective functions. Collectively, these works have profoundly shaped the landscape of supervised learning by introducing mathematically rigorous frameworks for classification, regression, deep learning, and optimization. Their theoretical contributions continue to inform modern advancements, ensuring that machine learning models remain both statistically sound and computationally efficient.

<table><caption>Table 16.15: Summary of Contributions in Supervised Learning</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Vapnik (1995) [143]</td><td>Introduced statistical learning theory, including the Vapnik-Chervonenkis (VC) dimension, structural risk minimization (SRM), and support vector machines (SVMs). Established the kernel trick for computing inner products in high-dimensional feature spaces.</td></tr><tr><td>Bishop (2006) [124]</td><td>Provided a probabilistic treatment of supervised learning, including Bayesian inference, Gaussian processes, Bayesian linear regression, and probabilistic neural networks. Discussed mixture models and the Expectation-Maximization (EM) algorithm.</td></tr><tr><td>Breiman (2001) [754]</td><td>Introduced Random Forests and bootstrap aggregation (bagging) for variance reduction. Theoretical justification of random feature selection and the out-of-bag (OOB) error estimation method.</td></tr><tr><td>Friedman, Hastie, and Tibshirani (2000) [1056]</td><td>Provided a statistical interpretation of boosting, demonstrating its connection to stagewise optimization of an exponential loss function. Established the foundation for gradient boosting machines (GBMs).</td></tr></tbody></table>
<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Schapire (1990) [1058]</td><td>Proved that weak classifiers can be transformed into strong classifiers through boosting. Developed a mathematical framework for iterative sample reweighting to reduce error bounds.</td></tr><tr><td>LeCun et al. (1998) [1055]</td><td>Introduced convolutional neural networks (CNNs) with applications to handwritten digit recognition. Provided rigorous derivations of backpropagation and weight sharing for reducing parameters.</td></tr><tr><td>Srivastava et al. (2014) [141]</td><td>Proposed dropout as a regularization technique in deep learning, providing a probabilistic interpretation as model averaging. Demonstrated improvements in generalization performance.</td></tr><tr><td>Rosenblatt (1958) [1057]</td><td>Introduced the perceptron algorithm and proved the perceptron convergence theorem for linearly separable data. Inspired the development of multi-layer perceptrons (MLPs) and deep networks.</td></tr><tr><td>Hastie, Tibshirani, and Friedman (2009) [139]</td><td>Provided a rigorous mathematical treatment of supervised learning, including regularization methods (ridge regression, Lasso), bias-variance tradeoff, kernel methods, decision trees, and ensemble learning. Theoretical analyses of algorithm convergence properties.</td></tr><tr><td>Kingma and Ba (2014) [176]</td><td>Developed the Adam optimization algorithm, combining Adam and RMSProp principles. Provided a mathematical derivation of Adam's update rules for adaptive learning rates in deep learning models.</td></tr></tbody></table>

## 16.2.2 Recent Literature Review of Supervised Learning

Supervised learning has found extensive applications across multiple domains, enhancing decision-making, classification, and predictive analytics through well-labeled datasets. Raikwar and Gupta (2025) [1046] developed an AI-driven trust management framework that integrates both supervised and unsupervised learning to classify security levels in wireless ad hoc networks. Their model improves the robustness of decentralized communication by leveraging machine learning-based trust computation, enhancing the detection of malicious nodes and mitigating vulnerabilities in self-organizing networks. Similarly, Rafiei et al. (2025) [1059] employed supervised multi-output classification models, including Random Forest and Support Vector Machines, to optimize lipid nanoparticle design for mRNA delivery. Their research demonstrates how machine learning can refine drug formulation by predicting the best combination of lipid components to enhance delivery efficiency. This showcases the growing role of supervised learning in biotechnology and precision medicine.

In the field of remote sensing and agriculture, Pei et al. (2025) [1060] introduced a weakly supervised learning approach for segmenting vegetation from UAV images. Traditional supervised models often struggle with limited labeled datasets in complex outdoor environments, but their model incorporates spectral reconstruction techniques to improve classification accuracy in field conditions. This advancement significantly enhances agricultural monitoring and precision farming. Likewise, Efendi et al. (2025) [1061] designed an IoT-based health monitoring system where supervised learning algorithms were used to classify and predict health anomalies in elderly patients. Their study highlights how cloud computing and machine learning can be combined to
provide real-time health insights, ultimately enabling better remote patient care.

Another critical challenge in supervised learning is the scarcity of labeled data, which was addressed by Pang et al. (2025) [1062] in their research on protein transition pathway prediction. They introduced DeepPath, a framework that integrates active learning with supervised deep learning, allowing models to efficiently identify uncertain data points for labeling. This method enhances model accuracy while reducing the dependency on extensive labeled datasets, making it particularly useful for complex biological simulations. In a different application, Curry et al. (2025) [1063] explored supervised classification techniques in geoscience, comparing their effectiveness with unsupervised clustering methods for analyzing ignimbrite flare-up patterns. Their work provides insights into how machine learning models can be optimized for geological pattern detection, improving hazard assessment and geological forecasting.

In the realm of computational drug discovery, Li et al. (2025) [1064] developed a deep learning-based framework called π-PhenoDrug, which employs supervised learning for phenotypic drug screening. The model utilizes transfer learning strategies and neural networks to classify drug interactions, significantly accelerating the identification of promising drug candidates. Similarly, Liu et al. (2025) [1065] integrated supervised learning with molecular docking and dynamic simulations to identify ASGR1 and HMGCR dual-target inhibitors. By combining computational chemistry with machine learning, they were able to streamline the drug discovery process, demonstrating the efficiency of supervised models in pharmaceutical research. These contributions highlight the transformative potential of supervised learning in biomedical applications.

Beyond healthcare and science, supervised learning is also playing a crucial role in business and engineering. Dutta and Karmakar (2025) [1067] investigated the application of the Random Forest algorithm in business analytics, demonstrating its superior accuracy in predictive modeling compared to other machine learning approaches. Their findings illustrate how machine learning can optimize decision-making processes in organizational contexts. Finally, Ekanayake (2025) [1054] explored the use of supervised deep learning models for enhancing Magnetic Resonance Imaging (MRI) reconstruction and super-resolution. Their study addresses the challenges of lengthy MRI scan times by leveraging artificial intelligence to improve image quality, thereby advancing medical diagnostics and reducing patient discomfort.

These studies collectively demonstrate the broad impact of supervised learning across various disciplines, from security and healthcare to remote sensing, drug discovery, and business analytics. The integration of machine learning with domain-specific challenges has led to significant breakthroughs, highlighting the adaptability and efficiency of supervised learning models. As researchers continue to refine these techniques, the future of artificial intelligence-driven decision-making appears increasingly promising. The ability of supervised learning to extract meaningful patterns from structured data is proving invaluable in solving real-world problems, paving the way for further advancements in machine learning research and its practical applications.

<table><caption>Table 16.16: Summary of Recent Contributions in Supervised Learning</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Raikwar and Gupta (2025) [1046]</td><td>Developed an AI-driven trust management framework integrating supervised and unsupervised learning to classify security levels in wireless ad hoc networks. Enhanced decentralized communication robustness and improved detection of malicious nodes.</td></tr></tbody></table>

*Continued on next page*
*Continued from previous page*

<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Rafiei et al. (2025) [1059]</td><td>Applied supervised multi-output classification models, including Random Forest and SVMs, to optimize lipid nanoparticle design for mRNA delivery, improving drug formulation predictions.</td></tr><tr><td>Pei et al. (2025) [1060]</td><td>Proposed a weakly supervised learning approach for segmenting vegetation from UAV images, enhancing classification accuracy in precision farming.</td></tr><tr><td>Efendi et al. (2025) [1061]</td><td>Designed an IoT-based health monitoring system that uses supervised learning algorithms to classify and predict health anomalies in elderly patients, improving remote patient care.</td></tr><tr><td>Pang et al. (2025) [1062]</td><td>Introduced DeepPath, a supervised deep learning framework integrating active learning for protein transition pathway prediction, reducing dependency on extensive labeled datasets.</td></tr><tr><td>Curry et al. (2025) [1063]</td><td>Compared supervised classification techniques and unsupervised clustering in geoscience, optimizing machine learning models for geological pattern detection and hazard assessment.</td></tr><tr><td>Li et al. (2025) [1064]</td><td>Developed π-PhenoDrug, a deep learning-based framework employing supervised learning for phenotypic drug screening, accelerating drug candidate identification.</td></tr><tr><td>Liu et al. (2025) [1065]</td><td>Integrated supervised learning with molecular docking and simulations to identify ASGR1 and HMGCR dual-target inhibitors, streamlining drug discovery.</td></tr><tr><td>Dutta and Karmakar (2025) [1067]</td><td>Investigated Random Forest applications in business analytics, demonstrating superior predictive modeling accuracy for optimizing organizational decision-making.</td></tr><tr><td>Ekanayake (2025) [1054]</td><td>Applied supervised deep learning models to enhance MRI reconstruction and super-resolution, improving medical diagnostics by reducing scan times.</td></tr></tbody></table>

### 16.2.3 Mathematical Analysis of Supervised Learning

Supervised learning is a fundamental paradigm in machine learning wherein a model is trained using labeled data, consisting of input-output pairs $\{(x_i, y_i)\}_{i=1}^N$. Here, $x_i \in \mathbb{R}^d$ represents the feature vector of the $i$th data point, and $y_i$ represents the corresponding label, which can be either continuous (regression) or discrete (classification). The goal of supervised learning is to learn a function $f: \mathbb{R}^d \to \mathbb{R}$ or $f: \mathbb{R}^d \to \mathcal{Y}$ such that for an unseen input $\mathbf{x}$, the model produces an accurate prediction $\hat{y} = f(\mathbf{x})$. Mathematically, the model aims to approximate the conditional probability distribution $P(Y | \mathbf{X})$, i.e.,

$$ \hat{y} = f(\mathbf{x}) \approx \mathbb{E}[Y | \mathbf{X} = \mathbf{x}] \qquad (16.78) $$

where the expectation is taken with respect to the true underlying data distribution $P(\mathbf{X}, Y)$. The optimal function $f^*$ minimizes the expected loss over the data distribution,

$$ f^* = \arg \min_{f \in \mathcal{F}} \mathbb{E}_{(\mathbf{x}, Y) \sim P} [\ell(Y, f(\mathbf{X}))] \qquad (16.79) $$

where $\mathcal{F}$ denotes the hypothesis space of functions and $\ell(y, \hat{y})$ is a loss function quantifying the error between the true and predicted values. In empirical risk minimization (ERM), the expectation is
<figure><img src="image_175.png" /><figcaption>Figure 16.7: Illustration of Supervised Learning in regression. Blue points represent training data, the dashed green line is the true underlying function, and the red line is the model learned from data</figcaption></figure>

approximated using the training set, leading to

$$ \hat{f} = \arg \min_{f \in \mathcal{F}} \frac{1}{N} \sum_{i=1}^{N} \ell(y_i, f(\mathbf{x}_i)). \qquad (16.80) $$

For regression tasks, common choices for $\ell(y, \hat{y})$ include the squared loss,

$$ \ell(y, \hat{y}) = (y - \hat{y})^2, \qquad (16.81) $$

which leads to the minimization of the mean squared error (MSE),

$$ \hat{f} = \arg \min_{f \in \mathcal{F}} \frac{1}{N} \sum_{i=1}^{N} (y_i - f(\mathbf{x}_i))^2. \qquad (16.82) $$

For classification, a frequently used loss function is the cross-entropy loss for a model outputting class probabilities $p_k(\mathbf{x})$,

$$ \ell(y, \hat{y}) = - \sum_{k=1}^{K} \mathbb{1}(y = k) \log p_k(\mathbf{x}), \qquad (16.83) $$

where K is the number of classes and $\mathbb{1}(y = k)$ is the indicator function that is 1 if $y = k$ and 0 otherwise. The classifier aims to minimize the empirical risk,

$$ \hat{f} = \arg \min_{f \in \mathcal{F}} - \frac{1}{N} \sum_{i=1}^{N} \sum_{k=1}^{K} \mathbb{1}(y_i = k) \log p_k(\mathbf{x}_i). \qquad (16.84) $$
To optimize $\hat{f}$, gradient-based methods are commonly used, particularly in deep learning where $f$ is parameterized by a neural network with weights $\theta$,

$$f(\mathbf{x}; \theta) = \sigma(W_L \sigma(W_{L-1} \cdots \sigma(W_1 \mathbf{x} + b_1) + b_{L-1}) + b_L), \quad (16.85)$$

where $W_l$ and $b_l$ are weight matrices and bias vectors at layer $l$, and $\sigma(\cdot)$ is an activation function. The optimization process follows stochastic gradient descent (SGD), updating parameters iteratively as follows,

$$\theta^{(t+1)} = \theta^{(t)} - \eta \nabla_{\theta} \mathcal{L}(\theta), \quad (16.86)$$

where $\eta$ is the learning rate and $\mathcal{L}(\theta)$ is the loss function. The gradient $\nabla_{\theta} \mathcal{L}(\theta)$ is computed via backpropagation,

$$\frac{\partial \mathcal{L}}{\partial \theta} = \sum_{i=1}^{N} \frac{\partial \ell(y_i, f(\mathbf{x}_i; \theta))}{\partial \theta}. \quad (16.87)$$

Regularization techniques, such as L2 regularization (ridge regression),

$$\mathcal{L}(\theta) = \frac{1}{N} \sum_{i=1}^{N} (y_i - f(\mathbf{x}_i; \theta))^2 + \lambda \|\theta\|_2^2, \quad (16.88)$$

are used to prevent overfitting by penalizing large weights. In logistic regression, the probability of a binary label $y \in \{0, 1\}$ is modeled using the sigmoid function,

$$p(y = 1 | \mathbf{x}; \theta) = \frac{1}{1 + e^{-\mathbf{x}^T \theta}}. \quad (16.89)$$

The loss function in this case is the binary cross-entropy loss,

$$\mathcal{L}(\theta) = -\frac{1}{N} \sum_{i=1}^{N} [y_i \log p(y_i) + (1 - y_i) \log(1 - p(y_i))]. \quad (16.90)$$

Thus, supervised learning encompasses various frameworks, each employing rigorous mathematical formulations to optimize predictive performance.

### 16.2.4 Python Code to generate Figure 16.7 illustrating the Supervised Learning in Regression

The Python code below produces the Figure 16.7 illustrating the Supervised Learning in Regression.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

# Generate synthetic data
np.random.seed(0)
X = 2 * np.randomrand(50, 1)
y = 4 + 3 * X + np.randomrand(50, 1)  # true relation y = 4 + 3x + noise

# Train a supervised learning model (linear regression)
model = LinearRegression()
model.fit(X, y)
y_pred = modelPredict(X)

# Create line for visualization
```
```python
16 X_new = np.linspace(0, 2, 100).reshape(100, 1)
17 y_new = model Predict(X_new)
18
19 # Plot
20 plt.figure(figsize=(8, 6))
21 plt. scatter(X, y, color='blue', label="Training Data")
22 plt. plot(X_new, y_new, color='red', linewidth=2, label="Learned Model")
23 plt. plot(X_new, 4 + 3*X_new, color='green', linewidth=2, label="True Function")
24
25 plt. xlabel("Feature (X)")
26 plt. ylabel("Target (y)")
27 plt. title("Supervised Learning: Regression Example")
28 plt. legend()
29 plt. grid(True)
30 plt. tight_layout()
31 plt. show()
```

## 16.2.5 Bias-Variance Tradeoff of Supervised Learning

### 16.2.5.1 Literature Review of Bias-Variance Tradeoff of Supervised Learning

The bias-variance tradeoff is a crucial concept in supervised learning, dictating how model complexity impacts predictive performance. Geman, Bienenstock, and Doursat (1992) [882] introduced the bias-variance decomposition in the context of neural networks, rigorously establishing that overly simple models suffer from high bias due to underfitting, while excessively complex models exhibit high variance due to overfitting. This foundational work provided a theoretical framework that has guided decades of machine learning research. Hastie, Tibshirani, and Friedman (2001) [139] further expanded upon this concept in their comprehensive textbook, where they provided mathematical formulations, empirical analyses, and practical implications of the tradeoff across various learning algorithms. Their work remains one of the most widely referenced sources for understanding the balance between bias and variance in real-world applications. More recently, Belkin et al. (2019) [883] challenged the classical understanding of the tradeoff by introducing the "double descent" risk curve, demonstrating that in highly over-parameterized models, test error initially increases but eventually decreases beyond a critical complexity threshold. This counterintuitive result, grounded in empirical evidence and theoretical analysis, reshaped the conventional wisdom regarding model selection and generalization in modern machine learning.

In the context of deep learning, Neal et al. (2018) [884] revisited the bias-variance tradeoff, rigorously analyzing why larger neural networks often achieve superior generalization despite their over-parameterization. They argued that implicit regularization, induced by optimization algorithms such as stochastic gradient descent, plays a crucial role in preventing excessive variance growth. Rocks and Mehta (2020) [885] further investigated over-parameterized models using statistical physics methods, deriving explicit analytical expressions for bias and variance in settings such as linear regression and shallow neural networks. Their work provided a deeper theoretical understanding of how interpolation affects generalization, revealing a phase transition where the training error collapses to zero while test error exhibits non-trivial behavior due to the complex interaction of bias and variance. Meanwhile, Doroudi and Rastegar (2023) [886] extended the bias-variance framework beyond machine learning, applying it to cognitive science models. Their work emphasized that cognitive processes also operate under similar tradeoffs, where overly rigid or overly flexible models fail to accurately capture human learning and reasoning, thereby providing an interdisciplinary perspective on the problem.
The practical implications of the bias-variance tradeoff extend to domains such as label noise, knowledge distillation, and ensemble learning. Almeida et al. (2020) [887] focused on mitigating class boundary label uncertainty, proposing an approach that estimates label noise and adjusts training sample weights to simultaneously reduce both bias and variance. Their method provided empirical evidence that accounting for uncertainty near decision boundaries leads to improved generalization performance. Zhou et al. (2021) [888] explored the role of soft labels in knowledge distillation, demonstrating that these labels act as a form of implicit regularization that influences the bias-variance tradeoff in student-teacher learning paradigms. They proposed a novel method of weighting soft labels to dynamically balance bias and variance at the sample level. Gupta et al. (2022) [889] examined ensemble methods from a bias-variance perspective, showing that ensembling reduces variance without significantly increasing bias. Their theoretical and empirical results highlighted why ensemble-based approaches often outperform single models in practice. Finally, Ranglani (2024) [890] conducted an extensive empirical analysis across different machine learning algorithms, quantifying the bias-variance decomposition and providing insights into the tradeoff dynamics in regression and classification tasks. Their study offered practical guidelines for optimizing machine learning models by rigorously understanding how bias and variance manifest across various architectures.

Collectively, these works illustrate the evolution of the bias-variance tradeoff from its classical roots to its modern reinterpretation in deep learning and beyond. They provide rigorous theoretical insights and empirical validations that have fundamentally shaped the field of machine learning. The contributions span diverse perspectives, from statistical learning theory to interdisciplinary applications, and challenge traditional notions of model complexity and generalization. The continued exploration of this tradeoff remains a central theme in machine learning research, influencing advancements in algorithm design, optimization techniques, and real-world deployment strategies.

<table><caption>Table 16.17: Summary of Contributions to the Bias-Variance Tradeoff in Supervised Learning</caption><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Geman, Bienenstock, and Doursat (1992)</td><td>Introduced the bias-variance decomposition in neural networks, demonstrating how model complexity influences predictive error. They formulated a theoretical framework explaining the tradeoff between underfitting (high bias) and overfitting (high variance).</td></tr><tr><td>Hastie, Tibshirani, and Friedman (2001)</td><td>Provided a comprehensive treatment of the bias-variance tradeoff in statistical learning, including rigorous mathematical derivations, practical insights, and empirical analyses across multiple machine learning models.</td></tr><tr><td>Belkin et al. (2019)</td><td>Challenged the classical U-shaped bias-variance curve by introducing the "double descent" phenomenon, showing that over-parameterized models can exhibit improved generalization after an initial increase in test error.</td></tr><tr><td>Neal et al. (2018)</td><td>Analyzed the bias-variance tradeoff in deep learning, arguing that implicit regularization induced by stochastic gradient descent prevents excessive variance growth in large neural networks, explaining their surprising generalization ability.</td></tr></tbody></table>

(Continued on next page)
(Continued from previous page)

<table><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Rocks and Mehta (2020)</td><td>Derived analytical expressions for bias and variance in over-parameterized models using statistical physics, uncovering a phase transition where increasing model complexity leads to test error reduction despite interpolation.</td></tr><tr><td>Guest and Martin (2021)</td><td>Extended the bias-variance framework to cognitive science, demonstrating how cognitive models suffer from similar tradeoffs between flexibility and generalization. They argued that model complexity in human cognition follows analogous patterns.</td></tr><tr><td>Almeida et al. (2020)</td><td>Developed a method for mitigating label uncertainty near class boundaries, proposing an adaptive weighting mechanism to reduce both bias and variance, thereby improving model generalization.</td></tr><tr><td>Zhou et al. (2021)</td><td>Investigated knowledge distillation through the lens of the bias-variance tradeoff, demonstrating that soft labels act as implicit regularizers and proposing an optimal weighting scheme to balance bias and variance at the sample level.</td></tr><tr><td>Gupta et al. (2022)</td><td>Provided a rigorous analysis of ensemble methods, proving that ensembling reduces variance without substantially increasing bias, explaining why ensemble learning consistently outperforms single-model approaches.</td></tr><tr><td>Ranglani (2024)</td><td>Conducted an extensive empirical study of the bias-variance tradeoff across various supervised learning models, quantifying bias and variance components and offering practical guidelines for model optimization.</td></tr></tbody></table>

### 16.2.5.2 Recent Literature Review of Bias-Variance Tradeoff of Supervised Learning

The bias-variance tradeoff is a fundamental concept in supervised learning, balancing model complexity and generalization performance. Recent literature explores its implications across various domains, including classical statistical learning, deep learning, and ensemble methods. Rahman and Rahman (2024) provide a foundational overview, discussing its role in linear classifiers and logistic regression, where regularization is used to mitigate overfitting. Tran et al. (2024) extend this idea to power systems, demonstrating how a learnable weighted-ensemble neural network optimally manages the tradeoff, particularly in handling real-world power flow optimization. Similarly, George (2024) examines how bias-variance adjustments impact handwriting recognition using the Kaggle digits dataset, emphasizing the importance of tuning model complexity for improved character classification accuracy.

In the context of statistical modeling, Polson and Sokolov (2024) rigorously analyze hierarchical linear models, showcasing how ridge and lasso regression act as regularization techniques to minimize variance while maintaining sufficient flexibility to prevent high bias. This is further explored by Jogo (2025), who provides a broader statistical perspective, linking the bias-variance tradeoff to support vector machines, unsupervised learning, and computational efficiency in high-dimensional spaces. Additionally, Du et al. (2025) introduce a mathematical framework that incorporates margin theory and optimization-based strategies to manage bias-variance tradeoffs in ensemble learning, highlighting computational trade-offs associated with different model architectures.

A particularly interesting development in deep learning is the challenge to the traditional U-shaped bias-variance curve, as presented by Wang and Pope (2025). Their work on the double descent phenomenon suggests that increasing model complexity beyond a certain threshold can actually
improve generalization, contradicting classical theory. In a related vein, Chen et al. (2024) investigate the role of graph convolutional networks in regression tasks, providing a theoretical framework to understand how different convolutional layers impact the bias-variance tradeoff. Meanwhile, Obster et al. (2024) take a more interpretability-focused approach, proposing a scoring system to balance predictive accuracy and model transparency while managing the bias-variance relationship.

Finally, ensemble-based techniques continue to be a crucial area of exploration in mitigating bias-variance issues. Owen et al. (2024) revisit bagging and stochastic algorithms, extending the bias-variance decomposition to demonstrate how ensemble methods enhance generalization. Their work aligns with recent trends advocating for hybrid model selection strategies that optimize variance reduction without sacrificing interpretability. The collective findings of these studies underscore the evolving nature of the bias-variance tradeoff in supervised learning, from classical statistical interpretations to novel deep learning insights, and highlight its persistent influence on model performance across diverse applications.

<table><thead><tr><td>Reference</td><td>Summary of Contribution</td></tr></thead><tbody><tr><td>Rahman & Rahman (2024)</td><td>Provides a foundational introduction to the bias-variance tradeoff in machine learning, focusing on logistic regression, linear classifiers, and regularization techniques. Discusses how adjusting model complexity impacts generalization performance.</td></tr><tr><td>Tran, Mitra, & Nguyen (2024)</td><td>Explores how an ensemble-based neural network effectively balances the bias-variance tradeoff in power system optimization. Demonstrates the model's ability to enhance stability and accuracy in real-world energy distribution scenarios.</td></tr><tr><td>George (2024)</td><td>Investigates how different machine learning models handle the bias-variance tradeoff in character recognition. Uses the Kaggle digits dataset to optimize performance through regularization and model selection strategies.</td></tr><tr><td>Du et al. (2025)</td><td>Develops a theoretical framework linking bias-variance tradeoff to margin theory and optimization techniques. Proposes computational trade-offs to improve model selection in ensemble learning methods.</td></tr><tr><td>Polson & Sokolov (2024)</td><td>Explains the role of ridge and lasso regression in controlling the bias-variance tradeoff. Provides empirical studies demonstrating how hyperparameter tuning influences model performance and generalization.</td></tr><tr><td>Jogo (2025)</td><td>Covers the statistical foundations of the bias-variance tradeoff, linking it to support vector machines, unsupervised learning, and computational efficiency in high-dimensional spaces.</td></tr><tr><td>Wang & Pope (2025)</td><td>Challenges the traditional U-shaped bias-variance tradeoff by showing that increased complexity beyond a certain point can improve generalization in deep learning models.</td></tr><tr><td>Chen, Schmidt-Hieber, & Donnat (2024)</td><td>Analyzes the impact of graph convolutional networks (GCNs) on bias-variance tradeoff, providing a theoretical framework for evaluating convolutional layer depth in regression models.</td></tr><tr><td>Obster, Ciolacu, & Humpe (2024)</td><td>Investigates the tradeoff between predictive accuracy and interpretability in machine learning models. Introduces a scoring system for model complexity optimization while maintaining an optimal bias-variance balance.</td></tr></tbody></table>
<table><thead><tr><td>Reference</td><td>Summary of Contribution</td></tr></thead><tbody><tr><td>Owen, Dick, &amp; Whigham (2024)</td><td>Extends the bias-variance decomposition for stochastic learning algorithms, demonstrating how bagging-based ensemble methods improve generalization. Highlights hybrid model selection strategies for variance reduction.</td></tr></tbody></table>

### 16.2.5.3 Mathematical Analysis of Bias-Variance Tradeoff of Supervised Learning

In supervised learning, the bias-variance tradeoff is a fundamental concept that governs the performance of a model in terms of its ability to generalize to unseen data. Mathematically, the total expected error of a model, often measured as the mean squared error (MSE), can be decomposed as follows:

$$ \mathbb{E}[(Y - \hat{f}(X))^2] = \underbrace{(f(X) - \mathbb{E}[\hat{f}(X)])^2}_{\text{Bias}^2} + \underbrace{\mathbb{E}[(\hat{f}(X) - \mathbb{E}[\hat{f}(X)])^2]}_{\text{Variance}} + \underbrace{\sigma^2}_{\text{Irreducible Error}} \quad (16.91) $$

where $Y$ is the true response variable, $X$ is the input feature, $f(X)$ is the true underlying function, and $\hat{f}(X)$ is the estimated function learned by the model. The decomposition highlights three key sources of error:

1. **Bias**: This term quantifies how far the expected prediction $\mathbb{E}[\hat{f}(X)]$ is from the true function $f(X)$. Formally, it is defined as

$$ \text{Bias}(\hat{f}(X)) = \mathbb{E}[\hat{f}(X)] - f(X). \quad (16.92) $$

A high-bias model makes systematic errors because it fails to capture the complexity of the data. This often occurs in underfitting, where the model is too simple to represent the underlying structure of the data.

2. **Variance**: This term quantifies the variability of the model's predictions around its expected value, given by

$$ \text{Var}(\hat{f}(X)) = \mathbb{E}[(\hat{f}(X) - \mathbb{E}[\hat{f}(X)])^2]. \quad (16.93) $$

A high-variance model is highly sensitive to fluctuations in the training data and does not generalize well to unseen data. This typically occurs in overfitting, where the model captures noise instead of the true signal.

3. **Irreducible Error**: The term $\sigma^2 = \mathbb{E}[(Y - f(X))^2]$ represents noise inherent in the data that no model can eliminate.

Since the total error is the sum of these components, an increase in one term often leads to a decrease in another. For instance, using a more flexible model (e.g., a high-degree polynomial) can reduce bias but at the cost of increased variance. Conversely, using a simple model (e.g., linear regression) reduces variance but increases bias. This tradeoff is captured by minimizing the expected loss function

$$ \min_{\hat{f}} \mathbb{E}[(Y - \hat{f}(X))^2], \quad (16.94) $$

where the goal is to balance the bias and variance terms to achieve optimal generalization. To analyze this tradeoff further, consider a simple linear model where the target function is quadratic:

$$ Y = X^2 + \epsilon, \quad \text{where } \epsilon \sim \mathcal{N}(0, \sigma^2). \quad (16.95) $$
<figure><img src="image_176.png" /><figcaption>Figure 16.8: Bias-Variance Tradeoff in Supervised Learning. Bias² decreases with model complexity, variance increases, training error decreases, and test error initially decreases then increases due to overfitting</figcaption></figure>

If we fit a linear regression model $\hat{f}(X) = aX + b$, then the bias is

$$ \text{Bias}^2 = (\mathbb{E}[aX + b] - X^2)^2. \qquad (16.96) $$

For a high-degree polynomial model $\hat{f}(X) = \sum_{i=0}^{d} c_i X^i$, the variance term increases with $d$, leading to:

$$ \text{Var}(\hat{f}(X)) = \sum_{i=0}^{d} \text{Var}(c_i) X^{2i}. \qquad (16.97) $$

Minimizing the expected generalization error requires selecting a model complexity $d$ such that

$$ \frac{\partial}{\partial d} (\text{Bias}^2 + \text{Var} + \sigma^2) = 0. \qquad (16.98) $$

To further illustrate, consider a dataset of size $n$ and a model parameterized by $\theta$. The variance of the estimator $\hat{\theta}$ is given by

$$ \text{Var}(\hat{\theta}) = \mathbb{E}[(\hat{\theta} - \mathbb{E}[\hat{\theta}])^2], \qquad (16.99) $$

whereas the bias squared is

$$ \text{Bias}^2(\hat{\theta}) = (\mathbb{E}[\hat{\theta}] - \theta_{\text{true}})^2. \qquad (16.100) $$

The total mean squared error (MSE) of the estimator is then

$$ \text{MSE}(\hat{\theta}) = \text{Bias}^2(\hat{\theta}) + \text{Var}(\hat{\theta}). \qquad (16.101) $$
As the complexity of the model increases, the bias decreases, but variance increases. This behavior is described by the function

$$ \text{Error}(d) = Ad^{-p} + Bd^{q} + C, \qquad (16.102) $$

where A, B, C, p, q are constants depending on the dataset and learning algorithm, and d represents model complexity. The optimal complexity d* satisfies

$$ \frac{\partial \text{Error}(d)}{\partial d} = 0. \qquad (16.103) $$

For neural networks, increasing the number of layers and neurons often leads to decreased bias but significantly increased variance due to overparameterization. The expected loss for a neural network with weights W trained on data (X, Y) is given by

$$ \mathbb{E}[\mathcal{L}(W)] = \mathbb{E}[(Y - f_W(X))^2] = \text{Bias}^2 + \text{Var} + \sigma^2. \qquad (16.104) $$

Regularization techniques such as ridge regression introduce a penalty term $\lambda||W||^2$, leading to the optimization problem

$$ \min_{W} \sum_{i=1}^{n} (y_i - f_W(x_i))^2 + \lambda ||W||^2. \qquad (16.105) $$

This reduces variance at the cost of increasing bias, effectively controlling model complexity to achieve an optimal bias-variance tradeoff.

### 16.2.5.4 Python Code to generate Figure 16.8 illustrating the Bias-Variance Tradeoff of Supervised Learning

The Python code below produces the Figure 16.8 illustrating the Bias-Variance Tradeoff of Supervised Learning.

```python
import numpy as np
import matplotlib.pyplot as plt

# Simulate model complexity
complexity = np.arange(1, 11)

# Simulate Bias^2, Variance, Training error, Test error
bias2 = np.exp(-0.5 * complexity) # Decreasing bias^2
variance = 0.1 * complexity**1.5 # Increasing variance
noise = 0.2 # Irreducible error
train_error = bias2 + 0.5 * variance # Simulated training error
test_error = bias2 + variance + noise # Simulated test error

# Plot
plt.figure(figsize=(8, 6))
plt.plot(complexity, bias2, 'r-o', label='Bias^2')
plt.plot(complexity, variance, 'b-o', label='Variance')
plt.plot(complexity, train_error, 'g--s', label='Training Error')
plt.plot(complexity, test_error, 'k--s', label='Test Error')
plt.xlabel('Model Complexity')
plt.ylabel('Error')
plt.title('Bias-Variance Tradeoff in Supervised Learning')
plt.legend()
plt.grid(True)
plt.tight_layout()
plt.show()
```
16.2.6 Support vector machine

16.2.6.1 Literature Review of Support vector machine

Support Vector Machines (SVMs) are rooted in Vladimir Vapnik’s “The Nature of Statistical Learn-
ing Theory” (1995) [143], which introduced the fundamental principle of structural risk minimiza-
tion (SRM). Unlike empirical risk minimization, which solely minimizes training error, SRM aims
to balance the complexity of the model and its ability to generalize to unseen data, thereby pre-
venting overfitting. This theoretical framework laid the foundation for the development of SVMs
as a powerful tool for classification and regression problems. Schölkopf and Smola’s ”Learning
with Kernels” (2002) [823] provided a mathematically rigorous treatment of kernel methods, which
form the core of SVMs by enabling them to operate in high-dimensional feature spaces without
explicit transformation via the kernel trick. This book formalized the mathematical underpinnings
of reproducing kernel Hilbert spaces (RKHS), Mercer’s theorem, and various kernel functions such
as Gaussian and polynomial kernels. Cristianini and Shawe-Taylor’s ”An Introduction to Sup-
port Vector Machines” (2000) [824] played a crucial role in making these complex mathematical
ideas accessible. It not only explained the theoretical aspects of SVMs but also provided insights
into their practical implementation, making it a key reference for researchers and practitioners alike.

The statistical properties of SVMs were rigorously analyzed in Christmann and Steinwart’s ”Sup-
port Vector Machines” (2008) [825], which focused on their consistency, robustness, and learning
rates. This work addressed fundamental questions regarding the asymptotic behavior of SVM
classifiers, including their convergence properties under different loss functions. Furthermore, the
edited volume by Schölkopf, Burges, and Smola, ”Advances in Kernel Methods” (1999) [826], com-
piled several significant advancements in SVM research, including extensions of SVMs to regression
problems (Support Vector Regression, SVR), one-class SVMs for anomaly detection, and novel
kernel functions suited for different applications. The book highlighted both theoretical develop-
ments and practical applications, illustrating the versatility of SVMs across various domains such
as image recognition, bioinformatics, and financial modeling.

The extension of SVMs beyond binary classification was a crucial milestone. Drucker et al.’s ”Support Vector Regression Machines” (1997) [827] formalized Support Vector Regression (SVR), which adapted the SVM framework to predict continuous-valued outputs by introducing an ε-insensitive loss function. This work demonstrated how SVMs could be effectively used for time series prediction and function approximation. Joachims’ ”Transductive Inference for Text Classification” (1999) [828] introduced Transductive SVMs (TSVMs), which exploit both labeled and unlabeled data to enhance classification performance, particularly in text classification problems where labeled data is limited. By incorporating unlabeled data, TSVMs leverage the structure of the input distribution, making them particularly useful in semi-supervised learning.

Another fundamental contribution came from Schölkopf, Smola, and Müller’s ”Nonlinear Com-
ponent Analysis as a Kernel Eigenvalue Problem” (1998) [829], which extended Principal Compo-
nent Analysis (PCA) to nonlinear settings using kernels (Kernel PCA). This demonstrated how
kernel-based methods, including SVMs, could be applied beyond classification and regression to
dimensionality reduction and feature extraction, which are critical for handling high-dimensional
data. Burges’ ”A Tutorial on Support Vector Machines for Pattern Recognition” (1998) [830] pro-
vided an intuitive yet mathematically detailed explanation of SVMs, covering Lagrange duality,
convex optimization, and margin maximization. This tutorial remains an essential resource for
researchers seeking to understand both the theoretical foundations and practical aspects of imple-
menting SVMs.

Finally, Schölkopf et al.'s "Estimating the Support of a High-Dimensional Distribution" (2001) [831]
introduced one-class SVMs for anomaly detection, where the goal is to find a decision boundary that encloses the majority of data points while rejecting outliers. This formulation is particularly useful in fraud detection, network security, and fault diagnosis, where anomalies are rare but significant. Collectively, these contributions have shaped the development of SVMs as a mathematically rigorous and practically powerful machine learning technique. The blend of theoretical depth, statistical robustness, and practical versatility has cemented SVMs as one of the most influential algorithms in the history of machine learning.

<table><caption>Table 16.19: Summary of Contributions in Support Vector Machines</caption><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Vladimir N. Vapnik (1995)</td><td>Introduced Structural Risk Minimization (SRM), the foundational principle behind SVMs. Established the theoretical basis for SVMs within statistical learning theory.</td></tr><tr><td>Bernhard Schölkopf and Alexander J. Smola (2002)</td><td>Provided a rigorous mathematical treatment of kernel methods, including reproducing kernel Hilbert spaces (RKHS) and Mercer's theorem, formalizing the kernel trick.</td></tr><tr><td>Nello Cristianini and John Shawe-Taylor (2000)</td><td>Made SVM theory accessible by providing a practical introduction to the mathematical foundations and applications of SVMs.</td></tr><tr><td>Ingo Steinwart and Andreas Christmann (2008)</td><td>Offered a statistical analysis of SVMs, covering consistency, robustness, and learning rates, providing insight into their asymptotic properties.</td></tr><tr><td>Bernhard Schölkopf, Christopher J.C. Burges, and Alexander J. Smola (1999)</td><td>Compiled major advances in kernel methods, including extensions of SVMs to regression (SVR) and novel kernel functions. Showcased applications in image processing and bioinformatics.</td></tr><tr><td>Harris Drucker et al. (1997)</td><td>Developed Support Vector Regression (SVR), adapting SVMs for continuous-valued predictions. Introduced the ε-insensitive loss function.</td></tr><tr><td>Thorsten Joachims (1999)</td><td>Introduced Transductive SVMs (TSVMs), which leverage both labeled and unlabeled data for improved classification, particularly in text mining applications.</td></tr><tr><td>Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller (1998)</td><td>Developed Kernel Principal Component Analysis (Kernel PCA), extending SVM-related methods to nonlinear dimensionality reduction and feature extraction.</td></tr><tr><td>Christopher J.C. Burges (1998)</td><td>Provided an accessible and mathematically detailed tutorial on SVMs, explaining concepts such as margin maximization, convex optimization, and duality theory.</td></tr><tr><td>Bernhard Schölkopf et al. (2001)</td><td>Introduced One-Class SVMs for anomaly detection, providing a method to estimate the support of a high-dimensional distribution. Applied in fraud detection and network security.</td></tr></tbody></table>

### 16.2.6.2 Recent Literature Review of Support vector machine

Support Vector Machines (SVMs) have demonstrated remarkable versatility across a wide range of domains, from biomedical imaging and sentiment analysis to engineering and hydrological modeling. In the medical domain, Guo and Sun (2025) [872] utilized SVMs to analyze neuroimaging
data, assessing stroke rehabilitation effects in brain tumor patients. Their study emphasized SVM's
capability in clinical decision-making and biomedical imaging. Similarly, Diao et al. (2025) [873]
investigated lung cancer detection by optimizing Bi-LSTM networks with hand-crafted features.
Their findings revealed that SVMs achieved the highest classification accuracy when extracting
Gray-Level Co-Occurrence Matrix (GLCM) features, reinforcing their robustness in medical image
classification. Another pivotal study by Lin et al. (2025) [874] combined deep transfer learning
with SVM-based radiomics for sinonasal malignancy detection in MRI scans, achieving an impres-
sive 92.6 percent accuracy, underscoring SVM's potential in computer-aided diagnosis and medical
image processing. Çetintaş (2025) [875] further extended SVM applications in healthcare by em-
ploying an optimized SVM model via Grid Search for monkeypox detection, significantly improving
classification performance on imbalanced datasets.

In the realm of natural language processing and text analytics, Wang and Zhao (2025) [876] compared sentiment lexicon and machine learning methods for citation sentiment identification, demonstrating SVM's superior generalization ability in text classification tasks. Muralinath et al. (2025) [877] explored multichannel EEG classification using spectral graph kernels, where SVMs played a crucial role in robust epilepsy detection. Additionally, Hu et al. (2025) [878] addressed the class imbalance problem in sarcasm detection by leveraging an ensemble-based oversampling technique, proving SVM's efficacy in handling skewed datasets. These studies confirm SVM's strength in high-dimensional, sparse data environments, making it a go-to method in various text processing applications.

Engineering and applied sciences have also benefited greatly from SVM-based models. Wang et al. (2025) [879] demonstrated how Support Vector Regression (SVR) effectively predicts the tensile properties of automotive steels, validating its use in materials science and mechanical engineering. Similarly, Husain et al. (2025) [880] employed SVR to model shear thickening fluid behavior, showcasing its ability to forecast nonlinear behaviors in physics and engineering applications. Iqbal and Siddiqi (2025) [881] integrated SVM in a hybrid deep learning model to enhance seasonal streamflow prediction, proving SVM's utility in hydrological and environmental modeling. These studies collectively highlight the adaptability of SVM-based regression techniques in predicting complex, nonlinear systems across scientific disciplines.

From biomedical diagnostics to predictive analytics in engineering, these studies illustrate how Support Vector Machines remain a fundamental tool in machine learning research. The ability of SVMs to handle high-dimensional spaces, work with limited data, and maintain strong generalization capabilities makes them highly relevant across numerous fields. Whether used for medical imaging, fraud detection, text mining, or physical modeling, SVMs continue to provide state-of-the-art solutions that rival deep learning techniques while maintaining interpretability and computational efficiency.
<table><caption>Table 16.20: Summary of Recent Contributions in Support Vector Machines</caption><thead><tr><td>Study</td><td>Contribution</td><td>Domain</td></tr></thead><tbody><tr><td>Guo & Sun (2025)</td><td>Applied SVM to analyze neuroimaging data, assessing stroke rehabilitation effects in brain tumor patients. Demonstrated SVM's potential in biomedical imaging and clinical decision-making.</td><td>Medical Imaging, Neuroscience</td></tr><tr><td>Diao et al. (2025)</td><td>Optimized Bi-LSTM networks for lung cancer detection. Found that SVM achieved the highest classification accuracy when extracting GLCM features.</td><td>Medical Diagnosis, Deep Learning</td></tr><tr><td>Lin et al. (2025)</td><td>Integrated SVM with deep transfer learning in MRI-based sinonasal malignancy detection, achieving 92.6% accuracy. Highlighted SVM's potential in radiomics.</td><td>Radiomics, MRI Analysis</td></tr><tr><td>Çetintaş (2025)</td><td>Used an optimized SVM model with Grid Search for monkeypox detection. Improved classification performance on imbalanced datasets.</td><td>Medical Image Classification</td></tr><tr><td>Wang & Zhao (2025)</td><td>Compared sentiment lexicon and machine learning methods for citation sentiment identification. Demonstrated SVM's effectiveness in text classification.</td><td>NLP, Sentiment Analysis</td></tr><tr><td>Muralinath et al. (2025)</td><td>Explored multichannel EEG classification using spectral graph kernels. Showed SVM's robustness in epilepsy detection.</td><td>EEG Analysis, Neurology</td></tr><tr><td>Hu et al. (2025)</td><td>Addressed class imbalance in sarcoma detection using ensemble-based oversampling techniques. Showed SVM's high performance in NLP.</td><td>NLP, Social Media Analysis</td></tr><tr><td>Wang et al. (2025)</td><td>Developed an SVR-based predictive model for tensile properties of automotive steels. Proved its effectiveness in mechanical engineering.</td><td>Materials Science, Engineering</td></tr><tr><td>Husain et al. (2025)</td><td>Applied SVR for modeling shear thickening fluid behavior, showing its ability to forecast nonlinear physical behaviors.</td><td>Physics, Fluid Mechanics</td></tr></tbody></table>

*Continued on next page*
(Continued from previous page)

<table><thead><tr><td>Study</td><td>Contribution</td><td>Domain</td></tr></thead><tbody><tr><td>Iqbal & Siddiqi (2025)</td><td>Integrated SVM into a hybrid deep learning model for seasonal streamflow prediction, demonstrating SVM's utility in hydrological modeling.</td><td>Hydrology, Environmental Science</td></tr></tbody></table>

### 16.2.6.3 Mathematical Analysis of Support vector machine

The Support Vector Machine (SVM) is a supervised learning model used for classification and regression. Given a training set $\{(\mathbf{x}_i, y_i)\}_{i=1}^N$, where $\mathbf{x}_i \in \mathbb{R}^d$ are the feature vectors and $y_i \in \{-1, 1\}$ are the class labels, the goal is to find a hyperplane that maximally separates the two classes. The hyperplane is represented as

$$ \mathbf{w}^\top \mathbf{x} + b = 0 \qquad (16.106) $$

where $\mathbf{w} \in \mathbb{R}^d$ is the normal vector to the hyperplane and $b \in \mathbb{R}$ is the bias term. For linearly separable data, the constraints ensuring correct classification are

$$ y_i(\mathbf{w}^\top \mathbf{x}_i + b) \geq 1, \quad \forall i \in \{1, 2, \dots, N\} \qquad (16.107) $$

The distance from a point $\mathbf{x}_i$ to the hyperplane is given by

$$ \frac{|\mathbf{w}^\top \mathbf{x}_i + b|}{\|\mathbf{w}\|} \qquad (16.108) $$

The **margin**, which is the distance between the two parallel supporting hyperplanes that pass through the closest data points of each class, is given by

$$ \frac{2}{\|\mathbf{w}\|} \qquad (16.109) $$

The **optimal separating hyperplane** maximizes this margin, leading to the following **convex optimization problem**

$$ \min_{\mathbf{w},b} \frac{1}{2} \|\mathbf{w}\|^2 \qquad (16.110) $$

subject to

$$ y_i(\mathbf{w}^\top \mathbf{x}_i + b) \geq 1, \quad \forall i \in \{1, 2, \dots, N\} \qquad (16.111) $$

To solve this constrained optimization problem, the **Lagrangian** is introduced:

$$ \mathcal{L}(\mathbf{w}, b, \alpha) = \frac{1}{2} \|\mathbf{w}\|^2 - \sum_{i=1}^{N} \alpha_i (y_i(\mathbf{w}^\top \mathbf{x}_i + b) - 1) \qquad (16.112) $$

where $\alpha = (\alpha_1, \alpha_2, \dots, \alpha_N)$ are the **Lagrange multipliers** satisfying $\alpha_i \geq 0$. Taking derivatives and setting them to zero gives the **Karush-Kuhn-Tucker (KKT) conditions**:

$$ \mathbf{w} = \sum_{i=1}^{N} \alpha_i y_i \mathbf{x}_i \qquad (16.113) $$

$$ \sum_{i=1}^{N} \alpha_i y_i = 0 \qquad (16.114) $$

$$ \alpha_i (y_i(\mathbf{w}^\top \mathbf{x}_i + b) - 1) = 0, \quad \forall i \in \{1, 2, \dots, N\} \qquad (16.115) $$
<figure><img src="image_177.png" /><figcaption>Figure 16.9: Support Vector Machine (SVM) with linear kernel. The solid line represents the decision boundary, dashed lines show the margins, and circled points are the support vectors</figcaption></figure>

Substituting **w** into the Lagrangian yields the dual problem:

$$ \max_{\alpha} \sum_{i=1}^{N} \alpha_i - \frac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_i \alpha_j y_i y_j \mathbf{x}_i^{\top} \mathbf{x}_j \quad (16.116) $$

subject to

$$ \sum_{i=1}^{N} \alpha_i y_i = 0, \quad \alpha_i \ge 0, \quad \forall i \in \{1, 2, \dots, N\} \quad (16.117) $$

The optimal **support vectors** are the points for which $\alpha_i > 0$. The bias term $b$ is computed using any support vector $\mathbf{x}_s$ as

$$ b = y_s - \sum_{i=1}^{N} \alpha_i y_i \mathbf{x}_i^{\top} \mathbf{x}_s \quad (16.118) $$

For non-linearly separable data, the SVM allows some misclassification using **slack variables** $\xi_i \ge 0$, leading to the soft-margin SVM formulation

$$ \min_{\mathbf{w}, b, \xi} \frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{i=1}^{N} \xi_i \quad (16.119) $$

subject to

$$ y_i(\mathbf{w}^{\top}\mathbf{x}_i + b) \ge 1 - \xi_i, \quad \xi_i \ge 0, \quad \forall i \in \{1, 2, \dots, N\} \quad (16.120) $$
<figure><img src="image_178.png" /><figcaption>Figure 16.10: Illustration of the Support Vector Machine</figcaption></figure>

Image Credit: By Larhmam - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=73710028

This illustration shows an SVM trained on two-class data, highlighting the maximum-margin hyperplane and its margins. The data points that rest on the margin are referred to as support vectors

where $C > 0$ is a regularization parameter. The corresponding **dual problem** remains

$$ \max \sum_{i=1}^{N} \alpha_i - \frac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{N} \alpha_i \alpha_j y_i y_j \mathbf{x}_i^{\top} \mathbf{x}_j \quad (16.121) $$

subject to

$$ \sum_{i=1}^{N} \alpha_i y_i = 0, \quad 0 \le \alpha_i \le C, \quad \forall i \in \{1, 2, \dots, N\} \quad (16.122) $$

The **final classifier** is

$$ f(\mathbf{x}) = \mathrm{sign} \left( \sum_{i=1}^{N} \alpha_i y_i K(\mathbf{x}_i, \mathbf{x}) + b \right) \quad (16.123) $$
Thus, SVM constructs an optimal decision boundary by solving a quadratic optimization problem, ensuring a maximum-margin separator, and leveraging kernel functions to extend the model to non-linearly separable cases.

### 16.2.6.4 Python Code to generate Figure 16.9 illustrating the Support Vector Machine (SVM) with linear kernel

The Python code below produces the Figure 16.9 illustrating the Support Vector Machine (SVM) with linear kernel.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn import datasets
fromsklearn.svm import SVC

# Generate toy dataset
X, y = datasets.make_blobs(n Samples=50, centers=2, random_state=6)

# Train SVM classifier
clf = SVC kernel='linear', C=1000)
clf.fit(X, y)

# Plot decision function
plt.figure(figsize=(8, 6))

# Scatter plot of points
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap=plt.cm.Paired)

# Get current axis
ax = plt.gca()
xlim = ax.get_xlim()
ylim = ax.get_ylim()

# Create grid to evaluate model
xx = npolspace(xlim[0], ylim[1], 30)
yy = npolspace(ylim[0], ylim[1], 30)
YY, XX = np meshgrid(xx, yy)
xy = npstack([XX.ravel(), YY.ravel()]).T
Z =clf.decision_function(xy).reshape(XX.shape)

# Plot decision boundary and margins
ax.contour(XX, YY, Z, colors='k',
           levels=[-1, 0, 1], alpha=0.7,
           linestyles=['--', -, '--'])

# Plot support vectors
ax.scatter(clf support vectors [:, 0],
           clf support vectors [:, 1],
           s=150, linewidth=1, facecolors='none', edgecolors='k')

plt.title("Support Vector Machine (SVM) with Linear Kernel")
plt.xlabel("Feature 1")
plt.ylabel("Feature 2")
plt.tight_layout()
plt.show()
```
16.2.7 Linear regression

16.2.7.1 Literature Review of Linear regression

The development of linear regression as a rigorous statistical tool can be traced back to the foun-
dational works of Gauss (1809) [832] and Legendre (1806) [833], who independently introduced
the method of least squares. This method provides a systematic way to estimate parameters by
minimizing the sum of squared residuals, ensuring the best linear fit to a given set of data points.
While Legendre initially proposed the method in the context of astronomical orbit calculations,
Gauss later provided a more comprehensive theoretical justification, demonstrating that under the
assumption of normally distributed errors, the least squares estimators are best linear unbiased
estimators (BLUE), meaning they have the lowest variance among all linear unbiased estimators.
This result was later formalized in what is now known as the Gauss-Markov theorem. Additionally,
Gauss extended these ideas to incorporate the normality assumption, which established a strong
probabilistic foundation for regression analysis by linking least squares estimation to maximum
likelihood estimation (MLE) in the presence of Gaussian errors.

The geometric interpretation of regression was further expanded by Pearson (1901) [834], who
developed principal component analysis (PCA) as a method for dimensionality reduction and data
compression. Pearson’s work highlighted the connection between linear regression and orthogonal
projections, providing insights into how regression techniques relate to eigenvalue decomposition
and singular value decomposition (SVD). Fisher (1922) [835] played a crucial role in rigorously
formulating the statistical inference framework for linear regression, developing methods for es-
timating standard errors, constructing confidence intervals, and conducting hypothesis tests on
regression coefficients. Fisher’s approach introduced key concepts such as the sampling distribu-
tion of regression estimators and the use of likelihood-based inference, which remain central to
modern regression analysis.

Building upon these foundational principles, Koopmans (1937) [836] extended linear regression
techniques to time series analysis, addressing challenges such as serial correlation, heteroskedastic-
ity, and multicollinearity in economic forecasting. This laid the groundwork for the generalized least
squares (GLS) method, which was later formalized by Goldberger (1991) [837]. Goldberger pro-
vided a rigorous treatment of violations of ordinary least squares (OLS) assumptions, particularly
when dealing with correlated error structures and non-constant variance in regression models. Rao
(1973) [838] further expanded the theory of regression by introducing ridge regression, a technique
that stabilizes coefficient estimates in the presence of multicollinearity by introducing a penalty
term to control variance. His work unified linear regression within the broader framework of mul-
tivariate statistical inference, contributing to the development of generalized regression models.

Huber (1992) [839] introduced a significant advancement by developing robust regression tech-
niques, which addressed the sensitivity of classical least squares estimation to outliers and depar-
tures from normality. His work on M-estimation provided a framework for obtaining regression
estimates that remain stable under heavy-tailed distributions and heteroskedastic errors. Finally,
modern developments in regression analysis were driven by Hastie, Tibshirani, and Friedman (2009)
[139], who introduced regularized regression techniques such as ridge regression and LASSO (Least
Absolute Shrinkage and Selection Operator). These methods added penalization terms to the least
squares objective function, effectively preventing overfitting and improving model generalization in
high-dimensional settings.

Collectively, these contributions transformed linear regression from a simple numerical fitting
method into a rigorous statistical and mathematical framework with broad applicability across
various scientific and engineering disciplines. The integration of inferential statistics, robustness
techniques, and regularization strategies has ensured that linear regression remains a cornerstone of modern statistical analysis and predictive modeling.

<table><caption>Table 16.21: Summary of Contributions to Linear Regression</caption><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Legendre (1805)</td><td>Introduced the least squares method for estimating parameters by minimizing the sum of squared residuals, initially applied to astronomical data.</td></tr><tr><td>Gauss (1809, 1821)</td><td>Formally justified the least squares method under normal error assumptions, proving its best linear unbiased estimation (BLUE) property and laying the foundation for the Gauss-Markov theorem. Developed statistical inference for regression.</td></tr><tr><td>Pearson (1901)</td><td>Developed principal component analysis (PCA), establishing the geometric relationship between regression and orthogonal projections, which later connected to singular value decomposition (SVD).</td></tr><tr><td>Fisher (1922)</td><td>Formalized the statistical inference framework for regression, deriving the sampling distributions of regression coefficients, hypothesis testing procedures, and maximum likelihood estimation (MLE) methods.</td></tr><tr><td>Koopmans (1937)</td><td>Extended regression to time series analysis, addressing issues such as serial correlation, heteroskedasticity, and multicollinearity, which are critical in econometric models.</td></tr><tr><td>Goldberger (1964)</td><td>Developed generalized least squares (GLS) to handle correlated errors and non-constant variance, providing a rigorous treatment of assumption violations in ordinary least squares (OLS).</td></tr><tr><td>Rao (1973)</td><td>Expanded regression theory by introducing ridge regression to handle multicollinearity, unifying regression within the broader framework of multivariate statistical inference.</td></tr><tr><td>Huber (1964)</td><td>Developed robust regression techniques, particularly M-estimation, which mitigates the impact of outliers and non-Gaussian error distributions, improving model stability.</td></tr><tr><td>Hastie, Tibshirani, and Friedman (2009)</td><td>Introduced regularized regression techniques such as ridge regression and LASSO, incorporating penalization terms to prevent overfitting and improve generalization in high-dimensional data.</td></tr></tbody></table>

### 16.2.7.2 Recent Literature Review of Linear regression

Linear regression, as a foundational statistical tool, has been extensively employed across various disciplines, from environmental monitoring and healthcare research to economic policy and agricultural sciences. Recent advancements illustrate how traditional regression techniques have been adapted and integrated with modern methodologies to enhance analytical precision and predictive power. Ramadhan and Ali (2025) [840] propose a novel multivariate wavelet shrinkage approach in quantile regression models, effectively improving estimation accuracy in high-dimensional data scenarios. Their work highlights how wavelet-based methods refine traditional regression by addressing noise reduction and variability, making it particularly useful in financial modeling and
risk assessment. Similarly, Zhou et al. (2025) [841] leverage linear regression within a hybrid ma-
chine learning framework that combines Partial Least Squares (PLS) and Decision Trees (DT) for
real-time chemical degradation monitoring. This application underscores regression's role in en-
vironmental science, where predictive analytics can optimize industrial and ecological interventions.

Healthcare research has also significantly benefited from linear regression models, particularly in understanding human behavior and institutional efficiency. Zhong et al. (2025) [842] employ multiple linear regression to analyze factors influencing nurses’ attitudes and practices in postural management for premature infants. Their study highlights how regression can quantify behavioral determinants, aiding in the design of targeted medical training programs. Likewise, Liu et al. (2025) [843] use regression analysis to assess the research capabilities of pediatric clinical nurses, identifying key skill gaps and the variables influencing scientific output in medical institutions. These findings emphasize the importance of regression-based methodologies in human resource optimization and professional development within healthcare. A crucial addition to the healthcare research stream is the study by Dietze et al. (2025) [845], which was previously omitted. Their research examines the impact of the UNODC/WHO SOS (Stop Overdose Safely) training program on opioid overdose knowledge and attitudes. Using multivariable linear regression, they analyze the relationship between demographic characteristics and training effectiveness. Their findings reveal how linear regression can provide deep insights into behavioral interventions, policy evaluations, and addiction treatment effectiveness.

In the economic and policy domain, regression techniques serve as powerful tools for evaluating regulatory frameworks and socioeconomic disparities. Ming-jun and Jian-ya (2025) [844] construct multiple linear regression models to examine the Porter hypothesis in environmental taxation, revealing complex interdependencies between tax policies and corporate behavior. This application of regression provides empirical support for policy decisions aimed at balancing environmental sustainability with economic growth. Cox et. al. (2022) [1468] extended Halbeisen's and Hungerbühler's optimal bounds for Collatz cycle lengths to the more general 3n + c sequences, where c is an odd integer not divisible by 3. It analyzes the structure of these cycles using parity vectors derived from ceiling and floor functions to identify relationships between the smallest and largest odd elements within each cycle. The study reveals rotational and congruential properties of these parity vectors, suggesting underlying uniformity in the distribution of elements across generalized Collatz cycles. Hasan and Ghosal (2025) [846] extend this approach to public health, using regression to quantify inequities in healthcare access across different regions in West Bengal, India. Their findings contribute to the formulation of equitable healthcare policies by identifying key determinants of accessibility, such as affordability and geographical distribution.

Agricultural and biomedical research have also leveraged linear regression for predictive modeling and diagnostic improvements. Zeng et al. (2025) [848] employ LASSO (Least Absolute Shrinkage and Selection Operator) regression to enhance maize yield predictions under stress conditions. By integrating regression with remote sensing data, they improve crop forecasting models, demonstrating the method’s importance in precision agriculture. In orthopedic research, Baird et al. (2025) [849] use regression to analyze long-term trends in surgical procedures, particularly in anterior cruciate ligament reconstruction and hip arthroscopy. Their work demonstrates how regression-driven AI analysis can refine medical decision-making by identifying treatment patterns and patient outcomes over time.

Finally, the application of regression in veterinary and agronomic sciences further highlights its interdisciplinary utility. Overton and Eicker (2025) [850] analyze milk production and fertility trends in Holstein dairy cows using a combination of linear and logistic regression models. Their study provides critical insights into dairy farm efficiency, showing how regression techniques can
optimize breeding programs and livestock management strategies. Collectively, these studies illustrate the remarkable adaptability of linear regression, as it continues to evolve through integration with machine learning and modern statistical techniques, further solidifying its relevance in both academic research and practical applications.

<table><caption>Table 16.22: Summary of Recent Contributions on Linear Regression</caption><thead><tr><td>Paper</td><td>Key Contribution</td></tr></thead><tbody><tr><td>Ramadhan & Ali (2025)</td><td>Introduces a multivariate wavelet shrinkage approach for quantile regression, improving accuracy in high-dimensional data.</td></tr><tr><td>Zhou et al. (2025)</td><td>Integrates linear regression with machine learning techniques (PLS, DT) for real-time environmental monitoring of chemical degradation.</td></tr><tr><td>Zhong et al. (2025)</td><td>Uses multiple linear regression to analyze factors influencing nurses' attitudes and practices in postural management of premature infants.</td></tr><tr><td>Liu et al. (2025)</td><td>Applies regression analysis to evaluate research capabilities of pediatric clinical nurses, identifying key influencing factors.</td></tr><tr><td>Dietze et al. (2025)</td><td>Examines the impact of opioid overdose prevention training using multivariable regression, evaluating demographic influences on knowledge retention.</td></tr><tr><td>Ming-jun & Jian-ya (2025)</td><td>Constructs multiple linear regression models to analyze the economic effects of environmental taxation policies.</td></tr><tr><td>Hasan & Ghosal (2025)</td><td>Uses regression techniques to quantify inequities in healthcare access, identifying determinants like affordability and availability.</td></tr><tr><td>Zeng et al. (2025)</td><td>Employs LASSO regression for maize yield prediction, integrating remote sensing data for enhanced agricultural forecasting.</td></tr><tr><td>Baird et al. (2025)</td><td>Uses AI-driven regression to analyze orthopedic surgery trends, particularly in ACL reconstruction and hip arthroscopy.</td></tr><tr><td>Overton & Eicker (2025)</td><td>Applies linear and logistic regression to assess fertility and milk production efficiency in Holstein dairy cows.</td></tr></tbody></table>

### 16.2.7.3 Mathematical Analysis of Linear regression

Linear regression is a fundamental statistical method used to model the relationship between a dependent variable *y* and one or more independent variables *x*₁, *x*₂, ..., *x*ₙ. The goal of linear regression is to determine the best-fitting linear function that describes this relationship by minimizing the error between the predicted values and the actual observations. In its most general form, the linear regression model can be written as

$$y = \beta_0 + \sum_{i=1}^{n} \beta_i x_i + \epsilon \quad (16.124)$$

where *y* is the dependent variable, *x*ᵢ are the independent variables, $\beta_0$ is the intercept term, $\beta_i$ are the regression coefficients corresponding to each independent variable, and $\epsilon$ represents the error
<figure><img src="image_179.png" /><figcaption>Figure 16.11: Framework of the Linear Regression</figcaption></figure>

Image Credit: By Krishnavedala - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=15462765

In linear regression, an underlying relationship (blue) exists between the dependent variable (y) and the independent variable (x), with the observed data points (red) resulting from random variations (green) around this trend

term, which accounts for the variability in y that is not explained by the independent variables. If there is only one independent variable, the model reduces to simple linear regression:

$$y = \beta_0 + \beta_1 x + \epsilon. \tag{16.125}$$

In matrix notation, for a dataset consisting of m observations, we define

$$y = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_m \end{bmatrix}, \quad X = \begin{bmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1n} \\ 1 & x_{21} & x_{22} & \cdots & x_{2n} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{m1} & x_{m2} & \cdots & x_{mn} \end{bmatrix}, \quad \beta = \begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \vdots \\ \beta_n \end{bmatrix}, \quad \epsilon = \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_m \end{bmatrix}. \tag{16.126}$$
<figure><img src="image_180.png" /><figcaption>Figure 16.12: Simple Linear Regression</figcaption></figure>

Image Credit: By Sewaqu - Own work, Public Domain, https://commons.wikimedia.org/w/index.php?curid=11967659

This is an example of simple linear regression, characterized by having only one independent variable.

Thus, the linear regression model can be written compactly as

$$y = X\beta + \epsilon. \qquad (16.127)$$

The fundamental objective of linear regression is to estimate the coefficient vector $\beta$ in such a way that the sum of squared errors (SSE) is minimized. The sum of squared errors is given by

$$S(\beta) = \sum_{i=1}^{m} \epsilon_{i}^{2} = \sum_{i=1}^{m} (y_{i} - x_{i}^{T}\beta)^{2}. \qquad (16.128)$$

In matrix form, the sum of squared errors can be written as

$$S(\beta) = (y - X\beta)^{T}(y - X\beta). \qquad (16.129)$$

To find the optimal $\beta$, we take the derivative of $S(\beta)$ with respect to $\beta$ and set it equal to zero:

$$\frac{\partial S(\beta)}{\partial \beta} = -2X^{T}(y - X\beta) = 0. \qquad (16.130)$$

Solving for $\beta$, we obtain the normal equation:

$$X^{T}X\beta = X^{T}y. \qquad (16.131)$$

If $X^T X$ is invertible, the solution for $\beta$ is

$$\beta = (X^T X)^{-1} X^T y. \qquad (16.132)$$
<figure><img src="image_181.png" /><figcaption>Figure 16.13: Cubic Polynomial Regression</figcaption></figure>

Image Credit: By Skbkekas - Own work, CC BY 3.0, https://commons.wikimedia.org/w/index.php?curid=6457163

This example demonstrates cubic polynomial regression, which falls under the category of linear regression. Although it models data with a curved function, it is considered statistically linear because the regression function $E(y|x)$ depends linearly on the unknown parameters being estimated. Therefore, polynomial regression is recognized as a specific form of multiple linear regression

The coefficient of determination $R^2$ is given by

$$R^2 = 1 - \frac{\sum_{i=1}^{m} (y_i - \hat{y}_i)^2}{\sum_{i=1}^{m} (y_i - \bar{y})^2}. \qquad (16.133)$$

The ridge regression estimate of $\beta$ is given by

$$\beta_{\text{ridge}} = (\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}, \qquad (16.134)$$

while the Lasso regression estimate is obtained by solving

$$\beta_{\text{lasso}} = \arg \min_{\beta} (\|\mathbf{y} - \mathbf{X}\beta\|^2 + \lambda \|\beta\|_1). \qquad (16.135)$$

The linear regression model can also be extended to a probabilistic framework:

$$y|\mathbf{x} \sim \mathcal{N}(\mathbf{x}^T\beta, \sigma^2), \qquad (16.136)$$

which allows for Bayesian interpretations where a prior distribution is placed on $\beta$ and updated using observed data to obtain a posterior distribution, leading to Bayesian linear regression.
16.2.8 Logistic regression

16.2.8.1 Literature Review of Logistic regression

Logistic regression has evolved through a rigorous theoretical foundation established by key con-
tributions in statistics, probability modeling, and machine learning. The earliest seminal work by
Cox (1958) [804] introduced the logistic function as a probability model for binary outcomes, rigor-
ously deriving the logit transformation to linearize the probability structure and enable the use of
maximum likelihood estimation (MLE) for parameter inference. This foundational work not only
formalized the statistical methodology but also provided a justification for using logistic regres-
sion over linear probability models, addressing issues such as non-linearity and heteroskedasticity.
Later, Nelder and Wedderburn (1972) [806] expanded upon this foundation by introducing Gener-
alized Linear Models (GLMs), of which logistic regression is a specific case. Their work rigorously
defined the role of exponential family distributions, formalizing the link function approach and
introducing a unified statistical framework where regression models could be generalized beyond
normally distributed response variables. This pivotal development allowed logistic regression to be
understood within a broader statistical paradigm, establishing it as a standard technique for binary
classification.

Subsequent research focused on model assessment and diagnostic methods to ensure the reliability
of logistic regression results. Haberman (1990) [807] rigorously developed deviance-based tests,
allowing analysts to assess goodness-of-fit and compare nested models using likelihood ratio tests.
This work introduced the notion of model adequacy beyond traditional R-squared metrics used in
linear regression. Similarly, Hosmer and Lemeshow (1980) [808] introduced the Hosmer-Lemeshow
test, which rigorously quantified calibration errors by partitioning observations into quantile groups
and comparing predicted and observed probabilities. Their methodology provided a practical yet
statistically rigorous tool for validating logistic regression models in real-world applications. The
rigorous mathematical structure of logistic regression was further explored by McCullagh (2019)
[809], who formalized the theory of iteratively reweighted least squares (IRLS) for estimating model
parameters and proved asymptotic properties of logistic regression coefficients. Their contributions
solidified logistic regression as a theoretically sound and computationally efficient method in sta-
tistical modeling.

With the increasing complexity of datasets and computational advancements, logistic regression
was further extended into machine learning and regularization frameworks. Hastie, Tibshirani,
and Friedman (2001) [139] rigorously analyzed logistic regression as a classification tool, compar-
ing it with support vector machines (SVMs) and decision trees. They introduced regularization
techniques, such as ridge regression (L2 penalty) and LASSO (L1 penalty), to combat issues of over-
fitting and multicollinearity. In parallel, Green and Silverman (1994) expanded logistic regression
to nonparametric settings, developing methods for spline smoothing and kernel-based regression,
allowing the model to capture nonlinear relationships without relying on rigid parametric assump-
tions. Another significant computational advancement came from Firth (1993) [810], who proposed
a bias reduction method using penalized maximum likelihood estimation (PMLE). This technique,
known as Firth’s logistic regression, rigorously mitigates the small-sample bias that arises in rare
event classification by modifying the score equations of MLE, thereby preventing infinite estimates
in perfectly separated data.

Finally, extensions of logistic regression into hierarchical and Bayesian settings have been rigor-
ously explored to account for structured data and uncertainty in parameter estimation. King and
Zeng (2001) [811] addressed a fundamental issue in rare event data, where traditional logistic regres-
sion tends to underestimate event probabilities due to data imbalance. Their corrected probability
estimation approach adjusted the likelihood function to provide more accurate parameter estimates
in imbalanced datasets, such as those found in epidemiology and fraud detection. Gelman and Hill (2007) [812] extended logistic regression into Bayesian hierarchical models, introducing random effects logistic regression to handle grouped data structures. They rigorously developed Bayesian priors for logistic regression coefficients, allowing for shrinkage estimation and improved parameter stability in small-sample settings. These contributions have collectively transformed logistic regression from a simple binary classification method into a statistically rigorous, computationally efficient, and widely applicable tool used in fields ranging from biomedical sciences to artificial intelligence.

<table><caption>Table 16.23: Summary of Contributions in Logistic Regression</caption><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Cox (1958)</td><td>Formulated logistic regression using the logit function and MLE.</td></tr><tr><td>Nelder & Wedderburn (1972)</td><td>Introduced GLMs, formalizing logistic regression in a unified statistical framework.</td></tr><tr><td>Haberman (1973)</td><td>Developed deviance tests for logistic regression goodness-of-fit.</td></tr><tr><td>Hosmer & Lemeshow (1980)</td><td>Introduced the Hosmer-Lemeshow test for model calibration.</td></tr><tr><td>McCullagh & Nelder (1983)</td><td>Provided a rigorous theoretical foundation for GLMs, including logistic regression.</td></tr><tr><td>Hastie, Tibshirani & Friedman (2001)</td><td>Discussed regularization methods and logistic regression in the context of statistical learning.</td></tr><tr><td>Green & Silverman (1994)</td><td>Extended logistic regression to nonparametric settings.</td></tr><tr><td>Firth (1993)</td><td>Proposed bias reduction techniques for small-sample logistic regression (Firth's correction).</td></tr><tr><td>King & Zeng (2001)</td><td>Addressed logistic regression's biases in rare event data.</td></tr><tr><td>Gelman & Hill (2007)</td><td>Developed Bayesian and hierarchical logistic regression models.</td></tr></tbody></table>

### 16.2.8.2 Recent Literature Review of Logistic regression

Logistic regression is widely used across diverse fields for modeling binary outcomes, and the latest research underscores its broad applicability. For instance, Sani et al. (2025) [813] applied logistic regression to analyze sociodemographic factors affecting contraceptive use among Nigerian women, revealing significant regional disparities. Their findings indicate that education and economic status strongly influence contraceptive intent, offering insights for public health interventions. Similarly, Dorsey et al. (2025) [814] employed logistic regression to investigate how piping plovers select nesting sites, demonstrating that broader viewsheds enhance predator detection and nest survival. This study contributes to ecological conservation by highlighting the role of visibility in avian habitat selection. In the realm of linguistics, Slawny et al. (2025) [815] leveraged logistic regression to explore the impact of language dominance and ability on bilingual parent-child communication, showing that linguistic environments significantly shape language transmission patterns within families.

In medical research, logistic regression plays a crucial role in identifying risk factors for diseases and treatment outcomes. Waller et al. (2025) [815] utilized logistic regression to examine the association between maternal diarrhea during the periconceptional period and birth defects, identifying key risk factors for fetal abnormalities. Similarly, Beyeler et al. (2025) [817] investigated the susceptibility vessel sign (SVS) in stroke patients undergoing thrombectomy, demonstrating that certain SVS characteristics significantly influence treatment efficacy. Another notable study by Yedavalli et al. (2025) [818] applied logistic regression to assess the relationship between hypoperfusion intensity and stroke recovery, revealing that a hypoperfusion intensity ratio below 0.4 correlates with favorable patient outcomes. These findings are critical in guiding clinical decisions for ischemic stroke treatment. Ghosh (2020) [1385] discussed an attempted proof of the Riemann
Hypothesis, a central unsolved problem in number theory concerning the nontrivial zeros of the Riemann zeta function. It aimed to provide a theoretical justification for why all nontrivial zeros should have a real part equal to one-half. The work reflects an effort to contribute to one of mathematics' most significant and long-standing open problems. Additionally, Yang et al. (2025) [820] explored the link between left ventricular function and cerebral small vessel disease, showing that cardiac health can serve as a biomarker for neurological risks. This research advances our understanding of cardiovascular-neurological interactions and their implications for stroke prevention.

Beyond healthcare, logistic regression is instrumental in behavioral and environmental studies. Aarakit et al. (2025) [819] examined how social networks and neighborhood effects impact solar energy adoption, employing logistic regression to determine the influence of community engagement on renewable energy choices. Their findings suggest that policy interventions should target social cohesion to increase solar panel adoption rates. Similarly, Cortese (2025) [821] investigated maternal and neonatal outcomes in women diagnosed with ADHD, uncovering heightened perinatal risks through logistic regression analysis. This study contributes to maternal health research by emphasizing the need for tailored healthcare strategies for expectant mothers with ADHD. Furthermore, Gaspar et al. (2025) [822] utilized logistic regression to analyze risk factors for bleeding in patients with thrombotic antiphospholipid syndrome, providing valuable clinical insights for optimizing anticoagulant therapy. These studies illustrate the versatility of logistic regression in both social sciences and healthcare, reinforcing its role in predictive modeling and risk assessment.

In conclusion, the recent literature showcases the far-reaching applications of logistic regression, from public health and medical research to environmental science and behavioral studies. Whether identifying predictors of contraceptive use, evaluating ecological selection mechanisms, or assessing treatment outcomes in stroke patients, logistic regression remains a powerful statistical tool. Its ability to model complex relationships and quantify risk factors makes it indispensable in research and decision-making across disciplines. As studies continue to refine logistic regression methodologies and incorporate advanced modeling techniques, its utility in predictive analytics and policy formulation will only expand. These advancements underscore the necessity of rigorous statistical approaches in tackling real-world challenges, ultimately driving more effective and data-informed solutions.

<table><thead><tr><td>Study Title</td><td>Main Contribution</td></tr></thead><tbody><tr><td>Sani et al. 2025</td><td>Logistic regression used to study sociodemographic predictors of contraception adoption.</td></tr><tr><td>Dorsey et al. 2025</td><td>Assesses how visual exposure influences shorebird nesting behavior.</td></tr><tr><td>Slawny et al. 2025</td><td>Examined how language dominance affects bilingual family communication.</td></tr><tr><td>Waller et al. 2025</td><td>Identifies birth defect risk factors using logistic regression analysis.</td></tr><tr><td>Beyeler et al. 2025</td><td>Evaluated impact of vessel characteristics on stroke interventions.</td></tr><tr><td>Yedavalli et al. 2025</td><td>Uses logistic regression to assess hypoperfusion intensity ratio and stroke outcomes.</td></tr><tr><td>Aarakit et al. 2025</td><td>Examines social network effects on renewable energy choices.</td></tr><tr><td>Yang et al. 2025</td><td>Studied how cardiac health influences neurological risks.</td></tr><tr><td>Cortese 2025</td><td>Identifies increased perinatal risks in ADHD-diagnosed mothers.</td></tr></tbody></table>

*Continued on next page*
<table><caption>Table 16.24: Summary of Recent Contributions in Logistic Regression</caption><thead><tr><td>Study Title</td><td>Main Contribution</td></tr></thead><tbody><tr><td>Gaspar et al. 2025</td><td>Evaluated factors affecting bleeding risks under anticoagulant therapy.</td></tr></tbody></table>

<figure><img src="image_182.png" /><figcaption>Figure 16.14: Example of a logistic regression curve fitted to data</figcaption></figure>

Image Credit: By Canley - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=116449187

This is an example of a logistic regression curve applied to data, depicting how the estimated probability of passing an exam (a binary dependent variable) varies with the number of hours spent studying (a single independent variable)

### 16.2.8.3 Mathematical Analysis of Logistic regression

Logistic regression is a fundamental statistical and machine learning method used for binary classification. Given an input feature vector $\mathbf{x} = (x_1, x_2, \dots, x_n)$, the logistic regression model aims to predict the probability that a binary outcome $y \in \{0, 1\}$ takes the value 1, given $\mathbf{x}$. Mathematically, the probability of class 1 is modeled using the logistic (sigmoid) function applied to a linear combination of the input features:

$$P(y = 1 | \mathbf{x}) = \sigma(\mathbf{w}^{\top}\mathbf{x} + b) \qquad (16.137)$$

where $\mathbf{w} = (w_1, w_2, \dots, w_n)$ is the weight vector, $b$ is the bias term, and $\sigma(z)$ is the sigmoid function given by:

$$\sigma(z) = \frac{1}{1 + e^{-z}} \qquad (16.138)$$
Thus, the probability of class 0 is:

$$P(y = 0 | \mathbf{x}) = 1 - P(y = 1 | \mathbf{x}) = 1 - \sigma(\mathbf{w}^{\top}\mathbf{x} + b) \quad (16.139)$$

The logistic regression model is trained by maximizing the likelihood function, which represents the probability of observing the given set of labeled data points ($\mathbf{x}_i, y_i$) for $i = 1, \dots, m$. Assuming that the training examples are independently and identically distributed (i.i.d.), the likelihood function is:

$$L(\mathbf{w}, b) = \prod_{i=1}^{m} P(y_i | \mathbf{x}_i) \quad (16.140)$$

Expanding this using the probabilities defined above, the likelihood function is:

$$L(\mathbf{w}, b) = \prod_{i=1}^{m} \sigma(\mathbf{w}^{\top}\mathbf{x}_i + b)^{y_i} (1 - \sigma(\mathbf{w}^{\top}\mathbf{x}_i + b))^{1-y_i} \quad (16.141)$$

Instead of maximizing the likelihood, it is more convenient to maximize the log-likelihood function:

$$\ell(\mathbf{w}, b) = \sum_{i=1}^{m} \left[ y_i \log \sigma(\mathbf{w}^{\top}\mathbf{x}_i + b) + (1 - y_i) \log(1 - \sigma(\mathbf{w}^{\top}\mathbf{x}_i + b)) \right] \quad (16.142)$$

To find the optimal values of $\mathbf{w}$ and $b$, we compute the gradient of the log-likelihood function. The derivative of the sigmoid function is:

$$\sigma'(z) = \sigma(z)(1 - \sigma(z)) \quad (16.143)$$

Using this, the gradient of the log-likelihood with respect to $w_j$ is:

$$\frac{\partial \ell}{\partial w_j} = \sum_{i=1}^{m} (y_i - \sigma(\mathbf{w}^{\top}\mathbf{x}_i + b)) x_{ij} \quad (16.144)$$

Similarly, the gradient with respect to $b$ is:

$$\frac{\partial \ell}{\partial b} = \sum_{i=1}^{m} (y_i - \sigma(\mathbf{w}^{\top}\mathbf{x}_i + b)) \quad (16.145)$$

To optimize $\mathbf{w}$ and $b$, we use gradient ascent, updating the parameters iteratively as follows:

$$w_j^{(t+1)} = w_j^{(t)} + \alpha \sum_{i=1}^{m} (y_i - \sigma(\mathbf{w}^{(t)\top}\mathbf{x}_i + b^{(t)})) x_{ij} \quad (16.146)$$

$$b^{(t+1)} = b^{(t)} + \alpha \sum_{i=1}^{m} (y_i - \sigma(\mathbf{w}^{(t)\top}\mathbf{x}_i + b^{(t)})) \quad (16.147)$$

If we include regularization, the loss function becomes:

$$J(\mathbf{w}) = - \sum_{i=1}^{m} \left[ y_i \log \sigma(\mathbf{w}^{\top}\mathbf{x}_i + b) + (1 - y_i) \log(1 - \sigma(\mathbf{w}^{\top}\mathbf{x}_i + b)) \right] + \frac{\lambda}{2} \|\mathbf{w}\|^2 \quad (16.148)$$

For multi-class classification, logistic regression is extended using the softmax function:

$$P(y = k | \mathbf{x}) = \frac{e^{\mathbf{w}_k^{\top}\mathbf{x} + b_k}}{\sum_{j=1}^{K} e^{\mathbf{w}_j^{\top}\mathbf{x} + b_j}} \quad (16.149)$$
with the corresponding loss function:

$$J(\mathbf{W}) = - \sum_{i=1}^{m} \sum_{k=1}^{K} y_{ik} \log P(y = k | \mathbf{x}_i) \qquad (16.150)$$

where the parameter update rule follows:

$$\mathbf{w}_k^{(t+1)} = \mathbf{w}_k^{(t)} + \alpha \sum_{i=1}^{m} (y_{ik} - P(y = k | \mathbf{x}_i))\mathbf{x}_i \qquad (16.151)$$

In conclusion, logistic regression provides a mathematically rigorous yet computationally efficient approach for binary and multi-class classification problems.

## 16.2.9 Linear discriminant analysis

### 16.2.9.1 Literature Review of Linear discriminant analysis

Linear Discriminant Analysis (LDA) is a fundamental statistical and machine learning technique, originating from Fisher's work in 1936 [784], which introduced the Fisher Discriminant to maximize class separability while minimizing intra-class variance. This foundational principle was later extended by Anderson in 1958 [785], who provided a multivariate statistical analysis of LDA, addressing the Gaussian assumption for class distributions and proving its connection to the Mahalanobis distance and Bayesian classification. Meanwhile, Rao in 1948 [786] formalized the canonical discriminant analysis framework, generalizing LDA to multiple discriminant functions and rigorously proving its statistical properties under various conditions. These early works laid the theoretical groundwork for LDA, establishing its optimality under homoscedastic Gaussian assumptions, a condition that remains a key analytical criterion in statistical pattern recognition. Subsequent advancements in LDA focused on expanding its applicability beyond classical statistics. Duda and Hart in 2001 [787] systematically compared LDA to alternative classification techniques such as Quadratic Discriminant Analysis (QDA) and Nearest Neighbor Classifiers, emphasizing LDA's geometric and probabilistic interpretations. McLachlan in 2004 [788] further extended the discussion by addressing regularized LDA, kernelized versions of LDA, and mixture models, providing a mathematically rigorous analysis of the method's limitations in high-dimensional settings. In parallel, Hastie, Tibshirani, and Friedman in 2009 [139] positioned LDA within the broader landscape of machine learning, exploring its connections with logistic regression, support vector machines (SVMs), and shrinkage methods such as Diagonal LDA. These developments significantly broadened LDA's scope, making it a crucial tool in modern classification theory and applied data science.

Beyond statistical and machine learning applications, LDA has played a critical role in computer vision and nonlinear classification problems. A significant application was introduced by Belhumeur et al. in 1997 [789], who developed the Fisherfaces method, leveraging LDA for robust face recognition that outperformed Principal Component Analysis (PCA)-based methods under varying lighting conditions and facial expressions. Ghosh (2019) [1352] introduced a Markov Chain Monte Carlo (MCMC) based Bayesian framework for adaptive sample size reestimation in confirmatory clinical trials. It overcomes the limitations of conventional point estimate methods by modeling uncertainty through sequential updating of posterior distributions at each interim stage. The approach enables more accurate and flexible sample size adjustments under varying trial conditions, demonstrated through application to a depression study. The scope of LDA was further expanded by Mika et al. in 1999 [790] with the introduction of Kernel Fisher Discriminant Analysis (KFDA), which mapped input data into a high-dimensional feature space via kernel functions before applying Fisher's criterion, making LDA suitable for nonlinear classification problems. More recent advancements, such as those by Ye and Yu in 2005 [791] and Sugiyama in 2007 [792], addressed
challenges in high-dimensional, low-sample-size problems and multimodal data distributions, respectively. Ye's Generalized Discriminant Analysis (GDA) provided theoretical solutions to issues arising when within-class covariance matrices are singular, while Sugiyama's Local Fisher Discriminant Analysis (LFDA) introduced a localized version of LDA that effectively preserves both global and local structures in complex datasets. These refinements further strengthened LDA's theoretical robustness and adaptability to real-world, high-dimensional, and structured data scenarios.

Thus, the trajectory of LDA research has evolved from Fisher's statistical classification framework to high-dimensional machine learning applications, bridging the gap between classical multivariate statistics and modern computational intelligence. The method has been rigorously analyzed, generalized, and extended across various domains, ensuring its continued relevance in statistical learning, pattern recognition, and artificial intelligence.

<table><caption>Table 16.25: Summary of Contributions to Linear Discriminant Analysis (LDA)</caption><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Fisher (1936)</td><td>Introduced LDA, maximizing inter-class separation while minimizing intra-class variance.</td></tr><tr><td>Anderson (1958)</td><td>Established LDA's statistical foundation under multivariate Gaussian assumptions.</td></tr><tr><td>Rao (1948)</td><td>Generalized LDA via canonical discriminant analysis and established its statistical properties.</td></tr><tr><td>Duda, Hart & Stork (2001)</td><td>Compared LDA to other classifiers and provided geometric/probabilistic insights.</td></tr><tr><td>McLachlan (2004)</td><td>Discussed regularized LDA, kernel LDA, and mixture model extensions.</td></tr><tr><td>Hastie, Tibshirani & Friedman (2009)</td><td>Presented LDA in a machine learning context, linking it to logistic regression and SVMs.</td></tr><tr><td>Belhumeur et al. (1997)</td><td>Developed Fisherfaces, applying LDA to face recognition for robustness.</td></tr><tr><td>Mika et al. (1999)</td><td>Proposed Kernel Fisher Discriminant Analysis (KFDA) for nonlinear classification.</td></tr><tr><td>Ye (2005)</td><td>Analyzed Generalized Discriminant Analysis (GDA) for high-dimensional, low-sample-size (HDLSS) problems.</td></tr><tr><td>Sugiyama (2007)</td><td>Developed Local Fisher Discriminant Analysis (LFDA) for multimodal classification.</td></tr></tbody></table>

### 16.2.9.2 Recent Literature Review of Linear discriminant analysis

Linear Discriminant Analysis (LDA) has demonstrated remarkable versatility across multiple domains, including medical diagnostics, psychology, bioinformatics, and deep learning applications. Hartmann et al. (2025) [793] applied LDA to distinguish emotional states based on bodily sensation mapping, revealing distinct activation patterns for different emotions. This underscores LDA's effectiveness in psychophysiological research, as it enables precise classification of bodily responses to emotional states. Similarly, Garrido-Tamayo et al. (2025) [794] leveraged LDA in malaria diagnos-
tics, achieving a 91.67 percentage classification accuracy in distinguishing P. falciparum-infected red blood cells based on autofluorescence spectra. This research exemplifies LDA's potential in biomedical imaging and pathology, particularly for early disease detection. Meanwhile, Li and Jiang (2025) [795] integrated LDA with graph convolutional networks for petroleum reservoir analysis, demonstrating how LDA enhances feature selection and classification in geophysical applications. In psychological and public health research, Nyembwe et al. (2025) [796] utilized LDA to assess how perceived discrimination affects blood pressure in Black mothers, showcasing its application in behavioral sciences. On the other hand, Singh et al. (2025) [797] employed LDA as a dimensionality reduction technique in facial expression recognition using CNN-BiLSTM networks. Their findings highlighted LDA's ability to enhance deep learning performance by filtering discriminative features while reducing computational complexity. Akter et al. (2025) [798] further explored LDA's capabilities in food quality assessment, comparing it with SVM and PLS-DA for detecting fruit surface defects via hyperspectral imaging. The study concluded that LDA, although effective, may be outperformed by more complex models when dealing with high-dimensional spectral data. Beyond human-centered applications, Feng et al. (2025) [799] demonstrated LDA's utility in oncology, classifying causes of death in colorectal and lung cancer patients, reinforcing its relevance in medical prognosis and predictive analytics. Similarly, Chick et al. (2025) [801] utilized LDA in microbiome analysis, identifying bacterial strains associated with gut inflammation in broiler chickens, showcasing its efficacy in bioinformatics and microbial classification. Meanwhile, Miao et al. (2025) [802] introduced an LDA-PCA hybrid model for breast cancer molecular subtyping, illustrating its role in cancer diagnostics when combined with spectral imaging data. Finally, Rohan et al. (2025) [803] compared LDA with ensemble AI techniques for heart disease prediction, revealing that while LDA remains a strong statistical classifier, modern ensemble models often outperform it in complex, non-linear data environments. In summary, these studies reinforce LDA's adaptability across disciplines, from psychophysiology and medicine to machine learning and geophysics. While LDA remains a powerful tool for classification and dimensionality reduction, recent advancements suggest that it performs best when integrated with more sophisticated models like CNNs, random forests, and ensemble learning techniques. Its continued application in high-impact research areas highlights its relevance in both traditional statistical analysis and contemporary AI-driven methodologies.

<table><caption>Table 16.26: Summary of Recent Contributions to Linear Discriminant Analysis (LDA)</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>W. Wolff, C.S. Martarelli (2025)</td><td>Used LDA for emotion classification via bodily sensation mapping, revealing distinct bodily activation patterns for different emotional states.</td></tr><tr><td>A. Rincón Santamaría, F.E. Hoyos (2025)</td><td>Achieved 91.67% accuracy in classifying P. falciparum-infected red blood cells using LDA, demonstrating its effectiveness in medical diagnostics.</td></tr><tr><td>B. Li, S. Jiang (2025)</td><td>Applied LDA with graph convolutional networks for petroleum reservoir fluid classification, improving feature selection in geophysical studies.</td></tr><tr><td>B.A. Caceres, A. Nyembwe (2025)</td><td>Used LDA to analyze the psychological and physiological effects of discrimination on blood pressure among young Black mothers.</td></tr><tr><td>S.K. Singh, M. Kumar (2025)</td><td>Integrated LDA for dimensionality reduction in a CNN-BiLSTM model, enhancing facial expression recognition performance.</td></tr></tbody></table>

*Continued on next page*
(Continued from previous page)

<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>T. Akter, M.A. Faqeerzada (2025)</td><td>Compared LDA with SVM and PLS-DA for hyperspectral imaging-based fruit defect classification, validating LDA's effectiveness in food quality assessment.</td></tr><tr><td>F. Deng, L. Zhang (2025)</td><td>Applied LDA to classify causes of death in colorectal and lung cancer patients, showcasing its utility in predictive healthcare analytics.</td></tr><tr><td>H.M. Chick, N. Sparks (2025)</td><td>Used LDA in microbiome analysis to classify bacterial strains linked to gut inflammation in broiler chickens.</td></tr><tr><td>X. Miao, L. Xu (2025)</td><td>Developed an LDA-PCA hybrid model for breast cancer molecular subtyping using spectral imaging data, improving diagnostic accuracy.</td></tr><tr><td>D. Rohan, G.P. Reddy (2025)</td><td>Compared LDA with ensemble AI techniques for heart disease prediction, demonstrating LDA's limitations in non-linear data analysis.</td></tr></tbody></table>

### 16.2.9.3 Mathematical Analysis of Linear discriminant analysis

Linear Discriminant Analysis (LDA) is a fundamental technique in statistical pattern recognition and machine learning, particularly used for dimensionality reduction and classification. It seeks to find a linear transformation that maximizes the separation between multiple classes in a dataset by projecting the data onto a lower-dimensional space while preserving as much of the class-discriminative information as possible. Mathematically, given a dataset with $N$ samples, each represented as a $d$-dimensional vector $\mathbf{x}_i \in \mathbb{R}^d$, and assuming the data belongs to $C$ distinct classes labeled $\{1, 2, \dots, C\}$, LDA finds an optimal projection matrix $\mathbf{W}$ such that the transformed data maximizes class separability.

To achieve this, LDA constructs two scatter matrices: the **within-class scatter matrix** $\mathbf{S}_W$ and the **between-class scatter matrix** $\mathbf{S}_B$. The within-class scatter matrix is defined as the sum of the covariance matrices of each class:

$$ \mathbf{S}_W = \sum_{c=1}^{C} \sum_{\mathbf{x}_i \in \mathcal{X}_c} (\mathbf{x}_i - \boldsymbol{\mu}_c)(\mathbf{x}_i - \boldsymbol{\mu}_c)^T \quad (16.152) $$

where $\boldsymbol{\mu}_c$ is the mean of class $c$, given by

$$ \boldsymbol{\mu}_c = \frac{1}{N_c} \sum_{\mathbf{x}_i \in \mathcal{X}_c} \mathbf{x}_i \quad (16.153) $$

and $N_c$ is the number of samples in class $c$. The between-class scatter matrix is defined as

$$ \mathbf{S}_B = \sum_{c=1}^{C} N_c (\boldsymbol{\mu}_c - \boldsymbol{\mu})(\boldsymbol{\mu}_c - \boldsymbol{\mu})^T \quad (16.154) $$

where $\boldsymbol{\mu}$ is the global mean of all data points:

$$ \boldsymbol{\mu} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{x}_i. \quad (16.155) $$

The objective of LDA is to maximize the ratio of the determinant of $\mathbf{S}_B$ to the determinant of $\mathbf{S}_W$, which leads to the following optimization problem:

$$ \mathbf{W}^* = \arg \max_{\mathbf{W}} \frac{\det(\mathbf{W}^T \mathbf{S}_B \mathbf{W})}{\det(\mathbf{W}^T \mathbf{S}_W \mathbf{W})}. \quad (16.156) $$
<figure><img src="image_183.png" /><figcaption>Figure 16.15: Linear discriminant analysis on a two dimensional space with two classes</figcaption></figure>

Image Credit: CC BY-SA 4.0, https://en.wikipedia.org/w/index.php?curid=76945442

This example illustrates linear discriminant analysis in a two-dimensional space with two classes. The true data-generating parameters define the Bayes boundary, whereas the realized data points are used to estimate the boundary

This is solved by finding the eigenvectors $\mathbf{w}_i$ corresponding to the largest eigenvalues $\lambda_i$ of the generalized eigenvalue problem:

$$S_B \mathbf{w} = \lambda S_W \mathbf{w}. \qquad (16.157)$$

Since $S_W$ is symmetric and positive definite under typical conditions, it is invertible, leading to the equivalent formulation:

$$S_W^{-1} S_B \mathbf{w} = \lambda \mathbf{w}. \qquad (16.158)$$

The eigenvectors corresponding to the largest $C-1$ eigenvalues form the columns of $\mathbf{W}$, giving the optimal projection that maximizes class separability in the lower-dimensional space. The transformed feature vectors are then given by

$$\mathbf{y}_i = \mathbf{W}^T \mathbf{x}_i. \qquad (16.159)$$

The projection preserves the information necessary for classification while reducing dimensionality. Given a new sample $\mathbf{x}$, classification can be performed using a simple distance metric such as the
Mahalanobis distance:

$$d_c(\mathbf{x}) = (\mathbf{W}^T\mathbf{x} - \mathbf{W}^T\boldsymbol{\mu}_c)^T\boldsymbol{\Sigma}_c^{-1}(\mathbf{W}^T\mathbf{x} - \mathbf{W}^T\boldsymbol{\mu}_c), \quad (16.160)$$

where $\boldsymbol{\Sigma}_c$ is the covariance matrix of the projected class distributions. The decision rule is then:

$$\hat{c} = \arg \min_c d_c(\mathbf{x}). \quad (16.161)$$

When the class distributions are assumed to be Gaussian with identical covariances, LDA can be interpreted as finding the optimal decision boundary that minimizes classification error under the Bayes decision framework. In the two-class case, LDA reduces to Fisher's Linear Discriminant, where the optimal direction $\mathbf{w}$ is obtained as

$$\mathbf{w} = \mathbf{S}_W^{-1}(\boldsymbol{\mu}_1 - \boldsymbol{\mu}_2). \quad (16.162)$$

The decision boundary is a hyperplane defined by

$$\mathbf{w}^T\mathbf{x} = b, \quad (16.163)$$

where $b$ is a threshold obtained by projecting the mean of the two classes:

$$b = \frac{1}{2} \mathbf{w}^T (\boldsymbol{\mu}_1 + \boldsymbol{\mu}_2). \quad (16.164)$$

Thus, LDA provides a theoretically optimal solution for classification under Gaussian assumptions with equal covariances and achieves dimensionality reduction while maintaining class separability.

### 16.2.9.4 Multiclass Linear discriminant analysis

Let $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^N$ be a dataset where each feature vector $\mathbf{x}_i \in \mathbb{R}^d$ is associated with a class label $y_i \in \{1, 2, \dots, C\}$. We assume the class-conditional distributions follow a Gaussian form, meaning that for each class $c$, the feature vectors are drawn from a normal distribution $N(\boldsymbol{\mu}_c, \boldsymbol{\Sigma})$ with class-dependent mean $\boldsymbol{\mu}_c$ and a shared covariance matrix $\boldsymbol{\Sigma}$. Thus, for each class,

$$p(\mathbf{x} | y = c) = \frac{1}{(2\pi)^{d/2} |\boldsymbol{\Sigma}|^{1/2}} \exp \left( -\frac{1}{2} (\mathbf{x} - \boldsymbol{\mu}_c)^T \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu}_c) \right). \quad (16.165)$$

The prior probability of class $c$ is denoted as $P(y = c) = \frac{N_c}{N}$, where $N_c$ is the number of samples in class $c$ and $N$ is the total number of samples. The global mean vector of the dataset is given by

$$\boldsymbol{\mu} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{x}_i. \quad (16.166)$$

For the purpose of classification, the goal of LDA is to maximize the separability between different classes while minimizing the variance within each class. This is achieved by defining two scatter matrices: the within-class scatter matrix and the between-class scatter matrix. The within-class scatter matrix is given by

$$\mathbf{S}_W = \sum_{c=1}^{C} \sum_{\mathbf{x}_i \in C_c} (\mathbf{x}_i - \boldsymbol{\mu}_c)(\mathbf{x}_i - \boldsymbol{\mu}_c)^T. \quad (16.167)$$

Alternatively, we can define per-class scatter matrices

$$\mathbf{S}_c = \sum_{\mathbf{x}_i \in C_c} (\mathbf{x}_i - \boldsymbol{\mu}_c)(\mathbf{x}_i - \boldsymbol{\mu}_c)^T, \quad (16.168)$$
so that the within-class scatter matrix can be written as

$$ \mathbf{S}_W = \sum_{c=1}^{C} \mathbf{S}_c. \qquad (16.169) $$

The between-class scatter matrix is defined as

$$ \mathbf{S}_B = \sum_{c=1}^{C} N_c(\boldsymbol{\mu}_c - \boldsymbol{\mu})(\boldsymbol{\mu}_c - \boldsymbol{\mu})^T. \qquad (16.170) $$

These matrices satisfy the fundamental relationship

$$ \mathbf{S}_T = \mathbf{S}_W + \mathbf{S}_B, \qquad (16.171) $$

where $\mathbf{S}_T$ is the total scatter matrix given by

$$ \mathbf{S}_T = \sum_{i=1}^{N} (\mathbf{x}_i - \boldsymbol{\mu})(\mathbf{x}_i - \boldsymbol{\mu})^T. \qquad (16.172) $$

To find the optimal transformation, LDA seeks to maximize the following objective function:

$$ J(\mathbf{W}) = \text{tr}((\mathbf{W}^T \mathbf{S}_W \mathbf{W})^{-1}(\mathbf{W}^T \mathbf{S}_B \mathbf{W})). \qquad (16.173) $$

The solution to this maximization problem involves solving the **generalized eigenvalue problem**

$$ \mathbf{S}_B \mathbf{w} = \lambda \mathbf{S}_W \mathbf{w}. \qquad (16.174) $$

Equivalently, if $\mathbf{S}_W$ is invertible, we rewrite it as

$$ \mathbf{S}_W^{-1} \mathbf{S}_B \mathbf{w} = \lambda \mathbf{w}. \qquad (16.175) $$

Since the rank of $\mathbf{S}_B$ is at most $C-1$, there are at most $C-1$ nonzero eigenvalues, implying that the optimal dimensionality of the transformed space is at most $C-1$. To extract the projection matrix $\mathbf{W}$, we sort the eigenvectors of $\mathbf{S}_W^{-1}\mathbf{S}_B$ corresponding to the largest eigenvalues. The optimal projection matrix is then

$$ \mathbf{W} = [\mathbf{w}_1, \mathbf{w}_2, \dots, \mathbf{w}_{C-1}]. \qquad (16.176) $$

Once the transformation is applied, the projected data points are given by

$$ \mathbf{z} = \mathbf{W}^T \mathbf{x}. \qquad (16.177) $$

The classification rule in the transformed space follows from Bayes' Theorem:

$$ P(y = c | \mathbf{z}) \propto P(y = c)p(\mathbf{z} | y = c). \qquad (16.178) $$

If $\mathbf{z}$ follows a Gaussian distribution,

$$ p(\mathbf{z} | y = c) \sim \mathcal{N}(\mathbf{W}^T \boldsymbol{\mu}_c, \mathbf{W}^T \boldsymbol{\Sigma} \mathbf{W}). \qquad (16.179) $$

The decision boundary between classes $c$ and $c'$ is determined by solving

$$ \log P(y = c | \mathbf{z}) - \log P(y = c' | \mathbf{z}) = 0. \qquad (16.180) $$

When the class covariances are equal, this results in a **linear decision boundary**, given by

$$ \mathbf{W}^T(\boldsymbol{\mu}_c - \boldsymbol{\mu}_{c'}) + \frac{1}{2}(\boldsymbol{\mu}_c^T \mathbf{W} \mathbf{W}^T \boldsymbol{\mu}_c - \boldsymbol{\mu}_{c'}^T \mathbf{W} \mathbf{W}^T \boldsymbol{\mu}_{c'}) = 0. \qquad (16.181) $$
<figure><img src="image_184.png" /><figcaption>Figure 16.16: Visualisation for one-versus-all LDA axes for 4 classes in 3d</figcaption></figure>

Image Credit: By Amélia Oliveira Freitas da Silva - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=104693008

Alternatively, when the class covariances differ, the boundary is quadratic, requiring solving

$$ (\mathbf{z} - \boldsymbol{\mu}_c)^T \boldsymbol{\Sigma}_c^{-1} (\mathbf{z} - \boldsymbol{\mu}_c) - (\mathbf{z} - \boldsymbol{\mu}_{c'})^T \boldsymbol{\Sigma}_{c'}^{-1} (\mathbf{z} - \boldsymbol{\mu}_{c'}) = 0. \qquad (16.182) $$

The relationship between LDA and Fisher's Discriminant Analysis is seen by considering the case $C = 2$, where maximizing the ratio

$$ J(\mathbf{w}) = \frac{\mathbf{w}^T \mathbf{S}_B \mathbf{w}}{\mathbf{w}^T \mathbf{S}_W \mathbf{w}} \qquad (16.183) $$

yields an equivalent generalized eigenvalue problem. This follows from the trace expansion

$$ J(\mathbf{W}) = \sum_{i=1}^{C-1} \lambda_i. \qquad (16.184) $$

To address cases where $\mathbf{S}_W$ is singular, a regularized version is used:

$$ \mathbf{S}_W^* = \mathbf{S}_W + \alpha \mathbf{I}, \qquad (16.185) $$

where $\alpha$ is a small positive constant. The overall asymptotic properties of LDA follow from the concentration of measure in high-dimensional settings, where the **expected misclassification rate** follows a chi-squared distribution under Gaussianity.

**Asymptotic behavior of Linear Discriminant Analysis:** The asymptotic behavior of LDA
<figure><img src="image_185.png" /><figcaption>Figure 16.17: Projections along linear discriminant axes for 4 classes</figcaption></figure>

Image Credit: By Amélia Oliveira Freitas da Silva - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=104693007

classification error is a crucial aspect of its theoretical performance. Given a new sample $\mathbf{x}$, its classification is determined by

$$ \hat{y} = \arg \max_{c} P(y = c | \mathbf{x}). \qquad (16.186) $$

For large sample sizes, the classification error converges to the **Bayes error rate**, which can be expressed in terms of the Mahalanobis distance between class means:

$$ P_{\text{error}} \approx Q \left( \frac{1}{2} \sqrt{ \sum_{c=1}^{C} \sum_{c' \neq c} \frac{N_c N_{c'}}{N_c + N_{c'}} (\boldsymbol{\mu}_c - \boldsymbol{\mu}_{c'})^T \boldsymbol{\Sigma}^{-1} (\boldsymbol{\mu}_c - \boldsymbol{\mu}_{c'}) } \right), \qquad (16.187) $$

where $Q(x)$ is the **tail probability** of the standard normal distribution:

$$ Q(x) = \frac{1}{\sqrt{2\pi}} \int_{x}^{\infty} e^{-t^2/2} dt. \qquad (16.188) $$

**Eigenvalue Perturbation Analysis:** Since LDA involves solving the generalized eigenvalue problem

$$ S_B \mathbf{w} = \lambda S_W \mathbf{w}, \qquad (16.189) $$

the perturbation of eigenvalues due to sampling variability is given by

$$ \lambda_i^* = \lambda_i + \epsilon \mathbf{w}_i^T \mathbf{E} \mathbf{w}_i + O(\epsilon^2), \qquad (16.190) $$

where $\mathbf{E}$ represents a small perturbation in $S_B$ due to finite-sample effects. Using first-order per-
<figure><img src="image_186.png" /><figcaption>Figure 16.18: Eigenvalue Perturbation Analysis of Linear Discriminant Analysis (LDA). The dashed lines represent the original eigenvalues, while the solid curves show perturbed eigenvalues as noise strength increases. This illustrates the stability of discriminant directions under perturbations</figcaption></figure>

turbation theory, the perturbed eigenvector satisfies

$$ \mathbf{w}_i^* = \mathbf{w}_i + \sum_{j \neq i} \frac{\mathbf{w}_j^T \mathbf{E} \mathbf{w}_i}{\lambda_i - \lambda_j} \mathbf{w}_j + O(\epsilon^2). \quad (16.191) $$

In high-dimensional regimes where $d \gg N$, the Marčenko-Pastur law governs the eigenvalue distribution of the sample covariance matrix:

$$ \rho(\lambda) = \frac{1}{2\pi\lambda} \sqrt{(\lambda_+ - \lambda)(\lambda - \lambda_-)}, \quad \lambda \in [\lambda_-, \lambda_+], \quad (16.192) $$

where

$$ \lambda_{\pm} = \sigma^2(1 \pm \sqrt{\gamma})^2, \quad \gamma = \frac{d}{N}. \quad (16.193) $$

The smallest eigenvalue of $S_W$ follows a Tracy-Widom distribution, leading to an asymptotic bound on the condition number:

$$ \kappa(S_W) \approx \frac{(1 + \sqrt{\gamma})^2}{(1 - \sqrt{\gamma})^2}. \quad (16.194) $$

This result suggests that as $d/N \to 1$, $S_W$ becomes ill-conditioned, degrading LDA performance.

**Spectral Decomposition of the Discriminant Space:** To understand the projection structure, we decompose the Fisher matrix

$$ F = S_W^{-1} S_B. \quad (16.195) $$
<figure><img src="image_187.png" /><figcaption>Figure 16.19: Spectral decomposition of the discriminant space in Linear Discriminant Analysis (LDA). Left: original data projected onto the first two raw features. Right: data projected onto the discriminant space obtained from eigen-decomposition of the LDA problem. The eigenvalue ratios indicate the discriminative strength of each axis</figcaption></figure>

Since $S_B$ has rank at most $C-1$, its nonzero eigenvalues $\lambda_1, \dots, \lambda_{C-1}$ determine the energy distribution in the discriminant subspace. The total variance captured in the projection is

$$ \sum_{i=1}^{C-1} \lambda_i. \qquad (16.196) $$

The spectral radius of $\mathbf{F}$ determines the maximal discriminative power of LDA:

$$ \rho(\mathbf{F}) = \max_i \lambda_i. \qquad (16.197) $$

From Davis-Kahan theorem, the perturbation in the discriminant subspace is bounded by

$$ \sin \Theta(\mathbf{W}, \mathbf{W}^*) \le \frac{\|\mathbf{E}\|_2}{\lambda_{\min}(\mathbf{S}_W^{-1}\mathbf{S}_B)}, \qquad (16.198) $$

where $\Theta$ represents the principal angle between the estimated and true discriminant subspaces.

**Asymptotic Convergence of the Decision Boundary:** In the large-sample limit $N \to \infty$, the empirical scatter matrices satisfy

$$ \frac{1}{N}\mathbf{S}_W \to \mathbb{E}[(\mathbf{x} - \boldsymbol{\mu}_y)(\mathbf{x} - \boldsymbol{\mu}_y)^T], \qquad (16.199) $$

$$ \frac{1}{N}\mathbf{S}_B \to \sum_{c=1}^{C} P(y=c)(\boldsymbol{\mu}_c - \boldsymbol{\mu})(\boldsymbol{\mu}_c - \boldsymbol{\mu})^T. \qquad (16.200) $$

Thus, the empirical discriminant directions converge to the true optimal directions up to a normalization factor. The generalization error scales as

$$ \varepsilon \approx \frac{C-1}{N} \sum_{i=1}^{C-1} \frac{1}{\lambda_i}. \qquad (16.201) $$

This indicates that when the smallest discriminative eigenvalue $\lambda_{\min}$ is small, LDA exhibits poor generalization.
Asymptotic Convergence of LDA Decision Boundary

<figure><img src="image_188.png" /><figcaption>Figure 16.20: Asymptotic convergence of the decision boundary in Linear Discriminant Analysis (LDA). The black solid line is the true Bayes-optimal boundary. The blue dashed line shows the estimated LDA boundary for different sample sizes. As the number of training samples increases, the LDA boundary converges to the true boundary</figcaption></figure>

**Exact Asymptotic Bounds for the Misclassification Probability:** The Bayes error rate for multiclass classification is characterized by the probability that a sample from class c is assigned to a different class c'. In LDA, this error is asymptotically governed by the generalized Mahalanobis distance between class means, given by

$$D_{cc'}^2 = (\mu_c - \mu_{c'})^T \Sigma^{-1} (\mu_c - \mu_{c'}).$$

For asymptotically optimal classifiers, the misclassification probability can be approximated using the Chernoff bound:

$$P_{\text{error}} \le \sum_{c \ne c'} \exp \left( -\frac{1}{8} D_{cc'}^2 \right).$$

In the high-dimensional limit where $d \gg N$, we incorporate the Marčenko-Pastur correction for the empirical covariance matrix $S_W$, leading to the refined bound

$$P_{\text{error}} \le \sum_{c \ne c'} \exp \left( -\frac{1}{8} \frac{D_{cc'}^2}{1 + \gamma} \right),$$
<figure><img src="image_189.png" /><figcaption>Figure 16.21: Exact asymptotic bounds for the misclassification probability in Linear Discriminant Analysis (LDA). The black dashed line represents the Bayes error (optimal bound), the red dashed line shows the asymptotic upper bound of order $O(1/\sqrt{n})$, and the blue curve indicates the empirical misclassification probability. As the sample size increases, the empirical curve converges tightly between the bounds</figcaption></figure>

where $\gamma = d/N$ is the dimensionality ratio. As $\gamma \to 1$, the error rate approaches that of random guessing due to the eigenvalue collapse of $S_W$.

**Further Spectral Properties:** To fully characterize LDA, we analyze the spectral structure of the Fisher matrix

$$F = S_W^{-1}S_B.$$

Since $S_B$ has at most rank $C-1$, its eigenvalue spectrum consists of $C-1$ nonzero eigenvalues, denoted as $\lambda_1, \dots, \lambda_{C-1}$, and a bulk of zero eigenvalues. The total discriminative variance captured by LDA is given by the sum

$$\sum_{i=1}^{C-1} \lambda_i.$$

Using random matrix perturbation theory, we quantify the stability of the eigenvalues under sampling noise. For small perturbations $\mathbf{E}$ in $S_B$, the first-order correction to each eigenvalue satisfies

$$\lambda_i^* = \lambda_i + \mathbf{w}_i^T \mathbf{E} \mathbf{w}_i + O(\|\mathbf{E}\|^2).$$

The condition number of the Fisher matrix is

$$\kappa(\mathbf{F}) = \frac{\lambda_{\max}(\mathbf{S}_W^{-1}\mathbf{S}_B)}{\lambda_{\min}(\mathbf{S}_W^{-1}\mathbf{S}_B)}.$$
<figure><img src="image_190.png" /><figcaption>Figure 16.22: Condition number of the Fisher matrix $F = S_W^{-1} S_B$ in Linear Discriminant Analysis (LDA). The x-axis shows the number of samples per class (log scale), while the y-axis shows the condition number (log scale). A high condition number indicates numerical instability, while larger sample sizes improve stability</figcaption></figure>
For high-dimensional data, the smallest eigenvalue $\lambda_{min}$ obeys the Tracy-Widom law, leading to an expected conditioning bound

$$ \kappa(\mathbf{F}) \approx \frac{(1 + \sqrt{\gamma})^2}{(1 - \sqrt{\gamma})^2}. $$

Thus, LDA becomes ill-conditioned as $d/N \to 1$, reducing its effectiveness.

**Random Matrix Theory Perspective:** From a random matrix theory (RMT) viewpoint, we analyze the spectrum of $S_W^{-1}S_B$. Assuming that the entries of $X$ are Gaussian-distributed, the

<figure><img src="image_191.png" /><figcaption>Figure 16.23: Random Matrix Theory (RMT) perspective on Linear Discriminant Analysis (LDA). The blue histogram shows the empirical eigenvalue distribution of the Fisher matrix $F = S_W^{-1}S_B$, while the red curve represents the Marchenko-Pastur law predicted by RMT. Outlier eigenvalues outside the bulk correspond to informative class-separating directions, while the bulk follows the universal random-matrix spectrum</figcaption></figure>

eigenvalue distribution of $S_W$ follows the Marčenko-Pastur law:

$$ \rho(\lambda) = \frac{1}{2\pi\lambda} \sqrt{(\lambda_+ - \lambda)(\lambda - \lambda_-)}, \quad \lambda \in [\lambda_-, \lambda_+], $$

where

$$ \lambda_{\pm} = \sigma^2(1 \pm \sqrt{\gamma})^2. $$

For the generalized eigenvalue problem

$$ S_B w = \lambda S_W w, $$

the spectral properties of $S_W^{-1}S_B$ follow from the spiked covariance model:

$$ \lambda_i = \begin{cases} \frac{(\beta_i+1)(\beta_i+\gamma)}{\beta_i}, & \text{if } \beta_i > \sqrt{\gamma}, \\ 1 + O(N^{-1/2}), & \text{otherwise.} \end{cases} $$
where $\beta_i = \frac{1}{d}w_i^T S_B w_i$ represents the population discriminability. Thus, when $\beta_i$ is small, LDA fails to extract meaningful directions.

**Exact Rate of Eigenvalue Concentration:** For a given sample covariance matrix $S_W$, we analyze the eigenvalue concentration of $S_W^{-1}S_B$ by considering the extreme eigenvalues in the high-dimensional regime $d, N \to \infty$ with $\gamma = d/N$ fixed. Using random matrix theory (RMT), the

<figure><img src="image_192.png" /><figcaption>Figure 16.24: Exact rate of eigenvalue concentration in Linear Discriminant Analysis (LDA). The blue curve shows the empirical maximum deviation of eigenvalues from their mean across multiple simulations. The red dashed line represents the theoretical rate $O(\sqrt{\log p/n})$. As $n$ increases, the eigenvalues concentrate tightly around their expectation in agreement with Random Matrix Theory</figcaption></figure>

empirical eigenvalues $\lambda_i$ of $S_W^{-1}S_B$ satisfy the Baik-Ben Arous-Péché (BBP) phase transition:

$$ \lambda_i = \begin{cases} \theta_i + \frac{\gamma\sigma^2}{\theta_i}, & \text{if } \theta_i > \sqrt{\gamma}, \\ (1 + \sqrt{\gamma})^2 + O(N^{-2/3}), & \text{otherwise.} \end{cases} $$

where $\theta_i$ represents the population eigenvalues of $S_B$. The rate of concentration of non-spiked eigenvalues follows the Tracy-Widom law:

$$ \mathbb{P}(\lambda_{\max} \le (1 + \sqrt{\gamma})^2 + tN^{-2/3}) \approx F_1(t), $$

where $F_1(t)$ is the Tracy-Widom distribution of order 1. This implies that the eigenvalues are tightly concentrated around their expected values, with deviations of order $N^{-2/3}$, ensuring that the Fisher discriminant ratio remains stable in sufficiently large dimensions.

**Advanced Perturbation Bounds:** To quantify sensitivity to noise, we analyze the perturbation bounds of the Fisher matrix $F = S_W^{-1}S_B$. Using Davis-Kahan theorem, the perturbation in
eigenvectors satisfies:

$$ \sin \Theta(\mathbf{w}_i, \mathbf{w}_i^*) \leq \frac{\|\mathbf{E}\|}{\lambda_i - \lambda_{\text{bulk}}} $$

where $\mathbf{w}_i$ is an eigenvector of the unperturbed Fisher matrix, $\mathbf{w}_i^*$ is the perturbed version, and $\lambda_{\text{bulk}}$ is the largest non-outlier eigenvalue. This bound quantifies how robust the discriminant directions are to sampling noise. For small perturbations $\mathbf{E}$ in $S_B$, the first-order correction to the eigenvalues

<figure><img src="image_193.png" /><figcaption>Figure 16.25: Weyl bounds on the eigenvalues of the Fisher matrix $F = S_W^{-1}S_B$ in Linear Discriminant Analysis (LDA). Blue points denote the original eigenvalues, red points denote the perturbed eigenvalues under a random perturbation of $S_B$, and gray dashed intervals show the Weyl bounds [$\lambda_i(F) - \|E\|_2, \lambda_i(F) + \|E\|_2$]. The perturbed eigenvalues lie within the predicted Weyl intervals</figcaption></figure>

is given by the Weyl bound:

$$ |\lambda_i^* - \lambda_i| \leq \|\mathbf{E}\|. $$

For moderate perturbations, the Ky Fan norm bound applies:

$$ \sum_{i=1}^{C-1} |\lambda_i^* - \lambda_i| \leq \|\mathbf{E}\|_{\text{tr}}. $$

For large perturbations, eigenvalue shifts satisfy the Bauer-Fike theorem:

$$ \frac{|\lambda_i^* - \lambda_i|}{|\lambda_i|} \leq \kappa(S_W) \|\mathbf{E}\|_{\infty}. $$

where $\kappa(S_W)$ is the condition number of $S_W$. These bounds ensure LDA's stability under sampling noise, provided $\kappa(S_W)$ is well-conditioned.
<figure><img src="image_194.png" /><figcaption>Figure 16.26: Ky Fan norm bound of eigenvalues of the Linear Discriminant Analysis (LDA). The curve shows the cumulative sum of the top-k eigenvalues, with the dashed red line indicating the trace bound (sum of all eigenvalues)</figcaption></figure>

**Non-Asymptotic Generalization Analysis:** For finite samples, we analyze the generalization error of LDA using PAC-Bayesian bounds. Define the expected classification risk as:

$$R(h) = \mathbb{P}(h(\mathbf{x}) \neq y),$$

where $h(\mathbf{x})$ is the LDA decision rule. The empirical risk $\hat{R}(h)$ based on training data satisfies:

$$\mathbb{P}(|R(h) - \hat{R}(h)| \geq \epsilon) \leq 2 \exp \left( -\frac{N\epsilon^2}{2C} \right).$$

Thus, the generalization error is controlled by:

$$R(h) \leq \hat{R}(h) + O\left(\sqrt{\frac{C}{N}}\right).$$

For high-dimensional LDA, the error is governed by the effective rank of $S_W$, defined as:

$$r_{\text{eff}} = \frac{\text{Tr}(S_W)}{\lambda_{\max}(S_W)}.$$

If $r_{\text{eff}} \ll d$, then the generalization error degrades. Using local Rademacher complexities, we obtain the bound:

$$R(h) \leq \hat{R}(h) + O\left(\frac{\sqrt{d \log C}}{\sqrt{N}}\right),$$

showing that LDA's performance deteriorates if $d \gg N$, unless regularization is applied.
<figure><img src="image_195.png" /><figcaption>Figure 16.27: Bauer-Fike theorem applied to eigenvalues of the Fisher matrix $F = S_W^{-1}S_B$ in LDA. Blue points denote the original eigenvalues, red crosses denote perturbed eigenvalues, and gray shaded regions indicate the Bauer-Fike bounds [$\lambda_i - \kappa(V)\|E\|_2, \lambda_i + \kappa(V)\|E\|_2$]. All perturbed eigenvalues lie within the predicted intervals</figcaption></figure>

### 16.2.9.5 Python Code to generate Figure 16.18 illustrating the Eigenvalue Perturbation Analysis of Linear Discriminant Analysis (LDA)

The Python code below produces the Figure 16.18 illustrating the Eigenvalue Perturbation Analysis of Linear Discriminant Analysis (LDA).

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn datasets import makeClassification

# Generate synthetic classification dataset
X, y = makeClassification(n Samples=200, n features=5, n informative=3,
                           n redundant=0, n classes=3, random state=42)

# Compute scatter matrices for LDA
classes = np.unique(y)
n_features = X.shape[1]

overall_mean = np.mean(X, axis=0)
S_W = np(zeros((n_features, n_features))) # Within-class scatter
S_B = np(zeros((n_features, n_features))) # Between-class scatter

for c in classes:
    X_c = X[y == c]
```
<figure><img src="image_196.png" /><figcaption>Figure 16.28: Generalization error of Linear Discriminant Analysis (LDA) using PAC-Bayesian bounds. The blue curve shows the empirical misclassification error, while the red dashed curve represents the PAC-Bayesian upper bound. As the sample size increases, the bound tightens, demonstrating improved generalization guarantees</figcaption></figure>

```python
mean_c = np.mean(X_c, axis=0)
S_W += (X_c - mean_c).T @ (X_c - mean_c)
n_c = X_c.shape[0]
mean_diff = (mean_c - overall_mean).reshape(-1, 1)
S_B += n_c * (mean_diff @ mean_diff.T)

# Solve generalized eigenvalue problem
eigvals, eigvecs = np.lINALG.eig(np.lINALG.PINV(S_W) @ S_B)
eigvals = np.real(eigvals)
eigvals = np.sort(eigvals)[::-1]

# Perturbation analysis
perturb_strengths = np.linspace(0, 1.0, 20)  # increasing noise
eigvals_perturbed = []  # define list before appending

for eps in perturb_strengths:
    noise = eps * np.randomEye(*S_B.shape)
    S_B_perturbed = S_B + noise @ noise.T * 0.01  # symmetric perturbation
    eigvals_pert, _ = np.lINALG.eig(np.lINALG.PINV(S_W) @ S_B_perturbed)
    eigvals_perturbed.append(np.sort(np.real(eigvals_pert))[::-1])

eigvals_perturbed = np.array(eigvals_perturbed)
```
```python
41
42 # Plot
43 plt.figure(figsize=(8, 6))
44 for i in range(min(3, eigvals_perturbed.shape[1])): # plot top 3
45     eigenvalues
46     plt.plot(perturb_strengths, eigvals_perturbed[:, i], '-o', label='f' +
47             'Perturbed Eigenvalue {i+1}')
47
48 plt axhline(y=eigvals[0], color='r', linewidth='--', label='Original Top
49 Eigenvalue')
50
51 plt axhline(y=eigvals[1], color='g', linewidth='--', label='Original 2nd
52 Eigenvalue')
53
54 plt.tight_layout()
55
56 # Plot spectral decomposition
57 X, y = makeClassification(nSamples=150, nFeatures=5, nInformative=3,
58                           nRedundant=0, nClasses=3,
59                           nClustersPerClass=1,
60                           randomState=42)
61
62 # Fit LDA
63 X lavd = LDA(nComponents=2)
64 X lavd.fitTransform(X, y)
65
66 # Eigen decomposition (spectral decomposition)
67 # lavd.scalings_ are eigenvectors (discriminant directions)
68 # lavd.explainedVarianceRatio_ are normalized eigenvalues
69 eigvectors = lavd.scalings_
70 eigvals = lavd.explainedVarianceRatio_
71
72 # Plot top 2 features
73 X top1, X top2 = X[:2], X[2]
74
75 # Left: original data projected onto first two features
76 plt Subplot(1, 2, 1)
77 plt scatter(X[:, 0], X[:, 1], c=y, cmap=plate.cm.Set1, edgecolor='k', s
78       =50)
```

### 16.2.9.6 Python Code to generate Figure 16.19 illustrating the Spectral decomposition of the discriminant space in Linear Discriminant Analysis (LDA)

The Python code below produces the Figure 16.19 illustrating the Spectral decomposition of the discriminant space in Linear Discriminant Analysis (LDA).

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3 fromsklearn datasets import makeclassification
4 fromsklearn discriminant_analysis import LinearDiscriminantAnalysis as
5 LDA
6
7 # Generate synthetic dataset (3 classes, 5 features)
8 X, y = makeclassification(nSamples=150, n_features=5, n_informative=3,
9                           n_redundant=0, n_classes=3,
10                          n clusters per class=1,
11                          random_state=42)
12
13 # Fit LDA
14 ldalda = LDA(n_components=2)
15 X ldalda = ldalda.fit_transform(X, y)
16
17 # Eigen decomposition (spectral decomposition)
18 # ldalda.scalings_ are eigenvectors (discriminant directions)
19 # ldalda.explainedVarianceRatio_ are normalized eigenvalues
20 eigvectors = ldalda.scalings_
21 eigvals = ldalda.explainedVarianceRatio_
22
23 # Plot spectral decomposition
24 plt figure(figsize=(12, 5))
25
26 # Left: original data projected onto first two features
27 pltsubplot(1, 2, 1)
28 plt scatter(X[:, 0], X[:, 1], c=y, cmap=plate.cm.Set1, edgecolor='k', s
29           =50)
```
```python
27 plt.title("Original Data (first two features)")
28 plt.xlabel("Feature 1")
29 plt.ylabel("Feature 2")
30
31 # Right: LDA discriminant space (projection on eigenvectors)
32 pltsubplot(1, 2, 2)
33 plt.scatter(X_lda[:, 0], X_lda[:, 1], c=y, cmap= plt.cm. Set1, edgecolor='k', s=50)
34 plt.title("Spectral Decomposition: LDA Discriminant Space")
35 plt.xlabel(f"LD1 (eigval ratio={eigvals[0]:.2f})")
36 plt.ylabel(f"LD2 (eigval ratio={eigvals[1]:.2f})")
37
38 plt.tight_layout()
39 plt.show()
```

### 16.2.9.7 Python Code to generate Figure 16.20 illustrating the Asymptotic convergence of the decision boundary in Linear Discriminant Analysis (LDA)

The Python code below produces the Figure 16.20 illustrating the Asymptotic convergence of the decision boundary in Linear Discriminant Analysis (LDA).

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3 from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
4
5 # True parameters for two Gaussian classes
6 np.random.seed(42)
7 mean1 = [0, 0]
8 mean2 = [2, 2]
9 cov = [[1, 0.5], [0.5, 1]] # shared covariance
10
11 # Generate test grid
12 xx, yy = np.meshgrid(np.linspace(-4, 6, 20), np.linspace(-4, 6, 20))
13 grid = np.c_[xx.ravel(), yy.ravel()]
14
15 # True Bayes decision boundary (linear, since cov is equal)
16 w_true = np.linalg.inv(cov) @ (np.array(mean2) - np.array(mean1))
17 b_true = -0.5 * (np.array(mean2) + np.array(mean1)) @ w_true
18
19 # Sample sizes to illustrate asymptotic convergence
20 sample_sizes = [20, 50, 200, 1000]
21
22 plt.figure(figsize=(12, 10))
23
24 for i, n in enumerate(sampleSizes, 1):
25     # Generate samples
26     X1 = np.random.multivariate_normal(mean1, cov, n)
27     X2 = np.random.multivariate_normal(mean2, cov, n)
28     X = np.vstack([X1, X2])
29     y = np.array([0]*n + [1]*n)
30
31     # Fit LDA
32     lda = LDA()
33     lda.fit(X, y)
```
```python
34 Z =lda.decision_function(grid).reshape(xx.shape)
35
36 # Plot
37 pltsubplot(2, 2, i)
38 plt.contour(xx, yy, Z, levels=[0], linewidths=2, colors='b',
39             linestyles='--')
39 plt.scatter(X1[:,0], X1[:,1], c='red', s=20, label='Class 0',
40             alpha=0.6)
41
42 # True boundary
43 true_boundary = -(w_true[0]/w_true[1])*xx - b_true/w_true[1]
44 plt plot(xx[0, :], true_boundary, 'k-', linewidth=2)
45
46 plt.title(f"LDA Decision Boundary (n={n} per class)")
47 plt.xlim(-4, 6)
48 plt.ylim(-4, 6)
49 plt.legend()
50
51 plt.suptitle("Asymptotic Convergence of LDA Decision Boundary", fontsize
51 =16)
52 plt.tight_layout rect=[0, 0.03, 1, 0.95])
53 plt.show()
```

### 16.2.9.8 Python Code to generate Figure 16.21 illustrating the Exact asymptotic bounds for the misclassification probability in Linear Discriminant Analysis (LDA)

The Python code below produces the Figure 16.21 illustrating the Exact asymptotic bounds for the misclassification probability in Linear Discriminant Analysis (LDA).

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3 from sklearn discriminant_analysis import LinearDiscriminantAnalysis
4 from scipy.stats import norm
5
6 # True parameters
7 np.random.seed(42)
8 mean1 = np.array([0, 0])
9 mean2 = np.array([2, 2])
10 cov = np.array([[1, 0.5], [0.5, 1]])
11
12 # Mahalanobis distance
13 delta2 = (mean1 - mean2).T @ np.lINALG.INV(COV) @ (mean1 - mean2)
14 delta = npSQRT delta2
15
16 # Bayes optimal error
17 P_star = norm.cdf(-0.5 * delta)
18
19 # Sample sizes
20 sampleSizes = np.logspace(1, 3, 10, type=int) # 10 to 1000
21 empirical_errors = []
22
23 # Monte Carlo approximation of empirical misclassification probability
```
```python
24 for n in sample_sizes:
25     errs = []
26     for _ in range(50): # 50 repetitions
27         X1 = np.random.multivariate_normal(mean1, cov, n)
28         X2 = np.random.multivariate_normal(mean2, cov, n)
29         X = np.vstack([X1, X2])
30         y = np.array([0]*n + [1]*n)
31         lda = LinearDiscriminantAnalysis()
32         lda.fit(X, y)
33
34     # Test on large fresh dataset
35     X1_test = np.random.multivariate_normal(mean1, cov, 5000)
36     X2_test = np.random.multivariate_normal(mean2, cov, 5000)
37     X_test = np.vstack([X1_test, X2_test])
38     y_test = np.array([0]*5000 + [1]*5000)
39     y_pred = lda Predict(X_test)
40     errs.append(np.mean(y_pred != y_test))
41 empiCal_errors.append(np.mean(errs))
42
43 # Asymptotic bounds: P* and P* + C/sqrt(n)
44 C = 0.2 # scaling constant for upper bound (tunable for illustration)
45 upper_bound = [P_star + C/npsqrt(n) for n in sample_sizes]
46 lower_bound = [P_star for n in sample_sizes]
47
48 # Plot
49 plt.figure(figsize=(8,6))
50 plt.Plot(sample_sizes, empiCal_errors, 'o-', label="EmpiCal
Misclassification Probability", color='blue')
51 plt.Plot(sample_sizes, lower_bound, 'k--', label="Bayes Error (Lower
Bound)")
52 plt.Plot(sample_sizes, upper_bound, 'r--', label="Asymptotic Upper Bound
")
53
54 plt.xscale("log")
55 plt.xlabel("Sample size per class (log scale)")
56 plt.ylabel("Misclassification Probability")
57 plt.title("Exact Asymptotic Bounds for Misclassification Probability")
58 plt.legend()
59 plt.grid(True, which="both", ls="--")
60 plt.tight_layout()
61 plt.show()
```

### 16.2.9.9 Python Code to generate Figure 16.22 illustrating the Condition number of the Fisher matrix in Linear Discriminant Analysis (LDA)

The Python code below produces the Figure 16.22 illustrating the Condition number of the Fisher matrix in Linear Discriminant Analysis (LDA).

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3 from numpy import lINALg import cond
4
5 # Random seed for reproducibility
6 np.random.seed(42)
7
```
```python
# Define two Gaussian-distributed classes
mean1 = np.array([0, 0])
mean2 = np.array([2, 2])
cov = np.array([[1, 0.5], [0.5, 1]])

sample_sizes = np.logspace(1, 3, 10, type=int)  # from 10 to 1000
condition_numbers = []

for n in sample_sizes:
    # Sample data
    X1 = np.random.multivariate_normal(mean1, cov, n)
    X2 = np.random.multivariate_normal(mean2, cov, n)
    X = np.vstack([X1, X2])
    y = np.array([0]*n + [1]*n)

    # Compute class means
    m1 = X1.mean
    m2 = X2.mean self
    overall_mean = X.mean self

    # Between-class scatter S_B
    S_B = np outer m1 - overall_mean, m1 - overall_mean) * n \n
         + np outer m2 - overall_mean, m2 - overall_mean) * n

    # Within-class scatter S_W
    S_W = np cov(X1, rowvar=False) * (n-1) + np cov(X2, rowvar=False) *
         (n-1)

    # Fisher matrix
    Fisher = np linalg pi nv(S_W) @ S_B

    # Condition number (using 2-norm, ratio of largest to smallest
    singular value)
    kappa = cond(Fisher)
    condition_numbers.append kappa

# Plot condition number vs sample size
plt figure figsize=(8,6)
plt plot(sample_sizes, condition_numbers, 'o-', color='purple', label="")
    Condition number of Fisher matrix")

    plt.xscale("log")
    plt. yscale("log")
    plt. ylabel("Sample size per class (log scale)")
    plt. title("Condition Number of the Fisher Matrix in LDA")
    plt. legend()
    plt. grid(True, which="both", ls="--")
    plt.tight_layout()
    plt. show()
```
16.2.9.10 Python Code to generate Figure 16.23 illustrating the Random Matrix Theory (RMT) perspective on Linear Discriminant Analysis (LDA)

The Python code below produces the Figure 16.23 illustrating the Random Matrix Theory (RMT) perspective on Linear Discriminant Analysis (LDA).

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy import lINALg import npnv, eigvals

# Random seed
np.random.seed(42)

# Parameters
n = 500 # samples per class
p = 200 # dimension (high-dimensional case, p/n ~ 0.4)
mean1 = np(zeros(p))
mean2 = np(zeros(p); mean2[:5] = 1.0 # small separation in first 5 coordinates
cov = np.eye(p)

# Generate data
X1 = np.random.multivariate_normal(mean1, cov, n)
X2 = np.random.multivariate_normal(mean2, cov, n)
X = np.vstack([X1, X2])
y = np.array([0]*n + [1]*n)

# Compute scatter matrices
m1 = X1.mean
m2 = X2.mean su
overall_mean = X.mean su

S_B = np outer(m1 - overall_mean, m1 - overall_mean) * n \
     + np outer(m2 - overall_mean, m2 - overall_mean) * n
S_W = np cov(X1, rowvar=False) * (n-1) + np cov(X2, rowvar=False) * (n-1)

# Fisher matrix
Fisher = npnv(S_W) @ S_B

# Eigenvalues
eigvals_F = np real(eigvals Fisher))

# Marchenko Pastur distribution
q = p / (2*n) # ratio dimension / total samples
lambda_PLUS = (1 + np sqrt(q))**2
lambdaminus = (1 - np sqrt(q))**2
lambdas = np linspace(max(0, lambda_minus - 0.1), lambda_PLUS + 0.5,
                       200)
mp_density = np sqrt((lambda_PLUS - lambdas) * (lambdas - lambdaminus)))
    / (2 * np * q * lambdas)
mp_density = np nan_to_num mp_density

# Plot eigenvalue spectrum
plt figure figsize=(8,6)
plt hist(eigvals_F, bins=50, density=True, alpha=0.6, color='blue',
```
```python
label="Emπrical spectrum of Fisher matrix")
plt.plot(lambdas, mp_density, 'r-', lw=2, label="Marchenko Pastur
prediction")

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

834

835

836

837

838

839

840

841

842

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

860

861

862

863

864

865

866

867

868

869

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

892

893

894

895

896

897

898

899

900

901

902

903

904

905

906

907

908

909

910

911

912

913

914

915

916

917

918

919

920

921

922

923

924

925

926

927

928

929

930

931

932

933

934

935

936

937

938

939

940

941

942

943

944

945

946

947

948

949

950

951

952

953

954

955

956

957

958

959

960

961

962

963

964

965

966

967

968

969

970

971

972

973

974

975

976

977

978

979

980

981

982

983

984

985

986

987

988

989

990

991

992

993

994

995

996

997

998

999

100
```

### 16.2.9.11 Python Code to generate Figure 16.24 illustrating the Exact rate of eigenvalue concentration in Linear Discriminant Analysis (LDA)

The Python code below produces the Figure 16.24 illustrating the Exact rate of eigenvalue concentration in Linear Discriminant Analysis (LDA).

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy import lINALg import eigvals, pi

# Parameters
np.random.seed(42)
p = 50 # dimension
n_list = np.logspace(2, 3.3, 8, type=int) # sample sizes: 100 -> 2000
trials = 30 # Monte Carlo trials

# True means and covariance
mean1 = np.zeros(p)
mean2 = np zeros(p); mean2[:5] = 1.0 # signal in first 5 features
cov = np .eye(p)

# Store deviations
max_deviations = []

for n in n_list:
    eigvals_trials = []
    for _ in range(trials):
        # Simulate data
        X1 = np.random.multivariate_normal(mean1, cov, n)
        X2 = np.random.multivariate_normal(mean2, cov, n)
        X = np .vstack([X1, X2])
        overall_mean = X .mean

        # Between-class scatter
        m1 = X1 .mean axis=0)
        m2 = X2 .mean axis=0)
        S_B = np .outer(m1 - overall_mean, m1 - overall_mean) * n \ + np .outer(m2 - overall_mean, m2 - overall_mean) * n

        # Within-class scatter
        S_W = np .cov(X1, rowvar=False) * (n-1) + np .cov(X2, rowvar=False) * (n-1)

        # Fisher matrix and eigenvalues
```
2. Red curve: eigenvalues after perturbation.

3. Gray dashed vertical intervals: Weyl bounds, showing the range each eigenvalue could move under perturbation.

4. The red points always fall within these intervals, verifying Weyl's theorem.

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy import lINALg import eigvals, npv, norm

# Random seed
np.random.seed(42)

# Parameters
n, p = 200, 20
mean1 = np(zeros(p))
mean2 = np(zeros(p)); mean2[:5] = 1.0
cov = np.eye(p)

# Generate data
X1 = np.random.multivariate_normal(mean1, cov, n)
X2 = np.random.multivariate_normal(mean2, cov, n)
X = np.vstack([X1, X2])
overall_mean = X.mean

# Compute scatter matrices
m1 = X1.mean(axis=0)
m2 = X2.mean axis=0)

S_B = np outer(m1 - overall_mean, m1 - overall_mean) * n \ 
         + np outer(m2 - overall_mean, m2 - overall_mean) * n
S_W = np cov(X1, rowvar=False) * (n-1) + np cov(X2, rowvar=False) * (n-1)

# Fisher matrix and eigenvalues
F = npv(S_W) @ S_B
eigvals_F = np.sort(np real(eigvals_F))[::-1]

# Perturbation
E = np.random normal(0, 0.05, size=S_B.shape) # small Gaussian noise
S_B_pert = S_B + E
F_pert = npv(S_W) @ S_B_pert
eigvals_F_pert = np.sort(np real(eigvals_F_pert))[::-1]

# Weyl bound width
pert_norm = norm(E, 2)

# Plot eigenvalues and bounds
plt figure(figsize=(8,6))
idx = np arange(len(eigvals_F))

plt plot(idx, eigvals_F, 'bo-', label="Original eigenvalues")
plt plot(idx, eigvals_F_pert, 'ro--', label="Perturbed eigenvalues")
```
```python
# Weyl bounds (intervals)
for i in range(len(eigvals_F)):
    plt.vlines(i, eigvals_F[i] - pert_norm, eigvals_F[i] + pert_norm,
               colors='gray', linestyles='dashed')

plt.xlabel("Eigenvalue index (sorted)")
plt.ylabel("Eigenvalue")

plt.title("Weyl Bound on Eigenvalues of LDA Fisher Matrix")

plt.legend()

plt.grid(True, ls="--")

plt.tight_layout()

plt.show()
```

### 16.2.9.13 Python Code to generate Figure 16.26 illustrating the Ky Fan norm bound of the eigenvalues of the Linear Discriminant Analysis (LDA)

The Python code below produces the Figure 16.26 illustrating the Ky Fan norm bound of the eigenvalues of the Linear Discriminant Analysis (LDA). The Ky Fan norm (also called the Ky Fan $k$-norm) of a Hermitian matrix is defined as the sum of the $k$ largest eigenvalues. In the context of Linear Discriminant Analysis (LDA), we can compute the between-class scatter matrix ($S_B$), within-class scatter matrix ($S_W$), and analyze the generalized eigenvalues of the matrix ($S_W^{-1}S_B$). Then, we can compute the Ky Fan $k$-norm bounds and plot them.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn datasets import makeClassification

# Generate synthetic data
X, y = makeClassification(n Samples=300, n features=5, n informative=3,
                           n redundant=0, n classes=3, random state=42)

# Compute class means and overall mean
classes = np.unique(y)
overall_mean = np.mean(X, axis=0)

# Initialize scatter matrices
S_W = np(zeros((X.shape[1], X.shape[1]))
S_B = np(zeros((X.shape[1], X.shape[1]))

for c in classes:
    X_c = X[y == c]
    mean_c = np.mean(X_c, axis=0)
    # Within-class scatter
    S_W += (X_c - mean_c).T @ (X_c - mean_c)
    # Between-class scatter
    n_c = X_c.shape[0]
    mean_diff = (mean_c - overall_mean).reshape(-1, 1)
    S_B += n_c * (mean_diff @ mean_diff.T)

# Solve generalized eigenvalue problem
eigvals, eigvecs = np.lINALG.eig(np.lINALG.PNV(S_W) @ S_B)
eigvals = np.real(np.sort(eigvals)[::-1])  # sort descending

# Compute Ky Fan norms for k=1,...,d
ky_fan_norms = [np.sum(eigvals[:k]) for k in range(1, len(eigvals)+1)]
```
```python
33
34 # Plot
35 plt.figure(figsize=(8,6))
36 plt.plot(range(1, len(eigvals)+1), ky_fan_norms, 'o-', label="Ky Fan k-
37 norm")
38 plt axhline(np.sum(eigvals), color='red', linewidth='--', label="Trace
39 bound (sum of all eigenvalues)")
40 plt.ylabel("k (number of top eigenvalues)")
41 plt.title("Ky Fan Norm Bound of Eigenvalues in LDA")
42 plt.grid(True)
43 plt.show()
```

### 16.2.9.14 Python Code to generate Figure 16.27 illustrating the Bauer-Fike theorem applied to eigenvalues of the Fisher matrix in LDA

The Python code below produces the Figure 16.27 illustrating the Bauer-Fike theorem applied to eigenvalues of the Fisher matrix $F = S_W^{-1}S_B$ in LDA. If we perturb $F$ slightly due to finite samples or noise:

$$F' = F + E$$

The Bauer-Fike theorem states for a diagonalizable matrix $F$ (with eigenvectors matrix $V$):

$$\forall \lambda' \in \text{eig}(F'), \exists \lambda \in \text{eig}(F) \text{ s.t. } |\lambda' - \lambda| \le \kappa(V)|E|_2$$

where $\kappa(V) = |V|_2|V^{-1}|_2$ is the condition number of the eigenvector matrix, and $|E|_2$ is the spectral norm of the perturbation. This tells us how eigenvalues shift under perturbations.

In the plot

1. Blue points: original eigenvalues of $F = S_W^{-1}S_B$

2. Red crosses: eigenvalues after perturbation $F + E$

3. Gray shaded interval: Bauer-Fike bound $\lambda_i \pm \kappa(V)|E|_2$

All perturbed eigenvalues lie within the predicted bounds, verifying Bauer-Fike.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3 from numpy import eig, npv, norm
4
5 np.random.seed(42)
6
7 # Parameters
8 n, p = 200, 10
9 mean1 = np.zeros(p)
10 mean2 = np zeros(p); mean2[:3] = 1.0
11 cov = np. eye(p)
12
13 # Generate synthetic data
14 X1 = np.random.multivariate_normal(mean1, cov, n)
15 X2 = np.random.multivariate_normal(mean2, cov, n)
16 X = np.vstack([X1, X2])
17 overall_mean = X. mean(axis=0)
```
```python
18
19 # Scatter matrices
20 m1 = X1.mean( axis=0 )
21 m2 = X2.mean( axis=0 )
22
23 S_B = np outer( m1 - overall_mean, m1 - overall_mean ) * n + \
24               np outer( m2 - overall_mean, m2 - overall_mean ) * n
25
26 S_W = np cov( X1, rowvar=False ) * (n-1) + np cov( X2, rowvar=False ) * (n
27 F = nv( S_W ) @ S_B
28
29 # Eigen decomposition
30 eigvals_F, V = eig( F )
31 eigvals_F = np real( eigvals_F ) # Correctly take real part of eigenvalues
32 V_inv = np lINALG. INV( V )
33 kappa_V = norm( V, 2 ) * norm( V_inv, 2 )
34
35 # Perturbation
36 E = np random normal( 0, 0.05, size=F.shape ) # small Gaussian noise
37 F_pert = F + E
38 eigvals_F_pert, _ = eig( F_pert ) # Correctly extract eigenvalues
39 eigvals_F_pert = np real( eigvals_F_pert ) # Correctly take real part of
40 eigenvalues
41
42 # Bauer-Fike bound
43 bound = kappa_V * norm( E, 2 )
44
45 # Plot
46 plt figure( figsize=(8,6) )
47 plt scatter( range( len( eigvals_F ) ), eigvals_F, color='blue', label='Original eigenvalues', s=50 )
48 plt scatter( range( len( eigvals_F_pert ) ), eigvals_F_pert, color='red',
49             marker='x', label='Perturbed eigenvalues', s=60 )
50
51 # Plot Bauer-Fike intervals
52 for i, val in enumerate( eigvals_F ):
53     plt hlines( val, i-0.2, i+0.2, colors='gray', linestyles='dashed')
54     plt fill-between( [i-0.2, i+0.2], val-bound, val+bound, color='gray',
55                       alpha=0.2 )
56
57 plt xlabel( "Eigenvalue index" )
58 plt ylabel( "Eigenvalue" )
59 plt title( "Bauer-Fike Theorem: Eigenvalue Shifts in LDA" )
60 plt legend( )
61 plt grid( True, ls="--" )
62 plt tight_layout( )
63 plt show( )
```
### 16.2.9.15 Python Code to generate Figure 16.28 illustrating the Generalization error of Linear Discriminant Analysis (LDA) using PAC-Bayesian bounds

The Python code below produces the Figure 16.28 illustrating the Generalization error of Linear Discriminant Analysis (LDA) using PAC-Bayesian bounds. In Linear Discriminant Analysis (LDA), the empirical error is the misclassification probability on training data:

$$ \hat{R}(h) = \frac{1}{n} \sum_{i=1}^{n} 1h(x_i) \neq y_i \quad (16.202) $$

PAC-Bayesian theory provides a probabilistic upper bound on the generalization error (R(h)):

$$ \forall \delta \in (0, 1), \quad R(h) \le \hat{R}(h) + \sqrt{\frac{\mathrm{KL}(Q|P) + \ln \frac{2\sqrt{n}}{\delta}}{2n}} \quad (16.203) $$

where:

1. $P$ = prior distribution over classifiers,

2. $Q$ = posterior over classifiers,

3. $KL(Q|P) = KL$ divergence,

4. $n$ = sample size.

We shall plot:

1. Empirical error ($\hat{R}(h)$)

2. PAC-Bayesian bound as a function of sample size.

In the plot

1. Blue curve = empirical misclassification error (test error)

2. Red dashed curve = PAC-Bayesian upper bound on generalization error

As sample size increases, the bound becomes tighter, and the empirical error usually decreases.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.model_selection import train_test_split

np.random.seed(42)

# Generate synthetic 2-class Gaussian data
n_features = 5
mean1 = np.zeros(n_features)
mean2 = np.zeros(n_features); mean2[:2] = 1.0
cov = npEye(n_features)
n_max = 500

def generate_data(n Samples):
    X1 = np.random.multivariate_normal(mean1, cov, n Samples)
    X2 = np.random.multivariate_normal(mean2, cov, n Samples)
    X = npstack([X1, X2])
    y = np.array([0]*n Samples + [1]*n samples)
```
established the utility of Naïve Bayes in probabilistic reasoning, even before modern machine learning frameworks popularized it.

Subsequent studies rigorously investigated the theoretical underpinnings of the classifier, particularly concerning its surprising effectiveness despite the strong feature independence assumption. Domingos and Pazzani (1997) [768] mathematically analyzed the performance of Naïve Bayes under zero-one loss and proved that the classifier could be optimal even when attributes exhibit strong dependencies. Their work provided theoretical justification for why Naïve Bayes performs well in many real-world settings. Hand and Yu (2001) [769] further argued that, despite its simplicity, Naïve Bayes remains highly competitive in classification tasks. They analyzed its robustness and derived theoretical explanations for its empirical success, showing that when attributes are conditionally independent given the class, Naïve Bayes achieves optimal classification performance. Rish (2001) [770] expanded upon this by conducting an extensive empirical study, identifying the specific conditions under which Naïve Bayes fails and where it excels. This research solidified the classifier's status as a baseline method that often performs remarkably well in practical applications.

Further refinements and comparative studies explored the limits of the independence assumption and its effect on classification accuracy. Ng and Jordan (2002) [771] rigorously compared Naïve Bayes with logistic regression, demonstrating that while Naïve Bayes converges more rapidly with fewer training samples, logistic regression achieves superior asymptotic performance. This highlighted the fundamental trade-off between generative and discriminative classifiers. Webb, Boughton, and Wang (2005) [772] proposed the Averaged One-Dependence Estimators (AODE), which relaxes the independence assumption by allowing each attribute to depend on one other attribute. Their study provided a pathway for enhancing Naïve Bayes through partial dependency modeling, improving classification accuracy while retaining computational efficiency. Similarly, Boulle (2007) [773] introduced a compression-based Bayesian regularization technique that selects the most probable subset of features while adhering to the Naïve Bayes assumption, mitigating overfitting and improving generalization performance. Larsen and Aone (1999) [774] also extended the classifier's capabilities by introducing a refined document clustering framework, reducing bias in class probability estimation.

Collectively, these contributions illustrate the extensive theoretical and empirical development of the Naïve Bayes classifier, from its foundational probabilistic framework to sophisticated refinements addressing its assumptions. The classifier's resilience, despite its simplifying assumptions, has been well-documented, and various studies have sought to either explain or mitigate its limitations. These advancements have cemented Naïve Bayes as a cornerstone of statistical learning, particularly in text classification, medical diagnosis, spam filtering, and other probabilistic reasoning applications. While modern machine learning methods have evolved beyond Naïve Bayes in many domains, its efficiency, interpretability, and theoretical elegance ensure its continued relevance in contemporary research and practice.

<table><caption>Table 16.27: Summary of Contributions to Naïve Bayes Classifier</caption><thead><tr><td>Reference</td><td>Key Contribution</td></tr></thead><tbody><tr><td>Maron (1961)</td><td>Introduced probabilistic indexing for document retrieval, laying the foundation for Bayesian-based text classification methods.</td></tr><tr><td>Minsky (1961)</td><td>Discussed the use of Bayesian probability in early artificial intelligence, setting the stage for Naïve Bayes applications in pattern recognition.</td></tr></tbody></table>

*Continued on next page*
*Continued from previous page*

<table><thead><tr><td>Reference</td><td>Key Contribution</td></tr></thead><tbody><tr><td>Mosteller and Wallace (1963)</td><td>Applied Bayesian inference for authorship attribution in the Federalist Papers, demonstrating Naïve Bayes' utility in text classification.</td></tr><tr><td>Domingos and Paz-zani (1997)</td><td>Theoretically proved that Naïve Bayes performs optimally under zero-one loss, even when attribute independence is violated.</td></tr><tr><td>Hand and Yu (2001)</td><td>Argued that Naïve Bayes is not "stupid" despite its simplifying assumptions, providing theoretical insights into its effectiveness.</td></tr><tr><td>Rish (2001)</td><td>Conducted an empirical analysis of Naïve Bayes, highlighting its strengths and weaknesses across different datasets.</td></tr><tr><td>Ng and Jordan (2002)</td><td>Compared Naïve Bayes with logistic regression, showing that Naïve Bayes converges faster with limited data but is asymptotically sub-optimal.</td></tr><tr><td>Webb et al. (2005)</td><td>Proposed Averaged One-Dependence Estimators (AODE) to relax the independence assumption, improving classification accuracy.</td></tr><tr><td>Boulle (2007)</td><td>Introduced a compression-based Bayesian regularization technique for feature selection, reducing overfitting in Naïve Bayes models.</td></tr><tr><td>Larsen and Aone (1999)</td><td>Developed an enhanced document clustering approach that reduces bias in probability estimation, extending Naïve Bayes for text mining.</td></tr></tbody></table>

### 16.2.10.2 Recent Literature Review of Naïve Bayes classifier

The *Naïve Bayes classifier* has been extensively applied across various domains, but its effectiveness is highly dependent on the nature of the dataset and the assumptions underlying the model. Recent research highlights both its strengths and weaknesses in areas such as text classification, geotechnical engineering, facial recognition, cybersecurity, and medical diagnosis. This paper rigorously examines the classifier's contributions across multiple fields, providing insights into when it is most effective and when it struggles.

Usman et al. (2025) [758] investigated the use of Naïve Bayes in retail sales prediction but found that the classifier performed poorly due to its inability to handle complex dependencies among product features. Similarly, Shannaq (2025) [775] explored its effectiveness in Arabic text classification, showing that while Naïve Bayes achieves **85-90% accuracy** on large datasets, its performance degrades significantly for smaller datasets. This limitation is attributed to the classifier's reliance on feature independence assumptions, which often do not hold in real-world scenarios. Goldstein et al. (2025) [776] further investigated the classifier's application in geotechnical characterization, concluding that Naïve Bayes is unreliable in high-uncertainty environments. This study highlights the model's inadequacy for engineering applications where probabilistic reasoning must be robust.

Beyond traditional applications, researchers have evaluated Naïve Bayes in transportation modeling, facial recognition, and fault detection. Ntamwiza and Bwire (2025) [777] compared it to ensemble models for predicting biking preferences, concluding that Naïve Bayes underperformed due to its inability to capture complex, non-linear relationships. In the domain of facial recognition, El Fadel (2025) [778] found that Naïve Bayes struggles with high-dimensional image data, making it significantly less effective than deep learning-based classifiers. Meanwhile, RaviKumar et al. (2025) [779] examined its performance in fault diagnosis for electric vehicles (EVs), demonstrating that while Naïve Bayes showed moderate success, it was inferior to deep learning methods due to its oversimplified assumptions.
Naïve Bayes has been widely employed in *text and sentiment analysis*, but with mixed results. Kavitha et al. (2025) [780] applied it to *fake review detection* and found that while computationally efficient, the model is **susceptible to misclassification** in the presence of sarcasm or linguistic subtleties. Nusantara (2025) [781] explored its use in *Twitter sentiment analysis for banking services*, concluding that it works well for **binary classification** but struggles with **multi-class sentiment analysis**. Ahmadi et al. (2025) [782] tested Naïve Bayes for *SMS spam detection in cybersecurity* and found that while it provides a **strong baseline**, it is **outperformed by deep learning models** that can capture semantic meaning.

The classifier's application in *medical and security domains* further highlights its strengths and weaknesses. Takaki et al. (2025) [783] tested it in an *AI-assisted respiratory classification system for chest X-rays*, concluding that while it is **computationally efficient**, its accuracy is significantly lower than deep learning methods such as EfficientNet and GoogleNet. Similarly, Abdullahi et al. (2025) [763] examined its effectiveness in *IoT attack detection*, showing that its performance was near **random chance** (AUC ≈ 0.50). These findings reinforce that while Naïve Bayes remains a viable choice for baseline comparisons and simple applications, **it is increasingly outclassed by advanced machine learning methods**.

While the Naïve Bayes classifier is a valuable tool due to its **computational efficiency**, **intertability**, and **ease of implementation**, its reliance on **independence assumptions** makes it unsuitable for complex, high-dimensional data. Across various applications, including *text classification*, *fault diagnosis*, *sentiment analysis*, and *medical imaging*, Naïve Bayes has been shown to be effective primarily in structured, low-dimensional datasets. However, as data complexity grows, more sophisticated models such as *deep learning and ensemble classifiers* significantly outperform it. Future research should focus on **hybrid approaches** that integrate Naïve Bayes with deep learning models to improve its robustness in complex settings.

<table><caption>Table 16.28: Summary of Recent Contributions on Naïve Bayes Classifier</caption><thead><tr><td>Study</td><td>Application Domain</td><td>Contribution</td><td>Key Finding</td></tr></thead><tbody><tr><td>Usman et al. (2025)</td><td>Retail Sales Prediction</td><td>Evaluated Naïve Bayes for identifying best-selling products based on historical data.</td><td>Poor accuracy due to failure in modeling complex dependencies.</td></tr><tr><td>Shannaq (2025)</td><td>Arabic Text Classification</td><td>Examined impact of dataset size on Naïve Bayes accuracy for Arabic texts.</td><td>Performs well (85-90%) on large datasets but struggles with small ones.</td></tr></tbody></table>

*Continued on next page*
<table><caption>Table 16.28 (Continued): Summary of Contributions</caption><thead><tr><td>Study</td><td>Application Domain</td><td>Contribution</td><td>Key Finding</td></tr></thead><tbody><tr><td>Goldstein et al. (2025)</td><td>Geotechnical Characterization</td><td>Used Naïve Bayes to classify geotechnical data from drilling operations.</td><td>Ineffective in high-uncertainty environments; poor generalization.</td></tr><tr><td>Ntamwiza &amp; Bwire (2025)</td><td>Transportation Modeling</td><td>Compared Naïve Bayes with ensemble models for predicting biking preferences.</td><td>Underperforms due to inability to handle complex feature interactions.</td></tr><tr><td>El Fadel (2025)</td><td>Facial Recognition</td><td>Systematic review of Naïve Bayes in facial recognition.</td><td>Performs poorly on high-dimensional image data; deep learning is superior.</td></tr><tr><td>RaviKumar et al. (2025)</td><td>EV Fault Diagnosis</td><td>Applied Naïve Bayes for detecting operational anomalies in electric vehicles.</td><td>Moderate success but inferior to deep learning-based fault detection.</td></tr><tr><td>Kavitha et al. (2025)</td><td>Fake Review Detection</td><td>Used Naïve Bayes for classifying fake reviews in e-commerce platforms.</td><td>Efficient but struggles with sarcasm and contextual cues.</td></tr><tr><td>Nusantara (2025)</td><td>Twitter Sentiment Analysis</td><td>Analyzed banking-related tweets using Naïve Bayes.</td><td>Good for binary classification but weak in multi-class sentiment detection.</td></tr><tr><td>Ahmadi et al. (2025)</td><td>Cybersecurity (Spam Detection)</td><td>Evaluated Naïve Bayes against deep learning for SMS spam classification.</td><td>Strong baseline but lacks contextual awareness, deep learning outperforms.</td></tr></tbody></table>

*Continued on next page*
<table><caption>Table 16.28 (Continued): Summary of Contributions</caption><thead><tr><th>Study</th><th>Application Domain</th><th>Contribution</th><th>Key Finding</th></tr></thead><tbody><tr><td>Takaki et al. (2025)</td><td>Medical Diagnosis</td><td>Applied Naïve Bayes for AI-assisted respiratory condition classification.</td><td>Computationally efficient but significantly less accurate than CNNs.</td></tr></tbody></table>

### 16.2.10.3 Mathematical Analysis of Naïve Bayes classifier

The Naïve Bayes classifier is a probabilistic model that applies Bayes' theorem under the assumption of conditional independence among features. Given a dataset consisting of $n$ feature vectors $\mathbf{x} = (x_1, x_2, \dots, x_n)$ and corresponding class labels $y \in \{C_1, C_2, \dots, C_k\}$, the objective is to determine the posterior probability $P(y | \mathbf{x})$ and assign the most probable class. Bayes' theorem states that

$$P(y | \mathbf{x}) = \frac{P(\mathbf{x} | y)P(y)}{P(\mathbf{x})} \qquad (16.204)$$

where $P(y)$ is the prior probability of class $y$, $P(\mathbf{x} | y)$ is the likelihood of observing $\mathbf{x}$ given class $y$, and $P(\mathbf{x})$ is the evidence, computed as

$$P(\mathbf{x}) = \sum_{y \in \{C_1, C_2, \dots, C_k\}} P(\mathbf{x} | y)P(y). \qquad (16.205)$$

Since the denominator $P(\mathbf{x})$ is independent of $y$, the classifier assigns $\mathbf{x}$ to the class maximizing the numerator, i.e.,

$$\hat{y} = \arg \max_{y \in \{C_1, C_2, \dots, C_k\}} P(\mathbf{x} | y)P(y). \qquad (16.206)$$

The Naïve Bayes assumption posits that the features $x_i$ are conditionally independent given $y$, such that

$$P(\mathbf{x} | y) = \prod_{i=1}^{n} P(x_i | y). \qquad (16.207)$$

Thus, the decision rule simplifies to

$$\hat{y} = \arg \max_{y \in \{C_1, C_2, \dots, C_k\}} P(y) \prod_{i=1}^{n} P(x_i | y). \qquad (16.208)$$

In practice, computing products of small probabilities may result in numerical underflow. To mitigate this, the logarithm of the probabilities is used, transforming the decision rule into

$$\hat{y} = \arg \max_{y \in \{C_1, C_2, \dots, C_k\}} \left( \log P(y) + \sum_{i=1}^{n} \log P(x_i | y) \right). \qquad (16.209)$$

The estimation of $P(y)$ and $P(x_i | y)$ depends on the type of data. For categorical data, the probabilities are estimated using the frequency counts:

$$P(y) = \frac{\text{count}(y)}{\text{total samples}}, \qquad (16.210)$$

$$P(x_i | y) = \frac{\text{count}(x_i, y)}{\text{count}(y)}. \qquad (16.211)$$
<figure><img src="image_197.png" /><figcaption>Figure 16.29: Decision boundary of a Gaussian Naïve Bayes classifier. The contour regions represent predicted classes, while the scatter points show the training data. Naïve Bayes assumes feature independence and Gaussian distributions for features</figcaption></figure>
For continuous data, a common approach is to model $P(x_i | y)$ as a Gaussian distribution with parameters estimated from the training data:

$$P(x_i | y) = \frac{1}{\sqrt{2\pi\sigma_{i,y}^2}} \exp\left(-\frac{(x_i - \mu_{i,y})^2}{2\sigma_{i,y}^2}\right), \qquad (16.212)$$

where $\mu_{i,y}$ and $\sigma_{i,y}^2$ are the mean and variance of feature $x_i$ for class $y$, computed as

$$\mu_{i,y} = \frac{1}{N_y} \sum_{j=1}^{N_y} x_{i,j}, \qquad (16.213)$$

$$\sigma_{i,y}^2 = \frac{1}{N_y} \sum_{j=1}^{N_y} (x_{i,j} - \mu_{i,y})^2. \qquad (16.214)$$

For robustness, Laplace smoothing is often applied in the categorical case, modifying the estimates as

$$P(x_i | y) = \frac{\text{count}(x_i, y) + \alpha}{\text{count}(y) + \alpha|V_i|} \qquad (16.215)$$

where $\alpha > 0$ is the smoothing parameter and $|V_i|$ is the number of possible values of $x_i$. The model is trained by computing these probabilities from the dataset and classifies new instances by evaluating the logarithmic sum of probabilities across all classes.

### 16.2.10.4 Python Code to generate Figure 16.29

The Python code below produces the Figure 16.29 illustrating the Naïve Bayes classifier. We shall visualize a Naïve Bayes classifier for a 2D dataset so it can be plotted easily. We'll use Gaussian Naïve Bayes for simplicity.

In the plot

1. The contour plot shows the predicted class regions (decision boundary).

2. Scatter points show the training data.

Gaussian Naïve Bayes assumes feature independence and models each feature per class as a Gaussian.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn datasets import makeClassification
from sklearn.naive_bayes import GaussianNB

# Generate synthetic 2D classification data
X, y = makeclassification(n Samples=200, n features=2, n informative=2,
                           n redundant=0, n clusters per class=1,
                           random state=42)

# Train Gaussian Naive Bayes
gnb = GaussianNB()
gnb.fit(X, y)

# Plot decision boundary
x_min, x_max = X[: ,0].min() - 1, X[: ,0].max() + 1
y_min, y_max = X[: ,1].min() - 1, X[: ,1].max() + 1
```
```python
17 xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
                       np.linspace(y_min, y_max, 200))
18
19 Z = gnb Predict(np.c_[xx.ravel(), yy.ravel()])
20 Z = Z.reshape(xx.shape)
21
22 plt.figure(figsize=(8,6))
23 plt.contourf(xx, yy, Z, alpha=0.3, cmap= plt.cm.Paired)
24 plt scatter(X[:,0], X[:,1], c=y, edgecolor='k', cmap= plt.cm.Paired)
25 plt xlabel("Feature 1")
26 plt ylabel("Feature 2")
27 plt title("Naive Bayes Classifier Decision Boundary")
28 plt grid(True, ls='--')
29 plt tight layout()
30 plt show()
```

## 16.2.11 Decision tree learning

### 16.2.11.1 Literature Review of Decision tree learning

Decision tree learning has undergone significant theoretical and methodological advancements, with seminal contributions that have shaped the field. Quinlan's ID3 algorithm (1986) [748] laid the foundation for decision tree construction by employing a greedy, top-down approach that recursively partitions the dataset based on attributes that maximize information gain. However, ID3 was limited to categorical attributes and lacked mechanisms for handling missing values. Quinlan's later work C4.5 (2014) [749] extended ID3 by incorporating strategies for dealing with continuous attributes through thresholding, handling missing values using probability-based assignment, and introducing pruning techniques to combat overfitting. Parallel to this, Breiman et al. (2017) [750] introduced a robust decision tree methodology that accommodated both classification and regression tasks. Unlike ID3 and C4.5, which used information gain as the splitting criterion, CART employed Gini impurity for classification and least-squares error minimization for regression. Moreover, it established a rigorous framework for cost-complexity pruning, which systematically balances model complexity and generalization.

Beyond core decision tree algorithms, significant research has focused on improving their performance through feature selection, ensemble learning, and data stream adaptation. Kohavi and John's Wrapper Method for Feature Selection (1997) [751] rigorously analyzed how feature selection impacts decision tree accuracy, demonstrating that an optimal feature subset can significantly improve predictive performance while reducing computational costs. Breiman's Bagging (1996) [752] further refined decision tree stability by introducing bootstrap aggregation, an ensemble method that constructs multiple trees on bootstrapped samples and aggregates their predictions to mitigate variance. Freund and Schapire's AdaBoost (1997) [753] revolutionized ensemble methods by proposing an adaptive boosting strategy, where successive weak decision trees are trained on reweighted datasets that emphasize misclassified instances. This approach led to strong generalization properties and became a cornerstone of ensemble-based decision tree methods.

Further methodological advancements have enabled decision trees to handle large-scale and streaming data. Breiman's Random Forests (2001) [754] combined the principles of bagging with randomized feature selection, ensuring diversity among individual trees and improving robustness against overfitting. Domingos and Hulten's Very Fast Decision Tree (VFDT) (2000) [755] algorithm addressed real-time data stream mining by using Hoeffding bounds to construct trees incrementally while maintaining computational efficiency. Additionally, Freund and Mason's Alternating Decision Tree (ADTree) (1999) [756] integrated decision trees with boosting techniques, producing
more compact and interpretable models that outperform traditional tree-based classifiers. Quinlan's Oblique Decision Trees (1993) [757] expanded decision tree expressiveness by introducing linear combination-based decision boundaries, overcoming the axis-aligned partitioning limitations of classical decision trees.

Collectively, these contributions have rigorously enhanced decision tree learning by addressing fundamental challenges related to feature selection, overfitting, scalability, and decision boundary flexibility. The evolution from simple, greedy tree induction algorithms to sophisticated ensemble and streaming methodologies has solidified decision trees as a powerful tool for machine learning. The theoretical underpinnings of these developments have not only improved predictive accuracy but also deepened the mathematical understanding of tree-based models, ensuring their continued relevance in modern data-driven applications.

<table><caption>Table 16.29: Summary of Contributions to Decision tree learning</caption><thead><tr><td>Reference</td><td>Key Contribution</td></tr></thead><tbody><tr><td>Quinlan (1986)</td><td>Introduced the ID3 algorithm, a fundamental top-down greedy approach that selects attributes based on maximum information gain, enabling efficient decision tree construction.</td></tr><tr><td>Quinlan (1993)</td><td>Developed the C4.5 algorithm, which extended ID3 by incorporating continuous attributes, handling missing values, and implementing pruning techniques to mitigate overfitting.</td></tr><tr><td>Breiman et al. (1984)</td><td>Proposed the CART methodology, which introduced binary decision trees, the Gini impurity measure, cost-complexity pruning, and regression trees, laying a rigorous statistical foundation.</td></tr><tr><td>Kohavi & John (1997)</td><td>Introduced the wrapper method for feature selection, demonstrating how optimal feature subset selection improves decision tree accuracy and computational efficiency.</td></tr><tr><td>Breiman (1996)</td><td>Developed the bagging (bootstrap aggregating) technique, which improves decision tree stability and reduces variance by averaging predictions from multiple bootstrapped models.</td></tr><tr><td>Freund & Schapire (1997)</td><td>Introduced the AdaBoost algorithm, an adaptive boosting approach that iteratively adjusts training sample weights to focus on misclassified instances, significantly enhancing decision tree performance.</td></tr><tr><td>Breiman (2001)</td><td>Created the Random Forests algorithm, which enhances decision trees using ensemble learning by constructing multiple randomized trees and aggregating their predictions.</td></tr><tr><td>Domingos & Hulten (2000)</td><td>Developed the Very Fast Decision Tree (VFDT) algorithm, designed for real-time data stream mining using Hoeffding bounds for incremental tree construction with fixed memory constraints.</td></tr><tr><td>Freund & Mason (1999)</td><td>Proposed the Alternating Decision Tree (ADTree), integrating decision trees with boosting to create interpretable models with superior generalization properties.</td></tr><tr><td>Quinlan (1993)</td><td>Developed Oblique Decision Trees, which introduce linear combination-based decision boundaries, improving classification performance by allowing non-axis-aligned splits.</td></tr></tbody></table>

### 16.2.11.2 Recent Literature Review of Decision tree learning

Decision tree learning is a fundamental method in machine learning, offering an interpretable and structured approach to classification and regression problems. Recent advancements have demon-
strated its effectiveness across diverse domains, including healthcare, retail forecasting, materials science, and agriculture. Usman et al. (2025) [758] leveraged decision trees to predict the best-selling products in retail, highlighting the advantage of ensemble methods like random forests over single decision trees in improving prediction accuracy. Similarly, Abbas et al. (2025) [759] explored the role of decision tree classifiers in diagnosing low back pain among students, demonstrating their capability to uncover intricate patterns in medical data that traditional statistical methods might overlook. In the commercial sector, Deng et al. (2025) [760] proposed an improved decision tree ensemble where successive trees focus on correcting errors made by previous ones, significantly enhancing retail demand forecasting accuracy.

In medical and clinical applications, decision tree learning has shown substantial promise. Eili et al. (2025) [761] developed a machine learning framework integrating decision trees with Markov models to predict patient treatment pathways for traumatic brain injuries (TBI). This application underscores the utility of decision trees in dynamic decision-making environments such as healthcare. Furthermore, Yin et al. (2025) [762] conducted a comparative analysis between decision trees and logistic regression for predicting cancer treatment response, revealing that tree-based models more effectively capture nonlinear relationships between biomarkers and treatment outcomes. Liu et al. (2025) [731] extended this approach to the dental field, using decision trees to analyze bond strength in lithium disilicate-reinforced ceramics, reinforcing the value of tree-based models in precision material engineering.

Beyond clinical settings, decision trees have been instrumental in environmental and agricultural research. Barghouthi et al. (2025) [732] fused decision trees with K-nearest neighbors and extreme gradient boosting to create a multi-channel predictive model for pressure injuries in hospitalized patients, demonstrating the robustness of tree-based models in handling high-dimensional data. In agronomy, Jewan (2025) [733] applied decision tree classifiers to predict crop yields in Bambara groundnut and grapevines, effectively processing remote sensing data to model environmental influences on agricultural productivity. Ghosh (2020) [692] examined Chebyshev's estimates of the prime counting function, highlighting their mathematical and historical importance in the development of analytic number theory. It explains how Chebyshev's bounds provided an early and crucial step toward understanding the distribution of prime numbers. The work emphasizes the foundational role these estimates played in motivating the later formulation of the Prime Number Theorem. Similarly, Abdullahi et al. (2025) [763] explored the use of decision trees in sound analysis for indoor localization, presenting a novel approach for integrating hierarchical classification with feature extraction, proving its adaptability beyond traditional use cases.

Lastly, Mokan et al. (2025) [764] illustrated the power of decision tree classifiers in medical imaging by developing a model capable of segmenting retinal vasculature into arteries and veins. This application demonstrates how decision trees can effectively handle pixel-wise classification tasks, enhancing diagnostic precision in ophthalmology. Collectively, these studies showcase the wide applicability and adaptability of decision tree learning, reinforcing its status as a versatile tool across disciplines. The method's ability to structure complex decision boundaries, interpret data hierarchically, and integrate seamlessly with ensemble learning approaches ensures its continued relevance in contemporary machine learning applications.
<table><caption>Table 16.30: Summary of Recent Contributions in Decision Tree Learning</caption><thead><tr><td>Authors (Year)</td><td>Title</td><td>Contribution</td></tr></thead><tbody><tr><td>Usman et al. (2025)</td><td>Identifying the Best-Selling Product using Machine Learning Algorithms</td><td>Explores decision trees for product sales forecasting, comparing single decision trees with ensemble models like random forests to enhance predictive accuracy.</td></tr><tr><td>Abbas et al. (2025)</td><td>Low Back Pain Among Health Sciences Undergraduates</td><td>Uses decision tree classifiers to predict low back pain patterns in students, demonstrating applications in healthcare analytics.</td></tr><tr><td>Deng et al. (2025)</td><td>Prediction of Retail Commodity Hot-Spots</td><td>Investigates boosting techniques where successive decision trees correct errors from previous ones, improving retail demand forecasting.</td></tr><tr><td>Eili et al. (2025)</td><td>Predicting Clinical Pathways of Traumatic Brain Injuries (TBI)</td><td>Integrates decision trees with Markov models for patient treatment pathway prediction, showcasing applications in dynamic decision-making.</td></tr><tr><td>Yin et al. (2025)</td><td>Gamma-Glutamyl Transferase Plus Carcinoembryonic Antigen Ratio Index</td><td>Compares decision trees with logistic regression for predicting cancer treatment responses, showing tree-based methods' effectiveness in handling nonlinear relationships.</td></tr><tr><td>Liu et al. (2025)</td><td>The Influence of Different Factors on the Bond Strength of Lithium Disilicate Glass-Ceramics to Resin</td><td>Applies decision trees in dental materials research, analyzing bond strength factors and highlighting feature importance.</td></tr><tr><td>Barghouthi et al. (2025)</td><td>A Fused Multi-Channel Prediction Model of Pressure Injury</td><td>Develops a hybrid model integrating decision trees with K-nearest neighbors and gradient boosting for improved predictive healthcare analytics.</td></tr><tr><td>Jewan (2025)</td><td>Remote Sensing Technology and Machine Learning for Crop Yield Prediction</td><td>Utilizes decision tree classifiers for agricultural forecasting, modeling the influence of environmental factors on crop productivity.</td></tr><tr><td>Akbal et al. (2025)</td><td>Accurate Indoor Home Location Classification through Sound Analysis</td><td>Implements decision trees in indoor localization using sound analysis, demonstrating its effectiveness in hierarchical feature classification.</td></tr></tbody></table>

*Continued on next page*