By integrating the domain-invariant representation $g_\theta(X)$ with causal inference models, the con-founding bias induced by distribution shift is mitigated, allowing for unbiased estimation of the causal effect in the target domain. Furthermore, the potential outcome framework in deep learning-based causal domain adaptation estimates the counterfactual outcomes $Y^0, Y^1$ using a shared feature encoder $f_\theta(X)$ and domain-invariant treatment-response functions $h_{\theta_0}(X)$ and $h_{\theta_1}(X)$:

$$ \hat{Y}^0 = h_{\theta_0}(f_\theta(X)), \quad \hat{Y}^1 = h_{\theta_1}(f_\theta(X)). \qquad (14.36) $$

The individualized treatment effect (ITE) is then estimated as:

$$ \hat{\tau}(X) = \hat{Y}^1 - \hat{Y}^0. \qquad (14.37) $$

To ensure domain generalization, recent advances incorporate representation learning techniques such as contrastive learning and information bottlenecks, where the feature extractor $g_\theta(X)$ is regularized to minimize the mutual information between domain-specific variations and the learned representations:

$$ \min_{\theta} \mathbb{E}_{X \sim P_S(X) \cup P_{\tau}(X)} \left[ \|g_{\theta}(X) - g_{\theta}(\tilde{X})\|_{2}^{2} \right], \qquad (14.38) $$

where $\tilde{X}$ represents an augmented version of $X$ through domain-specific transformations. The overall objective function in deep causal domain adaptation thus integrates causal loss, domain invariance constraints, and representation learning regularization:

$$ \min_{\theta, \theta_0, \theta_1} \sum_{i=1}^{n_S} w(X_i) \ell(h_{\theta_{A_i}}(f_\theta(X_i)), Y_i) + \lambda \sum_{j=1}^{n_\tau} \log(1 - D_\phi(g_\theta(X_j))) + \beta \mathbb{E}_{X, \tilde{X}} \left[ \|g_\theta(X) - g_\theta(\tilde{X})\|_2^2 \right], \qquad (14.39) $$

where $\lambda$ and $\beta$ are hyperparameters controlling the trade-off between causal effect estimation, domain adaptation, and representation alignment. By leveraging deep learning-based approaches for domain adaptation in causal inference, robust estimation of treatment effects is achieved even in the presence of significant distributional shifts, thereby enhancing the reliability of causal effect estimation in real-world applications.

## 14.4 Invariant Risk Minimization (IRM) in causal inference for deep neural networks

Invariant Risk Minimization (IRM) is a paradigm in causal inference and deep learning that aims to learn representations of data that lead to predictors that are invariant across different environments. Consider a dataset collected from multiple environments $\mathcal{E}$, each defined as a probability distribution $P_e(X, Y)$, where $X$ denotes the input features, and $Y$ represents the target variable. The goal of IRM is to learn a predictor $f: X \to Y$ that maintains its predictive performance across all possible environments, thereby capturing the causal relationship between $X$ and $Y$ rather than spurious correlations specific to particular environments.

Mathematically, let $\mathcal{H}$ denote the hypothesis space, which includes functions $h$ parameterized by a deep neural network. We define an invariant predictor $h$ that minimizes the worst-case risk over all observed environments:

$$ \min_{h \in \mathcal{H}} \max_{e \in \mathcal{E}} R_e(h), \qquad (14.40) $$

where the empirical risk in environment $e$ is given by:

$$ R_e(h) = \mathbb{E}_{(X,Y) \sim P_e} [\ell(h(X), Y)], \qquad (14.41) $$

with $\ell$ being a loss function, commonly the squared error loss for regression:

$$ \ell(h(X), Y) = (h(X) - Y)^2, \qquad (14.42) $$
or the cross-entropy loss for classification:

$$ \ell(h(X), Y) = -Y \log h(X) - (1 - Y) \log(1 - h(X)). \quad (14.43) $$

A crucial observation is that empirical risk minimization (ERM), which minimizes the sum of risks over all environments,

$$ \min_{h \in \mathcal{H}} \sum_{e \in \mathcal{E}} R_e(h), \quad (14.44) $$

often leads to predictors that exploit spurious correlations present in specific environments rather than capturing the invariant relationship. To address this, IRM introduces an additional constraint to enforce invariance in the predictor. Specifically, it seeks a representation function $\Phi: X \to Z$ such that a classifier $w$ trained on $Z$ remains optimal across all environments:

$$ \Phi^*, w^* = \arg \min_{\Phi, w} \sum_{e \in \mathcal{E}} R_e(w \circ \Phi), \quad (14.45) $$

subject to the invariance constraint:

$$ w \in \arg \min_{\tilde{w}} R_e(\tilde{w} \circ \Phi) \quad \forall e \in \mathcal{E}. \quad (14.46) $$

To relax this constraint into a differentiable objective, the IRM penalty is introduced:

$$ \mathcal{L}_{\text{IRM}} = \sum_{e \in \mathcal{E}} R_e(w \circ \Phi) + \lambda \| \nabla_w R_e(w \circ \Phi) |_{w=1} \|^2. \quad (14.47) $$

Here, the gradient term ensures that the classifier $w$ remains optimal across all environments by enforcing that its gradient with respect to $w$ vanishes. The hyperparameter $\lambda$ controls the strength of this regularization. Expanding the penalty term,

$$ \| \nabla_w R_e(w \circ \Phi) |_{w=1} \|^2 = \sum_{e \in \mathcal{E}} \| \mathbb{E}_{(X,Y) \sim P_e} [\nabla_w \ell(w\Phi(X), Y)] \|^2, \quad (14.48) $$

ensuring that the gradients of the risks are aligned across environments. A deeper connection to causality emerges when considering the Structural Equation Model (SEM):

$$ Y = f^*(X) + \epsilon, \quad X = g(Z, S), \quad (14.49) $$

where $Z$ are the causal features, $S$ are spurious features, and $\epsilon$ is an independent noise term. An ERM-trained deep network might learn $S$ because it improves empirical risk minimization in observed environments, while IRM discourages dependence on $S$ by enforcing that the predictor generalizes across all environments. A theoretical justification for IRM follows from an analysis of the optimal invariant predictor. Define the optimal classifier for a given representation $\Phi$:

$$ w_e^* = \arg \min_w R_e(w \circ \Phi). \quad (14.50) $$

For a truly invariant representation $\Phi^*$, the classifier should be identical across all environments:

$$ w_e^* = w^* \quad \forall e \in \mathcal{E}. \quad (14.51) $$

This leads to the condition:

$$ \nabla_w R_e(w \circ \Phi) = 0 \quad \forall e \in \mathcal{E}, \quad (14.52) $$

which is precisely the IRM penalty. Empirically, training with IRM often results in improved out-of-distribution generalization compared to standard deep learning approaches. Finally, an alternative interpretation of IRM emerges from considering a variational bound on the mutual information between the learned representation and the causal variables:

$$ I(Z; Y) = H(Y) - H(Y|Z). \quad (14.53) $$

IRM implicitly maximizes this information while minimizing the conditional entropy $H(Y|Z)$, leading to representations that capture causal relationships rather than spurious correlations.
## 14.5 Empirical Risk Minimization (ERM) in causal inference for deep neural networks

Empirical Risk Minimization (ERM) is a fundamental principle in statistical learning theory, providing a framework for optimizing predictive models, particularly within deep neural networks (DNNs). In the context of causal inference, ERM is adapted to account for the complexities introduced by confounding variables, treatment assignment mechanisms, and counterfactual estimation. Given a dataset $\mathcal{D} = \{(X_i, T_i, Y_i)\}_{i=1}^N$, where $X_i \in \mathbb{R}^d$ represents covariates, $T_i \in \{0, 1\}$ denotes the binary treatment assignment, and $Y_i \in \mathbb{R}$ is the observed outcome, the goal of causal inference is to estimate the Individual Treatment Effect (ITE), defined as

$$ \tau(X) = \mathbb{E}[Y | X, T=1] - \mathbb{E}[Y | X, T=0]. \quad (14.54) $$

The empirical risk in classical supervised learning is given by

$$ \hat{R}(h) = \frac{1}{N} \sum_{i=1}^{N} \ell(h(X_i), Y_i), \quad (14.55) $$

where $h: \mathbb{R}^d \to \mathbb{R}$ is the hypothesis function parameterized by a deep neural network and $\ell(\cdot, \cdot)$ is a loss function, such as the squared loss

$$ \ell(h(X_i), Y_i) = (h(X_i) - Y_i)^2. \quad (14.56) $$

However, in causal inference, the presence of treatment assignment $T$ necessitates a modified objective that accounts for the potential outcome framework. A common approach is to estimate the potential outcomes $Y(0)$ and $Y(1)$ using separate deep neural networks, $h_0(X; \theta_0)$ and $h_1(X; \theta_1)$, parameterized by $\theta_0, \theta_1$, leading to the empirical risk formulation

$$ \hat{R}(\theta_0, \theta_1) = \frac{1}{N} \sum_{i=1}^{N} \left( T_i \ell(h_1(X_i; \theta_1), Y_i) + (1-T_i) \ell(h_0(X_i; \theta_0), Y_i) \right). \quad (14.57) $$

This objective function is a weighted empirical risk based on the observed treatment assignment, ensuring that only the factual outcome is used in training. The challenge arises due to the imbalance in treatment assignments, leading to potential covariate shift between the treated and control groups. To mitigate this, inverse propensity weighting (IPW) is often incorporated, where the empirical risk is modified as

$$ \hat{R}_{\text{IPW}}(\theta_0, \theta_1) = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{T_i}{e(X_i)} \ell(h_1(X_i; \theta_1), Y_i) + \frac{1-T_i}{1-e(X_i)} \ell(h_0(X_i; \theta_0), Y_i) \right), \quad (14.58) $$

where $e(X) = P(T=1 | X)$ is the propensity score. This adjustment ensures that the empirical distribution of covariates in both treatment groups better approximates the underlying population distribution. However, direct inverse weighting can lead to high variance, motivating the use of balancing representations via domain adaptation methods such as representation learning with deep neural networks. Specifically, a feature representation function $\Phi: \mathbb{R}^d \to \mathbb{R}^m$ is learned such that the treated and control groups become indistinguishable in the transformed space. The empirical risk in this case is

$$ \hat{R}_{\Phi}(\theta_0, \theta_1, \phi) = \frac{1}{N} \sum_{i=1}^{N} \left( T_i \ell(h_1(\Phi(X_i; \phi); \theta_1), Y_i) + (1-T_i) \ell(h_0(\Phi(X_i; \phi); \theta_0), Y_i) \right), \quad (14.59) $$
with an additional discrepancy term penalizing differences between the distributions of $\Phi(X)$ under $T = 1$ and $T = 0$, such as the Maximum Mean Discrepancy (MMD),

$$ \mathcal{D}(\Phi) = \left\| \frac{1}{N_T} \sum_{i:T_i=1} k(\Phi(X_i), \cdot) - \frac{1}{N_C} \sum_{i:T_i=0} k(\Phi(X_i), \cdot) \right\|_{\mathcal{H}}, \qquad (14.60) $$

where $k(\cdot, \cdot)$ is a kernel function and $\mathcal{H}$ is the reproducing kernel Hilbert space. The final objective function then combines empirical risk minimization with regularization for balancing:

$$ \min_{\theta_0, \theta_1, \phi} \hat{R}_{\Phi}(\theta_0, \theta_1, \phi) + \lambda \mathcal{D}(\Phi). \qquad (14.61) $$

An alternative regularization strategy is adversarial balancing, where a discriminator $D(X)$ is trained to distinguish between treated and control units, and $\Phi(X)$ is trained adversarially to make this discrimination difficult. The objective for $\Phi$ and $D$ is formulated as a min-max problem:

$$ \min_{\theta_0, \theta_1, \phi} \max_{D} \hat{R}_{\Phi}(\theta_0, \theta_1, \phi) - \lambda \sum_{i=1}^{N} \log D(\Phi(X_i))^{T_i} (1 - D(\Phi(X_i)))^{1-T_i}. \qquad (14.62) $$

This ensures that the learned representation is treatment-agnostic, improving generalizability to counterfactual estimation. Additionally, doubly robust methods integrate both outcome modeling and inverse weighting by introducing a correction term based on residuals from a separate outcome model $m(X)$:

$$ \hat{R}_{\text{DR}}(\theta_0, \theta_1, \phi) = \frac{1}{N} \sum_{i=1}^{N} \left( \ell(h_{T_i}(\Phi(X_i; \phi); \theta_{T_i}), Y_i) + \frac{T_i - e(X_i)}{e(X_i)(1 - e(X_i))} (Y_i - m(X_i)) \right). \qquad (14.63) $$

This formulation ensures consistency even when either the outcome model or the propensity score model is misspecified. Ultimately, the optimization of these empirical risk formulations in deep neural networks involves gradient-based methods, with stochastic gradient descent (SGD) or Adam optimizing $\theta_0, \theta_1, \phi$, while the propensity score model $e(X)$ is learned via logistic regression or a separate neural network. The complexity of ERM in causal deep learning thus lies in balancing factual accuracy, counterfactual generalization, and representation learning to ensure robust ITE estimation.
# 15 Network Architecture Search (NAS) in Deep Neural Networks

Network Architecture Search (NAS) in Deep Neural Networks (DNNs) is an optimization problem that seeks to automatically design the best neural network architecture for a given task, typically involving a search space $\mathcal{A}$, a search strategy $\mathcal{S}$, and a performance evaluation metric $\mathcal{E}$. Mathematically, the NAS problem can be framed as finding the optimal architecture $\alpha^*$ that maximizes a performance function $\mathcal{F}$, which can be formally written as

$$ \alpha^* = \arg \max_{\alpha \in \mathcal{A}} \mathcal{F}(\alpha, \mathcal{D}) \qquad (15.1) $$

where $\mathcal{D}$ represents the dataset used for training and evaluation. The search space $\mathcal{A}$ defines the possible neural network architectures, which can be represented as a directed acyclic graph (DAG) $G = (V, E)$, where nodes $V$ correspond to layers (such as convolutional, fully connected, or recurrent layers), and edges $E$ represent connections between layers. The function $\mathcal{F}$ is typically defined in terms of accuracy, loss, computational efficiency, and resource constraints. The search process can be formulated as a reinforcement learning (RL) problem, where an agent explores the architecture space and updates its policy using rewards derived from model performance. Let $\pi_{\theta}(a_t | s_t)$ be a policy parameterized by $\theta$, which selects an action $a_t$ (modifying the architecture) given state $s_t$ (current architecture configuration). The expected reward function is then

$$ J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}} \left[ \sum_{t=0}^{T} \gamma^t r_t \right] \qquad (15.2) $$

where $\tau = (s_0, a_0, r_0, s_1, a_1, r_1, \dots)$ is a trajectory sampled from policy $\pi_{\theta}$, $\gamma$ is a discount factor, and $r_t$ is the reward (e.g., validation accuracy). Policy gradient methods such as REINFORCE update the policy via gradient ascent,

$$ \nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}} \left[ \sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t | s_t) R(\tau) \right] \qquad (15.3) $$

where $R(\tau)$ is the cumulative reward of a sampled trajectory. Alternatively, evolutionary algorithms model the search as a population-based optimization problem, evolving architectures through crossover and mutation operators. The fitness function in this context is given by

$$ f(\alpha) = \mathcal{E}(\alpha, \mathcal{D}) \qquad (15.4) $$

where $\mathcal{E}$ is an evaluation metric such as classification accuracy or mean squared error. Another approach is gradient-based NAS, where the architecture is parameterized by continuous variables $\alpha_i$ and optimized via gradient descent. If $w$ denotes network weights and $L(w, \alpha)$ is the loss function, then the architecture optimization problem is

$$ \min_{\alpha} L(w^*(\alpha), \alpha) \qquad (15.5) $$
where $w^*(\alpha)$ are optimal weights obtained by solving

$$w^*(\alpha) = \arg\min_{w} L(w, \alpha). \qquad (15.6)$$

The optimization follows a bilevel formulation, where the outer optimization updates $\alpha$ while the inner optimization solves for $w$. The gradient of $L(w^*(\alpha), \alpha)$ with respect to $\alpha$ is computed using implicit differentiation,

$$\nabla_{\alpha} L(w^*(\alpha), \alpha) = \nabla_{\alpha} L(w, \alpha) - \nabla_{w} L(w, \alpha) (\nabla_{w}^{2} L(w, \alpha))^{-1} \nabla_{\alpha} w. \qquad (15.7)$$

A continuous relaxation of discrete architectures is achieved using softmax over candidate operations $o_i$,

$$o_{\text{mixed}}(x) = \sum_{i} \frac{\exp(\alpha_i)}{\sum_{j} \exp(\alpha_j)} o_i(x). \qquad (15.8)$$

The optimal architecture is obtained by discretizing $\alpha_i$ after training. The computational complexity of NAS is often mitigated by weight sharing across architectures, leading to one-shot NAS methods where a supernet $\mathcal{N}$ encompasses all possible subnets $\alpha$, and training occurs over a shared weight space $W$,

$$\min_{W} \mathbb{E}_{\alpha \sim \mathcal{A}} L(W, \alpha). \qquad (15.9)$$

After training, the best subnet is selected by evaluating candidate architectures sampled from $\mathcal{A}$. The performance of a neural architecture is influenced by hyperparameters $\lambda$, which affect layer depth, width, and activation functions. A common formulation includes multi-objective optimization,

$$\max_{\alpha} \mathbb{E}[\mathcal{F}(\alpha)] - \lambda \cdot C(\alpha) \qquad (15.10)$$

where $C(\alpha)$ represents computational cost (e.g., FLOPs or latency). The search can be constrained using Lagrange multipliers,

$$\mathcal{L}(\alpha, \lambda) = \mathcal{F}(\alpha) - \lambda C(\alpha), \qquad (15.11)$$

with the optimal trade-off achieved by solving

$$\frac{\partial \mathcal{L}}{\partial \alpha} = 0, \quad \frac{\partial \mathcal{L}}{\partial \lambda} = 0. \qquad (15.12)$$

By iteratively refining $\hat{\mathcal{F}}$, the search converges to architectures with high performance and low computational cost. The theoretical underpinnings of NAS rely on neural tangent kernels (NTKs) to approximate training dynamics, where the NTK matrix $\Theta$ evolves as

$$\frac{d\Theta}{dt} = -\eta\Theta^2, \qquad (15.13)$$

where $\eta$ is the learning rate. A stable architecture satisfies the condition

$$\lambda_{\max}(\Theta) \ll 1. \qquad (15.14)$$

Thus, NAS systematically optimizes architectures through structured exploration of the search space, reinforcement learning, evolutionary algorithms, differentiable search, and surrogate modeling, ensuring optimality under computational constraints.

# 15.1 Evolutionary Algorithms in Network Architecture Search

Evolutionary Algorithms (EAs) have emerged as a powerful approach for automating Network Architecture Search (NAS) in Deep Neural Networks (DNNs), leveraging principles of natural
evolution to optimize network topologies. The core idea is to represent a neural network architecture as a set of hyperparameters and connection structures, encoding them into a genetic representation that undergoes evolutionary processes such as selection, mutation, and crossover to explore the vast search space efficiently. Given a population of architectures $P = \{N_1, N_2, \dots, N_N\}$, where each $N_i$ represents a candidate neural network, we define the fitness function $f: N \to \mathbb{R}$, which evaluates the performance of a network based on metrics such as validation accuracy, computational efficiency, or robustness to perturbations. Each neural network architecture $N_i$ can be described by a directed

<figure><img src="image_163.png" /><figcaption>Figure 15.1: Evolution of Architectures in NAS using Evolutionary Algorithm</figcaption></figure>

acyclic graph (DAG) $G_i = (V_i, E_i)$, where $V_i$ represents the set of layers (e.g., convolutional layers, pooling layers, fully connected layers), and $E_i$ represents the connections between them. A typical encoding scheme maps $G_i$ to a vector representation $x_i \in \mathbb{R}^d$, where each element represents hyperparameters such as the number of filters $F_l$ in a convolutional layer $l$, kernel sizes $K_l$, and activation functions $\sigma_l$. The architecture search problem can thus be formulated as an optimization problem:

$$ \max_{x_i \in \mathcal{X}} f(x_i) \qquad (15.15) $$

where $\mathcal{X}$ is the feasible set of architectures defined by design constraints (e.g., hardware constraints, FLOP limitations). The evolutionary process begins with the initialization of a population $\mathcal{P}^{(0)}$, where architectures are randomly generated or sampled from prior knowledge. At each generation $t$, a selection operator $S$ chooses a subset of architectures $\mathcal{P}_s^{(t)} \subset \mathcal{P}^{(t)}$ based on their fitness scores:

$$ \mathcal{P}_s^{(t)} = S(\mathcal{P}^{(t)}, f) \qquad (15.16) $$

Common selection methods include tournament selection and rank-based selection. The selected architectures undergo crossover and mutation operations. Crossover combines two parent architectures $N_a$ and $N_b$ to produce an offspring $N_c$ using a recombination function $C$:

$$ N_c = C(N_a, N_b) \qquad (15.17) $$

For instance, if architectures are encoded as binary strings representing layer connections, one-point or uniform crossover can be applied:

$$ x_c = (x_a^{(1:k)}, x_b^{(k+1:d)}) \qquad (15.18) $$
where k is a randomly selected crossover point. Mutation introduces random variations in the

<figure><img src="image_164.png" /><figcaption>Figure 15.2: Evolutionary Algorithm in Neural Architecture Search</figcaption></figure>

offspring architecture to maintain diversity in the population. Given a mutation rate $p_m$, a mutation operator $M$ perturbs the architecture encoding:

$$x'_{i} = M(x_{i}, p_{m}) \qquad (15.19)$$

Common mutation operations include altering the number of filters $F_l$, modifying kernel sizes $K_l$, or randomly inserting/deleting layers. The updated population $\mathcal{P}^{(t+1)}$ for the next generation is formed by selecting the top-performing architectures:

$$\mathcal{P}^{(t+1)} = \text{Elitism}(\mathcal{P}_{s}^{(t)} \cup \mathcal{P}_{c}^{(t)}, f) \qquad (15.20)$$

where $\mathcal{P}_{c}^{(t)}$ is the set of offspring architectures and elitism ensures that the best architectures persist across generations. The process repeats until a termination criterion is met, such as a maximum number of generations $T$ or convergence in fitness scores:

$$|f(\mathcal{N}^{(t)}) - f(\mathcal{N}^{(t-1)})| < \epsilon \qquad (15.21)$$

where $\epsilon$ is a predefined tolerance threshold. The search efficiency of EAs can be enhanced using surrogate models $\hat{f}$ that approximate the fitness function:

$$\hat{f}(x) = \sum_{i=1}^{n} w_{i}K(x, x_{i}) \qquad (15.22)$$

where $K(x, x_i)$ is a kernel function and $w_i$ are weights learned from evaluated architectures. Bayesian optimization and neural predictors are often employed to guide the search towards promising regions of $\mathcal{X}$. To further improve convergence speed, weight inheritance strategies transfer trained weights from parent networks to offspring, reducing the need for full training cycles. Given a parent-offspring pair $(\mathcal{N}_p, \mathcal{N}_c)$, weight inheritance is expressed as:

$$W_c = \mathcal{T}(W_p, \mathcal{N}_p, \mathcal{N}_c) \qquad (15.23)$$
where $W_p$ and $W_c$ denote the weight matrices of the parent and child networks, and $\mathcal{T}$ is a transformation function that maps weights based on structural similarities. Evolutionary NAS has demonstrated competitive performance against reinforcement learning-based and gradient-based approaches, particularly in discovering novel architectures for convolutional neural networks (CNNs) and transformers. The combination of evolutionary search with differentiable search spaces, such as in Differentiable Architecture Search (DARTS), leads to hybrid methods where architectures evolve within a continuous relaxation of the search space:

$$ \mathcal{L}(\alpha) = \sum_{i} w_{i} \sigma(\alpha_{i}) \qquad (15.24) $$

where $\alpha$ represents architecture parameters and $\sigma(\cdot)$ is a softmax function ensuring differentiability. By integrating EAs with differentiable optimization, recent methods achieve state-of-the-art performance while maintaining exploration capabilities.

## 15.2 Reinforcement Learning in Network Architecture Search

Reinforcement Learning (RL) plays a pivotal role in Network Architecture Search (NAS) for Deep Neural Networks (DNNs) by formulating the problem as a sequential decision-making process, wherein a policy optimizes the selection of architectural components to maximize a predefined performance metric, typically validation accuracy. The process is modeled as a Markov Decision

<figure><img src="image_165.png" /><figcaption>Figure 15.3: Reinforcement Learning in Neural Architecture Search (NAS)</figcaption></figure>

Process (MDP), defined by the tuple $(S, A, P, R, \gamma)$, where $S$ represents the state space corresponding to partially constructed architectures, $A$ is the action space defining possible modifications to architectures, $P$ is the transition probability governing the evolution of architectures, $R$ denotes the reward function that quantifies model performance, and $\gamma$ is the discount factor controlling long-term reward contributions. The objective is to optimize the policy $\pi(a|s; \theta)$, parameterized by $\theta$, such that the expected cumulative reward is maximized:

$$ J(\theta) = \mathbb{E}_{\pi} \left[ \sum_{t=0}^{T} \gamma^t R_t \right]. \qquad (15.25) $$
To optimize $J(\theta)$, policy gradient methods are employed, where the gradient of the expected reward with respect to policy parameters is given by:

$$ \nabla_{\theta} J(\theta) = \mathbb{E}_{\pi} \left[ \sum_{t=0}^{T} \nabla_{\theta} \log \pi(a_t|s_t; \theta) R_t \right]. \quad (15.26) $$

Using Monte Carlo estimates, the policy parameters are updated via gradient ascent:

$$ \theta \leftarrow \theta + \alpha \sum_{t=0}^{T} \nabla_{\theta} \log \pi(a_t|s_t; \theta) R_t. \quad (15.27) $$

A common approach to implementing NAS with RL is the use of a Recurrent Neural Network (RNN)-based controller that generates candidate architectures sequentially. The hidden state $h_t$ of the RNN encodes information about the architecture decisions made up to step $t$, and the action $a_t$ at each step is sampled from a softmax distribution:

$$ a_t \sim \text{Softmax}(Wh_t + b). \quad (15.28) $$

The reward signal is obtained by training and evaluating the generated architecture on a validation set, yielding an accuracy score $A$, which serves as the reward:

$$ R = A - b, \quad (15.29) $$

where $b$ is a baseline used in variance reduction techniques such as REINFORCE. The training of the controller follows the REINFORCE algorithm with baseline subtraction:

$$ \theta \leftarrow \theta + \alpha(R - b) \sum_{t=0}^{T} \nabla_{\theta} \log \pi(a_t|s_t; \theta). \quad (15.30) $$

To further stabilize learning, advantage estimation methods such as Generalized Advantage Estimation (GAE) can be employed, where the advantage function $A_t$ is computed using the value function $V(s_t)$:

$$ A_t = R_t + \gamma V(s_{t+1}) - V(s_t). \quad (15.31) $$

An alternative approach is to model the NAS problem as a Q-learning task, where the Q-value represents the expected future reward given a state-action pair:

$$ Q(s_t, a_t) = \mathbb{E} \left[ \sum_{k=0}^{\infty} \gamma^k R_{t+k} \mid s_t, a_t \right]. \quad (15.32) $$

The Q-values are updated using the Bellman equation:

$$ Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha \left( R_t + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t) \right). \quad (15.33) $$

In practical implementations, Deep Q-Networks (DQN) are utilized, where a neural network approximates the Q-function $Q(s, a; \theta)$, and updates are made using the loss function:

$$ L(\theta) = \mathbb{E}_{(s,a,r,s') \sim D} \left[ \left( r + \gamma \max_{a'} Q(s', a'; \theta^{-}) - Q(s, a; \theta) \right)^2 \right], \quad (15.34) $$

where $\theta^{-}$ represents target network parameters and $D$ is an experience replay buffer. To address the instability of Q-learning, Double Q-learning is often employed, where two networks, $Q_1$ and $Q_2$, are used to decouple action selection and evaluation:

$$ Q(s_t, a_t) = r_t + \gamma Q_{\min}(s_{t+1}, a_{t+1}), \quad (15.35) $$
where $Q_{\min}$ selects the minimum Q-value from two estimators to mitigate overestimation bias. In actor-critic methods such as Proximal Policy Optimization (PPO), the actor updates the policy while the critic evaluates its performance using the objective:

$$L(\theta) = \mathbb{E} \left[ \min \left( \frac{\pi(a_t|s_t; \theta)}{\pi(a_t|s_t; \theta_{\text{old}})} A_t, \text{clip} \left( \frac{\pi(a_t|s_t; \theta)}{\pi(a_t|s_t; \theta_{\text{old}})}, 1-\epsilon, 1+\epsilon \right) A_t \right) \right]. \quad (15.36)$$

This clipped objective prevents large policy updates, ensuring stability in the RL-based NAS framework. By iteratively refining the architecture using these RL techniques, NAS can effectively discover high-performing network topologies that outperform manually designed architectures.

## 15.3 Policy gradient methods in Network Architecture Search

Policy gradient methods provide a mathematically rigorous framework for optimizing network architectures in deep neural networks by formulating the architecture search as a reinforcement learning (RL) problem. The core idea is to define a probability distribution over possible network architectures, represented as a parametric policy $\pi_{\theta}(a|s)$, where $s$ represents the state (e.g., the current design choices or hyperparameters of the network), and $a$ represents an action (e.g., selecting a particular layer type, activation function, or connection pattern). The objective is to maximize

<figure><img src="image_166.png" /><figcaption>Figure 15.4: Policy Gradient Methods in Neural Architecture Search (NAS)</figcaption></figure>

the expected performance $J(\theta)$ of the architecture, which is commonly formulated as the expected reward obtained from training and evaluating the selected network:

$$J(\theta) = \mathbb{E}_{\tau \sim p_{\theta}(\tau)} [R(\tau)] \quad (15.37)$$

where $\tau = (s_0, a_0, s_1, a_1, \dots, s_T, a_T)$ is a trajectory of states and actions sampled according to the policy $\pi_{\theta}(a|s)$, and $R(\tau)$ represents the cumulative reward (e.g., validation accuracy of the trained architecture). The distribution over trajectories is given by

$$p_{\theta}(\tau) = p(s_0) \prod_{t=0}^{T} \pi_{\theta}(a_t|s_t)p(s_{t+1}|s_t, a_t) \quad (15.38)$$
where $p(s_{t+1}|s_t, a_t)$ is the environment transition function, often deterministic in the context of architecture search. To optimize $J(\theta)$, the policy gradient theorem provides an unbiased estimator of the gradient using the likelihood ratio trick:

$$ \nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim p_{\theta}(\tau)} [R(\tau) \nabla_{\theta} \log p_{\theta}(\tau)]. \qquad (15.39) $$

Expanding the gradient term, we obtain

$$ \nabla_{\theta} \log p_{\theta}(\tau) = \sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t|s_t). \qquad (15.40) $$

Thus, the gradient estimate simplifies to

$$ \nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim p_{\theta}(\tau)} \left[ R(\tau) \sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t|s_t) \right]. \qquad (15.41) $$

In practice, this expectation is estimated via Monte Carlo sampling over a set of architectures, leading to the update rule:

$$ \theta \leftarrow \theta + \alpha \sum_{\tau \sim p_{\theta}(\tau)} R(\tau) \sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t|s_t). \qquad (15.42) $$

To improve convergence, a baseline $b(s_t)$ is often introduced to reduce variance:

$$ \nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim p_{\theta}(\tau)} \left[ \sum_{t=0}^{T} (R(\tau) - b(s_t)) \nabla_{\theta} \log \pi_{\theta}(a_t|s_t) \right]. \qquad (15.43) $$

A common choice for $b(s_t)$ is a learned value function $V^\pi(s_t)$, leading to an advantage function $A(s_t, a_t) = R(\tau) - V^\pi(s_t)$, which forms the basis for actor-critic methods. The reward function $R(\tau)$ is typically designed to reflect the generalization ability of the architecture, often incorporating the validation accuracy of the trained model:

$$ R(\tau) = \frac{1}{N} \sum_{i=1}^{N} \ell(y_i, f_{\mathcal{A}}(x_i; w^*)) \qquad (15.44) $$

where $\mathcal{A}$ denotes the architecture defined by the trajectory $\tau$, $f_{\mathcal{A}}$ represents the network function, and $w^*$ are the trained weights obtained by minimizing the training loss

$$ w^* = \arg \min_{w} \sum_{i=1}^{N} \ell(y_i, f_{\mathcal{A}}(x_i; w)). \qquad (15.45) $$

To improve exploration, entropy regularization is often added to the objective:

$$ J'(\theta) = J(\theta) + \lambda H(\pi_{\theta}) \qquad (15.46) $$

where

$$ H(\pi_{\theta}) = - \sum_{t=0}^{T} \sum_{a} \pi_{\theta}(a|s_t) \log \pi_{\theta}(a|s_t) \qquad (15.47) $$

encourages diverse architecture sampling. In a real implementation, the policy $\pi_{\theta}$ is usually parameterized by a recurrent neural network (RNN) such as an LSTM, where the hidden state encodes past architecture decisions. The architecture parameters $\theta$ are updated using stochastic gradient descent with policy gradient updates, while the sampled architectures are trained using standard
backpropagation. The convergence of policy gradient methods in architecture search relies on ensuring sufficient exploration and avoiding premature convergence to suboptimal architectures. Techniques such as proximal policy optimization (PPO), trust region policy optimization (TRPO), and natural gradient methods are often used to stabilize updates and improve sample efficiency. These methods modify the standard gradient update by introducing constraints such as

$$D_{\mathrm{KL}}(\pi_{\theta_{\mathrm{old}}}|\pi_{\theta}) \leq \delta \qquad (15.48)$$

where $D_{\mathrm{KL}}$ is the Kullback-Leibler divergence, ensuring that the updated policy does not deviate too drastically from the previous policy, thereby maintaining stability in the optimization process.

## 15.4 Neural tangent kernels (NTKs) in Network Architecture Search

Neural Tangent Kernels (NTKs) have emerged as a powerful theoretical framework for understanding the training dynamics and generalization properties of deep neural networks in the infinite-width limit. In the context of Network Architecture Search (NAS), NTKs provide a mathematically rigorous way to evaluate the expressivity and trainability of different neural network architectures without explicitly training them, thereby significantly reducing computational costs associated with traditional NAS methods. The NTK is formally defined as the Gram matrix of the Jacobian

<figure><img src="image_167.png" /><figcaption>Figure 15.5: NTKs, Eigenvalue Spectra, and Condition Numbers for NAS Architectures</figcaption></figure>

of the neural network outputs with respect to its parameters, capturing how perturbations in parameters influence the output function. Mathematically, if $f_{\theta}(x)$ denotes the output of a neural network with parameters $\theta$ for an input $x$, then the NTK is given by

$$\Theta(x, x') = \mathbb{E}_{\theta} \left[ \nabla_{\theta} f_{\theta}(x) \cdot \nabla_{\theta} f_{\theta}(x')^{T} \right]. \qquad (15.49)$$

As the width of the network approaches infinity, the NTK remains constant during training under gradient descent, allowing the network to be effectively modeled as a Gaussian process. This property enables rapid evaluation of the convergence behavior and generalization ability of a given architecture by analyzing the eigenvalues of the NTK matrix. The training dynamics of an infinitely
wide neural network can be described using the evolution of the function $f_t(x)$ under gradient descent with a learning rate $\eta$, which follows

$$ \frac{df_t(x)}{dt} = -\eta \sum_{i=1}^{n} \Theta(x, x_i)(f_t(x_i) - y_i). \qquad (15.50) $$

Since the NTK remains constant in the infinite-width regime, this equation can be solved explicitly using

$$ f_t(x) = f_0(x) + \Theta(x, X)\Theta(X, X)^{-1}(e^{-\eta\Theta(X,X)t}(y - f_0(X))), \qquad (15.51) $$

where $X = \{x_1, \dots, x_n\}$ denotes the training set inputs, and $y = \{y_1, \dots, y_n\}$ are the corresponding targets. The rate of convergence is determined by the smallest eigenvalue $\lambda_{\min}$ of $\Theta(X, X)$, implying that architectures with larger minimum eigenvalues of their NTKs converge faster and are thus more trainable. The NTK perspective enables a principled approach to NAS by evaluating candidate architectures based on their kernel spectra, particularly the condition number $\kappa(\Theta) = \frac{\lambda_{\max}}{\lambda_{\min}}$, which governs the stability of gradient descent. Generalization in deep learning is often characterized by the smoothness and complexity of learned functions. The NTK controls the function space induced by a given architecture, allowing estimation of generalization error through the kernel ridge regression formula

$$ \mathbb{E}_{\text{test}} \left[ (f(x_{\text{test}}) - y_{\text{test}})^2 \right] \approx \sigma^2 \text{Tr} \left( \Theta(X, X)^{-1} \Theta(X_{\text{test}}, X_{\text{test}}) \Theta(X, X)^{-1} \right), \qquad (15.52) $$

where $\sigma^2$ is the noise variance. Architectures with better generalization properties exhibit smaller trace norms of the NTK inverse. In the infinite-width limit, NTKs of different architectures can

<figure><img src="image_168.png" /><figcaption>Figure 15.6: CNN NTKs and Eigenvalue Spectra for NAS Architectures</figcaption></figure>

be computed analytically by considering the recursive propagation of covariance matrices through network layers. For a fully connected network with ReLU activation, the NTK at depth $L$ is given by

$$ \Theta^{(L)}(x, x') = \Theta^{(L-1)}(x, x') \dot{K}^{(L)}(x, x'), \qquad (15.53) $$
where $\dot{K}^{(L)}(x, x')$ is the derivative of the activation function's covariance kernel. Different architectures, such as convolutional neural networks (CNNs), modify this recurrence relation by incorporating weight-sharing and local connectivity constraints. The NTK for a CNN layer with filter size $k$ is

$$ \Theta_{\text{conv}}^{(L)}(x, x') = \frac{1}{k^2} \sum_{i,j} \Theta^{(L-1)}(x_i, x'_j) \dot{K}^{(L)}(x_i, x'_j). \qquad (15.54) $$

This allows NAS to be performed analytically by evaluating how architectural choices influence the NTK and its spectral properties, bypassing the need for expensive training runs. Recent work has extended NTK-based NAS to attention-based architectures, such as transformers, where the self-attention mechanism induces a structured NTK

$$ \Theta_{\text{attn}}(x, x') = \mathbb{E} \left[ \text{softmax} \left( \frac{Q(x)K(x')^T}{\sqrt{d}} \right) V(x)V(x')^T \right], \qquad (15.55) $$

where $Q, K, V$ are the query, key, and value matrices. The spectrum of $\Theta_{\text{attn}}$ determines the expressivity and trainability of transformer-based models, enabling NTK-based architectural optimization in sequence learning tasks. The NTK framework provides an elegant, mathematically rigorous approach to NAS by linking architectural choices to trainability and generalization through spectral analysis of the kernel.
# 16 Learning paradigms

## 16.1 Unsupervised learning

### 16.1.1 Literature Review of Unsupervised Learning

Unsupervised learning has evolved significantly through foundational contributions in clustering, probabilistic modeling, neural representations, and generative techniques. One of the earliest and most fundamental methods in clustering was introduced by MacQueen in 1967 [1035] with the k-means algorithm. This algorithm partitions a set of *n* observations into *k* clusters in such a way that the intra-cluster variance is minimized. The iterative nature of k-means, involving cluster centroid updates and reassignment of points based on their Euclidean distances, remains a cornerstone of clustering techniques. Complementing k-means, the Expectation-Maximization (EM) algorithm, rigorously formalized by Dempster, Laird, and Rubin in 1977 [1036] provided a general framework for maximizing likelihood estimates in models with latent variables. EM underpins Gaussian Mixture Models (GMMs), allowing for probabilistic clustering by modeling data as a weighted sum of Gaussian distributions. These models account for uncertainty and have led to advancements in density estimation and soft clustering.

Another major breakthrough in unsupervised learning emerged with the self-organizing maps (SOMs) developed by Kohonen in 1982 [1037]. These neural-inspired models provide a competitive learning framework where neurons adjust their weights to form low-dimensional representations of input data while preserving topological relationships. This biologically motivated approach to clustering has been instrumental in feature extraction and visualization, especially in applications involving high-dimensional data. Parallel to SOMs, spectral techniques for manifold learning and dimensionality reduction were rigorously developed, with Belkin and Niyogi [1038] introducing Laplacian Eigenmaps in 2003. This method constructs a graph Laplacian to capture local geometric properties of data manifolds and embed them into a lower-dimensional space while maintaining neighborhood relationships. This has provided a mathematically principled foundation for spectral clustering and nonlinear dimensionality reduction.

The rigorous treatment of information-theoretic principles in unsupervised learning was significantly advanced by the Information Bottleneck (IB) method proposed by Tishby, Pereira, and Bialek in 2000 [1039]. The IB principle establishes a trade-off between compression and predictive efficiency by optimizing a mutual information objective. This formulation has deeply influenced representation learning, particularly in autoencoders and deep latent variable models. A major advancement in unsupervised neural representations was introduced by Hinton and Salakhutdinov in 2006 [1040], demonstrating that deep belief networks (DBNs) could learn hierarchical representations via layer-wise training of restricted Boltzmann machines (RBMs). This work laid the foundation for the resurgence of deep learning by showing that unsupervised pretraining significantly improves the performance of neural networks. The probabilistic formulation of deep unsupervised learning was further developed by Kingma and Welling in 2013 [1041] with the introduction of variational autoencoders (VAEs). By leveraging variational inference, VAEs provide a rigorous framework for learning latent variable models, enabling efficient probabilistic generative modeling
of complex data distributions.

A more adversarial approach to generative modeling was pioneered by Goodfellow et al. in 2020 [121] with the introduction of generative adversarial networks (GANs). GANs consist of a generator and discriminator competing in a minimax game, where the generator learns to produce realistic samples while the discriminator improves its ability to distinguish between real and generated data. This adversarial framework has profoundly impacted image generation, domain adaptation, and semi-supervised learning by producing high-quality synthetic data. The visualization of high-dimensional data was further enhanced by van der Maaten and Hinton in 2008 [1042] with the development of t-distributed stochastic neighbor embedding (t-SNE). This method provides a probabilistic approach to mapping data points into a lower-dimensional space by minimizing the Kullback-Leibler divergence between high- and low-dimensional distributions, preserving local similarities. t-SNE has become a widely used technique in exploratory data analysis due to its ability to reveal meaningful structure in complex datasets.

Together, these contributions have shaped the field of unsupervised learning by introducing rigorous mathematical formulations for clustering, probabilistic modeling, and representation learning. The foundational algorithms such as k-means and EM provide essential tools for unsupervised data analysis, while information-theoretic and spectral methods offer deeper insights into structure preservation and feature extraction. The integration of neural networks into unsupervised learning, through self-organizing maps, deep belief networks, and variational autoencoders, has led to a significant expansion in the capabilities of machine learning models. The development of adversarial training with GANs and the application of information bottleneck principles further illustrate the growing sophistication of unsupervised learning techniques. These advancements continue to push the boundaries of artificial intelligence, enabling models to learn rich, structured representations of data without the need for explicit supervision.

Roweis and Saul (2000) [1043] introduced Locally Linear Embedding (LLE), a nonlinear dimensionality reduction algorithm designed to uncover low-dimensional structures within high-dimensional data. The core idea behind LLE is to preserve the local geometric relationships between neighboring data points while embedding them into a lower-dimensional space. Unlike linear techniques such as Principal Component Analysis (PCA), which assume that the global structure of the data can be well represented using a linear subspace, LLE focuses on preserving the local structure by considering each data point and its nearest neighbors. The algorithm operates in three main steps. First, for each data point, LLE identifies a set of nearest neighbors based on a distance metric, usually Euclidean distance. Second, it computes local reconstruction weights by expressing each data point as a linear combination of its neighbors, minimizing the reconstruction error while enforcing the constraint that the weights sum to one. This step ensures that the local geometric structure is encoded in an invariant manner. Third, LLE determines a lower-dimensional embedding by finding a new set of coordinates that best preserves the reconstruction weights obtained in the previous step. This is achieved by minimizing a quadratic cost function that ensures the lower-dimensional representation maintains the same local linear relationships as the original high-dimensional data. The main advantage of LLE over traditional linear techniques is its ability to recover nonlinear manifolds, making it particularly effective for data sets where the underlying structure is curved or highly nonlinear. Unlike methods such as Multidimensional Scaling (MDS), which rely on pairwise distances between all points and can be computationally expensive, LLE leverages only local neighborhoods, making it more scalable to larger data sets. Additionally, since LLE does not impose any parametric assumptions on the manifold, it is capable of adapting to a wide variety of data distributions.

However, LLE also has some limitations. Its reliance on nearest-neighbor selection makes it sensitive
to noise and parameter choices, particularly the number of neighbors. Furthermore, the embeddings produced by LLE are often unnormalized, meaning that distances in the lower-dimensional space may not have a straightforward interpretation. Despite these challenges, LLE has had a significant impact on machine learning and data visualization, providing a powerful tool for uncovering the intrinsic structure of high-dimensional data in applications ranging from image processing to bioinformatics. Its introduction marked a shift toward more flexible, geometry-preserving dimensionality reduction techniques, influencing the development of subsequent manifold learning methods such as Isomap and t-SNE.

Bell and Sejnowski (1995) [1044] introduced Independent Component Analysis (ICA) as an information-theoretic approach to blind source separation (BSS), which allows for the decomposition of mixed signals into statistically independent components without prior knowledge of the mixing process. Their work was motivated by real-world scenarios where multiple signals, such as different voices in a crowded room or overlapping audio sources in a recording, become mixed together, making it difficult to recover the original sources. Unlike traditional statistical methods such as Principal Component Analysis (PCA), which rely on second-order statistics and assume orthogonality of components, ICA leverages higher-order statistical properties to separate signals that are non-Gaussian and statistically independent. The fundamental principle behind their method is the maximization of information transfer in a neural network, also known as the infomax principle. This principle is based on the idea that a system should be optimized to maximize the mutual information between its input and output, ensuring that the output representation is as statistically independent as possible. To achieve this, they formulated a neural learning rule that iteratively adjusts the network's weight parameters using a nonlinear function, which enhances the separation of independent sources. The key insight was that statistical independence implies that the joint probability distribution of the recovered signals should factorize into the product of their individual distributions, which is not the case for mixtures of dependent signals. To enforce this independence, their method used a contrast function derived from information theory, ensuring that the recovered sources were as non-Gaussian as possible, since mixtures of independent sources tend to become more Gaussian due to the Central Limit Theorem. This approach enabled the successful separation of mixed signals in a variety of applications, including audio processing, biomedical signal analysis, and image processing. One of the most famous demonstrations of ICA, often referred to as the “cocktail party problem,” involves separating multiple overlapping speech signals recorded by different microphones. Their algorithm was able to recover individual voices from the mixed recording with remarkable accuracy, highlighting the effectiveness of ICA in practical scenarios. Additionally, ICA found significant applications in neuroscience, particularly in electroencephalography (EEG) and functional magnetic resonance imaging (fMRI), where it helped isolate meaningful brain activity patterns from background noise.

Despite its power, ICA has limitations, including its sensitivity to the choice of nonlinear functions and the assumption that the number of independent sources does not exceed the number of observed mixtures. Furthermore, ICA assumes that the sources are statistically independent, which may not always hold in real-world data. Nonetheless, Bell and Sejnowski's work laid the foundation for subsequent advancements in unsupervised learning, influencing modern approaches in deep learning, signal processing, and latent variable modeling. Their method provided a robust mathematical framework for data decomposition that has since been expanded and refined in numerous fields, demonstrating the lasting impact of their contribution to unsupervised learning.
<table><caption>Table 16.1: Summary of Contributions to the Unsupervised Learning</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>MacQueen (1967) [1035]</td><td>Introduced the k-means algorithm, a foundational clustering method that minimizes intra-cluster variance through iterative centroid updates and point reassignment based on Euclidean distance.</td></tr><tr><td>Dempster, Laird, and Rubin (1977) [1036]</td><td>Developed the Expectation-Maximization (EM) algorithm, a general framework for maximizing likelihood estimates in models with latent variables, forming the basis for Gaussian Mixture Models (GMMs).</td></tr><tr><td>Kohonen (1982) [1037]</td><td>Proposed self-organizing maps (SOMs), a neural-inspired model for competitive learning, which preserves topological relationships and has been instrumental in feature extraction.</td></tr><tr><td>Belkin and Niyogi (2003) [1038]</td><td>Introduced Laplacian Eigenmaps, which use a graph Laplacian to capture local geometric properties of data manifolds, providing a foundation for spectral clustering and nonlinear dimensionality reduction.</td></tr><tr><td>Tishby, Pereira, and Bialek (2000) [1039]</td><td>Proposed the Information Bottleneck (IB) method, optimizing mutual information to balance compression and predictive efficiency, influencing representation learning in autoencoders.</td></tr><tr><td>Hinton and Salakhutdinov (2006) [1040]</td><td>Demonstrated deep belief networks (DBNs), where layer-wise training of restricted Boltzmann machines (RBMs) enables hierarchical unsupervised representation learning.</td></tr><tr><td>Kingma and Welling (2013) [1041]</td><td>Developed variational autoencoders (VAEs), leveraging variational inference for probabilistic generative modeling of complex data distributions.</td></tr><tr><td>Goodfellow et al. (2020) [121]</td><td>Introduced generative adversarial networks (GANs), an adversarial framework where a generator and discriminator compete, leading to advances in synthetic data generation.</td></tr><tr><td>van der Maaten and Hinton (2008) [1042]</td><td>Developed t-distributed stochastic neighbor embedding (t-SNE), a probabilistic approach for high-dimensional data visualization that preserves local similarities.</td></tr><tr><td>Roweis and Saul (2000) [1043]</td><td>Introduced Locally Linear Embedding (LLE), a nonlinear dimensionality reduction technique that preserves local geometric relationships and is effective for manifold learning.</td></tr><tr><td>Bell and Sejnowski (1995) [1044]</td><td>Developed Independent Component Analysis (ICA), an information-theoretic method for blind source separation, leveraging higher-order statistics to extract statistically independent signals.</td></tr></tbody></table>

## 16.1.2 Recent Literature Review of Unsupervised Learning

Unsupervised learning has emerged as a powerful tool in various domains, offering novel solutions where labeled data is scarce or unavailable. In semiconductor manufacturing, Parmar (2020) [1045] introduced an unsupervised learning framework aimed at identifying previously unknown defects. By leveraging clustering techniques and anomaly detection, this approach enhances quality control without relying on predefined defect categories. The ability to discover novel defect patterns autonomously makes this method particularly valuable in industries where manufacturing processes constantly evolve, and new defect types emerge. In a related field, Raikwar and Gupta (2025) [1046] developed an AI-driven trust management framework for wireless ad hoc networks. This system
integrates unsupervised and supervised learning to classify network nodes based on trustworthiness, helping identify potentially malicious nodes. Such frameworks are critical in decentralized environments where security threats are unpredictable and dynamically evolving. The application of unsupervised learning in this context reduces reliance on predefined attack signatures, improving network security in real-time.

Structural health monitoring has also benefited from unsupervised learning techniques. Moustakidis et al. (2025) [1047] proposed a deep learning autoencoder framework for fast Fourier transform (FFT)-based clustering, designed to analyze acoustic emission data from composite materials. This method enables automatic detection of structural damage over time, allowing for proactive maintenance and risk mitigation in infrastructure management. By employing autoencoders to extract meaningful features from raw sensor data, the approach outperforms traditional inspection methods that require extensive manual interpretation. Feature selection is another area where unsupervised learning has made a significant impact. Liu et al. (2025) [1048] introduced an unsupervised feature selection algorithm using L2, p-norm feature reconstruction. This method effectively reduces redundant features while preserving essential data structures, thereby improving the efficiency and accuracy of clustering algorithms. High-dimensional datasets, such as those used in bioinformatics and financial modeling, greatly benefit from this technique, as it enables better data representation and pattern discovery without the need for labeled guidance.

In the medical and healthcare sector, unsupervised learning has been leveraged for disease risk assessment and biomarker discovery. Zhou et al. (2025) [1049] applied clustering techniques to analyze metabolic profiles, uncovering hidden subtypes of hypertriglyceridemia associated with varying disease risks. This research demonstrates the potential of unsupervised learning in personalized medicine, where identifying metabolic subgroups can help tailor treatments to individual patients rather than applying a one-size-fits-all approach. Similarly, Lin et al. (2025) [1050] employed unsupervised learning for risk control in health insurance fund management. By analyzing claims data and identifying anomalous patterns, this approach enhances fraud detection and risk assessment, ultimately leading to better resource allocation and cost reduction. The ability to detect fraud without labeled examples is a significant advantage in an industry where fraudulent activities are often sophisticated and constantly evolving.

Beyond healthcare, unsupervised learning has shown promise in improving real-world object detection systems. Huang et al. (2025) [1051] proposed a novel unsupervised domain adaptation technique to enhance open-world object detection. Unlike traditional supervised models that require large amounts of labeled data, this approach enables object detection models to generalize across different environments with minimal supervision. Such advancements are crucial in autonomous navigation, surveillance, and robotics, where training data may not always be representative of real-world conditions. In a different engineering application, Wu and Liu (2025) [1052] introduced a VQ-VAE-2-based algorithm for detecting cracks in concrete structures. This unsupervised approach automates structural health monitoring, reducing dependence on costly manual inspections while improving the accuracy and efficiency of damage assessments.

Natural language processing and medical imaging have also seen substantial contributions from unsupervised learning. Nagelli and Saleena (2025) [1053] developed an aspect-based sentiment analysis model using self-attention mechanisms. This model enables the automatic extraction of sentiment-related features from multilingual datasets, allowing businesses to analyze customer feedback without requiring labeled sentiment data. Such models are essential for real-time sentiment analysis in global markets where user-generated content is vast and diverse. Meanwhile, Ekanayake (2025) [1054] applied deep learning techniques for MRI reconstruction and super-resolution enhancement. This research significantly reduces MRI scan times while preserving high image quality,
offering a transformative solution to the challenges of medical imaging. The use of unsupervised learning in this context enables more efficient data-driven reconstruction techniques, reducing dependency on expensive and time-consuming manually labeled training datasets. These diverse applications illustrate the expanding role of unsupervised learning in solving complex real-world problems, demonstrating its adaptability and potential to drive further innovation across multiple industries.

<table><caption>Table 16.2: Summary of Recent Contributions in Unsupervised Learning</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Parmar (2025) [1045]</td><td>Introduced an unsupervised learning framework for identifying unknown defects in semiconductor manufacturing, leveraging clustering and anomaly detection to improve quality control in industrial settings.</td></tr><tr><td>Raikwar and Gupta (2025) [1046]</td><td>Developed an AI-driven trust management framework for wireless ad hoc networks, combining unsupervised and supervised learning to classify network nodes based on trustworthiness and detect malicious activity.</td></tr><tr><td>Moustakidis et al. (2025) [1047]</td><td>Proposed deep learning autoencoders for FFT-based clustering in structural health monitoring, enabling automated detection of temporal damage evolution in composite materials.</td></tr><tr><td>Liu et al. (2025) [1048]</td><td>Designed an unsupervised feature selection algorithm using L2, p-norm feature reconstruction, reducing redundant features and improving clustering performance for high-dimensional datasets.</td></tr><tr><td>Zhou et al. (2025) [1049]</td><td>Applied unsupervised clustering techniques to metabolic profiles, identifying hidden metabolic subtypes associated with hypertriglyceridemia and disease risks, advancing personalized medicine.</td></tr><tr><td>Lin et al. (2025) [1050]</td><td>Developed an unsupervised learning-based risk control model for health insurance fund management, effectively identifying high-risk groups and fraudulent claims through anomaly detection.</td></tr><tr><td>Huang et al. (2025) [1051]</td><td>Proposed an unsupervised domain adaptation method for open-world object detection, enabling models to generalize across different environments without extensive labeled datasets.</td></tr><tr><td>Wu and Liu (2025) [1052]</td><td>Designed a VQ-VAE-2-based unsupervised detection algorithm for concrete crack identification, automating structural health monitoring and reducing manual inspection efforts.</td></tr><tr><td>Nagelli and Saleena (2025) [1053]</td><td>Developed an aspect-based sentiment analysis model using self-attention mechanisms, enabling multilingual sentiment analysis without labeled training data.</td></tr><tr><td>Ekanayake (2025) [1054]</td><td>Applied deep learning-based unsupervised learning for MRI reconstruction and super-resolution, reducing scan times while maintaining high image quality in medical imaging.</td></tr></tbody></table>
### 16.1.3 Mathematical Analysis of Unsupervised Learning

Unsupervised learning is a fundamental paradigm in machine learning, in which a model is trained on a dataset $\mathcal{D} = \{\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_N\}$ without explicit supervision in the form of labeled outputs. Mathematically, given a dataset consisting of $N$ data points, where each $\mathbf{x}_i$ is a vector in $\mathbb{R}^d$, the goal is to find an underlying structure, distribution, or latent representation of the data, often formulated as a function $f: \mathbb{R}^d \to \mathbb{R}^k$ where $k$ may be lower than $d$, capturing essential properties of the data manifold. The data is assumed to be drawn from an unknown probability distribution $p(\mathbf{x})$, and the central objective in unsupervised learning is to model or approximate this distribution using a function $p_\theta(\mathbf{x})$, where $\theta$ represents the parameters of the model. This can be done using probabilistic modeling, clustering, dimensionality reduction, and manifold learning.

One of the most rigorous formulations in unsupervised learning arises in the estimation of probability density functions, where the likelihood function of the observed data is given by

$$L(\theta) = \prod_{i=1}^{N} p_{\theta}(\mathbf{x}_i). \qquad (16.1)$$

The optimal parameters $\theta^*$ are obtained by maximizing the log-likelihood function:

$$\theta^* = \arg \max_{\theta} \sum_{i=1}^{N} \log p_{\theta}(\mathbf{x}_i). \qquad (16.2)$$

In clustering problems, an objective function is often defined to minimize intra-cluster variance while maximizing inter-cluster separation. Given a set of $K$ cluster centers $\{\mathbf{c}_1, \mathbf{c}_2, \dots, \mathbf{c}_K\}$, the assignment of each point $\mathbf{x}_i$ to a cluster is determined by

$$r_{i,k} = \begin{cases} 1, & \text{if } k = \arg \min_j \|\mathbf{x}_i - \mathbf{c}_j\|^2, \\ 0, & \text{otherwise.} \end{cases} \qquad (16.3)$$

The objective function to minimize is given by

$$J(\mathbf{c}) = \sum_{i=1}^{N} \sum_{k=1}^{K} r_{i,k} \|\mathbf{x}_i - \mathbf{c}_k\|^2. \qquad (16.4)$$

Expectation-Maximization (EM) algorithms generalize clustering in a probabilistic framework, where data is assumed to be generated by a mixture of distributions:

$$p(\mathbf{x}) = \sum_{k=1}^{K} \pi_k p(\mathbf{x} | \theta_k), \qquad (16.5)$$

where $\pi_k$ are the mixture weights such that $\sum_{k=1}^{K} \pi_k = 1$, and $p(\mathbf{x} | \theta_k)$ are component distributions, often taken as Gaussian:

$$p(\mathbf{x} | \theta_k) = \frac{1}{(2\pi)^{d/2} |\Sigma_k|^{1/2}} \exp \left( -\frac{1}{2} (\mathbf{x} - \mu_k)^T \Sigma_k^{-1} (\mathbf{x} - \mu_k) \right). \qquad (16.6)$$

Dimensionality reduction techniques such as Principal Component Analysis (PCA) seek to find a lower-dimensional representation that maximizes variance. Given a centered data matrix $X \in \mathbb{R}^{N \times d}$, PCA finds the eigenvectors of the covariance matrix

$$C = \frac{1}{N} X^T X. \qquad (16.7)$$
The principal components are obtained by solving the eigenvalue problem

$$C\mathbf{v} = \lambda\mathbf{v}. \qquad (16.8)$$

Only the top $k$ eigenvectors corresponding to the largest eigenvalues are retained, reducing the data to a lower-dimensional subspace. The transformation to the new basis is given by

$$\mathbf{z}_i = V^T \mathbf{x}_i, \qquad (16.9)$$

where $V$ is the matrix of top eigenvectors. More advanced methods such as autoencoders learn a mapping from input data $\mathbf{x}$ to a latent representation $\mathbf{z}$ using an encoder function

$$\mathbf{z} = f_{\text{enc}}(\mathbf{x}; \theta_{\text{enc}}) \qquad (16.10)$$

and reconstruct the input via a decoder function

$$\hat{\mathbf{x}} = f_{\text{dec}}(\mathbf{z}; \theta_{\text{dec}}). \qquad (16.11)$$

The reconstruction loss is typically measured as

$$\mathcal{L}(\theta_{\text{enc}}, \theta_{\text{dec}}) = \sum_{i=1}^{N} \|\mathbf{x}_i - f_{\text{dec}}(f_{\text{enc}}(\mathbf{x}_i))\|^2. \qquad (16.12)$$

For probabilistic models, Variational Autoencoders (VAEs) extend autoencoders by introducing a latent variable model with a prior distribution $p(\mathbf{z})$, an approximate posterior $q_{\phi}(\mathbf{z} | \mathbf{x})$, and a reconstruction likelihood $p_{\theta}(\mathbf{x} | \mathbf{z})$. The objective function in VAEs is to maximize the evidence lower bound (ELBO):

$$\mathcal{L}(\theta, \phi) = \mathbb{E}_{q_{\phi}(\mathbf{z}|\mathbf{x})}[\log p_{\theta}(\mathbf{x} | \mathbf{z})] - D_{\text{KL}}(q_{\phi}(\mathbf{z} | \mathbf{x}) \| p(\mathbf{z})), \qquad (16.13)$$

where $D_{\text{KL}}$ is the Kullback-Leibler divergence. Generative models such as Generative Adversarial Networks (GANs) learn a mapping from a simple latent space $\mathbf{z} \sim p(\mathbf{z})$ to the data distribution via a generator function $G: \mathbb{R}^k \to \mathbb{R}^d$, while an adversarial discriminator $D$ learns to distinguish between real and generated samples. The objective function of a GAN is given by

$$\min_{G} \max_{D} \mathbb{E}_{\mathbf{x} \sim p_{\text{data}}} [\log D(\mathbf{x})] + \mathbb{E}_{\mathbf{z} \sim p(\mathbf{z})} [\log(1 - D(G(\mathbf{z})))]. \qquad (16.14)$$

### 16.1.4 Information Bottleneck (IB) method

#### 16.1.4.1 Literature Review of Information Bottleneck (IB) method

The Information Bottleneck (IB) method, introduced by Tishby, Pereira, and Bialek (1999) [1068], provides a principled approach to extracting relevant information from a random variable $X$ concerning another variable $Y$. The method formulates an optimization problem that seeks to find a compressed representation $T$ of $X$ that retains the maximal information about $Y$, while minimizing the redundancy in $X$. This is achieved by balancing the mutual information terms $I(X; T)$ and $I(T; Y)$, where a Lagrange multiplier controls the trade-off between compression and relevance. The authors derived an iterative algorithm based on variational principles to solve this optimization problem and demonstrated its utility in clustering and data representation tasks. This work laid the foundation for a vast array of subsequent research efforts that have expanded the scope of the IB method to different domains and problem settings, providing deeper insights into the nature of information-theoretic compression and representation learning.

One of the key extensions of the IB method was its application to continuous-valued variables,
particularly those following a Gaussian distribution. Chechik, Globerson, Tishby, and Weiss (2005) [1069] rigorously examined the IB method in the context of jointly Gaussian random variables, deriving analytical solutions that revealed the intrinsic connections between the IB formulation and classical techniques such as canonical correlation analysis. Their results demonstrated that the optimal representation in the Gaussian case can be expressed in terms of principal component analysis (PCA)-like transformations, thereby bridging the IB method with traditional dimensionality reduction approaches. This work has had significant implications for signal processing and statistical learning, as it provides a mathematically rigorous foundation for applying the IB principle to real-world datasets that exhibit Gaussian-like characteristics. In a related extension, Chechik and Tishby (2002) [1070] introduced a variant of the IB framework that incorporates side information, allowing for the extraction of representations that retain information about one target variable while being uninformative about another. This formulation has proven particularly useful in privacy-preserving machine learning and fairness-aware representations, where one seeks to ensure that a learned representation encodes task-relevant information while obfuscating sensitive attributes.

In the context of deep learning, Tishby and Zaslavsky (2015) [1071] proposed that the training process of deep neural networks (DNNs) can be interpreted through the lens of the IB principle. They hypothesized that DNNs undergo two distinct phases during training: an initial "fitting" phase where the mutual information between successive layers and the target variable *Y* increases, followed by a "compression" phase where the network progressively reduces the mutual information between intermediate layers and the input variable *X*. This perspective provides a theoretical justification for why deep networks generalize well despite their over-parameterization, as the compression phase effectively filters out irrelevant variations in the input while preserving task-relevant features. However, Saxe et. al. (2019) [1072] critically examined this hypothesis and argued that the presence of the compression phase is highly dependent on architectural choices, particularly the activation functions used in deep networks. Their empirical findings demonstrated that the IB theory does not universally apply to all neural network architectures, suggesting that while information-theoretic compression may play a role in some settings, it is not a fundamental principle governing the training dynamics of all deep networks.

Further empirical support for the IB perspective in deep learning was provided by Shwartz-Ziv and Tishby (2017) [1073], who conducted a detailed analysis of information flow in deep neural networks. Using information plane visualizations, they showed that the training process of DNNs aligns with the two-phase description proposed in earlier work. Their findings provided strong evidence for the compression phenomenon in networks trained with stochastic gradient descent (SGD), reinforcing the argument that deep networks learn compact representations of the input. However, a major challenge in applying the IB framework to high-dimensional data is the difficulty of accurately estimating mutual information, especially in deep networks where the latent representations are highly non-Gaussian. To address this issue, Noshad et. al. (2019) [1074] introduced a novel mutual information estimator based on dependence graphs, enabling more scalable and robust estimations of information flow in deep learning models. Their approach has opened new avenues for applying the IB principle to real-world machine learning problems where traditional mutual information estimators fail due to the curse of dimensionality.

The theoretical implications of information bottleneck in neural networks were further explored by Goldfeld et. al. (2018) [1075], who developed refined techniques for estimating mutual information in deep networks. Their work provided new tools for analyzing the role of compression in neural representations, offering rigorous mathematical justifications for why certain layers in deep networks tend to exhibit strong compression effects. More recently, Geiger (2021) [1077] presented a comprehensive review of information plane analyses in neural network classifiers, evaluating the
strengths and limitations of the IB framework in this setting. His analysis raised critical questions about the general applicability of IB-based insights in deep learning, highlighting cases where the information plane approach fails to provide accurate characterizations of training dynamics. Building on these theoretical developments, Kawaguchi, Deng, Ji, and Huang (2023) [1078] rigorously analyzed the generalization properties of neural networks under the IB framework, providing a mathematical link between information compression and generalization error bounds. Their results establish that controlling the information bottleneck can serve as a regularization mechanism, leading to improved generalization performance in deep learning models. Collectively, these contributions underscore the profound impact of the IB principle across a wide range of disciplines, from statistical signal processing to modern artificial intelligence.

<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Tishby et al. (1999) [1068]</td><td>Introduced the Information Bottleneck (IB) method, formulating an optimization problem that balances mutual information terms to extract relevant information from a random variable while minimizing redundancy. Developed an iterative variational algorithm for solving the IB problem, demonstrating its application in clustering and representation learning.</td></tr><tr><td>Chechik et al. (2003) [1069]</td><td>Extended the IB method to jointly Gaussian variables, deriving analytical solutions that connect IB with canonical correlation analysis and PCA. Provided a rigorous foundation for applying IB to real-world Gaussian data.</td></tr><tr><td>Chechik and Tishby (2002) [1070]</td><td>Developed a variant of the IB framework incorporating side information, enabling extraction of representations that retain information about one target while obfuscating another. Applied to privacy-preserving and fairness-aware machine learning.</td></tr><tr><td>Tishby and Zaslavsky (2015) [1071]</td><td>Proposed that deep neural network training follows an IB perspective, consisting of an initial fitting phase followed by a compression phase, explaining generalization through information-theoretic principles.</td></tr><tr><td>Saxe et al. (2019) [1072]</td><td>Critically examined the IB hypothesis in deep learning, showing that the presence of a compression phase depends on network architecture and activation functions, challenging the universality of IB in training dynamics.</td></tr><tr><td>Shwartz-Ziv and Tishby (2017) [1073]</td><td>Conducted empirical analysis of information flow in deep networks using information plane visualizations, providing evidence for compression in networks trained with SGD and reinforcing IB-based interpretations.</td></tr><tr><td>Noshad et al. (2019) [1074]</td><td>Developed a new mutual information estimator using dependence graphs to improve the scalability and accuracy of IB-based analyses in high-dimensional settings, addressing limitations of traditional estimators.</td></tr><tr><td>Goldfeld et al. (2018) [1075]</td><td>Provided refined mutual information estimation techniques for deep networks, offering rigorous mathematical justifications for compression effects in neural representations.</td></tr></tbody></table>
<table><tr><td>Geiger (2021) [1077]</td><td>Reviewed information plane analyses in neural classifiers, evaluating the strengths and weaknesses of IB interpretations, highlighting cases where IB fails to accurately characterize training dynamics.</td></tr><tr><td>Kawaguchi et al. (2023) [1078]</td><td>Analyzed generalization properties of neural networks under IB, linking information compression to generalization error bounds and establishing IB as a regularization mechanism for improved performance.</td></tr></table>

### 16.1.4.2 Recent Literature Review of Information Bottleneck (IB) method

The Information Bottleneck (IB) method has been widely applied across various disciplines, contributing significantly to fields such as machine learning, reinforcement learning, adversarial robustness, plant metabolomics, and quantum computing. One of the most notable contributions comes from Dardour et al. (2025) [1079], who introduced a novel approach to enhance adversarial robustness in stochastic neural networks. By leveraging inter-separability and intra-concentration, their study demonstrated that using the IB principle to constrain the representation space allows neural networks to learn more robust latent features, effectively mitigating the effects of adversarial perturbations. Similarly, Krinner et al. (2025) [1080] applied IB principles to reinforcement learning by designing state-space world models that accelerate learning efficiency. Their work showed that IB-based methods enable an agent to discard irrelevant environmental noise while retaining essential features, thereby improving exploration efficiency and overall performance in model-based reinforcement learning tasks. Yildirim et. al. (2025) [1081] explored how IB constraints affect StyleGAN-based image editing and demonstrated that GAN-based inversion techniques often suffer from detail loss due to excessive compression, thus proposing refined inversion methods that better preserve fine-grained information.

Yang et al. (2025) [1082] introduced a cognitive-load-aware activation mechanism for large language models (LLMs), significantly improving efficiency by dynamically activating only the necessary model parameters. Their study leveraged IB principles to ensure that LLMs retain only the most relevant contextual representations while discarding redundant computations, reducing computational overhead without sacrificing accuracy. Similarly, Liu et al. (2025) [1083] incorporated IB principles in their Vision Mamba network for crack segmentation in infrastructure, designing a structure-aware model that efficiently filters out redundant spatial information. By applying IB techniques, they achieved enhanced computational efficiency and superior segmentation accuracy, which is crucial for real-time applications in structural health monitoring. Stierle and Valtere (2025) [1084] took a different approach by applying IB theory to medical innovation, examining how bottlenecks in information access within regulatory and patent frameworks slow down gene therapy advancements. Their work provided a comprehensive analysis of how information bottlenecks in medical research and policy impede technological progress, emphasizing the necessity of optimized regulatory frameworks.

Another significant study by Chen et al. (2025) [1085] applied IB concepts to quantum computing, particularly in optimizing construction supply chains. Their work demonstrated that quantum models, when integrated with IB techniques, could efficiently compress relevant data while filtering out extraneous information, leading to improved decision-making and enhanced scheduling flexibility. Yuan et al. (2025) [1086] extended IB applications to plant metabolomics, where they proposed a novel feature selection approach to retain highly informative metabolite interactions while discarding non-essential data. This method improved interpretability in plant metabolic studies, allowing researchers to better understand metabolite interactions without being overwhelmed by excessive data complexity. In a related domain, Dey et al. (2025) [1087] utilized IB principles in
spatio-temporal prediction models for NDVI (Normalized Difference Vegetation Index), a critical measure for rice crop yield forecasting. Their IB-augmented neural network significantly enhanced prediction performance by filtering out irrelevant environmental variables while maintaining the most crucial features for accurate yield assessment.

Further expanding IB applications, Li (2025) [1088] employed IB principles in robotic path planning by developing an optimized method for navigation path extraction in mobile robots. Their approach utilized IB constraints to eliminate irrelevant environmental noise while preserving the most crucial navigational data, thereby improving the efficiency and reliability of robotic movement. This research has significant implications for autonomous navigation systems, where maintaining a compact yet informative representation of the surrounding environment is essential. Finally, Krinner et al. (2025) [1080] extended IB applications to reinforcement learning, emphasizing the importance of information retention in decision-making models. Their proposed IB-based reinforcement learning framework demonstrated superior generalization capabilities compared to traditional approaches, as it effectively retained task-relevant information while discarding unnecessary complexity. This improvement in learning efficiency has the potential to enhance autonomous systems across various applications, including robotics, finance, and large-scale industrial automation.

Overall, these studies underscore the broad applicability and impact of the IB method in diverse research areas. From adversarial robustness and reinforcement learning to plant metabolomics and robotic navigation, the IB principle continues to provide a powerful framework for optimizing information processing across various domains. The ability of IB-based models to extract the most salient features while eliminating redundancies has been instrumental in enhancing computational efficiency and decision-making accuracy. Future research in IB methodologies is likely to further refine and extend its applications, unlocking new possibilities in artificial intelligence, scientific research, and complex system optimization.

<table><caption>Table 16.4: Summary of Recent Contributions in Information Bottleneck Research</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Dardour et al. (2025) [1079]</td><td>Introduced a novel approach to enhance adversarial robustness in stochastic neural networks. By leveraging inter-separability and intra-concentration, their study demonstrated that IB constraints help neural networks learn more robust latent features, effectively mitigating adversarial perturbations.</td></tr><tr><td>Krinner et al. (2025) [1080]</td><td>Applied IB principles to reinforcement learning by designing state-space world models that accelerate learning efficiency. Their study showed that IB-based methods help an agent discard irrelevant environmental noise while retaining essential features, leading to improved exploration efficiency.</td></tr><tr><td>Yildirim et al. (2024) [1081]</td><td>Explored how IB constraints affect StyleGAN-based image editing. They demonstrated that GAN-based inversion techniques often suffer from excessive compression-induced detail loss, proposing refined inversion methods that better preserve fine-grained image features.</td></tr></tbody></table>

*Continued on next page*
<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Yang et al. (2025) [1082]</td><td>Developed a cognitive-load-aware activation mechanism for large language models (LLMs), improving efficiency by dynamically activating only the necessary model parameters. Their study used IB principles to retain relevant contextual representations while discarding redundant computations, reducing computational overhead.</td></tr><tr><td>Liu et al. (2025) [1083]</td><td>Incorporated IB principles in a structure-aware Vision Mamba network for crack segmentation in infrastructure. Their method efficiently filters out redundant spatial information, enhancing computational efficiency and segmentation accuracy, making it crucial for real-time applications in structural health monitoring.</td></tr><tr><td>Stierle and Valtere (2025) [1084]</td><td>Applied IB theory to medical innovation, examining how information bottlenecks in regulatory and patent frameworks slow down gene therapy advancements. Their work analyzed how such bottlenecks in medical research and policy impede technological progress.</td></tr><tr><td>Chen et al. (2025) [1085]</td><td>Applied IB concepts to quantum computing, particularly in optimizing construction supply chains. Their work demonstrated that quantum models integrated with IB techniques efficiently compress relevant data while filtering out extraneous information, improving decision-making processes.</td></tr><tr><td>Yuan et al. (2025) [1086]</td><td>Extended IB applications to plant metabolomics by proposing a novel feature selection approach that retains highly informative metabolite interactions while discarding non-essential data. This method improved interpretability in plant metabolic studies.</td></tr><tr><td>Dey et al. (2025) [1087]</td><td>Utilized IB principles in spatio-temporal prediction models for NDVI (Normalized Difference Vegetation Index), which is crucial for rice crop yield forecasting. Their IB-augmented neural network improved prediction accuracy by filtering out irrelevant environmental variables.</td></tr><tr><td>Li (2025) [1088]</td><td>Applied IB principles in robotic path planning, developing an optimized method for navigation path extraction in mobile robots. Their approach eliminated irrelevant environmental noise while preserving crucial navigational data, improving robotic movement efficiency.</td></tr></tbody></table>

### 16.1.4.3 Mathematical Analysis of Information Bottleneck (IB) method

The **Information Bottleneck (IB) method** is a highly rigorous information-theoretic framework that seeks to optimally compress an input variable *X* while retaining the most relevant information about an output variable *Y*. Mathematically, the IB method formulates the problem as one of finding a compressed representation *T* of *X*, such that *T* retains as much mutual information with *Y* as possible while minimizing the redundant information from *X*. This is achieved by solving an optimization problem that balances the competing objectives of compression and prediction
accuracy, fundamentally rooted in Shannon's information theory.

The IB method is expressed through the following constrained optimization problem:

$$ \min_{p(t|x)} I(X; T) - \beta I(T; Y), \qquad (16.15) $$

where:

* $I(X; T)$ is the mutual information between the input $X$ and the compressed representation $T$, which measures the amount of information retained about $X$ in $T$.

* $I(T; Y)$ is the mutual information between $T$ and the target variable $Y$, ensuring that the compressed representation remains useful for predicting $Y$.

* $\beta$ is a Lagrange multiplier that controls the trade-off between compression and prediction accuracy.

The mutual information terms are given by:

$$ I(X; T) = \sum_{x,t} p(x,t) \log \frac{p(x,t)}{p(x)p(t)} \qquad (16.16) $$

and

$$ I(T; Y) = \sum_{t,y} p(t,y) \log \frac{p(t,y)}{p(t)p(y)}. \qquad (16.17) $$

These quantities describe the dependencies between variables, and minimizing $I(X; T)$ ensures maximal compression while maximizing $I(T; Y)$ preserves relevant predictive information. To find the optimal encoding distribution $p(t|x)$, we introduce a **variational formulation** using Lagrange multipliers, leading to the following self-consistent equations:

$$ p(t|x) = \frac{p(t)}{Z(x, \beta)} \exp(-\beta D_{\text{KL}}(p(y|x) \| p(y|t))), \qquad (16.18) $$

where:

* $Z(x, \beta)$ is a normalization constant ensuring that $p(t|x)$ is a valid probability distribution.

* $D_{\text{KL}}(p(y|x) \| p(y|t))$ is the Kullback-Leibler (KL) divergence between the posterior distributions $p(y|x)$ and $p(y|t)$, ensuring that $T$ retains relevant information about $Y$.

By iterating the above equation along with the updates:

$$ p(t) = \sum_{x} p(x)p(t|x), \qquad (16.19) $$

$$ p(y|t) = \sum_{x} p(y|x)p(x|t), \qquad (16.20) $$

we can numerically solve for $p(t|x)$ in an iterative fashion until convergence. A crucial aspect of the IB method is the **information plane**, where solutions are analyzed in terms of the trade-off between $I(X; T)$ and $I(T; Y)$. The optimal trade-off curve is derived by maximizing:

$$ \mathcal{L}(p(t|x)) = I(T; Y) - \beta I(X; T), \qquad (16.21) $$
which provides a **Pareto-optimal frontier** representing the most efficient trade-offs between compression and predictive power. For multivariate Gaussian variables $X$ and $Y$, where $(X, Y)$ follows a joint Gaussian distribution:

$$ \begin{bmatrix} X \\ Y \end{bmatrix} \sim \mathcal{N} \left( \begin{bmatrix} \mu_X \\ \mu_Y \end{bmatrix}, \begin{bmatrix} \Sigma_{XX} & \Sigma_{XY} \\ \Sigma_{YX} & \Sigma_{YY} \end{bmatrix} \right), \qquad (16.22) $$

the IB solution can be expressed explicitly in terms of covariance matrices. The optimal bottleneck variable $T$ satisfies:

$$ \Sigma_{TT} = \Sigma_{XX} - \Sigma_{XY}\Sigma_{YY}^{-1}\Sigma_{YX}, \qquad (16.23) $$

where the optimal compression ratio is determined by the eigenvalues of the information-preserving covariance transformation. In modern applications, the IB principle has been extensively applied to deep neural networks (DNNs), where hidden layer representations $T$ are trained to maximize information retention about the target $Y$. The information-theoretic loss function:

$$ \mathcal{L}(\theta) = I(T_{\theta}; Y) - \beta I(X; T_{\theta}) \qquad (16.24) $$

is used in variational autoencoders (VAEs) and other deep learning models to enforce minimal sufficient representations.

In conclusion, The Information Bottleneck method provides a rigorous and mathematically principled approach to optimal data compression with maximal information retention. It is deeply rooted in Shannon's information theory and has extensive applications in signal processing, neural networks, and statistical learning. Its iterative updates, self-consistency equations, and variational derivations make it a powerful tool for understanding fundamental limits in machine learning and information processing.

## 16.1.5 Restricted Boltzmann machines (RBMs)

### 16.1.5.1 Literature Review of Restricted Boltzmann machines (RBMs)

Restricted Boltzmann Machines (RBMs) have played a fundamental role in the advancement of machine learning, particularly in the context of unsupervised learning, feature extraction, and probabilistic modeling. The origins of RBMs can be traced back to the work of Smolensky (1986)[1090], who introduced the concept of the Harmonium, a precursor to modern RBMs. This work laid the theoretical foundation for energy-based models and provided a framework for understanding how neural networks could learn representations based on statistical mechanics principles. Smolensky's formulation established the notion that a system could be described in terms of an energy function, where lower energy states correspond to more likely configurations of the model. The importance of this contribution is underscored by its influence on subsequent developments in probabilistic graphical models and deep learning architectures. Building upon these early theoretical insights, Hinton and Salakhutdinov (2006)[1040] demonstrated how RBMs could be stacked to form Deep Belief Networks (DBNs), enabling efficient unsupervised pretraining for deep neural networks. Their work showed that RBMs could be used for dimensionality reduction in a manner analogous to Principal Component Analysis (PCA), but with the added advantage of capturing complex, non-linear dependencies in data. The significance of this approach was evident in its ability to improve the training of deep networks, reducing overfitting and making it feasible to train deep architectures without the need for vast amounts of labeled data.

One of the critical challenges in training RBMs is the efficient computation of gradients for weight updates, given the intractability of exact maximum likelihood estimation. To address this issue, Carreira-Perpiñán and Hinton (2005)[1091] analyzed the Contrastive Divergence (CD) algorithm, a
popular method for approximating the likelihood gradient. Their study provided a rigorous exam-
ination of the convergence properties of CD and highlighted its strengths and limitations. While
CD allows for fast and efficient training of RBMs, they showed that it does not always lead to
unbiased estimates of the likelihood gradient, which can impact the learned representations. Hin-
ton (2012)[1092] later expanded on these ideas by providing a practical guide to training RBMs,
detailing essential hyperparameter selection strategies, initialization techniques, and empirical best
practices. His work served as an invaluable resource for researchers and practitioners aiming to im-
plement RBMs effectively, covering both theoretical and experimental considerations. In a broader
context, Fischer and Igel (2014)[1093] presented a comprehensive introduction to the training and
theoretical underpinnings of RBMs, consolidating knowledge from previous research into a struc-
tured and accessible form. Their work not only explained the fundamental mechanics of RBMs but
also explored various extensions and applications, making it an essential reference for those seeking
to understand both the theoretical and applied aspects of RBMs.

The versatility of RBMs extends beyond unsupervised learning, as demonstrated by Larochelle and Bengio (2008)[1094], who introduced a discriminative variant of RBMs specifically tailored for classification tasks. Their approach modified the standard RBM training objective to optimize for discriminative performance, demonstrating that RBMs could be used effectively for supervised learning tasks as well. This contribution was crucial in showcasing the adaptability of RBMs to different learning paradigms. In another important application, Salakhutdinov, Mnih, and Hinton (2007)[1095] utilized RBMs for collaborative filtering, where they modeled user-item interactions in recommender systems. Their study demonstrated that RBMs could outperform traditional approaches like matrix factorization in certain settings, particularly in handling sparse and high-dimensional data. This work provided a strong argument for the applicability of RBMs in real-world systems where interactions between entities need to be learned from limited observed data. The ability of RBMs to extract latent features from data proved useful in various domains, including recommendation systems, document modeling, and image processing.

Another significant direction in RBM research involved understanding their effectiveness in un-
supervised feature learning. Coates, Lee, and Ng (2011)[1096] conducted an extensive analysis of
single-layer neural networks, including RBMs, to assess their ability to learn meaningful represen-
tations from raw data. Their findings highlighted the potential of RBMs in learning hierarchical
feature representations and provided empirical evidence that RBMs could achieve competitive per-
formance compared to other feature-learning approaches. This study influenced subsequent research
in deep learning by emphasizing the importance of structured feature extraction from data. In a
different vein, Hinton and Salakhutdinov (2009)[1097] proposed the Replicated Softmax model, an
extension of RBMs designed to model word counts in documents. Their work bridged the gap
between RBMs and topic models, enabling RBMs to be applied to natural language processing
tasks. This development demonstrated the flexibility of RBMs in handling different types of data
distributions, further expanding their applicability beyond conventional structured data.

In recent years, RBM research has intersected with advancements in quantum computing, as
explored by Adachi and Henderson (2015)[1098], who investigated the application of quantum
annealing to the training of deep neural networks. Their study explored how quantum hardware
could potentially accelerate the learning process in RBMs by leveraging quantum parallelism. This
work opened new avenues for research at the intersection of quantum computing and machine
learning, suggesting that RBMs could benefit from novel optimization techniques unavailable to
classical computing paradigms. Overall, the contributions of these works collectively demonstrate
the breadth of RBM research, from theoretical foundations and training methodologies to applica-
tions in diverse domains such as recommender systems, natural language processing, and quantum
machine learning. The progression of RBMs from their early formulations to their modern applica-
tions underscores their significance as a foundational tool in the development of deep learning and probabilistic modeling frameworks.

<table><caption>Table 16.5: Summary of Contributions on Restricted Boltzmann Machines (RBMs)</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Smolensky (1986)[1090]</td><td>Introduced the concept of the Harmonium, providing the theoretical foundation for energy-based models and probabilistic representations in neural networks.</td></tr><tr><td>Hinton and Salakhutdinov (2006)[1040]</td><td>Demonstrated how RBMs could be stacked to form Deep Belief Networks (DBNs), enabling efficient unsupervised pretraining and improving deep learning architectures.</td></tr><tr><td>Carreira-Perpiñán and Hinton (2005)[1091]</td><td>Analyzed the Contrastive Divergence (CD) algorithm, providing insights into its convergence properties and limitations for RBM training.</td></tr><tr><td>Hinton (2012)[1092]</td><td>Provided a practical guide for training RBMs, detailing hyper-parameter tuning, initialization strategies, and best practices.</td></tr><tr><td>Fischer and Igel (2014)[1093]</td><td>Offered a comprehensive introduction to RBMs, covering theoretical foundations, training methodologies, and practical applications.</td></tr><tr><td>Larochelle and Bengio (2008)[1094]</td><td>Introduced a discriminative variant of RBMs tailored for classification tasks, demonstrating their adaptability to supervised learning.</td></tr><tr><td>Salakhutdinov, Mnih, and Hinton (2007)[1095]</td><td>Applied RBMs to collaborative filtering, showing their effectiveness in recommender systems by capturing latent user-item interactions.</td></tr><tr><td>Coates, Lee, and Ng (2011)[1096]</td><td>Analyzed RBMs for unsupervised feature learning, demonstrating their ability to extract hierarchical representations from raw data.</td></tr><tr><td>Salakhutdinov and Hinton (2009)[1097]</td><td>Proposed the Replicated Softmax model, extending RBMs for modeling word counts in natural language processing tasks.</td></tr><tr><td>Adachi and Henderson (2015)[1098]</td><td>Investigated the use of quantum annealing for RBM training, exploring potential acceleration of learning using quantum computing techniques.</td></tr></tbody></table>

### 16.1.5.2 Recent Literature Review of Restricted Boltzmann machines (RBMs)

Restricted Boltzmann Machines (RBMs) have been extensively studied and applied across various domains, leading to significant advancements in both theoretical understanding and practical implementations. One of the most notable works in this area is by Salloum, Nayal, and Mazzara (2024) [1099], who explored the comparative performance of classical RBMs and their quantum counterparts in MNIST classification. Their study highlights how quantum annealing techniques, particularly quantum-restricted Boltzmann machines, offer improved performance over classical models in specific scenarios, mainly due to their ability to explore the solution space more effectively. This research underscores the potential for quantum machine learning to outperform traditional neural networks, particularly in complex optimization problems where classical RBMs struggle due to their inherent limitations in representing certain distributions.

Building upon the foundational aspects of RBMs, Joudaki (2025) [1100] conducted a comprehensive review of their applications in human action recognition, particularly in combination with Deep Belief Networks (DBNs). This study systematically categorizes existing research and iden-
tifies key challenges, such as overfitting and slow convergence, while also discussing how RBMs facilitate hierarchical feature extraction. A complementary perspective is provided by Prat Pou, Romero, Martí, and Mazzanti (2025) [1101], who focus on improving the computational efficiency of RBMs in evaluating partition functions using annealed importance sampling. Their work is particularly relevant in statistical physics, where accurate estimation of partition functions is crucial for modeling spin systems and understanding phase transitions. They demonstrate that their proposed initialization method enhances the robustness of the sampling process, significantly reducing variance in the estimated probabilities.

Further theoretical advancements in RBMs were made by Decelle, Gómez, and Seoane (2025) [1102], who investigated the ability of RBMs to infer high-order dependencies in complex systems. Their work presents a framework for mapping RBMs onto higher-order interactions, particularly in domains such as protein interaction networks and spin glasses, where conventional machine learning models struggle to capture intricate relationships. In a related study, Savitha, Kannan, and Logeswaran (2025) [1103] integrate RBMs within DBNs for cardiovascular disease prediction, leveraging optimization techniques such as the Harris Hawks Search algorithm. Their research emphasizes the role of RBMs in medical diagnosis, showing that the extracted features from unsupervised pretraining significantly improve classification accuracy in deep learning pipelines. These contributions highlight the versatility of RBMs in both theoretical modeling and real-world applications.

Efforts to enhance the efficiency of RBM training and sampling have been explored by Béreux, Decelle, and Furtlehner (2025) [1104], who propose a novel training strategy that accelerates convergence without compromising generalization performance. Their method is particularly effective for large-scale learning problems where traditional contrastive divergence methods are computationally expensive. Thériault et. al. (2024) [1105] further examine the structured learning properties of RBMs in a teacher-student setting, demonstrating that incorporating structured priors enhances the network's ability to generalize beyond seen data. These studies collectively address the computational bottlenecks associated with RBM training, making them more viable for practical machine learning applications.

Another notable application of RBMs is in feature learning for non-traditional data types. Manimurugan, Karthikeyan, and Narmatha (2024) [1106] introduce a hybrid approach that combines Bi-LSTM networks with RBMs for underwater object detection, demonstrating how RBMs effectively capture spatial dependencies in sonar and optical imagery. Similarly, Hossain, Showkat Ara, and Han (2025) [1107] benchmark RBMs against classical and deep learning models for human activity recognition, finding that RBMs provide a unique advantage in extracting latent representations. Extending RBM applications to neuromorphic computing, Qin, Peng, Miao, Chen, Ouyang, and Yang (2025) [1108] integrate RBMs with magnetic tunnel junctions for enhanced magnetic anomaly detection. This interdisciplinary work bridges the gap between neuromorphic architectures and probabilistic learning models, demonstrating a path towards more energy-efficient and compact AI systems.

Collectively, these studies highlight the ongoing evolution of RBMs, from fundamental theoretical advances to diverse applications in physics, medicine, security, and beyond. The research spans improvements in training efficiency, inference capabilities, and integration with other deep learning techniques, demonstrating the continued relevance of RBMs in modern AI research. While challenges such as mode collapse and slow training persist, the integration of RBMs with quantum computing, neuromorphic architectures, and hybrid deep learning models presents promising directions for future work.
<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Salloum et al. (2024) [1099]</td><td>Compared classical RBMs with quantum-restricted Boltzmann machines for MNIST classification, demonstrating that quantum models exhibit superior performance in certain optimization scenarios.</td></tr><tr><td>Joudaki (2025) [1100]</td><td>Conducted a comprehensive literature review on RBMs and Deep Belief Networks (DBNs) for human action recognition, identifying key challenges such as overfitting and slow convergence.</td></tr><tr><td>Prat Pou et al. (2025) [1101]</td><td>Proposed an improved method for evaluating the partition function in RBMs using annealed importance sampling, which enhances accuracy in statistical physics applications.</td></tr><tr><td>Decelle et al. (2025) [1102]</td><td>Investigated the ability of RBMs to infer high-order dependencies in complex systems, particularly in protein interaction networks and spin glasses.</td></tr><tr><td>Savitha et al. (2025) [1103]</td><td>Integrated RBMs within DBNs for cardiovascular disease prediction, leveraging optimization techniques such as the Harris Hawks Search algorithm to improve diagnostic accuracy.</td></tr><tr><td>Béreux et al. (2025) [1104]</td><td>Developed an efficient training strategy for RBMs that accelerates convergence while maintaining strong generalization capabilities in large-scale machine learning problems.</td></tr><tr><td>Thériault et al. (2024) [1105]</td><td>Explored structured learning in RBMs within a teacher-student setting, demonstrating that incorporating structured priors enhances generalization beyond seen data.</td></tr><tr><td>Manimurugan et al. (2024) [1106]</td><td>Combined Bi-LSTM networks with RBMs for underwater object detection, showcasing the ability of RBMs to effectively capture spatial dependencies in sonar and optical imagery.</td></tr><tr><td>Hossain et al. (2025) [1107]</td><td>Benchmarked RBMs against classical and deep learning models for human activity recognition, highlighting their effectiveness in extracting latent features.</td></tr><tr><td>Qin et al. (2025) [1108]</td><td>Integrated RBMs with magnetic tunnel junctions for magnetic anomaly detection, demonstrating their potential in neuromorphic computing for energy-efficient AI systems.</td></tr></tbody></table>

### 16.1.5.3 Mathematical Analysis of Restricted Boltzmann machines (RBMs)

A **Restricted Boltzmann Machine (RBM)** is a generative stochastic artificial neural network that consists of a **visible layer** and a **hidden layer**, forming a **bipartite graph** with no intra-layer connections. This characteristic differentiates it from more general **Boltzmann Machines**, where visible-visible and hidden-hidden connections are allowed. Mathematically, an RBM models a **joint probability distribution** over a set of visible variables **v** and hidden variables **h** using an **energy-based approach**. The energy function associated with the RBM, which determines the probability of a given configuration, is given by:

$$E(\mathbf{v}, \mathbf{h}) = - \sum_{i} a_{i}v_{i} - \sum_{j} b_{j}h_{j} - \sum_{i,j} v_{i}W_{ij}h_{j} \quad (16.25)$$
where $v_i$ represents the state of the i-th visible unit, $h_j$ represents the state of the j-th hidden unit, $W_{ij}$ is the weight connecting visible unit i to hidden unit j, $a_i$ and $b_j$ are the biases of the visible and hidden units, respectively. The probability distribution of a visible-hidden configuration ($\mathbf{v}, \mathbf{h}$) is governed by the **Boltzmann distribution**, given by:

$$P(\mathbf{v}, \mathbf{h}) = \frac{e^{-E(\mathbf{v}, \mathbf{h})}}{Z} \qquad (16.26)$$

where Z is the **partition function** ensuring normalization:

$$Z = \sum_{\mathbf{v},\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})} \qquad (16.27)$$

Since the **hidden units are conditionally independent given the visible units**, the conditional probability distribution of a hidden unit given the visible units follows a **sigmoid activation function**:

$$P(h_j = 1 | \mathbf{v}) = \sigma \left( b_j + \sum_i W_{ij} v_i \right) = \frac{1}{1 + e^{-(b_j + \sum_i W_{ij} v_i)}} \qquad (16.28)$$

Similarly, the probability of a visible unit given the hidden units is:

$$P(v_i = 1 | \mathbf{h}) = \sigma \left( a_i + \sum_j W_{ij} h_j \right) = \frac{1}{1 + e^{-(a_i + \sum_j W_{ij} h_j)}} \qquad (16.29)$$

The marginal probability of a visible vector $\mathbf{v}$ is obtained by summing over all possible hidden states:

$$P(\mathbf{v}) = \sum_{\mathbf{h}} P(\mathbf{v}, \mathbf{h}) = \frac{1}{Z} \sum_{\mathbf{h}} e^{-E(\mathbf{v}, \mathbf{h})} \qquad (16.30)$$

By factoring out terms dependent only on $\mathbf{v}$, we define the **free energy function** as:

$$F(\mathbf{v}) = - \sum_{i} a_i v_i - \sum_{j} \log(1 + e^{b_j + \sum_i W_{ij} v_i}) \qquad (16.31)$$

which leads to:

$$P(\mathbf{v}) = \frac{e^{-F(\mathbf{v})}}{Z} \qquad (16.32)$$

RBMs are trained using **gradient descent** to minimize the negative log-likelihood:

$$\mathcal{L} = - \sum_{\mathbf{v} \in \text{training data}} \log P(\mathbf{v}) \qquad (16.33)$$

The gradient of this likelihood function with respect to the weight matrix $W_{ij}$ is given by:

$$\frac{\partial \mathcal{L}}{\partial W_{ij}} = \mathbb{E}_{\text{data}}[v_i h_j] - \mathbb{E}_{\text{model}}[v_i h_j] \qquad (16.34)$$

where $\mathbb{E}_{\text{data}}[v_i h_j]$ is the expectation over the training data, $\mathbb{E}_{\text{model}}[v_i h_j]$ is the expectation under the model distribution. The **weight updates** are then performed using:

$$\Delta W_{ij} = \eta (\mathbb{E}_{\text{data}}[v_i h_j] - \mathbb{E}_{\text{model}}[v_i h_j]) \qquad (16.35)$$

RBMs use **stochastic gradient descent (SGD)** for optimization. The weight updates in SGD follow:

$$W_{ij}^{(t+1)} = W_{ij}^{(t)} + \eta \left( v_i^{(\text{data})} h_j^{(\text{data})} - v_i^{(\text{model})} h_j^{(\text{model})} \right) \qquad (16.36)$$

RBMs serve as the **building blocks** for **Deep Belief Networks (DBNs)**, where multiple RBMs are **stacked** to form deep architectures. Each layer is pre-trained in an **unsupervised manner** using contrastive divergence before fine-tuning using **backpropagation**.
<figure><img src="image_169.png" /><figcaption>Figure 16.1: Diagram of a restricted Boltzmann machine</figcaption></figure>

Image Credit: By Qwertyus - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=22717044

Illustration of a restricted Boltzmann machine featuring three observable units and four concealed units, excluding bias units

16.1.6 Deep belief networks (DBNs)

16.1.6.1 Literature Review of Deep belief networks (DBNs)

Deep Belief Networks (DBNs) have played a transformative role in deep learning by enabling effi-
cient unsupervised pre-training and hierarchical feature extraction. One of the foundational works
in this area is by Hinton et al. (2006) [854], who introduced a fast learning algorithm for DBNs that
leverages a greedy layer-wise pre-training strategy using Restricted Boltzmann Machines (RBMs).
This work addressed the long-standing vanishing gradient problem in deep networks by initializing
weights in a way that preserves meaningful feature representations before fine-tuning with super-
vised learning. Their contribution established DBNs as a key component of early deep learning
architectures and laid the groundwork for further explorations into deep generative models. Lee et
al. (2009) [1109] extended the standard DBN framework by incorporating convolutional structures,
leading to the development of Convolutional Deep Belief Networks (CDBNs). By introducing local
receptive fields and weight sharing, CDBNs enabled the automatic discovery of spatial hierarchies
in data, making them particularly suitable for image and speech processing applications. This
work demonstrated that DBNs could be adapted to structured data representations, enhancing
their scalability and generalization capability.
In the domain of speech recognition, Mohamed et al. (2012) [1111] pioneered the application of DBNs for acoustic modeling, demonstrating that these networks could effectively capture complex audio patterns. Their work provided empirical evidence that DBN-based models significantly outperformed traditional Gaussian Mixture Models (GMMs) when used in conjunction with Hidden Markov Models (HMMs) for automatic speech recognition. This breakthrough accelerated the adoption of deep learning in the field of speech processing and motivated subsequent research into deep neural network-based acoustic modeling. Similarly, Zhang and Zhao (2017) [1113] explored the use of DBNs for fault diagnosis in chemical processes, highlighting their ability to model intricate dependencies within multivariate industrial datasets. Their work demonstrated that DBNs could learn compact feature representations from noisy sensor data, leading to improved fault detection accuracy. By applying DBNs to real-world industrial systems, they provided compelling evidence of their practical utility in process monitoring and control.

Further expanding the use of DBNs in fault diagnostics, Peng et al. (2019) [1112] introduced a health indicator construction framework based on DBNs for bearing fault diagnosis. Their approach enabled the automatic extraction of degradation features from vibration signals, allowing for early detection of mechanical failures. The health indicator developed in their study demonstrated superior predictive performance compared to traditional statistical methods, underscoring the power of DBNs in prognostics and health management applications. Zhang et al. (2018) [1115] extended the use of DBNs to the medical field by integrating them with feature selection and extraction methods for predicting clinical outcomes in lung cancer patients. Their study illustrated that DBNs could capture complex interactions between clinical variables, improving the interpretability and accuracy of predictive models in oncology. By leveraging deep learning for medical prognosis, this work demonstrated the potential of DBNs to advance personalized healthcare and decision support systems.

Zhong et. al. (2017) [1118] examined the problem of image classification with limited training data and demonstrated that DBNs could learn meaningful feature representations even in data-scarce settings. Their work emphasized the ability of DBNs to leverage unsupervised pre-training, enabling them to generalize well even when labeled data is insufficient. This contribution reinforced the importance of DBNs in applications where obtaining large-scale labeled datasets is challenging. Financial time series analysis is another domain where DBNs have demonstrated utility. Liu (2018) [1114] proposed a hybrid prediction model that combined DBNs with the Autoregressive Integrated Moving Average (ARIMA) model for stock trend forecasting. Their approach leveraged the pattern recognition capabilities of DBNs along with the time-series forecasting strengths of ARIMA, resulting in improved predictive performance for financial markets. This work showcased the effectiveness of deep learning in modeling complex temporal dependencies and provided insights into how hybrid models could enhance financial decision-making. Jormakka and Ghosh (2021) [1452] introduce a novel method for solving certain stochastic processes using generating functions, derive a general theorem for generating functions of a particular type, and apply this method to obtain polynomial-time algorithms for specific partitions; they further generalize the approach to a broader class of linear transformations and utilize it to develop a theoretical polynomial-time algorithm for the knapsack problem. Ghosh (2021) [1465] provides an accessible exploration of the Arithmetic Mean-Geometric Mean (AM-GM) inequality, offering both elementary and advanced proofs—including an elegant derivation using the exponential function—and demonstrates its applications through classical inequalities such as Nesbitt’s inequality, thereby enhancing understanding of fundamental inequalities in mathematics. Cox and Ghosh (2020) [1479] extend the study of the Collatz conjecture by introducing the concept of generalized dead limbs and analyzing $3n+c$ cycles with attachment points, demonstrating that all known cycles for $|c| < 53$ lie within these generalized dead limbs, and suggesting that this containment is a necessary condition for the existence
of such cycles. Abhyankar et. al. (2002) [1490] provided a comprehensive overview of the Zeta function, detailing its historical development, key properties, and applications in number theory, and offered an accessible introduction to its significance in modern mathematics. Finally, Hoang and Kang (2018) [1116] developed a novel fault diagnosis framework by integrating DBNs with Dempster-Shafer evidence theory. Their study highlighted how DBNs could serve as a powerful feature extractor while Dempster-Shafer theory facilitated the fusion of information from multiple sources, enhancing fault detection accuracy. This interdisciplinary approach demonstrated the potential of combining deep learning with probabilistic reasoning for more reliable fault diagnosis in engineering systems.

Collectively, these studies illustrate the broad impact of DBNs across multiple domains, including speech recognition, industrial fault diagnosis, medical prognosis, uncertainty quantification, and financial prediction. The ability of DBNs to learn hierarchical representations from high-dimensional data has established them as a powerful tool for tackling complex machine learning problems. The advancements made in adapting DBNs to convolutional architectures, hybrid models, and ensemble methods have further expanded their applicability, ensuring their continued relevance in deep learning research. As interest in deep generative models and representation learning grows, DBNs continue to serve as a foundational framework for understanding and developing more sophisticated deep learning architectures.

<table><caption>Table 16.7: Summary of Contributions on Deep Belief Networks (DBNs)</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Hinton et al. (2006) [854]</td><td>Introduced a fast learning algorithm for DBNs using a greedy layer-wise pre-training strategy based on Restricted Boltzmann Machines (RBMs). Addressed the vanishing gradient problem and established DBNs as foundational deep learning architectures.</td></tr><tr><td>Lee et al. (2009) [1109]</td><td>Developed Convolutional Deep Belief Networks (CDBNs) by incorporating convolutional structures into DBNs, introducing local receptive fields and weight sharing for improved scalability in image and speech processing.</td></tr><tr><td>Mohamed et al. (2012) [1111]</td><td>Pioneered the application of DBNs for acoustic modeling in speech recognition, demonstrating superior performance over traditional Gaussian Mixture Models (GMMs) in conjunction with Hidden Markov Models (HMMs).</td></tr><tr><td>Zhang and Zhao (2017) [1113]</td><td>Applied DBNs for fault diagnosis in chemical processes, showing that DBNs effectively model dependencies in multivariate datasets and enhance fault detection accuracy.</td></tr><tr><td>Peng et al. (2019) [1112]</td><td>Developed a DBN-based health indicator construction framework for bearing fault diagnosis, enabling automatic extraction of degradation features from vibration signals for early failure detection.</td></tr><tr><td>Zhang et al. (2018) [1115]</td><td>Integrated DBNs with feature selection methods for predicting clinical outcomes in lung cancer patients, enhancing predictive accuracy and interpretability in medical prognosis.</td></tr></tbody></table>

*Continued on next page*
*Continued from previous page*

<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Zhong et. al. (2017) [1118]</td><td>Demonstrated that DBNs could learn meaningful representations even with limited training data, reinforcing their utility in scenarios with scarce labeled datasets.</td></tr><tr><td>Liu (2018) [1114]</td><td>Combined DBNs with the Autoregressive Integrated Moving Average (ARIMA) model for stock trend forecasting, leveraging DBNs' pattern recognition capabilities with ARIMA's time-series forecasting strengths.</td></tr><tr><td>Hoang and Kang (2018) [1116]</td><td>Developed a novel fault diagnosis framework by integrating DBNs with Dempster-Shafer evidence theory, enhancing fault detection accuracy through probabilistic reasoning.</td></tr></tbody></table>

### 16.1.6.2 Recent Literature Review of Deep belief networks (DBNs)

Deep Belief Networks (DBNs) have emerged as a powerful deep learning architecture capable of hierarchical feature learning and unsupervised pre-training. These networks, based on a stack of Restricted Boltzmann Machines (RBMs), have been widely used in various fields, ranging from human activity recognition and medical diagnosis to cybersecurity and agricultural applications. Joudaki (2025) [1100] provides an extensive literature review on the theoretical underpinnings of DBNs and RBMs, emphasizing their role in human action recognition. The study highlights how DBNs are particularly suited for tasks requiring high-dimensional feature extraction and learning temporal dependencies, outperforming traditional machine learning models. The hierarchical representation learned by DBNs allows them to capture complex patterns in human gestures and postures, making them ideal for applications such as motion tracking and gesture-based interface design.

Alzughaibi (2025) [1119] presents an innovative application of DBNs in pest detection, where they are integrated with a modified artificial hummingbird algorithm. The research demonstrates how deep learning-based pattern recognition models can significantly enhance the accuracy of pest classification in agricultural settings. The model is trained on large datasets of pest images, leveraging the hierarchical feature extraction capabilities of DBNs to distinguish between different species effectively. Similarly, Savitha et al. (2025) [1103] apply DBNs to cardiovascular disease prediction by integrating them with the Harris Hawks Search optimization algorithm. This approach optimizes feature selection and classification accuracy, showing that DBNs can be successfully adapted for medical diagnosis by leveraging their deep hierarchical structure. The study underscores the potential of DBNs in clinical decision support systems, where accurate and timely diagnoses are crucial.

The intrinsic hierarchical structure of DBNs has also been studied by Tausani et al. (2025) [1120], who investigate their top-down inference capabilities compared to other deep generative models. Their research explores how DBNs can simulate human-like cognition and learning, making them suitable for applications in artificial intelligence and cognitive computing. By analyzing the internal feature representations of DBNs, the study provides insights into their interpretability and efficiency in generative tasks. Kumar and Ravi (2025) [1121] further contribute to the field by introducing XDATE, an explainable deep learning framework that combines DBNs with auto-encoders. The study addresses the issue of interpretability in deep learning by employing the Garson Algorithm to improve feature attribution, demonstrating that DBNs can achieve a balance between accuracy and explainability in classification tasks.
In the field of medical image analysis, Alhajlah (2024) [1122] applies DBNs for automated lesion detection in gastrointestinal endoscopic images. The research integrates DBNs with a genetic algorithm-based segmentation technique, significantly improving diagnostic precision. By leveraging the feature extraction capabilities of DBNs, the proposed system outperforms conventional image processing techniques, reducing false positives and improving lesion classification accuracy. Hossain et al. (2025) [1107] evaluate the performance of DBNs in human activity recognition, benchmarking them against various classical and deep learning models. Their study finds that DBNs, despite being unsupervised in their initial training phase, can achieve competitive accuracy on small and medium-sized datasets, demonstrating their robustness and adaptability.

The application of DBNs extends to cybersecurity, where Pavithra et al. (2025) [1123] develop a hybrid RNN-DBN model for detecting IoT attacks. This approach captures temporal dependencies in network traffic, allowing for effective anomaly detection and threat mitigation. The fusion of DBNs with recurrent networks enables better sequence modeling, making them highly effective in cybersecurity applications. In a similar vein, Bhadane and Verma (2024) [1124] explore the role of DBNs in personality trait classification, comparing their performance with CNNs and RNNs. Their study highlights the advantages of DBNs in handling high-dimensional psychological datasets, where deep hierarchical structures enable the capture of complex personality-related patterns. Lastly, Keivanimehr and Akbari (2025) [1125] examine how DBNs can be applied in edge computing for cardiovascular disease monitoring. Their research discusses the feasibility of deploying DBN-based models in TinyML environments, where computational efficiency and real-time processing are critical.

Collectively, these studies demonstrate the versatility and efficacy of DBNs in various domains, from healthcare and cybersecurity to agricultural automation and cognitive computing. The hierarchical feature learning capabilities of DBNs, combined with their ability to leverage unsupervised pre-training, make them suitable for a wide range of complex tasks. Future research is likely to explore further hybrid architectures that integrate DBNs with more advanced deep learning models, enhancing their adaptability and efficiency in real-world applications.

<table><caption>Table 16.8: Summary of Contributions</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Joudaki (2025) [1100]</td><td>Provides an extensive literature review on the theoretical foundations of DBNs and RBMs, emphasizing their role in human action recognition. Demonstrates their effectiveness in capturing complex patterns in human gestures and postures for applications such as motion tracking and gesture-based interface design.</td></tr><tr><td>Alzughaibi (2025) [1119]</td><td>Applies DBNs in pest detection, integrating them with a modified artificial hummingbird algorithm. Enhances pest classification accuracy using deep hierarchical feature extraction on large image datasets.</td></tr><tr><td>Savitha et al. (2025) [1103]</td><td>Employs DBNs for cardiovascular disease prediction, integrating them with the Harris Hawks Search optimization algorithm. Demonstrates improved feature selection and classification accuracy in medical diagnosis applications.</td></tr></tbody></table>

(Continued on next page)
(Continued from previous page)

<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Tausani et al. (2025) [1120]</td><td>Investigates the top-down inference capabilities of DBNs compared to other deep generative models. Explores their interpretability and efficiency in artificial intelligence and cognitive computing tasks.</td></tr><tr><td>Kumar and Ravi (2025) [1121]</td><td>Introduces XDATE, an explainable deep learning framework that combines DBNs with auto-encoders. Uses the Garson Algorithm to enhance feature attribution, balancing accuracy and interpretability in classification tasks.</td></tr><tr><td>Alhajlah (2024) [1122]</td><td>Applies DBNs in medical image analysis for automated lesion detection in gastrointestinal endoscopic images. Integrates DBNs with a genetic algorithm-based segmentation technique to enhance diagnostic precision and reduce false positives.</td></tr><tr><td>Hossain et al. (2025) [1107]</td><td>Benchmarks DBNs against classical and deep learning models for human activity recognition. Demonstrates DBNs' robustness and adaptability, particularly in small and medium-sized datasets.</td></tr><tr><td>Pavithra et al. (2025) [1123]</td><td>Develops a hybrid RNN-DBN model for IoT attack detection, capturing temporal dependencies in network traffic for anomaly detection and threat mitigation.</td></tr><tr><td>Bhadane and Verma (2024) [1124]</td><td>Explores DBNs for personality trait classification, comparing their performance with CNNs and RNNs. Highlights the advantages of DBNs in processing high-dimensional psychological datasets.</td></tr><tr><td>Keivanimehr and Akbari (2025) [1125]</td><td>Investigates DBNs for edge computing applications in cardiovascular disease monitoring. Discusses feasibility in TinyML environments for computational efficiency and real-time processing.</td></tr></tbody></table>

### 16.1.6.3 Mathematical Analysis of Deep belief networks (DBNs)

A Deep Belief Network (DBN) is a generative graphical model composed of multiple layers of stochastic latent variables, typically represented as **Restricted Boltzmann Machines (RBMs)** stacked hierarchically. The fundamental structure of a DBN is built upon the principles of **probabilistic modeling**, **unsupervised pretraining**, and **layer-wise greedy learning** to form a **deep hierarchical representation** of data. Mathematically, a DBN is a composition of multiple RBMs, each trained independently in a bottom-up fashion, followed by fine-tuning using backpropagation when supervised learning is required. The probabilistic nature of DBNs allows them to model complex **joint probability distributions** over observed and latent variables, making them highly effective for feature extraction, dimensionality reduction, and generative modeling.

Let $\mathbf{x} \in \mathbb{R}^d$ denote the input data vector, where $d$ is the dimensionality of the input space. A DBN consists of **multiple layers of hidden variables $\mathbf{h}^{(l)}$** connected in a directed fashion in the upper layers and undirected in the lower layers. The network defines a **joint probability distribution** over the visible units $\mathbf{x}$ and hidden units $\mathbf{h}^{(1)}, \mathbf{h}^{(2)}, \dots, \mathbf{h}^{(L)}$ as follows:

$$P(\mathbf{x}, \mathbf{h}^{(1)}, \mathbf{h}^{(2)}, \dots, \mathbf{h}^{(L)}) = P(\mathbf{x}|\mathbf{h}^{(1)})P(\mathbf{h}^{(1)}|\mathbf{h}^{(2)})\cdots P(\mathbf{h}^{(L-1)}|\mathbf{h}^{(L)})P(\mathbf{h}^{(L)}) \quad (16.37)$$

where:
1. $P(\mathbf{x}|\mathbf{h}^{(1)})$ represents the conditional distribution of the visible layer given the first hidden layer.

2. $P(\mathbf{h}^{(l-1)}|\mathbf{h}^{(l)})$ represents the conditional dependency between consecutive layers.

3. $P(\mathbf{h}^{(L)})$ represents the top-layer prior, which is modeled as a Restricted Boltzmann Machine (RBM).

<figure><img src="image_170.png" /><figcaption>Figure 16.2: Diagram of a Deep belief network</figcaption></figure>

Image Credit: By Qwertyus - Own work, CC0, https://commons.wikimedia.org/w/index.php?curid=30324766

Diagrammatic representation of a deep belief network, where the arrows indicate directed connections within the corresponding graphical model

## Restricted Boltzmann Machine (RBM) as a Building Block

Each RBM consists of a layer of **visible units** $\mathbf{v}$ and a layer of **hidden units** $\mathbf{h}$, with energy function given by:

$$E(\mathbf{v}, \mathbf{h}) = - \sum_{i} b_{i}v_{i} - \sum_{j} c_{j}h_{j} - \sum_{i,j} v_{i}W_{ij}h_{j} \qquad (16.38)$$

where $\mathbf{v} \in \{0, 1\}^d$ are the binary visible units, $\mathbf{h} \in \{0, 1\}^m$ are the binary hidden units, $W_{ij}$ is the **weight matrix** connecting visible and hidden units, $b_i$ and $c_j$ are the **bias terms** for visible and hidden units, respectively. The probability distribution over $(\mathbf{v}, \mathbf{h})$ is given by the **Boltzmann distribution**:

$$P(\mathbf{v}, \mathbf{h}) = \frac{1}{Z}e^{-E(\mathbf{v}, \mathbf{h})} \qquad (16.39)$$

where $Z$ is the **partition function**:

$$Z = \sum_{\mathbf{v},\mathbf{h}} e^{-E(\mathbf{v},\mathbf{h})} \qquad (16.40)$$

Since **exact inference** in RBMs is intractable due to the exponential summation in $Z$, we use **Contrastive Divergence (CD)** to approximate the gradient:

$$\frac{\partial \log P(\mathbf{v})}{\partial W_{ij}} \approx \mathbb{E}_{\text{data}}[v_i h_j] - \mathbb{E}_{\text{model}}[v_i h_j] \qquad (16.41)$$
## Layer-wise Pretraining of DBNs

DBNs are trained using a **layer-wise greedy learning algorithm**, which first trains each RBM independently and then stacks them hierarchically. Given an input dataset $\{\mathbf{x}_n\}_{n=1}^N$, the pretraining procedure follows:

1. Train the first RBM with input $\mathbf{x}$ to obtain hidden activations:

$$P(h_j^{(1)} = 1 | \mathbf{x}) = \sigma \left( \sum_i W_{ij}^{(1)} x_i + c_j^{(1)} \right) \qquad (16.42)$$

2. Use hidden activations as input for training the second RBM:

$$P(h_j^{(2)} = 1 | \mathbf{h}^{(1)}) = \sigma \left( \sum_i W_{ij}^{(2)} h_i^{(1)} + c_j^{(2)} \right) \qquad (16.43)$$

## Fine-Tuning with Backpropagation

Once pretraining is complete, DBNs can be **fine-tuned** using **backpropagation** if labeled data is available. A cost function such as **cross-entropy loss** is used:

$$\mathcal{L} = - \sum_{n=1}^{N} \sum_{k=1}^{K} y_k^{(n)} \log \hat{y}_k^{(n)} \qquad (16.44)$$

The gradients are computed using:

$$\frac{\partial \mathcal{L}}{\partial W_{ij}^{(l)}} = \frac{\partial \mathcal{L}}{\partial h_j^{(l)}} \frac{\partial h_j^{(l)}}{\partial W_{ij}^{(l)}} \qquad (16.45)$$

Deep Belief Networks (DBNs) serve as **powerful generative models** capable of **capturing hierarchical representations** of data through layer-wise pretraining of RBMs.

### 16.1.7 t-distributed stochastic neighbor embedding (t-SNE)

#### 16.1.7.1 Literature Review of t-distributed stochastic neighbor embedding

T-distributed Stochastic Neighbor Embedding (t-SNE) has been widely studied and refined since its introduction, with various works expanding its theoretical foundations, applications, and practical optimizations. The foundational work of van der Maaten and Hinton (2008) [1042] introduced t-SNE as an improvement over Stochastic Neighbor Embedding (SNE), primarily by incorporating a symmetric probability distribution in the low-dimensional space and replacing the Gaussian-based similarity measure with a heavy-tailed Student's t-distribution to address the crowding problem. This innovation enabled better preservation of both local and global structures when mapping high-dimensional data into lower dimensions. The study established the mathematical formulation of t-SNE by minimizing the Kullback-Leibler divergence between joint probability distributions of data points in high and low dimensions, leading to a widely adopted visualization tool in machine learning and data science.

Several studies have analyzed and improved the robustness of t-SNE, particularly concerning its application to large datasets and biological data visualization. Kobak and Berens (2019) [1126] addressed the challenges in single-cell transcriptomics by developing "opt-SNE," an automated
parameter selection framework that fine-tunes the perplexity and learning rate to enhance visual-
ization reliability. Their study demonstrated that proper parameter tuning significantly impacts
the quality of embeddings, thereby reducing the risk of misleading interpretations. Similarly, Belk-
ina et al. (2019) [1127] proposed an automated approach for optimizing t-SNE parameters for large
datasets, emphasizing the importance of reproducibility and interpretability in real-world applica-
tions. Their work provided empirical evidence that parameter selection can drastically affect the
clustering structure in t-SNE visualizations, necessitating automated approaches for better stan-
dardization.

Beyond empirical studies, some researchers have sought to establish a more rigorous theoretical understanding of t-SNE's behavior in clustering and manifold learning. Linderman and Steinerberger (2019) [1128] provided a mathematical proof that t-SNE effectively recovers well-separated clusters under certain conditions, offering a theoretical foundation for why t-SNE performs well in clustering applications despite not being explicitly designed for that purpose. Their work bridged the gap between empirical success and mathematical justification, thereby enhancing the credibility of t-SNE in scientific applications. Amorim and Mirkin (2012) [1129] extended this discussion by exploring the role of distance metrics in clustering and feature weighting, providing an alternative approach that incorporated Minkowski metrics and anomaly detection within the t-SNE-reduced space. Their work showed that selecting an appropriate distance function could further refine the clustering properties of dimensionality reduction methods, leading to improved segmentation of data points. Additionally, Wattenberg et al. (2016) [1130] provided a detailed discussion on how to use t-SNE effectively, emphasizing the interpretational pitfalls and practical considerations when applying t-SNE to real-world data. Their work highlighted the limitations of t-SNE in preserving global structure and warned against misinterpreting distance relationships in the lower-dimensional space.

In response to computational challenges, Pezzotti et al. (2016) [1131] proposed an approximation method that enabled real-time, user-steerable t-SNE for progressive visual analytics. Their work significantly reduced the computational burden of t-SNE by introducing an interactive framework that allowed users to refine embeddings dynamically, making t-SNE more practical for large-scale applications. Similarly, Kobak and Linderman (2021) [1132] investigated the importance of initialization strategies for preserving global data structures in both t-SNE and UMAP embeddings. Their findings demonstrated that different initialization schemes can lead to drastically different embeddings, underscoring the need for careful selection of initialization methods to ensure meaningful visualizations.

Finally, alternative approaches to t-SNE have been explored, with studies comparing its perfor-
mance against other dimensionality reduction techniques. Becht et al. (2019) [1133] introduced
Uniform Manifold Approximation and Projection (UMAP) as an alternative to t-SNE, arguing
that UMAP offers improved scalability and better preservation of global structures. Their study
provided empirical comparisons between UMAP and t-SNE, demonstrating that UMAP performs
comparably in clustering while requiring significantly less computational time. Likewise, Moon
et al. (2019) [1134] proposed PHATE (Potential of Heat-diffusion for Affinity-based Transition
Embedding), which was designed to capture both local and global structures more effectively than
t-SNE. Their study showcased PHATE's ability to preserve continuous trajectories in biological
data, highlighting its advantages in studying cell differentiation processes. Collectively, these stud-
ies illustrate the ongoing efforts to refine and expand upon t-SNE, both theoretically and practically,
ensuring its continued relevance in high-dimensional data visualization.
<table><caption>Table 16.9: Summary of Contributions in t-SNE Literature Review</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>van der Maaten and Hinton (2008) [1042]</td><td>Introduced t-SNE as an extension of Stochastic Neighbor Embedding (SNE) by incorporating a Student's t-distribution to address the crowding problem. This modification improved the preservation of both local and global structures and formulated t-SNE as an optimization problem minimizing Kullback-Leibler divergence.</td></tr><tr><td>Kobak and Berens (2019) [1126]</td><td>Developed "opt-SNE," an automated parameter selection framework for fine-tuning perplexity and learning rate in t-SNE. Their work demonstrated that proper parameter selection significantly affects embedding quality and reduces misleading visualizations in single-cell transcriptomics.</td></tr><tr><td>Belkina et al. (2019) [1127]</td><td>Proposed an optimized pipeline for selecting t-SNE parameters, focusing on reproducibility and interpretability in large-scale biological datasets. Their work emphasized the importance of standardizing parameter selection to improve clustering outcomes in t-SNE applications.</td></tr><tr><td>Linderman and Steinerberger (2019) [1128]</td><td>Provided a mathematical proof that t-SNE reliably recovers well-separated clusters under specific conditions, bridging the gap between empirical observations and theoretical guarantees in clustering applications.</td></tr><tr><td>Amorim and Mirkin (2012) [1129]</td><td>Investigated the role of distance metrics in t-SNE clustering, proposing the use of Minkowski metrics and feature weighting to refine cluster separation. Their work highlighted how different distance functions influence embedding outcomes.</td></tr><tr><td>Wattenberg et al. (2016) [1130]</td><td>Analyzed interpretational pitfalls in t-SNE, cautioning against misinterpretation of distances in low-dimensional embeddings. Their study outlined best practices for applying t-SNE in real-world data visualization tasks.</td></tr><tr><td>Pezzotti et al. (2016) [1131]</td><td>Developed a real-time, user-steerable t-SNE method that enabled progressive visual analytics. Their approach reduced computational overhead and allowed for interactive exploration of embeddings in large-scale applications.</td></tr><tr><td>Kobak and Linderman (2021) [1132]</td><td>Investigated initialization strategies for t-SNE and UMAP, demonstrating that different initialization schemes lead to vastly different embeddings. Their study emphasized the importance of careful initialization to preserve global data structures.</td></tr><tr><td>Becht et al. (2019) [1133]</td><td>Introduced UMAP as an alternative to t-SNE, showing that UMAP provides improved scalability and better preservation of global structures while requiring significantly less computational time.</td></tr></tbody></table>

*Continued on next page*
<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Moon et al. (2019) [1134]</td><td>Proposed PHATE as a dimensionality reduction technique designed to capture both local and global structures more effectively than t-SNE. Their study demonstrated PHATE's superiority in visualizing continuous biological trajectories, such as cell differentiation processes.</td></tr></tbody></table>

### 16.1.7.2 Recent Literature Review of t-distributed stochastic neighbor embedding

t-Distributed Stochastic Neighbor Embedding (t-SNE) has emerged as a powerful tool in high-dimensional data visualization, finding extensive applications across various domains, including biomedical sciences, material science, deep learning, and geophysics. Rivera and Deniega (2025) [1135] demonstrated the efficacy of t-SNE in automating flow cytometry gating, where the method was combined with clustering techniques such as DBSCAN and PCA. Their study highlighted how t-SNE facilitates better separation and classification of cell populations, ultimately improving computational efficiency in biomedical applications. Similarly, Chang (2025) [1136] provided a rigorous survey of dimensionality reduction techniques, discussing the advantages and limitations of t-SNE compared to traditional methods like PCA and newer alternatives like UMAP. The study outlined the scenarios where t-SNE performs optimally, particularly in non-linear data distributions, making it an essential reference for researchers selecting appropriate dimensionality reduction techniques.

In the domain of defect detection in industrial settings, Chern et al. (2025) [1137] utilized t-SNE for visualizing metal defect classification in deep learning models, improving interpretability in YOLO-based defect detection frameworks. Their study showed how t-SNE enhances the understanding of feature similarities among defect classes, leading to more refined model improvements. A similar approach was adopted by Li et al. (2025) [1138] in the field of food safety, where t-SNE was applied to olfactory sensor data to analyze aflatoxin B1 contamination in wheat. By employing t-SNE for dimensionality reduction, the researchers effectively visualized sensor response variations, demonstrating the method's utility in complex chemical data analysis. In another domain, Singh and Singh (2025) [1139] developed a hybrid approach for medical image retrieval by integrating deep learning features with t-SNE. Their method showed that t-SNE enhances feature clustering, thereby improving the accuracy and efficiency of gastric image retrieval systems, which is particularly beneficial in computer-aided diagnosis applications.

Sun et al. (2025) [1142] explored the application of t-SNE in biomechanics, specifically in detecting muscle fatigue during lower limb isometric contraction tasks. Their study employed t-SNE to reduce the dimensionality of electromyography (EMG) data before applying machine learning-based classification, leading to improved performance in fatigue detection models. Meanwhile, Su et al. (2025) [1143] incorporated t-SNE into seismic fragility analysis for earth-rock dams, integrating it within a deep residual shrinkage network to enhance predictive accuracy. Their research illustrated how t-SNE, when combined with deep learning techniques, can refine the interpretation of heterogeneous material behavior under seismic loading conditions. In the biomedical domain, Yousif and Al-Sarray (2025) [1144] integrated t-SNE with spectral clustering via convex optimization to enhance breast cancer gene classification. Their work established that this combination yields superior clustering performance compared to conventional methods, contributing significantly to genomics research and precision medicine.

Park et al. (2025) [1145] assessed the clinical applicability of t-SNE in flow cytometry, specifically for hematologic malignancies. Their study compared t-SNE with UMAP in reducing high-dimensional cytometric data and concluded that t-SNE provides superior local structure preservation, which is
crucial for identifying rare cell populations in clinical diagnostics. Qiao et al. (2025) [1146] used t-SNE to analyze cancer-associated fibroblasts (CAFs) in pancreatic ductal adenocarcinoma patients, identifying subclusters that exhibited distinct inflammatory gene expression patterns. This study underscored the value of t-SNE in uncovering hidden patterns in complex biological datasets, aiding in the stratification of cancer patients based on gene expression profiles. Furthermore, t-SNE has been utilized in aerospace engineering, as demonstrated by Su et al. (2025) [1143], who employed the technique in damage quantification for aircraft structures. Their study integrated t-SNE into an end-to-end deep learning framework to enhance defect localization, demonstrating how the method can improve structural health monitoring systems.

Overall, these studies collectively emphasize the versatility of t-SNE across diverse fields, ranging from biomedical sciences and geophysics to industrial defect detection and aerospace engineering. While t-SNE's strength lies in its ability to preserve local structures in high-dimensional data, these studies also highlight its limitations, such as high computational cost and sensitivity to parameter tuning. Future research should focus on optimizing t-SNE's efficiency while maintaining its robust visualization capabilities. By integrating t-SNE with deep learning architectures and hybrid clustering techniques, researchers can further expand its applications in fields requiring advanced data analysis and interpretation. The increasing adoption of t-SNE across disciplines highlights its ongoing relevance, making it a crucial tool for researchers handling complex, high-dimensional datasets.

<table><caption>Table 16.10: Summary of Recent Contributions in t-SNE Literature Review</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Rivera and Deniega (2025) [1135]</td><td>Demonstrated the efficacy of t-SNE in automating flow cytometry gating, improving cell population classification using clustering techniques such as DBSCAN and PCA.</td></tr><tr><td>Chang (2025) [1136]</td><td>Provided a survey of dimensionality reduction techniques, analyzing the strengths and weaknesses of t-SNE compared to PCA and UMAP, emphasizing its performance on non-linear data distributions.</td></tr><tr><td>Chern et al. (2025) [1137]</td><td>Applied t-SNE for visualizing metal defect classification in YOLO-based deep learning models, enhancing interpretability in industrial defect detection.</td></tr><tr><td>Li et al. (2025) [1138]</td><td>Utilized t-SNE for olfactory sensor data analysis in detecting aflatoxin B1 contamination in wheat, demonstrating its utility in chemical data visualization.</td></tr><tr><td>Singh and Singh (2025) [1139]</td><td>Developed a hybrid medical image retrieval approach by integrating deep learning features with t-SNE, improving clustering and accuracy in gastric image retrieval systems.</td></tr><tr><td>Sun et al. (2025) [1142]</td><td>Investigated t-SNE's application in biomechanics, reducing the dimensionality of electromyography (EMG) data for improved muscle fatigue classification.</td></tr><tr><td>Su et al. (2025) [1143]</td><td>Incorporated t-SNE in seismic fragility analysis of earth-rock dams, enhancing predictive accuracy when integrated into a deep residual shrinkage network.</td></tr></tbody></table>

*Continued on next page*
<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Yousif and Al-Sarray (2025) [1144]</td><td>Combined t-SNE with spectral clustering via convex optimization for breast cancer gene classification, achieving superior clustering performance over conventional methods.</td></tr><tr><td>Park et al. (2025) [1145]</td><td>Assessed the use of t-SNE in flow cytometry for hematologic malignancies, highlighting its superiority in preserving local structures compared to UMAP.</td></tr><tr><td>Qiao et al. (2025) [1146]</td><td>Applied t-SNE to analyze cancer-associated fibroblasts (CAFs) in pancreatic ductal adenocarcinoma, identifying gene expression subclusters for patient stratification.</td></tr><tr><td>Su et al. (2025) [1143]</td><td>Employed t-SNE for damage quantification in aircraft structures, integrating it into a deep learning framework to enhance structural health monitoring.</td></tr></tbody></table>

### 16.1.7.3 Mathematical Analysis of t-distributed stochastic neighbor embedding

The t-distributed stochastic neighbor embedding (t-SNE) algorithm, developed by Laurens van der Maaten and Geoffrey Hinton, is a nonlinear dimensionality reduction technique particularly well-suited for visualizing high-dimensional datasets in lower dimensions (typically 2D or 3D). At its core, t-SNE aims to preserve the local structure of data points by modeling pairwise similarities in both high-dimensional and low-dimensional spaces and minimizing the discrepancy between these similarities. The method builds upon *Stochastic Neighbor Embedding (SNE)* by incorporating a **Student's t-distribution with a single degree of freedom (i.e., a Cauchy distribution)** as the low-dimensional similarity function, significantly mitigating the crowding problem of SNE. The mathematical formulation of t-SNE is highly intricate and involves defining probability distributions over pairwise relationships, constructing a cost function based on Kullback-Leibler (KL) divergence, and employing gradient-based optimization methods such as *gradient descent* to find an embedding that best preserves local structures.

#### Constructing the High-Dimensional Probability Distribution

Let $\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_N \in \mathbb{R}^D$ be a set of $N$ data points in a high-dimensional space of dimension $D$. We define a conditional probability distribution that represents the similarity between points $\mathbf{x}_i$ and $\mathbf{x}_j$ based on a Gaussian distribution centered at $\mathbf{x}_i$. The probability that $\mathbf{x}_j$ is a neighbor of $\mathbf{x}_i$ is given by:

$$p_{j|i} = \frac{\exp\left(-\frac{\|\mathbf{x}_i - \mathbf{x}_j\|^2}{2\sigma_i^2}\right)}{\sum_{k \neq i} \exp\left(-\frac{\|\mathbf{x}_i - \mathbf{x}_k\|^2}{2\sigma_k^2}\right)} \quad (16.46)$$

where $\sigma_i$ is the *perplexity-dependent bandwidth parameter* for point $\mathbf{x}_i$, which controls how much influence distant points have on similarity measures. The **perplexity**, denoted as $\mathcal{P}$, is defined as:

$$\mathcal{P}(i) = 2^{H(P_i)} \quad (16.47)$$

where $H(P_i)$ is the Shannon entropy of the probability distribution:

$$H(P_i) = - \sum_{j \neq i} p_{j|i} \log_2 p_{j|i} \quad (16.48)$$

We then symmetrize these conditional probabilities to obtain a symmetric joint probability distribution:

$$p_{ij} = \frac{p_{j|i} + p_{i|j}}{2N} \quad (16.49)$$
<figure><img src="image_171.png" /><figcaption>Figure 16.3: T-SNE representation of word embeddings derived from 19th-century literary texts</figcaption></figure>

Image Credit: By Siobhán Grayson, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=64541584

This ensures that $p_{ij} = p_{ji}$, making optimization easier and avoiding directed relationships between points.

## Constructing the Low-Dimensional Probability Distribution

We now define a probability distribution in the lower-dimensional space (typically 2D or 3D) with corresponding mapped points $y_1, y_2, \dots, y_N \in \mathbb{R}^d$ (where $d \ll D$). Instead of a Gaussian kernel, t-SNE employs a **Student's t-distribution with one degree of freedom (a Cauchy distribution)** to measure similarities:

$$q_{ij} = \frac{(1 + \|y_i - y_j\|^2)^{-1}}{\sum_{k \neq l} (1 + \|y_k - y_l\|^2)^{-1}} \quad (16.50)$$

This choice of the t-distribution helps mitigate the *crowding problem*, where points in high-dimensional space that are moderately distant tend to collapse together in low-dimensional space when using Gaussian similarities.

## The Kullback-Leibler Divergence Cost Function

To ensure that the probability distributions $p_{ij}$ and $q_{ij}$ match as closely as possible, we minimize the *Kullback-Leibler (KL) divergence*, which measures how much information is lost when using $q_{ij}$ to approximate $p_{ij}$:

$$C = \sum_{i \neq j} p_{ij} \log \frac{p_{ij}}{q_{ij}} \quad (16.51)$$
<figure><img src="image_172.png" /><figcaption>Figure 16.4: T-SNE embeddings of MNIST dataset</figcaption></figure>

Image Credit: By Kyle McDonald - https://www.flickr.com/photos/kylemcdonald/26620503329/, CC BY 2.0, https://commons.wikimedia.org/w/index.php?curid=115726949

This function is minimized using *gradient descent*, where the gradient with respect to each low-dimensional point $\mathbf{y}_i$ is given by:

$$ \frac{\partial C}{\partial \mathbf{y}_i} = 4 \sum_{j \neq i} (p_{ij} - q_{ij}) (1 + \|\mathbf{y}_i - \mathbf{y}_j\|^2)^{-1} (\mathbf{y}_i - \mathbf{y}_j) \quad (16.52) $$

## Optimization Using Momentum-Based Gradient Descent

To find an optimal configuration of points in the lower-dimensional space, we iteratively update $\mathbf{y}_i$ using an *adaptive learning rate* and a *momentum term* to prevent getting stuck in local minima:

$$ \mathbf{y}_i^{(t+1)} = \mathbf{y}_i^{(t)} + \eta \frac{\partial C}{\partial \mathbf{y}_i} + \alpha (\mathbf{y}_i^{(t)} - \mathbf{y}_i^{(t-1)}) \quad (16.53) $$

where $\eta$ is the **learning rate**, $\alpha$ is the **momentum coefficient**, and $\mathbf{y}_i^{(t)}$ represents positions at different iterations.

In summary, t-SNE is a powerful, highly nonlinear technique for dimensionality reduction that excels at visualizing high-dimensional data while preserving local structures. Its formulation relies on constructing two probability distributions, using a *t-distribution kernel*, and optimizing the **KL divergence cost function** through gradient descent. However, due to its computational complexity, improvements such as **Barnes-Hut t-SNE** and **FIt-SNE** have been developed for large datasets.
16.1.8 Locally Linear Embedding (LLE)

16.1.8.1 Literature Review of Locally Linear Embedding (LLE)

Locally Linear Embedding (LLE), introduced by Roweis and Saul (2000) [1043], is a nonlinear dimensionality reduction method that aims to preserve the local geometric properties of high-dimensional data while embedding it into a lower-dimensional space. The core idea behind LLE is to represent each data point as a linear combination of its nearest neighbors and then determine the low-dimensional representation that best preserves these local linear reconstructions. Unlike methods such as Principal Component Analysis (PCA) or Multidimensional Scaling (MDS), LLE does not assume a global linear structure but instead relies on local affine invariances to extract meaningful low-dimensional embeddings. Roweis and Roweis (2000) [1147] later provided an in-depth mathematical exposition of the LLE algorithm, detailing its optimization formulation and demonstrating its capability to uncover nonlinear manifold structures. Their work emphasized that LLE effectively preserves local symmetries without requiring an explicit parametrization of the data manifold, thus making it suitable for a wide range of applications in machine learning and data analysis.

Following the introduction of LLE, Polito and Perona (2001) [1148] extended its application to the problem of clustering and dimensionality reduction, demonstrating that LLE could naturally group data points based on their intrinsic geometric properties. Their study highlighted the algorithm's ability to perform soft clustering, where different regions of the embedded space correspond to distinct clusters in the high-dimensional space. This property is particularly useful in vision tasks where the underlying data often exhibit complex nonlinear structures. Zhang and Zha (2004) [1150] proposed an alternative approach to nonlinear dimensionality reduction known as Local Tangent Space Alignment (LTSA), which sought to align local tangent spaces rather than simply preserving locally linear relationships. By focusing on the consistency of local tangent approximations, LTSA addressed certain limitations of LLE, particularly its sensitivity to variations in neighborhood density. LTSA improved the quality of embeddings by ensuring a more faithful reconstruction of the global manifold structure, thereby making it a competitive alternative to LLE for high-dimensional data analysis.

Further refinements to LLE were made by Donoho and Grimes (2003) [1151], who introduced Hessian Eigenmaps as a variation of LLE that utilized Hessian-based quadratic forms to better capture local curvature information. Their work showed that Hessian Eigenmaps could outperform standard LLE in cases where the data exhibited significant variations in local density, thereby reducing distortion in the learned embeddings. Another modification, introduced by Zhang and Wang (2006) [1152], was the development of Modified Locally Linear Embedding (MLLE), which incorporated multiple weights in each neighborhood to address issues related to the conditioning of the local weight matrix. The authors demonstrated that by introducing multiple weight constraints, MLLE produced embeddings that were less prone to numerical instability, thus leading to improved robustness and consistency in the low-dimensional representations. These advancements collectively enhanced the stability and general applicability of LLE-based methods, reinforcing their utility in real-world machine learning tasks.

Beyond theoretical refinements, LLE found applications in natural language processing, where Liang (2005) [1153] explored its role in semi-supervised learning. By leveraging the geometric structure of unlabeled data, LLE facilitated the discovery of meaningful feature representations, which proved useful in learning linguistic patterns with limited labeled samples. Coates and Ng (2012) [1154] provided a broader perspective on feature learning by comparing LLE with other unsupervised learning techniques such as K-means clustering. Their study examined the strengths and weaknesses of LLE in the context of automatic feature extraction, highlighting its capacity to learn meaningful data representations without explicit supervision. These contributions under-
scored the versatility of LLE and its relevance across different domains, including computer vision, speech processing, and text analysis.

In the broader context of feature extraction and representation learning, Hyvärinen and Oja (2000) [1155] explored Independent Component Analysis (ICA) as an alternative method for learning structured representations of high-dimensional data. While ICA seeks to identify statistically independent components, LLE preserves local geometric relationships, making them complementary approaches to dimensionality reduction. Lee et al. (2006) [1156] further advanced the study of feature learning by developing efficient sparse coding algorithms, which provided insights into the underlying structures of data representations. Their work discussed the differences between sparse coding techniques and manifold learning approaches such as LLE, emphasizing the advantages of sparsity constraints in generating interpretable features. Collectively, these contributions illustrate the ongoing evolution of nonlinear dimensionality reduction techniques, with LLE serving as a foundational method that continues to inspire research in machine learning and data science.

<table><caption>Table 16.11: Summary of Contributions in Locally Linear Embedding (LLE)</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Roweis and Saul (2000) [1043]</td><td>Introduced Locally Linear Embedding (LLE), a nonlinear dimensionality reduction method that preserves local geometric properties while embedding high-dimensional data into a lower-dimensional space. LLE represents each data point as a linear combination of its nearest neighbors and determines an embedding that best preserves these local reconstructions.</td></tr><tr><td>Saul and Roweis (2000) [1147]</td><td>Provided an in-depth mathematical exposition of the LLE algorithm, detailing its optimization formulation and demonstrating its effectiveness in uncovering nonlinear manifold structures without requiring explicit parametrization of the data manifold.</td></tr><tr><td>Polito and Perona (2001) [1148]</td><td>Extended LLE to clustering and dimensionality reduction, showing that LLE naturally groups data points based on intrinsic geometric properties, allowing for soft clustering useful in vision tasks.</td></tr><tr><td>Zhang and Zha (2004) [1150]</td><td>Proposed Local Tangent Space Alignment (LTSA), which aligns local tangent spaces rather than preserving local linear relationships, addressing LLE's sensitivity to variations in neighborhood density and improving global manifold reconstruction.</td></tr><tr><td>Donoho and Grimes (2003) [1151]</td><td>Introduced Hessian Eigenmaps, a variation of LLE utilizing Hessian-based quadratic forms to capture local curvature, reducing distortion in embeddings for data with significant variations in local density.</td></tr><tr><td>Zhang and Wang (2006) [1152]</td><td>Developed Modified Locally Linear Embedding (MLLE), incorporating multiple weights in each neighborhood to improve numerical stability and robustness in low-dimensional representations.</td></tr><tr><td>Liang (2005) [1153]</td><td>Applied LLE to semi-supervised learning in natural language processing, leveraging the geometric structure of unlabeled data to discover meaningful feature representations.</td></tr><tr><td>Coates and Ng (2012) [1154]</td><td>Compared LLE with other unsupervised learning techniques, such as K-means clustering, highlighting LLE's strengths and weaknesses in automatic feature extraction and representation learning.</td></tr></tbody></table>
<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Hyvärinen and Oja (2000) [1155]</td><td>Explored Independent Component Analysis (ICA) as an alternative method for structured representation learning, contrasting ICA's focus on statistical independence with LLE's preservation of local geometric relationships.</td></tr><tr><td>Lee et al. (2006) [1156]</td><td>Developed efficient sparse coding algorithms, discussing differences between sparse coding techniques and manifold learning approaches like LLE, emphasizing the advantages of sparsity constraints in generating interpretable features.</td></tr></tbody></table>

### 16.1.8.2 Recent Literature Review of Locally Linear Embedding (LLE)

Locally Linear Embedding (LLE) has been widely studied in recent years, finding applications in diverse fields such as tourism analysis, machine learning, geophysics, remote sensing, and industrial diagnostics. Yang et al. (2025) [1157] utilized LLE in the domain of international tourism competitiveness, where they integrated it with an entropy-TOPSIS and GRA model to analyze city-level data. By leveraging the ability of LLE to uncover non-linear structures in high-dimensional spaces, they were able to enhance the ranking system for assessing tourism competitiveness, which would have been difficult using traditional linear techniques. Wang et al. (2025) [1158] introduced a hybrid model that integrates LLE with Transformer and LightGBM to predict the thermal conductivity of natural rock materials, a crucial parameter for geothermal energy applications. In their study, LLE was used to perform dimensionality reduction, allowing the Transformer model to better capture latent structural patterns in the data. The combination of these methods led to improved prediction accuracy, highlighting the role of LLE in optimizing feature selection for complex geophysical modeling. Jin et al. (2025) [1159] proposed an improved version of LLE called Neighbor-Adapted LLE (NALLE) for synthetic aperture radar (SAR) image processing. Their model effectively captured the structural properties of SAR images, facilitating zero-shot learning, where models are trained without requiring large labeled datasets. By adapting LLE to work efficiently in image processing tasks, they demonstrated its applicability in remote sensing, particularly for classifying maritime objects.

In the field of optimization and algorithmic advancements, Li et al. (2024) [1160] proposed a novel variation of LLE that modifies its original L2 norm-based distance metric using the Ali Baba and The Forty Thieves Algorithm, an optimization technique inspired by metaheuristics. This modification improved LLE's computational efficiency while maintaining accuracy in capturing data manifold structures. Similarly, Jafari et al. (2025) [1161] provided an extensive review of LLE and its variants in machine learning, emphasizing their role in feature extraction, non-linear dimensionality reduction, and data visualization. Their work serves as a valuable resource for understanding the theoretical and practical developments of LLE, particularly in the context of biological data analysis and big data applications. Additionally, Zhou et al. (2025) [1162] demonstrated the practical application of LLE in nondestructive testing (NDT) of thermal barrier coatings. Their study showed that LLE could extract useful features from terahertz imaging data, significantly improving the accuracy of stress detection in high-temperature material coatings.

Beyond scientific and industrial applications, LLE has been successfully employed in engineering diagnostics and predictive maintenance. Dou et al. (2024) [1164] proposed an LLE-based method for detecting faults in high-speed train traction systems. By transforming high-dimensional sensor data into a lower-dimensional representation, LLE allowed for the early detection of system failures, ensuring the reliability and safety of train operations. Similarly, Bagherzadeh et al. (2021) [1165] combined LLE with K-means clustering to optimize test case prioritization in software testing. Their method improved the efficiency of defect detection in software systems, reducing testing
costs and accelerating the debugging process. Liu et al. (2025) [1166] proposed an intelligent recognition algorithm for analyzing substation secondary wiring diagrams using a denoised variant of LLE (D-LLE). Their approach enhanced the accuracy of connection identification in complex electrical circuits, improving power system automation and maintenance.

These studies collectively illustrate the versatility and adaptability of LLE in a wide range of fields, from fundamental algorithmic research to practical industrial applications. The continuous improvements and novel adaptations of LLE, such as the development of Neighbor-Adapted LLE for SAR images, its integration with Transformer models for geothermal applications, and its optimization using heuristic algorithms, demonstrate its evolving role in modern data science. The effectiveness of LLE in dimensionality reduction, feature extraction, and manifold learning underscores its significance in handling high-dimensional data across diverse disciplines. Whether in tourism, energy modeling, material testing, fault detection, or machine learning, LLE continues to be a powerful tool for solving complex, non-linear problems, paving the way for future advancements in data-driven decision-making.

<table><caption>Table 16.12: Summary of Recent Contributions in Locally Linear Embedding (LLE)</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Yang et al. (2025) [1157]</td><td>Utilized LLE in international tourism competitiveness analysis, integrating it with entropy-TOPSIS and GRA models to enhance ranking systems for city-level data.</td></tr><tr><td>Wang et al. (2025) [1158]</td><td>Introduced a hybrid model combining LLE with Transformer and LightGBM to predict thermal conductivity of natural rock materials, improving feature selection in geothermal energy applications.</td></tr><tr><td>Jin et al. (2025) [1159]</td><td>Proposed Neighbor-Adapted LLE (NALLE) for SAR image processing, enabling zero-shot learning and improving maritime object classification in remote sensing.</td></tr><tr><td>Li et al. (2024) [1160]</td><td>Developed a novel variation of LLE using the Ali Baba and The Forty Thieves Algorithm, enhancing computational efficiency while preserving data manifold accuracy.</td></tr><tr><td>Jafari et al. (2025) [1161]</td><td>Conducted an extensive review of LLE and its variants, focusing on feature extraction, non-linear dimensionality reduction, and data visualization in biological and big data applications.</td></tr><tr><td>Zhou et al. (2025) [1162]</td><td>Applied LLE in nondestructive testing (NDT) of thermal barrier coatings, demonstrating improved feature extraction from terahertz imaging data for stress detection.</td></tr><tr><td>Dou et al. (2024) [1164]</td><td>Proposed an LLE-based method for fault detection in high-speed train traction systems, facilitating early system failure detection and enhancing train safety.</td></tr><tr><td>Bagherzadeh et al. (2021) [1165]</td><td>Combined LLE with K-means clustering for test case prioritization in software testing, improving defect detection efficiency and reducing debugging costs.</td></tr><tr><td>Liu et al. (2025) [1166]</td><td>Developed an intelligent recognition algorithm for analyzing substation secondary wiring diagrams using a denoised LLE (D-LLE) variant, enhancing power system automation and maintenance.</td></tr></tbody></table>
### 16.1.8.3 Mathematical Analysis of Locally Linear Embedding (LLE)

Locally Linear Embedding (LLE) is a **manifold learning technique** that seeks to uncover the **intrinsic geometric structure** of high-dimensional data by **preserving local neighborhoods**. It assumes that the data lies on or near a **low-dimensional manifold** embedded within a high-dimensional space. The core idea is that each data point and its nearest neighbors lie on or near a locally linear patch of the manifold. Thus, LLE constructs a **low-dimensional embedding** that preserves these local linear relationships. To achieve this, LLE follows three key steps: (1) finding nearest neighbors, (2) computing linear reconstruction weights, and (3) computing the low-dimensional embedding. Below, we describe these steps in an **extremely rigorous, highly mathematical, and very detailed manner with maximal equations and derivations**.

#### Step 1: Finding Nearest Neighbors

Given a dataset $\mathbf{X} = \{\mathbf{x}_1, \mathbf{x}_2, \dots, \mathbf{x}_N\}$, where each data point $\mathbf{x}_i$ belongs to $\mathbb{R}^D$ (i.e., the data is embedded in a high-dimensional space of dimension $D$), the first step in LLE is to identify the **K-nearest neighbors** of each point. Mathematically, for each point $\mathbf{x}_i$, we find a set of **K nearest neighbors** $\mathcal{N}(i)$, such that:

<figure><img src="image_173.png" /><figcaption>Figure 16.5: Locally Linear Embedding (LLE) applied to a Swiss roll dataset. Left: original 3D Swiss roll. Right: 2D LLE embedding preserving local structure</figcaption></figure>

$$ \mathcal{N}(i) = \{\mathbf{x}_{j_1}, \mathbf{x}_{j_2}, \dots, \mathbf{x}_{j_K}\} \qquad (16.54) $$

where the neighbors are selected based on **Euclidean distance**:

$$ \|\mathbf{x}_i - \mathbf{x}_j\|^2 = (\mathbf{x}_i - \mathbf{x}_j)^T (\mathbf{x}_i - \mathbf{x}_j). \qquad (16.55) $$

#### Step 2: Computing Reconstruction Weights

Once the neighbors of each point are found, the next step is to compute **reconstruction weights** that best express each data point as a linear combination of its nearest neighbors:

$$ \mathbf{x}_i \approx \sum_{j \in \mathcal{N}(i)} w_{ij} \mathbf{x}_j. \qquad (16.56) $$

The goal is to determine the **weights** $w_{ij}$ that **minimize the reconstruction error**:

$$ \mathcal{E}(W) = \sum_{i=1}^{N} \left\| \mathbf{x}_i - \sum_{j \in \mathcal{N}(i)} w_{ij} \mathbf{x}_j \right\|^2. \qquad (16.57) $$
To ensure **invariance to translations**, we impose the **constraint**:

$$ \sum_{j \in \mathcal{N}(i)} w_{ij} = 1. \qquad (16.58) $$

This leads to the **local covariance matrix**:

$$ C_{jk}^{(i)} = (\mathbf{x}_i - \mathbf{x}_j)^T (\mathbf{x}_i - \mathbf{x}_k). \qquad (16.59) $$

The optimal weights $w_{ij}$ are obtained by solving the **quadratic programming problem**:

$$ \min_{w_{ij}} \sum_{i=1}^{N} \sum_{j,k \in \mathcal{N}(i)} w_{ij} C_{jk}^{(i)} w_{ik}, \quad \text{subject to} \quad \sum_{j \in \mathcal{N}(i)} w_{ij} = 1. \qquad (16.60) $$

The solution is:

$$ \mathbf{w}_i = \frac{\mathbf{C}^{(i)-1} \mathbf{1}}{\mathbf{1}^T \mathbf{C}^{(i)-1} \mathbf{1}}. \qquad (16.61) $$

### Step 3: Computing the Low-Dimensional Embedding

After computing the weights $w_{ij}$, we find low-dimensional coordinates $\mathbf{Y} = \{\mathbf{y}_1, \mathbf{y}_2, \dots, \mathbf{y}_N\}$ such that:

$$ \mathbf{y}_i = \sum_{j \in \mathcal{N}(i)} w_{ij} \mathbf{y}_j. \qquad (16.62) $$

This leads to the **quadratic cost function**:

$$ \Phi(\mathbf{Y}) = \sum_{i=1}^{N} \left\| \mathbf{y}_i - \sum_{j \in \mathcal{N}(i)} w_{ij} \mathbf{y}_j \right\|^2. \qquad (16.63) $$

Rewriting in matrix form:

$$ \Phi(\mathbf{Y}) = \mathrm{Tr}(\mathbf{Y}^T \mathbf{M} \mathbf{Y}), \qquad (16.64) $$

where $\mathbf{M} = (\mathbf{I} - \mathbf{W})^T (\mathbf{I} - \mathbf{W})$ is the **embedding cost matrix**. The final embedding is obtained by solving:

$$ \mathbf{M} \mathbf{y} = \lambda \mathbf{y}. \qquad (16.65) $$

The final solution is:

$$ \mathbf{Y} = [\mathbf{y}_1, \mathbf{y}_2, \dots, \mathbf{y}_N]^T. \qquad (16.66) $$

This ensures that the **local linear relationships are preserved** while capturing the **global structure** of the data.

### 16.1.8.4 Python Code to generate Figure 16.5 illustrating the Locally Linear Embedding (LLE) applied to a Swiss roll dataset

The Python code below produces the Figure 16.5 illustrating the Locally Linear Embedding (LLE) applied to a Swiss roll dataset.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.manifold import LocallyLinearEmbedding
from sklearn datasets import make_swiss roll

# Seed for reproducibility
np.random.seed(0)
```
```python
9 # Generate Swiss roll data
10 n Samples = 1500
11 X, color = make_swiss roll(n Samples)
12
13 # Perform Locally Linear Embedding
14 n neighbors = 12
15 n_components = 2
16 lle = LocallyLinearEmbedding(n neighbors=n neighbors, n_components=n_components, method='standard')
17 X_r = lle.fit transform(X)
18
19 # Plot original 3D Swiss roll
20 fig = plt .figure(figsize=(12, 5))
21
22 ax = fig.addsubplot(1, 2, 1, projection='3d')
23 ax. scatter(X[:, 0], X[:, 1], X[:, 2], c=color, cmap= plt.cm.Spectral )
24 ax. set_title("Original Swiss Roll (3D)")
25
26 # Plot LLE embedding (2D)
27 ax2 = fig.addsubplot(1, 2, 2)
28 ax2. scatter(X_r[:, 0], X_r[:, 1], c=color, cmap= plt.cm.Spectral )
29 ax2. set_title("LLE Embedding (2D)")
30
31 plt .tight_layout()
32 plt .show()
```

16.1.9 Independent Component Analysis (ICA)

16.1.9.1 Literature Review of Independent Component Analysis (ICA)

Independent Component Analysis (ICA) has developed into a powerful technique in signal pro-
cessing, statistics, and machine learning. Its theoretical foundations were laid by Comon (1994)
[1167], who rigorously introduced the concept of ICA as a method for blind source separation. He
provided a detailed mathematical framework, demonstrating how statistical independence, rather
than uncorrelatedness, is the key criterion for successful source separation. By linking ICA with
higher-order statistics, particularly kurtosis, he established a formal justification for ICA as a gener-
alization of Principal Component Analysis (PCA). This work also introduced the concept of mutual
information minimization as a means of achieving independence, offering a theoretical perspective
that would influence many later algorithms. Around the same time, Jutten and Herault (1991)
[1168] developed an adaptive learning algorithm based on neuromorphic principles, which laid the
groundwork for ICA as an iterative computational approach. Their method used nonlinear func-
tions to iteratively adjust the weight matrix, leveraging higher-order statistical moments to achieve
source separation, marking one of the first practical implementations of ICA in neural networks.

A major breakthrough came with the development of computationally efficient ICA algorithms,
notably the FastICA method by Hyvärinen and Oja (1997) [1169]. They formulated an elegant
fixed-point approach that significantly improved the convergence speed of ICA. Instead of using slow
gradient-based updates, their method maximized non-Gaussianity using a negentropy approxima-
tion, which enabled rapid convergence to independent components. This algorithm incorporated a
preprocessing step involving PCA-based whitening, which decorrelated the observed signals before
ICA extraction, further enhancing stability and efficiency. In parallel, Bell and Sejnowski (1995)
[1044] introduced an information-theoretic approach to ICA, deriving the Infomax algorithm based
on maximum likelihood estimation (MLE). Their work established that maximizing the entropy
of the output neurons under a nonlinear activation function naturally leads to ICA, providing a fundamental link between ICA and unsupervised neural learning. This insight positioned ICA as a biologically plausible mechanism for sensory processing, particularly in neural coding and vision research.

Several alternative approaches to ICA were also developed, focusing on different statistical principles. Cardoso and Souloumiac (1993) [1170] introduced the JADE algorithm, which employed joint approximate diagonalization of eigenmatrices to separate independent sources. By exploiting fourth-order cumulants, their method avoided gradient descent altogether, leading to a robust and efficient separation procedure, especially for overdetermined mixtures. Meanwhile, Amari et al. (1995) [1171] introduced an ICA algorithm based on natural gradient learning, drawing on concepts from information geometry. They demonstrated that a Riemannian metric on the parameter space of ICA solutions enables more efficient optimization, leading to faster convergence without suffering from the plateaus and slowdowns of traditional gradient descent methods. This information-geometric perspective provided deeper insights into the structure of the ICA optimization landscape and inspired further advances in adaptive ICA algorithms.

A significant refinement of ICA occurred with the extension of Infomax to handle both sub-Gaussian and super-Gaussian source distributions by Lee et al. (1999) [1172]. Their method introduced a nonlinear adaptive function that dynamically adjusted based on the statistical properties of the data, making ICA applicable to a broader range of real-world signals. This extension proved particularly useful in biomedical applications, where sources often exhibit mixed statistical distributions. Around the same time, Pham and Garat (1997) [1173] presented a quasi-maximum likelihood estimation (QMLE) approach to ICA, rigorously formulating the problem as an optimization task within the framework of statistical estimation theory. Their work provided a more systematic theoretical foundation for ICA and improved its robustness in practical scenarios, particularly when dealing with noisy or low-sample data environments.

More recent developments in ICA have explored probabilistic and Bayesian approaches. Højen-Sørensen et al. (2002) [1174] introduced a variational Bayesian framework for ICA, using mean-field approximations to estimate the posterior distribution of independent components. This approach allowed ICA to be extended into a probabilistic setting, making it more robust in the presence of noise and missing data. Their work was particularly influential in applications requiring uncertainty quantification, such as brain imaging and financial modeling. In addition to these algorithmic advancements, Stone (2004) [1175] provided a comprehensive textbook that rigorously detailed the mathematical principles underlying ICA. His work systematically explored the relationship between ICA, mutual information, higher-order statistics, and practical signal processing applications, serving as an authoritative reference for both theoretical research and practical implementation. Together, these contributions have solidified ICA as a fundamental tool in signal processing, machine learning, and neuroscience, with ongoing developments continuing to refine its theoretical underpinnings and expand its applications.

<table><caption>Table 16.13: Summary of Contributions in Independent Component Analysis (ICA) Literature</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Comon (1994) [1167]</td><td>Established the theoretical framework of ICA, emphasizing statistical independence over uncorrelatedness for blind source separation. Introduced mutual information minimization as a means to achieve independence and linked ICA with higher-order statistics, particularly kurtosis.</td></tr></tbody></table>
(Continued from previous page)

<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Jutten and Herault (1991) [1168]</td><td>Developed an adaptive learning algorithm for ICA using neuromorphic principles. Introduced nonlinear functions to iteratively adjust the weight matrix based on higher-order statistical moments. This work laid the foundation for iterative computational approaches in ICA.</td></tr><tr><td>Hyvärinen and Oja (1997) [1169]</td><td>Proposed the FastICA algorithm, a fixed-point method maximizing non-Gaussianity using negentropy approximation. Improved convergence speed and incorporated PCA-based whitening for signal decorrelation.</td></tr><tr><td>Bell and Sejnowski (1995) [1044]</td><td>Developed the Infomax algorithm using maximum likelihood estimation, linking ICA to entropy maximization in neural networks. Provided a biological perspective on ICA and its role in sensory processing.</td></tr><tr><td>Cardoso and Souloumiac (1993) [1170]</td><td>Introduced the JADE algorithm, employing joint approximate diagonalization of eigenmatrices to separate independent sources. Used fourth-order cumulants to achieve robust and efficient source separation.</td></tr><tr><td>Amari et al. (1995) [1171]</td><td>Developed an ICA algorithm based on natural gradient learning, utilizing a Riemannian metric for efficient optimization. Provided an information-geometric perspective, improving convergence speed.</td></tr><tr><td>Lee et al. (1999) [1172]</td><td>Extended the Infomax algorithm to handle both sub-Gaussian and super-Gaussian source distributions using a nonlinear adaptive function. Enhanced ICA applicability to biomedical signals with mixed statistical distributions.</td></tr><tr><td>Pham and Garat (1997) [1173]</td><td>Introduced a quasi-maximum likelihood estimation (QMLE) approach to ICA, formulating it as an optimization problem within statistical estimation theory. Improved robustness in noisy and low-sample data environments.</td></tr><tr><td>Højen-Sørensen et al. (2002) [1174]</td><td>Proposed a variational Bayesian framework for ICA using mean-field approximations. Enabled ICA to incorporate uncertainty quantification and handle missing data, extending its robustness.</td></tr><tr><td>Stone (2004) [1175]</td><td>Authored a comprehensive textbook on ICA, detailing its mathematical principles, mutual information, higher-order statistics, and applications in signal processing and machine learning. Provided a systematic exploration of ICA's theoretical and practical aspects.</td></tr></tbody></table>

### 16.1.9.2 Recent Literature Review of Independent Component Analysis (ICA)

Independent Component Analysis (ICA) has found diverse applications across multiple domains, ranging from neuroscience and medical imaging to financial modeling and power systems. Bezhadfar et al. (2025) [1176] introduced a novel multi-frequency ICA-based approach to process functional MRI (fMRI) data. Their study focused on extracting meaningful components while effectively removing non-gray matter signals, thereby refining the accuracy of frequency-based brain
imaging. The method demonstrated improved signal clarity and more precise voxel-wise frequency
difference estimation, making it a valuable tool in neuroimaging research. Similarly, Eierud et al.
(2025) [1177] developed the NeuroMark PET ICA framework, which employs ICA to decompose
whole-brain PET signals into distinct networks, aiding in the construction of multivariate molecu-
lar imaging brain atlases. This technique allows researchers to analyze complex brain connectivity
patterns with greater precision, enhancing the study of neurodegenerative disorders and other neu-
rological conditions.

Expanding ICA's application in hydrology, Wang et al. (2025) [1178] leveraged the technique to analyze terrestrial water storage anomaly (TWSA) trends in the Yangtze River Basin. Their research identified statistically independent spatial trends within hydrological data, offering insights into climate change and its impact on water resource management. By distinguishing significant patterns from background noise, ICA facilitated a more accurate assessment of water distribution and hydrological cycles over time. Similarly, Heurtebise et al. (2025) [1179] used ICA to stabilize estimators in hydrological dataset analysis, improving the reliability of multivariate mutual information measurements. These advancements underscore ICA's utility in environmental sciences, particularly in large-scale water resource monitoring.

In the field of computational neuroscience, Ouyang and Li (2025) [1180] developed a protocol that integrates ICA with Principal Component Analysis (PCA) for semi-automated EEG preprocessing. Their approach streamlines the removal of artifacts and enhances signal quality, making it easier for researchers to conduct large-scale EEG studies with minimal manual intervention. Zhang and Luck (2025) [1181] further explored ICA's impact on brain-computer interfaces by assessing the effect of artifact correction on the performance of support vector machine (SVM)-based EEG decoding. Their study demonstrated that ICA-based correction significantly improved classification accuracy, reinforcing its role as a crucial preprocessing tool for neurophysiological data.

ICA has also shown promise in financial modeling and power system monitoring. Kirsten and Süssmuth (2025) [1182] applied ICA to financial time-series data, demonstrating its ability to filter noise and identify independent market-driving factors in cryptocurrency price movements. Their findings indicated that ICA, combined with ARIMA modeling, improved predictive accuracy in highly volatile market environments. Meanwhile, Jung et al. (2025) [1183] developed a hybrid fault detection system that integrates ICA with auto-associative kernel regression (AAKR) for power plant monitoring. Their model effectively isolated independent fault components, enabling more accurate anomaly detection and predictive maintenance strategies.

In signal processing and acoustic applications, Wang et al. (2025) [1184] implemented ICA for noise filtering in passive acoustic localization, particularly for underwater object detection. Their method significantly enhanced the clarity of spatial positioning signals, reducing the impact of environmental noise. Luo et al. (2025) [1185] extended ICA's utility to brain-computer interfaces (BCIs), where they used the technique to eliminate electrical noise and enhance the transmission of neural signals. Their research demonstrated ICA's ability to improve the reliability of noninvasive BCIs, paving the way for more accurate and responsive brain-controlled devices.

These studies collectively illustrate ICA's versatility across disciplines, from improving medical imaging and neurophysiological data analysis to enhancing hydrological assessments, financial forecasting, and fault detection. As ICA continues to evolve, its integration with machine learning techniques and its application in multi-sensor data fusion are likely to expand, offering even greater potential for scientific and industrial advancements. Future research should focus on optimizing ICA algorithms to handle increasingly complex datasets while maintaining computational efficiency.
<table><caption>Table 16.14: Summary of Recent Contributions in Independent Component Analysis (ICA) Literature</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Behzadfar et al. (2025) [1176]</td><td>Proposed a multi-frequency ICA-based approach for fMRI data processing, enhancing component extraction and eliminating non-gray matter signals for improved frequency-based brain imaging accuracy.</td></tr><tr><td>Eierud et al. (2025) [1177]</td><td>Developed the NeuroMark PET ICA framework, enabling ICA decomposition of whole-brain PET signals into networks to construct multivariate molecular imaging brain atlases.</td></tr><tr><td>Wang et al. (2025) [1178]</td><td>Applied ICA to analyze terrestrial water storage anomaly (TWSA) trends in the Yangtze River Basin, identifying independent spatial trends and improving hydrological cycle assessments.</td></tr><tr><td>Heurtebise et al. (2025) [1179]</td><td>Utilized ICA to stabilize estimators in hydrological dataset analysis, enhancing the reliability of multivariate mutual information measurements.</td></tr><tr><td>Ouyang and Li (2025) [1180]</td><td>Integrated ICA with PCA for semi-automated EEG preprocessing, facilitating artifact removal and improving signal quality in large-scale EEG studies.</td></tr><tr><td>Zhang and Luck (2025) [1181]</td><td>Investigated ICA-based artifact correction in brain-computer interfaces, demonstrating significant improvements in SVM-based EEG decoding accuracy.</td></tr><tr><td>Kirsten and Süssmuth (2025) [1182]</td><td>Applied ICA to financial time-series data, filtering noise and identifying independent market-driving factors, enhancing cryptocurrency price prediction when combined with ARIMA modeling.</td></tr><tr><td>Jung et al. (2025) [1183]</td><td>Developed a hybrid fault detection system integrating ICA with auto-associative kernel regression (AAKR) for power plant monitoring, improving anomaly detection and predictive maintenance.</td></tr><tr><td>Wang et al. (2025) [1184]</td><td>Implemented ICA for noise filtering in passive acoustic localization, significantly enhancing underwater object detection through improved signal clarity.</td></tr><tr><td>Luo et al. (2025) [1185]</td><td>Applied ICA in brain-computer interfaces (BCIs) to eliminate electrical noise and enhance neural signal transmission, improving the reliability of noninvasive BCIs.</td></tr></tbody></table>

### 16.1.9.3 Mathematical Analysis of Independent Component Analysis (ICA)

Independent Component Analysis (ICA) is a powerful statistical technique used for **blind source separation** (BSS), where the goal is to recover a set of **statistically independent** source signals from their **linear mixtures** without any prior information about the mixing process. Mathematically, we assume that we are given an *n*-dimensional observation vector **x**, which is related to an unknown set of source signals **s** through an **unknown mixing matrix A** as follows:

$$ \mathbf{x} = \mathbf{A}\mathbf{s} \qquad (16.67) $$

where $\mathbf{x} \in \mathbb{R}^n$ is the observed signal vector, $\mathbf{s} \in \mathbb{R}^n$ is the original source vector, and $\mathbf{A} \in \mathbb{R}^{n \times n}$ is the **mixing matrix** that describes how the sources are combined. The objective of ICA is to estimate
a demixing matrix $\mathbf{W}$ such that the transformed signals $\hat{s}$ are as statistically independent as possible:

$$ \hat{s} = \mathbf{Wx} \qquad (16.68) $$

where ideally $\mathbf{W} \approx \mathbf{A}^{-1}$, leading to an approximate recovery of the independent source signals. The fundamental assumption that allows ICA to work is that the source signals $s_1, s_2, ..., s_n$ are **statistically independent**, meaning their joint probability density function (PDF) factorizes:

$$ p(s_1, s_2, ..., s_n) = p_1(s_1)p_2(s_2) \cdots p_n(s_n). \qquad (16.69) $$

This assumption of statistical independence is crucial because it allows the identification of the source signals based on **higher-order statistical properties**, such as kurtosis and negentropy. Furthermore, ICA relies on the **non-Gaussianity** of the source signals, as the **central limit**

<figure><img src="image_174.png" /><figcaption>Figure 16.6: Independent Component Analysis (ICA) of mixed signals. The top panel shows mixed signals, the middle panel shows true sources, and the bottom panel shows ICA recovered signals</figcaption></figure>

**theorem (CLT)** states that the sum of independent random variables tends toward a Gaussian distribution. Thus, if the sources were Gaussian, they could not be separated from their mixtures because Gaussian distributions are completely characterized by second-order statistics (mean and variance), and higher-order statistics would provide no additional information. To estimate the demixing matrix $\mathbf{W}$, we must find a transformation that maximizes the **statistical independence** of the recovered signals $\hat{s}$. Several measures of statistical independence exist, including **kurtosis**, **negentropy**, and **mutual information**, which lead to different optimization formulations of
ICA. One approach is to use **kurtosis**, which is defined as the fourth central moment of a random variable:

$$ \text{Kurt}(y) = \mathbb{E}[y^4] - 3(\mathbb{E}[y^2])^2. \qquad (16.70) $$

A Gaussian distribution has a kurtosis of zero, while non-Gaussian distributions have nonzero kurtosis. Since ICA relies on the assumption that the sources are **non-Gaussian**, we can maximize the absolute value of kurtosis to obtain independent components. Another widely used measure is **negentropy**, which is derived from information theory and defined as:

$$ J(y) = H(y_{\text{gauss}}) - H(y), \qquad (16.71) $$

where $H(y)$ is the differential entropy, given by:

$$ H(y) = - \int p(y) \log p(y) dy. \qquad (16.72) $$

Since entropy measures randomness, negentropy quantifies how far a given distribution is from Gaussianity. The higher the negentropy, the more non-Gaussian the distribution, making it a useful objective function for ICA. An alternative approach is to use **mutual information**, which measures the statistical dependence between variables:

$$ I(y_1, y_2, \dots, y_n) = \sum_{i=1}^{n} H(y_i) - H(y). \qquad (16.73) $$

By minimizing mutual information, we can maximize the statistical independence of the estimated sources. In practical ICA implementations, preprocessing is often necessary to improve performance. A common preprocessing step is **whitening**, which ensures that the observed signals are uncorrelated and have unit variance. Whitening is achieved by first computing the covariance matrix of $x$:

$$ C = \mathbb{E}[xx^T]. \qquad (16.74) $$

Performing an **eigenvalue decomposition (EVD):**

$$ C = \mathbf{EDE}^T, \qquad (16.75) $$

where $\mathbf{E}$ is the matrix of eigenvectors and $\mathbf{D}$ is the diagonal matrix of eigenvalues. The whitened signals are then computed as:

$$ \mathbf{x}' = \mathbf{D}^{-1/2} \mathbf{E}^T \mathbf{x}. \qquad (16.76) $$

This transformation ensures that the covariance of $\mathbf{x}'$ is the identity matrix:

$$ \mathbb{E}[\mathbf{x}'\mathbf{x}^T] = \mathbf{I}. \qquad (16.77) $$

ICA has broad applications in signal processing, including **EEG and fMRI signal separation**, **speech and audio processing**, **financial time series analysis**, and **image processing**. It is particularly useful in solving the **cocktail party problem**, where multiple speakers' voices are separated from mixed audio recordings. The mathematical foundation of ICA makes it one of the most robust techniques for **blind signal separation**, and its various optimization formulations offer flexibility depending on the application.

### 16.1.9.4 Python Code to generate Figure 16.6 illustrating the Independent Component Analysis (ICA) of mixed signals

The Python code below produces the Figure 16.6 illustrating the Independent Component Analysis (ICA) of mixed signals.