Continued from previous page

<table><thead><tr><th>Authors (Year)</th><th>Title</th><th>Contribution</th></tr></thead><tbody><tr><td>Mokan et al. (2025)</td><td>Pixel-Wise Classification of the Retinal Vasculature into Arteries and Veins</td><td>Uses decision trees for medical image segmentation, distinguishing arteries and veins in retinal scans, improving diagnostic precision.</td></tr></tbody></table>

### 16.2.11.3 Mathematical Analysis of Decision tree learning

Decision tree learning is a fundamental supervised learning algorithm used for classification and regression tasks. It recursively partitions the feature space into distinct regions by selecting the most informative feature at each step based on a splitting criterion. The goal of a decision tree is to create a model that predicts the value of a target variable by learning simple decision rules inferred from data features. Given a dataset $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^N$, where each $\mathbf{x}_i \in \mathbb{R}^d$ is a feature vector and $y_i$ is the corresponding output (either categorical for classification or continuous for regression), a decision tree recursively partitions $\mathbb{R}^d$ into disjoint regions $R_m$, where each region corresponds to a leaf node containing a prediction for $y$. The function learned by the decision tree can be expressed as a piecewise constant function:

$$f(\mathbf{x}) = \sum_{m=1}^{M} c_m \mathbb{1}(\mathbf{x} \in R_m), \qquad (16.216)$$

where $M$ is the total number of leaf nodes, $c_m$ is the prediction assigned to region $R_m$, and $\mathbb{1}(\mathbf{x} \in R_m)$ is an indicator function that is 1 if $\mathbf{x} \in R_m$ and 0 otherwise. The splitting criterion for a decision tree involves selecting the feature $j$ and threshold $s$ that best separate the data at each step. For classification, the impurity of a node is measured using a criterion such as the Gini impurity, defined as

$$G(R) = \sum_{k=1}^{K} p_k(1 - p_k), \qquad (16.217)$$

where $p_k$ is the proportion of samples in region $R$ belonging to class $k$, and $K$ is the total number of classes. Another common impurity measure is entropy, given by

$$H(R) = - \sum_{k=1}^{K} p_k \log p_k. \qquad (16.218)$$

For regression, the variance reduction is typically used, and the impurity at a node is given by the mean squared error (MSE):

$$\text{MSE}(R) = \frac{1}{|R|} \sum_{\mathbf{x}_i \in R} (y_i - \bar{y}_R)^2, \qquad (16.219)$$

where $\bar{y}_R$ is the mean of the target values in region $R$. The optimal split is found by maximizing the information gain, which is computed as

$$\Delta I = I(R) - \left( \frac{|R_L|}{|R|} I(R_L) + \frac{|R_R|}{|R|} I(R_R) \right), \qquad (16.220)$$

where $R_L$ and $R_R$ are the left and right child nodes obtained after splitting $R$, and $I(R)$ is the impurity measure (Gini, entropy, or MSE). The algorithm selects the feature $j^*$ and threshold $s^*$ that maximize $\Delta I$:

$$(j^*, s^*) = \arg \max_{j,s} \Delta I. \qquad (16.221)$$
Survival of passengers on the Titanic

<figure><img src="image_198.png" /><figcaption>Figure 16.30: Framework of the Decision Tree Learning</figcaption></figure>

Image Credit: By Gilgoldm - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=90405437

The diagram depicts passenger survival on the Titanic, with "sibsp" referring to the count of spouses or siblings aboard. Beneath each leaf, the figures represent the probability of survival and the percentage of passengers in that category. In essence, a higher chance of survival applied to those who were either (i) female or (ii) male, no older than 9.5 years, and traveling with fewer than three siblings.

The recursive splitting process continues until a stopping criterion is met, such as a maximum depth *D*, a minimum number of samples per leaf *n*<sub>min</sub>, or an impurity threshold *ε*. The depth of the tree, denoted *D*, determines the complexity of the model, and the number of terminal nodes *M* satisfies

$$M \le 2^D. \qquad (16.222)$$
Pruning is performed to prevent overfitting, and one common approach is cost complexity pruning, which minimizes the function

$$C(T) = \sum_{m=1}^{M} |R_m| H(R_m) + \alpha M, \qquad (16.223)$$

where $\alpha$ is a regularization parameter controlling the trade-off between tree complexity and accuracy. The optimal subtree $T^*$ is obtained by

$$T^* = \arg \min_{T} C(T). \qquad (16.224)$$

Decision tree learning is computationally efficient, with a worst-case training complexity of

$$O(Nd \log N), \qquad (16.225)$$

where $N$ is the number of samples and $d$ is the number of features. Despite its simplicity, decision trees can suffer from high variance, making ensemble methods such as random forests and gradient boosting necessary for improving generalization.

## 16.2.12 k-nearest neighbors algorithm

### 16.2.12.1 Literature Review of k-nearest neighbors (KNN) algorithm

Fix and Hodges (1951) [718] introduced the k-NN algorithm, laying the groundwork for non-parametric classification methods by proposing a technique where classification is based on the closest training examples in the feature space. Cover and Hart (1967) [719] provided a rigorous analysis of the k-NN algorithm's properties, demonstrating that as the size of the dataset approaches infinity, the error rate of the k-NN classifier is at most twice the Bayes error rate, establishing its strong consistency. Devroye et. al. (2013) [720] offered an in-depth theoretical analysis of pattern recognition methods, including k-NN, and discusses their probabilistic properties and performance bounds. Toussaint (2005) [721] explored the use of geometric proximity graphs, such as Voronoi diagrams and Delaunay triangulations, to enhance the efficiency and accuracy of k-NN classifiers by structuring the data to reflect inherent geometric relationships. Arya et. al. (1998) [723] introduced an efficient algorithm for approximate nearest neighbor search, addressing the computational challenges of k-NN in high-dimensional spaces by allowing approximate solutions with provable bounds on their accuracy. Terrell and Scott (1992) [724] discussed variable kernel density estimation techniques, which are closely related to k-NN methods, providing insights into adaptive methods for density estimation that can improve k-NN performance in varying data densities. Samworth (2012) [725] investigated the weighting schemes in k-NN classifiers, proposing optimal weighting strategies that enhance classification performance, especially in situations where the assumption of uniformity in data distribution does not hold. Bremner et. al. (2005) [726] presented algorithms that compute the decision boundaries of k-NN classifiers more efficiently, making the application of k-NN more practical for large datasets by focusing computational efforts on the most critical regions of the feature space. Ramaswamy et. al. (2000) [727] introduced methods for detecting outliers using k-NN concepts, which are essential for identifying anomalies in large datasets and have applications in fraud detection, network security, and data cleaning. Cover (1999) [728] provided a comprehensive introduction to information theory, including discussions on k-NN and its connections to concepts such as entropy and mutual information, offering a theoretical foundation for understanding the behavior of k-NN in the context of information theory.
<table><caption>Table 16.31: Summary of Contributions to k-nearest neighbors algorithm</caption><thead><tr><td>Reference</td><td>Key Contribution</td></tr></thead><tbody><tr><td>Fix and Hodges (1951)</td><td>Introduced the k-NN algorithm as a non-parametric classification method, forming the foundation of instance-based learning.</td></tr><tr><td>Cover and Hart (1967)</td><td>Provided theoretical analysis proving that the k-NN error rate is at most twice the Bayes error, establishing its consistency.</td></tr><tr><td>Devroye, Györfi and Lugosi (1996)</td><td>Developed a probabilistic framework for pattern recognition, including bounds on k-NN performance.</td></tr><tr><td>Toussaint (2005)</td><td>Explored geometric proximity graphs to improve k-NN efficiency by structuring data spatially.</td></tr><tr><td>Arya et al. (1998)</td><td>Introduced an optimal approximate nearest neighbor search algorithm for high-dimensional spaces.</td></tr><tr><td>Terrell and Scott (1992)</td><td>Discussed variable kernel density estimation, offering insights into adaptive density-based k-NN improvements.</td></tr><tr><td>Samworth (2012)</td><td>Proposed optimal weighted k-NN classifiers, improving performance in non-uniform distributions.</td></tr><tr><td>Bremner et al. (2005)</td><td>Developed output-sensitive algorithms for computing k-NN decision boundaries efficiently.</td></tr><tr><td>Ramaswamy et al. (2000)</td><td>Applied k-NN for outlier detection in large datasets, aiding in anomaly detection applications.</td></tr><tr><td>Cover and Thomas (1991)</td><td>Connected k-NN to information theory concepts such as entropy and mutual information.</td></tr></tbody></table>

### 16.2.12.2 Recent Literature Review of k-nearest neighbors (KNN) algorithm

Alaca and Emin (2024) [729] evaluated KNN as part of hybrid models for medical kidney image classification. It compares KNN with Support Vector Machines (SVM) and Random Forest (RF), showcasing its strengths and weaknesses in medical image analysis. Chen et. al. (2025) [730] proposed a Probability-Integrated Projection (PIP)-based KNN algorithm for epidemic management, improving classification accuracy in medical datasets by optimizing distance metrics. Liu et. al. (2025) [731] used KNN for predicting bond strength in dental materials. The study demonstrates that KNN provides robust accuracy when combined with kernel-based methods like SVM. Barghouthi et. al. (2025) [732] introduced a multi-channel fusion approach using KNN for predicting pressure injuries in hospitalized patients, integrating KNN with deep learning models. Jewan (2025) [733] examined KNN's effectiveness in remote sensing applications for crop yield prediction using UAV images, comparing it with Decision Trees and Random Forest. Moldovanu et. al. (2025) [734] studied how data corruption affects KNN's accuracy, offering methods to improve KNN's robustness using feature transformation. HosseinpourFardi and Alizadeh (2025) [735] proposed a hardware-accelerated KNN model for incremental learning, optimizing nearest-neighbor calculations in embedded systems. Afrin et. al. (2025) [736] used KNN to classify oil pipeline failure causes, demonstrating its reliability in industrial failure prediction. Hussain et. al. (2025) [737] evaluated KNN in geospatial flood susceptibility prediction, comparing it with RF and Extreme Gradient Boosting (XGBoost). Reddy and Murthy (2025) [738] combined Particle Swarm Optimization (PSO) with KNN to enhance accuracy in cardiovascular disease prediction.
<table><caption>Table 16.32: Summary of Recent Contributions to k-nearest neighbors algorithm</caption><thead><tr><td>Author (Year)</td><td>Summary of Contributions</td></tr></thead><tbody><tr><td>Alaca & Emin (2024)</td><td>Evaluates KNN within hybrid models for medical kidney image classification, comparing it with SVM and RF. Demonstrates how KNN performs in medical imaging tasks.</td></tr><tr><td>Chen, Hung & Yang (2025)</td><td>Proposes a Probability-Integrated Projection (PIP)-based KNN algorithm, improving classification accuracy for epidemic spread prediction through optimized distance metrics.</td></tr><tr><td>Liu et al. (2025)</td><td>Uses KNN for predicting bond strength of dental materials. Highlights KNN's performance against kernel-based methods like SVM.</td></tr><tr><td>Barghouthi et al. (2025)</td><td>Develops a multi-channel fusion model using KNN for pressure injury prediction in hospitalized patients, integrating it with deep learning techniques.</td></tr><tr><td>Jewan (2025)</td><td>Examines KNN's **effectiveness in remote sensing applications** for crop yield prediction using UAV images, comparing it with Decision Trees and RF.</td></tr><tr><td>Moldovanu et al. (2025)</td><td>Investigates how data corruption affects KNN's accuracy, proposing feature transformation techniques to improve robustness.</td></tr><tr><td>HosseinpourFardi & Alizadeh (2025)</td><td>Introduces a hardware-accelerated KNN model for incremental learning, optimizing KNN's performance in embedded systems.</td></tr><tr><td>Afrin et al. (2025)</td><td>Utilizes KNN to classify oil pipeline failure causes, demonstrating its effectiveness in industrial failure prediction.</td></tr><tr><td>Hussain et al. (2025)</td><td>Applies KNN to geospatial flood susceptibility prediction, comparing it with RF and XGBoost, showing its viability in disaster risk assessment.</td></tr><tr><td>Reddy & Murthy (2025)</td><td>Combines Particle Swarm Optimization (PSO) with KNN to enhance accuracy in cardiovascular disease prediction, demonstrating KNN's adaptability in medical applications.</td></tr></tbody></table>

### 16.2.12.3 Mathematical Analysis of k-nearest neighbors (KNN) algorithm

The k-nearest neighbors (KNN) algorithm is a fundamental instance-based learning method for classification and regression. It operates by determining the k nearest data points in the feature space and making predictions based on these neighbors. Formally, consider a dataset $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^N$, where $\mathbf{x}_i \in \mathbb{R}^d$ represents a $d$-dimensional feature vector, and $y_i$ represents the corresponding target variable. Given a new query point $\mathbf{x}_q$, the algorithm finds the k points $\{\mathbf{x}_{(i)}\}_{i=1}^k$ in $\mathcal{D}$ that are closest to $\mathbf{x}_q$ under a chosen distance metric.

Mathematically, the distance function $d(\mathbf{x}_q, \mathbf{x}_i)$ is commonly defined as the Euclidean distance:

$$d(\mathbf{x}_q, \mathbf{x}_i) = \|\mathbf{x}_q - \mathbf{x}_i\|_2 = \sqrt{\sum_{j=1}^{d} (x_{qj} - x_{ij})^2} \qquad (16.226)$$
<figure><img src="image_199.png" /><figcaption>Figure 16.31: Framework of the k-nearest neighbors (KNN) algorithm</figcaption></figure>

Image Credit: By Antti Ajanki AnAj - Own work, CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=2170282

The green dot represents the test sample, which needs to be classified as either a blue square or a red triangle. If the classification is based on $k = 3$ (solid-line circle), the sample is assigned to the red triangles because there are more triangles (2) than squares (1) within this region. However, with $k = 5$ (dashed-line circle), the sample is classified as a blue square, as the outer circle contains three squares and only two triangles

Alternatively, the Minkowski distance of order $p$ generalizes the Euclidean and Manhattan distances:

$$d(\mathbf{x}_q, \mathbf{x}_i) = \left( \sum_{j=1}^{d} |x_{qj} - x_{ij}|^p \right)^{\frac{1}{p}} \quad (16.227)$$

where $p = 1$ corresponds to the Manhattan distance and $p = 2$ corresponds to the Euclidean distance. For classification, the prediction is typically made using majority voting among the $k$ nearest neighbors. Let $\mathcal{N}_k(\mathbf{x}_q)$ be the set of indices of the $k$ nearest neighbors of $\mathbf{x}_q$. The predicted class label $\hat{y}_q$ is determined as:

$$\hat{y}_q = \arg \max_{c \in \mathcal{C}} \sum_{i \in \mathcal{N}_k(\mathbf{x}_q)} \mathbb{1}(y_i = c) \quad (16.228)$$
where $\mathbb{1}(\cdot)$ is the indicator function, and $\mathcal{C}$ represents the set of all possible class labels. If weighted voting is used, the contribution of each neighbor can be weighted by the inverse distance:

$$w_i = \frac{1}{d(\mathbf{x}_q, \mathbf{x}_i) + \epsilon} \qquad (16.229)$$

where $\epsilon$ is a small positive number to prevent division by zero. The weighted class probability estimate is then given by:

$$P(y_q = c) = \frac{\sum_{i \in \mathcal{N}_k(\mathbf{x}_q)} w_i \mathbb{1}(y_i = c)}{\sum_{i \in \mathcal{N}_k(\mathbf{x}_q)} w_i} \qquad (16.230)$$

and the final classification decision is:

$$\hat{y}_q = \arg \max_{c \in \mathcal{C}} P(y_q = c) \qquad (16.231)$$

For regression, the predicted value $\hat{y}_q$ is typically the mean of the k nearest neighbors' target values:

$$\hat{y}_q = \frac{1}{k} \sum_{i \in \mathcal{N}_k(\mathbf{x}_q)} y_i \qquad (16.232)$$

or, when distance weighting is applied,

$$\hat{y}_q = \frac{\sum_{i \in \mathcal{N}_k(\mathbf{x}_q)} w_i y_i}{\sum_{i \in \mathcal{N}_k(\mathbf{x}_q)} w_i} \qquad (16.233)$$

Computational complexity is a critical aspect of KNN. A naive search for the nearest neighbors requires computing distances from $\mathbf{x}_q$ to all $N$ data points, leading to a time complexity of $O(Nd)$. If a spatial data structure such as a k-d tree or a ball tree is used, the query time can be reduced to $O(\log N)$ in low-dimensional spaces. However, for high-dimensional data, the curse of dimensionality makes these structures less effective, often reverting to the brute-force $O(Nd)$ complexity. The choice of $k$ significantly affects KNN performance. A small $k$ can lead to high variance, whereas a large $k$ smooths the decision boundary but may introduce bias. The optimal $k$ is often determined via cross-validation, where the classification error is minimized as:

$$\hat{k} = \arg \min_k \sum_{i=1}^{N} \mathbb{1}(\hat{y}_i^{(k)} \neq y_i) \qquad (16.234)$$

The theoretical foundation of KNN can be analyzed in terms of consistency. Under mild assumptions, as $N \to \infty$ and $k \to \infty$ while $k/N \to 0$, the KNN classification error converges to the Bayes error rate:

$$\lim_{N \to \infty} \mathbb{E}[\mathbb{1}(\hat{y}_q \neq y_q)] = R^* \qquad (16.235)$$

For regression, under similar asymptotic conditions, the expected squared error converges to:

$$\lim_{N \to \infty} \mathbb{E}[(\hat{y}_q - y_q)^2] = \sigma^2 \qquad (16.236)$$

In high-dimensional spaces, the effectiveness of KNN diminishes due to the concentration of distances:

$$\lim_{d \to \infty} \frac{\max_i d(\mathbf{x}_q, \mathbf{x}_i) - \min_i d(\mathbf{x}_q, \mathbf{x}_i)}{\min_i d(\mathbf{x}_q, \mathbf{x}_i)} = 0 \qquad (16.237)$$

The KNN decision boundary is a piecewise linear approximation of the true decision surface. In the limit as $k \to 1$, KNN forms a Voronoi tessellation:

$$V_i = \{ \mathbf{x} \in \mathbb{R}^d \mid d(\mathbf{x}, \mathbf{x}_i) < d(\mathbf{x}, \mathbf{x}_j), \forall j \neq i \} \qquad (16.238)$$

For $k > 1$, decision regions are obtained by aggregating Voronoi cells. Given a sample $\mathbf{x}_q$, the probability of class $c$ is estimated as:

$$P(y_q = c | \mathbf{x}_q) = \frac{k_c}{k} \qquad (16.239)$$

Thus, KNN is a flexible, non-parametric method that relies on local structure for decision-making.
## 16.2.13 Similarity learning

### 16.2.13.1 Literature Review of Similarity learning

Similarity training has evolved as a fundamental concept in machine learning, influencing areas such as classification, ranking, image retrieval, and natural language processing. The work of Chen et al. (2009) [739] provided a rigorous foundation for similarity-based classification by analyzing different approaches for converting similarities into kernel functions. Their study systematically explored the mathematical properties of similarity measures and their effect on classification performance, offering insights into optimal ways to utilize nearest neighbor weights. Chechik et al. (2010) [740] expanded upon this by introducing OASIS, an online learning algorithm designed to handle large-scale image similarity tasks efficiently. By leveraging a bilinear similarity function and optimizing with a large-margin criterion, OASIS achieved state-of-the-art performance in ranking tasks while maintaining computational feasibility. Similarly, Huang et al. (2013) [741] extended similarity learning into the domain of content-based image retrieval (CBIR) by integrating relative comparisons, an approach grounded in ranking theory. This refinement aligned image retrieval more closely with real-world applications, ensuring that retrieved images were ordered based on their true visual similarities rather than absolute feature distances.

The theoretical underpinnings of similarity-based learning were further explored by Kar and Jain (2011) [744], who proposed a framework for mapping similarity functions to data-driven embeddings. Their work addressed the crucial question of how similarity functions can be interpreted in terms of data separability, providing a bridge between metric learning and traditional feature-based classifiers. In a different but related direction, Xiao et al. (2011) [743] tackled the problem of positive and unlabeled learning (PU learning), where similarity functions were used to weight ambiguous examples and improve the classification of data points with uncertain labels. This approach demonstrated how similarity-based techniques could enhance learning in scenarios where labeled data is scarce or incomplete, making it particularly relevant for applications such as anomaly detection and biomedical classification.

With the advent of deep learning, similarity learning has expanded into more complex and high-dimensional data spaces. Yang et al. (2024) [742] conducted an extensive survey on deep learning approaches for similarity computation, covering applications ranging from sequence matching to graph-based similarity models. Their review provided a critical examination of various neural network architectures designed to capture similarity relationships in data, highlighting key challenges such as overfitting, interpretability, and computational efficiency. Additionally, contributions from Wikipedia contributors [746] have documented the broader theoretical framework of semantic similarity, including traditional node-based and edge-based methods for quantifying textual similarity. Co-citation proximity analysis, as explored in recent research, introduced an innovative way to determine document similarity by leveraging citation networks, demonstrating how similarity measures can be extended beyond direct content analysis to relational data structures.

From a practical implementation perspective, PingCAP (2024) [745] evaluated a range of tools for computing semantic similarity in natural language processing, including transformer-based models like BERT and classical vector representations such as Word2Vec. Their analysis underscored the trade-offs between model complexity, computational cost, and accuracy in different NLP applications. Finally, Choi (2022) [747] applied similarity scoring techniques to document retrieval and clustering, demonstrating how fine-grained textual similarity assessments can enhance information retrieval systems. By incorporating deep learning models, they showcased improvements in contextual understanding, making similarity-based approaches increasingly vital in modern AI applications. Collectively, these works highlight the evolution of similarity training, demonstrating its growing importance across disciplines while underscoring the interplay between theoretical
advancements and real-world applications.

<table><caption>Table 16.33: Summary of Contributions in Similarity Training</caption><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Chen et al. (2009)</td><td>Established a mathematical foundation for similarity-based classification by systematically converting similarity measures into kernels and evaluating their performance in various learning scenarios.</td></tr><tr><td>Chechik et al. (2010)</td><td>Developed OASIS, an efficient online algorithm that learns a bilinear similarity function for large-scale image ranking, optimizing a margin-based criterion to enhance ranking performance.</td></tr><tr><td>Wang et al. (2013)</td><td>Proposed a similarity learning framework for content-based image retrieval (CBIR) that incorporates relative comparisons, aligning retrieval results with human perception of image similarity.</td></tr><tr><td>Kar and Jain (2011)</td><td>Introduced a similarity embedding framework that maps similarity functions into data-driven feature spaces, bridging the gap between similarity learning and traditional classification approaches.</td></tr><tr><td>Liu et al. (2011)</td><td>Addressed positive and unlabeled (PU) learning by leveraging similarity-based weighting of uncertain examples, improving classification performance when labeled data is scarce.</td></tr><tr><td>Zhang et al. (2024)</td><td>Conducted an extensive survey on deep learning techniques for similarity learning, covering applications in sequence modeling, graph-based learning, and high-dimensional data similarity computation.</td></tr><tr><td>Wikipedia Contributors</td><td>Documented theoretical aspects of semantic similarity, including classical methods such as node-based and edge-based similarity computations, and their applications in knowledge representation.</td></tr><tr><td>PingCAP (2024)</td><td>Explored NLP tools like Word2Vec and BERT for semantic similarity computation, analyzing trade-offs between computational complexity and accuracy in various language processing applications.</td></tr><tr><td>ResearchGate Contributors (2023)</td><td>Demonstrated the application of similarity-based scoring techniques in document retrieval and clustering, showcasing improvements in contextual understanding using deep learning models.</td></tr><tr><td>Co-citation Proximity Analysis</td><td>Investigated citation-based similarity measures, introducing co-citation proximity analysis to quantify the relationship between academic articles based on their citation network structures.</td></tr></tbody></table>

### 16.2.13.2 Recent Literature Review of Similarity learning

Nanyonga et. al. (2025) [707] presented a transformer-based approach for predicting causes of aviation incidents using similarity training. It introduces a multi-head attention mechanism that evaluates patterns in historical incident data to improve predictive accuracy. The model achieves a
similarity score of 0.697 with a standard deviation of ±0.153, highlighting the effectiveness of the similarity training approach. Fan and Chung (2025) [708] leveraged similarity-based training to classify crops in UAV-captured images. It employs RGB and vegetation indices (VARI) for training, optimizing performance by balancing training and testing datasets with an 80/20 split. The model improves classification accuracy by using similarity metrics to refine feature extraction. Bakaev et. al. (2025) [709] explored similarity-based training methods to enhance synthetic text generation by Large Language Models (LLMs). Using cosine similarity and Mahalanobis distance, the authors analyze how closely generated text matches human-authored content, showcasing the effectiveness of similarity-based methods in controlling model outputs. Ahn et. al. (2025) [711] employed similarity training using the Dice Similarity Coefficient to fine-tune a deep learning model for medical image segmentation. The proposed method significantly improves precision in identifying standard imaging planes, demonstrating how similarity metrics enhance training effectiveness. Peng et. al. (2025) [712] introduced similarity label supervision to refine visual place recognition by improving re-ranking mechanisms. By integrating descriptor similarity into the training process, the model enhances recognition performance even in challenging environmental conditions. Zhao et. al. (2025) [713] employed similarity-based global distance measures to cluster data while preserving privacy in federated learning. The authors integrate Generative Adversarial Networks (GANs) to refine the training process, ensuring robust clustering in distributed environments. Wang et. al. (2025) [714] developed a similarity matrix (W matrix) to predict genomic traits in maize hybrids. By incorporating similarity measures into training, the model improves yield and moisture content predictions across different environmental conditions. Xu et. al. (2025) [715] introduced a similarity loss function in medical image registration. By optimizing image similarity during training, the authors achieve enhanced alignment accuracy for medical imaging applications. Sun et. al. (2025) [716] investigated how similarity training affects text generation in LLMs. It evaluates ROUGE-1 similarity scores to measure how synthetic text diverges from human-generated content, guiding training strategies for better alignment with human language patterns. Liang et. al. (2025) [717] presented a similarity-based training approach to improve lip-to-speech synthesis. The model learns to generate natural-sounding speech without explicit speaker embeddings, achieving high speaker similarity through deep learning methods.

<table><caption>Table 16.34: Summary of Recent Contributions in Similarity Training</caption><thead><tr><td>Reference</td><td>Title</td><td>Contribution</td></tr></thead><tbody><tr><td>Nanyonga et al. (2025)</td><td>Multi-Head Attention-Based Transformer Model for Predicting Causes in Aviation Incidents</td><td>Introduces a transformer-based similarity training approach to predict aviation incidents with a multi-head attention mechanism, achieving a similarity score of 0.697.</td></tr><tr><td>Fan & Chung (2025)</td><td>Integrating Image Processing Technology and Deep Learning to Identify Crops in UAV Orthoimages</td><td>Uses similarity training to classify crops in UAV images, leveraging RGB and vegetation indices (VARI) to improve accuracy.</td></tr><tr><td>Bakaev et al. (2025)</td><td>Who Will Author the Synthetic Texts?</td><td>Implements cosine similarity and Mahalanobis distance for evaluating synthetic text similarity, improving text coherence in LLMs.</td></tr><tr><td>Ahn et al. (2025)</td><td>Deep Learning-Based Automated Guide for Developmental Dysplasia of the Hip Screening</td><td>Employs the Dice Similarity Coefficient to optimize deep learning-based segmentation in medical imaging.</td></tr></tbody></table>

*Continued on next page*
(Continued from previous page)

<table><thead><tr><td>Reference</td><td>Title</td><td>Contribution</td></tr></thead><tbody><tr><td>Peng et al. (2025)</td><td>Range and Bird's Eye View Fused Cross-Modal Visual Place Recognition</td><td>Introduces similarity label supervision to refine visual place recognition through descriptor similarity search.</td></tr><tr><td>Zhao et al. (2025)</td><td>Privacy-Preserved Federated Clustering with Non-IID Data via GANs</td><td>Uses similarity-based clustering and GANs to enhance privacy-preserved federated learning.</td></tr><tr><td>Wang et al. (2025)</td><td>Accurate Genomic Prediction for Maize Hybrids Using Multi-Environment Data</td><td>Develops a similarity matrix (W matrix) for genomic prediction, improving maize yield and moisture content predictions.</td></tr><tr><td>Xu et al. (2025)</td><td>Medical Image Registration Meets Vision Foundation Model: Prototype Learning and Contour Awareness</td><td>Introduces similarity loss functions to optimize image registration in medical imaging applications.</td></tr><tr><td>Sun et al. (2025)</td><td>Idiosyncrasies in Large Language Models</td><td>Evaluates similarity training effects on LLM-generated text, analyzing divergence from human-authored content using ROUGE-1 similarity scores.</td></tr><tr><td>Liang et al. (2025)</td><td>NaturalL2S: High-Quality Multi-Speaker Lip-to-Speech Synthesis</td><td>Employs similarity-based training to enhance lip-to-speech synthesis, achieving high speaker similarity.</td></tr></tbody></table>

### 16.2.13.3 Mathematical Analysis of Similarity learning

Similarity learning is a fundamental paradigm in machine learning and mathematical optimization that focuses on learning a function that maps inputs to a representation space where similar inputs are placed closer together, while dissimilar inputs are pushed farther apart. Mathematically, given a dataset $\mathcal{D} = \{(\mathbf{x}_i, \mathbf{x}_j, y_{ij})\}_{i,j=1}^N$, where $\mathbf{x}_i, \mathbf{x}_j \in \mathbb{R}^d$ are feature vectors and $y_{ij} \in \{0, 1\}$ is a similarity label (where $y_{ij} = 1$ indicates similarity and $y_{ij} = 0$ indicates dissimilarity), similarity learning aims to learn a function $f: \mathbb{R}^d \to \mathbb{R}^m$ such that a distance metric $D(f(\mathbf{x}_i), f(\mathbf{x}_j))$ satisfies

$$D(f(\mathbf{x}_i), f(\mathbf{x}_j)) \leq \epsilon \quad \text{if } y_{ij} = 1 \qquad (16.240)$$

$$D(f(\mathbf{x}_i), f(\mathbf{x}_j)) > \epsilon \quad \text{if } y_{ij} = 0 \qquad (16.241)$$

for some threshold $\epsilon > 0$. A common choice for $D$ is the Euclidean distance,

$$D(f(\mathbf{x}_i), f(\mathbf{x}_j)) = \|f(\mathbf{x}_i) - f(\mathbf{x}_j)\|_2. \qquad (16.242)$$

A central approach in similarity learning is metric learning, where the goal is to learn a distance metric $M$ that parameterizes a Mahalanobis-like distance

$$D_M(\mathbf{x}_i, \mathbf{x}_j) = \sqrt{(\mathbf{x}_i - \mathbf{x}_j)^T M (\mathbf{x}_i - \mathbf{x}_j)} \qquad (16.243)$$

where $M$ is a positive semi-definite (PSD) matrix ($M \succeq 0$), ensuring that $D_M$ satisfies the properties of a metric:

$$D_M(\mathbf{x}_i, \mathbf{x}_j) \geq 0, \quad D_M(\mathbf{x}_i, \mathbf{x}_j) = 0 \iff \mathbf{x}_i = \mathbf{x}_j \qquad (16.244)$$

$$D_M(\mathbf{x}_i, \mathbf{x}_j) = D_M(\mathbf{x}_j, \mathbf{x}_i), \quad D_M(\mathbf{x}_i, \mathbf{x}_k) \leq D_M(\mathbf{x}_i, \mathbf{x}_j) + D_M(\mathbf{x}_j, \mathbf{x}_k). \qquad (16.245)$$
<figure><img src="image_200.png" /><figcaption>Figure 16.32: Autoencoder architecture</figcaption></figure>

Image Credit: https://developers.google.com/machine-learning/clustering/dnn-clustering/overview

Metric learning can be formulated as an optimization problem where we minimize a loss function that enforces similarity constraints. One such loss function is the contrastive loss

$$L = \sum_{i,j} y_{ij} D_M(\mathbf{x}_i, \mathbf{x}_j)^2 + (1 - y_{ij}) \max(0, \alpha - D_M(\mathbf{x}_i, \mathbf{x}_j))^2 \quad (16.246)$$

where $\alpha$ is a margin parameter. This loss ensures that similar samples are pulled together while dissimilar samples are pushed apart beyond a margin. Another widely used loss function is the triplet loss, which operates on triplets $(\mathbf{x}_a, \mathbf{x}_p, \mathbf{x}_n)$, where $\mathbf{x}_a$ is an anchor, $\mathbf{x}_p$ is a positive example, and $\mathbf{x}_n$ is a negative example. The triplet loss is defined as

$$L = \sum_{(a,p,n)} \max(0, D(f(\mathbf{x}_a), f(\mathbf{x}_p)) - D(f(\mathbf{x}_a), f(\mathbf{x}_n)) + \alpha). \quad (16.247)$$

Here, the objective is to ensure that the distance between the anchor and the positive is smaller than the distance between the anchor and the negative by at least a margin $\alpha$. If we represent the embedding function $f(\mathbf{x})$ as a deep neural network parameterized by $\theta$, then similarity learning becomes a deep learning problem, where the network parameters are optimized via stochastic gradient descent to minimize one of the loss functions described above. A particularly effective approach in similarity learning is the use of a Siamese network, where two identical neural networks $f_\theta$ share weights and process two input vectors $\mathbf{x}_i$ and $\mathbf{x}_j$. The network learns a representation such that the Euclidean distance

$$D(f_{\theta}(\mathbf{x}_i), f_{\theta}(\mathbf{x}_j)) = \|f_{\theta}(\mathbf{x}_i) - f_{\theta}(\mathbf{x}_j)\|_2 \quad (16.248)$$
<figure><img src="image_201.png" /><figcaption>Figure 16.33: Predictor architecture</figcaption></figure>

Image Credit:

https://developers.google.com/machine-learning/clustering/dnn-clustering/overview

is small for similar pairs and large for dissimilar pairs. The optimization is driven by minimizing a loss function such as the contrastive or triplet loss. Another approach is the use of graph-based similarity learning, where a similarity graph $G = (V, E)$ is constructed over the dataset, and embeddings are learned by enforcing that connected nodes are closer together in the embedding space. This can be formulated as

$$L = \sum_{(i,j) \in E} w_{ij} D(f(\mathbf{x}_i), f(\mathbf{x}_j))^2 \qquad (16.249)$$

where $w_{ij}$ is a weight representing the strength of similarity between $\mathbf{x}_i$ and $\mathbf{x}_j$. Thus, similarity learning encompasses a vast array of methods, from classical metric learning with Mahalanobis distances to deep learning approaches with Siamese networks, triplet loss, and self-supervised contrastive learning, all of which fundamentally rely on defining and optimizing a notion of similarity in a mathematically rigorous manner.

# 16.3 Self learning

## 16.3.1 Literature Review of Self Learning

Self-learning in artificial intelligence has evolved through various paradigms, including reinforcement learning, self-supervised learning, meta-learning, and curiosity-driven exploration. A foundational work in this domain is Schmidhuber's (1991) [123] research on curiosity-driven model-building, which introduced the concept of **intrinsic motivation** in learning systems. He proposed
that an agent should actively seek **novel experiences** by maximizing learning progress, thereby enhancing its predictive capabilities. This principle has been extensively applied in reinforcement learning (RL) to encourage **efficient exploration** in complex environments, forming the basis of modern **exploration bonuses** used in deep RL. Reinforcement learning itself was rigorously formalized by Sutton (2018) [286] and Barto (2021) [287], who introduced the **Markov Decision Process (MDP)** framework and core algorithms like **temporal difference (TD) learning** and **actor-critic architectures**. Their work established **self-learning via trial-and-error**, wherein an agent refines its policy through **rewards and penalties**, ultimately enabling autonomous decision-making without explicit supervision. Building upon these foundations, Silver et al. (2017) [851] demonstrated how **self-play** in AlphaZero allows an agent to master games like chess, shogi, and Go **without human data**, utilizing deep RL with **Monte Carlo Tree Search (MCTS)** to iteratively refine strategies through self-competition.

Beyond reinforcement-based methods, **self-supervised learning (SSL)** has emerged as a crucial paradigm for learning **without labeled data**. Bengio et al. (2009) [283] introduced **curriculum learning**, which simulates the human cognitive process by training models on simpler tasks before progressing to complex ones. This approach has been shown to significantly **improve convergence rates and generalization** in deep learning. He et al. (2020) [852] further advanced SSL through **contrastive learning**, where Momentum Contrast (MoCo) enables a model to **differentiate instances** based on learned feature representations, even in the absence of labels. Grill et al. (2020) [853] introduced **Bootstrap Your Own Latent (BYOL)**, a breakthrough demonstrating that self-learning representations do not require **negative pairs** (a key contrastive learning element). Instead, BYOL employs an **online and target network** paradigm to iteratively refine embeddings, proving that deep neural networks can **self-learn meaningful features** without explicit contrastive mechanisms. This non-contrastive learning methodology has been a significant step toward **fully autonomous feature extraction**, eliminating the need for carefully curated negative samples in self-learning tasks.

In the context of **hierarchical self-learning**, Hinton et al. (2006) [854] proposed **Deep Belief Networks (DBNs)**, pioneering **layer-wise unsupervised pretraining**, which allows deep networks to learn robust feature hierarchies. This method was pivotal in **self-learning hierarchical representations** and directly influenced architectures like **autoencoders**, **GANs**, and **modern deep networks**. Similarly, **meta-learning**, or “learning how to learn,” was formalized by Finn et al. (2017) [855] through **Model-Agnostic Meta-Learning (MAML)**. Their framework demonstrated that deep networks could be optimized to rapidly adapt to **new tasks** using only a few gradient steps, showcasing self-learning’s potential in **few-shot learning and robotics**. Additionally, Jaderberg et al. (2017) [856] introduced **auxiliary self-learning objectives** in RL, where an agent trains on **unsupervised auxiliary tasks**, such as predicting future states, to **enhance representation learning** and sample efficiency. This method demonstrated that reinforcement learning agents could **self-discover useful intermediate goals**, thereby accelerating policy convergence and improving generalization across multiple tasks.

Finally, the **transformer revolution** has brought self-learning capabilities to vision tasks, as demonstrated by Dosovitskiy et al. (2020) [857] with **Vision Transformers (ViTs)**. They showed that **self-attention mechanisms**—originally developed for natural language processing—could replace convolutional architectures, allowing deep networks to **self-learn image representations** without handcrafted priors. This introduced a paradigm shift where models no longer rely on **spatial inductive biases**, instead learning **global dependencies** in a self-supervised manner. Collectively, these contributions have rigorously established self-learning as a fundamental principle in artificial intelligence, spanning multiple disciplines and application areas. From curiosity-driven exploration and reinforcement-based self-play to self-supervised feature learning and meta-
learning, self-learning methodologies now underpin **state-of-the-art AI systems**, enabling them to **autonomously acquire knowledge, optimize representations, and generalize across domains** with minimal or no human intervention.

<table><caption>Table 16.35: Summary of Contributions in Self-Learning</caption><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Schmidhuber (1991)</td><td>Introduced curiosity-driven learning and intrinsic motivation for self-learning agents.</td></tr><tr><td>Sutton and Barto (1998)</td><td>Formalized reinforcement learning through MDPs, TD learning, and actor-critic architectures.</td></tr><tr><td>Silver et al. (2017)</td><td>Demonstrated self-play in AlphaZero, mastering games without human input using MCTS.</td></tr><tr><td>Bengio et al. (2009)</td><td>Introduced curriculum learning, simulating human cognitive progression in deep learning.</td></tr><tr><td>He et al. (2020)</td><td>Developed contrastive learning through Momentum Contrast (MoCo) for self-supervised learning.</td></tr><tr><td>Grill et al. (2020)</td><td>Introduced BYOL, proving self-supervised learning without negative pairs.</td></tr><tr><td>Hinton et al. (2006)</td><td>Pioneered deep belief networks (DBNs) for hierarchical self-learning representations.</td></tr><tr><td>Finn et al. (2017)</td><td>Formalized model-agnostic meta-learning (MAML) for few-shot learning.</td></tr><tr><td>Jaderberg et al. (2017)</td><td>Proposed auxiliary self-learning objectives in RL to enhance policy learning.</td></tr><tr><td>Dosovitskiy et al. (2020)</td><td>Developed Vision Transformers (ViTs), enabling self-learning representations in vision tasks.</td></tr></tbody></table>

## 16.3.2 Recent Literature Review of Self Learning

Self-learning artificial intelligence (AI) has made significant advances across diverse domains, including ethical considerations, scientific research, healthcare, and education. The first major contribution is from Mousavi (2025) [858], who explores the ethical implications of self-aware AGI, questioning whether such AI systems should be granted moral consideration akin to conscious beings. This work utilizes fuzzy logic to determine whether AI self-awareness equates to possessing a "soul," raising crucial debates about data deletion, AI rights, and governance. Similarly, Cui et al. (2025) [860] propose a dual-level self-supervised learning framework to enhance the generalization capabilities of AI in interatomic potential modeling. By improving AI's ability to adapt to out-of-distribution data, their work has a profound impact on computational chemistry and materials science. Meanwhile, Jia et al. (2025) [861] apply self-supervised graph learning techniques to molecular property prediction. Their research focuses on fragment-level feature fusion, using retrosynthetic fragmentation algorithms to improve drug discovery by learning molecular representations without labeled data. These studies highlight the expanding role of self-learning AI in scientific and ethical discussions.

In healthcare and medical applications, self-learning AI continues to demonstrate transformative capabilities. Liu et al. (2025) [863] introduce a self-optimized reinforcement learning algorithm for elective surgery scheduling, significantly enhancing hospital efficiency by dynamically optimizing multi-specialty, multi-stage procedures. Additionally, Song et al. (2025) [864] develop a deep self-supervised learning framework for time series classification, optimizing AI's ability to detect anomalies in various domains, including finance and industrial monitoring. In another critical medical application, Chaudary et al. (2025) [870] design an EEG-based emotion recognition system uti-
lizing explainable AI. Their model interprets neural activity patterns to enhance human-computer interaction, which is vital for mental health monitoring and affective computing. Expanding further into neuroscience, Tautan et al. (2025) [871] present a systematic review of unsupervised learning methods applied to epilepsy detection using EEG data. Their work highlights AI's ability to classify seizures with minimal human intervention, improving diagnostic accuracy and reducing the burden on neurologists. These contributions showcase AI's ability to autonomously learn from complex biomedical signals, paving the way for intelligent decision-making in healthcare.

The role of self-learning AI extends beyond the medical field into education and cognitive sciences. Hou (2025) [862] investigates the integration of AI-supported learning environments with students' psychological factors such as self-esteem and mindfulness. Their findings suggest that AI-driven adaptive learning platforms foster personalized education, optimizing student performance through tailored feedback mechanisms. A related study by Li et al. (2025) [869] explored generative AI's role in real-time adaptive scaffolding for self-regulated learning. By analyzing student behaviors in real time, AI dynamically generates learning resources that adapt to individual needs, significantly enhancing personalized education strategies. Furthermore, Bjerregaard et al. (2025) [859] examine how self-supervised learning can be applied to structural biology, particularly in protein folding studies. Their research provides a foundation for AI-driven drug discovery and molecular engineering by leveraging AI models that can learn protein structures from vast amounts of unlabeled data. These studies highlight the increasing influence of self-learning AI in shaping education, scientific research, and personalized cognitive support.

Across these domains, self-learning AI is emerging as a revolutionary tool capable of independently acquiring knowledge, recognizing patterns, and making informed decisions without human intervention. Its impact ranges from ethical AI governance to improving biomedical diagnostics, optimizing industrial and financial systems, and reshaping education. The convergence of self-learning AI with reinforcement learning, self-supervised learning, and generative AI models demonstrates its versatility in tackling real-world challenges. By continuously refining its ability to learn from limited data, detect anomalies, and optimize decision-making, self-learning AI is poised to drive future innovations in science, healthcare, and education. These rigorous contributions underscore the growing necessity of developing AI systems that not only automate tasks but also refine their understanding of complex environments, leading to more autonomous and intelligent machines.

<table><caption>Table 16.36: Summary of Recent Contributions in Self-Learning</caption><thead><tr><td>Paper</td><td>Contribution</td></tr></thead><tbody><tr><td>Mousavi (2025)</td><td>Examines the ethical implications of self-aware AGI and whether it possesses moral standing. Utilizes fuzzy logic to assess AI consciousness, influencing AI governance debates.</td></tr><tr><td>Bjerregaard et al. (2025)</td><td>Demonstrates the application of self-supervised learning to structural biology, improving molecular structure prediction and advancing computational drug discovery.</td></tr><tr><td>Cui et al. (2025)</td><td>Develops a dual-level self-supervised learning model to enhance generalization in physics-based AI, particularly in interatomic potential modeling.</td></tr><tr><td>Jia et al. (2025)</td><td>Introduces a graph-based self-supervised learning model for molecular property prediction, utilizing retrosynthetic fragmentation to improve AI-driven drug design.</td></tr></tbody></table>

*Continued on next page*
(Continued from previous page)

<table><thead><tr><td>Paper</td><td>Contribution</td></tr></thead><tbody><tr><td>Hou (2025)</td><td>Investigates the psychological effects of AI-driven adaptive learning on self-esteem and academic mindfulness, highlighting AI's role in personalized education.</td></tr><tr><td>Liu et al. (2025)</td><td>Proposes a reinforcement learning-based scheduling system for elective surgeries, optimizing hospital efficiency and reducing patient wait times.</td></tr><tr><td>Song et al. (2025)</td><td>Develops a deep self-supervised learning framework for anomaly detection in time-series data, improving AI applications in finance and industry.</td></tr><tr><td>Li et al. (2025)</td><td>Explores generative AI's ability to provide real-time adaptive scaffolding for personalized self-regulated learning, enhancing online education strategies.</td></tr><tr><td>Chaudary et al. (2025)</td><td>Presents an EEG-based AI model for emotion recognition using self-learning algorithms, improving human-computer interaction and mental health monitoring.</td></tr><tr><td>Tautan et al. (2025)</td><td>Conducts a systematic review of unsupervised learning methods for epilepsy detection using EEG data, showcasing AI's diagnostic potential in neurology.</td></tr></tbody></table>

### 16.3.3 Mathematical Analysis of Self Learning

Self-learning is a process by which an entity, be it biological or artificial, acquires knowledge, refines understanding, and adapts behavior without direct external instruction. Mathematically, self-learning can be framed in the context of optimization, function approximation, and iterative improvement of hypotheses, drawing upon formalism from information theory, machine learning, and dynamical systems. Given an unknown target function $f: X \to Y$, self-learning aims to construct an approximation $\hat{f}$ such that

$$ \hat{f} \approx f \qquad (16.250) $$

in some well-defined sense, typically minimizing an objective function $J(\hat{f})$ over an appropriate hypothesis space $\mathcal{H}$. If we let $x_i \in X$ denote inputs and $y_i \in Y$ denote corresponding outputs, then a fundamental formulation of self-learning can be described as an empirical risk minimization (ERM) problem:

$$ \min_{\hat{f} \in \mathcal{H}} \sum_{i=1}^{N} L(y_i, \hat{f}(x_i)) \qquad (16.251) $$

where $L$ is a loss function that quantifies the discrepancy between the actual outcome $y_i$ and the predicted outcome $\hat{f}(x_i)$. In an idealized self-learning system, the function $\hat{f}$ evolves dynamically as additional information is acquired, which can be described by an update rule:

$$ \hat{f}^{(t+1)} = \hat{f}^{(t)} - \eta \nabla J(\hat{f}^{(t)}) \qquad (16.252) $$

where $\eta$ is a learning rate and $\nabla J(\hat{f})$ denotes the gradient of the loss function with respect to the model parameters. A crucial aspect of self-learning is the incorporation of feedback mechanisms, whereby an entity refines its internal model based on past errors. This can be framed in terms of stochastic gradient descent (SGD):

$$ \theta^{(t+1)} = \theta^{(t)} - \eta \nabla_{\theta} L(y_i, f_{\theta}(x_i)) \qquad (16.253) $$

where $\theta$ represents the parameters of the function approximator $f_{\theta}$. If the system has access to reinforcement signals rather than explicit input-output pairs, self-learning aligns with reinforcement
<figure><img src="image_202.png" /><figcaption>Figure 16.34: Decision boundary of a Self-Learning (Self-Training) classifier. The classifier is initially trained on labeled data, predicts labels on unlabeled data, and iteratively updates itself using confident predictions. The contour regions represent predicted classes, and scatter points show the original data points</figcaption></figure>

learning, where the goal is to maximize an expected cumulative reward:

$$J(\pi) = \mathbb{E} \left[ \sum_{t=0}^{\infty} \gamma^t r_t \mid \pi \right] \qquad (16.254)$$

where $\pi$ is a policy mapping states to actions, $r_t$ is a reward function, and $\gamma$ is a discount factor. The optimal policy satisfies the Bellman equation:

$$Q^*(s, a) = \mathbb{E} \left[ r + \gamma \max_{a'} Q^*(s', a') \mid s, a \right] \qquad (16.255)$$

which recursively expresses the expected return of a state-action pair. In an unsupervised setting, self-learning often manifests through clustering or density estimation, where the objective is to model the underlying distribution $P(X)$. One way to formalize this is through maximum likelihood estimation:

$$\max_{\theta} \sum_{i=1}^{N} \log P(x_i | \theta) \qquad (16.256)$$

which seeks to find model parameters $\theta$ that best explain the observed data. A more flexible formulation involves variational inference, where we approximate the posterior distribution using a tractable function $q_{\phi}(Z | X)$ and minimize the Kullback-Leibler (KL) divergence:

$$D_{\text{KL}}(q_{\phi}(Z | X) \parallel P(Z | X)) = \mathbb{E}_{q_{\phi}} \left[ \log q_{\phi}(Z | X) - \log P(Z | X) \right] \qquad (16.257)$$
which ensures that $q_\phi$ is close to the true posterior $P(Z | X)$. Self-learning in neural systems can be framed using Hebbian learning principles, where synaptic weights $w_{ij}$ evolve according to activity correlations:

$$ \Delta w_{ij} = \eta x_i x_j \qquad (16.258) $$

or more generally through spike-timing-dependent plasticity (STDP):

$$ \Delta w_{ij} = \eta e^{-\tau/\tau_0} (x_i x_j) \qquad (16.259) $$

where $\tau$ represents the temporal offset between presynaptic and postsynaptic activations. In deep learning, self-learning often involves self-supervised contrastive learning, which optimizes an objective of the form:

$$ \min_{\theta} \sum_{(x,x^+)} -\log \frac{\exp(\mathrm{sim}(f_{\theta}(x), f_{\theta}(x^+))/\tau)}{\sum_{x^-} \exp(\mathrm{sim}(f_{\theta}(x), f_{\theta}(x^-))/\tau)} \qquad (16.260) $$

where $x^+$ is a positive sample, $x^-$ represents negative samples, and $\mathrm{sim}(\cdot, \cdot)$ measures similarity in representation space. This encourages self-learned representations to be invariant under transformations. Mathematically, self-learning is deeply connected to information-theoretic principles, particularly in maximizing the mutual information between learned representations $Z$ and the data $X$:

$$ I(Z; X) = H(Z) - H(Z | X) \qquad (16.261) $$

which captures the reduction in uncertainty about $Z$ given knowledge of $X$. A self-learning system thus seeks to construct representations that maximize $I(Z; X)$ while discarding task-irrelevant noise. More generally, if self-learning operates within a Bayesian framework, it continuously updates a posterior belief over hypotheses $\mathcal{H}$ using Bayes' theorem:

$$ P(h | D) = \frac{P(D | h)P(h)}{P(D)} \qquad (16.262) $$

where $D$ represents the accumulated data. Finally, in the context of dynamical systems, self-learning can be modeled as a time-dependent process governed by differential equations, such as the Riccati equation in adaptive control:

$$ \frac{dP}{dt} = A^T P + PA - PBR^{-1}B^T P + Q \qquad (16.263) $$

where $P$ represents a learning-induced adaptation of system dynamics. Self-learning is thus an iterative, evolving process that refines internal models through optimization, probabilistic inference, feedback-driven adaptation, and information maximization.

### 16.3.4 Python Code to generate Figure 16.34

The Python code below produces the Figure 16.34 illustrating the Self-Learning.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn datasets import makeClassification
from sklearn.naive_bayes import GaussianNB
from sklearn.semi_supervised import SelfTrainingClassifier

# Generate synthetic 2D classification data
X, y = makeClassification(n Samples=200, n features=2, n informative=2,
                           n redundant=0, n clusters per class=1,
                           random state=42)

# Mask some labels to simulate unlabeled data
```
```python
12     rng = np.random.RandomState(42)
13     mask_unlabeled = rngrand(len(y)) < 0.7  # 70% unlabeled
14     y_missing = y.copy()
15     y_missing[mask_unlabeled] = -1  # -1 indicates unlabeled in sklearn
16
17     # Base classifier
18     base_clf = GaussianNB()
19
20     # Self-training classifier
21     self_training_clf = SelfTrainingClassifier(base_clf, criterion='k_best',
22                                             k_best=50)
23     self_training_clf.fit(X, y_missing)
24
25     # Plot decision boundary
26     x_min, x_max = X[: ,0].min() - 1, X[: ,0].max() + 1
27     y_min, y_max = X[: ,1].min() - 1, X[: ,1].max() + 1
28     xx, yy = np.meshgrid(np.linspace(x_min, x_max, 200),
29                         np.linspace(y_min, y_max, 200))
29     Z = self_training_clf Predict(np.c[xx ravel(), yy ravel()])
30     Z = Z reshape(xx.shape)
31
32     plt.figure(figsize=(8,6))
33     plt.contourf(xx, yy, Z, alpha=0.3, cmap= plt.cm.Paired)
34     plt scatter(X[:,0], X[:,1], c=y, edgecolor='k', cmap= plt.cm.Paired)
35     plt.xlabel("Feature 1")
36     plt.ylabel("Feature 2")
37     plt.title("Self-Learning (Self-Training) Classifier Decision Boundary")
38     plt.grid(True, ls='--')
39     plt.tight_layout()
40     plt.show()
```

## 16.4 Reinforcement learning

### 16.4.1 Literature Review of Reinforcement learning

Reinforcement Learning (RL) has evolved from fundamental theoretical concepts to sophisticated algorithms capable of solving high-dimensional decision-making problems. The foundational work by Bellman (1954) [891] introduced Dynamic Programming (DP), establishing the Bellman equations that recursively compute optimal value functions in Markov Decision Processes (MDPs). This formulation became the mathematical backbone for reinforcement learning, particularly in value iteration and policy iteration methods. Decades later, Sutton and Barto (1998, 2018) [286] [287] formalized RL as a computational framework for sequential decision-making, introducing core concepts such as temporal difference (TD) learning, policy evaluation, and actor-critic architectures. Their seminal book remains the primary reference for both theoretical research and practical applications, covering both classical algorithms like Q-learning (Watkins and Dayan, 1992) [893] and deep reinforcement learning techniques. The work of Watkins and Dayan (1992) [893] is particularly crucial, as it introduced Q-learning, a model-free off-policy RL algorithm that enables an agent to learn an optimal policy without requiring an explicit model of the environment. The proof of convergence of Q-learning under certain conditions laid the foundation for later deep RL algorithms such as Deep Q Networks (DQN).

A major milestone in RL came with Mnih et al. (2015) [292], who introduced Deep Q Net-
works (DQN), combining deep neural networks with Q-learning to solve high-dimensional control problems. Their method incorporated experience replay to break correlation in training data and target networks for stable Q-value updates, allowing for the first superhuman performance on Atari games using raw pixel inputs. The introduction of deep RL opened the door to applications in robotics, game AI, and autonomous systems. The subsequent work of Silver et al. (2016) [892] extended deep RL into combinatorial search problems, with AlphaGo integrating Monte Carlo Tree Search (MCTS) with deep policy and value networks. This demonstrated the potential of RL in solving long-horizon planning problems and led to AlphaZero, which generalizes the method to various board games, achieving superhuman performance in Go, chess, and shogi without human supervision. Parallel to these advances, Konda and Tsitsiklis (2000) [293] introduced actor-critic algorithms, which separate the value function estimator (critic) from the policy update mechanism (actor), leading to more stable policy learning. This framework influenced modern policy optimization techniques such as Proximal Policy Optimization (PPO), developed by Schulman et al. (2017) [894], which uses a clipped surrogate objective to ensure stable updates without excessive deviation from the current policy.

In the realm of continuous control, Lillicrap et al. (2016) [895] introduced Deep Deterministic Policy Gradient (DDPG), an off-policy actor-critic method that enables RL in high-dimensional continuous action spaces. Unlike DQN, which operates in discrete action spaces, DDPG employs a deterministic policy with target networks and batch normalization for stable learning. Building on these ideas, Haarnoja et al. (2018) [291] proposed Soft Actor-Critic (SAC), incorporating entropy maximization to encourage exploration and improve sample efficiency. SAC's stochastic policy formulation and automatic entropy adjustment mechanism set a new standard for continuous control tasks in RL. Meanwhile, Levine et al. (2016) [294] demonstrated the feasibility of end-to-end learning for robotics, directly mapping visual inputs to control actions using deep reinforcement learning. Their guided policy search method combined RL with supervised learning to improve sample efficiency, paving the way for RL-based autonomous robotic systems. These collective advancements, spanning from fundamental RL principles to modern deep learning integration, have established reinforcement learning as a powerful tool for solving complex decision-making and control problems in diverse applications.

<table><caption>Table 16.37: Summary of Key Contributions in Reinforcement Learning</caption><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Bellman (1957)</td><td>Introduced Dynamic Programming, laying the mathematical foundation for reinforcement learning. Developed the Bellman equation, which enables recursive computation of optimal policies in Markov Decision Processes (MDPs).</td></tr><tr><td>Sutton and Barto (1998, 2018)</td><td>Formalized reinforcement learning as a computational framework. Introduced key concepts such as temporal difference (TD) learning, actor-critic methods, and policy evaluation. Their textbook serves as the primary resource for both theoretical and applied RL.</td></tr><tr><td>Watkins and Dayan (1992)</td><td>Developed Q-learning, a model-free off-policy algorithm that enables agents to learn optimal policies without requiring an explicit model of the environment. Provided proof of Q-learning's convergence under certain conditions.</td></tr><tr><td>Mnih et al. (2015)</td><td>Introduced Deep Q Networks (DQN), combining deep learning with Q-learning to handle high-dimensional state spaces. Innovations include experience replay and target networks, leading to stable and sample-efficient training. Achieved superhuman performance on Atari games.</td></tr></tbody></table>
<table><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Silver et al. (2016)</td><td>Developed AlphaGo, which integrated Monte Carlo Tree Search (MCTS) with deep reinforcement learning. Demonstrated the ability to learn complex planning tasks with deep policy and value networks. Paved the way for AlphaZero, which generalized the approach to chess and shogi.</td></tr><tr><td>Konda and Tsitsiklis (2000)</td><td>Proposed actor-critic methods, separating policy learning (actor) from value estimation (critic). Their framework improved policy stability and inspired modern policy gradient methods such as Proximal Policy Optimization (PPO).</td></tr><tr><td>Schulman et al. (2017)</td><td>Developed Proximal Policy Optimization (PPO), a policy gradient method using a clipped objective function to stabilize training. PPO is widely used due to its balance between sample efficiency and simplicity.</td></tr><tr><td>Lillicrap et al. (2016)</td><td>Introduced Deep Deterministic Policy Gradient (DDPG), extending reinforcement learning to continuous action spaces. Utilized deterministic policies, target networks, and batch normalization to improve stability.</td></tr><tr><td>Haarnoja et al. (2018)</td><td>Developed Soft Actor-Critic (SAC), which incorporates entropy maximization for improved exploration and stability. SAC's stochastic policy formulation and automatic entropy adjustment enhanced sample efficiency.</td></tr><tr><td>Levine et al. (2016)</td><td>Applied reinforcement learning to robotics using guided policy search. Combined RL with supervised learning to improve sample efficiency, demonstrating end-to-end learning from raw sensory inputs to control outputs.</td></tr></tbody></table>

## 16.4.2 Recent Literature Review of Reinforcement learning

Reinforcement Learning (RL) has seen substantial advancements in recent research, spanning diverse applications from security and AI planning to industrial systems and environmental monitoring. A key development in RL is its application to cybersecurity, as highlighted by Shah's (2025) [896] research on adversarial vulnerabilities in RL models. This study systematically investigates how RL-based AI systems can be compromised through adversarial attacks and proposes mitigation strategies to enhance robustness. This security-oriented approach is complemented by Ajanovi et al. (2025) [897], who bridge the gap between AI planning and RL by integrating structured planning methods with reinforcement learning to create interpretable and reliable decision-making frameworks. Their work addresses a long-standing challenge in AI: ensuring that RL-based decision models remain understandable and controllable, a crucial aspect in mission-critical applications. Similarly, Oliveira et al. (2025) [898] introduce spatial cluster detection using RL, revolutionizing geospatial analysis by allowing RL models to identify non-regular spatial clusters, which traditional statistical techniques struggle to detect.

Beyond planning and security, RL is making strides in multi-agent decision-making systems. Hengzhi et al. (2025) [899] leverage multi-agent reinforcement learning (MARL) for UAV relay covert communication, optimizing real-time transmission strategies for secure aerial data networks. This work highlights the potential of RL in decentralized, high-stakes environments where autonomous agents must coordinate under uncertainty. The concept of multi-task learning is further expanded by Pan et al. (2025) [900], who propose a Markov Decision Process (MDP)-based task grouping approach that enables RL models to learn multiple tasks efficiently. This innovation addresses a fundamental
limitation of RL: its need for extensive task-specific training, making it more scalable across diverse problem domains. Similarly, Liu et al. (2025) [901] explore multi-hop knowledge graph reasoning, applying RL to enhance automated reasoning processes in knowledge systems, significantly improving the efficiency of complex decision chains.

RL is also proving instrumental in energy and infrastructure resilience. Chen et al. (2025) [902] propose an interpretable RL framework for building energy management, ensuring that deep RL models remain transparent while optimizing energy consumption in smart building systems. Their work addresses a key limitation of black-box AI models by extracting human-readable decision rules, bridging the gap between efficiency and interpretability. Anwar and Akber (2025) [903] extend RL applications to structural resilience, using multi-agent deep RL to enhance the durability of buildings under extreme environmental conditions. This work is pivotal in modern urban planning, demonstrating how RL can optimize interconnected physical infrastructures for enhanced resilience. Zhao et al. (2025) [904] apply RL-enhanced long short-term memory (LSTM) networks to predictive maintenance, improving industrial fault detection by optimizing data-driven health assessments of rolling bearings. By integrating RL with deep learning models, they achieve superior early fault detection, reducing operational risks in industrial automation.

Finally, RL is shaping mental health support and empathetic AI. Soman et al. (2025) [905] introduce reinforcement learning-enhanced retrieval-augmented generation (RAG) models for mental health AI agents, where RL fine-tunes AI-generated responses to provide more personalized and empathetic support. This integration is crucial in human-AI interactions, ensuring that mental health support systems align with human emotional needs. Overall, these studies collectively demonstrate the growing versatility of reinforcement learning across disciplines, tackling challenges in security, multi-agent coordination, AI transparency, and intelligent automation. As RL techniques continue to evolve, they are poised to redefine decision-making, infrastructure resilience, and human-centric AI interactions in unprecedented ways.

<table><caption>Table 16.38: Summary of Key Contributions in Recent Reinforcement Learning</caption><thead><tr><td>Authors, Year, and Source</td><td>Key Contribution</td></tr></thead><tbody><tr><td>H. Shah (2025), ResearchGate</td><td>Explores the security vulnerabilities of reinforcement learning (RL) models to adversarial attacks and proposes mitigation techniques for model robustness.</td></tr><tr><td>B. Hengzhi, W. Haichao, H. Rongrong, et al. (2025), Chinese Journal of Aeronautics (Elsevier)</td><td>Uses multi-agent reinforcement learning (MARL) to optimize UAV relay covert communication, improving security and efficiency in real-time transmission.</td></tr><tr><td>R. Pan, Q. Yuan, G. Luo, B. Chen, et al. (2025), SSRN</td><td>Introduces a novel Markov Decision Process (MDP) graph-based approach to improve sample efficiency in multi-task reinforcement learning (MTRL).</td></tr><tr><td>G. Soman, M.V. Judy, A.M. Abou (2025), Cognitive Systems Research (Elsevier)</td><td>Applies reinforcement learning to enhance Retrieval-Augmented Generation (RAG) for AI-driven mental health support systems.</td></tr><tr><td>D.R.X. Oliveira, G.J.P. Moreira, A.R. Duarte (2025), Environmental and Ecological Statistics (Springer)</td><td>Develops RL-based spatial cluster detection techniques to improve geospatial data analysis beyond traditional statistical methods.</td></tr></tbody></table>

(Continued on next page)
(Continued from previous page)

<table><thead><tr><td>Authors, Year, and Source</td><td>Key Contribution</td></tr></thead><tbody><tr><td>Z. Ajanovi, T. Gros, F. Den Hengst, et al. (2025), AAAI Conference on Artificial Intelligence (IBM Research)</td><td>Integrates AI planning techniques with RL to create more interpretable and structured decision-making models.</td></tr><tr><td>H. Chen, W. Guo, W. Bao, et al. (2025), Energy and Buildings (Elsevier)</td><td>Introduces an interpretable RL framework for energy management in smart buildings, ensuring transparency in decision-making.</td></tr><tr><td>H. Liu, D. Li, B. Zeng, Y. Xu (2025), Applied Intelligence (Springer)</td><td>Enhances multi-hop reasoning tasks by using RL to optimize knowledge graph reasoning efficiency.</td></tr><tr><td>W. Zhao, Y. Lv, K.M. Lee, et al. (2025), Computers and Industrial Engineering (Elsevier)</td><td>Uses reinforcement learning-enhanced LSTM models to improve predictive maintenance and fault detection in industrial systems.</td></tr><tr><td>G.A. Anwar, M.Z. Akber (2025), Computers and Structures (Elsevier)</td><td>Applies multi-agent RL to optimize structural resilience under extreme environmental conditions, improving infrastructure durability.</td></tr></tbody></table>

### 16.4.3 Mathematical Analysis of Reinforcement learning

Reinforcement Learning (RL) is a framework for optimal decision-making in sequential environments where an agent interacts with a stochastic system to maximize a cumulative reward signal. Mathematically, RL is modeled as a Markov Decision Process (MDP), which is defined by the tuple $(S, A, P, R, \gamma)$, where $S$ is the state space, $A$ is the action space, $P(s'|s, a)$ is the transition probability of reaching state $s'$ from state $s$ after taking action $a$, $R(s, a)$ is the reward function that defines the immediate reward received after executing action $a$ in state $s$, and $\gamma \in [0, 1]$ is the discount factor that determines the weight of future rewards. The objective is to find an optimal policy $\pi^*$ that maximizes the expected cumulative reward over time, expressed as the return $G_t$, defined by

$$G_t = \sum_{k=0}^{\infty} \gamma^k R(s_{t+k}, a_{t+k}). \qquad (16.264)$$

The value function of a policy $\pi$, denoted as $V^\pi(s)$, represents the expected return starting from state $s$ while following policy $\pi$,

$$V^\pi(s) = \mathbb{E}^\pi \left[ \sum_{k=0}^{\infty} \gamma^k R(s_{t+k}, a_{t+k}) \mid s_t = s \right]. \qquad (16.265)$$

Similarly, the action-value function $Q^\pi(s, a)$ gives the expected return for taking action $a$ in state $s$ and subsequently following policy $\pi$,

$$Q^\pi(s, a) = \mathbb{E}^\pi \left[ \sum_{k=0}^{\infty} \gamma^k R(s_{t+k}, a_{t+k}) \mid s_t = s, a_t = a \right]. \qquad (16.266)$$

The Bellman equation characterizes the value function recursively,

$$V^\pi(s) = \sum_{a \in A} \pi(a | s) \sum_{s' \in S} P(s' | s, a) [R(s, a) + \gamma V^\pi(s')]. \qquad (16.267)$$

Similarly, the Bellman equation for the action-value function is given by

$$Q^\pi(s, a) = \sum_{s' \in S} P(s' | s, a) \left[ R(s, a) + \gamma \sum_{a' \in A} \pi(a' | s') Q^\pi(s', a') \right]. \qquad (16.268)$$
<figure><img src="image_203.png" /><figcaption>Figure 16.35: Framework of the Reinforcement Learning</figcaption></figure>

Image Credit: By Megajuice - Own work, CC0, https://commons.wikimedia.org/w/index.php?curid=57895741

An optimal policy $\pi^*$ satisfies

$$V^*(s) = \max_{\pi} V^{\pi}(s), \qquad (16.269)$$

and the optimal state-value function satisfies the Bellman optimality equation,

$$V^*(s) = \max_{a \in A} \sum_{s' \in S} P(s' | s, a) [R(s, a) + \gamma V^*(s')]. \qquad (16.270)$$

Similarly, the optimal action-value function satisfies

$$Q^*(s, a) = \sum_{s' \in S} P(s' | s, a) \left[ R(s, a) + \gamma \max_{a' \in A} Q^*(s', a') \right]. \qquad (16.271)$$

The policy iteration algorithm alternates between policy evaluation, which computes $V^\pi(s)$, and policy improvement, which updates the policy as

$$\pi'(s) = \arg \max_{a \in A} Q^\pi(s, a). \qquad (16.272)$$
<figure><img src="image_204.png" /><figcaption>Figure 16.36: Q-Learning policy in a 4x4 grid world. Each cell shows the optimal action learned by the agent (↑, →, ↓, ←) to reach the goal. The agent learns the optimal path using reinforcement learning</figcaption></figure>

In contrast, value iteration directly updates $V(s)$ using the Bellman optimality equation iteratively,

$$V_{k+1}(s) = \max_{a \in A} \sum_{s' \in S} P(s' | s, a) [R(s, a) + \gamma V_k(s')]. \quad (16.273)$$

Temporal Difference (TD) learning updates value estimates based on observed transitions, using

$$V(s_t) \leftarrow V(s_t) + \alpha [R(s_t, a_t) + \gamma V(s_{t+1}) - V(s_t)]. \quad (16.274)$$

The Q-learning algorithm updates the action-value function using the rule

$$Q(s_t, a_t) \leftarrow Q(s_t, a_t) + \alpha \left[ R(s_t, a_t) + \gamma \max_{a'} Q(s_{t+1}, a') - Q(s_t, a_t) \right]. \quad (16.275)$$

where $\alpha$ is the learning rate. In deep reinforcement learning, function approximators such as neural networks parameterized by $\theta$ approximate $Q$, where updates follow

$$\theta \leftarrow \theta + \alpha \nabla_{\theta} \left( R(s_t, a_t) + \gamma \max_{a'} Q_{\theta}(s_{t+1}, a') - Q_{\theta}(s_t, a_t) \right). \quad (16.276)$$
The policy gradient method optimizes a stochastic policy $\pi_{\theta}(a | s)$ by following the gradient of the expected return, given by

$$ \nabla_{\theta} J(\theta) = \mathbb{E} \left[ \sum_{t=0}^{\infty} \nabla_{\theta} \log \pi_{\theta}(a_t | s_t) G_t \right]. \qquad (16.277) $$

### 16.4.4 Python Code to generate Figure 16.36

The Python code below produces the Figure 16.36 illustrating the Q-Learning policy.

```python
import numpy as np
import matplotlib.pyplot as plt

# Simple 4x4 grid world
grid_size = 4
n_states = grid_size * grid_size
nActions = 4  # up, right, down, left

# Rewards: goal at bottom-right corner
R = np(zeros((n_states, nActions)))
goal = n_states - 1

for s in range(n_states):
    for a in range(nActions):
        next_state = s
        row, col = divmod(s, grid_size)
        if a == 0 and row > 0: # up
            next_state -= grid_size
        elif a == 1 and col < grid_size-1: # right
            next_state += 1
        elif a == 2 and row < grid_size-1: # down
            next_state += grid_size
        elif a == 3 and col > 0: # left
            next_state -= 1
        if next_state == goal:
            R[s,a] = 1.0

# Q-learning parameters
Q = np(zeros((n_states, nActions)))
alpha = 0.1
gamma = 0.9
epsilon = 0.2
n_ePodes = 5000

# Q-learning algorithm
for _ in range(n_ePodes):
    s = 0  # start state
    while s != goal:
        if np.random rand() < epsilon:
            a = np.random.choice(nActions)
        else:
            a = np maxim(Q[s])
        # Compute next state
        row, col = divmod(s, grid_size)
        if a == 0 and row > 0: next_s = s - grid_size
        elif a == 1 and col < grid_size-1: next_s = s + 1
```
```python
elif a == 2 and row < grid_size-1: next_s = s + grid_size
elif a == 3 and col > 0: next_s = s - 1
else: next_s = s
# Reward
r = R[s,a]
# Q update
Q[s,a] = Q[s,a] + alpha*(r + gamma*np.max(Q(next_s]) - Q[s,a])
s = next_s

# Extract policy
policy = [policy_arrows(npargmax(Q[s])) for s in range(n_states)]
policy_grid = np.array(policy).reshape((grid_size, grid_size))

# Plot policy
plt.figure(figsize=(6,6))
for i in range(grid_size):
    for j in range(grid_size):
        plt.text(j, grid_size-1-i, policy_grid[i,j], ha='center', va='center')
        plt.xlim(-0.5, grid_size-0.5)
        plt.ylim(-0.5, grid_size-0.5)
        plt.xticks(range(grid_size))
        plt.yticks(range(grid_size))
        plt.grid(True)
        plt.title("Q-Learning Policy in 4x4 Grid World")
        plt.show()
```

## 16.4.5 Policy Gradient Method

### 16.4.5.1 Literature Review of Policy Gradient Method

Policy gradient methods form a crucial class of reinforcement learning algorithms that directly optimize policies using gradient ascent techniques. The foundational work by Sutton et al. (1999) [906] introduced policy gradient methods in the context of function approximation, rigorously deriving an unbiased estimate of the gradient of the expected return. This work established the basis for parameterizing policies explicitly and developing gradient-based optimization techniques that ensure convergence under certain assumptions. Kakade (2001) [908] extended this framework by introducing the Natural Policy Gradient (NPG) method, which leverages the Fisher information matrix to normalize policy updates, thereby making learning more efficient and stable. The key contribution of NPG lies in its ability to make gradient updates invariant to policy parameterization, which enhances convergence properties and mitigates issues related to vanishing or exploding gradients.

Building upon these theoretical foundations, Schulman et al. (2015) [909] proposed Trust Region Policy Optimization (TRPO), which ensures monotonic improvement in policy performance by constraining the step size via the Kullback-Leibler (KL) divergence between successive policies. This theoretically motivated approach prevents drastic policy updates that could lead to performance degradation. TRPO was further refined by Schulman et al. (2017) [894] through Proximal Policy Optimization (PPO), which simplifies the optimization process by employing a clipped surrogate objective function. PPO strikes a balance between sample efficiency and computational feasibility, making it one of the most widely adopted algorithms in deep reinforcement learning. In parallel, Agarwal et al. (2021) [910] rigorously analyzed the optimality and sample complexity of policy gradient methods, particularly investigating how function approximation errors and distribution
shifts affect their theoretical performance guarantees. Their work provided valuable insights into the conditions under which policy gradient methods converge to optimal or near-optimal policies.

Recent theoretical advancements, such as the work by Liu et al. (2024) [911], have further deepened our understanding of policy optimization by analyzing projected policy gradients and natural policy gradients in the context of discounted Markov Decision Processes (MDPs). By deriving novel convergence bounds, they provided an elementary yet rigorous framework that elucidates the dynamics of policy updates in large-scale reinforcement learning tasks. Meanwhile, Lorberbom et al. (2020) [912] proposed Direct Policy Gradients (DirPG), an alternative approach specifically designed for discrete action spaces. Their method optimizes policies by directly maximizing expected return-to-go trajectories, offering a novel perspective that integrates domain knowledge into policy optimization. Complementary to these efforts, McCracken et al. (2020) [913] studied policy gradient methods in exactly solvable Partially Observable Markov Decision Processes (POMDPs), deriving analytical results that characterize their probabilistic convergence behavior. Their findings are significant as they provide a rigorous theoretical foundation for understanding policy gradients in partially observable settings.

Lastly, practical considerations and comparative studies have played a crucial role in refining policy gradient methodologies. A definitive guide by Lehmann (2024) [914] systematically explores on-policy policy gradient methods in deep reinforcement learning, presenting a rigorous discussion of entropy regularization, KL divergence constraints, and their impact on stability. Furthermore, comparative analyses of policy-gradient algorithms done by Sutton et. al. (2000) [916] provide both theoretical and empirical evaluations of their efficiency and convergence properties, guiding practitioners in selecting the most suitable methods for various reinforcement learning applications. Collectively, these contributions have significantly advanced the theoretical underpinnings and practical implementations of policy gradient methods, shaping their role as a fundamental tool in modern reinforcement learning.

<table><caption>Table 16.39: Summary of Key Contributions in Policy Gradient Methods</caption><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Sutton et al. (1999)</td><td>Introduced policy gradient methods with function approximation, deriving an unbiased gradient estimator for direct policy optimization. Established the foundation for parameterized policies independent of value functions.</td></tr><tr><td>Kakade (2001)</td><td>Developed Natural Policy Gradient (NPG), which utilizes the Fisher information matrix to normalize gradient updates, making learning invariant to policy parameterization. Improved stability and efficiency in policy optimization.</td></tr><tr><td>Schulman et al. (2015)</td><td>Proposed Trust Region Policy Optimization (TRPO), a theoretically motivated approach that enforces a KL divergence constraint on policy updates, ensuring monotonic performance improvement and preventing catastrophic performance drops.</td></tr><tr><td>Schulman et al. (2017)</td><td>Introduced Proximal Policy Optimization (PPO), which simplifies TRPO by using a clipped surrogate objective, striking a balance between computational efficiency and stability, making it widely adopted in deep reinforcement learning.</td></tr><tr><td>Agarwal et al. (2021)</td><td>Provided theoretical analysis on policy gradient optimality, sample complexity, and performance under distribution shift, giving insights into when policy gradient methods effectively converge to near-optimal solutions.</td></tr></tbody></table>

*Continued on next page*
*Continued from previous page*

<table><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Liu et al. (2024)</td><td>Conducted a rigorous theoretical study on projected and natural policy gradients in discounted Markov Decision Processes (MDPs), deriving convergence rates and properties of different policy optimization methods.</td></tr><tr><td>Lorberbom et al. (2020)</td><td>Developed Direct Policy Gradients (DirPG), an approach optimized for discrete action spaces that maximizes return-to-go trajectories, allowing integration of domain knowledge into policy learning.</td></tr><tr><td>McCracken et al. (2020)</td><td>Analyzed policy gradient methods in exactly solvable Partially Observable Markov Decision Processes (POMDPs), deriving analytical results on value distributions and probabilistic convergence behavior.</td></tr><tr><td>Lehmann (2024)</td><td>Provided a definitive theoretical guide to policy gradients in deep reinforcement learning, covering entropy regularization, KL divergence constraints, and their impact on sample efficiency and stability.</td></tr><tr><td>Sutton et. al. (2000)</td><td>Conducted a comparative study on policy-gradient algorithms, evaluating their theoretical convergence properties and empirical efficiency, guiding practitioners in selecting appropriate methods.</td></tr></tbody></table>

### 16.4.5.2 Recent Literature Review of Policy Gradient Method

Policy Gradient Methods in Reinforcement Learning (RL) have been widely adopted across various domains, leading to significant advancements in optimization, decision-making, and automation. One of the most notable implementations is in vehicular networks, where Mustafa et al. (2025) [917] leverage Proximal Policy Optimization (PPO) to manage computation offloading in vehicular communication systems. Their policy gradient-based framework optimally distributes computational resources across edge devices, reducing latency and improving system efficiency. Similarly, Yang et al. (2025) [918] employ a Deep Deterministic Policy Gradient (DDPG) in a hierarchical reinforcement learning (HRL) setup to dynamically optimize vehicular edge computing resources, outperforming traditional heuristics in real-time environments. These approaches collectively demonstrate the power of policy gradient methods in optimizing network performance under dynamic conditions. Additionally, Jamshidiha et al. (2025) [919] integrate Deep Deterministic Policy Gradient (DDPG) with Graph Neural Networks (GNNs) to improve mobile user association in cellular networks, dynamically optimizing traffic distribution and resource allocation. These approaches collectively demonstrate the power of policy gradient methods in optimizing network performance under dynamic conditions.

In the realm of robotic and autonomous control, policy gradient methods have enabled significant progress in task-specific learning. Raei et al. (2025) [920] introduce a DDPG-based reinforcement learning framework for robotic nonprehensile manipulation, particularly focusing on object sliding. Their study presents an efficient policy optimization technique that enhances robotic dexterity in handling objects without grasping them. Extending this concept to UAV (Unmanned Aerial Vehicle) applications, Ting-Ting et al. (2025) [921] propose a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) approach, which allows multiple UAVs to make collaborative, autonomous decisions while operating under strict communication constraints. Similarly, Zhang et al. (2025) [922] integrate Deep Deterministic Policy Gradient (DDPG) with neuromorphic computing to develop a novel Hybrid Deep Deterministic Policy Gradient (Neuro-HDDPG) algorithm. This approach enhances obstacle avoidance for spherical underwater robots by mimicking the decision-making process of biological neural networks, significantly improving their autonomy in complex
underwater environments.

Policy gradient methods have also found profound applications in reinforcement learning for natural language processing (NLP) and constrained decision-making problems. Nguyen et al. (2025) [923] explore REINFORCE and RELAX policy gradient algorithms to fine-tune text-to-SQL models, effectively improving structured query generation accuracy by optimizing reward-based learning. Additionally, Chathuranga Brahmanage et al. (2025) [924] extend policy gradient techniques to action-constrained reinforcement learning by leveraging constraint violation signals, allowing agents to optimize their decision-making while adhering to predefined operational constraints. These studies highlight how policy gradient approaches can be adapted to complex, structured learning tasks that require precision and rule adherence.

Another major advancement in policy gradient methods is in federated learning and network optimization. Huang et al. (2025) [925] introduce the Knowledge Collaboration Actor-Critic Policy Gradient (KCACPG) algorithm, which facilitates knowledge transfer in cooperative traffic scheduling for intelligent transportation networks, ensuring seamless traffic management with minimal delays. Finally, Li et al. (2025) [926] propose FedDDPG, a federated learning variant of Deep Deterministic Policy Gradient, tailored for vehicle trajectory prediction in decentralized learning environments. This framework enables autonomous vehicles to learn optimal path planning strategies without centralizing sensitive data, ensuring enhanced privacy and efficiency in large-scale intelligent transportation systems. Together, these contributions showcase the robustness of policy gradient methods across diverse applications, reinforcing their critical role in advancing deep reinforcement learning frameworks for real-world challenges.

<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Mustafa et al. (2025)</td><td>Utilizes Proximal Policy Optimization (PPO) to optimize offloading decisions in vehicular communication networks, reducing latency and enhancing efficiency.</td></tr><tr><td>Huang et al. (2025)</td><td>Introduces the Knowledge Collaboration Actor-Critic Policy Gradient (KCACPG) method to optimize reinforcement learning in traffic management using knowledge transfer.</td></tr><tr><td>Yang et al. (2025)</td><td>Employs Deep Deterministic Policy Gradient (DDPG) within a hierarchical reinforcement learning framework to optimize vehicular resource allocation.</td></tr><tr><td>Jamshidiha et al. (2025)</td><td>Combines Graph Neural Networks (GNN) with DDPG to improve mobile user association in cellular networks through dynamic traffic-aware optimization.</td></tr><tr><td>Raei et al. (2025)</td><td>Develops a DDPG-based framework for robotic nonprehensile manipulation, enabling efficient object sliding with adaptive policy gradient optimization.</td></tr><tr><td>Ting-Ting et al. (2025)</td><td>Proposes a Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm to optimize UAV coordination under limited communication conditions.</td></tr><tr><td>Zhang et al. (2025)</td><td>Integrates neuromorphic computing with reinforcement learning to develop a Hybrid Deep Deterministic Policy Gradient (Neuro-HDDPG) for underwater robot navigation.</td></tr></tbody></table>

*Continued on next page*
*Continued from previous page*

<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Nguyen et al. (2025)</td><td>Implements REINFORCE and RELAX policy gradient algorithms to improve text-to-SQL model fine-tuning with reward optimization.</td></tr><tr><td>Chathuranga Brahmanage et al. (2025)</td><td>Introduces a constraint-aware policy gradient algorithm that learns from violation signals, optimizing decision-making within strict operational constraints.</td></tr><tr><td>Li et al. (2025)</td><td>Proposes FedDDPG, a federated learning extension of Deep Deterministic Policy Gradient (DDPG) to optimize vehicle trajectory prediction while preserving data privacy.</td></tr></tbody></table>

### 16.4.5.3 Mathematical Analysis of Policy Gradient Method

The policy gradient method is a fundamental approach in reinforcement learning that directly optimizes the policy $\pi_{\theta}(a | s)$, parameterized by $\theta$, through gradient ascent on an objective function that represents expected cumulative rewards. The policy $\pi_{\theta}(a | s)$ defines a probability distribution over actions given a state, and the goal is to adjust $\theta$ such that the expected return is maximized. The objective function, representing the expected return, is given by

$$J(\theta) = \mathbb{E}_{\tau \sim p_{\theta}(\tau)} [R(\tau)] \qquad (16.278)$$

where $\tau = (s_0, a_0, s_1, a_1, ...)$ represents a trajectory sampled from the environment following policy $\pi_{\theta}$, and $p_{\theta}(\tau)$ is the probability distribution over trajectories under policy $\pi_{\theta}$, given by the product of the initial state distribution $p(s_0)$, the policy probability, and the state transition dynamics:

$$p_{\theta}(\tau) = p(s_0) \prod_{t=0}^{T-1} \pi_{\theta}(a_t | s_t)p(s_{t+1} | s_t, a_t). \qquad (16.279)$$

The reward function $R(\tau)$ is typically defined as the sum of discounted rewards:

$$R(\tau) = \sum_{t=0}^{T-1} \gamma^t r(s_t, a_t), \qquad (16.280)$$

where $r(s_t, a_t)$ is the reward received at time step $t$ and $\gamma \in (0, 1]$ is the discount factor. The gradient of $J(\theta)$ with respect to $\theta$ is computed using the likelihood ratio trick:

$$\nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim p_{\theta}(\tau)} [R(\tau) \nabla_{\theta} \log p_{\theta}(\tau)]. \qquad (16.281)$$

Expanding the logarithm,

$$\nabla_{\theta} \log p_{\theta}(\tau) = \sum_{t=0}^{T-1} \nabla_{\theta} \log \pi_{\theta}(a_t | s_t), \qquad (16.282)$$

we obtain

$$\nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim p_{\theta}(\tau)} \left[ R(\tau) \sum_{t=0}^{T-1} \nabla_{\theta} \log \pi_{\theta}(a_t | s_t) \right]. \qquad (16.283)$$

This expectation is estimated using Monte Carlo sampling over trajectories:

$$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} R(\tau_i) \sum_{t=0}^{T-1} \nabla_{\theta} \log \pi_{\theta}(a_t^i | s_t^i), \qquad (16.284)$$
<figure><img src="image_205.png" /><figcaption>Figure 16.37: Policy Gradient Method: The policy network (green) outputs a distribution over actions given the state. The environment (gray) produces trajectories and rewards, which are stored (orange). The policy gradient update (red) adjusts the policy parameters using the gradient $\nabla_{\theta}J(\theta) = \mathbb{E}_{\pi_{\theta}}[\nabla_{\theta} \log \pi_{\theta}(a|s)R]$</figcaption></figure>

where $\tau_i$ are sampled trajectories. Instead of using raw returns $R(\tau)$, it is common to replace them with an advantage function $A(s_t, a_t)$ to reduce variance:

$$A(s_t, a_t) = Q(s_t, a_t) - V(s_t), \qquad (16.285)$$

where $Q(s_t, a_t)$ is the action-value function,

$$Q(s_t, a_t) = \mathbb{E} \left[ \sum_{t'=t}^{T-1} \gamma^{t'-t} r(s_{t'}, a_{t'}) | s_t, a_t \right], \qquad (16.286)$$

and $V(s_t)$ is the state-value function,

$$V(s_t) = \mathbb{E}_{a \sim \pi_{\theta}(\cdot|s_t)}[Q(s_t, a)]. \qquad (16.287)$$

Replacing $R(\tau)$ with $A(s_t, a_t)$, the gradient estimate becomes

$$\nabla_{\theta}J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \sum_{t=0}^{T-1} A(s_t^i, a_t^i) \nabla_{\theta} \log \pi_{\theta}(a_t^i | s_t^i). \qquad (16.288)$$

The parameters are updated using gradient ascent:

$$\theta \leftarrow \theta + \alpha \nabla_{\theta}J(\theta), \qquad (16.289)$$
where $\alpha > 0$ is the learning rate. A commonly used baseline function $b(s_t)$, often taken as $V(s_t)$, is subtracted to further reduce variance without introducing bias:

$$ \nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^{N} \sum_{t=0}^{T-1} (Q(s_t^i, a_t^i) - V(s_t^i)) \nabla_{\theta} \log \pi_{\theta}(a_t^i | s_t^i). \quad (16.290) $$

This formulation forms the basis of policy gradient methods, including REINFORCE, actor-critic algorithms, and proximal policy optimization (PPO), which introduce additional constraints and refinements to improve stability and efficiency.

### 16.4.5.4 Python Code to generate Figure 16.37

The Python code below produces the Figure 16.37 illustrating the Policy Gradient Method.

```python
import matplotlib.pyplot as plt
```

```python
import networkx as nx
```

```python
def build_pg_graph():
```

```python
G = nx.DiGraph()
```

```python
# Nodes
```

```python
G.add_node("Environment", type="env")
```

```python
G.add_node("Policy Network\nπθ(a|s)", type="actor")
```

```python
G.add_node("Trajectory / Rewards", type="memory")
```

```python
G.add_node("Policy Gradient Update", type="loss")
```

```python
# Edges
```

```python
G.add_edge("Policy Network\nπθ(a|s)", "Environment", label="Action a")
```

```python
G.add_edge("Environment", "Trajectory / Rewards", label="(s,a,r,s)")
```

```python
G.add_edge("Trajectory / Rewards", "Policy Gradient Update", label="Compute R")
```

```python
G.add_edge("Policy Gradient Update", "Policy Network\nπθ(a|s)", label="Update θ")
```

```python
return G
```

```python
def plot_pg.save_as=None):
```

```python
G = build_pg_graph()
```

```python
pos = {
```

```python
    "Policy Network\nπθ(a|s)": (0,2),
```

```python
    "Environment": (3,2),
```

```python
    "Trajectory / Rewards": (3,1),
```

```python
    "Policy Gradient Update": (6,1.5),
```

```python
}
```

```python
# Colors
```

```python
color_map = []
```

```python
for n, d in G.nodes(data=True):
```

```python
    if d["type"] == "actor":
```

```python
color_map.append("lightgreen")
```

```python
    elif d["type"] == "memory":
```

```python
color_map.append("orange")
```

```python
    elif d["type"] == "loss":
```

```python
color_map.append("salmon")
```
```python
elif d["type"] == "env":
    color_map.append("lightgrey")
else:
    color_map.append("white")

# Draw nodes
nx.draw_networkx_nodes(G, pos, node_color color_map, edgecolors="k",
                       node_size=2800)
nx.draw_networkx labels(G, pos, font_size=9)

# Draw edges with labels
ax = plt.gca()
for u, v, d in GEdges(data=True):
    ax annotate("",
              xy(pos[v], xycoordinates="data",
                  xytext(pos[u], textcoordinates="data",
                          arrowprops=dict(arrowstyle="->", color="black", lw
                                                =1.5)))
    if "label" in d:
        mid_x = (pos[u][0] + pos[v][0]) / 2
        mid_y = (pos[u][1] + pos[v][1]) / 2
        plt.text(mid_x, mid_y+0.15, d["label"], fontsize=8, ha="center")

plt.title("Policy Gradient Method Architecture", fontsize=12)
plt.axis("off")

if save_as:
    plt.savefig save_as, dπ=300, bbox_inches="tight")
    plt.show()

if __name__ == "__main__":
    plot pg save_as="policygradient.png")
```

## 16.4.6 Deep Reinforcement Learning

### 16.4.6.1 Literature Review of Deep Reinforcement Learning

Deep Reinforcement Learning (DRL) has undergone significant advancements over the past decade, driven by key innovations in value-based methods, policy optimization, and distributional approaches. The introduction of Deep Q-Networks (DQN) by Mnih et al. (2015) [292] marked a turning point, demonstrating that deep neural networks could successfully approximate Q-values for high-dimensional state spaces. The stability of DQN was ensured through experience replay, which mitigated correlation among consecutive updates, and target networks, which provided a stable reference for temporal difference updates. However, DQN was restricted to discrete action spaces, prompting Lillicrap et al. (2016) [895] to introduce the Deep Deterministic Policy Gradient (DDPG), an actor-critic method designed for continuous control problems. DDPG extended Q-learning to continuous actions by utilizing an off-policy approach with target smoothing, improving convergence in high-dimensional environments. Meanwhile, Schulman et al. (2015) [909], Schulman et al. (2017) [894] introduced Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO), which formulated policy optimization with explicit constraints on divergence from previous policies. PPO's clipped objective made it computationally efficient and less sensitive to hyperparameters, establishing it as a standard for training deep RL agents.
Building upon these foundations, Haarnoja et al. (2018) [291] proposed Soft Actor-Critic (SAC), which incorporated entropy maximization to encourage exploration and robustness in policy learning. Unlike traditional policy gradient methods, SAC optimized a trade-off between reward accumulation and policy stochasticity, leading to superior performance in complex robotic tasks. Simultaneously, Hessel et al. (2018) [927] developed Rainbow DQN, a unification of multiple enhancements to DQN, including Double DQN, Prioritized Experience Replay, Dueling Networks, Multi-step Learning, Noisy Networks, and Distributional RL. This integration provided a comprehensive framework for improving sample efficiency and generalization. Reinforcement learning also made breakthroughs in strategic decision-making, as demonstrated by Silver et al. (2016) [892] with AlphaGo, which leveraged Monte Carlo Tree Search (MCTS) with deep policy and value networks to achieve superhuman performance in the game of Go. AlphaGo's success was a milestone in planning-based RL, where deep neural networks provided heuristic guidance for complex search-based decision-making.

Another significant area of advancement in DRL was its application to robotic control and perception-driven decision-making. Levine et al. (2016) [928] pioneered end-to-end deep visuomotor policies, allowing robots to learn manipulation skills directly from raw visual inputs. Their work integrated guided policy search with deep learning, enabling robots to generalize across unseen scenarios with improved sample efficiency. Parallel to these practical advancements, theoretical contributions were made by Bellemare et al. (2017) [929] in Distributional RL, which proposed predicting a distribution of future rewards rather than a single expected return. This perspective significantly improved stability and sample efficiency, leading to the development of quantile-based methods like QR-DQN and Implicit Quantile Networks (IQN). Additionally, the foundational work by Sutton (2018) [286] provided a rigorous mathematical exposition of value iteration, policy iteration, and temporal difference learning, forming the theoretical bedrock of modern RL research. Their book continues to be the definitive reference for both theoretical exploration and practical implementation of reinforcement learning algorithms.

Through these fundamental breakthroughs, DRL has evolved into a robust field that spans value-based methods, policy gradient approaches, entropy-regularized reinforcement learning, and distributional perspectives. The fusion of deep learning with reinforcement learning has enabled unprecedented progress in areas such as robotics, strategic decision-making, and autonomous systems. The field continues to expand, with ongoing research exploring meta-reinforcement learning, multi-agent RL, and offline reinforcement learning to push the boundaries of intelligent decision-making in complex and uncertain environments.

<table><caption>Table 16.41: Summary of Contributions in Deep Reinforcement Learning</caption><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Mnih et al. (2015)</td><td>Introduced Deep Q-Learning for high-dimensional state spaces using deep neural networks. Proposed experience replay to break correlation among samples and target networks for stable Q-value updates. Enabled deep reinforcement learning for discrete action spaces.</td></tr><tr><td>Lillicrap et al. (2016)</td><td>Extended Q-learning to continuous action spaces using an actor-critic framework. Leveraged off-policy learning with target smoothing to improve convergence and stability in high-dimensional environments.</td></tr></tbody><tfoot><tr><td colspan="2">Continued on next page</td></tr></tfoot></table>
(Continued from previous page)

<table><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Schulman et al. (2015)</td><td>Developed a stable policy gradient method with constraints on policy divergence using a trust region approach. Ensured monotonic policy improvement and was particularly effective in robotic control tasks.</td></tr><tr><td>Schulman et al. (2017)</td><td>Simplified TRPO by introducing a clipped surrogate objective for stable and efficient policy updates. Achieved state-of-the-art performance while being computationally efficient and less sensitive to hyperparameters.</td></tr><tr><td>Haarnoja et al. (2018)</td><td>Introduced entropy-regularized reinforcement learning to encourage exploration. Optimized a stochastic policy while balancing reward accumulation and entropy maximization, improving sample efficiency and robustness.</td></tr><tr><td>Hessel et al. (2018)</td><td>Unified multiple improvements to DQN, including Double DQN, Prioritized Experience Replay, Dueling Networks, Multi-step Learning, Noisy Networks, and Distributional RL. Provided a comprehensive framework with enhanced sample efficiency and stability.</td></tr><tr><td>Silver et al. (2016)</td><td>Combined Monte Carlo Tree Search (MCTS) with deep policy and value networks to achieve superhuman performance in the game of Go. Demonstrated the power of planning-based reinforcement learning.</td></tr><tr><td>Levine et al. (2016)</td><td>Integrated deep learning with guided policy search for end-to-end robotic manipulation. Enabled robots to learn complex tasks directly from raw visual inputs.</td></tr><tr><td>Bellemare et al. (2017)</td><td>Proposed predicting the distribution of future rewards instead of the expected value. Led to improved stability and sample efficiency, forming the foundation for quantile-based RL methods such as QR-DQN and IQN.</td></tr><tr><td>Sutton and Barto (2018)</td><td>Provided a rigorous mathematical foundation for reinforcement learning. Covered fundamental topics such as value iteration, policy iteration, and temporal difference learning. Served as a definitive reference for both theoretical and practical RL research.</td></tr></tbody></table>

### 16.4.6.2 Recent Literature Review of Deep Reinforcement Learning

Deep reinforcement learning (DRL) has emerged as a powerful tool for solving complex decision-making and control problems across various domains. One of its key applications is in task offloading and resource allocation in mobile edge computing (MEC) networks, particularly in UAV-assisted environments. The work by Xue et al. (2025) [930] proposes a novel DRL-based algorithm to optimize computation offloading and multi-cache placement, addressing efficiency bottlenecks in UAV-assisted MEC systems. This is complemented by Amodu et al. (2025) [931], who present a comprehensive review of DRL applications for UAV-based optimization in MEC and IoT applications. Their work outlines the core challenges of DRL, including sample inefficiency, high computational costs, and model generalizability, while also suggesting future directions such as transfer learning and meta-reinforcement learning to improve adaptability in real-world systems. Another critical area where DRL has proven useful is autonomous energy management, as demonstrated in the study by Sunder et al. (2025) [932], who introduce the SmartAPM framework. This system leverages DRL to optimize power consumption in wearable devices, allowing for extended battery life through adaptive power control strategies based on environmental and user activity data.
Beyond network optimization and energy management, DRL is also making significant strides in autonomous control systems and cyber defense. For instance, Sarigül and Bayezit (2025) [933] apply DRL to fixed-wing aircraft heading control, showing how deep policy optimization can enhance maneuverability and operational autonomy compared to classical control techniques. Similarly, Mustafa et al. (2025) [917] focus on vehicular networks, proposing a Proximal Policy Optimization (PPO)-based DRL framework for computation offloading. Their findings highlight improved latency reduction and energy efficiency, making real-time decision-making more feasible for intelligent transport systems. Meanwhile, Mukhamadiarov (2025) [934] explores the application of DRL in stochastic dynamical systems, demonstrating how reinforcement learning can be effectively integrated with traditional control theory to enhance stability in non-linear environments. In cybersecurity, Ali and Wallace (2025) [935] introduce a DRL-powered autonomous cyber defense system for Security Operations Centers (SOCs), where self-learning agents continuously refine threat detection and mitigation strategies in adversarial settings. This research underscores the increasing role of DRL in adaptive cyber resilience, as it allows security systems to proactively identify and neutralize cyber threats in real-time.

Another promising area of DRL research lies in cross-domain transfer learning and real-world deployment, particularly in fluid dynamics and energy optimization. The study by Yan et al. (2025) [936] applies a mutual information-based transfer learning approach in active flow control for three-dimensional bluff body flows, significantly reducing training costs by enabling efficient knowledge transfer between different aerodynamic configurations. In a different sector, Silvestri et al. (2025) [937] investigated the deployment of DRL in real-world building control systems, where they incorporate imitation learning to address the sample inefficiency problem that plagues traditional DRL methods. Their approach enhances energy efficiency in buildings while maintaining occupant comfort, demonstrating a viable path for scaling DRL beyond theoretical applications. Similarly, Alajaji et al. (2025) [938] provide a scoping review of DRL in medical imaging, particularly in infrared spectroscopy-based cancer diagnosis. Their findings reveal that DRL can significantly improve the sensitivity and specificity of early cancer detection by automatically identifying abnormal patterns in spectroscopy data.

Collectively, these studies showcase the remarkable versatility and potential of DRL in solving real-world challenges. Whether in network optimization, cyber defense, autonomous control, medical imaging, or fluid dynamics, DRL continues to redefine the boundaries of machine learning by offering adaptive, scalable, and self-improving solutions. However, challenges such as high sample complexity, lack of explainability, and computational overhead remain significant hurdles that researchers are actively working to overcome. Future research in DRL is likely to focus on hybrid learning approaches, transfer learning techniques, and improved generalization methods to ensure that DRL models can perform effectively across diverse, unpredictable environments. The continued evolution of DRL in these domains not only paves the way for more robust and intelligent autonomous systems but also holds promise for breakthroughs in fields ranging from aerospace engineering to healthcare and cybersecurity.

<table><caption>Table 16.42: Summary of Recent Contributions in Deep Reinforcement Learning Research</caption><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>K. Xue, L. Zhai, Y. Li, Z. Lu, W. Zhou (2025)</td><td>Proposes a DRL-based algorithm to optimize task offloading and caching in UAV-assisted MEC networks, addressing computational efficiency and resource allocation.</td></tr></tbody></table>

*Continued on next page*
Continued from previous page

<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>O.A. Amodu, R.A.R. Mahmood, H. Althumali (2025)</td><td>Provides a comprehensive review of DRL applications in UAV-based MEC and IoT systems, identifying challenges and proposing future research directions.</td></tr><tr><td>A. Silvestri, D. Coraci, S. Brandi, A. Capozzoli (2025)</td><td>Investigates imitation learning as an alternative to DRL for real-world building control systems to enhance energy efficiency while maintaining occupant comfort.</td></tr><tr><td>R. Sunder, U.K. Lilhore, A.K. Rai, E. Ghith, M. Tlija (2025)</td><td>Introduces SmartAPM, a DRL-based adaptive power management system that optimizes battery life based on environmental and user activity data.</td></tr><tr><td>L. Yan, Q. Wang, G. Hu, W. Chen, B.R. Noack (2025)</td><td>Develops a mutual information-based transfer learning approach for DRL in active flow control, reducing training costs and improving aerodynamic simulations.</td></tr><tr><td>E. Mustafa, J. Shuja, F. Rehman, A. Namoun (2025)</td><td>Proposes a PPO-based DRL framework for vehicular computation offloading, significantly improving latency and energy efficiency in intelligent transport systems.</td></tr><tr><td>S.A. Alajaji, R. Sabzian, Y. Wang, A.S. Sultan, R. Wang (2025)</td><td>Reviews the use of DRL in medical imaging, highlighting its potential in infrared spectroscopy-based cancer diagnosis for improved accuracy and early detection.</td></tr><tr><td>N. Ali, G. Wallace (2025)</td><td>Explores DRL applications in cybersecurity, particularly in autonomous cyber defense for SOCs, enhancing threat detection and response capabilities.</td></tr><tr><td>F.A. Sarigül, I. Bayezit (2025)</td><td>Applies DRL to fixed-wing aircraft heading control, improving maneuverability and autonomy over traditional control techniques.</td></tr><tr><td>R. Mukhamadiarov (2025)</td><td>Investigates the application of DRL for controlling stochastic dynamical systems by integrating reinforcement learning with control theory to optimize stability.</td></tr></tbody></table>

### 16.4.6.3 Mathematical Analysis of Deep Reinforcement Learning

Deep Reinforcement Learning (DRL) is a mathematical framework that extends classical Reinforcement Learning (RL) by leveraging deep neural networks to approximate complex functions, enabling the efficient handling of high-dimensional state and action spaces. Mathematically, DRL is rooted in the Markov Decision Process (MDP), which is formally defined as the tuple

$$ \mathcal{M} = (\mathcal{S}, \mathcal{A}, P, R, \gamma) \qquad (16.291) $$

where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $P(s'|s, a)$ represents the transition probability function defining the probability of transitioning to state $s'$ given the current state $s$ and action $a$, $R(s, a)$ is the reward function specifying the immediate reward received after taking action $a$ in state $s$, and $\gamma \in [0, 1]$ is the discount factor that determines the present value of future rewards. The objective of DRL is to find an optimal policy $\pi^*(a|s)$ that maximizes the expected cumulative
reward, given by the return function:

$$G_t = \sum_{k=0}^{\infty} \gamma^k R(s_{t+k}, a_{t+k}). \qquad (16.292)$$

The optimal policy is derived by maximizing the state-value function $V^\pi(s)$ or the action-value function $Q^\pi(s, a)$, which are respectively defined as:

$$V^\pi(s) = \mathbb{E}_\pi \left[ \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t) \middle| s_0 = s \right], \qquad (16.293)$$

$$Q^\pi(s, a) = \mathbb{E}_\pi \left[ \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t) \middle| s_0 = s, a_0 = a \right]. \qquad (16.294)$$

In Deep Q-Networks (DQN), a deep neural network parameterized by $\theta$ is used to approximate the optimal Q-function:

$$Q(s, a; \theta) \approx Q^*(s, a). \qquad (16.295)$$

The network parameters are updated using the loss function derived from the Bellman equation:

$$L(\theta) = \mathbb{E}_{(s,a,r,s') \sim D} \left[ (y - Q(s, a; \theta))^2 \right], \qquad (16.296)$$

where $y$ is the target value computed using the Bellman optimality equation:

$$y = r + \gamma \max_{a'} Q(s', a'; \theta^{-}), \qquad (16.297)$$

where $\theta^{-}$ represents the parameters of a target network that is updated periodically to stabilize training. Policy-based methods directly optimize the policy $\pi_\theta(a|s)$ without relying on a Q-function. The objective function for policy gradient methods is given by:

$$J(\theta) = \mathbb{E}_{\pi} \left[ \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t) \right]. \qquad (16.298)$$

The gradient of $J(\theta)$ with respect to the policy parameters is computed using the policy gradient theorem:

$$\nabla_{\theta} J(\theta) = \mathbb{E}_{\pi} \left[ \nabla_{\theta} \log \pi_{\theta}(a|s) Q^{\pi}(s, a) \right]. \qquad (16.299)$$

The REINFORCE algorithm estimates this gradient using Monte Carlo sampling:

$$\nabla_{\theta} J(\theta) \approx \sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t|s_t) G_t. \qquad (16.300)$$

Actor-Critic methods combine value-based and policy-based approaches by introducing an actor network $\pi_\theta(a|s)$ and a critic network $V_w(s)$, where $w$ represents the critic parameters. The critic updates its parameters using the Temporal Difference (TD) error:

$$\delta_t = R_t + \gamma V_w(s_{t+1}) - V_w(s_t). \qquad (16.301)$$

The policy gradient is then computed using an advantage function $A^\pi(s, a)$, where:

$$A^\pi(s, a) = Q^\pi(s, a) - V^\pi(s). \qquad (16.302)$$

In Advantage Actor-Critic (A2C), the policy gradient update is:

$$\nabla_{\theta} J(\theta) \approx \sum_{t=0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_t|s_t) \delta_t. \qquad (16.303)$$
<figure><img src="image_206.png" /><figcaption>Figure 16.38: General pipeline of Deep Reinforcement Learning (DRL): The agent, represented by a deep neural network (blue), interacts with the environment (gray) by producing actions. The environment generates experiences, which are stored in the replay buffer (orange). Mini-batches from the buffer are used to compute predictions and targets for the loss function (red). The optimizer (violet) updates network weights through gradient descent</figcaption></figure>

Deep Deterministic Policy Gradient (DDPG) extends policy gradients to continuous action spaces by parameterizing the policy as a deterministic function:

$$a = \mu_{\theta}(s). \qquad (16.304)$$

The policy update follows the deterministic policy gradient theorem:

$$\nabla_{\theta} J(\theta) = \mathbb{E}_{s \sim \rho^{\pi}} \left[ \nabla_{\theta} \mu_{\theta}(s) \nabla_{a} Q^{\pi}(s, a) \bigg|_{a=\mu_{\theta}(s)} \right]. \qquad (16.305)$$

Proximal Policy Optimization (PPO) improves stability by enforcing a trust region constraint on policy updates. The surrogate objective function is:

$$L(\theta) = \mathbb{E} \left[ \min \left( r_t(\theta) A_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) A_t \right) \right], \qquad (16.306)$$

where $r_t(\theta) = \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ is the probability ratio between the new and old policies. Soft Actor-Critic (SAC) introduces an entropy regularization term to encourage exploration:

$$J(\theta) = \mathbb{E} \left[ \sum_{t=0}^{\infty} \gamma^t (R(s_t, a_t) + \alpha \mathcal{H}(\pi(\cdot|s_t))) \right], \qquad (16.307)$$

where $\mathcal{H}(\pi(\cdot|s_t)) = - \sum_a \pi(a|s) \log \pi(a|s)$ represents the entropy of the policy, and $\alpha$ is a temperature parameter.
### 16.4.6.3.1 Python Code to generate Figure 16.38

The Python code below produces the Figure 16.38 illustrating the Deep Reinforcement Learning.

```python
import matplotlib.pyplot as plt
import networkx as nx

def build_drl_graph():
    G = nx.DiGraph()

    # Nodes
    G.add_node("Environment", type="env")
    G.add_node("Replay Buffer", type="memory")
    G.add_node("Deep Neural Network", label="network")
    G.add_node("Loss Function", type="loss")
    G.add_node("Optimizer", type="opt")

    # Edges
    G.add_edge("Deep Neural Network", label="Environment", "Environment", "Action a")
    G.add_edge("Environment", "Replay Buffer", label="(s,a,r,s)")
    G.add_edge("Replay Buffer", "Deep Neural Network", label="Sample minibatch")
    G.add_edge("Deep Neural Network", label="Loss Function", "Loss Function", "Prediction")
    G.add_edge("Replay Buffer", "Loss Function", label="Target")
    G.add_edge("Loss Function", "Optimizer", label="Compute Gradients")
    G.add_edge("Optimizer", "Deep Neural Network", label="Update Weights")

    return G

def plot_drl.save_as=None):
    G = build_drl_graph()
    pos = {
        "Environment": (0,2),
        "Replay Buffer": (0,1),
        "Deep Neural Network": (3,1.5),
        "Loss Function": (6,2),
        "Optimizer": (6,1),
    }

    # Colors
    color_map = []
    for n, d in G.nodes(data=True):
        if d["type"] == "network":
            color_map.append("lightblue")
        elif d["type"] == "memory":
            color_map.append("orange")
        elif d["type"] == "loss":
            color_map.append("salmon")
        elif d["type"] == "opt":
            color_map.append("violet")
        elif d["type"] == "env":
```
```python
color_map.append("lightgrey")
else:
    color_map.append("white")

# Draw nodes
nx.draw_networkx_nodes(G, pos, node_color color_map, edgecolors="k",
                       node_size=3000)
nx.draw_networkx labels(G, pos, font_size=9)

# Draw edges with labels
ax = plt.gca()
for u, v, d in GEdges(data=True):
    ax annotate("",
              xy(pos[v], xycoordinates="data",
                  xytext(pos[u], textcoordinates="data",
                          arrowprops=dict(arrowstyle="->", color="black", lw
                                                =1.5)))
    if "label" in d:
        mid_x = (pos[u][0] + pos[v][0]) / 2
        mid_y = (pos[u][1] + pos[v][1]) / 2
        plt.text(mid_x, mid_y+0.15, d["label"], fontsize=7, ha="center")

plt.title("Deep Reinforcement Learning (DRL) Architecture", fontsize
          =12)
plt.axis("off")

if save_as:
    plt.savefig save_as, dπ=300, bbox_inches="tight")
    plt.show()

if __name__ == "__main__":
    plot_drl (save_as="drl_architecture.png")
```

### 16.4.6.4 Advantage Actor-Critic (A2C)

#### 16.4.6.4.1 Literature Review of Advantage Actor-Critic (A2C)

The Advantage Actor-Critic (A2C) algorithm has been extensively studied and applied in various domains, contributing to both theoretical advancements and practical implementations. Mnih et al. (2016) [949] laid the foundational work for A2C through their development of the Asynchronous Advantage Actor-Critic (A3C) method, which utilized multi-threaded parallelization to stabilize policy gradient methods. A2C emerged as a synchronous variant that improved sample efficiency while retaining the stability benefits of A3C. Wang et al. (2022) [950] further refined A2C by introducing Recursive Least Squares (RLS) methods to improve sample efficiency and convergence stability. Their research focused on optimizing the learning process using adaptive step sizes, effectively reducing training time and enhancing the robustness of deep reinforcement learning models.

The adaptability of A2C has enabled its application across diverse fields, including financial market trading and multi-agent reinforcement learning. Rubell Marion Lincy et al. (2023) [951] explored the integration of A2C with technical analysis indicators to improve decision-making in stock trading. Their approach demonstrated that A2C could effectively learn optimal buy and sell strategies while adapting to market fluctuations. Meanwhile, Paczolay et al. (2020) [952] extended A2C to
multi-agent environments, addressing key challenges in coordination and competition. They developed an improved training mechanism to stabilize interactions between multiple learning agents, significantly enhancing performance in decentralized decision-making settings. Zhang et al. (2024) [953] leveraged A2C for industrial applications, specifically in disassembly line balancing, where they optimized cycle time minimization. By formulating the optimization problem as a reinforcement learning task, they showcased the potential of A2C to streamline complex industrial processes.

Beyond conventional applications, researchers have also explored novel enhancements to A2C, such as quantum computing and adversarial training. Kölle et al. (2024) [955] introduced Quantum Advantage Actor-Critic (QA2C) and Hybrid Advantage Actor-Critic (HA2C), benchmarking them against classical A2C implementations. Their findings indicated that quantum models could improve computational efficiency, making reinforcement learning more scalable for high-dimensional problems. Benhamou (2019) [956] focused on addressing one of the fundamental challenges in policy gradient methods—variance reduction. By formulating a rigorous mathematical framework, they optimized control variate estimators, effectively minimizing gradient variance and improving learning stability. Similarly, Peng et al. (2018) [957] introduced Adversarial A2C, where a discriminator was incorporated to enhance exploration strategies in dialogue policy learning. Their approach demonstrated significant improvements in reinforcement learning-based conversational agents, particularly in human-like task completion.

Finally, A2C has proven to be a powerful tool in robotics and adaptive control systems. Van Veldhuizen (2022) [958] explored its application in tuning Proportional-Integral-Derivative (PID) controllers for robotic systems. Their research demonstrated that A2C could autonomously optimize PID parameters, significantly improving the precision of control tasks such as apple harvesting. The ability of A2C to generalize across different control environments underscores its flexibility as a reinforcement learning framework. Collectively, these contributions illustrate the extensive theoretical advancements and practical innovations enabled by A2C, solidifying its position as a cornerstone in deep reinforcement learning.

<table><caption>Table 16.43: Summary of Contributions in A2C Research</caption><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Mnih et al. (2016)</td><td>Introduced the Asynchronous Advantage Actor-Critic (A3C) algorithm, which inspired the synchronous A2C algorithm by stabilizing policy gradient methods through multi-threaded training.</td></tr><tr><td>Wang et al. (2022)</td><td>Developed RLS-based A2C variants that improve sample efficiency and learning stability in deep reinforcement learning environments.</td></tr><tr><td>Rubell Marion Lincy et al. (2023)</td><td>Applied A2C for stock trading using technical indicators, showing improved decision-making for buy and sell strategies.</td></tr><tr><td>Paczolay et al. (2020)</td><td>Adapted A2C for multi-agent scenarios, addressing coordination and competition among agents with improved training mechanisms.</td></tr><tr><td>Zhang et al. (2024)</td><td>Applied A2C for industrial optimization problems, demonstrating enhanced cycle time minimization in disassembly line balancing.</td></tr></tbody><tfoot><tr><td colspan="2">Continued on next page</td></tr></tfoot></table>
<table><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Kölle et al. (2024)</td><td>Explored quantum implementations of A2C, benchmarking QA2C and HA2C against classical A2C architectures for efficiency gains.</td></tr><tr><td>Benhamou (2019)</td><td>Provided a mathematical framework for reducing variance in A2C methods, optimizing control variate estimators for policy gradient algorithms.</td></tr><tr><td>Peng et al. (2018)</td><td>Introduced Adversarial A2C, incorporating a discriminator to enhance exploration in dialogue policy learning.</td></tr><tr><td>van Veldhuizen (2022)</td><td>Applied A2C for adaptive control in robotics, optimizing PID tuning for an apple-harvesting robot.</td></tr></tbody></table>

### 16.4.6.4.2 Recent Literature Review of Advantage Actor-Critic (A2C)

Advantage Actor-Critic (A2C) reinforcement learning has emerged as a powerful tool across diverse domains, providing superior adaptability and stability in decision-making compared to traditional reinforcement learning methods. In the financial sector, A2C has been leveraged for optimizing investment strategies and risk-sensitive portfolio management. Wang and Liu (2025) [939] introduced a transformer-enhanced A2C model that refines petroleum futures trading by dynamically adjusting trading decisions based on real-time risk assessments. Their study demonstrates how combining transformers with A2C can enhance financial modeling accuracy, outperforming static risk-assessment methods. Similarly, Thongkairat and Yamaka (2025) [940] applied A2C in stock market trading, proving its effectiveness in managing highly volatile stock portfolios. Unlike Deep Q-Networks (DQN) and other value-based methods, A2C's policy gradient approach enables more stable convergence and better long-term decision-making. By leveraging actor-critic optimization, A2C effectively reduces erratic investment behaviors and optimizes portfolio allocations under stochastic market conditions.

Beyond finance, A2C has been pivotal in cybersecurity and autonomous decision-making. Dey and Ghosh (2025) [941] demonstrated A2C's potential in intrusion detection, developing a reinforcement learning-based framework to mitigate QUIC-based Denial of Service (DoS) attacks. Traditional signature-based cybersecurity systems often struggle against evolving attack patterns, but A2C adapts dynamically by continuously learning from network behavior, thereby improving real-time threat detection. In the realm of autonomous vehicles and drones, Zhao et al. (2025) [942] proposed an A2C-based UAV trajectory optimization method, optimizing real-time flight paths and task allocation based on environmental feedback. Their study underscores A2C's ability to handle high-dimensional control problems, ensuring efficient drone operations while minimizing energy consumption. Similarly, Mounesan et al. (2025) [943] introduced Infer-EDGE, an A2C-driven system for optimizing deep learning inference in edge-AI applications. Their research highlights the growing synergy between reinforcement learning and AI-driven computational efficiency, demonstrating how A2C can dynamically balance trade-offs between computation cost, latency, and model accuracy in resource-constrained environments.

In energy systems and industrial automation, A2C has proven instrumental in optimizing resource allocation and performance forecasting. Hou et al. (2025) [944] developed a multi-agent cooperative A2C framework (MAC-A2C) for fuel cell degradation prediction, significantly improving the lifespan and efficiency of fuel cells in automotive applications. This research showcases how reinforcement learning can improve predictive maintenance, reducing operational costs in high-performance energy systems. In nuclear reactor safety, Radaideh et al. (2025) [945] applied asynchronous A2C algorithms to optimize neutron flux distribution, enhancing reactor control and minimizing the risk
of critical failures. Their findings suggest that reinforcement learning could revolutionize nuclear engineering by introducing adaptive and autonomous reactor safety mechanisms. Meanwhile, Li et al. (2025) [946] compared A2C with Soft Actor-Critic (SAC) for task offloading in edge computing, demonstrating that A2C's on-policy approach offers superior learning stability in low-resource environments. Their work highlights A2C's effectiveness in mobile cloud computing, where efficient resource distribution is essential.

Furthermore, A2C has been successfully employed in large-scale combinatorial optimization problems. Khan et al. (2025) [947] applied A2C for route optimization in dairy product logistics, reducing delivery costs while ensuring timely and efficient distribution. Unlike heuristic optimization approaches, reinforcement learning-based solutions can adapt to real-time traffic conditions and supply chain fluctuations, making A2C a promising tool for logistics and transportation industries. Yuan et al. (2025) [948] extended A2C by incorporating transformers (TR-A2C) to enhance multi-user semantic communication in vehicle networks. Their study demonstrates how transformer-enhanced reinforcement learning can significantly improve the efficiency of information exchange in connected vehicles, paving the way for next-generation intelligent transportation systems. These applications illustrate A2C's growing impact across multiple disciplines, cementing its role as a versatile and robust reinforcement learning framework. By enabling real-time adaptability, policy optimization, and multi-agent coordination, A2C continues to drive advancements in machine learning, industrial automation, and autonomous decision-making.

<table><caption>Table 16.44: Summary of Recent Contributions in A2C Research</caption><thead><tr><td>Authors and Year</td><td>Contribution and Key Findings</td></tr></thead><tbody><tr><td>Wang and Liu (2025)</td><td>Developed a transformer-enhanced A2C model for portfolio optimization in petroleum futures trading. Their approach improves risk-sensitive decision-making by dynamically adjusting trading strategies, outperforming traditional risk-assessment methods.</td></tr><tr><td>Thongkairat and Yamaka (2025)</td><td>Applied A2C for automated stock trading, demonstrating its superiority over Deep Q-Networks (DQN) in handling volatile markets. The model enables more stable convergence and improved long-term returns through policy gradient optimization.</td></tr><tr><td>Dey and Ghosh (2025)</td><td>Proposed an A2C-based intrusion detection system for QUIC-based Denial of Service (DoS) attacks. Unlike static cybersecurity systems, their model adapts dynamically to evolving network threats, achieving higher detection accuracy.</td></tr><tr><td>Zhao et al. (2025)</td><td>Designed an A2C-based UAV trajectory optimization system, enabling drones to optimize real-time flight paths and task offloading. The approach minimizes energy consumption while improving autonomous navigation efficiency.</td></tr><tr><td>Mounesan et al. (2025)</td><td>Introduced Infer-EDGE, an A2C-driven deep learning inference optimization system. The model dynamically balances computational cost, latency, and resource allocation in edge-AI environments, enhancing performance efficiency.</td></tr></tbody><tfoot><tr><td colspan="2">Continued on next page</td></tr></tfoot></table>
<table><caption>Table 16.44 – Continued from previous page</caption><thead><tr><td>Authors and Year</td><td>Contribution and Key Findings</td></tr></thead><tbody><tr><td>Hou et al. (2025)</td><td>Developed a multi-agent cooperative A2C (MAC-A2C) framework for fuel cell degradation prediction in automotive applications. The model extends fuel cell lifespan and reduces operational costs by optimizing predictive maintenance.</td></tr><tr><td>Radaideh et al. (2025)</td><td>Applied asynchronous A2C algorithms to optimize neutron flux distribution in nuclear microreactors. Their model enhances reactor control strategies, reducing the risk of critical failures through reinforcement learning-based adaptation.</td></tr><tr><td>Li et al. (2025)</td><td>Compared A2C with Soft Actor-Critic (SAC) for edge computing task offloading. The results indicate that A2C's on-policy learning approach provides greater stability and efficiency in resource-constrained cloud environments.</td></tr><tr><td>Khan et al. (2025)</td><td>Used A2C for optimizing route planning in supply chain logistics, particularly in dairy product distribution. Their reinforcement learning approach reduces delivery costs and dynamically adapts to traffic conditions for improved efficiency.</td></tr><tr><td>Yuan et al. (2025)</td><td>Developed a transformer-enhanced A2C (TR-A2C) framework to improve multi-user semantic communication in vehicle networks. The approach enhances data transmission efficiency and ensures better adaptation to real-time network changes.</td></tr></tbody></table>

### 16.4.6.4.3 Mathematical Analysis of Advantage Actor-Critic (A2C)

The Advantage Actor-Critic (A2C) algorithm is a policy gradient method that improves sample efficiency by utilizing both value-based and policy-based learning techniques. Let $s_t$ denote the state of the environment at time step $t$, $a_t$ the action taken, and $r_t$ the immediate reward received. The objective of the reinforcement learning agent is to maximize the expected cumulative discounted reward, given by

$$J(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t=0}^{\infty} \gamma^t r_t \right], \qquad (16.308)$$

where $\pi_{\theta}(a_t | s_t)$ represents the policy parameterized by $\theta$, and $\gamma$ is the discount factor ($0 \le \gamma \le 1$). The policy is updated using the policy gradient theorem, which states

$$\nabla_{\theta} J(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t=0}^{\infty} \nabla_{\theta} \log \pi_{\theta}(a_t | s_t) G_t \right], \qquad (16.309)$$

where $G_t$ is the return from time step $t$ onwards. Instead of using the full return $G_t$, which has high variance, A2C employs the advantage function

$$A^{\pi}(s_t, a_t) = Q^{\pi}(s_t, a_t) - V^{\pi}(s_t), \qquad (16.310)$$

where $Q^{\pi}(s_t, a_t)$ is the action-value function,

$$Q^{\pi}(s_t, a_t) = \mathbb{E}_{\pi} \left[ \sum_{k=0}^{\infty} \gamma^k r_{t+k} | s_t, a_t \right], \qquad (16.311)$$
<figure><img src="image_207.png" /><figcaption>Figure 16.39: Architecture of the Advantage Actor-Critic (A2C) algorithm. The actor network (green) outputs actions based on states. The environment (gray) generates trajectories stored in a buffer (orange). The critic (blue) estimates state values, which together with rewards are used in advantage estimation (violet). The loss function (red) updates both actor and critic networks synchronously</figcaption></figure>

and $V^\pi(s_t)$ is the state-value function,

$$V^\pi(s_t) = \mathbb{E}_\pi \left[ \sum_{k=0}^{\infty} \gamma^k r_{t+k} \mid s_t \right]. \qquad (16.312)$$

This leads to the gradient update

$$\nabla_{\theta} J(\theta) = \mathbb{E}_{\pi_{\theta}} \left[ \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi}(s_t, a_t) \right]. \qquad (16.313)$$

A2C estimates $V^\pi(s_t)$ using a parameterized critic $V_\phi(s_t)$, trained by minimizing the squared temporal difference (TD) error

$$L_V(\phi) = \mathbb{E} \left[ (r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t))^2 \right]. \qquad (16.314)$$

The advantage function is then approximated as

$$A^{\pi}(s_t, a_t) \approx r_t + \gamma V_{\phi}(s_{t+1}) - V_{\phi}(s_t). \qquad (16.315)$$

The actor updates the policy parameters $\theta$ using the advantage-weighted policy gradient

$$\theta \leftarrow \theta + \alpha \nabla_{\theta} \log \pi_{\theta}(a_t \mid s_t) A^{\pi}(s_t, a_t), \qquad (16.316)$$
where $\alpha$ is the learning rate. The critic updates its parameters $\phi$ using gradient descent

$$ \phi \leftarrow \phi - \beta \nabla_{\phi} L_{V}(\phi), \qquad (16.317) $$

where $\beta$ is the critic's learning rate. The training process is synchronized across multiple agents, ensuring a more stable and deterministic update compared to the asynchronous variant A3C. This synchronous nature of A2C prevents stale gradients and allows for efficient batch updates.

### 16.4.6.4.4 Python Code to generate Figure 16.39

The Python code below produces the Figure 16.39 illustrating the Advantage Actor-Critic (A2C).

```python
import matplotlib.pyplot as plt
import networkx as nx

def build_a2c_graph():
    G = nx.DiGraph()

    # Nodes
    G.add_node("Environment", type="env")
    G.add_node("Replay (Trajectory)", type="memory")
    G.add_node("Actor (πθ)", type="actor")
    G.add_node("Critic (VΦ)", type="critic")
    G.add_node("Advantage Estimation", type="advantage")
    G.add_node("Loss Function", type="loss")

    # Edges
    G.add_edge("Actor (πθ)", "Environment", label="a ~ πθ(s)")
    G.add_edge("Environment", "Replay (Trajectory)", label="(s,a,r,s)")
    G.add_edge("Replay (Trajectory)", "Actor (πθ)", label="s")
    G.add_edge("Replay (Trajectory)", "Critic (VΦ)", label="s, r, s")
    G.add_edge("Critic (VΦ)", "Advantage Estimation", label="VΦ(s), VΦ(s)")
    G.add_edge("Replay (Trajectory)", "Advantage Estimation", label="r")
    G.add_edge("Advantage Estimation", "Loss Function", label="A(s,a)")
    G.add_edge("Loss Function", "Actor (πθ)", label="update θ")
    G.add_edge("Loss Function", "Critic (VΦ)", label="update Φ")

    return G

def plot_a2c.save_as=None):
    G = build_a2c_graph()

    pos = {
        "Environment": (0,2),
        "Replay (Trajectory)": (0,1),
        "Actor (πθ)": (2,2),
        "Critic (VΦ)": (2,1),
        "Advantage Estimation": (4,1.5),
        "Loss Function": (6,1.5),
    }

    # Colors
    color_map = []

    for n, d in G.nodes(data=True):
```
```python
if d["type"] == "actor":
    color_map.append("lightgreen")
elif d["type"] == "critic":
    color_map.append("lightblue")
elif d["type"] == "advantage":
    color_map.append("violet")
elif d["type"] == "loss":
    color_map.append("salmon")
elif d["type"] == "memory":
    color_map.append("orange")
elif d["type"] == "env":
    color_map.append("lightgrey")
else:
    color_map.append("white")

# Draw nodes
nx.draw_networkx_nodes(G, pos, node_color.color_map, edgecolors="k",
                       node_size=2500)
nx.draw_networkx_labels(G, pos, font_size=9)

# Draw edges
ax = plt.gca()
for u, v, d in GEdges(data=True):
    ax annotate("",
               xy(pos[v], xycoordinates="data",
                  xytext(pos[u], textcoordinates="data",
                         arrowprops=dict(arrowstyle="->", color="black", lw
                                               =1.5)))
    if "label" in d:
        mid_x = (pos[u][0] + pos[v][0]) / 2
        mid_y = (pos[u][1] + pos[v][1]) / 2
        plt.text(mid_x, mid_y+0.15, d["label"], fontsize=7, ha="center")

plt.title("Advantage Actor-Critic (A2C) Architecture", fontsize=12)
plt.axis("off")

if save_as:
    plt.savefig.save_as, dπ=300, bbox_inches="tight")
    plt.show()

if __name__ == "__main__":
    plot_a2c.save_as="a2c_architecture.png")
```

### 16.4.6.5 Deep Deterministic Policy Gradient

#### 16.4.6.5.1 Literature Review of Deep Deterministic Policy Gradient

Deep Deterministic Policy Gradient (DDPG) has emerged as a fundamental algorithm in reinforcement learning (RL) for continuous action spaces, building upon deterministic policy gradients and deep Q-learning. The pioneering work by Lillicrap et al. (2015) [895] introduced the DDPG framework, integrating an actor-critic architecture with experience replay and target networks to stabilize training. This algorithm extends the Deterministic Policy Gradient (DPG) theorem from Silver et al. (2014) [851], which rigorously established that deterministic policy gradients can be computed
with lower variance than their stochastic counterparts, making them particularly effective in high-
dimensional continuous control tasks. While DDPG demonstrated its efficacy in simulated physics
environments, subsequent research focused on addressing its limitations, particularly in off-policy
learning, stability, and exploration. Cicek et al. (2021) [959] tackled the issue of off-policy correc-
tion by introducing Batch Prioritized Experience Replay, a technique that prioritizes data points
based on their likelihood of being generated by the current policy. This enhancement improved
sample efficiency by reducing policy divergence during updates, a critical challenge in off-policy
deep RL methods.

Building upon these foundational ideas, Han et al. (2021) [960] proposed the Regularly Updated Deterministic (RUD) Policy Gradient Algorithm, which introduced a structured update mechanism to mitigate overestimation bias and variance in Q-value estimates. This was further refined by Pan et al. (2020) [961] in the Softmax Deep Double Deterministic Policy Gradient (SD3) method, which incorporated a Boltzmann softmax operator to smooth the optimization landscape and combat overestimation errors, leading to more reliable policy convergence. Addressing another major weakness of DDPG—exploration inefficiency—Luck et al. (2019) [962] proposed an approach that integrates latent trajectory optimization, where model-based trajectory estimation improves exploration in sparse-reward settings. Their work demonstrated that combining deep RL with latent variable models leads to improved generalization and efficiency in complex, high-dimensional environments. The practical applications of DDPG have also been expanded, with Dong et al. (2023) [963] developing an enhanced version for robotic arm control by integrating adaptive reward shaping and improved experience replay, yielding superior performance in dexterous manipulation tasks.

Beyond robotics, DDPG has been successfully applied in autonomous navigation and medical decision-making. Jesus et al. (2019) [964] extended the DDPG framework to mobile robot navigation, demonstrating its effectiveness in dynamic obstacle-avoidance scenarios without the need for explicit localization or map-based planning. Their results highlight DDPG's ability to learn control policies in real-world environments where stochastic disturbances pose significant challenges. In a different domain, Lin et al. (2023) [965] employed DDPG for personalized medicine dosing strategies, illustrating how reinforcement learning can optimize complex decision-making processes in healthcare. Their approach aimed to replicate expert medical decisions, showcasing the potential for RL-based models to revolutionize treatment protocols by adapting to patient-specific characteristics. Lastly, the work of Sumalatha et al. (2024) [966] provided a rigorous overview of the evolution, enhancements, and applications of DDPG, consolidating various algorithmic advancements and practical implementations across multiple fields.

Collectively, these contributions form a rigorous and multifaceted body of research that extends DDPG's theoretical foundations, addresses its key limitations, and explores novel applications in diverse domains. The enhancements in off-policy correction, policy update mechanisms, and exploration strategies underscore the continuous evolution of deep reinforcement learning methods to achieve greater efficiency, stability, and real-world applicability. As research in deep RL progresses, the integration of DDPG with meta-learning, hierarchical RL, and model-based approaches is expected to further enhance its capabilities, opening new frontiers in both theoretical and applied reinforcement learning.
<table><caption>Table 16.45: Summary of Contributions in Deep Deterministic Policy Gradient (DDPG) Research</caption><thead><tr><td>Authors & Year</td><td>Contribution & Key Improvements</td></tr></thead><tbody><tr><td>Lillicrap et al. (2015)</td><td>Introduced Deep Deterministic Policy Gradient (DDPG), combining deterministic policy gradients with deep Q-learning in an actor-critic framework. Enabled off-policy learning for high-dimensional continuous action spaces, stabilized training using target networks and experience replay.</td></tr><tr><td>Silver et al. (2014)</td><td>Developed the deterministic policy gradient theorem, proving that deterministic policies yield lower variance gradients compared to stochastic policy gradients. Provided the theoretical foundation for DDPG, ensuring efficient learning in continuous control environments.</td></tr><tr><td>Cicek et al. (2021)</td><td>Proposed Batch Prioritized Experience Replay (BPER) to improve off-policy learning by prioritizing data points based on their likelihood of being generated by the current policy. This reduced policy divergence and improved sample efficiency.</td></tr><tr><td>Han et al. (2021)</td><td>Developed the Regularly Updated Deterministic (RUD) Policy Gradient Algorithm to mitigate Q-value overestimation bias and variance. Introduced a structured update mechanism that improved stability in policy learning.</td></tr><tr><td>Pan et al. (2020)</td><td>Introduced Softmax Deep Double Deterministic Policy Gradient (SD3), incorporating a Boltzmann softmax operator to smooth policy updates. This reduced overestimation errors in Q-learning and improved stability during training.</td></tr><tr><td>Luck et al. (2019)</td><td>Integrated latent trajectory optimization with DDPG to enhance exploration in sparse-reward environments. Improved exploration by leveraging model-based trajectory estimation, enabling better generalization in complex environments.</td></tr><tr><td>Dong et al. (2023)</td><td>Extended DDPG for robotic arm control by integrating adaptive reward shaping and improved experience replay mechanisms. Increased policy robustness and efficiency in real-world dexterous manipulation tasks.</td></tr><tr><td>Jesus et al. (2019)</td><td>Applied DDPG to autonomous mobile robot navigation, demonstrating its effectiveness in real-time obstacle avoidance. Showed that learning-based control could be achieved without explicit localization or map-based planning.</td></tr><tr><td>Lin et al. (2023)</td><td>Used DDPG for personalized medicine dosing strategies to optimize treatment decisions. Demonstrated reinforcement learning's capability in healthcare applications by personalizing medical treatment plans based on patient-specific characteristics.</td></tr></tbody></table>

*Continued on next page*
<table><thead><tr><td>Authors & Year</td><td>Contribution & Key Improvements</td></tr></thead><tbody><tr><td>Sumalatha et al. (2024)</td><td>Provided a comprehensive survey of DDPG advance-ments, consolidating various algorithmic improvements and practical applications across multiple domains. An-alyzed the evolution of DDPG and its role in real-world implementations.</td></tr></tbody></table>

### 16.4.6.5.2 Recent Literature Review of Deep Deterministic Policy Gradient

Deep Deterministic Policy Gradient (DDPG) has demonstrated significant advancements in diverse domains, particularly in optimizing resource allocation, decision-making, and control systems across vehicular networks, robotics, and UAV coordination. Recent studies have leveraged DDPG and its variations, such as Twin Delayed DDPG (TD3) and Multi-Agent Deep Deterministic Policy Gradient (MADDPG), to enhance performance in complex, real-time environments. For instance, in vehicular edge computing networks, Yang et al. (2025) [967] proposed a hierarchical optimization framework integrating DDPG to optimize driving mode selection and resource allocation, significantly reducing latency and improving efficiency. Similarly, Jamshidiha and Pourahmadi (2025) incorporated DDPG into a traffic-aware graph neural network for cellular networks, demonstrating improved user association and network adaptability. Furthermore, Tian et al. (2025) [968] introduced a Multi-State Iteration DDPG (SIDDPG) for the Internet of Vehicles (IoV), optimizing offloading strategies to minimize delays and energy consumption. These studies collectively highlight DDPG's ability to dynamically optimize decision-making processes in vehicular and communication networks, ensuring enhanced network performance and resource utilization.

In addition to vehicular networks, DDPG has been extensively applied in robotics, UAV coordination, and cloud-edge computing. Raei et al. (2025) [920] introduced a DDPG-based reinforcement learning framework for non-prehensile robotic manipulation, demonstrating superior efficiency in dynamic control environments. In another application, Chen et al. (2025) [969] proposed a UAV-based computation offloading system utilizing DDPG, which optimizes task scheduling in cloud-edge collaborative environments, reducing latency and enhancing computational efficiency. Moreover, Ting-Ting et al. (2025) [921] developed a MADDPG-based approach for UAV clusters, ensuring robust decision-making under communication constraints. These studies illustrate DDPG's adaptability to real-time, multi-agent environments, allowing for optimized coordination and resource management in robotics and aerial systems. By integrating reinforcement learning with real-world constraints, these approaches have successfully enhanced operational efficiency in autonomous systems.

Furthermore, advancements in DDPG have facilitated its application in emerging network and cloud computing paradigms. Deng et al. (2025) [970] expanded DDPG into a multi-agent reinforcement learning paradigm for non-terrestrial networks, enabling distributed coordination in satellite communications. Similarly, Anwar and Akber (2025) [903] incorporated MADDPG into structural engineering, optimizing building resilience by considering utility interactions under extreme conditions. Additionally, Zhang et al. (2025) [971] employed TD3 for multi-vehicle and multi-edge resource orchestration, optimizing computational tasks and minimizing system delays. These studies highlight the robustness of DDPG in optimizing decision-making processes across distributed networks, structural systems, and multi-agent interactions. By addressing dynamic, high-dimensional control problems, DDPG and its extensions continue to drive innovations in artificial intelligence and engineering.

In conclusion, the reviewed studies establish DDPG as a versatile and powerful reinforcement learn-
ing technique capable of handling continuous action-space optimization in diverse domains. From vehicular edge computing and UAV-based coordination to robotics and cloud-edge resource management, DDPG has demonstrated remarkable efficiency in optimizing real-time decision-making. With further advancements in multi-agent learning, federated reinforcement learning, and adaptive policy optimization, DDPG is poised to remain at the forefront of artificial intelligence research, revolutionizing complex control and resource allocation problems across various industries.

<table><thead><tr><td>Study</td><td>Contribution</td></tr></thead><tbody><tr><td>Yang et al. (2025)</td><td>Introduced a three-stage hierarchical optimization framework (3SHO) where DDPG optimizes resource allocation, improving efficiency in vehicular networks.</td></tr><tr><td>Jamshidiha and Pourahmadi (2025)</td><td>Employed DDPG within a traffic-aware graph neural network to optimize user association, enhancing network adaptability in cellular networks.</td></tr><tr><td>Tian et al. (2025)</td><td>Developed Multi-State Iteration DDPG (SIDDPG) to optimize task partitioning and reduce latency and energy consumption in Internet of Vehicles (IoV).</td></tr><tr><td>Saad et al. (2025)</td><td>Applied Twin Delayed DDPG (TD3) for edge server selection in 5G-enabled industrial applications, minimizing computation delays.</td></tr><tr><td>Deng et al. (2025)</td><td>Proposed a Multi-Agent DDPG (MADDPG) for satellite communication networks, improving distributed coordination in non-terrestrial networks.</td></tr><tr><td>Raei et al. (2025)</td><td>Designed a DDPG-based reinforcement learning framework for robotic manipulation through sliding, improving control efficiency.</td></tr><tr><td>Ting-Ting et al. (2025)</td><td>Developed MADDPG with inter-agent communication for UAV clusters under constrained communication environments.</td></tr><tr><td>Zhang et al. (2025)</td><td>Utilized TD3 to optimize multi-vehicle and multi-edge computing resource allocation, reducing system delays in vehicular edge computing.</td></tr><tr><td>Chen et al. (2025)</td><td>Integrated DDPG into UAV-based computation offloading to improve task scheduling and computational efficiency in cloud-edge collaborative computing.</td></tr><tr><td>Anwar and Akber (2025)</td><td>Applied MADDPG to enhance building resilience through multi-agent reinforcement learning, optimizing utility interactions in structural engineering.</td></tr></tbody></table>

### 16.4.6.5.3 Mathematical Analysis of Deep Deterministic Policy Gradient

Deep Deterministic Policy Gradient (DDPG) is a mathematically rigorous reinforcement learning (RL) algorithm that extends the classical Deterministic Policy Gradient (DPG) method to high-dimensional continuous action spaces by incorporating function approximators, particularly deep neural networks, to estimate both the policy and the value function. The algorithm is formulated within the framework of an infinite-horizon Markov Decision Process (MDP) characterized by a state space $S$, an action space $A$, a state transition probability distribution $p(s'|s, a)$, a reward function $r: S \times A \rightarrow \mathbb{R}$, and a discount factor $\gamma \in (0, 1]$. The agent's objective is to learn a policy $\pi: S \rightarrow A$ that maximizes the expected cumulative reward, given by the return function

$$R_t = \sum_{k=0}^{\infty} \gamma^k r(s_{t+k}, a_{t+k}). \qquad (16.318)$$
<figure><img src="image_208.png" /><figcaption>Figure 16.40: Architecture of the Deep Deterministic Policy Gradient (DDPG) algorithm. The actor network (green) outputs continuous actions. The environment (gray) produces transitions, which are stored in a replay buffer (orange). The critic (blue) and target networks (violet) evaluate Q-values and compute target values. The loss function (red) updates both actor and critic networks via policy gradients and temporal-difference learning</figcaption></figure>

The action-value function, or Q-function, is defined as the expected return starting from a state s and taking an action a, while following policy π:

$$Q^{\pi}(s, a) = \mathbb{E}[R_t | s_t = s, a_t = a, \pi] = \mathbb{E} \left[ \sum_{k=0}^{\infty} \gamma^k r(s_{t+k}, a_{t+k}) | s_t = s, a_t = a, \pi \right]. \quad (16.319)$$

The Bellman equation for the Q-function under an optimal policy satisfies

$$Q^*(s, a) = \mathbb{E} \left[ r(s, a) + \gamma \max_{a'} Q^*(s', a') | s, a \right]. \quad (16.320)$$

Unlike traditional Q-learning approaches, which solve for discrete action spaces by iterating over all possible actions, DDPG employs an actor-critic architecture where the actor $\pi_{\theta}$ is a deterministic policy parameterized by $\theta$ that maps states directly to actions:

$$a = \pi_{\theta}(s). \quad (16.321)$$

The critic function $Q_{\phi}(s, a)$ approximates the true Q-function and is parameterized by $\phi$. The optimal policy parameters are obtained by maximizing the Q-value under the policy:

$$\theta^* = \arg \max_{\theta} \mathbb{E}_{s \sim \rho^{\pi}} [Q_{\phi}(s, \pi_{\theta}(s))] . \quad (16.322)$$
The deterministic policy gradient theorem states that the gradient of the expected return with respect to the policy parameters is

$$ \nabla_{\theta} J(\theta) = \mathbb{E}_{s \sim \rho^{\pi}} \left[ \nabla_{\theta} \pi_{\theta}(s) \nabla_{a} Q_{\phi}(s, a) \bigg|_{a=\pi_{\theta}(s)} \right]. \qquad (16.323) $$

The Q-function is learned via the Bellman equation, leading to the following loss function for the critic:

$$ L(\phi) = \mathbb{E}_{(s,a,r,s') \sim D} \left[ (Q_{\phi}(s,a) - y)^2 \right], \qquad (16.324) $$

where the target y is computed using a target Q-network $Q_{\phi'}$ and a target policy network $\pi_{\theta'}$:

$$ y = r + \gamma Q_{\phi'}(s', \pi_{\theta'}(s')). \qquad (16.325) $$

To stabilize training, DDPG employs two techniques: (1) target networks, which are slow-moving versions of the policy and value networks, updated with soft updates

$$ \theta' \leftarrow \tau\theta + (1 - \tau)\theta', \qquad (16.326) $$

$$ \phi' \leftarrow \tau\phi + (1 - \tau)\phi', \qquad (16.327) $$

where $\tau \ll 1$; and (2) an experience replay buffer $D$, where past transitions $(s, a, r, s')$ are stored and randomly sampled for training, breaking correlations between consecutive updates and improving sample efficiency.

### 16.4.6.5.4 Python Code to generate Figure 16.40

DDPG is an actor-critic, off-policy RL algorithm with:

1. Actor (policy network) outputs continuous actions.

2. Critic (Q-network) evaluates $Q(s, a)$.

3. Target networks stabilize learning.

4. Replay buffer stores transitions.

5. Environment generates experience.

The Python code below produces the Figure 16.40 illustrating the Deep Deterministic Policy Gradient.

```python
import matplotlib.pyplot as plt
import networkx as nx

def build_ddpg_graph():
    G = nx.DiGraph()

    # Nodes
    G.add_node("Environment", type="env")
    G.add_node("Replay Buffer", type="memory")
    G.add_node("Actor (πθ)", type="actor")
    G.add_node("Critic (QΦ)", type="critic")
    G.add_node("Target Actor (πθ)", type="target")
    G.add_node("Target Critic (QΦ)", type="target")
    G.add_node("Loss Function", type="loss")

    # Edges
```
G.add_edge("Actor (πθ)", "Environment", label="a = πθ(s)")
G.add_edge("Environment", "Replay Buffer", label="(s,a,r,s)")
G.add_edge("Replay Buffer", "Actor (πθ)", label="s")
G.add_edge("Replay Buffer", "Critic (QΦ)", label="(s,a)")
G.add_edge("Replay Buffer", "Target Actor (πθ)", label="s")
G.add_edge("Target Actor (πθ)", "Target Critic (QΦ)", label="a = πθ(s)")
G.add_edge("Target Critic (QΦ)", "Loss Function", label="y=r+γQΦ(s,a)")
G.add_edge("Critic (QΦ)", "Loss Function", label="QΦ(s,a)")
G.add_edge("Loss Function", "Critic (QΦ)", label="update Φ")
G.add_edge("Critic (QΦ)", "Actor (πθ)", label="grad a Q(s,a)")
G.add_edge("Loss Function", "Actor (πθ)", label="update θ")

return G

def plot_ddpg save_as=None ):
    G = build_ddpg_graph()
    pos ={ "Environment" : (0,2), "Replay Buffer" : (0,1), "Actor (πθ)" : (2,2), "Critic (QΦ)" : (2,1), "Target Actor (πθ)" : (4,2), "Target Critic (QΦ)" : (4,1), "Loss Function" : (6,1.5), }
    # Colors
    color_map =[]
    for n, d in G.nodes(data=True) :
        if d[ "type" ] == "actor" :
            color_map.append("lightgreen")
        elif d[ "type" ] == "critic" :
            color_map.append("lightblue")
        elif d[ "type" ] == "target" :
            color_map.append("violet")
        elif d[ "type" ] == "loss" :
            color_map.append("salmon")
        elif d[ "type" ] == "memory" :
            color_map.append("orange")
        elif d[ "type" ] == "env" :
            color_map.append("lightgrey")
        else :
            color_map.append("white")
    # Draw nodes
    nx.draw_networkx_nodes(G, pos, node_color= color_map, edgecolors="k", node_size=2500)
    nx.draw_networkx labels(G, pos, font_size=9)
    # Draw edges
    ax = plt.gca()
    for u, v, d in GEdges(data=True) :
        ax. annotate("",
```python
xy(pos[v], xycoordinates="data",
   xytext(pos[u], textcoordinates="data",
          arrowprops=dict(arrowstyle="->", color="black", lw
              =1.5))

if "label" in d:
    mid_x = (pos[u][0] + pos[v][0]) / 2
    mid_y = (pos[u][1] + pos[v][1]) / 2
    plt.text(mid_x, mid_y+0.15, d("label"), fontsize=7, ha="center")

plt.title("Deep Deterministic Policy Gradient (DDPG) Architecture",
          fontsize=12)
plt.axis("off")

if save_as:
    plt.savefig.save_as(dπ=300, bbox_inches="tight")
    plt.show()

if __name__ == "__main__":
    plot_ddpg.save_as("ddpg_architecture.png")
```

### 16.4.6.6 Proximal Policy Optimization

#### 16.4.6.6.1 Literature Review of Proximal Policy Optimization

Proximal Policy Optimization (PPO) has become one of the most widely used policy gradient methods due to its balance between computational efficiency and stable performance. Schulman et al. (2017) laid the theoretical foundation for PPO by introducing a first-order optimization technique that constrains policy updates through a clipped surrogate objective, ensuring stable and reliable learning without requiring the complex second-order optimization steps of Trust Region Policy Optimization (TRPO). Following its introduction, researchers identified several critical implementation details that significantly impact PPO's performance. Huang and Dossa (2022) compiled a comprehensive list of 37 key implementation details, highlighting often-overlooked hyperparameter choices, network architectures, and training strategies necessary for PPO to perform consistently across different tasks. These insights provided a crucial guideline for practitioners and researchers to reproduce PPO's results accurately, preventing performance discrepancies arising from subtle implementation variations.

Several studies have proposed refinements to PPO's clipping mechanism to enhance training stability and sample efficiency. Zhang et al. (2023) addressed the issue of fixed clipping bounds by introducing a dynamic adjustment mechanism that utilizes task-specific feedback, thereby ensuring that policy updates remain within an optimal range. This adaptation mitigates the issue of overly aggressive policy updates in early training stages while preventing excessive conservatism in later stages. Similarly, Zhang et al. (2020) proposed an alternative dynamic clipping strategy to fine-tune the trade-off between exploration and exploitation, further improving PPO's adaptability. Another notable contribution in this domain is Kobayashi's (2022) introduction of an adaptive threshold for PPO using the symmetric relative density ratio, which optimally adjusts the clipping bound based on the scale of the policy update error. This method improves training stability and ensures a more theoretically grounded approach to threshold selection. Kobayashi (2020) extended this idea by formulating PPO using relative Pearson divergence, which provides an intuitive and principled way to constrain policy updates, thereby achieving smoother and more stable training dynamics.
Beyond single-agent reinforcement learning, PPO has also been extended to multi-agent systems. Piao and Zhuo (2021) introduced Coordinated Proximal Policy Optimization (CoPPO), which adapts step sizes dynamically for multiple interacting agents, mitigating instability issues caused by independent updates in decentralized training settings. The coordinated adaptation mechanism significantly enhances PPO's applicability in complex environments where multiple agents must learn simultaneously while maintaining stability in learning dynamics. Additionally, Wang et al. (2019) proposed Truly Proximal Policy Optimization (TPPO), which integrates a trust region constraint directly into the objective function, thereby ensuring that updates remain within a theoretically justified range and reducing the likelihood of catastrophic policy collapse. These modifications further reinforced PPO's theoretical robustness by explicitly incorporating policy divergence constraints.

Finally, a broader perspective on PPO's reliability was provided by Henderson et al. (2018), who critically examined deep reinforcement learning methodologies and demonstrated that performance metrics in literature are highly sensitive to hyperparameter tuning and implementation details. Their findings underscored the necessity of rigorous benchmarking and reproducibility standards to ensure that performance improvements claimed by various PPO enhancements are not merely the result of uncontrolled hyperparameter tuning. Collectively, these contributions showcase the continuous evolution of PPO, from its theoretical foundation to practical improvements in policy update mechanisms, stability guarantees, and applicability in multi-agent reinforcement learning. They also highlight the importance of principled algorithm design and robust evaluation methodologies to advance reinforcement learning research in a scientifically rigorous manner.

<table><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Schulman et al. (2017)</td><td>Introduced Proximal Policy Optimization (PPO), a first-order policy gradient method that balances stability and efficiency. PPO replaces the trust region constraint in TRPO with a clipped surrogate objective, simplifying implementation while maintaining strong empirical performance across various reinforcement learning tasks.</td></tr><tr><td>Huang and Dossa (2022)</td><td>Provided a comprehensive checklist of 37 crucial implementation details necessary for reproducing PPO's reported performance. This work highlighted the impact of hyperparameter choices, architectural considerations, and training strategies on PPO's consistency and reliability.</td></tr><tr><td>Zhang et al. (2023)</td><td>Proposed a dynamic clipping strategy where the clipping threshold adapts based on task feedback, ensuring optimal constraint selection for policy updates. This method improves training stability and sample efficiency while addressing PPO's sensitivity to fixed clipping bounds.</td></tr><tr><td>Zhang et al. (2020)</td><td>Developed an exploration-enhancing variant of PPO that incorporates uncertainty estimation. Their approach improves sample efficiency by encouraging exploration in regions of high uncertainty, particularly in continuous control tasks.</td></tr><tr><td>Kobayashi (2022)</td><td>Introduced a threshold adaptation mechanism for PPO using the symmetric relative density ratio, leading to a more theoretically grounded method for selecting the clipping bound. This modification enhances stability and improves policy update efficiency.</td></tr><tr><td>Kobayashi (2020)</td><td>Formulated PPO using relative Pearson divergence, providing a principled policy divergence constraint that ensures smoother and more stable training updates, improving theoretical soundness.</td></tr></tbody></table>
<table><thead><tr><td>Reference</td><td>Contribution</td></tr></thead><tbody><tr><td>Piao and Zhuo (2021)</td><td>Extended PPO to multi-agent reinforcement learning by introducing Coordinated Proximal Policy Optimization (CoPPO), which adapts step sizes dynamically for multiple interacting agents. This approach improves training stability and performance in decentralized multi-agent environments.</td></tr><tr><td>Zhang and Wang (2020)</td><td>Proposed a dynamic clipping bound method that adjusts the threshold throughout training, further refining the balance between exploration and exploitation in PPO-based learning.</td></tr><tr><td>Wang et al. (2019)</td><td>Introduced Truly Proximal Policy Optimization (TPPO), which explicitly incorporates a trust region constraint into PPO's objective function, leading to more reliable and theoretically grounded policy updates.</td></tr><tr><td>Henderson et al. (2018)</td><td>Critically examined the reproducibility of deep reinforcement learning algorithms, including PPO. The study demonstrated the sensitivity of performance metrics to hyperparameter tuning and implementation details, emphasizing the need for rigorous benchmarking and standardization in reinforcement learning research.</td></tr></tbody></table>

### 16.4.6.6.2 Recent Literature Review of Proximal Policy Optimization

Proximal Policy Optimization (PPO) has emerged as a robust reinforcement learning algorithm with wide-ranging applications across multiple domains, from space exploration to autonomous robotics and networking optimization. One of its notable implementations is in interplanetary trajectory design, where PPO is leveraged to optimize orbital maneuvers, ensuring efficient fuel management and trajectory planning under stringent mission constraints (Cuéllar et al., 2024) [972]. This showcases PPO's ability to handle high-dimensional decision-making problems. Similarly, in autonomous underwater robotics, PPO is employed for three-dimensional trajectory planning by integrating fluid dynamics-based motion strategies, allowing underwater vehicles to adapt to dynamic marine environments and external perturbations (Liu et al., 2025) [973]. The adaptability of PPO in handling complex physical constraints further extends to humanoid robotics, where it enables fast multimodal gait learning, allowing bipedal robots to adjust walking patterns across varying terrains through reinforcement-based self-learning mechanisms (Figueroa et al., 2025) [974]. These applications underscore the strength of PPO in real-world motion planning and adaptive control.

In networking and communication systems, PPO has demonstrated its efficacy in cognitive radio networks and edge computing. For instance, a Lyapunov-guided PPO-based resource allocation model optimizes spectrum sharing and power distribution in dynamic radio environments, reducing latency and maximizing throughput (Xu et al., 2025) [975]. Similarly, vehicular networks benefit from PPO-driven computation offloading, where the algorithm efficiently schedules tasks between cloud and edge nodes to enhance latency-sensitive applications (Mustafa et al., 2025) [976]. PPO has also been employed in anti-jamming wireless communication to dynamically allocate spectrum resources in adversarial settings, ensuring robust and resilient connectivity (Li et al., 2025) [976]. These applications illustrate how PPO enhances real-time decision-making in complex, distributed systems that require efficient task execution and resource utilization.

Beyond robotics and communication, PPO has been instrumental in optimizing industrial processes and energy systems. In cloud computing, PPO has been integrated with graph neural networks (GNNs) to optimize dynamic workflow scheduling, reducing both computational overhead and energy consumption in large-scale cloud environments (Chandrasiri and Meedeniya, 2025) [977]. Moreover, a smart building energy management system leverages PPO to dynamically regulate
heating, ventilation, and air conditioning (HVAC) operations, significantly improving carbon effi-
ciency (Wu and Xie, 2025) [979]. In the manufacturing sector, Guan et al. (2025) [980] proposed
a multi-agent PPO-based approach for real-time production scheduling and AGV selection, ensur-
ing just-in-time logistics and maximizing supply chain efficiency. These implementations highlight
PPO's impact on autonomous decision-making for sustainability and operational efficiency.

Finally, PPO's utility in decision-making under uncertainty is evident in financial markets and transportation systems. In stock portfolio management, PPO is integrated with reinforcement-based market simulations to optimize trade execution, balancing risk and return in volatile environments (Zhang et al., 2025) [981]. Additionally, in autonomous vehicle trajectory planning, PPO is combined with control barrier function techniques to ensure collision-free navigation in high-speed driving scenarios, refining behavior planning strategies at complex intersections (Zhang et al., 2025) [982]. These cases demonstrate PPO's capability in learning optimal policies in dynamic, stochastic environments, making it an invaluable tool for real-time AI decision-making. Across all these domains, PPO continues to drive innovation by balancing exploration and exploitation, allowing intelligent systems to adapt to ever-changing real-world constraints efficiently.

<table>
   <thead>
    <tr>
     <td>
      Author(s) &amp; Year
     </td>
     <td>
      Contribution
     </td>
    </tr>
   </thead>
   <tbody>
    <tr>
     <td>
      Cuéllar et al., 2024
     </td>
     <td>
      Uses PPO for optimizing interplanetary trajectory design, ensuring efficient orbital maneuvers with fuel management optimization.
     </td>
    </tr>
    <tr>
     <td>
      Guan et al., 2025
     </td>
     <td>
      Integrates PPO with a transformer-based attention mechanism for vehicle routing, improving convergence and efficiency.
     </td>
    </tr>
    <tr>
     <td>
      Wu &amp; Xie, 2025
     </td>
     <td>
      Proposes PPO for optimizing HVAC energy efficiency in smart buildings, significantly reducing carbon footprint.
     </td>
    </tr>
    <tr>
     <td>
      Chandrasiri &amp; Mee-deniya, 2025
     </td>
     <td>
      Combines PPO with Graph Neural Networks to optimize cloud workflow scheduling, reducing latency and energy use.
     </td>
    </tr>
    <tr>
     <td>
      Liu et al., 2025
     </td>
     <td>
      Employs PPO to optimize real-time navigation for underwater robots, adapting to environmental disturbances.
     </td>
    </tr>
    <tr>
     <td>
      Mustafa et al., 2025
     </td>
     <td>
      Uses PPO for dynamic task offloading in edge-assisted vehicular networks, enhancing real-time resource management.
     </td>
    </tr>
    <tr>
     <td>
      Figueroa et al., 2025
     </td>
     <td>
      Applies PPO for humanoid robot gait adaptation, enabling robots to walk efficiently across different terrains.
     </td>
    </tr>
    <tr>
     <td>
      Xu et al., 2025
     </td>
     <td>
      Develops a PPO-based Lyapunov-guided model for optimizing resource allocation in cognitive radio networks.
     </td>
    </tr>
    <tr>
     <td>
      Bukhari et al., 2025
     </td>
     <td>
      Integrates PPO with causal learning for natural language-based robotic control, improving adaptability in robotics.
     </td>
    </tr>
    <tr>
     <td>
      Dai et al., 2025
     </td>
     <td>
      Utilizes PPO with control barrier functions for autonomous vehicle trajectory planning, ensuring safe and efficient merging.
     </td>
    </tr>
   </tbody>
  </table>

16.4.6.6.3 Mathematical Analysis of Proximal Policy Optimization

Proximal Policy Optimization (PPO) is a policy gradient method used in reinforcement learning that seeks to improve the stability and efficiency of policy updates while maintaining sample effi- ciency. Given a Markov Decision Process (MDP) defined by the tuple (S, A, P, r, γ), where S is the state space, A is the action space, P(s' | s, a) represents the transition probabilities, r(s, a) is the reward function, and γ is the discount factor, PPO aims to find an optimal policy π_θ(a | s) pa- rameterized by θ, maximizing the expected cumulative discounted reward. The objective function for policy optimization is given by the expected advantage-weighted likelihood ratio:

$$
J(\theta) = \mathbb{E}_{s,a \sim \pi_{\theta_{\text{old}}}} \left[ \frac{\pi_{\theta}(a|s)}{\pi_{\theta_{\text{old}}}(a|s)} A^{\pi_{\theta_{\text{old}}}}(s,a) \right] \quad (16.328)
$$
where $A^{\pi_{\theta_{old}}}(s, a)$ is the advantage function that quantifies how much better an action is compared to the expected value of the state under the old policy. To prevent overly large updates that can lead to instability, PPO employs a clipped surrogate objective:

$$L^{\text{CLIP}}(\theta) = \mathbb{E}_{s,a \sim \pi_{\theta_{\text{old}}}} [\min (r_{\theta}(s, a)A^{\pi_{\theta_{\text{old}}}}(s, a), \text{clip}(r_{\theta}(s, a), 1 - \epsilon, 1 + \epsilon)A^{\pi_{\theta_{\text{old}}}}(s, a))] \quad (16.329)$$

where the importance sampling ratio is defined as

$$r_{\theta}(s, a) = \frac{\pi_{\theta}(a | s)}{\pi_{\theta_{\text{old}}}(a | s)} \qquad (16.330)$$

and $\epsilon > 0$ is a hyperparameter that controls how much the new policy can deviate from the old policy. The clipping mechanism ensures that if the policy update step would lead to a large change in action probability, the advantage function is modified to prevent large updates, thereby stabilizing training. The clipped term can be rewritten as

$$\text{clip}(r_{\theta}(s, a), 1 - \epsilon, 1 + \epsilon) A^{\pi_{\theta_{\text{old}}}}(s, a) = \begin{cases} (1 + \epsilon) A^{\pi_{\theta_{\text{old}}}}(s, a), & \text{if } r_{\theta}(s, a) > 1 + \epsilon \\ (1 - \epsilon) A^{\pi_{\theta_{\text{old}}}}(s, a), & \text{if } r_{\theta}(s, a) < 1 - \epsilon \\ r_{\theta}(s, a) A^{\pi_{\theta_{\text{old}}}}(s, a), & \text{otherwise} \end{cases} \quad (16.331)$$

This formulation ensures that the policy update is constrained within a predefined trust region without requiring explicit second-order optimization methods, unlike Trust Region Policy Optimization (TRPO). Additionally, PPO often includes a value function term to reduce variance while maintaining unbiased gradient estimates. The value loss function is given by

$$L^{\text{VF}}(\theta) = \mathbb{E}_{s \sim \pi_{\theta_{\text{old}}}} \left[ \left( V_{\theta}(s) - V^{\text{target}}(s) \right)^2 \right] \quad (16.332)$$

where $V_{\theta}(s)$ is the learned state-value function and $V^{\text{target}}(s)$ is the estimated target value function computed from bootstrapped returns, typically using Generalized Advantage Estimation (GAE):

$$V^{\text{target}}(s_t) = R_t + \gamma V_{\theta_{\text{old}}}(s_{t+1}) \quad (16.333)$$

where the return is

$$R_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \dots + \gamma^{T-t} r_T \quad (16.334)$$

and the advantage function is estimated using GAE:

$$A_t = \sum_{l=0}^{T-t} (\gamma \lambda)^l \delta_{t+l} \quad (16.335)$$

with the temporal difference residual

$$\delta_t = r_t + \gamma V_{\theta_{\text{old}}}(s_{t+1}) - V_{\theta_{\text{old}}}(s_t) \quad (16.336)$$

where $\lambda$ is a smoothing parameter that controls the bias-variance tradeoff. The final PPO objective function combines the clipped surrogate loss, value function loss, and an entropy bonus to encourage exploration:

$$L(\theta) = \mathbb{E} [L^{\text{CLIP}}(\theta) - c_1 L^{\text{VF}}(\theta) + c_2 S[\pi_{\theta}](s)] \quad (16.337)$$

where $c_1$ and $c_2$ are hyperparameters and $S[\pi_{\theta}](s)$ is the entropy term given by

$$S[\pi_{\theta}](s) = - \sum_{a \in A} \pi_{\theta}(a | s) \log \pi_{\theta}(a | s) \quad (16.338)$$

which ensures policy exploration by discouraging premature convergence to deterministic policies. The policy updates are performed iteratively using stochastic gradient ascent with Adam optimizer:

$$\theta \leftarrow \theta + \alpha \nabla_{\theta} L(\theta) \quad (16.339)$$

where $\alpha$ is the learning rate. Since PPO is an on-policy method, experience data is collected under the current policy and discarded after updates. The training follows a batch-wise paradigm where sampled trajectories are used to update the policy over multiple epochs. By balancing exploration, sample efficiency, and stability, PPO achieves robust policy optimization across diverse reinforcement learning tasks.
<figure><img src="image_209.png" /><figcaption>Figure 16.41: Architecture of the Proximal Policy Optimization (PPO) algorithm: The actor (green) outputs actions, interacting with the environment (gray). Rollouts are stored in a trajectory buffer (orange). The critic (blue) estimates value functions, and the advantage estimator (violet) computes advantages $A_t$. The clipped surrogate objective (red) combines policy ratios and advantages to update the actor stably</figcaption></figure>

16.4.6.6.4 Python Code to generate Figure 16.41

Proximal Policy Optimization (PPO) is one of the most widely used policy gradient algorithms. To visualize PPO, a good diagram shows the data flow between:

1. Environment → generates transitions Policy (Actor) → produces actions from states.

2. Value Function (Critic) → estimates state values

3. Advantage Estimator → computes using rewards and values.

4. Clipped Objective → updates the policy with constraints to prevent large steps.

5. Replay/Trajectory Buffer → stores collected rollouts.

The Python code below produces the Figure 16.41 illustrating the Proximal Policy Optimization (PPO).

1 import matplotlib.pyplot as plt

2 import networkx as nx

3
def build_ppo_graph():

G = nx.DiGraph()

# Nodes

G.add_node("Environment", type="env")

G.add_node("Trajectory Buffer", type="memory")

G.add_node("Actor (Policy πθ)", type="actor")

G.add_node("Critic (Value VΦ)", type="critic")

G.add_node("Advantage Estimator", type="adv")

G.add_node("Clipped Objective", type="loss")

# Edges

G.add_edge("Environment", "Trajectory Buffer", label="(s,a,r,s)")

G.add_edge("Actor (Policy πθ)", "Environment", label="a")

G.add_edge("Trajectory Buffer", "Actor (Policy πθ)", label="states s")

G.add_edge("Trajectory Buffer", "Critic (Value VΦ)", label="states s")

G.add_edge("Trajectory Buffer", "Advantage Estimator", label="rewards r")

G.add_edge("Critic (Value VΦ)", "Advantage Estimator", label="values V(s)")

G.add_edge("Actor (Policy πθ)", "Clipped Objective", label="πθ")

G.add_edge("Advantage Estimator", "Clipped Objective", label="A_t")

G.add_edge("Clipped Objective", "Actor (Policy πθ)", label="policy update")

return G

def plot_ppo save_as=None):

G = build_ppo_graph()

pos = {
    "Environment": (0,2),
    "Trajectory Buffer": (0,1),
    "Actor (Policy πθ)": (2,2),
    "Critic (Value VΦ)": (2,1),
    "Advantage Estimator": (4,1.5),
    "Clipped Objective": (6,1.5),
}

# Colors

color_map = []

for n, d in G.nodes(data=True):
    if d["type"] == "actor":
        color_map.append("lightgreen")
    elif d["type"] == "critic":
        color_map.append("lightblue")
    elif d["type"] == "adv":
        color_map.append("violet")
    elif d["type"] == "loss":
        color_map.append("salmon")
    elif d["type"] == "memory":
        color_map.append("orange")
    elif d["type"] == "env":
        color_map.append("lightgrey")
```python
else:
    color_map.append("white")

# Draw nodes
nx.draw_networkx_nodes(G, pos, node_color color_map, edgecolors="k",
                       node_size=2500)
nx.draw_networkx labels(G, pos, font_size=9)

# Draw edges
ax = plt.gca()
for u, v, d in GEdges(data=True):
    ax annotate("",
              xy(pos[v], xycoordinates="data",
                  xytext(pos[u], textcoordinates="data",
                          arrowprops=dict(arrowstyle="->", color="black", lw
                                                =1.5)))
    if "label" in d:
        mid_x = (pos[u][0] + pos[v][0]) / 2
        mid_y = (pos[u][1] + pos[v][1]) / 2
        plt.text(mid_x, mid_y+0.15, d["label"], fontsize=7, ha="center")

    plt.title("Proximal Policy Optimization (PPO) Architecture",
              fontsize=12)
    plt.axis("off")

    if save_as:
        plt.savefig save_as, dπ=300, bbox_inches="tight")
    plt.show()

if __name__ == "__main__":
    plot_ppo save_as="ppo_architecture.png")
```

### 16.4.6.7 Soft Actor-Critic (SAC) algorithm

#### 16.4.6.7.1 Literature Review of Soft Actor-Critic (SAC) algorithm

The Soft Actor-Critic (SAC) algorithm, originally proposed by Haarnoja et al. (2018), introduced a novel framework that integrates the maximum entropy principle with off-policy reinforcement learning, significantly enhancing sample efficiency, stability, and robustness of policy learning. The key contribution of SAC lies in its entropy-regularized objective, which encourages policies to not only maximize expected returns but also maintain high entropy, ensuring effective exploration and better generalization across unseen states. This approach was further refined in Haarnoja et al. (2019), where an automatic temperature tuning mechanism was introduced to dynamically balance the trade-off between exploration and exploitation, alleviating the need for exhaustive hyperparameter tuning. These developments enabled SAC to outperform existing algorithms such as Deep Deterministic Policy Gradient (DDPG) and Twin Delayed DDPG (TD3) on a variety of continuous control benchmarks, demonstrating superior performance in both sample efficiency and policy robustness.

Several extensions of SAC have been proposed to address specific challenges in robotics, autonomous navigation, and uncertainty estimation. Wu et al. (2023) applied SAC to LiDAR-based robot navigation, where the algorithm was adapted to dynamic obstacle avoidance scenarios, showing im-
proved training efficiency and navigation success rates. Similarly, Hossain et al. (2022) introduced an inhibitory network-based modification to SAC, designed to accelerate the retraining of UAV controllers, thereby enhancing adaptability in fast-changing environments. Another significant development came from Ishfaq et al. (2025), who introduced Langevin Soft Actor-Critic (LSAC), integrating Thompson sampling with distributional Langevin Monte Carlo updates to improve uncertainty estimation and exploration efficiency. These contributions reflect ongoing efforts to tailor SAC for real-world applications where stability and robustness are paramount.

Further theoretical advancements have sought to refine the critic network and improve value estimation in SAC. Verma et al. (2023) proposed the Soft Actor Retrospective Critic (SARC), which introduced an additional retrospective loss term for the critic network, accelerating convergence and stabilizing training dynamics. Addressing concerns regarding approximation errors, Tasdighi et al. (2023) introduced PAC-Bayesian Soft Actor-Critic, which incorporates a PAC-Bayesian objective into the critic's learning process, reducing uncertainty and improving sample efficiency. In a related effort, Duan (2021) proposed Distributional Soft Actor-Critic (DSAC), which learns a Gaussian distribution over stochastic returns, mitigating value overestimation errors commonly encountered in standard SAC implementations. These refinements collectively contribute to more reliable policy learning, especially in high-dimensional continuous control tasks.

Beyond algorithmic improvements, theoretical explanations and comparative studies have solidified SAC's role as a benchmark reinforcement learning algorithm. The Papers with Code analysis provides a detailed breakdown of SAC's fundamental principles, highlighting its superiority over traditional policy optimization methods due to its ability to capture multi-modal policy distributions. This is complemented by the in-depth analyses of Haarnoja et al., who emphasize the algorithm's strengths in handling stochasticity while maintaining computational tractability. Taken together, these contributions establish SAC as one of the most powerful and widely adopted actor-critic methods in deep reinforcement learning, continuing to inspire further research in robotics, autonomous systems, and sample-efficient deep reinforcement learning.

<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Haarnoja et al. (2018)</td><td>Introduced Soft Actor-Critic (SAC), integrating the maximum entropy principle into reinforcement learning to enhance sample efficiency, stability, and policy robustness. SAC introduced an entropy-regularized objective, encouraging high-entropy policies for better exploration and generalization.</td></tr><tr><td>Haarnoja et al. (2019)</td><td>Developed an automatic temperature tuning mechanism for SAC, dynamically adjusting the entropy coefficient to balance exploration and exploitation. This improvement alleviates the need for manual hyperparameter tuning, leading to more adaptive learning.</td></tr><tr><td>Wu et al. (2023)</td><td>Applied SAC to LiDAR-based robot navigation, demonstrating its effectiveness in dynamic obstacle avoidance. The study showed improved training efficiency and higher navigation success rates, making SAC more viable for real-world robotic applications.</td></tr><tr><td>Hossain et al. (2022)</td><td>Proposed an inhibitory network-based modification to SAC for UAV control, enhancing adaptability and retraining speed in fast-changing environments. This modification allows SAC to better handle non-stationary reinforcement learning settings.</td></tr></tbody></table>
<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Ishfaq et al. (2025)</td><td>Introduced Langevin Soft Actor-Critic (LSAC), combining SAC with Thompson sampling and distributional Langevin Monte Carlo updates to improve uncertainty estimation and exploration efficiency. This modification leads to more robust decision-making under uncertainty.</td></tr><tr><td>Verma et al. (2023)</td><td>Developed the Soft Actor Retrospective Critic (SARC), incorporating a retrospective loss term in the critic network to accelerate convergence and stabilize training dynamics, improving sample efficiency in high-dimensional problems.</td></tr><tr><td>Tasdighi et al. (2023)</td><td>Introduced PAC-Bayesian Soft Actor-Critic, integrating a PAC-Bayesian objective into SAC's critic network to reduce uncertainty and improve sample efficiency. This approach provides theoretical guarantees on policy performance.</td></tr><tr><td>Duan (2021)</td><td>Proposed Distributional Soft Actor-Critic (DSAC), learning a Gaussian distribution over stochastic returns to mitigate value overestimation errors in standard SAC implementations. This leads to more reliable policy learning in continuous control tasks.</td></tr><tr><td>Papers with Code Analysis</td><td>Provided a comprehensive breakdown of SAC's theoretical foundations, benchmarking its performance across various continuous control environments, and highlighting its superiority over traditional policy optimization methods.</td></tr><tr><td>Haarnoja et al. (Various Analyses)</td><td>Conducted in-depth studies on SAC's ability to handle stochasticity, demonstrating its advantages in learning multi-modal policy distributions while maintaining computational efficiency.</td></tr></tbody></table>

### 16.4.6.7.2 Recent Literature Review of Soft Actor-Critic (SAC) algorithm

Soft Actor-Critic (SAC) has emerged as a powerful deep reinforcement learning algorithm, demonstrating remarkable adaptability across various domains such as autonomous systems, financial markets, industrial automation, and transportation. One significant application is in search and rescue operations, where SAC outperforms traditional reinforcement learning methods due to its entropy-regularized policy. Ewers et al. (2025) implement SAC alongside Proximal Policy Optimization (PPO) in a recurrent autoencoder-based deep reinforcement learning system, revealing SAC's superiority in handling dynamic and uncertain environments. Similarly, in fluid dynamics, Yan et al. (2025) propose mutual information-based knowledge transfer learning (MIKT-SAC), enhancing SAC's ability to generalize across domains for active flow control in bluff body flows, demonstrating significant improvements in cross-domain transferability. Another critical industrial application is seen in Industry 5.0, where Asmat et al. (2025) develop a Digital Twin (DT) framework integrated with SAC, enabling intelligent cyber-physical systems for adaptive industrial transitions. Moreover, SAC is extensively utilized in telecommunications, as highlighted by Chao and Jiao (2025), who leverage SAC for network spectrum resource allocation, optimizing spectrum usage in dynamic wireless environments.

Beyond industrial applications, SAC is also making strides in autonomous navigation and adversarial learning. Ma et al. (2025) introduce SIE-SAC, an advanced SAC-based learning mechanism for UAV navigation under adversarial conditions, specifically GPS/INS-integrated spoofing scenarios. The study underscores SAC's capability to adapt and refine deception strategies, a crucial element in defensive aerospace applications. Additionally, SAC is being explored in financial markets, where
Walia et al. (2025) integrate SAC with causal generative adversarial networks (GANs) and large language models (LLMs) for liquidity-aware bond yield prediction. This approach significantly enhances financial forecasting by leveraging reinforcement learning's adaptability in complex, non-linear financial datasets. Lalor and Swishchuk (2025) further extend SAC's financial applications by applying it to non-Markovian market-making, proving its efficiency in handling long-term dependencies and stochastic pricing models.

SAC's utility is not limited to theoretical models but also extends into multi-agent cooperative learning and resource optimization. Zhang et al. (2025) propose a diffusion-based SAC framework for multi-UAV networks in the Metaverse, optimizing cooperative task allocation and resource distribution in edge-enabled virtual environments. Similarly, Zhao et al. (2025) apply SAC in energy management for hybrid storage systems in urban rail transit, enhancing the efficiency of traction power supply systems. This approach demonstrates SAC's potential in sustainable energy applications, where efficient power distribution is paramount. In the automotive sector, Tresca et al. (2025) utilize SAC to design adaptive energy management strategies for hybrid electric vehicles, optimizing fuel efficiency and battery longevity by dynamically adjusting energy consumption based on real-time driving conditions.

Overall, SAC continues to revolutionize deep reinforcement learning across various domains by providing sample-efficient, entropy-regularized learning strategies that balance exploration and exploitation effectively. From autonomous navigation and cybersecurity to industrial optimization and sustainable energy systems, SAC's ability to handle high-dimensional, complex decision-making problems ensures its widespread applicability. Whether applied in robotics, finance, or telecommunications, SAC demonstrates exceptional versatility, pushing the boundaries of intelligent decision-making in real-world systems.

<table><thead><tr><td>Authors (Year)</td><td>Contribution</td></tr></thead><tbody><tr><td>Ewers et al. (2025)</td><td>Implement SAC and PPO in a recurrent autoencoder-based reinforcement learning framework for search and rescue operations, demonstrating SAC's superior performance in dynamic and uncertain environments.</td></tr><tr><td>Yan et al. (2025)</td><td>Develop a mutual information-based knowledge transfer learning (MIKT-SAC) method to enhance SAC's generalization across domains, improving active flow control in bluff body flows.</td></tr><tr><td>Asmat et al. (2025)</td><td>Propose a Digital Twin (DT) framework integrated with SAC to enable intelligent cyber-physical systems, facilitating the transition from Industry 4.0 to Industry 5.0.</td></tr><tr><td>Chao & Jiao (2025)</td><td>Apply SAC for network spectrum resource allocation, optimizing spectrum usage in dynamic wireless environments to enhance telecommunications efficiency.</td></tr><tr><td>Ma et al. (2025)</td><td>Introduce SIE-SAC, a novel reinforcement learning mechanism for UAV navigation in adversarial conditions, particularly in GPS/INS-integrated spoofing scenarios.</td></tr><tr><td>Walia et al. (2025)</td><td>Combine SAC with causal generative adversarial networks (GANs) and large language models (LLMs) to improve bond yield predictions and enhance financial forecasting.</td></tr></tbody></table>

*Continued on next page*