units **h**, and **a**, **b** are bias vectors—is that the log-likelihood gradient for a single training datum **v**(0) involves an expectation over the model distribution $P_{\theta}(\mathbf{v}, \mathbf{h})$:

$$ \nabla_{\theta} \log P_{\theta}(\mathbf{v}^{(0)}) = \mathbb{E}_{P_{\theta}(\mathbf{h}|\mathbf{v}^{(0)})}[\nabla_{\theta} E(\mathbf{v}^{(0)}, \mathbf{h})] - \mathbb{E}_{P_{\theta}(\mathbf{v},\mathbf{h})}[\nabla_{\theta} E(\mathbf{v}, \mathbf{h})] \quad (10.322) $$

The first term, the positive phase, is tractable as it requires averaging over the conditional distribution $P_{\theta}(\mathbf{h}|\mathbf{v}^{(0)})$, which factorizes due to the bipartite structure of the RBM. The second term, the negative phase, is an expectation over the joint model distribution and is computationally prohibitive as it sums over all possible configurations of **v** and **h**. Contrastive Divergence approximates this negative phase expectation by starting from the data point **v**(0) and running a short Markov Chain Monte Carlo (MCMC) simulation for *k* steps (typically *k* = 1), rather than running the chain to convergence.

The CD-*k* algorithm proceeds as follows. First, given a training sample **v**(0), sample a hidden state from the conditional distribution:

$$ \mathbf{h}^{(0)} \sim P_{\theta}(\mathbf{h}|\mathbf{v}^{(0)}) \quad (10.323) $$

For binary units, this is

$$ P(h_j = 1|\mathbf{v}) = \sigma(b_j + \sum_i W_{ij}v_i) \quad (10.324) $$

where $\sigma$ is the sigmoid function. Then, for *k* steps, alternately sample visible and hidden states: sample $\mathbf{v}^{(t)}$ from $P_{\theta}(\mathbf{v}|\mathbf{h}^{(t-1)})$ where

$$ P(v_i = 1|\mathbf{h}) = \sigma(a_i + \sum_j W_{ij}h_j) \quad (10.325) $$

and then sample $\mathbf{h}^{(t)}$ from $P_{\theta}(\mathbf{h}|\mathbf{v}^{(t)})$. After *k* steps, the gradient approximation for the weight matrix **W** is:

$$ \nabla_{W_{ij}} \log P_{\theta}(\mathbf{v}^{(0)}) \approx \mathbb{E}_{P_{\theta}(\mathbf{h}|\mathbf{v}^{(0)})}[v_i^{(0)}h_j] - \mathbb{E}_{P_{\theta}(\mathbf{h}|\mathbf{v}^{(k)})}[v_i^{(k)}h_j] = v_i^{(0)}P(h_j = 1|\mathbf{v}^{(0)}) - v_i^{(k)}P(h_j = 1|\mathbf{v}^{(k)}) \quad (10.326) $$

For the biases, the approximations are

$$ \nabla_{a_i} \log P_{\theta}(\mathbf{v}^{(0)}) \approx v_i^{(0)} - v_i^{(k)} \quad (10.327) $$

and

$$ \nabla_{b_j} \log P_{\theta}(\mathbf{v}^{(0)}) \approx P(h_j = 1|\mathbf{v}^{(0)}) - P(h_j = 1|\mathbf{v}^{(k)}) \quad (10.328) $$

The CD loss is not an explicit function to be minimized but rather defines this stochastic gradient update rule. The parameter update becomes

$$ \theta \leftarrow \theta + \eta(\langle \cdot \rangle_{\text{data}} - \langle \cdot \rangle_{\text{reconstruction}}) \quad (10.329) $$

where $\langle \cdot \rangle_{\text{data}}$ is the positive phase expectation and $\langle \cdot \rangle_{\text{reconstruction}}$ is the expectation estimated after *k* steps of sampling. The mathematical justification for CD is that it follows the gradient of a difference of two divergences:

$$ CD_k = D_{\text{KL}}(P_0 \| P_{\theta}^{\infty}) - D_{\text{KL}}(P_k \| P_{\theta}^{\infty}) \quad (10.330) $$

where $P_0$ is the data distribution, $P_{\theta}^{\infty}$ is the equilibrium distribution of the Markov chain defined by the model, and $P_k$ is the distribution of the visible units after *k* steps of the chain starting from the data. While $CD_k$ is not the gradient of the log-likelihood, it has been proven to be a valid learning rule that minimizes the KL divergence under certain conditions. The bias introduced by using a finite *k* is significant; CD-1 does not converge to the maximum likelihood estimate but to a fixed
<figure><img src="image_155.png" /><figcaption>Figure 10.74: Contrastive Divergence (CD) Loss Landscape</figcaption></figure>

point that minimizes a different objective. However, in practice, CD-1 works remarkably well for pre-training deep belief networks. The gradient estimates are noisy but unbiased estimators of the true gradient only if the chain is at equilibrium, which it is not for finite $k$. Persistent Contrastive Divergence (PCD) is a variant that maintains a single persistent chain across parameter updates to better approximate the model distribution, but CD-$k$ remains the foundational algorithm. The loss, interpreted as the contrast between the data-driven configuration and the model-driven reconstruction after a brief "dream" cycle, provides a computationally feasible mechanism for training generative models with latent variables, bridging the gap between theoretical maximum likelihood and practical iterative parameter estimation.

### 10.5.3.9.9 Persistent Contrastive Divergence (PCD) Loss

Persistent Contrastive Divergence (PCD) is an advanced algorithm for training energy-based models like Restricted Boltzmann Machines (RBMs) that addresses a fundamental limitation of the standard Contrastive Divergence (CD) algorithm by maintaining a persistent Markov chain across parameter updates to provide a better approximation of the model distribution used in the negative phase of learning. The core objective remains the maximization of the log-likelihood of the training data, whose gradient for a single datum $\mathbf{v}^{(0)}$ is

$$ \nabla_{\theta} \log P_{\theta}(\mathbf{v}^{(0)}) = \mathbb{E}_{P_{\theta}(\mathbf{h}|\mathbf{v}^{(0)})}[\nabla_{\theta} E(\mathbf{v}^{(0)}, \mathbf{h})] - \mathbb{E}_{P_{\theta}(\mathbf{v},\mathbf{h})}[\nabla_{\theta} E(\mathbf{v}, \mathbf{h})] \quad (10.331) $$

The intractability of the negative phase expectation $\mathbb{E}_{P_{\theta}(\mathbf{v},\mathbf{h})}[\cdot]$ is circumvented by MCMC sampling. While CD-$k$ initializes a chain from each training example $\mathbf{v}^{(0)}$ and runs it for a small number of steps $k$ (often just one), this approach biases the estimate because the chain is far from equilibrium
Contrastive Divergence (CD) Loss Landscape (3D)

<figure><img src="image_156.png" /><figcaption>Figure 10.75: Contrastive Divergence (CD) Loss Landscape (3D)</figcaption></figure>

when the samples are taken. PCD eliminates this bias by maintaining a single, persistent fantasy particle or a set of particles {$\tilde{\mathbf{v}}^{(t)}$} that are not reset to training data between parameter updates. Instead, after each parameter update, the current state of the chain is retained and used as the initial state for the next MCMC step. The PCD algorithm initializes a set of fantasy particles {$\tilde{\mathbf{v}}^{(0)}$} from some initial distribution (e.g., random or from a previous model). For each iteration of learning, given a minibatch of training data {$\mathbf{v}_{1}^{(0)}, \dots, \mathbf{v}_{m}^{(0)}$}, the parameter update for the weight matrix $\mathbf{W}$ is computed as:

$$ \Delta W_{ij} = \eta \left( \frac{1}{m} \sum_{\mu=1}^{m} \mathbb{E}[v_i h_j | \mathbf{v}_{\mu}^{(0)}] - \mathbb{E}[v_i h_j | \tilde{\mathbf{v}}^{(t)}] \right), \quad (10.332) $$

where $\eta$ is the learning rate. The key difference from CD is that the negative sample $\tilde{\mathbf{v}}^{(t)}$ is not obtained by running a short chain from a data point but is the current state of the persistent chain. After the parameter update, the fantasy particles are updated by performing one or more Gibbs sampling steps according to the new parameters $\theta'$. A Gibbs step consists of sampling the hidden units given the visible units:

$$ \tilde{\mathbf{h}}^{(t)} \sim P_{\theta'}(\mathbf{h}|\tilde{\mathbf{v}}^{(t)}) \quad (10.333) $$

and then sampling the new visible units:

$$ \tilde{\mathbf{v}}^{(t+1)} \sim P_{\theta'}(\mathbf{v}|\tilde{\mathbf{h}}^{(t)}) \quad (10.334) $$

This update is formalized as:

$$ \tilde{\mathbf{v}}^{(t+1)} \sim T_{\theta'}(\tilde{\mathbf{v}}^{(t+1)}|\tilde{\mathbf{v}}^{(t)}), \quad (10.335) $$
<figure><img src="image_157.png" /><figcaption>Figure 10.76: Persistent Contrastive Divergence (PCD) Loss Landscape</figcaption></figure>

where $T_\theta$ is the transition operator of the Markov chain defined by the model (e.g., block Gibbs sampling). Because the chain is persistent, the fantasy particles $\tilde{\mathbf{v}}^{(t)}$ eventually converge to samples from the stationary distribution $P_{\theta'}(\mathbf{v})$ if the parameters change slowly enough. This makes the negative phase estimate $\mathbb{E}[v_i h_j | \tilde{\mathbf{v}}^{(t)}]$ an asymptotically unbiased estimator of the true negative phase expectation $\mathbb{E}_{P_{\theta}(\mathbf{v},\mathbf{h})}[v_i h_j]$, unlike the biased estimate from CD-k.

The mathematical foundation of PCD relies on the theory of Markov chains and stochastic approximation. The learning process can be viewed as a stochastic approximation algorithm that aims to find the root of the function

$$f(\theta) = \mathbb{E}_{P_{\text{data}}} [\mathbb{E}_{P_{\theta}(\mathbf{h}|\mathbf{v})}[\nabla_{\theta} E(\mathbf{v}, \mathbf{h})]] - \mathbb{E}_{P_{\theta}(\mathbf{v},\mathbf{h})}[\nabla_{\theta} E(\mathbf{v}, \mathbf{h})] = 0 \quad (10.336)$$

The PCD update rule is

$$\theta_{t+1} = \theta_t + \eta_t (\langle \nabla_{\theta} E \rangle_{\text{data}} - \langle \nabla_{\theta} E \rangle_{\tilde{\mathbf{v}}^{(t)}}) \quad (10.337)$$

where the sequence $\{\tilde{\mathbf{v}}^{(t)}\}$ is a homogeneous Markov chain whose transition kernel depends on the evolving parameter $\theta_t$. The convergence of this algorithm is non-trivial because the samples $\tilde{\mathbf{v}}^{(t)}$ are drawn from a non-stationary distribution. However, under the condition that the learning rate $\eta_t$ decreases appropriately for

$$\sum_t \eta_t = \infty, \sum_t \eta_t^2 < \infty \quad (10.338)$$

and the parameter changes are sufficiently small, the chain can track the slowly evolving model distribution, and the algorithm converges to a maximum likelihood estimate. A common improvement is PCD with fast weights, which uses a separate, faster-evolving set of parameters to define the Markov chain while the primary parameters are updated more slowly, enhancing stability. The
Persistent Contrastive Divergence (PCD) Loss Landscape (3D)

<figure><img src="image_158.png" /><figcaption>Figure 10.77: Persistent Contrastive Divergence (PCD) Loss Landscape (3D)</figcaption></figure>

PCD loss, therefore, is not a closed-form function but an algorithmic procedure for estimating the gradient of the log-likelihood, providing a more accurate and stable training method for RBMs than CD by reducing the bias in the negative phase approximation through persistent state tracking.

10.5.3.9.10 Hausdorff Distance Loss

The Hausdorff Distance Loss is a metric for comparing two point sets, $A = \{a_1, \dots, a_N\}$ and $B = \{b_1, \dots, b_M\}$, that is particularly useful in image segmentation and shape matching due to its sensitivity to the worst-case discrepancy between the sets, making it robust to outliers and focused on the global structure of the boundaries rather than pixel-wise overlap. The classical Hausdorff distance is defined as the maximum of the directed distances from each set to the other:

$$H(A, B) = \max(h(A, B), h(B, A)) \qquad (10.339)$$

where the directed Hausdorff distance $h(A, B)$ is given by

$$h(A, B) = \max_{a \in A} \min_{b \in B} \|a - b\| \qquad (10.340)$$

and $\| \cdot \|$ is a underlying norm, typically the Euclidean distance. This formulation is highly non-convex and non-differentiable, and extremely sensitive to a single outlier point; a single point in $A$ far from any point in $B$ will cause $h(A, B)$ to be large regardless of the proximity of all other points. To mitigate this for practical optimization, a more general, differentiable approximation is used, often based on computing statistics over the distances rather than the strict maximum. The
generalized loss function for a predicted set B against a ground truth set A is formulated using a smooth rank-based approximation. One common approach is to replace the max and min operators with their smooth counterparts, such as the log-sum-exp for the maximum and a simple average or a soft-minimum for the minimum. The directed distance can be approximated as

$$h_{\text{soft}}(A, B) = \frac{1}{|A|} \sum_{a \in A} \min_{b \in B} \|a - b\|^p \quad (10.341)$$

for a finite p, but a more robust and differentiable version uses the concept of the Hausdorff distance based on the k-th ranked distance rather than the maximum. The k-th ranked distance is defined as the k-th smallest value of $\min_{b \in B} \|a - b\|$ for $a \in A$. The loss is then

$$H^k(A, B) = \max(h^k(A, B), h^k(B, A)) \quad (10.342)$$

where

$$h^k(A, B) = \underset{a \in A}{\operatorname{k-th}} \max_{b \in B} \|a - b\| \quad (10.343)$$

This is known as the Partial Hausdorff Distance and is less sensitive to outliers. For integration with gradient-based deep learning, a fully differentiable loss is essential. This is achieved by expressing the minimum and ranked maximum operations in terms of continuous functions. The minimum distance from a point a to the set B can be approximated with a softmax function:

$$d_{\text{softmin}}(a, B) = \frac{\sum_{b \in B} \|a - b\| \cdot \exp(-\lambda \|a - b\|)}{\sum_{b' \in B} \exp(-\lambda \|a - b'\|)} \quad (10.344)$$

where $\lambda$ is a temperature parameter controlling the sharpness of the approximation; as $\lambda \to \infty$, this approaches the true minimum. Similarly, the directed distance $h(A, B)$ can be approximated by a softmax over the distances $d(a, B)$ for all $a \in A$:

$$h_{\text{soft}}(A, B) = \frac{\sum_{a \in A} d(a, B) \cdot \exp(\lambda d(a, B))}{\sum_{a' \in A} \exp(\lambda d(a', B))} \quad (10.345)$$

The complete symmetric Hausdorff loss is then

$$L_{\text{HD}} = h_{\text{soft}}(A, B) + h_{\text{soft}}(B, A) \quad (10.346)$$

The gradient of this loss with respect to a point $b \in B$ in the predicted set is complex due to the nested nature of the operations. It involves the chain rule through the softmax and softmax functions. The derivative of the softmax function $d_{\text{softmax}}(a, B)$ with respect to a point b is:

$$\frac{\partial d_{\text{softmax}}}{\partial b} = \frac{\exp(-\lambda \|a - b\|)}{\sum_{b'} \exp(-\lambda \|a - b'\|)} \cdot \left( \frac{\partial \|a - b\|}{\partial b} - (d_{\text{softmax}}(a, B) - \|a - b\|) \cdot \lambda \frac{\partial \|a - b\|}{\partial b} \right) \quad (10.347)$$

This gradient assigns a weight to each point b based on its proximity to a. The derivative of the softmax-directed distance $h_{\text{soft}}(A, B)$ with respect to $d(a, B)$ is given by the Jacobian of the softmax, which is

$$\frac{\partial h_{\text{soft}}}{\partial d(a, B)} = \frac{\exp(\lambda d(a, B))}{\sum_{a'} \exp(\lambda d(a', B))} (1 + \lambda(d(a, B) - h_{\text{soft}}(A, B))) \quad (10.348)$$

The full gradient $\frac{\partial L_{\text{HD}}}{\partial b}$ is then obtained by summing the contributions from all points $a \in A$ through the chain rule:

$$\frac{\partial L_{\text{HD}}}{\partial b} = \sum_{a \in A} \frac{\partial h_{\text{soft}}(A, B)}{\partial d(a, B)} \cdot \frac{\partial d(a, B)}{\partial b} + \frac{\partial h_{\text{soft}}(B, A)}{\partial b} \quad (10.349)$$

The second term requires a symmetric calculation for points in A. In practice, for image segmentation, the point sets A and B are the coordinates of the boundary pixels of the ground truth and predicted masks. The Hausdorff Distance Loss directly penalizes the largest segmentation error, encouraging the model to accurately capture the entire contour of an object. This makes it highly effective for medical image segmentation, where the precise delineation of organ boundaries is critical, and it complements pixel-wise losses like Dice or Cross-Entropy by focusing on the worst-case error rather than the average.
<figure><img src="image_159.png" /><figcaption>Figure 10.78: Hausdorff Distance Loss Visualization</figcaption></figure>

10.5.3.10 Python Code to generate Figure 10.13 illustrating 1D Wasserstein Distance of two distributions having parameters: Normal($\mu$ = -1, $\sigma$ = 1) and Normal($\mu$ = 1.5, $\sigma$ = 1)

The Python code below produces the Figure 10.13 illustrating 1D Wasserstein Distance of two distributions having parameters: Normal($\mu$ = -1, $\sigma$ = 1) and Normal($\mu$ = 1.5, $\sigma$ = 1).

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import wasserstein_distance

# Define x-axis
x = np.linspace(-7, 7, 1000)

# Define two Gaussian distributions
mu_p, sigma_p = -1.0, 1.0
mu_q, sigma_q = 1.5, 1.0
```
<figure><img src="image_160.png" /><figcaption>Figure 10.79: Hausdorff Distance Heatmap</figcaption></figure>

```python
p = np.exp(-0.5 * ((x - mu_p)/sigma_p)**2) / (sigma_p * npsqrt(2*np.pi))
q = np.exp(-0.5 * ((x - mu_q)/sigma_q)**2) / (sigma_q * np.sq
rt(2*np.pi))

# Normalize (not strictly needed since Gaussian is normalized, but good practice)
p /= np.trapezoid(p, x)
q /= np.trapezoid(q, x)

# Compute Wasserstein distance
wd = wasserstein_distance(x, x, p, q)

# Plot
fig, ax = plt.subplots(figsize=(8,5))
ax = plt plot(x, p, label="P: Normal(\mu={mu_p}, \sigma={sigma_p})", lw=2)
ax = plt plot(x, q, label="Q: Normal(\mu={mu_q}, \sigma={sigma_q})", lw=2)
ax = plt fill-between(x, p, q, color="gray", alpha=0.3, label="Difference")

# Labels and title
ax xlabel("x", fontsize=12)
ax ylabel("Probability Density", fontsize=12)
ax title("1D Wasserstein Distance", fontsize=14)
```
```python
33 # Wasserstein distance box inside plot
34 plt.text(0.63, 0.95, f"Wasserstein Distance = {wd:.4f}",
35          transform= plt.gca().transAxes,
36         fontsize=10, bbox = dict(facecolor="white", edgecolor="black",
37                          boxstyle="round,pad=0.3"))
37
38 plt.legend()
39
40 plt.grid()
41 plt.show()
```

### 10.5.3.11 Python Code to generate Figure 10.14 illustrating Multidimensional Wasserstein Distance of two distributions having parameters, Distribution A: $N(\mu = [0,0], \Sigma = I)$, Distribution B: $N(\mu = [3,3], \Sigma = I)$

The Python code below produces the Figure 10.14 illustrating Multidimensional Wasserstein Distance of two distributions having parameters, Distribution A: $N(\mu = [0,0], \Sigma = I)$, Distribution B: $N(\mu = [3,3], \Sigma = I)$.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3 import ot   # πp install POT
4
5 # Generate two 2D Gaussian point clouds
6 n = 200
7 mu_a, cov_a = [0, 0], np.eye(2)
8 mu_b, cov_b = [3, 3], np.eye(2)
9
10 a = np.random.multivariate_normal(mu_a, cov_a, n)
11 b = np.random.multivariate_normal(mu_b, cov_b, n)
12
13 # Uniform weights
14 wa = np.ones(n) / n
15 wb = np.ones(n) / n
16
17 # Compute Wasserstein distance (Earth Mover's Distance)
18 wd = ot.emd2(wa, wb, ot.dist(a, b))
19
20 # Plot
21 plt.figure(figsize=(6,6))
22 plt.scatter(a[:,0], a[:,1], alpha=0.7, label=f"A: Normal (μ={mu_a}, Σ={wd})")
23 plt scatter(b[:,0], b[:,1], alpha=0.7, label=f"B: Normal (μ={mu_b}, Σ={wd})")
24
25 # Labels and title
26 plt.xlabel("x1", fontsize=12)
27 plt.ylabel("x2", fontsize=12)
28 plt.title("Multidimensional Wasserstein Distance", fontsize=14)
29
30 # Wasserstein distance box inside plot
31 plt.text(0.02, 0.98, f"Wasserstein Distance = {wd:.4f}",
32          transform= plt.gca().transAxes,
33         fontsize=12, bbox = dict(facecolor="white", edgecolor="black",
34                          boxstyle="round,pad=0.3"))
34
```
35 plt.legend()
36 plt.grid()
37 plt.show()

### 10.5.3.12 Python Code to generate Figure 10.15 comparing Equal CDF, WGAN Critic, GAN Discriminator

The Python code below produces the Figure 10.15 comparing Equal CDF, WGAN Critic, GAN Discriminator.

1 import numpy as np
2 import matplotlib.pyplot as plt
3 from scipy.stats import norm
4
5 # 1D domain
6 x = np.linspace(-4, 4, 500)
7
8 # Reference distribution mu_ref: standard normal
9 mu_ref = norm.pdf(x, loc=0, scale=1)
10 cdf_ref = norm.cdf(x, loc=0, scale=1)
11
12 # Generator distribution mu_G: shifted normal
13 mu_G = norm.pdf(x, loc=1.5, scale=1)
14 cdf_G = norm.cdf(x, loc=1.5, scale=1)
15
16 # 1. Equal CDF line
17 equal_cdf = np zeros_like(x)
18
19 # 2. Optimal WGAN Critic (scaled)
20 D_WGAN = cdf_ref - cdf_G
21 D_WGAND_WGAN / np max(np abs(D_WGAN)) # scale to [-1,1]
22
23 # 3. Optimal GAN Discriminator (scaled)
24 D_gan = mu_ref / (mu_ref + mu_G)
25 D_ganче = (D_gan - 0.5) * 2 # scale to [-1,1]
26
27 # Highlight points
28 # WGAN critic = 0 to index closest to zero
29 idx_zero_wgan = np argmin(np abs(D_WGANче))
30 x_zero_wgan = x[idx_zero_wgan]
31 y_zero_wgan = D_WGANче[idx_zero_wgan]
32
33 # GAN discriminator = 0.5 to index closest to 0.5 before scaling
34 idx_gan_05 = np argmin(np abs(D_gan - 0.5))
35 x_gan_05 = x[idx_gan_05]
36 y_gan_05 = D_ganче[idx_gan_05]
37
38 # Plot
39 plt figure(figsize=(10,6))
40 plt plot(x, equal_cdf, label='Equal CDF', color='black', lw=2)
41 plt plot(x, D_WGANче, label='Optimal WGAN Critic $D_{WGAN}$', lw
42 =2, color='blue')
43 plt plot(x, D_ganче, label='Optimal GAN Discriminator $D$', lw=2,
44 linestyle='--', color='red')
```python
44
# Plot highlights
45
plt.scatter(x_zero_wgan, y_zero_wgan, color='blue', s=100, edgecolor='k',
            label='WGAN Critic = 0')
46
plt scatter(x_gan_05, y_gan_05, color='red', s=100, edgecolor='k', label
            ='GAN Discriminator = 0.5')
47
48
plt.tight_layout()
49
```

### 10.5.3.13 Python Code to generate Figure 10.16 illustrating Least Squares GAN (LSGAN) Loss Functions

The Python code below produces the Figure 10.16 illustrating Least Squares GAN (LSGAN) Loss Functions.

```python
1
import numpy as np
2
import matplotlib.pyplot as plt
3
4
# Tyπcal LSGAN parameters
5
a, b, c = 0, 1, 1
6
7
# D(x) output range
8
d_values = np.linspace(-1, 2, 500)
9
# Loss functions
10
D_real_loss = 0.5 * (d_values - b)**2  # for real samples
11
D fake_loss = 0.5 * (d_values - a)**2  # for fake samples
12
G_loss = 0.5 * (d_values - c)**2    # for generator
13
14
# Plot
15
plt.figure(figsize=(8,6))
16
17
plot(d_values, D_real_loss, label=r"$L_D$ (real): $\frac{1}{2}(D(x)-1)^2$", lw=2)
18
plot(d_values, D fake_loss, label=r"$L_D$ (fake): $\frac{1}{2}(D(G(z))-0)^2$", lw=2)
19
plot(d_values, G_loss, label=r"$L_G$: $\frac{1}{2}(D(G(z))-1)^2$", lw=2)
20
# Labels and formatting
21
plot(xlabel=r"Discriminator output $D(x)$", fontsize=12)
22
plot(ylabel="Loss", fontsize=12)
23
plot.title("Least Squares GAN (LSGAN) Loss Functions", fontsize=14)
24
plot.legend()
25
plot.grid(True)
26
plot.show()
27
```
```python
17 plt.plot(d, L_dissimilar, label=rf"Dissimilar pair ($y=1$): $\max(0, \{m
18 }-d)^2\$", lw=2)
19 # Labels and formatting
20 plt.xlabel("Euclidean Distance $d\$", fontsize=12)
21 plt.ylabel("Loss", fontsize=12)
22 plt.title("Contrastive Loss", fontsize=14)
23 plt.legend()
24 plt.grid(True)
25
26 # Mark margin
27 plt axvline(m, color="red", linewidth="--", label="Margin")
28 plt.legend()
29
30 plt.show()
```

### 10.5.3.16 Python Code to generate Figure 10.19 illustrating Contrastive Loss Landscapes

The Python code below produces the Figure 10.19 illustrating Contrastive Loss Landscapes.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 # -------------------------------
5 # Contrastive loss function
6 # -------------------------------
7 def contrastive_loss(x1, x2, y, margin=1.0):
8 """ x1, x2 : coordinates of two points (can be arrays)
9 y : similarity label (1=similar, 0=dissimilar)
10 """
11 D = np.abs(x1 - x2)  # L1 distance for 1D example
12 loss = y * D**2 + (1-y) * np maximum(0, margin - D)**2
14 return loss
15
16 # -------------------------------
17 # Experiment setup
18 # -------------------------------
19 x_range = np.linspace(-2, 3, 200)
20 X1, X2 = npgrid(x_range, x_range)
21
22 # Similar pair (y=1)
23 loss_similar = contrastive_loss(X1, X2, y=1, margin=1.0)
24 # Dissimilar pair (y=0)
25 loss_dissimilar = contrastive_loss(X1, X2, y=0, margin=1.0)
26
27 # -------------------------------
28 # Plot
29 # -------------------------------
30 plt.figure(figsize=(12,5))
31
32 # Similar
33 pltsubplot(1,2,1)
34 cont1 = plt.contourf(X1, X2, loss_similar, levels=50, cmap='viridis')
```
```python
35 plt.colorbar(cont1, label='Loss')
36 plt.title("Contrastive Loss Landscape (Similar Pair y=1)")
37 plt.xlabel("x1")
38 plt.ylabel("x2")
39
40 # Dissimilar
41 pltsubplot(1,2,2)
42 cont2 = plt.contourf(X1, X2, loss_dissimilar, levels=50, cmap='plasma')
43 plt.colorbar(cont2, label='Loss')
44 plt.title("Contrastive Loss Landscape (Dissimilar Pair y=0)")
45 plt.xlabel("x1")
46 plt.ylabel("x2")
47
48 plt.suptitle("Contrastive Loss Landscapes", fontsize=14)
49 plt.show()
```

### 10.5.3.17 Python Code to generate Figure 10.20 illustrating 3D Contrastive Loss Landscapes

The Python code below produces the Figure 10.20 illustrating 3D Contrastive Loss Landscapes.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3 from mpl_toolkits.mplot3d import Axes3D
4
5 # -------------------------------
6 # Contrastive loss function
7 # -------------------------------
8 def contrastive_loss(x1, x2, y, margin=1.0):
9     D = np.abs(x1 - x2)  # L1 distance for 1D example
10    loss = y * D**2 + (1-y) * np maximum(0, margin - D)**2
11    return loss
12
13 # -------------------------------
14 # Experiment setup
15 # -------------------------------
16 x_range = np.linspace(-2, 3, 200)
17 X1, X2 = npgrid(x_range, x_range)
18
19 # Loss surfaces
20 loss_similar = contrastive_loss(X1, X2, y=1, margin=1.0)
21 loss_dissimilar = contrastive_loss(X1, X2, y=0, margin=1.0)
22
23 # -------------------------------
24 # 3D Surface Plots
25 # -------------------------------
26 fig = plt.figure(figsize=(14,6))
27
28 # Similar pair (y=1)
29 ax1 = fig.add_subplot(1,2,1, projection='3d')
30 surf1 = ax1 plot Cole (X1, X2, loss_similar, cma = 'viridis', edgecolor
31 = 'none', alpha=0.9)
32 ax1.set_title("Contrastive Loss Surface (Similar Pair y=1)")
33 ax1.set_xlabel("x1")
34 ax1.set_ylabel("x2")
```
```python
34 ax1.set_zlabel("Loss")
35 fig.colorbar(surf1, ax=ax1, shrink=0.5, aspect=10)
36
37 # Dissimilar pair (y=0)
38 ax2 = fig.add_subplot(1,2,2, projection='3d')
39 surf2 = ax2 plot(X1, X2, loss_dissimilar, cmap='plasma',
40                       edgecolor='none', alpha=0.9)
41 ax2.set_title("Contrastive Loss Surface (Dissimilar Pair y=0)")
42 ax2.set_xlabel("x1")
43 ax2.set_ylabel("x2")
44 ax2.set_zlabel("Loss")
45 fig.colorbar(surf2, ax=ax2, shrink=0.5, aspect=10)
46
47 plt.suptitle("3D Contrastive Loss Landscapes", fontsize=14)
48 plt.show()
```

### 10.5.3.18 Python Code to generate Figure 10.21 illustrating Triplet Loss Function

The Python code below produces the Figure 10.21 illustrating Triplet Loss Function.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 # Anchor-positive distance (fixed)
5 d_ap = 0.5
6
7 # Anchor-negative distance (variable)
8 d_an = np.linspace(0, 2.0, 500)
9
10 # Margin
11 alpha = 0.5
12
13 # Triplet loss
14 triplet_loss = npassum(0, d_ap**2 - d_an**2 + alpha)
15
16 # Plot
17 plt.figure(figsize=(8,6))
18 plt plot(d_an, triplet_loss, lw=2, color='blue', label='Triplet Loss (
19         d_ap={d_ap}, alpha={alpha})')
20
21 # Labels and title
22 plt.xlabel("Anchor-Negative Distance $d_{an}$", fontsize=12)
23 plt.ylabel("Triplet Loss", fontsize=12)
24 plt.title("Triplet Loss Function", fontsize=14)
25 plt.grid(True)
26
27 # Highlight margin point
28 margin_point = npsqrt(d_ap**2 + alpha)
29 plt.axvline(margin_point, color='red', linewidth=1, label='Margin
30         Point')
31 plt.legend()
32 plt.show()
```
### 10.5.3.19 Python Code to generate Figure 10.22 illustrating Triplet Loss Landscape

The Python code below produces the Figure 10.22 illustrating Triplet Loss Landscape.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

# Define range of distances
d_ap = np.linspace(0, 2.0, 100)  # anchor-positive distance
d_an = np.linspace(0, 3.0, 100)  # anchor-negative distance

# Margin
alpha = 0.5

# Create meshgrid
DAP, DAN = np.meshgrid(d_ap, d_an)

# Compute Triplet Loss
triplet_loss = np maximum(0, DAP**2 - DAN**2 + alpha)

# Plot 3D surface
fig = plt. figure(figsize=(10,7))
ax = fig. add_subplot(111, projection='3d')
surf = ax. plot_surface(DAP, DAN, triplet_loss, cmap='viridis', edgecolor
                       = 'k', alpha=0.9)

ax. set_xlabel("Anchor-Positive Distance $d_{\{ap\}}$", fontsize=12)
ax. set_ylabel("Anchor-Negative Distance $d_{\{an\}}$", fontsize=12)
ax. set_zlabel("Triplet Loss", fontsize=12)
ax. set_title("Triplet Loss Landscape", fontsize=14)

fig. colorbar(surf, shrink=0.5, aspect=10, label='Triplet Loss')

plt. show()
```

### 10.5.3.20 Python Code to generate Figure 10.23 illustrating Triplet Loss Contour Plot

The Python code below produces the Figure 10.23 illustrating Triplet Loss Contour Plot.

```python
import numpy as np
import matplotlib. plot as plt

# Define range of distances
d_ap = np. linspace(0, 2.0, 500)  # anchor-positive distance
d_an = np. linspace(0, 3.0, 500)  # anchor-negative distance

# Margin
alpha = 0.5

# Create meshgrid
DAP, DAN = np. meshgrid(d_ap, d_an)

# Compute Triplet Loss
triplet_loss = np. maximum(0, DAP**2 - DAN**2 + alpha)
```
```python
17 # Plot contour
18 plt.figure(figsize=(8,6))
19 contour = plt.contourf(DAP, DAN, triplet_loss, levels=50, cmap='viridis')
20 plt.colorbar(contour, label='Triplet Loss')
21
22 # Plot margin boundary: d_an^2 = d_ap^2 + alpha
23 margin_dan = np Round(d_ap**2 + alpha)
24 plt. plot(d_ap, margin_dan, color='red', linewidth=2, label='Margin Boundary')
25
26 plt. xlabe l("Anchor-Positive Distance $d_{\{ap\}}$", fontsize=12)
27 plt. ylabel("Anchor-Negative Distance $d_{\{an\}}$", fontsize=12)
28 plt. title("Triplet Loss Contour Plot", fontsize=14)
29 plt. legend()
30 plt. grid(True)
31 plt. show()
```

### 10.5.3.21 Python Code to generate Figure 10.24 illustrating Center Loss Function

The Python code below produces the Figure 10.24 illustrating Center Loss Function.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 # Distance from feature to class center
d = np. linspace(0, 5, 500)
5
6 # Center loss: 0.5 * ||x - c||^2
center_loss = 0.5 * d**2
7
8 # Plot
9
10 plt. figure(figsize=(8,6))
11 plt. plot(d, center_loss, lw=2, color='purple', label='Center Loss 0.5*d^2')
12
13 # Labels and title
14
15 plt. xlabe l("Distance from Feature to Class Center ||x - c||", fontsize
16 =12)
17 plt. ylabel("Center Loss", fontsize=12)
18 plt. title("Center Loss Function", fontsize=14)
19 plt. grid(True)
20 plt. legend()
21 plt. show()
```

### 10.5.3.22 Python Code to generate Figure 10.25 illustrating Center Loss Visualization: Features vs Class Centers

The Python code below produces the Figure 10.25 illustrating Center Loss Visualization: Features vs Class Centers.

```python
1 import numpy as np
2 import matplotlib. plot as plt
3
4 # Number of classes
```
```python
5 num_classes = 3
6
7 # Class centers (2D for visualization)
8 centers = np.array([[0, 0], [3, 3], [6, 0]])
9
10 # Generate sample features around each center
11 np.random.seed(0)
12 features = []
13 labels = []
14
15 for i, center in enumerate(centers):
16     # 10 samples per class, random normal around center
17     feats = center + 0.8 * np.randomonde(10, 2)
18     features.append fea)
19     labels.extend([i]*10)
20
21 features = np.vstack(fea)
22 labels = np.array labels)
23
24 # Compute Center Loss for each feature
25 losses = np(zeros(fea.shape[0]))
26 for i, feature in enumerate(fea):
27     center = centers[labels[i]]
28     d = np.lndalg.norm(feature - center)
29     losses[i] = 0.5 * d**2
30
31 # Plot features and centers
32 plt.figure(figsize=(8,6))
33 for i in range(num_classes):
34     plt.scatter(fea[labels==i,0], fea[labels==i,1], label=f"Class {i} Samples", s=50)
35     plt.scatter(centers[i,0], centers[i,1], color='black', marker='X', s=200, label=f"Class {i} Center")
36
37 # Draw lines from features to their centers to visualize contribution to loss
38 for i in range(fea.shape[0]):
39     plt plot([features[i,0], centers[labels[i],0]], [features[i,1], centers[labels[i],1]], 'k--', alpha=0.5)
40
41 plt.xlabel("Feature Dimension 1")
42 plt.ylabel("Feature Dimension 2")
43 plt.title("Center Loss Visualization: Features vs Class Centers")
44 plt.grid(True)
45 plt.legend()
46 plt.show()
47
48 # Optional: print average loss
49 print(f"Average Center Loss: {np.mean(losses):.3f}")
```

### 10.5.3.23 Python Code to generate Figure 10.26 illustrating 3D Center Loss Surface

The Python code below produces the Figure 10.26 illustrating 3D Center Loss Surface.

```python
1 import numpy as np
```
13
14 plt.tight_layout() 15
16 plt.figure() 17
18 plt.title("Perceptual Loss Function", fontsize=14)
19 plt.grid(True)
20
21
22
23
24
25
26
27

10.5.3.25 Python Code to generate Figure 10.28 illustrating Pixel-wise MSE vs Per-
ceptual Loss (Nonlinear Features)

The Python code below produces the Figure 10.28 illustrating Pixel-wise MSE vs Perceptual Loss (Nonlinear Features).

1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 # Simulate πxel differences
5 πxel_diff = np.linspace(0, 5, 500)
6
7 # Simulate feature differences as a nonlinear function of πxel differences
8 # This represents perceptual loss capturing more semantic differences
9 feature_diff = npsqrt(πxel_diff) + 0.3 * npsin(2*πxel_diff)
10
11 # Pixel-wise MSE loss
12 mse_loss = πxel_diff**2
13
14 # Perceptual loss (L2 in feature space)
15 perceptual_loss = feature_diff**2
16
17 # Plot
18 plt.figure(figsize=(8,6))
19 plt plot(πxel_diff, mse_loss, lw=2, color='red', label='Pixel-wise MSE Loss')
20 plt plot(πxel_diff, perceptual_loss, lw=2, color='purple', label='Perceptual Loss')
21
22 plt xlabel("Pixel Difference", fontsize=12)
23 plt ylabel("Loss Value", fontsize=12)
24 plt title("Pixel-wise MSE vs Perceptual Loss (Nonlinear Features)", fontsize=14)
25 plt grid(True)
26 plt legend()
27 plt show()

10.5.3.26 Python Code to generate Figure 10.29 illustrating Style Loss Function in Neural Style Transfer

The Python code below produces the Figure 10.29 illustrating Style Loss Function in Neural Style Transfer.

1 import numpy as np
2 import matplotlib.pyplot as plt
3
```python
# Simulated feature correlation differences (from Gram matrices)
corr_diff = np.linspace(0, 5, 500)

# Style loss = squared Frobenius norm of Gram matrix difference
style_loss = corr_diff**2

# Plot
plt.figure(figsize=(8,6))
plt.plot(corr_diff, style_loss, 'r', 'o', markersize=4, label='Style Loss $||G(\phi(x)-G(\phi(y))||_F^2$')

# Plot
plt.figure(figsize=(8,6))
plt.plot(corr_diff, style_loss, 'r', 'o', markersize=4, label='Style Loss $||G(\phi(x)-G(\phi(y))||_F^2$')

# Plot
plt.figure(figsize=(8,6))
plt.plot(corr_diff, style_loss, 'r', 'o', markersize=4, label='Style Loss $||G(\phi(x)-G(\phi(y))||_F^2$')

# Plot
plt.figure(figsize=(8,6))
plt.plot(corr_diff, style_loss, 'r', 'o', markersize=4, label='Style Loss $||G(\phi(x)-G(\phi(y))||_F^2$')

# Plot
plt.figure(figsize=(8,6))
plt.plot(corr_diff, style_loss, 'r', 'o', markersize=4, label='Style Loss $||G(\phi(x)-G(\phi(y))||_F^2$')

# Plot
plt.figure(figsize=(8,6))
plt.plot(corr_diff, style_loss, 'r', 'o', markersize=4, label='Style Loss $||G(\phi(x)-G(\phi(y))||_F^2$')

# Plot
plt.figure(figsize=(8,6))
plt.plot(corr_diff, style_loss, 'r', 'o', markersize=4, label='Style Loss $||G(\phi(x)-G(\phi(y))||_F^2$')

# Plot
plt.figure(figsize=(8,6))
plt.plot(corr_diff, style_loss, 'r', 'o', markersize=4, label='Style Loss $||G(\phi(x)-G(\phi(y))||_F^2$')

# Plot
plt.figure(figsize=(8,6))
plt.plot(corr_diff, style_loss, 'r', 'o', markersize=4, label='Style Loss $||G(\phi(x)-G(\phi(y))||_F^2$')

# Plot
plt.figure(figsize=(8,6))
plt plot(scales, loss_values, color='purple')

# Plot
```

### 10.5.3.27 Python Code to generate Figure 10.30 illustrating Style Loss vs Scaling of Generated Features

The Python code below produces the Figure 10.30 illustrating Style Loss vs Scaling of Generated Features.

```python
import numpy as np
import matplotlib.pyplot as plt

def gram_matrix(F):
    C, H, W = F.shape
    F_flat = F.reshape(C, H*W)
    G = F_flat @ F_flat.T
    return G

def style_loss(F_gen, F_style):
    G_gen = gram_matrix(F_gen)
    G_style = gram_matrix(F_style)
    C, H, W = F_gen.shape
    return np.sum((G_gen - G_style)**2) / (4 * (C**2) * (H*W)**2)

# Simulate feature maps
C, H, W = 3, 8, 8  # 3 channels, small spatial size
F_style = np.random rand(C, H, W)

# Scaling factors for generated image features
scales = np.linspace(0.0, 2.0, 100)
loss_values = []

for s in scales:
    F_gen = s * F_style  # scale generated features
    loss_values.append(style_loss(F_gen, F_style))

# Plot

# Plot
```
```python
31 plt.xlabel('Scaling Factor of Generated Features')
32 plt.ylabel('Style Loss')
33 plt.title('Style Loss vs Scaling of Generated Features')
34 plt.grid(True)
35 plt.show()
```

### 10.5.3.28 Python Code to generate Figure 10.31 illustrating Total Variation (TV) Loss

The Python code below produces the Figure 10.31 illustrating Total Variation (TV) Loss.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def tv_loss_anisotroπc(img):
5     diff_x = np.abs(img[:, 1:] - img[:1, :]).sum()
6     diff_y = np abs(img[1:, :] - img[:-1, :]).sum()
7     return diff_x + diff_y
8
9 def tv_loss_isotroπc(img):
10     diff_x = img[:, 1:] - img[:1, :]  # (H, W-1)
11     diff_y = img[1:, :] - img[:-1, :]  # (H-1, W)
12     diff_x = diff_x[:-1, :]          # align to (H-1, W-1)
13     diff_y = diff_y[:-1, :]          # align to (H-1, W-1)
14     return npsqrt(diff_x**2 + diff_y**2).sum()
15
16 # Generate two images: smooth gradient and noisy
17 x = np.linspace(0, 1, 64)
18 y = np.linspace(0, 1, 64)
19 xx, yy = np.meshgrid(x, y)
20
21 smooth img = (xx + yy) / 2
22 noisy img = smooth img + 0.2 * np.randomonde(*smooth img.shape)
23
24 # Compute losses
25 tv.smooth_aniso = tv_loss_anisotroπc(smooth img)
26 tv.noisy_aniso = tv_loss_anisotroπc(noisy img)
27
28 tv.smooth_iso = tv_loss_isotroπc(smooth img)
29 tv.noisy_iso = tv_loss_isotroπc(noisy img)
30
31 # Plot images and TV loss values
32 plt.figure(figsize=(10,4))
33
34 pltsubplot(1,2,1)
35 plt.imshow(smooth img, cmap='gray')
36 plt.title(f"Smooth Image\nTV Loss (Aniso)={tv.smooth_aniso:.2f}, (Iso)={tv.smooth_iso:.2f}")
37 plt axis('off')
38
39 pltsubplot(1,2,2)
40 plt.imshow(noisy img, cmap='gray')
41 plt.title(f"Noisy Image\nTV Loss (Aniso)={tv.noisy_aniso:.2f}, (Iso)={tv.noisy_iso:.2f}")
42 plt axis('off')
```
```python
43
44 plt.tight_layout()
45 plt.show()
```

### 10.5.3.29 Python Code to generate Figure 10.32 illustrating Total Variation Loss vs Noise Level

The Python code below produces the Figure 10.32 illustrating Total Variation Loss vs Noise Level.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def tv_loss_anisotropic(img):
5     diff_x = np.abs(img[:, 1:] - img[:1, :]).sum()
6     diff_y = np abs(img[1:, :])
7     return diff_x + diff_y
8
9 def tv_loss_isotropic(img):
10     diff_x = img[:1, :] - img[:1, :-1]  # (H, W-1)
11     diff_y = img[1:, :] - img[:-1, :]  # (H-1, W)
12     diff_x = diff_x[:-1, :]          # align to (H-1, W-1)
13     diff_y = diff_y[:-1, :]          # align to (H-1, W-1)
14     return np.hypot(diff_x**2 + diff_y**2).sum()
15
16 # Base smooth image
17 size = 64
18 x = np.linspace(0, 1, size)
19 y = np.linspace(0, 1, size)
20 xx, yy = np.meshgrid(x, y)
21 smooth img = (xx + yy) / 2
22
23 # Noise levels
24 noiseLevels = np.linspace(0, 0.5, 20)
25 tv_aniso_values = []
26 tv iso values = []
27
28 # Compute TV loss for increasing noise
29 for sigma in noiseLevels:
30     noisy img = smooth img + sigma * np.randomonde(size, size)
31     tv_aniso_values.append (tv_loss_anisotropic (noisy img))
32     tv iso values.append (tv_loss_isotropic (noisy img))
33
34 # Plot
35 plt .figure (figsize=(8,6))
36 plt .plot (noiseLevels, tv_aniso_values, label='Anisotropic TV Loss',
37          marker='o')
38 plt .plot (noiseLevels, tv iso values, label='Isotropic TV Loss', marker='s')
39 plt .xlabel('Gaussian Noise Std Dev $\sigma$')
40 plt .ylabel('TV Loss')
41 plt .title('Total Variation Loss vs Noise Level')
42 plt .legend()
43 plt .grid (True)
44 plt .show()
```
### 10.5.3.30 Python Code to generate Figure 10.33 illustrating Evidence Lower Bound (ELBO) Loss during Training

The Python code below produces the Figure 10.33 illustrating Evidence Lower Bound (ELBO) Loss during Training.

```python
import numpy as np
import matplotlib.pyplot as plt

# Example dummy training data (replace with real values from your VAE
training loop)
epochs = np.arange(1, 51)
reconstruction_loss = np.exp(-0.1 * epochs) * 100  # dummy curve
kl_divergence = np.log(epochs + 1) * 2         # dummy curve

# ELBO = - (Reconstruction Loss + KL Divergence)
# (sign convention: some implementations minimize negative ELBO)
elbo = -(reconstruction_loss + kl_divergence)

plt.figure(figsize=(8,6))
plt.plot(epochs, elbo, label="ELBO Loss", linewidth=2)
plt plot(epochs, -reconstruction_loss, '--', label="Reconstruction Term")

16
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
```

### 10.5.3.31 Python Code to generate Figure 10.34 illustrating Negative Log-Likelihood (NLL) Loss

The Python code below produces the Figure 10.34 illustrating Negative Log-Likelihood (NLL) Loss.

```python
import numpy as np
import matplotlib.pyplot as plt

# Example dummy NLL values over 50 epochs (replace with actual training
logs)
epochs = np.arange(1, 51)
nll_loss = np.exp(-0.08 * epochs) * 2.0  # decreasing loss

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
```
### 10.5.3.32 Python Code to generate Figure 10.35 illustrating Domain Adversarial Loss during Training

The Python code below produces the Figure 10.35 illustrating Domain Adversarial Loss during Training.

```python
import numpy as np
import matplotlib.pyplot as plt

# Example dummy data (replace with real logs from your model)
epochs = np.arange(1, 51)
task_loss = np.exp(-0.08 * epochs) * 1.5          # source
    classification loss
domain_loss = (1 / (epochs + 1)) * 0.8          # domain
    classifier loss
lambda_val = 0.5                               # weighting factor

# Domain Adversarial Loss
dann_loss = task_loss - lambda_val * domain_loss

plt.figure(figsize=(8,6))
    plt plot(epochs, dann_loss, label="Domain Adversarial Loss", color="blue")
    plt linewidth=2)
    plt plot(epochs, task_loss, '--', label="Task Loss (classification)")
    color="green")
    plt plot(epochs, domain_loss, '--', label="Domain Loss", color="red")

    plt title("Domain Adversarial Loss during Training", fontsize=14)
    plt xlabel("Epoch", fontsize=12)
    plt ylabel("Loss Value", fontsize=12)
    plt legend()
    plt grid(True,olinestyle="--", alpha=0.6)
    plt show()
```

### 10.5.3.33 Python Code to generate Figure 10.36 illustrating Maximum Mean Discrepancy (MMD) Loss during Training

The Python code below produces the Figure 10.36 illustrating Maximum Mean Discrepancy (MMD) Loss during Training.

```python
import numpy as np
import matplotlib.pyplot as plt

# Gaussian RBF kernel
def rbf_kernel(x, y, gamma=1.0):
    x = x[:, None]
    y = y[None, :]
    return np.exp(-gamma * (x - y) ** 2)

# Compute MMD^2 between two samples
def compute_mmd(xSamples, ySamples, gamma=1.0):
    xx = rbf_kernel(xSamples, xSamples, gamma).mean()
    yy = rbf_kernel(ySamples, ySamples, gamma).mean()
    xy = rbf_kernel(xSamples, ySamples, gamma).mean()
    return xx + yy - 2 * xy
```
```python
# Simulate training: target distribution shifts closer to source
epochs = np.arange(1, 51)
mmd_loss = []

for epoch in epochs:
    # Source samples ~ N(0,1)
    source = np.random.normal(0, 1, 100)
    # Target samples ~ N(shift, 1), shift decreases with epoch
    shift = 2.0 / epoch
    target = np.random.normal(shift, 1, 100)

    mmd_val = compute_mmd(source, target, gamma=1.0)
    mmd_loss.append(mmd_val)

# Plot MMD Loss
plt.figure(figsize=(8,6))
plt.plot(epochs, mmd_loss, label="MMD Loss", color="purple", linewidth=2)

plt.title("Maximum Mean Discrepancy (MMD) Loss during Training",
          fontsize=14)

plt.xlabel("Epoch", fontsize=12)

plt.ylabel("Loss Value", fontsize=12)

plt.legend()

plt.grid(True, style="--", alpha=0.6)

plt.show()
```

### 10.5.3.34 Python Code to generate Figure 10.37 illustrating IoU Loss and Variants (GIoU Loss, DIoU Loss, CIoU Loss)

The Python code below produces the Figure 10.37 illustrating IoU Loss and Variants (GIoU Loss, DIoU Loss, CIoU Loss).

```python
import numpy as np
import matplotlib.pyplot as plt

# IoU
def iou.box1, box2):
    xi1 = max.box1[0], box2[0])
    yi1 = max.box1[1], box2[1])
    xi2 = min.box1[2], box2[2])
    yi2 = min.box1[3], box2[3])

    inter_area = max(0, xi2 - xi1) * max(0, yi2 - yi1)
    box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1])
    box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1])
    union_area = box1_area + box2_area - inter_area

    return inter_area / union_area if union_area > 0 else 0

# GIoU
def giou.box1, box2):
    iou_val = iou.box1, box2)
    xc1, yc1 = min.box1[0], box2[0]), min.box1[1], box2[1])
    xc2, yc2 = max.box1[2], box2[2]), max.box1[3], box2[3])
    area_c = (xc2 - xc1) * (yc2 - yc1)
    box1_area = (box1[2] - box1[0]) * (box1[3] - box1[1])
    box2_area = (box2[2] - box2[0]) * (box2[3] - box2[1])
```
```python
24 inter_area = iou_val * (box1_area + box2_area - iou_val)
25 union_area = box1_area + box2_area - inter_area
26 return iou_val - (area_c - union_area) / area_c
27
28 # DIoU
29 def diou.box1, box2):
30     iou_val = iou.box1, box2)
31     b1_x, b1_y = (box1[0] + box1[2]) / 2, (box1[1] + box1[3]) / 2
32     b2_x, b2_y = (box2[0] + box2[2]) / 2, (box2[1] + box2[3]) / 2
33     center_dist = (b1_x - b2_x)**2 + (b1_y - b2_y)**2
34     xc1, yc1 = min.box1[0], box2[0]), min.box1[1], box2[1])
35     xc2, yc2 = max.box1[2], box2[2]), max.box1[3], box2[3])
36     diag = (xc2 - xc1)**2 + (yc2 - yc1)**2
37     return iou_val - center_dist / (diag + 1e-7)
38
39 # CIoU
40 def ciou.box1, box2):
41     diou_val = diou.box1, box2)
42     iou_val = iou.box1, box2)
43     w1, h1 = box1[2] - box1[0], box1[3] - box1[1]
44     w2, h2 = box2[2] - box2[0], box2[3] - box2[1]
45     v = (4/np.pi**2) * (np.arctan(w1/h1) - np.arctan(w2/h2))**2
46     alpha = v / (1 - iou_val + v + 1e-7)
47     return diou_val - alpha * v
48
49 # Ground truth box
50 gt_box = [0, 0, 2, 2]
51
52 # Simulate: predicted box shifts and changes width/height
53 shifts = np.linspace(0, 2, 50)
54 iou vals, giou vals, diou vals, ciou vals = [], [], [], []
55
56 for i, s in enumerate(shifts):
57     width = 2 + 0.05*i  # slowly increasing width
58     height = 2 - 0.03*i  # slowly decreasing height
59     pred_box = [s, s, s+width, s+height]
60     iou vals.append(iou(gt_box, pred_box))
61     giou vals.append(giou(gt_box, pred_box))
62     diou vals.append(diou(gt_box, pred_box))
63     ciou vals.append(ciou(gt_box, pred_box))
64
65 # Convert to Loss
66 iou loss = [1 - v for v in iou vals]
67 giou loss = [1 - v for v in giou vals]
68 diou loss = [1 - v for v in diou vals]
69 ciou loss = [1 - v for v in ciou vals]
70
71 # Plot
72 plt.figure(figsize=(8,6))
73 plt plot(shifts, iou loss, label="IoU Loss", linewidth=2, color="blue")
74 plt plot(shifts, giou loss, label="GIoU Loss", linewidth=2, color="orange")
75 plt plot(shifts, diou loss, label="DIoU Loss", linewidth=2, color="green")
76 plt plot(shifts, ciou loss, label="CIoU Loss", linewidth=2, color="red")
```
linestyle="--", marker="x", markersize=3)

77 plt.title("IoU Loss and Variants (GIoU Loss, DIoU Loss, CIoU Loss)",
4 fontsize=14)
78 plt.tight_layout()  # Ensure the plot is saved with all elements visible
79 plt.xlabel("Shift of Predicted Box", fontsize=12)
80 plt.ylabel("Loss Value", fontsize=12)
81 plt.legend()
82 plt.grid(True, style="--", alpha=0.6)
83 plt.show()

### 10.5.3.35 Python Code to generate Figure 10.38 illustrating Dice Loss during Training

The Python code below produces the Figure 10.38 illustrating Dice Loss during Training.

1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 # Dice coefficient
5 def diceCoefficient(y_true, y_pred, smooth=1e-7):
6     intersection = np.sum(y_true * y_pred)
7     return (2. * intersection + smooth) / (np.sum(y_true) + np.sum(
8     y_pred) + smooth)
9
10 # Dice loss
11 def dice_loss(y_true, y_pred):
12     return 1 - diceCoefficient(y_true, y_pred)
13
14 # Simulate training
15 epochs = np.arange(1, 51)
16 dice_lost = []
17
18 for epoch in epochs:
19     # Ground truth mask (binary)
20     y_true = np.randomonde(0, 2, (64, 64))
21
22     # Prediction improves over time: closer to y_true
23     noise = np.randomonde(64, 64)
24     threshold = 0.5 - (epoch * 0.01)  # prediction improves
25     y_pred = (noise < (y_true * 0.7 + threshold)).astype(int)
26
27     loss = dice_loss(y_true, y_pred)
28     dice_lost.append(loss)
29
30 # Plot Dice Loss
31 plt.figure(figsize=(8,6))
32 plt plot(epochs, dice_lost, label="Dice Loss", color="purple",
33         linewidth=2)
34
35 plt title("Dice Loss during Training", fontsize=14)
36 plt.xlabel("Epoch", fontsize=12)
37 plt.ylabel("Loss Value", fontsize=12)
38 plt.legend()
39 plt grid(True, style="--", alpha=0.6)
40 plt show()
### 10.5.3.36 Python Code to generate Figure 10.39 illustrating Focal Loss vs Predicted Probability

The Python code below produces the Figure 10.39 illustrating Focal Loss vs Predicted Probability.

```python
import numpy as np
import matplotlib.pyplot as plt

def focal_loss(p, y, alpha=0.25, gamma=2.0):
    """
    Compute Focal Loss for a single example.
    p: predicted probability of class 1
    y: true label (0 or 1)
    alpha: weighting factor
    gamma: focusing parameter
    """
    pt = p if y == 1 else 1 - p
    return -alpha * (1 - pt)**gamma * np.log(np clip(pt, 1e-8, 1.0))

# Probability range from 0 to 1
p = np.linspace(0, 1, 1000)

# Compute Focal Loss for positive (y=1) and negative (y=0) classes
fl_pos = [focal_loss(π, 1) for π in p]
fl谓 = [focal_loss(π, 0) for π in p]

# Plot
fig, ax = plt.subplots(figsize=(8,6))
ax plot(p, fl_pos, 'r', label='Focal Loss y=1', color='blue')
ax plot(p, fl谓, 'g', label='Focal Loss y=0', color='red')
axxlabel('Predicted Probability p')
axylabel('Focal Loss')
axlegend()
axgrid(True)
axshow()
```

### 10.5.3.37 Python Code to generate Figure 10.40 illustrating Focal Loss vs Standard Binary Cross-Entropy

The Python code below produces the Figure 10.40 illustrating Focal Loss vs Standard Binary Cross-Entropy.

```python
import numpy as np
import matplotlib.pyplot as plt

def focal_loss(p, y, alpha=0.25, gamma=2.0):
    """
    pt = p if y == 1 else 1 - p
    return -alpha * (1 - pt)**gamma * np.log(np clip(pt, 1e-8, 1.0))


def bce_loss(p, y):
    """
    return -(y * np.log(np clip(p, 1e-8, 1.0)) +
              (1-y) * np.log(np clip(1-p, 1e-8, 1.0)))

# Probability range
p = np.linspace(0, 1, 1000)
```
```python
15 plt.figure(figsize=(8,6))
16 plt.plot(f, loss_pos, label='Hinge Loss (y=+1)', color='blue')
17 plt.plot(f, loss-negative, label='Hinge Loss (y=-1)', color='red')
18 plt axhline(0, color='black', linewidth=0.8, style='--')
19 plt axvline(0, color='black', linewidth=0.8, style='--')
20 plt xlabel('Prediction Score f(x)')
21 plt ylabel('Hinge Loss')
22 plt title('Hinge Loss vs Prediction Score')
23 plt legend()
24 plt grid(True)
25 plt show()
```

### 10.5.3.39 Python Code to generate Figure 10.42 illustrating Comparison of the Standard Hinge Loss and the Squared Hinge Loss

The Python code below produces the Figure 10.42 illustrating Comparison of the Standard Hinge Loss and the Squared Hinge Loss.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def hinge_loss(y, f):
5     return np maximum(0, 1 - y * f)
6
7 def squared_hinge_loss(y, f):
8     return np maximum(0, 1 - y * f) ** 2
9
10 # Prediction scores from -3 to +3
11 f = np linspace(-3, 3, 500)
12
13 # Compute losses for y = +1 and y = -1
14 loss_pos_hinge = hinge_loss(1, f)
15 loss-negative = hinge_loss(-1, f)
16
17 loss_pos_sq = squared_hinge_loss(1, f)
18 loss-negative_sq = squared_hinge_loss(-1, f)
19
20 # Plot results
21 plt.figure(figsize=(12,5))
22
23 # y = +1 case
24 pltsubplot(1,2,1)
25 plt plot(f, loss_pos_hinge, label='Hinge Loss (y=+1)', color='blue')
26 plt plot(f, loss_pos_sq, label='Squared Hinge Loss (y=+1)', color='blue',
27         linewidth=1)
28 plt axhline(0, color='black', linewidth=0.8, style='--')
29 plt axvline(0, color='black', linewidth=0.8, style='--')
30 plt xlabel('Prediction Score f(x)')
31 plt ylabel('Loss')
32 plt title('Loss Comparison for y=+1')
33 plt legend()
34 plt grid(True)
35
36 # y = -1 case
37 pltsubplot(1,2,2)
```
```python
37 plt.plot(f, loss谓neg_hinge, label='Hinge Loss (y=-1)', color='red')
38 plt plot(f, loss谓neg_sq, label='Squared Hinge Loss (y=-1)', color='red',
39          linestyle='--')
39 plt axhline(0, color='black', linewidth=0.8, style='--')
40 plt axvline(0, color='black', linewidth=0.8, style='--')
41 plt xlabel('Prediction Score f(x)')
42 plt ylabel('Loss')
43 plt title('Loss Comparison for y=-1')
44 plt legend()
45 plt grid(True)
46
47 plt.tight_layout()
48 plt show()
```

### 10.5.3.40 Python Code to generate Figure 10.43 illustrating Value Loss (MSE) for Different Targets

The Python code below produces the Figure 10.43 illustrating Value Loss (MSE) for Different Targets.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def value_loss(V_pred, R):
5     return (V_pred - R) ** 2
6
7 # Predicted value range
8 V_pred = np.linspace(-3, 3, 500)
9
10 # Choose some example targets
11 targets = [-2, 0, 2]
12
13 plt.figure(figsize=(8,6))
14
15 for R in targets:
16     loss = value_loss(V_pred, R)
17     plt plot(V_pred, loss, label=f'Target R = {R}')
18
19 plt xlabel('Predicted Value $V_{\\theta}(s)$')
20 plt ylabel('Value Loss')
21 plt title('Value Loss (MSE) for Different Targets')
22 plt legend()
23 plt grid(True)
24 plt show()
```

### 10.5.3.41 Python Code to generate Figure 10.44 illustrating Value Loss (MSE) vs Huber Loss

The Python code below produces the Figure 10.44 illustrating Value Loss (MSE) vs Huber Loss.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def value_loss(V_pred, R):
5     return (V_pred - R) ** 2
```
```python
def huber_loss(V_pred, R, delta=1.0):
    diff = V_pred - R
    abs_diff = np.abs(diff)
    return np where (abs_diff <= delta,
                     0.5 * diff**2,
                     delta * (abs_diff - 0.5 * delta))

# Predicted value range
V_pred = np linspace (-3, 3, 500)

# Target values
targets = [-2, 0, 2]

# Plot
plt figure (figsize=(12,5))

for R in targets:
    mse = value_loss(V_pred, R)
    huber = huber_loss(V_pred, R, delta=1.0)
    plt plot (V_pred, mse, label='MSE (R={R})',olinestyle='--')
    plt plot (V_pred, huber, label='Huber (R={R})')

plt xlabel('Predicted Value $V_\\theta(s)$')
plt ylabel('Loss')
plt title('Value Loss (MSE) vs Huber Loss')
plt legend()
plt grid (True)
plt show()
```

### 10.5.3.42 Python Code to generate Figure 10.45 illustrating Policy Loss vs Action Probability

The Python code below produces the Figure 10.45 illustrating Policy Loss vs Action Probability.

```python
import numpy as np
import matplotlib pyplot as plt

def policy_loss(p, advantage):
    return -np log(np clip(p, 1e-8, 1.0)) * advantage

# Action probability range
p = np linspace (0.01, 0.99, 500)

# Example advantages
advantages = [1.0, -1.0, 0.5, -0.5]

plt figure (figsize=(8,6))

for A in advantages:
    loss = policy_loss(p, A)
    plt plot (p, loss, label='Advantage = {A}')

plt xlabel('Policy Probability $\pi_\theta(a|s)$')
```
```python
20 plt.ylabel('Policy Loss')
21 plt.title('Policy Loss vs Action Probability')
22 plt.legend()
23 plt.grid(True)
24 plt.show()
```

### 10.5.3.43 Python Code to generate Figure 10.46 illustrating PPO Clipped Surrogate Objective

The Python code below produces the Figure 10.46 illustrating PPO Clipped Surrogate Objective.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def ppo_loss(r, A, eps=0.2):
5     clipped_r = np clipped r, 1 - eps, 1 + eps
6     return np minimum(r * A, clipped_r * A)
7
8 # Range of probability ratios
9 r = np linspace(0, 2, 500)
10
11 # Example advantages
12 advantages = [1.0, -1.0]
13
14 plt figure(figsize=(8,6))
15
16 for A in advantages:
17     loss = ppo_loss(r, A, eps=0.2)
18     plt plot(r, loss, label=f'Advantage = {A}')
19
20 # Reference line at r=1
21 plt axvline(1.0, color='black', linewidth=1, edgecolor='white')
22
23 plt xlabel('Probability Ratio $r_t(\|\theta)$')
24 plt ylabel('PPO Loss')
25 plt title('PPO Clipped Surrogate Objective')
26 plt legend()
27 plt grid(True)
28 plt show()
```

### 10.5.3.44 Python Code to generate Figure 10.47 illustrating PPO Loss Comparison

The Python code below produces the Figure 10.47 illustrating PPO Loss Comparison.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def unclipped_loss(r, A):
5     return r * A
6
7 def ppo_clipped_loss(r, A, eps=0.2):
8     clipped_r = np clipped r, 1 - eps, 1 + eps
9     return np minimum(r * A, clipped_r * A)
10
11 # Probability ratio range
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
```
```python
r = np.linspace(0, 2, 500)

# Advantages
advantages = [1.0, -1.0]

plt.figure(figsize=(12,5))

for i, A in enumerate(advantages, 1):
    pltsubplot(1,2,i)

    loss_unclipped = unclipped_loss(r, A)
    loss_clipped = ppo_clipped_loss(r, A, eps=0.2)

    plt plot(r, loss_unclipped, label='Unclipped (r*A)', lifestyle='--',
             color='black')
    plt plot(r, loss_clipped, label='PPO Clipped', color='blue')

    plt axvline(1.0, color='red', lifestyle='--', linewidth=0.8)
    plt axvline(1.0 - 0.2, color='gray', lifestyle=':', linewidth=0.8)
    plt axvline(1.0 + 0.2, color='gray', lifestyle=':', linewidth=0.8)

    plt xlabel('Probability Ratio $r_t(\|\theta)$')
    plt ylabel('Loss')
    plt title(f'PPO Loss Comparison (Advantage = {A})')
    plt legend()
    plt grid(True)

plt.tight_layout()
plt show()
```

### 10.5.3.45 Python Code to generate Figure 10.48 illustrating PPO Clipped Surrogate Loss with Entropy Regularization

The Python code below produces the Figure 10.48 illustrating PPO Clipped Surrogate Loss with Entropy Regularization.

```python
import numpy as np
import matplotlib.pyplot as plt

def ppo_clipped_loss(r, A, eps=0.2):
    clipped_r = np(r, 1 - eps, 1 + eps)
    return np.minimize(r * A, clipped_r * A)

def entropy(p):
    # Binary entropy for probability p
    p = npclip(p, 1e-8, 1-1e-8)
    return -(p * np.log(p) + (1 - p) * np.log(1 - p))

def ppo_with_entropy(r, A, beta=0.01, eps=0.2, p_action=0.5):
    return ppo_clipped_loss(r, A, eps) - beta * entropy(p_action)

# Probability ratio range
r = np.linspace(0, 2, 500)

# Example advantage
```
```python
20 A = 1.0
21 beta_values = [0.0, 0.01, 0.05] # different entropy weights
22
23 plt.figure(figsize=(8,6))
24
25 for beta in beta_values:
26     loss = [ppo_with_entropy(r, A, beta=beta, eps=0.2, p_action=0.5)
27         for ri in r]
28     plt.plot(r, loss, label=f'Beta = {beta}')
29
30 # Reference lines
31 plt axvline(1.0, color='black', linewidth=0.8)
32 plt axvline(1.0 - 0.2, color='gray', linewidth=0.8)
33 plt axvline(1.0 + 0.2, color='gray', linewidth=0.8)
34
35 plt xlabel('Probability Ratio $r_t(\|\theta)$')
36
37 plt ylabel('PPO Loss + Entropy')
38
39 plt title('PPO Clipped Surrogate Loss with Entropy Regularization')
40
41 plt legend()
42
43 plt grid(True)
44 plt show()
```

### 10.5.3.46 Python Code to generate Figure 10.49 comparing PPO Unclipped, PPO Clipped, PPO + Entropy

The Python code below produces the Figure 10.49 comparing PPO Unclipped, PPO Clipped, PPO + Entropy.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 # PPO loss functions
5 def ppo_unclipped(r, A):
6     return r * A
7
8 def ppo_clipped(r, A, eps=0.2):
9     clipped_r = np(r, 1 - eps, 1 + eps)
10    return np.minimize(r * A, clipped_r * A)
11
12 def entropy(p):
13     p = npclip(p, 1e-8, 1-1e-8)
14     return -(p * np.log(p) + (1 - p) * np.log(1 - p))
15
16 def ppo_with_entropy(r, A, beta=0.01, eps=0.2, p_action=0.5):
17     return ppo_clipped(r, A, eps) - beta * entropy(p_action)
18
19 # Probability ratio range
20 r = np.linspace(0, 2, 500)
21 A = 1.0 # Advantage
22 beta = 0.01
23 p_action = 0.5
24
25 # Compute losses
26 loss_unclipped = ppo_unclipped(r, A)
27 loss_clipped = ppo_clipped(r, A, eps=0.2)
```
```python
28 loss
loss_entropy = [ppo_with_entropy(ri, A, beta=beta, eps=0.2, p_action=
                          p_action) for ri in r]
29
30 # Plot
31 plt.figure(figsize=(10,6))
32 plt.plot(r, loss_unclipped, label='PPO Unclipped', color='blue',
          linewidth=2)
33 plt.plot(r, loss_clipped, label='PPO Clipped', color='green', linewidth
          =2)
34 plt.plot(r, loss_entropy, label='PPO + Entropy', color='purple',
          linewidth=2)
35
36 # Reference lines for clipπng
37 plt axvline(1.0, color='black', linewidth=-1, linewidth=1)
38 plt axvline(1.0 - 0.2, color='gray', linewidth=1, linewidth=1)
39 plt axvline(1.0 + 0.2, color='gray', linewidth=1, linewidth=1)
40
41 plt xlabel('Probability Ratio $r_t(\\theta)$')
42 plt ylabel('Loss')
43 plt title('Comparison: PPO Unclipped, PPO Clipped, PPO + Entropy')
44 plt legend()
45 plt grid(True)
46 plt show()
```

### 10.5.3.47 Python Code to generate Figure 10.50 illustrating Soft Actor-Critic (SAC) Policy Loss with Entropy Regularization

The Python code below produces the Figure 10.50 illustrating Soft Actor-Critic (SAC) Policy Loss with Entropy Regularization.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def sac_loss(p, Q, alpha=0.1):
5 """p: probability of chosen action
6 Q: soft Q-value for that action
7 alpha: entropy temperature
8 """p = np.clip(p, 1e-8, 1-1e-8) # avoid log(0)
9 entropy = -(p * np.log(p) + (1-p) * np.log(1-p))
10 return alpha * (-entropy) - Q # loss = alpha*log(π) - Q
11
12
13 # Action probability range
14 p = np.linspace(0.01, 0.99, 500)
15
16 # Example Q-values
17 Q_values = [0.5, 1.0, 2.0]
18 alpha_values = [0.05, 0.1, 0.2]
19
20 plt.figure(figsize=(10,6))
21
22 for Q in Q_values:
23     for alpha in alpha_values:
24         loss = sac_loss(p, Q, alpha)
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
```
```python
26     plt.plot(p, loss, label=f'Q={Q}', alpha={alpha:.2f})
27
28     plt.xlabel('Policy Probability $\\pi_\\theta(a|s)$')
29     plt.ylabel('SAC Policy Loss')
30     plt.title('Soft Actor-Critic (SAC) Policy Loss with Entropy Regularization')
31     plt.legend()
32     plt.grid(True)
33     plt.show()
```

### 10.5.3.48 Python Code to generate Figure 10.51 illustrating KL Divergence Loss as Sparsity Constraint

The Python code below produces the Figure 10.51 illustrating KL Divergence Loss as Sparsity Constraint.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def kl_divergence_loss(rho, rhoHat):
5     # KL divergence for sparsity constraint
6     rhoHat = np clipped rhoHat, 1e-8, 1-1e-8) # avoid log(0)
7     return rho * np log rho / rhoHat + (1 - rho) * np log((1 - rho) / (1 - rhoHat))
8
9 # Range of average activations
10 rhoHat = np.linspace(0.001, 0.999, 500)
11
12 # Desired sparsity values
13 rho_values = [0.05, 0.1, 0.2]
14
15 plt figure (figsize=(8,6))
16
17 for rho in rho_values:
18     loss = kl_divergence_loss(rho, rhoHat)
19     plt plot(rhoHat, loss, label=f'rho={rho}')
20
21 plt xlabel('Average Activation $\hat{\rho}$')
22 plt ylabel('KL Divergence Sparsity Loss')
23 plt title('KL Divergence Loss as Sparsity Constraint')
24 plt legend()
25 plt grid(True)
26 plt show()
```

### 10.5.3.49 Python Code to generate Figure 10.52 illustrating Total KL Sparsity Loss for 10 Hidden Units ($\rho = 0.05$)

The Python code below produces the Figure 10.52 illustrating Total KL Sparsity Loss for 10 Hidden Units ($\rho = 0.05$).

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def kl_divergence_loss(rho, rhoHat):
5     rhoHat = np clipped rhoHat, 1e-8, 1-1e-8)
```
<figure><img src="image_161.png" /></figure>

10.5.3.50 Python Code to generate Figure 10.53 illustrating Total KL Sparsity Loss for 10 Hidden Units with Random Activations ($\rho$ = 0.05)

The Python code below produces the Figure 10.53 illustrating Total KL Sparsity Loss for 10 Hidden Units with Random Activations ($\rho$ = 0.05).

<figure><img src="image_162.png" /></figure>
```python
np.random.seed(42) # for reproducibility

for rho_hat_center in rho_hat_values:
    # Randomize activations around rho_hat_center for each hidden unit
    rho_hats = rho_hat_center + 0.02 * np.randomiji(n_hidden) #
    small noise
    rho_hats = np.rho_hats, 1e-8, 1-1e-8)

# Sum KL loss over all hidden units
total_loss.append(npsum([kl_divergence_loss(rho, rh) for rh in
                          rho_hats]))

# Plot
plt.figure(figsize=(8,6))
plt plot(rho_hat_values, total_loss, color='blue')

# Central Average Activation of Hidden Units $\hat{\rho}$

# Total KL Divergence Sparsity Loss

# Total KL Sparsity Loss for {n_hidden} Hidden Units with
    Random Activations (ρ={rho})')

# Fixed correct class score
s_y = 2.0

# Range of incorrect class scores
s_j = np.linspace(0, 4, 500)

# Different margin values
deltas = [0.5, 1.0, 1.5]

# figure (figsize=(8,6))

for delta in deltas:
    loss = ssvm_loss(s_y, s_j, delta)
    plt plot(s_j, loss, label=f'Δ = {delta}')

# Correct Class Score

# Incorrect Class Score $s_j$

# Structured SVM Loss

# Structured SVM (SSVM) Hinge Loss
```

### 10.5.3.51 Python Code to generate Figure 10.54 illustrating Structured SVM (SSVM) Hinge Loss

The Python code below produces the Figure 10.54 illustrating Structured SVM (SSVM) Hinge Loss.

```python
import numpy as np
import matplotlib.pyplot as plt

def ssvm_loss(s correct, s incorrect, delta=1.0):
    return np maximum(0, s incorrect - s correct + delta)

# Fixed correct class score
s y = 2.0

# Range of incorrect class scores
s_j = np linspace(0, 4, 500)

# Different margin values
deltas = [0.5, 1.0, 1.5]

# figure (figsize=(8,6))

for delta in deltas:
    loss = ssvm_loss(s y, s_j, delta)
    plt plot(s_j, loss, label=f'Δ = {delta}')

# Correct Class Score

# Incorrect Class Score $s_j$

# Structured SVM Loss

# Structured SVM (SSVM) Hinge Loss
```
26 plt.legend()
27 plt.grid(True)
28 plt.show()

### 10.5.3.52 Python Code to generate Figure 10.55 illustrating Structured SVM Loss Surface ($\Delta = 1.0$)

The Python code below produces the Figure 10.55 illustrating Structured SVM Loss Surface ($\Delta = 1.0$).

1 import numpy as np
2 import matplotlib.pyplot as plt
3 from mpl_toolkits.mplot3d import Axes3D
4
5 def ssvm_loss(s correct, s incorrect, delta=1.0):
6     return np maximum(0, s incorrect - s correct + delta)
7
8 # Score ranges
9 s correct = np linspace(0, 4, 100)
10 s incorrect = np linspace(0, 4, 100)
11 S correct, S incorrect = np meshgrid(s correct, s incorrect)
12
13 # Compute loss surface
14 Delta = 1.0
15 Loss = ssvm_loss(S correct, S incorrect, delta=Delta)
16
17 # Plot 3D surface
18 fig = plt figure(figsize=(10,7))
19 ax = fig.add_subplot(111, projection='3d')
20 ax plot Cole's surface(S correct, S incorrect, Loss, cmap='viridis', alpha=0.9)
21
22 # Labels
23 ax.set xlabel('Correct Class Score $s_y$')
24 ax.set ylabel('Incorrect Class Score $s_j$')
25 ax.set zlabel('SSVM Loss')
26 ax.set title(f'Structured SVM Loss Surface ($\Delta$={Delta})')
27
28 plt.show()

### 10.5.3.53 Python Code to generate Figure 10.56 illustrating Structured SVM Loss Contour ($\Delta = 1.0$)

The Python code below produces the Figure 10.56 illustrating Structured SVM Loss Contour ($\Delta = 1.0$).

1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def ssvm_loss(s correct, s incorrect, delta=1.0):
5     return np maximum(0, s incorrect - s correct + delta)
6
7 # Score ranges
8 s correct = np linspace(0, 4, 200)
9 s incorrect = np linspace(0, 4, 200)
10 S correct, S incorrect = np meshgrid(s correct, s incorrect)
```python
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
```

### 10.5.3.54 Python Code to generate Figure 10.57 illustrating Simplified CRF Loss vs Correct Label Score

The Python code below produces the Figure 10.57 illustrating Simplified CRF Loss vs Correct Label Score.

```python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
```
29

plt.show()

### 10.5.3.55 Python Code to generate Figure 10.58 illustrating CRF Loss Surface (Single Timestep, 2 Labels)

The Python code below produces the Figure 10.58 illustrating CRF Loss Surface (Single Timestep, 2 Labels).

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

def crf_loss_2d(s correct, s other):
    """
    Simplified CRF loss for one timestep with one competing label
    s correct, s other: can be arrays of scores
    """
    partition = np.exp(s correct) + np.exp(s other)
    return - (s correct - np.log(partition))

# Score ranges
s correct = np.linspace(-2, 5, 100)
s other = np.linspace(-2, 5, 100)
S correct, S other = np.meshgrid(s correct, s other)

# Compute CRF loss surface
Loss = crf_loss_2d(S correct, S other)

# Plot 3D surface
fig = plt.figure(figsize=(10,7))
ax = fig.add_subplot(111, projection='3d')
surf = ax plot Cole (S correct, S other, Loss, cmap='viridis', alpha=0.9)

# Labels
ax.set xlabel('Correct Label Score $s_{correct}$')
ax.set ylabel('Competing Label Score $s_{other}$')
ax.set zlabel('CRF Loss')
ax.set title('CRF Loss Surface (Single Timestep, 2 Labels)')
fig colorbar(surf, shrink=0.6, label='CRF Loss')

plt show()
```

### 10.5.3.56 Python Code to generate Figure 10.59 illustrating CRF Loss Contour (Single Timestep, 2 Labels)

The Python code below produces the Figure 10.59 illustrating CRF Loss Contour (Single Timestep, 2 Labels).

```python
import numpy as np
import matplotlib.pyplot as plt

def crf_loss_2d(s correct, s other):
    """
    Simplified CRF loss for one timestep with one competing label
    """
```
```python
s correct, s other: can be arrays of scores
"""
partition = np.exp(s correct) + np.exp(s other)
return - (s correct - np.log(partition))

# Score ranges
s correct = np linspace(-2, 5, 200)
s other = np linspace(-2, 5, 200)
S correct, S other = np meshgrid(s correct, s other)

# Compute CRF loss surface
Loss = crf loss 2d(S correct, S other)

# Contour plot
plt .figure(figure=(8,6))
contour = plt .contourf(S correct, S other, Loss, levels =50, cmap ='viridis')
plt .colorbar(contour, label ='CRF Loss')

# Overlay zero-loss boundary (approximate)
plt .contour(S correct, S other, Loss, levels=[0], colors='red',
linewidths=2, linestyles='--')

plt .xlabel('Correct Label Score $s_{correct}$')
plt .ylabel('Competing Label Score $s_{other}$')
plt .title('CRF Loss Contour (Single Timestep, 2 Labels)')
plt .grid(True)
plt .show()
```

### 10.5.3.57 Python Code to generate Figure 10.60 illustrating CTC Loss vs Probability of Correct Sequence

The Python code below produces the Figure 10.60 illustrating CTC Loss vs Probability of Correct Sequence.

```python
import numpy as np
import matplotlib .pyplot as plt

def ctc_loss(sim prob):
    """
    Simplified CTC loss: negative log-probability of correct sequence
    sim prob: probability of correct alignment (0 to 1)
    """
    sim prob = np .clip(sim prob, 1e-8, 1.0) # avoid log(0)
    return -np .log(sim prob)

# Simulate model confidence in correct sequence
probs = np .linspace(0.001, 1.0, 500)

# Compute CTC loss
loss = ctc loss(probs)

# Plot
plt .figure(figure=(8,6))
plt .plot(probs, loss, color='teal', linewidth=2)
```
```python
21 plt.xlabel('Probability of Correct Alignment $P(y|X)$')
22 plt.ylabel('CTC Loss')
23 plt.title('CTC Loss vs Probability of Correct Sequence')
24 plt.grid(True)
25 plt.show()
```

### 10.5.3.58 Python Code to generate Figure 10.61 illustrating Approximate CTC Loss vs Per-Timestep Probability for Different Sequence Lengths

The Python code below produces the Figure 10.61 illustrating Approximate CTC Loss vs Per-Timestep Probability for Different Sequence Lengths.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 def ctc_loss_avg(prob, seq_len):
5     """
6         Approximate CTC loss for sequence of length seq_len
7         prob: average probability per timestep of correct label
8     """
9     prob = npanstClip(prob, 1e-8, 1.0)  # avoid log(0)
10    # Approximation: total probability of sequence ~ product of per-
11    timestepping probabilities
12    total prob = prob ** seq_len
13    return -np.log(total prob)
14
15 # Average per-timestep probabilities
16 prob = npsthispace(0.5, 1.0, 500)
17
18 # Different target sequence lengths
19 seq lengths = [5, 10, 20]
20
21 plt figure(figsize=(10,6))
22
23 for L in seq lengths:
24     loss = ctc loss avg(prob s, L)
25     plt plot(prob s, loss, label=f'Sequence Length = {L}', linewidth=2)
26
27 plt xlabel('Average Per-Timestep Probability of Correct Label')
28 plt ylabel('CTC Loss')
29 plt title('Approximate CTC Loss vs Per-Timestep Probability for
3 Different Sequence Lengths')
30
31 plt legend()
32 plt grid(True)
33 plt show()
```

### 10.5.3.59 Python Code to generate Figure 10.62 illustrating CTC Loss Surface vs Sequence Length and Per-Timestep Probability

The Python code below produces the Figure 10.62 illustrating CTC Loss Surface vs Sequence Length and Per-Timestep Probability.

```python
1 import numpy as np
2 import matplotlib.pytplot as plt
3
4 from mpl_toolkits.mplot3d import Axes3D
```
```python
def ctc_loss_avg(prob, seq_len):
    """
    Approximate CTC loss for sequence of length seq_len
    prob: average probability per timestep of correct label
    """
    prob = np clip(prob, 1e-8, 1.0) # avoid log(0)
    total prob = prob ** seq_len
    return -np log(total prob)

# Sequence length and per-timestep probability ranges
seq lengths = np linspace(1, 30, 100)
probs = np linspace(0.5, 1.0, 100)
Seq, Prob = np meshgrid(seq lengths,/probs)

# Compute CTC loss surface
Loss = ctc_loss_avg(Prob, Seq)

# Plot 3D surface
fig = plt figure(figsize=(12,7))
ax = fig.addsubplot(111, projection='3d')
surf = ax plot surface Seq, Prob, Loss, colormap='viridis', alpha=0.9)

# Labels
ax.set xlabel('Sequence Length')
ax.set ylabel('Per-Timestep Probability')
ax.set zlabel('CTC Loss')
ax.set title('CTC Loss Surface vs Sequence Length and Per-Timestep
Probability')
fig colorbar(surf, shrink=0.6, label='CTC Loss')

plt show()
```

### 10.5.3.60 Python Code to generate Figure 10.63 illustrating CTC Loss Contour vs Sequence Length and Per-Timestep Probability

The Python code below produces the Figure 10.63 illustrating CTC Loss Contour vs Sequence Length and Per-Timestep Probability.

```python
import numpy as np
import matplotlib pyplot as plt

def ctc_loss_avg(prob, seq_len):
    """
    Approximate CTC loss for sequence of length seq_len
    prob: average probability per timestep of correct label
    """
    prob = np clip(prob, 1e-8, 1.0) # avoid log(0)
    total prob = prob ** seq_len
    return -np log(total prob)

# Sequence length and per-timestep probability ranges
seq lengths = np linspace(1, 30, 200)
probs = np linspace(0.5, 1.0, 200)
Seq, Prob = np meshgrid(seq lengths,/probs)
```
```python
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
```

### 10.5.3.61 Python Code to generate Figure 10.64 illustrating Maximum Margin Markov Networks ($M^3N$) Loss

The Python code below produces the Figure 10.64 illustrating Maximum Margin Markov Networks ($M^3N$) Loss.

```python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
```
```python
29 plt.xlabel('Competing Output Score $s(\tilde{y})$')
30 plt.ylabel('M3N Loss')
31 plt.title('Maximum Margin Markov Networks (M3N) Loss')
32 plt.legend()
33 plt.grid(True)
34 plt.show()
```

### 10.5.3.62 Python Code to generate Figure 10.65 illustrating M³N Loss Surface ($\Delta = 1.0$)

The Python code below produces the Figure 10.65 illustrating M³N Loss Surface ($\Delta = 1.0$).

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3 from mpl_toolkits.mplot3d import Axes3D
4
5 def m3n_loss_3d(s correct, s incorrect, delta):
6     """
7     Structured hinge loss for M3N
8     """
9     return np.max(0, delta + sIncorrect - sCorrect)
10
11 # Score ranges
12 s correct = np.linspace(0, 4, 100)
13 s incorrect = np.linspace(0, 4, 100)
14 delta = 1.0
15
16 S correct, S incorrect = np.meshgrid(s correct, s incorrect)
17
18 # Compute loss surface
19 Loss = m3n_loss_3d(S correct, S incorrect, delta)
20
21 # Plot 3D surface
22 fig = plt.figure(figsize=(10,7))
23 ax = fig.add_subplot(111, projection='3d')
24 surf = ax.plot(S correct, S incorrect, Loss, cmap='viridis',
25                     alpha=0.9)
26
27 # Labels
28 ax.set_xlabel('Correct Output Score $s_y$')
29 ax.set_ylabel('Competing Output Score $s_{\tilde{y}}$')
30 ax.set_zlabel('M3N Loss')
31 ax.set_title(f'M3N Loss Surface ($\Delta={\delta}$)')
32 fig.colorbar(surf, shrink=0.6, label='M3N Loss')
33 plt.show()
```

### 10.5.3.63 Python Code to generate Figure 10.66 illustrating Contour Plot of M³N Loss ($\Delta = 1.0$)

The Python code below produces the Figure 10.66 illustrating Contour Plot of M³N Loss ($\Delta = 1.0$).

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
```
```python
def m3n_loss(s correct, s incorrect, delta):
    """
    Structured hinge loss for M3N
    """
    return np maximum(0, delta + s incorrect - s correct)

# Score ranges
s correct = np linspace(0, 4, 200)
s incorrect = np linspace(0, 4, 200)
delta = 1.0

S correct, S incorrect = np meshgrid(s correct, s incorrect)

# Compute loss values
Loss = m3n_loss(S correct, S incorrect, delta)

# Contour plot
plt figure(figsize=(8,6))
contour = plt contourf(S correct, S incorrect, Loss, levels=50, cmap='viridis')
cbar = plt colorbar(contour)
cbar set_label('M3N Loss')

# Add margin boundary line: where Loss = 0 to s incorrect = s correct - delta
plt plot(s correct, s correct - delta, 'r--', linewidth=2, label=f'Margin Boundary ($s_{\{\tilde{y}\}} = s_y - \Delta\$')')

# Labels
plt xlabel('Correct Output Score $s_y$')
plt ylabel('Competing Output Score $s_{\{\tilde{y}\}}$')
plt title(f'Contour Plot of M3N Loss ($\Delta={\{delta}}$')')
plt legend()

plt show()
```

### 10.5.3.64 Python Code to generate Figure 10.67 illustrating Listwise Ranking Losses (ListNet vs ListMLE)

The Python code below produces the Figure 10.67 illustrating Listwise Ranking Losses (ListNet vs ListMLE).

```python
import numpy as np
import matplotlib pyplot as plt
from scipy.special import softmax

# Listwise Loss Functions

def listnet_loss(scores, relevance):
    """
    ListNet loss: Cross-entropy between softmax(relevance) and softmax(scores)
    """
```
66

```python
plt.grid(True)
```

67

```python
plt.show()
```

### 10.5.3.65 Python Code to generate Figure 10.68 illustrating Listwise Ranking Losses (3D Surfaces)

The Python code below produces the Figure 10.68 illustrating Listwise Ranking Losses (3D Surfaces).

1

```python
import numpy as np
```

2

```python
import matplotlib.pyplot as plt
```

3

```python
from scipy.special import softmax
```

4

```python
from mpl_toolkits.mplot3d import Axes3D
```

5

```python
# Listwise Loss Functions
```

6

```python
# Listwise Loss Functions
```

7

```python
# Listwise Loss Functions
```

8

```python
def listnet_loss(scores, relevance):
```

10

```python
        """ListNet: Cross-entropy between softmax(relevance) and softmax(
        scores)"""
```

11

```python
            P_true = softmax(relevance)
```

12

```python
            P_pred = softmax(scores)
```

13

```python
        return -np.sum(P_true * np.log(P_pred + 1e-12))
```

14

15

```python
def listmle_loss(scores, relevance):
```

16

```python
        """ListMLE: Negative log-likelihood of ground truth permutation"""
```

17

```python
            idx = np.argsort(-relevance) # sort by relevance descending
```

18

```python
            sorted_scores = scores[idx]
```

19

20

```python
            loss = 0.0
```

21

```python
        for i in range(len(scores)):
```

22

```python
            denom = np.sum(np.exp(sorted_scores[i:]))
```

23

```python
            loss += -sorted_scores[i] + np.log(denom)
```

24

```python
        return loss
```

25

26

```python
# Experiment Setup
```

27

```python
# Experiment Setup
```

28

```python
# Experiment Setup
```

29

```python
# Ground truth relevance labels (e.g., 3 items: item0 most relevant,
# then item1, then item2)
```

30

```python
        relevance = np.array([3, 2, 0])
```

31

```python
# Range of predicted scores for two items, keep the third fixed
```

32

```python
            score_range = np.linspace(-2, 4, 100)
```

33

```python
        S0, S1 = np.meshgrid(score_range, score_range)
```

34

```python
# Fixed score for the third item
```

35

```python
        fixed_score2 = 1.0
```

36

```python
# Compute loss surfaces
```

37

```python
        listnet surface = np(zeros_like(S0))
```

38

```python
        listmle surface = np(zeros_like(S0))
```

39

40

41
```python
45 for i in range(S0.shape[0]):
46     for j in range(S0.shape[1]):
47         scores = np.array([S0[i,j], S1[i,j], fixed_score2])
48         listnet[i,j] = listnet_loss(scores, relevance)
49         listmle surface[i,j] = listmle_loss(scores, relevance)
50
51 # --------------------------
52 # Plot ListNet Loss Surface
53 # --------------------------
54 fig = plt.figure(figsize=(12,5))
55
56 ax1 = fig.add_subplot(1,2,1, projection="3d")
57 surf1 = ax1 Plot(S0, S1, listnet_surface, colormap="viridis", alpha
58                      =0.9)
59 ax1.set_title("ListNet Loss Surface")
60 ax1.set_xlabel("Predicted Score for Item 0")
61 ax1.set_ylabel("Predicted Score for Item 1")
62 ax1.set_zlabel("Loss")
63 fig.colorbar(surf1, ax=ax1, shrink=0.6)
64
65 # --------------------------
66 # Plot ListMLE Loss Surface
67 # --------------------------
68 ax2 = fig.add_subplot(1,2,2, projection="3d")
69 surf2 = ax2.Plotile(S0, S1, listmle surface, colormap="plasma", alpha
70                     =0.9)
71 ax2.set_title("ListMLE Loss Surface")
72 ax2.set_xlabel("Predicted Score for Item 0")
73 ax2.set_ylabel("Predicted Score for Item 1")
74 ax2.set_zlabel("Loss")
75 fig.colorbar(surf2, ax=ax2, shrink=0.6)
76
77 plt.suptitle("Listwise Ranking Losses (3D Surfaces)", fontsize=14)
78 plt.show()
```

### 10.5.3.66 Python Code to generate Figure 10.69 illustrating Listwise Ranking Loss Contours

The Python code below produces the Figure 10.69 illustrating Listwise Ranking Loss Contours.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3 from scipy.special import softmax
4
5 # --------------------------
6 # Listwise Loss Functions
7 # --------------------------
8
9 def listnet_loss(scores, relevance):
10     """ListNet: Cross-entropy between softmax(relevance) and softmax(
11         scores)"""
12     P_true = softmax(relevance)
13     P_pred = softmax(scores)
14     return -np.sum(P_true * np.log(P_pred + 1e-12))
```
```python
def listmle_loss(scores, relevance):
    """ListMLE: Negative log-likelihood of ground truth permutation"""
    idx = np argsort(-relevance) # sort by relevance descending
    sorted_scores = scores[idx]

    loss = 0.0
    for i in range(len(scores)):
        denom = np sum(np exp sorted_scores[i:]))
        loss += -sorted_scores[i] + np log denom
    return loss

# Experiment Setup
# Ground truth relevance labels
relevance = np array([3, 2, 0])

# Score ranges for item0 and item1
score_range = np linspace(-2, 4, 200)
S0, S1 = np meshgrid score_range, score_range

# Fixed score for item2
fixed_score2 = 1.0

# Compute loss surfaces
listnet = np zeros_like(S0)
listmle surface = np zeros_like(S0)

for i in range(S0.shape[0]):
    for j in range(S0.shape[1]):
        scores = np array([S0[i,j], S1[i,j], fixed_score2])
        listnet surface[i,j] = listnet_loss(scores, relevance)
        listmle surface[i,j] = listmle_loss(scores, relevance)

# Contour Plots
# ListNet contour
cont1 = plt contourf S0, S1, listnet surface, levels=50, cmap='viridis')
cont1 colorbar cont1, label='ListNet Loss')

# Predicted Score for Item 0

# Predicted Score for Item 1

# ListMLE contour
cont2 = plt contourf S0, S1, listmle surface, levels=50, cmap='plasma')
cont2 colorbar cont2, label='ListMLE Loss')

# Predicted Score for Item 0

# Predicted Score for Item 1

# ListMLE Loss Contour')
```
70
71 plt.suptitle('Listwise Ranking Loss Contours', fontsize=14)
72 plt.show()

### 10.5.3.67 Python Code to generate Figure 10.70 illustrating Pairwise Ranking Losses

The Python code below produces the Figure 10.70 illustrating Pairwise Ranking Losses.

1
import numpy as np
2
import matplotlib.pyplot as plt
3
4
# ----------------------------
5
# Pairwise Loss Functions
6
# ----------------------------
7
def pairwise_hinge_loss(s_diff, margin=1.0):
8
    """Pairwise hinge ranking loss"""
9
    return np maximum(0, margin - s_diff)
10
11
12
def pairwise_logistic_loss(s_diff):
13
    """Pairwise logistic loss (RankNet style)"""
14
    return np.log(1 + np. exp(-s_diff))
15
16
# ----------------------------
17
# Experiment Setup
18
# ----------------------------
19
# Score differences (s_i - s_j)
20
21 s_diff = np. linspace(-3, 3, 400)
22
23 hinge_lost s = pairwise_hinge_loss(s_diff)
24 log_lost s = pairwise_logistic_loss(s_diff)
25
26
# ----------------------------
27
# Plot Results
28
# ----------------------------
29
30 plt. figure (figsize=(8,6))
31 plt. plot(s_diff, hinge_lost s, label="Pairwise Hinge Loss", linewidth=2)
32 plt. plot(s_diff, log_lost s, label="Pairwise Logistic Loss (RankNet)",
           linewidth=2)
33
34 plt. xlabel("Score Difference (s_i - s_j)")
35 plt. ylabel("Loss Value")
36 plt. title("Pairwise Ranking Losses")
37 plt. legend()
38
39 plt. grid (True)
40 plt. show()

### 10.5.3.68 Python Code to generate Figure 10.71 illustrating Pairwise Ranking Loss Landscapes

The Python code below produces the Figure 10.71 illustrating Pairwise Ranking Loss Landscapes.

1
import numpy as np
import matplotlib.pyplot as plt

# ----------------------------

# Pairwise Loss Functions

# ----------------------------

def pairwise_hinge_loss(s_i, s_j, margin=1.0):

"""Pairwise hinge ranking loss"""

return np maximum(0, margin - (s_i - s_j))

def pairwise_logistic_loss(s_i, s_j):

"""Pairwise logistic loss (RankNet style)"""

return np. log(1 + np. exp(-(s_i - s_j)))

# ----------------------------

# Experiment Setup

# ----------------------------

# Score ranges

s_range = np. linspace(-3, 3, 200)

S_i, S_j = np. meshgrid(s_range, s_range)

# Compute losses

hinge surface = pairwise_hinge_loss(S_i, S_j)

log surface = pairwise_logistic_loss(S_i, S_j)

# ----------------------------

# Contour Plots

# ----------------------------

plt. figure( figsize=(12, 5))

# Pairwise hinge loss contour

plt.subplot(1, 2, 1)

cont1 = plt. contourf(S_i, S_j, hinge surface, levels=50, cmap='viridis')

plt. colorbar(cont1, label='Loss Value')

plt.xlabel("Score $s_i$")

plt.ylabel("Score $s_j$")

plt. title("Pairwise Hinge Ranking Loss Contour")

# Pairwise logistic loss contour

plt.subplot(1, 2, 2)

cont2 = plt. contourf(S_i, S_j, log surface, levels=50, cmap='plasma')

plt. colorbar(cont2, label='Loss Value')

plt.xlabel("Score $s_i$")

plt.ylabel("Score $s_j$")

plt. title("Pairwise Logistic Ranking Loss Contour (RankNet)")

plt. suptitle("Pairwise Ranking Loss Landscapes", fontsize=14)

plt. show()
### 10.5.3.69 Python Code to generate Figure 10.72 illustrating 3D Surfaces of Pairwise Ranking Losses

The Python code below produces the Figure 10.72 illustrating 3D Surfaces of Pairwise Ranking Losses.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

# ----------------------------
# Pairwise Loss Functions
# ----------------------------

def pairwise_hinge_loss(s_i, s_j, margin=1.0):
    """Pairwise hinge ranking loss"""
    return np maximum(0, margin - (s_i - s_j))

def pairwise_logistic_loss(s_i, s_j):
    """Pairwise logistic loss (RankNet style)"""
    return np.log(1 + np. exp(-(s_i - s_j)))

# ----------------------------
# Experiment Setup
# ----------------------------

# Score ranges
s_range = np. linspace(-3, 3, 100)
S_i, S_j = np. meshgrid(s_range, s_range)

# Compute losses
hingeSurface = pairwise_hinge_loss(S_i, S_j)
logSurface = pairwise_logistic_loss(S_i, S_j)

# ----------------------------
# 3D Surface Plots
# ----------------------------

fig = plt. figure(figsize=(12, 5))

# Pairwise hinge loss
ax1 = fig. add_subplot(1, 2, 1, projection="3d")
surf1 = ax1. plot(S_i, S_j, hingeSurface, cmap="viridis", alpha
                       =0.9)
ax1. set_title("Pairwise Hinge Ranking Loss Surface")
ax1. set_xlabel("Score $s_i$")
ax1. set_ylabel("Score $s_j$")
ax1. set_zlabel("Loss")
fig. colorbar(surf1, ax=ax1, shrink=0.6)

# Pairwise logistic loss
ax2 = fig. add_subplot(1, 2, 2, projection="3d")
surf2 = ax2. plot surface(S_i, S_j, logSurface, cmap="plasma", alpha
                          =0.9)
ax2. set_title("Pairwise Logistic Ranking Loss Surface (RankNet)")
ax2. set_xlabel("Score $s_i$")
```
```python
ax2.set_ylabel("Score $s_j$")
ax2.set_zlabel("Loss")
fig.colorbar(surf2, ax=ax2, shrink=0.6)

plt.suptitle("3D Surfaces of Pairwise Ranking Losses", fontsize=14)
plt.show()
```

### 10.5.3.70 Python Code to generate Figure 10.73 illustrating Contrastive Divergence (CD) Loss vs Weight

The Python code below produces the Figure 10.73 illustrating Contrastive Divergence (CD) Loss vs Weight.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.special import expi as sigmoid

# Synthetic CD Loss Example

def cd_loss(weight, data_v, data_h):
    """
    Contrastive Divergence loss for 1 visible and 1 hidden unit
    weight: scalar parameter
    data_v, data_h: observed binary data (averages)
    """
    # Positive phase (data expectation)
    pos = data_v * data_h

    # Negative phase (model expectation using reconstruction)
    # Model probability of hidden given visible ~ sigmoid(v * W)
    prob_h_given_v = sigmoid(data_v * weight)
    recon_v = prob_h_given_v  # simplified single-step reconstruction
    prob_h_given_recon_v = sigmoid(recon_v * weight)
    neg = recon_v * prob_h_given_recon_v

    return -(pos - neg)

# Experiment Setup

# Assume some "data statistics"
data_v = 1.0  # visible always on
data_h = 0.8  # hidden unit activated with prob 0.8

weights = np.linspace(-6, 6, 400)
losses = [cd_loss(w, data_v, data_h) for w in weights]

# Plot

plt.figure(figsize=(8,6))
```
```python
43 plt.plot weights, losses, label="CD Loss", linewidth=2, color="purple")
44 plt axhline 0, color="black", linewidth="--", alpha=0.7)
45
46 plt xlabel("Weight $W$")
47 plt ylabel("Contrastive Divergence Loss")
48 plt title("Contrastive Divergence (CD) Loss vs Weight")
49 plt legend()
50 plt grid(True)
51 plt show()
```

### 10.5.3.71 Python Code to generate Figure 10.74 illustrating Contrastive Divergence (CD) Loss Landscape

The Python code below produces the Figure 10.74 illustrating Contrastive Divergence (CD) Loss Landscape.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3 from scipy.special import expi as sigmoid
4
5 # --------------------------
6 # CD Loss with 1 visible + 2 hidden units
7 # --------------------------
8 def cd_loss_2d(w1, w2, data_v, data_h1, data_h2):
9 """
10 Contrastive Divergence loss with 1 visible, 2 hidden units
11 """
12 # Positive phase (data expectation)
13 pos = data_v * (data_h1 + data_h2)
14
15 # Negative phase (model expectation using reconstruction)
16 # Hidden activations given visible
17 prob_h1_given_v = sigmoid(data_v * w1)
18 prob_h2_given_v = sigmoid(data_v * w2)
19
20 # Reconstruct visible (mean-field style approximation)
21 recon_v = (prob_h1_given_v + prob_h2_given_v) / 2.0
22
23 # Hidden activations given reconstructed visible
24 prob_h1_given_recon_v = sigmoid(recon_v * w1)
25 prob_h2_given_recon_v = sigmoid(recon_v * w2)
26
27 neg = recon_v * (prob_h1_given_recon_v + prob_h2_given_recon_v)
28
29 return -(pos - neg)
30
# --------------------------
31 # Experiment Setup
32 # --------------------------
33 data_v = 1.0  # visible always on
34 data_h1 = 0.8  # hidden 1 activation prob
35 data_h2 = 0.6  # hidden 2 activation prob
36
37 w_range = np.linspace(-6, 6, 200)
38 W1, W2 = np.meshgrid(w_range, w_range)
```
```python
data_h1 = 0.8  # hidden 1 activation prob
data_h2 = 0.6  # hidden 2 activation prob

w_range = np.linspace(-6, 6, 100)
W1, W2 = np.meshgrid(w_range, w_range)
loss = cd_loss_2d(W1, W2, data_v, data_h1, data_h2)

# 3D Surface Plot
ax = fig.add_subplot(111, projection="3d")

surf = ax.plot(W1, W2, loss, colormap="viridis", edgecolor="none", alpha=0.9)
fig.colorbar(surf, ax=ax, shrink=0.5, aspect=10, label="CD Loss")

ax.set_xlabel("Weight $W_1$")
ax.set_ylabel("Weight $W_2$")
ax.set_zlabel("CD Loss")
ax.set_title("Contrastive Divergence (CD) Loss Landscape (3D)")

plt.show()
```

### 10.5.3.73 Python Code to generate Figure 10.76 illustrating Persistent Contrastive Divergence (PCD) Loss Landscape

The Python code below produces the Figure 10.76 illustrating Persistent Contrastive Divergence (PCD) Loss Landscape.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.special import expi as sigmoid

# PCD Loss with 1 visible + 2 hidden units
# def pcd_loss_2d(w1, w2, data_v, data_h1, data_h2, steps=50):
#     """
#     Persistent Contrastive Divergence loss
#     w1, w2 : weights (can be arrays)
#     data_v, data_h1, data_h2 : observed data stats
#     steps : number of Gibbs updates for persistent chain
#     """
#     Positive phase (data expectation)
#     pos = data_v * (data_h1 + data_h2)

# Initialize persistent visible state randomly
v_model = np.random rand(*np.shape(w1))

# Persistent chain Gibbs sampling
for _ in range(steps):
    # Hidden probabilities given visible
    prob_h1_given_v = sigmoid(v_model * w1)
    prob_h2_given_v = sigmoid(v_model * w2)
```
```python
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

# Sample hidden states
h1 = (np.randomrand(*np.shape(w1)) < prob_h1_given_v).astype(
       float)
h2 = (np.randomrand(*np.shape(w2)) < prob_h2_given_v).astype(
       float)

# Re-sample visible from hidden
v_model = sigmoid(h1 * w1 + h2 * w2)

# Negative phase expectation
prob_h1_given_vm = sigmoid(v_model * w1)
prob_h2_given_vm = sigmoid(v_model * w2)
neg = v_model * (prob_h1_given_vm + prob_h2_given_vm)

return -(pos - neg)

# -----------------------------
# Experiment Setup
# -----------------------------

data_v = 1.0   # visible always on
data_h1 = 0.8  # hidden unit 1 activation
data_h2 = 0.6  # hidden unit 2 activation

w_range = np.linspace(-6, 6, 80)
W1, W2 = np.meshgrid(w_range, w_range)
losssurface = pcd_loss_2d(W1, W2, data_v, data_h1, data_h2, steps=100)

# -----------------------------
# Plot
# -----------------------------

plt.figure(figsize=(8,6))
contour = plt.contourf(W1, W2, lossservice, levels=50, cmap="viridis")
plt.colorbar(contour, label="PCD Loss")

plt.xlabel("Weight $W_1$")
plt.ylabel("Weight $W_2$")

plt.title("Persistent Contrastive Divergence (PCD) Loss Landscape")

plt.grid(False)

plt.show()
```

### 10.5.3.74 Python Code to generate Figure 10.77 illustrating Persistent Contrastive Divergence (PCD) Loss Landscape (3D)

The Python code below produces the Figure 10.77 illustrating Persistent Contrastive Divergence (PCD) Loss Landscape (3D).

```python
1
2
3
4
5
6
7
8

import numpy as np
import matplotlib.pyplot as plt
from scipy.special import expt as sigmoid
from mpl_toolkits.mplot3d import Axes3D

# -----------------------------
# PCD Loss with 1 visible + 2 hidden units
# -----------------------------
```
```python
def pcd_loss_2d(w1, w2, data_v, data_h1, data_h2, steps=50):
    """
    Persistent Contrastive Divergence loss
    w1, w2 : weights (can be arrays)
    data_v, data_h1, data_h2 : observed data stats
    steps : number of Gibbs updates for persistent chain
    """
    # Positive phase (data expectation)
    pos = data_v * (data_h1 + data_h2)

    # Initialize persistent visible state randomly
    v_model = np.random rand(np.shape(w1))

    # Persistent chain Gibbs sampling
    for _ in range(steps):
        # Hidden probabilities given visible
        prob_h1_given_v = sigmoid(v_model * w1)
        prob_h2_given_v = sigmoid(v_model * w2)

        # Sample hidden states
        h1 = (np.random rand(np.shape(w1)) < prob_h1_given_v).astype(float)
        h2 = (np.random rand(np.shape(w2)) < prob_h2_given_v).astype(float)

        # Re-sample visible from hidden
        v_model = sigmoid(h1 * w1 + h2 * w2)

    # Negative phase expectation
    prob_h1_given_vm = sigmoid(v_model * w1)
    prob_h2_given_vm = sigmoid(v_model * w2)
    neg = v_model * (prob_h1_given_vm + prob_h2_given_vm)

    return -(pos - neg)

# -----------------------------
# Experiment Setup
# -----------------------------
data_v = 1.0   # visible always on
data_h1 = 0.8  # hidden unit 1 activation
data_h2 = 0.6  # hidden unit 2 activation

w_range = np.linspace(-6, 6, 60)  # fewer points for 3D rendering speed
W1, W2 = np.meshgrid(w_range, w_range)
losssurface = pcd_loss_2d(W1, W2, data_v, data_h1, data_h2, steps=100)

# -----------------------------
# 3D Surface Plot
# -----------------------------
fig = plt.figure(figsize=(10,7))
ax = fig.add_subplot(111, projection="3d")

surf = ax plot_surface(W1, W2, lossservice, cmap="viridis", edgecolor="none", alpha=0.9)
fig.colorbar(surf, ax=ax, shrink=0.5, aspect=10, label="PCD Loss")
```
```python
ax.set_xlabel("Weight $W_1$")
ax.set_ylabel("Weight $W_2$")
ax.set_zlabel("PCD Loss")
ax.set_title("Persistent Contrastive Divergence (PCD) Loss Landscape (3D )")

plt.show()
```

### 10.5.3.75 Python Code to generate Figure 10.78 illustrating Hausdorff Distance Loss Visualization

The Python code below produces the Figure 10.78 illustrating Hausdorff Distance Loss Visualization.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.spatial.distance import cdist

# -----------------------------
# Compute Hausdorff Distance
# -----------------------------

def hausdorff_distance(A, B):
    # Pairwise distances
    D = cdist(A, B, metric='euclidean')

    # Directed distances
    d_AB = np.max(np.min(D, axis=1)) # sup_a inf_b
    d_BA = np.max(np.min(D, axis=0)) # sup_b inf_a

    return max(d_AB, d_BA)

# -----------------------------
# Example point sets
# -----------------------------

# Ground truth: circle
theta = np.linspace(0, 2* np.pi, 100)
A = np.c_[np.cos theta, np.sin theta]]

# Prediction: shifted ellipse
phi = np.linspace(0, 2* np.pi, 100)
B = np.c_[1.2 * np.cos phi + 0.5, 0.8 * np.sin phi + 0.3]

# Compute Hausdorff Distance
hd = hausdorff_distance(A, B)

# -----------------------------
# Plot
# -----------------------------

plt.figure(figsize=(7,7))
plt plot(A[:,0], A[:,1], 'b-', label="Ground Truth (A)")
plt plot(B[:,0], B[:,1], 'r--', label="Prediction (B)")

# Highlight some closest points
D = cdist(A, B)
```
# 11 Few-Shot Learning

Few-shot learning (FSL) refers to the problem of training a model to generalize to new tasks using only a limited number of labeled examples. Mathematically, given a dataset $\mathcal{D} = \{(\mathbf{x}_i, y_i)\}_{i=1}^N$ consisting of $N$ labeled instances, a conventional supervised learning model aims to learn a mapping $f: \mathcal{X} \to \mathcal{Y}$ by minimizing an empirical risk functional of the form

$$ \mathcal{L}(f) = \sum_{i=1}^{N} \ell(f(\mathbf{x}_i), y_i) \qquad (11.1) $$

where $\ell(\cdot, \cdot)$ is an appropriate loss function such as the cross-entropy loss

$$ \ell(f(\mathbf{x}_i), y_i) = - \sum_{k=1}^{K} \mathbb{1}(y_i = k) \log P(y_i = k | \mathbf{x}_i, \theta) \qquad (11.2) $$

where $\theta$ denotes the model parameters. However, in a few-shot setting, the number of labeled examples per class is very small, typically denoted as $K$, where $K \ll N$, making it challenging to generalize in a conventional learning paradigm. Few-shot learning is often formalized using meta-learning approaches where the goal is to optimize a learning algorithm itself such that it can quickly adapt to new tasks using very few labeled samples. A common meta-learning formulation involves the optimization of a meta-objective defined over a distribution of tasks $\mathcal{T}$. Given a distribution $p(\mathcal{T})$, a meta-learner optimizes parameters $\theta$ such that the expected loss over tasks is minimized:

$$ \min_{\theta} \mathbb{E}_{\mathcal{T} \sim p(\mathcal{T})} [\mathcal{L}_{\mathcal{T}}(f_{\theta})] \qquad (11.3) $$

where each task $\mathcal{T}$ consists of a small training set $\mathcal{S}_{\text{train}}$ (the support set) and a validation set $\mathcal{S}_{\text{test}}$ (the query set). The loss function for a given task $\mathcal{T}$ is typically written as:

$$ \mathcal{L}_{\mathcal{T}}(f_{\theta}) = \sum_{(\mathbf{x}, y) \in \mathcal{S}_{\text{test}}} \ell(f_{\theta}^{\mathcal{S}_{\text{train}}}(\mathbf{x}), y) \qquad (11.4) $$

where $f_{\theta}^{\mathcal{S}_{\text{train}}}$ denotes the model adapted using the small support set $\mathcal{S}_{\text{train}}$. A well-known approach to FSL is Model-Agnostic Meta-Learning (MAML), where the optimization is performed by updating $\theta$ such that it rapidly adapts to new tasks via a few gradient updates. Specifically, MAML defines a bi-level optimization problem where an inner loop computes task-specific parameters $\theta'$ via gradient descent:

$$ \theta' = \theta - \alpha \nabla_{\theta} \mathcal{L}_{\mathcal{T}}(f_{\theta}) \qquad (11.5) $$

where $\alpha$ is the step size. The outer loop then updates $\theta$ based on the loss incurred on the query set:

$$ \theta \leftarrow \theta - \beta \sum_{\mathcal{T} \sim p(\mathcal{T})} \nabla_{\theta} \mathcal{L}_{\mathcal{T}}(f_{\theta'}) \qquad (11.6) $$

where $\beta$ is another learning rate. The goal of MAML is to find an initialization $\theta$ such that a small number of gradient steps suffices for good generalization. Another prominent FSL approach
is metric-based learning, where models learn an embedding function $g_\theta: \mathcal{X} \to \mathbb{R}^d$ such that a similarity function $S(g_\theta(\mathbf{x}_i), g_\theta(\mathbf{x}_j))$ enables effective classification in a low-data regime. A widely used method is prototypical networks, which represent each class in the support set with a prototype:

$$ \mathbf{c}_k = \frac{1}{|\mathcal{S}_{\text{train},k}|} \sum_{(\mathbf{x}_i, y_i) \in \mathcal{S}_{\text{train},k}} g_\theta(\mathbf{x}_i) \qquad (11.7) $$

where $\mathcal{S}_{\text{train},k}$ is the subset of the support set corresponding to class $k$. Classification is then performed using a softmax function over distances:

$$ P(y = k | \mathbf{x}) = \frac{\exp(-d(g_\theta(\mathbf{x}), \mathbf{c}_k))}{\sum_{k'} \exp(-d(g_\theta(\mathbf{x}), \mathbf{c}_{k'})}) \qquad (11.8) $$

where $d(\cdot, \cdot)$ is typically the squared Euclidean distance. The training objective is the negative log-likelihood:

$$ \mathcal{L} = - \sum_{(\mathbf{x}, y) \in \mathcal{S}_{\text{test}}} \log P(y | \mathbf{x}) \qquad (11.9) $$

Bayesian methods in FSL adopt a probabilistic framework where uncertainty in model parameters is explicitly modeled using a prior $p(\theta)$. Given a support set $\mathcal{S}_{\text{train}}$, a Bayesian model infers a posterior over parameters:

$$ p(\theta | \mathcal{S}_{\text{train}}) \propto p(\mathcal{S}_{\text{train}} | \theta) p(\theta) \qquad (11.10) $$

A predictive distribution over a new query point $\mathbf{x}^*$ is obtained by marginalizing over $\theta$:

$$ p(y^* | \mathbf{x}^*, \mathcal{S}_{\text{train}}) = \int p(y^* | \mathbf{x}^*, \theta) p(\theta | \mathcal{S}_{\text{train}}) d\theta \qquad (11.11) $$

Ultimately, the success of FSL hinges on leveraging shared structure across tasks, designing effective adaptation mechanisms, and optimizing inductive biases to mitigate data scarcity.

## 11.1 Meta-learning formulation in Few Shot Learning

Meta-learning, particularly in the context of Few-Shot Learning (FSL), aims to train a model such that it can generalize effectively to novel tasks with very few training examples. This is accomplished by optimizing the learning process itself, enabling the model to rapidly adapt to new tasks. Given a distribution over tasks $p(\mathcal{T})$, where each task $\mathcal{T}_i$ is characterized by a small support set $S_i = \{(\mathbf{x}_{i,j}, y_{i,j})\}_{j=1}^K$ and a corresponding query set $Q_i$, the objective of meta-learning is to train a model $f_\theta$ with parameters $\theta$ such that it minimizes the expected loss over new tasks drawn from $p(\mathcal{T})$.

$$ \min_{\theta} \mathbb{E}_{\mathcal{T} \sim p(\mathcal{T})} [\mathcal{L}_{\mathcal{T}}(f_{\theta})] \qquad (11.12) $$

where $\mathcal{L}_{\mathcal{T}}(f_\theta)$ represents the loss function associated with the task $\mathcal{T}$, typically computed over the query set after adapting the model using the support set. One of the most widely used meta-learning approaches is Model-Agnostic Meta-Learning (MAML), which optimizes for an initial parameter set $\theta$ that can be quickly adapted to a new task using only a few gradient steps. The inner-loop adaptation updates the model parameters as:

$$ \theta'_{i} = \theta - \alpha \nabla_{\theta} \mathcal{L}_{\mathcal{T}_{i}}(f_{\theta}) \qquad (11.13) $$

where $\alpha$ is the inner-loop learning rate. The meta-objective then minimizes the loss computed on the query set with respect to the updated parameters:

$$ \min_{\theta} \sum_{\mathcal{T}_i \sim p(\mathcal{T})} \mathcal{L}_{\mathcal{T}_i}(f_{\theta'_{i}}) \qquad (11.14) $$
which requires computing the meta-gradient:

$$ \nabla_{\theta} \sum_{\mathcal{T}_i} \mathcal{L}_{\mathcal{T}_i}(f_{\theta-\alpha \nabla_{\theta} \mathcal{L}_{\mathcal{T}_i}(f_{\theta})}) \qquad (11.15) $$

leading to the final meta-update:

$$ \theta \leftarrow \theta - \beta \sum_{\mathcal{T}_i} \nabla_{\theta} \mathcal{L}_{\mathcal{T}_i}(f_{\theta'_i}) \qquad (11.16) $$

where $\beta$ is the meta-learning rate. Another formulation follows metric-based meta-learning, where the model learns an embedding function $g_{\theta}$ that maps input samples to a feature space where classification can be performed using a simple distance metric. In Prototypical Networks, the class prototype for each class c in the support set is computed as:

$$ \mathbf{c}_c = \frac{1}{|S_c|} \sum_{(\mathbf{x}_j, y_j) \in S_c} g_{\theta}(\mathbf{x}_j) \qquad (11.17) $$

where $S_c$ is the set of examples belonging to class c. The probability of assigning a query sample $\mathbf{x}_q$ to class c is then given by:

$$ p(y_q = c | \mathbf{x}_q) = \frac{\exp(-d(g_{\theta}(\mathbf{x}_q), \mathbf{c}_c))}{\sum_{c'} \exp(-d(g_{\theta}(\mathbf{x}_q), \mathbf{c}_{c'}))} \qquad (11.18) $$

where $d(\cdot, \cdot)$ is a distance metric, often chosen as the squared Euclidean distance. Contrastive learning-based meta-learning methods instead optimize a loss function based on relative similarities. A common choice is the contrastive loss:

$$ \mathcal{L} = \sum_{i,j} \mathbb{I}[y_i = y_j] \cdot d(g_{\theta}(\mathbf{x}_i), g_{\theta}(\mathbf{x}_j)) - (1 - \mathbb{I}[y_i = y_j]) \cdot \max(0, m - d(g_{\theta}(\mathbf{x}_i), g_{\theta}(\mathbf{x}_j))) \qquad (11.19) $$

where $m$ is a margin parameter and $\mathbb{I}[\cdot]$ is an indicator function. Bayesian meta-learning approaches introduce a probabilistic treatment where the model parameters $\theta$ are sampled from a learned posterior distribution $p(\theta | D)$, updated via variational inference:

$$ q_{\phi}(\theta) = \arg \min_{q} D_{KL}(q(\theta) \parallel p(\theta | D)) \qquad (11.20) $$

where $D_{KL}(\cdot \parallel \cdot)$ denotes the Kullback-Leibler divergence. The expected posterior predictive distribution for a new task is then computed as:

$$ p(y_q | \mathbf{x}_q, S) = \mathbb{E}_{\theta \sim q_{\phi}(\theta)}[p(y_q | \mathbf{x}_q, \theta)] \qquad (11.21) $$

which can be approximated using Monte Carlo sampling. Gradient-based meta-learning methods can also be formulated using recurrent neural networks, where the meta-learner is a recurrent model that updates a task-specific hidden state $h_t$ over a sequence of examples. Given a sequence of support set updates, the parameterized recurrence is given by:

$$ h_t = f_{\psi}(h_{t-1}, \nabla_{\theta} \mathcal{L}_t) \qquad (11.22) $$

and the model parameters for a given task are inferred as:

$$ \theta_t = g_{\psi}(h_t) \qquad (11.23) $$

where $f_{\psi}$ and $g_{\psi}$ are recurrent function approximators parameterized by $\psi$. These meta-learning paradigms share a common objective: to minimize the expected generalization error of a model trained over a distribution of tasks, enabling rapid adaptation to new scenarios with minimal labeled data. The fundamental mechanism across formulations involves optimizing a bi-level objective where the outer loop adjusts meta-parameters and the inner loop performs task-specific adaptation. The optimization problem governing this framework can be expressed in a general form as:

$$ \min_{\theta} \mathbb{E}_{\mathcal{T} \sim p(\mathcal{T})} [\mathcal{L}_{\mathcal{T}}(f_{\theta-\nabla_{\theta} \mathcal{L}_{\mathcal{T}}(f_{\theta})})] \qquad (11.24) $$

which encapsulates the core philosophy of meta-learning: learning to learn.
## 11.2 Bayesian Methods in Few Shot Learning

Bayesian methods in Few-Shot Learning (FSL) provide a principled probabilistic framework for reasoning about uncertainty in the context of limited data. The core objective of FSL is to train a model on a set of "base" classes with abundant data such that it can rapidly generalize to a set of novel classes, for which only a few labeled examples (the support set $S$) are provided. The Bayesian approach treats the model parameters, or more commonly a task-specific context, as latent random variables, enabling the quantification of epistemic uncertainty arising from the scarcity of data. This is fundamentally expressed through Bayes' theorem. Let $D$ represent the support set for a novel task, and let $\theta$ represent the model parameters or task-specific adaptation parameters. The posterior distribution over these parameters is given by

$$P(\theta|D) = \frac{P(D|\theta)P(\theta)}{P(D)} \qquad (11.25)$$

where $P(\theta)$ is the prior distribution encapsulating knowledge acquired from the base classes, $P(D|\theta)$ is the likelihood of the support data, and

$$P(D) = \int P(D|\theta)P(\theta) d\theta \qquad (11.26)$$

is the model evidence. The intractability of this evidence integral necessitates approximate inference techniques.

A prominent Bayesian formulation for FSL involves placing a prior over the parameters of the classifier for novel classes, while the feature embedding network is considered fixed and learned during meta-training. For a $N$-way, $K$-shot task, the support set is

$$S = \{(x_i, y_i)\}_{i=1}^{N \times K} \qquad (11.27)$$

The goal is to infer the parameters $\phi$ of the novel-class classifier. A hierarchical Bayesian model posits

$$P(\phi|S) \propto P(S|\phi)P(\phi|\theta) \qquad (11.28)$$

where $\theta$ are hyperparameters learned from the base classes. The prior $P(\phi|\theta)$ regularizes the novel class classifier. For a linear classifier where $\phi = \{w_c\}_{c=1}^N$, a common prior is an isotropic Gaussian,

$$P(w_c|\theta) = N(w_c|\mu_\theta, \sigma_\theta^2 I) \qquad (11.29)$$

The likelihood for the support set, assuming a softmax classification model, is

$$P(S|\phi) = \prod_{i=1}^{NK} P(y_i|x_i, \phi) = \prod_{i=1}^{NK} \text{Categorical}(y_i|\text{softmax}(W^T f(x_i))) \qquad (11.30)$$

where $W$ is the matrix of weight vectors $\{w_c\}$ and $f(\cdot)$ is the feature embedding function. The posterior $P(\phi|S)$ is then used for prediction on a query point $x^*$ via

$$P(y^*|x^*, S) = \int P(y^*|x^*, \phi)P(\phi|S) d\phi \qquad (11.31)$$

Direct computation of the posterior $P(\phi|S)$ is analytically intractable due to the non-conjugacy of the softmax likelihood and the Gaussian prior. Therefore, variational inference is extensively employed. In this framework, a parameterized variational distribution $q_{\lambda}(\phi)$ is introduced to approximate the true posterior. The goal is to minimize the Kullback-Leibler (KL) divergence between
the variational approximation and the true posterior, KL($q_{\lambda}(\phi) || P(\phi|S)$). This is equivalent to maximizing the Evidence Lower Bound (ELBO),

$$ \mathcal{L}(\lambda) = \mathbb{E}_{q_{\lambda}(\phi)}[\log P(S|\phi)] - \text{KL}(q_{\lambda}(\phi) || P(\phi|\theta)) \quad (11.32) $$

The parameters $\lambda$ of the variational distribution are optimized during this inference step. For example, if

$$ q_{\lambda}(w_c) = \mathcal{N}(w_c|\mu_c, \Sigma_c) \quad (11.33) $$

then $\lambda = \{\mu_c, \Sigma_c\}_{c=1}^N$. The expectation is typically estimated via Monte Carlo sampling,

$$ \mathbb{E}_{q_{\lambda}(\phi)}[\log P(S|\phi)] \approx \frac{1}{T} \sum_{t=1}^{T} \sum_{i=1}^{NK} \log P(y_i|x_i, \phi^{(t)}) \quad (11.34) $$

where $\phi^{(t)} \sim q_{\lambda}(\phi)$. The predictive distribution for a query point becomes

$$ P(y^* = c|x^*, S) \approx \frac{1}{T} \sum_{t=1}^{T} \text{softmax}((W^{(t)})^T f(x^*))_c \quad (11.35) $$

marginalizing over the posterior uncertainty.

An alternative and highly influential Bayesian perspective is embodied by the framework of Bayesian meta-learning, particularly through models like the Neural Process and its variants. Here, the entire context of the support set is used to define a latent variable $z$ that represents the task. The generative model for a task is

$$ P(D) = \int P(z) \prod_{(x,y) \in D} P(y|x, z) dz \quad (11.36) $$

The conditional likelihood $P(y|x, z)$ is implemented by a decoder network. The posterior over the task variable given the support set is

$$ P(z|S) \propto P(S|z)P(z) \quad (11.37) $$

Inference is again performed approximately, often using a variational inference scheme where an encoder network $q_{\psi}(z|S)$ amortizes the inference process by mapping the entire support set to a distribution over $z$. The ELBO for this model is

$$ \mathcal{L}(\psi, \omega) = \mathbb{E}_{q_{\psi}(z|S)}[\log P_{\omega}(S|z)] - \text{KL}(q_{\psi}(z|S) || P(z)) \quad (11.38) $$

where $\omega$ are the decoder parameters. The predictive distribution for a query point is

$$ P(y^*|x^*, S) = \int P_{\omega}(y^*|x^*, z)q_{\psi}(z|S) dz \approx \frac{1}{T} \sum_{t=1}^{T} P_{\omega}(y^*|x^*, z^{(t)}) \quad (11.39) $$

with $z^{(t)} \sim q_{\psi}(z|S)$.

Gaussian Processes (GPs) offer another rigorously Bayesian non-parametric approach to FSL. A GP defines a prior directly over functions,

$$ f(x) \sim \mathcal{GP}(m(x), k(x, x')) \quad (11.40) $$

where $m(x)$ is the mean function and $k(x, x')$ is the covariance kernel function. In the FSL setting, the support set $S$ conditions this prior to yield a posterior process. The key equations for prediction
are derived from the joint Gaussian distribution of the support and query points. Let $X_S$ and $y_S$ be the inputs and labels of the support set, and $X_Q$ be the query inputs. The joint prior is

$$ \begin{bmatrix} f(X_S) \\ f(X_Q) \end{bmatrix} \sim \mathcal{N} \left( \begin{bmatrix} m(X_S) \\ m(X_Q) \end{bmatrix}, \begin{bmatrix} K_{SS} & K_{SQ} \\ K_{QS} & K_{QQ} \end{bmatrix} \right) \qquad (11.41) $$

where $[K_{SS}]_{ij} = k(x_i, x_j)$ for $x_i, x_j \in X_S$, and similarly for $K_{SQ}, K_{QS}, K_{QQ}$. For regression, the posterior predictive distribution is Gaussian:

$$ P(f(X_Q)|X_S, y_S, X_Q) = \mathcal{N}(f(X_Q)|\mu^*, \Sigma^*) \qquad (11.42) $$

with

$$ \mu^* = m(X_Q) + K_{QS}(K_{SS} + \sigma_n^2 I)^{-1}(y_S - m(X_S)) \qquad (11.43) $$

and

$$ \Sigma^* = K_{QQ} - K_{QS}(K_{SS} + \sigma_n^2 I)^{-1}K_{SQ} \qquad (11.44) $$

For classification, a likelihood $P(y|f(x))$, such as a Bernoulli-logit, is introduced, and the posterior becomes non-Gaussian, requiring approximation methods like Laplace approximation or expectation propagation.

The Bayesian treatment naturally allows for the integration of uncertainty into the learning process itself, a concept formalized in Bayesian optimization for hyperparameter tuning in FSL. The acquisition function, which guides the search for optimal hyperparameters $\eta$, leverages a surrogate model for the objective function $J(\eta)$. Given a prior $P(J)$ over functions, typically a GP, and observed data $\mathcal{D} = \{(\eta_i, J(\eta_i))\}$, the posterior $P(J|\mathcal{D})$ is used to compute an acquisition function $a(\eta)$ such as Expected Improvement (EI),

$$ \mathrm{EI}(\eta) = \mathbb{E}_{P(J|\mathcal{D})}[\max(0, J(\eta) - J(\eta^+))] \qquad (11.45) $$

where $\eta^+$ is the current best hyperparameter configuration. This rigorous probabilistic framework for hyperparameter selection is crucial in FSL where the risk of overfitting to the few shots is high. The unifying theme across all these Bayesian methods is the explicit representation of uncertainty through probability distributions, transforming the few-shot learning problem from a point-estimation task into one of full posterior inference, thereby enhancing robustness and calibration.

## 11.3 Prototypical networks in Few Shot Learning

Prototypical networks operate within the framework of few-shot classification by learning a non-linear embedding of input data into a metric space where classification can be performed by computing distances to prototype representations of each class. The foundational assumption is that there exists an embedding function $f_\phi: \mathcal{X} \to \mathbb{R}^d$ with parameters $\phi$ that maps inputs into a d-dimensional space such that points cluster around a single prototype representation for each class. Given a support set $S = \{(x_1, y_1), (x_2, y_2), \dots, (x_N, y_N)\}$ where $x_i$ is an input and $y_i \in \{1, \dots, K\}$ is its corresponding class label, the prototype for each class $k$ is computed as the mean vector of the embedded support points belonging to that class. This computation is formally expressed by the equation

$$ c_k = \frac{1}{|S_k|} \sum_{(x_i, y_i) \in S_k} f_\phi(x_i) \qquad (11.46) $$

where $S_k$ denotes the set of support examples labeled with class $k$.

For a query point $x$, the prototypical network produces a distribution over classes based on a
softmax over the negative distances from the embedded query point to the class prototypes. The distance metric is typically the squared Euclidean distance in the embedding space, defined as

$$d(f_{\phi}(x), c_k) = \|f_{\phi}(x) - c_k\|^2 \qquad (11.47)$$

The probability that the query point $x$ belongs to class $k$ is then given by

$$p_{\phi}(y=k|x) = \frac{\exp(-d(f_{\phi}(x), c_k))}{\sum_{k'} \exp(-d(f_{\phi}(x), c_{k'})}) \qquad (11.48)$$

This formulation aligns with the concept of a Gibbs distribution, where the negative distance acts as an energy function, making the probability proportional to the exponentiated similarity between the query embedding and the class prototype.

The training objective is to maximize the likelihood of the true class for all query points across training episodes, which is equivalent to minimizing the negative log-probability. For a single episode with support set $S$ and a query set $Q$, the loss function is defined as

$$J(\phi) = -\frac{1}{|Q|} \sum_{(x_j, y_j) \in Q} \log p_{\phi}(y = k^* | x_j) \qquad (11.49)$$

where $k^*$ is the true class label of the query point $x_j$. Substituting the probability expression, the loss becomes

$$J(\phi) = -\frac{1}{|Q|} \sum_{(x_j, y_j) \in Q} \log \left( \frac{\exp(-\|f_{\phi}(x_j) - c_{k^*}\|^2)}{\sum_k \exp(-\|f_{\phi}(x_j) - c_k\|^2)} \right) \qquad (11.50)$$

This can be simplified to

$$J(\phi) = \frac{1}{|Q|} \sum_{(x_j, y_j) \in Q} \left[ \|f_{\phi}(x_j) - c_{k^*}\|^2 + \log \sum_k \exp(-\|f_{\phi}(x_j) - c_k\|^2) \right] \qquad (11.51)$$

which is effectively a cross-entropy loss computed in the metric space.

The learning process occurs through episodic training. Each episode is constructed by randomly selecting a subset of classes $C \subset C_{train}$, then for each class $k \in C$, selecting a support set $S_k$ and a query set $Q_k$. The complete support set for the episode is $S = \cup_{k \in C} S_k$ and the query set is $Q = \cup_{k \in C} Q_k$. The prototypes $c_k$ for all $k \in C$ are computed from $S$, and the parameters $\phi$ of the embedding network are updated by performing a gradient descent step on the loss $J(\phi)$ computed on $Q$. The update rule is

$$\phi \leftarrow \phi - \alpha \nabla_{\phi} J(\phi) \qquad (11.52)$$

where $\alpha$ is the learning rate. The gradients flow through both the query point embeddings and the prototype computations, as the prototypes $c_k$ are functions of $\phi$ via the support examples. The gradient of the loss with respect to a prototype vector is

$$\nabla_{c_k} J = \frac{1}{|Q|} \sum_{(x_j, y_j) \in Q} 2(p_{\phi}(k|x_j) - \mathbb{1}\{y_j = k\})(f_{\phi}(x_j) - c_k) \qquad (11.53)$$

where $\mathbb{1}\{\cdot\}$ is the indicator function. The gradient with respect to the embedding of a support point $x_i \in S_k$ is

$$\nabla_{f_{\phi}(x_i)} J = \frac{1}{|S_k|} \nabla_{c_k} J \qquad (11.54)$$

and the gradient with respect to the embedding of a query point $x_j$ is

$$\nabla_{f_{\phi}(x_j)} J = \frac{2}{|Q|} \sum_k (\mathbb{1}\{y_j = k\} - p_{\phi}(k|x_j))(f_{\phi}(x_j) - c_k) \qquad (11.55)$$
The scientific rigor of this approach is rooted in its interpretation as a form of mixture density estimation on the embedded space. If each class-conditional distribution in the embedding space is a spherical Gaussian with a fixed, shared covariance matrix $\sigma^2 I$, then the probability density for a point $z = f_\phi(x)$ under class $k$ is

$$p(z|y=k) = \frac{1}{(2\pi\sigma^2)^{d/2}} \exp\left(-\frac{1}{2\sigma^2}\|z - c_k\|^2\right) \qquad (11.56)$$

Assuming equal prior probabilities for all classes, the posterior probability for class $k$ given $z$ is

$$p(y=k|z) = \frac{p(z|y=k)}{\sum_{k'} p(z|y=k')} = \frac{\exp(-\frac{1}{2\sigma^2}\|z - c_k\|^2)}{\sum_{k'} \exp(-\frac{1}{2\sigma^2}\|z - c_{k'}\|^2)} \qquad (11.57)$$

By setting $\sigma^2 = \frac{1}{2}$, this posterior simplifies exactly to the softmax over negative squared Euclidean distances used in the prototypical network,

$$p_{\phi}(y=k|x) = \frac{\exp(-\|f_{\phi}(x) - c_k\|^2)}{\sum_{k'} \exp(-\|f_{\phi}(x) - c_{k'}\|^2)} \qquad (11.58)$$

This probabilistic foundation provides a rigorous justification for the model's architecture and loss function, framing it as an approximate inference procedure under a specific generative model of the data in the embedding space. The model's efficacy is thus contingent upon the embedding network's ability to transform the input data such that its distribution in the latent space conforms to this isotropic Gaussian assumption per class.

## 11.4 Model-Agnostic Meta-Learning (MAML) in Few Shot Learning

Model-Agnostic Meta-Learning (MAML) is a fundamental optimization-based approach in few-shot learning, designed to enable models to quickly adapt to new tasks with minimal data. The core idea is to learn an initialization of model parameters that facilitates rapid adaptation through gradient-based updates. Given a distribution over tasks, denoted as $p(\mathcal{T})$, MAML optimizes for a set of parameters $\theta$ such that for a new task $\mathcal{T}_i \sim p(\mathcal{T})$, the adapted parameters $\theta'_i$ after a small number of gradient steps yield high performance on the task.

For a given task $\mathcal{T}_i$, we assume a dataset $\mathcal{D}_i$ composed of a support set $\mathcal{D}_i^{\text{train}}$ and a query set $\mathcal{D}_i^{\text{test}}$. Let the model be parameterized by $\theta$ and let the task-specific loss function be $\mathcal{L}_{\mathcal{T}_i}$. The adaptation to task $\mathcal{T}_i$ involves performing one or more gradient descent steps with respect to $\mathcal{L}_{\mathcal{T}_i}$ evaluated on $\mathcal{D}_i^{\text{train}}$. The standard first-order adaptation rule for a single gradient step is given by:

$$\theta'_{i} = \theta - \alpha \nabla_{\theta} \mathcal{L}_{\mathcal{T}_i}(\theta, \mathcal{D}_i^{\text{train}}) \qquad (11.59)$$

where $\alpha$ is the learning rate for task-specific updates. The meta-objective is to find an initialization $\theta$ such that after task-specific adaptation, the model performs well on the query set $\mathcal{D}_i^{\text{test}}$. The meta-loss is computed over the query set:

$$\mathcal{L}_{\text{meta}} = \sum_{\mathcal{T}_i \sim p(\mathcal{T})} \mathcal{L}_{\mathcal{T}_i}(\theta'_{i}, \mathcal{D}_{i}^{\text{test}}) \qquad (11.60)$$

which leads to the outer-level optimization of $\theta$:

$$\theta \leftarrow \theta - \beta \nabla_{\theta} \sum_{\mathcal{T}_i \sim p(\mathcal{T})} \mathcal{L}_{\mathcal{T}_i}(\theta'_{i}, \mathcal{D}_{i}^{\text{test}}) \qquad (11.61)$$
where $\beta$ is the meta-learning rate. Expanding the gradient term using the chain rule,

$$ \nabla_{\theta} \mathcal{L}_{\mathcal{T}_i}(\theta'_i, \mathcal{D}_i^{\text{test}}) = \nabla_{\theta'_i} \mathcal{L}_{\mathcal{T}_i}(\theta'_i, \mathcal{D}_i^{\text{test}}) \cdot \frac{d\theta'_i}{d\theta} \qquad (11.62) $$

with

$$ \frac{d\theta'_i}{d\theta} = I - \alpha \nabla_{\theta}^2 \mathcal{L}_{\mathcal{T}_i}(\theta, \mathcal{D}_i^{\text{train}}) \qquad (11.63) $$

where $I$ is the identity matrix and $\nabla_{\theta}^2 \mathcal{L}_{\mathcal{T}_i}$ is the Hessian of the loss function with respect to $\theta$. The optimization process involves computing second-order derivatives, which can be computationally expensive. A first-order approximation, called First-Order MAML (FOMAML), simplifies this update by ignoring the Hessian term:

$$ \theta \leftarrow \theta - \beta \sum_{\mathcal{T}_i \sim p(\mathcal{T})} \nabla_{\theta'_i} \mathcal{L}_{\mathcal{T}_i}(\theta'_i, \mathcal{D}_i^{\text{test}}) \qquad (11.64) $$

where $\theta'_i$ is still obtained using the standard gradient descent update. This approximation reduces computational overhead while still achieving strong meta-learning performance. The effectiveness of MAML depends on its ability to learn an initialization $\theta$ that allows rapid adaptation across a diverse set of tasks. Formally, we define the expected meta-loss over the task distribution:

$$ \mathbb{E}_{\mathcal{T}_i \sim p(\mathcal{T})} [\mathcal{L}_{\mathcal{T}_i}(\theta - \alpha \nabla_{\theta} \mathcal{L}_{\mathcal{T}_i}(\theta, \mathcal{D}_i^{\text{train}}), \mathcal{D}_i^{\text{test}})] \qquad (11.65) $$

which represents the expectation over the loss incurred after a single adaptation step. The meta-optimization aims to minimize this expected loss, ensuring that $\theta$ provides an effective starting point for task-specific fine-tuning. For multiple gradient adaptation steps, the parameter update for a given task follows an iterative process:

$$ \theta_i^{(k)} = \theta_i^{(k-1)} - \alpha \nabla_{\theta_i^{(k-1)}} \mathcal{L}_{\mathcal{T}_i}(\theta_i^{(k-1)}, \mathcal{D}_i^{\text{train}}) \qquad (11.66) $$

for $k = 1, \dots, K$, where $K$ denotes the number of inner-loop gradient updates. The final adapted parameters $\theta_i^{(K)}$ are then used to compute the meta-loss:

$$ \mathcal{L}_{\text{meta}} = \sum_{\mathcal{T}_i \sim p(\mathcal{T})} \mathcal{L}_{\mathcal{T}_i}(\theta_i^{(K)}, \mathcal{D}_i^{\text{test}}) \qquad (11.67) $$

which determines the outer-loop optimization of $\theta$. The full meta-gradient computation involves backpropagating through $K$ gradient updates, leading to higher-order derivative terms:

$$ \nabla_{\theta} \mathcal{L}_{\text{meta}} = \sum_{\mathcal{T}_i \sim p(\mathcal{T})} \nabla_{\theta_i^{(K)}} \mathcal{L}_{\mathcal{T}_i}(\theta_i^{(K)}, \mathcal{D}_i^{\text{test}}) \cdot \prod_{k=1}^{K} \frac{d\theta_i^{(k)}}{d\theta_i^{(k-1)}} \qquad (11.68) $$

where each term captures the second-order effects of iterative gradient updates.

## 11.5 Metric-based learning in Few Shot Learning

Metric-based learning in the context of few-shot learning is a paradigm that relies on learning a similarity metric to compare query examples against a set of labeled support examples. Given a support set $S = \{(x_i, y_i)\}_{i=1}^N$, where each $x_i$ is an input instance and $y_i$ is its corresponding label, and a query instance $x_q$, the goal is to predict $y_q$ based on a learned metric function $d_{\theta}(x_q, x_i)$ parameterized by $\theta$. The fundamental principle is that instances belonging to the same class should be mapped closer together in an embedding space, while instances from different classes
should be mapped further apart. Mathematically, this can be formulated as minimizing the intra-class variance while maximizing the inter-class variance, which can be written as

$$ \min_{\theta} \sum_{(x_i, y_i) \in S} \sum_{(x_j, y_j) \in S} \mathbb{I}(y_i = y_j) d_{\theta}(x_i, x_j) - \lambda \sum_{(x_i, y_i) \in S} \sum_{(x_j, y_j) \in S} \mathbb{I}(y_i \neq y_j) d_{\theta}(x_i, x_j), \quad (11.69) $$

where $\lambda$ is a weighting factor that controls the relative importance of inter-class separation. The metric function $d_{\theta}(x_i, x_j)$ is often implemented using a neural network that maps instances into an embedding space via an encoder $f_{\theta}: \mathcal{X} \to \mathbb{R}^d$, such that

$$ d_{\theta}(x_i, x_j) = \|f_{\theta}(x_i) - f_{\theta}(x_j)\|^2. \quad (11.70) $$

One of the key approaches in metric-based few-shot learning is prototypical networks, where each class in the support set is represented by a prototype $c_k$, computed as the mean of embedded support examples belonging to class k:

$$ c_k = \frac{1}{|S_k|} \sum_{(x_i, y_i) \in S_k} f_{\theta}(x_i). \quad (11.71) $$

The classification of a query example $x_q$ is then performed by computing its similarity to each prototype, typically using the squared Euclidean distance,

$$ p(y_q = k | x_q) = \frac{\exp(-d(f_{\theta}(x_q), c_k))}{\sum_{k'} \exp(-d(f_{\theta}(x_q), c_{k'}))}. \quad (11.72) $$

Another prominent approach is relation networks, which replace the explicit distance metric with a learned similarity function $g_{\phi}: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}$, parameterized by $\phi$. The probability of assigning query $x_q$ to class k is given by

$$ p(y_q = k | x_q) = \frac{\exp(g_{\phi}(f_{\theta}(x_q), c_k))}{\sum_{k'} \exp(g_{\phi}(f_{\theta}(x_q), c_{k'}))}. \quad (11.73) $$

Siamese networks approach metric learning through pairwise comparisons, where a shared encoder $f_{\theta}$ maps pairs of examples $(x_i, x_j)$ into an embedding space, and the similarity is computed as

$$ s_{\theta}(x_i, x_j) = \sigma(W \cdot |f_{\theta}(x_i) - f_{\theta}(x_j)|), \quad (11.74) $$

where $W$ is a learned weight vector and $\sigma$ is the sigmoid activation function. The model is trained to minimize the binary cross-entropy loss,

$$ \mathcal{L} = - \sum_{(x_i, y_i), (x_j, y_j) \in S} \left[ \mathbb{I}(y_i = y_j) \log s_{\theta}(x_i, x_j) + (1 - \mathbb{I}(y_i = y_j)) \log(1 - s_{\theta}(x_i, x_j)) \right]. \quad (11.75) $$

Triplet networks extend this by considering an anchor example $x_a$, a positive example $x_p$ (same class), and a negative example $x_n$ (different class), enforcing the constraint

$$ d(f_{\theta}(x_a), f_{\theta}(x_p)) + m < d(f_{\theta}(x_a), f_{\theta}(x_n)), \quad (11.76) $$

where $m$ is a margin hyperparameter. The loss function is given by

$$ \mathcal{L} = \sum_{\text{triplets}} \max(0, d(f_{\theta}(x_a), f_{\theta}(x_p)) - d(f_{\theta}(x_a), f_{\theta}(x_n)) + m). \quad (11.77) $$

By optimizing these loss functions, metric-based few-shot learning methods enable models to generalize well to novel classes, as the learned metric encodes class-invariant representations of instances. The embedding space acts as a structured manifold where geometric distances capture semantic similarities, allowing rapid adaptation to unseen tasks with minimal labeled data.
## 11.6 Bayesian methods in Few Shot Learning

Bayesian methods in Few-Shot Learning (FSL) provide a probabilistic framework to model uncertainty in learning from a small number of examples. Given a dataset $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$ with a limited number of training examples $N$, Bayesian approaches aim to infer a distribution over the model parameters $\theta$, rather than a single deterministic estimate, allowing the model to generalize effectively despite limited data. The core idea is to use Bayes' theorem to compute the posterior distribution over $\theta$:

$$p(\theta|\mathcal{D}) = \frac{p(\mathcal{D}|\theta)p(\theta)}{p(\mathcal{D})} \qquad (11.78)$$

where $p(\mathcal{D}|\theta)$ is the likelihood of the data given parameters $\theta$, $p(\theta)$ is the prior over the parameters, and $p(\mathcal{D})$ is the marginal likelihood or evidence. The posterior distribution encapsulates the uncertainty in parameter estimation, making it particularly useful for FSL where the data is scarce.

The Bayesian framework enables meta-learning by assuming that tasks in FSL are drawn from an unknown distribution over tasks $p(\mathcal{T})$. Given a set of tasks $\{\mathcal{T}_i\}$, each with a small support set $\mathcal{S}_i$ and query set $\mathcal{Q}_i$, the goal is to learn a distribution over task-specific parameters $\theta_i$ conditioned on the support set:

$$p(\theta_i|\mathcal{S}_i) = \frac{p(\mathcal{S}_i|\theta_i)p(\theta_i)}{p(\mathcal{S}_i)} \qquad (11.79)$$

A hierarchical Bayesian model treats the task-specific parameters $\theta_i$ as drawn from a global distribution with hyperparameters $\phi$:

$$p(\theta_i|\phi) = \mathcal{N}(\mu_{\phi}, \Sigma_{\phi}) \qquad (11.80)$$

where $\phi = (\mu_{\phi}, \Sigma_{\phi})$ defines a prior distribution over $\theta_i$. This enables the model to generalize across tasks by updating the prior $p(\phi)$ using multiple tasks:

$$p(\phi|\{\mathcal{S}_i\}) \propto p(\{\mathcal{S}_i\}|\phi)p(\phi) \qquad (11.81)$$

Given a new task with support set $\mathcal{S}_*$, the predictive distribution over labels $y_*$ for query inputs $x_*$ is obtained by marginalizing over $\theta$:

$$p(y_*|x_*, \mathcal{S}_*) = \int p(y_*|x_*, \theta)p(\theta|\mathcal{S}_*)d\theta \qquad (11.82)$$

Since exact inference of $p(\theta|\mathcal{D})$ is often intractable, variational inference is commonly used. We approximate $p(\theta|\mathcal{D})$ with a variational distribution $q(\theta|\lambda)$, parameterized by $\lambda$, minimizing the Kullback-Leibler (KL) divergence:

$$\mathcal{L}(\lambda) = \mathbb{E}_{q(\theta|\lambda)}[\log p(\mathcal{D}|\theta)] - D_{KL}(q(\theta|\lambda)||p(\theta)) \qquad (11.83)$$

This leads to an optimization problem where $\lambda$ is updated via gradient descent to minimize $\mathcal{L}(\lambda)$, ensuring the learned distribution $q(\theta|\lambda)$ approximates the true posterior. Another approach is Bayesian nonparametrics, where the distribution over functions $f: \mathcal{X} \to \mathcal{Y}$ is modeled using a Gaussian Process (GP), providing a principled way to quantify uncertainty. Given a prior GP:

$$f(x) \sim \mathcal{GP}(m(x), k(x, x')) \qquad (11.84)$$

where $m(x)$ is the mean function and $k(x, x')$ is the covariance function, the posterior over functions given training data $\mathcal{D}$ is:

$$p(f|\mathcal{D}) = \mathcal{GP}(\tilde{m}(x), \tilde{k}(x, x')) \qquad (11.85)$$

where $\tilde{m}(x)$ and $\tilde{k}(x, x')$ are the posterior mean and covariance functions derived using Bayes' rule. The predictive distribution for a new input $x_*$ follows:

$$p(y_*|x_*, \mathcal{D}) = \int p(y_*|f(x_*))p(f|\mathcal{D})df \qquad (11.86)$$
which yields a closed-form Gaussian predictive distribution due to the properties of GPs. Another key Bayesian technique is Bayesian Neural Networks (BNNs), where instead of learning a single weight matrix W, a distribution over weights is maintained:

$$p(W|\mathcal{D}) \propto p(\mathcal{D}|W)p(W) \qquad (11.87)$$

Inference in BNNs requires approximations like Monte Carlo Dropout, where approximate Bayesian inference is performed by applying dropout at both training and test time, yielding an empirical posterior:

$$p(y_*|x_*, \mathcal{D}) \approx \frac{1}{T} \sum_{t=1}^{T} p(y_*|x_*, W_t) \qquad (11.88)$$

where $W_t$ are sampled from the variational posterior $q(W)$. This captures uncertainty in predictions, crucial for robust few-shot learning. Bayesian optimization further refines FSL by selecting informative examples to maximize learning efficiency. Given an acquisition function $a(x)$ based on the posterior predictive distribution:

$$x^* = \arg\max_x a(x) \qquad (11.89)$$

a new query point is selected to minimize uncertainty and maximize information gain, enhancing generalization from few samples. Thus, Bayesian methods in FSL provide a mathematically rigorous probabilistic framework to model uncertainty, transfer knowledge across tasks, and make robust predictions even with limited training data.
# 12 Metric Learning

Metric learning is a fundamental concept in machine learning and mathematical optimization, concerned with learning a distance function or similarity measure that captures the underlying structure of data. Given a set of training samples $\mathcal{X} = \{\mathbf{x}_i\}_{i=1}^N$, where each $\mathbf{x}_i \in \mathbb{R}^d$, the objective is to learn a metric function $d: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}^+$ that quantifies the dissimilarity between any two points. A standard choice for $d(\mathbf{x}_i, \mathbf{x}_j)$ is the Mahalanobis distance, which generalizes the Euclidean distance and is defined as

$$d_{\mathbf{M}}(\mathbf{x}_i, \mathbf{x}_j) = \sqrt{(\mathbf{x}_i - \mathbf{x}_j)^T \mathbf{M} (\mathbf{x}_i - \mathbf{x}_j)} \quad (12.1)$$

where $\mathbf{M}$ is a positive semi-definite matrix, i.e., $\mathbf{M} \succeq 0$, ensuring that the distance satisfies the properties of a metric. Learning the matrix $\mathbf{M}$ is the core challenge in metric learning. When $\mathbf{M} = \mathbf{I}$, the metric reduces to the standard Euclidean distance:

$$d_{\mathbf{I}}(\mathbf{x}_i, \mathbf{x}_j) = \sqrt{(\mathbf{x}_i - \mathbf{x}_j)^T (\mathbf{x}_i - \mathbf{x}_j)} \quad (12.2)$$

However, metric learning seeks to optimize $\mathbf{M}$ such that semantically similar data points have smaller distances than dissimilar points. This is typically formulated as an optimization problem where we enforce constraints on distances between pairs of points. Given a set of pairs $\mathcal{S}$ (similar pairs) and $\mathcal{D}$ (dissimilar pairs), the goal is to satisfy

$$d_{\mathbf{M}}(\mathbf{x}_i, \mathbf{x}_j) \le u, \quad \forall (\mathbf{x}_i, \mathbf{x}_j) \in \mathcal{S} \quad (12.3)$$

$$d_{\mathbf{M}}(\mathbf{x}_i, \mathbf{x}_j) \ge l, \quad \forall (\mathbf{x}_i, \mathbf{x}_j) \in \mathcal{D} \quad (12.4)$$

for some thresholds $u$ and $l$. One widely used approach to metric learning is Large Margin Nearest Neighbors (LMNN), which seeks to optimize $\mathbf{M}$ by minimizing a hinge loss on triplets of points $(\mathbf{x}_i, \mathbf{x}_j, \mathbf{x}_k)$, where $\mathbf{x}_i$ is closer to $\mathbf{x}_j$ than $\mathbf{x}_k$, i.e.,

$$\sum_{(i,j,k)} \max(0, 1 + d_{\mathbf{M}}(\mathbf{x}_i, \mathbf{x}_j) - d_{\mathbf{M}}(\mathbf{x}_i, \mathbf{x}_k)) \quad (12.5)$$

where the triplet loss ensures that the distance between similar points is smaller than the distance between dissimilar ones by a margin of at least 1. An alternative approach is the information-theoretic metric learning (ITML) framework, which minimizes the Kullback-Leibler divergence between two Gaussian distributions defined by different Mahalanobis distances:

$$D_{\text{KL}}(p_{\mathbf{M}} \| p_{\mathbf{M}_0}) = \frac{1}{2} (\text{tr}(\mathbf{M}_0^{-1} \mathbf{M}) - \log \det(\mathbf{M}_0^{-1} \mathbf{M}) - d) \quad (12.6)$$

Deep metric learning extends these concepts by parameterizing the metric function using a neural network $f_{\theta}$ with learnable parameters $\theta$, leading to a learned embedding space where distances are computed as

$$d_{\theta}(\mathbf{x}_i, \mathbf{x}_j) = \|f_{\theta}(\mathbf{x}_i) - f_{\theta}(\mathbf{x}_j)\|^2 \quad (12.7)$$

Instead of learning a linear transformation as in Mahalanobis-based methods, deep metric learning implicitly learns a nonlinear mapping such that the Euclidean distance in the transformed space aligns with semantic similarity. The contrastive loss is commonly used in this context:

$$\sum_{(i,j) \in \mathcal{S}} d_{\theta}(\mathbf{x}_i, \mathbf{x}_j) + \sum_{(i,j) \in \mathcal{D}} \max(0, m - d_{\theta}(\mathbf{x}_i, \mathbf{x}_j))^2 \quad (12.8)$$
Alternatively, the triplet loss is defined as

$$ \sum_{(i,j,k)} \max(0, d_{\theta}(\mathbf{x}_i, \mathbf{x}_j) - d_{\theta}(\mathbf{x}_i, \mathbf{x}_k) + m) \qquad (12.9) $$

A more recent approach, the normalized temperature-scaled cross-entropy loss (NT-Xent), utilizes a softmax formulation:

$$ \ell = - \sum_{i} \log \frac{\exp(\mathrm{sim}(\mathbf{z}_i, \mathbf{z}_j) / \tau)}{\sum_{k \neq i} \exp(\mathrm{sim}(\mathbf{z}_i, \mathbf{z}_k) / \tau)} \qquad (12.10) $$

where $\mathrm{sim}(\mathbf{z}_i, \mathbf{z}_j)$ is the cosine similarity,

$$ \mathrm{sim}(\mathbf{z}_i, \mathbf{z}_j) = \frac{\mathbf{z}_i^T \mathbf{z}_j}{\|\mathbf{z}_i\| \|\mathbf{z}_j\|} \qquad (12.11) $$

By optimizing the metric under these constraints, metric learning produces feature representations that are well-structured for downstream tasks such as retrieval, verification, and clustering.

## 12.1 Large Margin Nearest Neighbors (LMNN) approach to Metric Learning

The Large Margin Nearest Neighbors (LMNN) approach to metric learning is a method that aims to learn a Mahalanobis distance metric that optimizes nearest neighbor classification by ensuring that points belonging to the same class remain close while different-class points are separated by a large margin. Given a dataset $\{(\mathbf{x}_i, y_i)\}_{i=1}^n$, where $\mathbf{x}_i \in \mathbb{R}^d$ represents a feature vector and $y_i$ denotes its class label, LMNN learns a transformation matrix $\mathbf{L} \in \mathbb{R}^{d \times d}$ such that the squared distance between any two points is given by

$$ d_{\mathbf{L}}^2(\mathbf{x}_i, \mathbf{x}_j) = \|\mathbf{L}(\mathbf{x}_i - \mathbf{x}_j)\|^2 = (\mathbf{x}_i - \mathbf{x}_j)^T \mathbf{M} (\mathbf{x}_i - \mathbf{x}_j) \qquad (12.12) $$

where $\mathbf{M} = \mathbf{L}^T \mathbf{L}$ is a positive semi-definite (PSD) matrix. The optimization problem is designed to minimize classification error by enforcing that each sample $\mathbf{x}_i$ remains close to its target neighbors, denoted by $\mathcal{N}_i$, while keeping different-class points at a distance. The objective function consists of two competing terms: a **pull term** that encourages proximity among target neighbors and a **push term** that enforces a large margin separation from impostors (samples of different classes that intrude upon the local neighborhood). The **pull term** is formulated as

$$ \sum_{i} \sum_{j \in \mathcal{N}_i} d_{\mathbf{L}}^2(\mathbf{x}_i, \mathbf{x}_j) = \sum_{i} \sum_{j \in \mathcal{N}_i} (\mathbf{x}_i - \mathbf{x}_j)^T \mathbf{M} (\mathbf{x}_i - \mathbf{x}_j). \qquad (12.13) $$

This term ensures that the squared distances between each point and its target neighbors remain small. The **push term**, on the other hand, ensures that impostors—points $\mathbf{x}_k$ of a different class than $\mathbf{x}_i$ but closer than its farthest target neighbor—are pushed away by at least a margin 1, formulated using hinge loss as

$$ \sum_{i} \sum_{j \in \mathcal{N}_i} \sum_{k} \xi_{ijk} \qquad (12.14) $$

subject to the constraints

$$ d_{\mathbf{L}}^2(\mathbf{x}_i, \mathbf{x}_k) \ge d_{\mathbf{L}}^2(\mathbf{x}_i, \mathbf{x}_j) + 1 - \xi_{ijk}, \quad \xi_{ijk} \ge 0. \qquad (12.15) $$

The total LMNN objective function combines these terms as

$$ \min_{\mathbf{M} \succeq 0} \sum_{i} \sum_{j \in \mathcal{N}_i} d_{\mathbf{L}}^2(\mathbf{x}_i, \mathbf{x}_j) + C \sum_{i} \sum_{j \in \mathcal{N}_i} \sum_{k} \xi_{ijk} \qquad (12.16) $$
where $C$ is a regularization parameter controlling the tradeoff between the pull and push terms. The constraint $\mathbf{M} \succeq 0$ ensures that the learned metric remains a valid Mahalanobis distance. The optimization is typically solved using semidefinite programming (SDP) or projected gradient descent methods while enforcing the PSD constraint on $\mathbf{M}$. To interpret the impact of the learned metric, consider an eigenvalue decomposition of $\mathbf{M}$, given by

$$ \mathbf{M} = \mathbf{U}\mathbf{\Lambda}\mathbf{U}^T \qquad (12.17) $$

where $\mathbf{U}$ is an orthonormal matrix and $\mathbf{\Lambda}$ is a diagonal matrix of eigenvalues. The transformation $\mathbf{L} = \mathbf{U}\mathbf{\Lambda}^{1/2}$ effectively maps the original data space into a new space where distances are reshaped to optimize nearest-neighbor classification performance. The LMNN framework ensures that different classes are well-separated while preserving local neighborhood structures, leading to improved generalization in classification tasks.

## 12.2 Information-theoretic metric learning (ITML) framework approach to Metric Learning

The Information-Theoretic Metric Learning (ITML) framework is a fundamental approach to metric learning that formulates the problem as an optimization task driven by information theory principles, particularly Kullback-Leibler (KL) divergence minimization. Given a set of data points $\{x_i\}_{i=1}^n \subset \mathbb{R}^d$, the goal of ITML is to learn a Mahalanobis distance metric parameterized by a positive semi-definite (PSD) matrix $\mathbf{M}$, so that distances between similar points are minimized while distances between dissimilar points are maintained above a certain threshold. The Mahalanobis distance between two points $x_i$ and $x_j$ under the learned metric $\mathbf{M}$ is defined as

$$ d_{\mathbf{M}}(x_i, x_j) = \sqrt{(x_i - x_j)^T \mathbf{M} (x_i - x_j)} \qquad (12.18) $$

where $\mathbf{M}$ is a symmetric positive semi-definite matrix, i.e., $\mathbf{M} \succeq 0$, ensuring that $d_{\mathbf{M}}(x_i, x_j)$ defines a proper distance function. The fundamental idea of ITML is to find a metric $\mathbf{M}$ that minimizes the divergence from a given prior metric $\mathbf{M}_0$, typically the identity matrix $\mathbf{I}$, subject to constraints that enforce distances between pairs of points to satisfy specified conditions. To formalize this, ITML minimizes the KL divergence between the distributions parameterized by $\mathbf{M}$ and $\mathbf{M}_0$. Given that Gaussian distributions parameterized by a Mahalanobis metric have a natural representation in terms of covariance matrices, the KL divergence between two Gaussian distributions with covariance matrices $\mathbf{M}_0^{-1}$ and $\mathbf{M}_0^{-1}$ is

$$ D_{\text{KL}}(\mathcal{N}(0, \mathbf{M}^{-1}) \| \mathcal{N}(0, \mathbf{M}_0^{-1})) = \frac{1}{2} (\text{tr}(\mathbf{M}_0^{-1}\mathbf{M}) - \log \det(\mathbf{M}_0^{-1}\mathbf{M}) - d) \qquad (12.19) $$

where $\text{tr}(\cdot)$ denotes the trace operator, and $\log \det(\cdot)$ is the logarithm of the determinant. The optimization problem in ITML is thus formulated as

$$ \min_{\mathbf{M} \succeq 0} \text{tr}(\mathbf{M}_0^{-1}\mathbf{M}) - \log \det(\mathbf{M}_0^{-1}\mathbf{M}) \qquad (12.20) $$

subject to pairwise constraints of the form

$$ (x_i - x_j)^T \mathbf{M} (x_i - x_j) \leq u, \quad \forall (x_i, x_j) \in \mathcal{S} \qquad (12.21) $$

$$ (x_i - x_j)^T \mathbf{M} (x_i - x_j) \geq \ell, \quad \forall (x_i, x_j) \in \mathcal{D} \qquad (12.22) $$

where $\mathcal{S}$ and $\mathcal{D}$ denote sets of similar and dissimilar pairs, respectively, and $u, \ell$ are predefined thresholds that control the tightness of the constraints. The use of KL divergence as the objective
function ensures that the learned metric remains close to the prior $\mathbf{M}_0$ while satisfying the constraints. To solve this optimization problem, a log-barrier method is commonly employed, leading to an iterative update for $\mathbf{M}$ of the form

$$ \mathbf{M}^{(t+1)} = \mathbf{M}^{(t)} + \gamma_t \mathbf{M}^{(t)}(x_i - x_j)(x_i - x_j)^T \mathbf{M}^{(t)} \quad (12.23) $$

where $\gamma_t$ is an adaptive step size chosen to satisfy the constraints. This iterative procedure ensures that $\mathbf{M}$ remains positive semi-definite at every step. The learned metric is influenced by the data-driven constraints while maintaining a balance between adaptation and preservation of the prior structure. The ITML framework exhibits strong theoretical properties, particularly in terms of generalization and convexity. Since the optimization problem is convex in $\mathbf{M}$, the solution is globally optimal, ensuring robustness. Moreover, the constraint formulation allows for a natural interpretation: if the prior $\mathbf{M}_0 = \mathbf{I}$, then the learned metric is a transformation of Euclidean space that deforms distances according to the prescribed similarity relationships. In practice, ITML is highly effective due to its ability to incorporate prior knowledge and adapt flexibly to data distributions. The reliance on KL divergence ensures stability in learning, preventing excessive deviation from the prior. By iteratively refining the metric with convex optimization, ITML efficiently learns a Mahalanobis metric that best captures the underlying relationships within the data.

## 12.3 Deep Metric Learning

Deep Metric Learning (DML) is a fundamental approach in machine learning that aims to learn an embedding function $f_\theta: \mathbb{R}^d \to \mathbb{R}^m$ that maps input data points from a high-dimensional space $\mathbb{R}^d$ to a lower-dimensional space $\mathbb{R}^m$, such that semantically similar points are brought closer together while dissimilar points are pushed farther apart. Formally, given a dataset $\mathcal{X} = \{x_i\}_{i=1}^N$ with labels $\mathcal{Y} = \{y_i\}_{i=1}^N$, the goal is to optimize the parameters $\theta$ of the embedding function $f_\theta$ such that for any triplet of points $(x_a, x_p, x_n)$ where $x_a$ (anchor) and $x_p$ (positive) belong to the same class ($y_a = y_p$), while $x_n$ (negative) belongs to a different class ($y_a \neq y_n$), the Euclidean distance in the embedding space satisfies

$$ \|f_\theta(x_a) - f_\theta(x_p)\|_2^2 + \alpha < \|f_\theta(x_a) - f_\theta(x_n)\|_2^2, \quad (12.24) $$

where $\alpha > 0$ is a margin that enforces a minimum separation between positive and negative pairs. To achieve this objective, DML often employs contrastive loss, which minimizes the Euclidean distance between positive pairs while maximizing it for negative pairs. Given a pair $(x_i, x_j)$, the contrastive loss function is

$$ \mathcal{L}_{\text{contrastive}} = (1 - y_{ij}) \max(0, m - d_{ij})^2 + y_{ij} d_{ij}^2, \quad (12.25) $$

where $y_{ij} = 1$ if $x_i$ and $x_j$ belong to the same class, $y_{ij} = 0$ otherwise, $d_{ij} = \|f_\theta(x_i) - f_\theta(x_j)\|_2$ is the Euclidean distance, and $m$ is a margin parameter. Another widely used loss function in DML is the triplet loss, given by

$$ \mathcal{L}_{\text{triplet}} = \sum_{(a,p,n) \in \mathcal{T}} \max(0, \|f_{\theta}(x_a) - f_{\theta}(x_p)\|_2^2 - \|f_{\theta}(x_a) - f_{\theta}(x_n)\|_2^2 + \alpha). \quad (12.26) $$

This loss function ensures that the anchor-positive distance is always smaller than the anchor-negative distance by at least $\alpha$, thereby enforcing clustering of similar instances while separating dissimilar ones. In deep learning-based metric learning, the embedding function $f_\theta$ is typically parameterized by a deep neural network such as a convolutional neural network (CNN) for image data or a recurrent neural network (RNN) for sequential data. Let $x$ be an input sample and let the network be denoted as $f_\theta(x)$. The network is optimized by minimizing the empirical risk

$$ \mathcal{L}(\theta) = \mathbb{E}_{(x_a, x_p, x_n) \sim \mathcal{D}} \mathcal{L}_{\text{triplet}}(x_a, x_p, x_n), \quad (12.27) $$
where $\mathcal{D}$ represents the data distribution. To further refine the embeddings, methods such as hard negative mining are employed, where the negative samples are chosen such that

$$d_{an} = \min_{n \in \mathcal{N}_a} \|f_{\theta}(x_n) - f_{\theta}(x_n)\|_2. \qquad (12.28)$$

This strategy ensures that the network focuses on difficult examples that are closer to the anchor, making learning more effective. To generalize beyond triplet-based losses, one can also consider the N-pair loss, which extends the triplet loss by incorporating multiple negatives:

$$\mathcal{L}_{\text{N-pair}} = \sum_{i=1}^{N} \log \left( 1 + \sum_{j \neq i} \exp(\|f_{\theta}(x_i) - f_{\theta}(x_j)\|_2^2 - \|f_{\theta}(x_i) - f_{\theta}(x_i^+)\|_2^2) \right), \qquad (12.29)$$

where $x_i^+$ is a positive sample corresponding to $x_i$. An alternative formulation is Proxy-based Deep Metric Learning, where trainable class proxies $p_c$ are introduced for each class c, and each input is encouraged to be close to its respective proxy while remaining far from proxies of other classes. The softmax-based proxy loss is

$$\mathcal{L}_{\text{proxy}} = - \sum_{i=1}^{N} \log \frac{\exp(-\|f_{\theta}(x_i) - p_{y_i}\|_2^2)}{\sum_{c} \exp(-\|f_{\theta}(x_i) - p_c\|_2^2)}. \qquad (12.30)$$

Another recent improvement in deep metric learning involves contrastive learning-based methods, where positive and negative pairs are dynamically generated using a memory bank or momentum encoder. The InfoNCE loss, a contrastive loss derived from mutual information maximization, is

$$\mathcal{L}_{\text{InfoNCE}} = -\mathbb{E} \left[ \log \frac{\exp(f_{\theta}(x_i) \cdot f_{\theta}(x_i^+)/\tau)}{\sum_j \exp(f_{\theta}(x_i) \cdot f_{\theta}(x_j^-)/\tau)} \right], \qquad (12.31)$$

where $\tau$ is a temperature parameter that controls the distribution of distances. Furthermore, a probabilistic perspective can be adopted, where embeddings are modeled as distributions rather than point estimates. This leads to the deep variational metric learning approach, where embeddings are represented as Gaussian distributions $q_{\theta}(z|x)$, and the distance metric is defined using the Kullback-Leibler divergence

$$D_{\text{KL}}(q_{\theta}(z|x_i) \| q_{\theta}(z|x_j)). \qquad (12.32)$$

The corresponding loss function is formulated as

$$\mathcal{L}_{\text{variational}} = \sum_{i,j} y_{ij} D_{\text{KL}}(q_{\theta}(z|x_i) \| q_{\theta}(z|x_j)) + (1 - y_{ij}) \max(0, m - D_{\text{KL}}(q_{\theta}(z|x_i) \| q_{\theta}(z|x_j))). \qquad (12.33)$$

Finally, the quality of the learned embeddings can be evaluated using retrieval-based metrics such as mean average precision (mAP), recall at k (R@k), and normalized mutual information (NMI), ensuring that the learned distance metric effectively captures semantic similarity.

## 12.4 Normalized temperature-scaled cross-entropy loss (NT-Xent) approach to Metric Learning

The Normalized Temperature-scaled Cross-Entropy Loss (NT-Xent) is a fundamental loss function in contrastive learning, particularly in self-supervised representation learning. The NT-Xent loss is designed to maximize agreement between similar (positive) pairs while pushing dissimilar (negative) pairs apart in the learned feature space. Mathematically, it builds upon softmax-based similarity scoring and temperature scaling, ensuring a well-calibrated probabilistic interpretation of similarity.
Given a batch of N samples, let each data point $x_i$ be augmented to obtain two correlated views, denoted as $x'_i$ and $x''_i$. These views are encoded into the feature space using a learned function $f_\theta(\cdot)$, which is typically a deep neural network. The feature representations of the two augmented views are given by

$$z'_i = f_\theta(x'_i), \quad z''_i = f_\theta(x''_i) \qquad (12.34)$$

where the output embeddings are L2-normalized to lie on a unit hypersphere:

$$\|z'_i\| = 1, \quad \|z''_i\| = 1. \qquad (12.35)$$

The similarity between two representations is computed using the cosine similarity, which is given by

$$\mathrm{sim}(z_i, z_j) = \frac{z_i \cdot z_j}{\|z_i\| \|z_j\|} = z_i^\top z_j. \qquad (12.36)$$

Since $z_i$ and $z_j$ are unit vectors, their dot product is directly the cosine of the angle between them. To define the NT-Xent loss, a temperature scaling parameter $\tau$ is introduced to control the sharpness of the similarity distribution. The temperature-scaled similarity is given by

$$s_{ij} = \frac{z_i^\top z_j}{\tau}. \qquad (12.37)$$

This scaling prevents collapse by adjusting the entropy of the softmax distribution. The probability of $z'_i$ matching with $z''_i$ (its positive counterpart) over all possible samples in the batch is computed using the softmax function

$$p_{i,i''} = \frac{\exp(s_{i,i''})}{\sum_{j \neq i} \exp(s_{i,j})}. \qquad (12.38)$$

The NT-Xent loss for a single positive pair $(z'_i, z''_i)$ is then formulated as

$$\ell_i = -\log p_{i,i''} = -\log \frac{\exp(s_{i,i''})}{\sum_{j \neq i} \exp(s_{i,j})}. \qquad (12.39)$$

To compute the full batch-wise NT-Xent loss, we sum over all samples and their corresponding augmentations, leading to

$$\mathcal{L} = \frac{1}{2N} \sum_{i=1}^{N} (\ell_i + \ell_{i''}) = -\frac{1}{2N} \sum_{i=1}^{N} \left( \log \frac{\exp(s_{i,i''})}{\sum_{j \neq i} \exp(s_{i,j})} + \log \frac{\exp(s_{i'',i})}{\sum_{j \neq i} \exp(s_{i'',j})} \right). \qquad (12.40)$$

The denominator in the softmax function acts as a contrastive term, where all other embeddings in the batch contribute as negative samples. Since the numerator contains only one positive pair, the loss forces these representations to be closer, while the denominator encourages separation from all other samples. A key insight into NT-Xent is its connection to InfoNCE loss (used in contrastive predictive coding), but it is fully symmetric, meaning both views of the same instance contribute equally. This symmetry helps in learning invariant representations by reinforcing consistent feature extraction across different augmentations. To ensure proper gradient flow, the temperature $\tau$ is crucial. If $\tau$ is too high, the softmax distribution becomes too uniform, leading to weak discrimination. Conversely, if $\tau$ is too low, the model learns overly sharp distributions, which can lead to overfitting. The gradient of NT-Xent loss with respect to the embedding $z_i$ can be derived using the softmax gradient formula

$$\frac{\partial \mathcal{L}}{\partial z_i} = \sum_j p_{i,j}(z_j - z_i). \qquad (12.41)$$

This update step ensures that positive pairs $(z_i, z_{i''})$ get pulled together while negative pairs are pushed apart in proportion to their softmax probabilities. In the large batch limit, NT-Xent can be approximated by Monte Carlo sampling of negative pairs, ensuring computational efficiency in practical self-supervised learning frameworks such as SimCLR. Thus, the NT-Xent loss serves as a fundamental tool for learning discriminative, invariant, and well-clustered embeddings in self-supervised contrastive learning frameworks.
# 13 Adversial Learning

Adversarial learning is a fundamental paradigm in machine learning where two models are trained in opposition to each other, leading to the generation of robust representations and improvements in generalization. The central concept underlying adversarial learning can be mathematically formulated in terms of a minimax optimization problem, where one model seeks to maximize an objective function while the other aims to minimize it. Formally, given a function $f(\theta)$ parameterized by $\theta$, adversarial learning can be framed as

$$ \min_{\theta} \max_{\delta} \mathcal{L}(f(\theta, x + \delta), y) \qquad (13.1) $$

where $\mathcal{L}$ is the loss function, $x$ is the input, $y$ is the ground truth label, and $\delta$ represents an adversarial perturbation optimized to maximize the loss. The adversarial perturbation $\delta$ is constrained within a perturbation bound $\epsilon$, ensuring that the perturbed example $x + \delta$ remains within a small neighborhood of $x$ according to a given norm constraint:

$$ \|\delta\|_p \le \epsilon \qquad (13.2) $$

where $\| \cdot \|_p$ represents the $p$-norm. The most common choice is the $\ell_\infty$-norm, which results in perturbations constrained as

$$ \|\delta\|_{\infty} \le \epsilon. \qquad (13.3) $$

A fundamental technique in adversarial learning is the generation of adversarial examples using gradient-based methods. The Fast Gradient Sign Method (FGSM) computes adversarial perturbations using the sign of the gradient of the loss function with respect to the input:

$$ \delta = \epsilon \cdot \mathrm{sign}(\nabla_x \mathcal{L}(f(\theta, x), y)). \qquad (13.4) $$

This results in the adversarial example:

$$ x' = x + \delta = x + \epsilon \cdot \mathrm{sign}(\nabla_x \mathcal{L}(f(\theta, x), y)). \qquad (13.5) $$

A more refined approach is the Projected Gradient Descent (PGD) method, which iteratively updates the adversarial example using gradient ascent:

$$ x_{t+1} = \Pi_{B_{\epsilon}(x)} (x_t + \alpha \cdot \mathrm{sign}(\nabla_x \mathcal{L}(f(\theta, x_t), y))), \qquad (13.6) $$

where $\Pi_{B_{\epsilon}(x)}$ denotes projection onto the $\ell_\infty$-ball of radius $\epsilon$ around $x$, and $\alpha$ is the step size. The adversarial training process modifies the learning objective by incorporating adversarially generated examples into the training procedure, leading to the following robust optimization problem:

$$ \min_{\theta} \mathbb{E}_{(x,y) \sim \mathcal{D}} \left[ \max_{\|\delta\|_p \le \epsilon} \mathcal{L}(f(\theta, x + \delta), y) \right], \qquad (13.7) $$

where $\mathcal{D}$ represents the data distribution. This formulation ensures that the model is trained on worst-case perturbations, improving robustness against adversarial attacks. A generative approach to adversarial learning is encapsulated in Generative Adversarial Networks (GANs), where a generator $G$ and a discriminator $D$ are trained in a two-player minimax game:

$$ \min_{G} \max_{D} \mathbb{E}_{x \sim p_{\mathrm{data}}(x)} [\log D(x)] + \mathbb{E}_{z \sim p_{z}(z)} [\log(1 - D(G(z)))] . \qquad (13.8) $$
Here, the generator $G$ aims to produce samples $G(z)$ that are indistinguishable from real samples, while the discriminator $D$ seeks to correctly classify real and generated samples. The optimal discriminator is given by

$$D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_G(x)}. \qquad (13.9)$$

The equilibrium of the game is attained when $p_G(x) = p_{\text{data}}(x)$, leading to the global minimum of the Jensen-Shannon divergence between real and generated distributions. Variants of adversarial learning include Wasserstein GANs (WGANs), which replace the standard GAN loss with the Wasserstein distance:

$$\min_{G} \max_{\|D\|_{L} \le 1} \mathbb{E}_{x \sim p_{\text{data}}}[D(x)] - \mathbb{E}_{z \sim p_{z}}[D(G(z))], \qquad (13.10)$$

where $D$ is constrained to be a 1-Lipschitz function. The Wasserstein distance provides a more stable training procedure by mitigating mode collapse. In reinforcement learning, adversarial learning is utilized in adversarial policy training, where an agent maximizes a reward function while an adversary perturbs the environment:

$$\max_{\pi} \min_{\mathcal{A}} \mathbb{E}_{\tau \sim P(\tau|\pi, \mathcal{A})} [R(\tau)]. \qquad (13.11)$$

Here, $\pi$ is the policy, $\mathcal{A}$ represents adversarial perturbations, and $R(\tau)$ is the reward function. Mathematically rigorous guarantees for adversarial robustness involve formulating certified defenses, which provide provable bounds on a model's robustness. For example, randomized smoothing transforms a classifier $f(x)$ into a smoothed classifier:

$$g(x) = \mathbb{E}_{\eta \sim \mathcal{N}(0, \sigma^2 I)}[f(x + \eta)]. \qquad (13.12)$$

A theoretical guarantee on robustness is given by

$$\mathbb{P}[g(x) = g(x + \delta)] \ge 1 - 2 \exp \left( - \frac{\|\delta\|_2^2}{2\sigma^2} \right). \qquad (13.13)$$

These mathematical formulations illustrate the depth of adversarial learning, emphasizing its fundamental role in improving model robustness and stability.

## 13.1 Fast Gradient Sign Method (FGSM) in Adversial Learning

The Fast Gradient Sign Method (FGSM) is a crucial technique in adversarial learning, a domain that investigates the vulnerability of deep learning models to adversarial perturbations. Formally, given a neural network classifier $f(\mathbf{x}; \boldsymbol{\theta})$ parameterized by $\boldsymbol{\theta}$, where $\mathbf{x} \in \mathbb{R}^d$ is an input and $y$ is the corresponding true label, the objective of FGSM is to generate an adversarial example $\mathbf{x}_{\text{adv}}$ by perturbing $\mathbf{x}$ along the gradient direction that maximizes the classification loss. The adversarial perturbation is computed using the gradient of the loss function $J(\boldsymbol{\theta}, \mathbf{x}, y)$ with respect to the input $\mathbf{x}$, given by

$$\mathbf{g} = \nabla_{\mathbf{x}} J(\boldsymbol{\theta}, \mathbf{x}, y). \qquad (13.14)$$

FGSM constructs an adversarial input by taking the sign of this gradient and scaling it by a perturbation magnitude $\epsilon$, leading to the perturbed input

$$\mathbf{x}_{\text{adv}} = \mathbf{x} + \epsilon \text{sign}(\mathbf{g}). \qquad (13.15)$$

Mathematically, this attack is formulated as an optimization problem where the goal is to maximize the loss function subject to a bounded perturbation constraint:

$$\mathbf{x}_{\text{adv}} = \arg \max_{\mathbf{x}'} J(\boldsymbol{\theta}, \mathbf{x}', y) \quad \text{subject to} \quad \|\mathbf{x}' - \mathbf{x}\|_{\infty} \le \epsilon. \qquad (13.16)$$
Since computing the exact solution to this optimization problem is computationally expensive, FGSM employs a first-order approximation using Taylor series expansion:

$$J(\theta, \mathbf{x} + \boldsymbol{\delta}, y) \approx J(\theta, \mathbf{x}, y) + \boldsymbol{\delta}^T \nabla_{\mathbf{x}} J(\theta, \mathbf{x}, y). \quad (13.17)$$

Maximizing this expression with respect to $\boldsymbol{\delta}$ under the $\ell_\infty$-norm constraint $\|\boldsymbol{\delta}\|_\infty \le \epsilon$ leads to the optimal perturbation:

$$\boldsymbol{\delta}^* = \epsilon \operatorname{sign}(\nabla_{\mathbf{x}} J(\theta, \mathbf{x}, y)). \quad (13.18)$$

Substituting this into the perturbed input yields the FGSM formulation:

$$\mathbf{x}_{\text{adv}} = \mathbf{x} + \epsilon \operatorname{sign}(\nabla_{\mathbf{x}} J(\theta, \mathbf{x}, y)). \quad (13.19)$$

FGSM exploits the local linearity property of neural networks, which can be seen by considering a first-order approximation of the neural network output near $\mathbf{x}$:

$$f(\mathbf{x} + \boldsymbol{\delta}; \boldsymbol{\theta}) \approx f(\mathbf{x}; \boldsymbol{\theta}) + \nabla_{\mathbf{x}} f(\mathbf{x}; \boldsymbol{\theta})^T \boldsymbol{\delta}. \quad (13.20)$$

Given that neural networks often have high-dimensional input spaces, even a small perturbation aligned with the gradient can induce significant changes in the output, leading to misclassification. If the network assigns class probabilities using a softmax function

$$P(y | \mathbf{x}; \boldsymbol{\theta}) = \frac{\exp(f_y(\mathbf{x}; \boldsymbol{\theta}))}{\sum_j \exp(f_j(\mathbf{x}; \boldsymbol{\theta}))}, \quad (13.21)$$

then the FGSM attack perturbs $\mathbf{x}$ in a way that increases the probability of an incorrect class $y'$, where

$$P(y' | \mathbf{x}_{\text{adv}}; \boldsymbol{\theta}) > P(y | \mathbf{x}_{\text{adv}}; \boldsymbol{\theta}). \quad (13.22)$$

The effectiveness of FGSM depends on $\epsilon$. A small $\epsilon$ may not cause misclassification, whereas a large $\epsilon$ may introduce perceptible distortions. The impact of FGSM is often analyzed using the decision boundary properties of neural networks. Consider a linear classifier with decision boundary defined by

$$\mathbf{w}^T \mathbf{x} + b = 0. \quad (13.23)$$

Applying FGSM perturbs $\mathbf{x}$ along the gradient direction, which in a linear setting is simply

$$\mathbf{x}_{\text{adv}} = \mathbf{x} + \epsilon \operatorname{sign}(\mathbf{w}). \quad (13.24)$$

If $\mathbf{x}$ is initially correctly classified, then after perturbation, the new decision function evaluation is

$$\mathbf{w}^T(\mathbf{x} + \epsilon \operatorname{sign}(\mathbf{w})) + b = \mathbf{w}^T(\mathbf{x} + b + \epsilon \|\mathbf{w}\|_1). \quad (13.25)$$

If the perturbation shifts this beyond zero, the classification flips. This explains why FGSM can be highly effective, even for small $\epsilon$, particularly in high-dimensional spaces where each small perturbation accumulates across dimensions. A key property of FGSM is its transferability across different models. Given two classifiers $f_1(\mathbf{x}; \boldsymbol{\theta}_1)$ and $f_2(\mathbf{x}; \boldsymbol{\theta}_2)$, adversarial examples crafted for $f_1$ often remain adversarial for $f_2$, which can be analyzed using the gradient similarity measure:

$$\mathbb{E}[\nabla_{\mathbf{x}} J_1(\boldsymbol{\theta}_1, \mathbf{x}, y) \cdot \nabla_{\mathbf{x}} J_2(\boldsymbol{\theta}_2, \mathbf{x}, y)] > 0. \quad (13.26)$$

This suggests that gradients of different models are often aligned, leading to the observed transferability phenomenon. FGSM can be countered using adversarial training, where the training objective is modified to incorporate adversarial examples:

$$\min_{\boldsymbol{\theta}} \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{D}} \left[ \max_{\|\boldsymbol{\delta}\|_{\infty} \le \epsilon} J(\boldsymbol{\theta}, \mathbf{x} + \boldsymbol{\delta}, y) \right]. \quad (13.27)$$
This formulation leads to robust classifiers that are less sensitive to adversarial perturbations. However, adversarial training increases computational costs, as it requires solving an inner maximization problem during training. Another defense mechanism is gradient masking, where modifications to the loss function result in non-informative gradients:

$$ \tilde{J}(\theta, \mathbf{x}, y) = J(\theta, \mathbf{x}, y) + \lambda \|\nabla_{\mathbf{x}} J(\theta, \mathbf{x}, y)\|^2. \quad (13.28) $$

While this suppresses adversarial perturbations, it does not fundamentally resolve the adversarial vulnerability, as adaptive attacks can circumvent such defenses by using alternative optimization techniques.

## 13.2 Projected Gradient Descent (PGD) method in Adversial Learning

Projected Gradient Descent (PGD) is a highly rigorous iterative optimization method employed in adversarial learning to generate adversarial examples that maximize the loss function of a given classifier while ensuring that the perturbation remains within a specified constraint set. The mathematical formulation of PGD follows directly from constrained optimization, where an adversarial perturbation $\delta$ is sought such that a perturbed input $\mathbf{x} + \delta$ induces a maximal classification error while remaining within a bounded perturbation set $S$, often defined by an $l_p$-norm ball. The adversarial attack is thus formalized as

$$ \max_{\delta \in S} \mathcal{L}(\mathbf{x} + \delta, y; \theta) \quad (13.29) $$

where $\mathcal{L}(\mathbf{x}, y; \theta)$ denotes the loss function (e.g., cross-entropy loss), $\mathbf{x}$ is the input sample, $y$ is the corresponding label, and $\theta$ represents the parameters of the model. The perturbation set $S$ is commonly chosen as the $l_p$-ball of radius $\epsilon$, given by

$$ S = \{ \delta \in \mathbb{R}^d \mid \|\delta\|_p \le \epsilon \}. \quad (13.30) $$

To solve the constrained optimization problem, PGD proceeds iteratively by updating $\delta$ in the direction of the gradient of $\mathcal{L}$ with respect to the input, followed by a projection step to enforce the constraint. Given a step size $\alpha$, the update rule for PGD at iteration $t$ is

$$ \delta^{(t+1)} = \Pi_S \left( \delta^{(t)} + \alpha \frac{\nabla_{\mathbf{x}} \mathcal{L}(\mathbf{x} + \delta^{(t)}, y; \theta)}{\| \nabla_{\mathbf{x}} \mathcal{L}(\mathbf{x} + \delta^{(t)}, y; \theta) \|_p} \right), \quad (13.31) $$

where $\Pi_S(\cdot)$ denotes the projection operator that ensures $\delta^{(t+1)}$ remains within $S$. The projection depends on the choice of the $l_p$-norm constraint. For the commonly used $l_\infty$-ball, the projection is simply

$$ \Pi_S(\delta) = \max(-\epsilon, \min(\delta, \epsilon)), \quad (13.32) $$

which clips each component of $\delta$ to lie within $[-\epsilon, \epsilon]$. For the $l_2$-ball, projection is achieved by normalizing and scaling the perturbation as

$$ \Pi_S(\delta) = \epsilon \frac{\delta}{\|\delta\|_2} \quad \text{if } \|\delta\|_2 > \epsilon. \quad (13.33) $$

The iterative application of this process ensures that the perturbation maximally increases the loss while adhering to the predefined perturbation budget. In the limit as the step size $\alpha \to 0$ and the number of iterations $T \to \infty$, PGD approximates the optimal solution of the constrained
maximization problem. This is in contrast to the Fast Gradient Sign Method (FGSM), which
performs only a single-step update of the form

$$
\delta = \epsilon \cdot \mathrm{sign}(\nabla_x \mathcal{L}(\mathbf{x}, y; \boldsymbol{\theta})). \tag{13.34}
$$

PGD is therefore considered a stronger attack, as it allows the adversarial perturbation to explore the loss landscape more effectively by iteratively refining the adversarial example. From a theoretical perspective, PGD can be interpreted as an approximate projected gradient ascent method for solving the constrained maximization problem. Formally, this corresponds to a Lagrangian formulation where the Karush-Kuhn-Tucker (KKT) conditions dictate the optimal perturbation:

$$
\nabla_{\delta} \mathcal{L}(\mathbf{x} + \delta, y; \theta) - \lambda \nabla_{\delta} g(\delta) = 0, \quad (13.35)
$$

where g(δ) defines the boundary of the perturbation set. The iterative updates in PGD approximate this equilibrium condition numerically. Moreover, when applied iteratively with different random initializations of δ⁽⁰⁾, PGD can better explore non-convex loss surfaces, making it an effective strategy against adversarially trained defenses.

13.3 Generative approach in Adversarial Learning

The generative approach in adversarial learning is fundamentally rooted in the interplay between
two networks: the generator $G$ and the discriminator $D$, which are trained in an adversarial man-
ner. The generator aims to synthesize data that is indistinguishable from real data, while the
discriminator attempts to distinguish real data from generated data. This setup can be rigorously
modeled as a minimax optimization problem, given by

$$
\min_G \max_D \mathbb{E}_{\mathbf{x} \sim p_{\text{data}}} [\log D(\mathbf{x})] + \mathbb{E}_{\mathbf{z} \sim p_{\mathbf{z}}} [\log(1 - D(G(\mathbf{z})))] \quad (13.36)
$$

where $p_{\text{data}}$ represents the true data distribution, and $p_{\mathbf{z}}$ is the prior distribution from which latent variables $\mathbf{z}$ are drawn. The generator learns a mapping $G: \mathbf{z} \to \mathbf{x}$, parameterized by $\theta_G$, such that the distribution of generated samples $p_G(\mathbf{x})$ approximates $p_{\text{data}}$. The discriminator, parameterized by $\theta_D$, learns a function $D: \mathbf{x} \to [0, 1]$, which estimates the probability that $\mathbf{x}$ is drawn from $p_{\text{data}}$. The training of the generator and discriminator proceeds iteratively, where $D$ is updated by maximizing

$$
J_D(\theta_D) = \mathbb{E}_{\mathbf{x} \sim p_{\text{data}}} [\log D(\mathbf{x})] + \mathbb{E}_{\mathbf{z} \sim p_{\mathbf{z}}} [\log(1 - D(G(\mathbf{z})))] \quad (13.37)
$$

and $G$ is updated by minimizing

$$
J_G(\theta_G) = \mathbb{E}_{\mathbf{z} \sim p_{\mathbf{z}}} [\log(1 - D(G(\mathbf{z})))] \quad (13.38)
$$

A fundamental result in adversarial learning is that, given sufficient capacity, the optimal discrim-
inator is

$$
D^*(\mathbf{x}) = \frac{p_{\text{data}}(\mathbf{x})}{p_{\text{data}}(\mathbf{x}) + p_G(\mathbf{x})} \qquad (13.39)
$$

Substituting $D^*$ into the minimax objective leads to the Jensen-Shannon divergence between $p_{\text{data}}$ and $p_G$, given by

$$
\text{JSD}(p_{\text{data}} \parallel p_G) = \frac{1}{2} D_{\text{KL}}(p_{\text{data}} \parallel M) + \frac{1}{2} D_{\text{KL}}(p_G \parallel M) \quad (13.40)
$$

where

$$
M = \frac{1}{2}(p_{\text{data}} + p_G) \tag{13.41}
$$

and $D_{\text{KL}}(P \parallel Q)$ denotes the Kullback-Leibler divergence

$$
D_{\text{KL}}(P \parallel Q) = \int P(\mathbf{x}) \log \frac{P(\mathbf{x})}{Q(\mathbf{x})} d\mathbf{x} \qquad (13.42)
$$
In practical settings, this divergence-based formulation leads to issues such as vanishing gradients when $p_G$ and $p_{\text{data}}$ do not overlap significantly. To address this, alternative formulations such as Wasserstein GANs (WGANs) optimize the Wasserstein distance

$$W(p_{\text{data}}, p_G) = \sup_{\|f\|_L \le 1} \mathbb{E}_{\mathbf{x} \sim p_{\text{data}}} [f(\mathbf{x})] - \mathbb{E}_{\mathbf{x} \sim p_G} [f(\mathbf{x})] \quad (13.43)$$

where $f$ is constrained to be 1-Lipschitz. The Wasserstein distance provides meaningful gradients even when $p_G$ and $p_{\text{data}}$ have disjoint support. The optimal transport formulation of the Wasserstein distance can be rewritten as

$$W(p_{\text{data}}, p_G) = \inf_{\gamma \in \Pi(p_{\text{data}}, p_G)} \mathbb{E}_{(\mathbf{x}, \mathbf{y}) \sim \gamma} [\|\mathbf{x} - \mathbf{y}\|] \quad (13.44)$$

where $\Pi(p_{\text{data}}, p_G)$ denotes the set of joint distributions with marginals $p_{\text{data}}$ and $p_G$. A crucial extension of adversarial learning is its application to conditional generation, where the generator learns a conditional distribution $p_G(\mathbf{x} | \mathbf{y})$. The objective function for a conditional GAN is given by

$$\min_G \max_D \mathbb{E}_{(\mathbf{x}, \mathbf{y}) \sim p_{\text{data}}} [\log D(\mathbf{x}, \mathbf{y})] + \mathbb{E}_{\mathbf{z} \sim p_{\text{z}}, \mathbf{y} \sim p_{\text{data}}} [\log(1 - D(G(\mathbf{z}, \mathbf{y}), \mathbf{y}))] \quad (13.45)$$

From a probabilistic standpoint, generative adversarial networks can be interpreted within the framework of energy-based models, where the discriminator implicitly defines an energy function

$$E(\mathbf{x}) = - \log D(\mathbf{x}) \quad (13.46)$$

Minimizing this energy aligns generated samples with the modes of the true distribution, reinforcing the generative modeling capability of adversarial learning.

## 13.4 Interpreting Generative Adversarial Networks within the framework of Energy-Based models

Generative Adversarial Networks (GANs) can be interpreted probabilistically within the framework of Energy-Based Models (EBMs) by considering their fundamental objective as a minimax optimization problem that implicitly defines an energy function governing the probability distribution of data and generated samples. The generator $G$ and the discriminator $D$ interact in a two-player game, which can be rigorously formulated in the language of probabilistic energy-based models. The discriminator learns to distinguish real data samples from generated ones, while the generator learns to produce samples that minimize the discriminator's ability to differentiate them.

To formally understand this, let $p_{\text{data}}(\mathbf{x})$ be the true data distribution and $p_G(\mathbf{x})$ be the generator's induced distribution. The discriminator is parameterized by $D_{\theta}(\mathbf{x})$, where $D_{\theta}(\mathbf{x})$ outputs the probability that $\mathbf{x}$ is a real sample. The optimal discriminator in the standard GAN formulation is derived as:

$$D^*(\mathbf{x}) = \frac{p_{\text{data}}(\mathbf{x})}{p_{\text{data}}(\mathbf{x}) + p_G(\mathbf{x})} \quad (13.47)$$

which emerges from the minimization of the following objective:

$$\mathbb{E}_{\mathbf{x} \sim p_{\text{data}}} [\log D(\mathbf{x})] + \mathbb{E}_{\mathbf{x} \sim p_G} [\log(1 - D(\mathbf{x}))] \quad (13.48)$$

In the energy-based interpretation, we introduce an energy function $E(\mathbf{x})$ that models the relative likelihood of a sample. The energy function is linked to probability distributions via the Gibbs measure:

$$p(\mathbf{x}) = \frac{e^{-E(\mathbf{x})}}{Z} \quad (13.49)$$
where Z is the partition function:

$$Z = \int e^{-E(\mathbf{x})} d\mathbf{x} \qquad (13.50)$$

We can define an energy-based discriminator as:

$$D_{\theta}(\mathbf{x}) = \sigma(-E_{\theta}(\mathbf{x})) = \frac{1}{1 + e^{E_{\theta}(\mathbf{x})}} \qquad (13.51)$$

where $\sigma(x) = \frac{1}{1+e^{-x}}$ is the sigmoid function. This formulation implies that minimizing the GAN objective is equivalent to learning an energy function $E_{\theta}(\mathbf{x})$ such that:

$$E_{\theta}(\mathbf{x}) \approx -\log \frac{p_{\text{data}}(\mathbf{x})}{p_{G}(\mathbf{x})} \qquad (13.52)$$

which is precisely the log-ratio of the real and generated distributions, making GANs inherently an instance of energy-based models. From a probabilistic standpoint, the generator learns to minimize the divergence between $p_G(\mathbf{x})$ and $p_{\text{data}}(\mathbf{x})$. Specifically, for a fixed discriminator, the generator's objective can be rewritten as minimizing the Jensen-Shannon divergence:

$$D_{\text{JS}}(p_{\text{data}} \parallel p_G) = \frac{1}{2} D_{\text{KL}}(p_{\text{data}} \parallel M) + \frac{1}{2} D_{\text{KL}}(p_G \parallel M) \qquad (13.53)$$

where $M = \frac{1}{2}(p_{\text{data}} + p_G)$. This reveals that GANs operate within an information-theoretic framework where they reduce an energy-based measure of dissimilarity between real and generated samples. The training dynamics can be interpreted through contrastive divergence, which appears naturally in energy-based models. The generator updates its parameters to produce samples that have lower energy, thereby implicitly performing a form of Markov Chain Monte Carlo (MCMC)-like sampling in the energy landscape defined by $E_{\theta}(\mathbf{x})$. The gradient update of the generator is:

$$\nabla_{\theta_G} \mathbb{E}_{\mathbf{z} \sim p_G} [\log(1 - D(G(\mathbf{z})))] = -\mathbb{E}_{\mathbf{z} \sim p_G} [\nabla_{\theta_G} E_{\theta}(G(\mathbf{z}))] \qquad (13.54)$$

which aligns with the contrastive divergence updates used in training energy-based models. This equivalence suggests that GANs effectively perform energy minimization by dynamically adjusting G such that its samples reduce the energy discrepancy between real and generated distributions. Furthermore, considering a continuous-time stochastic formulation, the discriminator can be reinterpreted using a Langevin-type evolution:

$$\frac{d\mathbf{x}}{dt} = -\nabla_{\mathbf{x}} E_{\theta}(\mathbf{x}) + \sqrt{2\beta^{-1}} \eta(t) \qquad (13.55)$$

where $\eta(t)$ is a white noise process and $\beta$ controls the temperature of the energy model. This perspective suggests that adversarial training implicitly simulates diffusion in an energy-based potential field, refining the generator's distribution $p_G$ over time. Thus, from a probabilistic viewpoint, generative adversarial networks fundamentally operate as an energy-based model where the discriminator estimates an energy function that guides the generator toward producing samples that match the statistics of real data. This establishes GANs as a probabilistic framework akin to energy minimization methods such as Boltzmann Machines, where adversarial training serves as an implicit mechanism for defining an energy landscape in the data space.
# 14 Casual Inference in Deep Neural Networks

Causal inference in deep neural networks involves the rigorous mathematical study of cause-and-effect relationships within high-dimensional representations learned by deep models. Unlike traditional statistical correlation, causal inference seeks to understand how perturbations in input variables propagate through the model to affect outcomes, disentangling spurious correlations from true causation. Given a set of random variables X, Y, and Z, a fundamental question in causal inference is whether X causally influences Y or whether their observed association is due to a confounding factor Z. The structural causal model (SCM) framework formalizes this using a set of structural equations:

$$Y = f_Y(X, U_Y) \qquad (14.1)$$

$$X = f_X(Z, U_X) \qquad (14.2)$$

where $f_Y$ and $f_X$ are deterministic functions encoding the causal mechanisms, and $U_Y$ and $U_X$ are exogenous noise variables, assumed to be mutually independent. The central problem in causal inference is identifying causal effects through interventions, denoted as $do(X = x)$, which replace the original structural equation of X with a fixed value x, leading to a new distribution:

$$P(Y|do(X = x)) = \sum_z P(Y|X = x, Z = z)P(Z) \qquad (14.3)$$

This contrasts with traditional conditional probabilities $P(Y|X)$, which do not eliminate confounding. In deep neural networks, causal inference is particularly challenging due to the high-dimensional, entangled nature of representations. Given a deep neural network parameterized by weights $\theta$, mapping an input X to an output Y through multiple layers:

$$H^{(l)} = \sigma(W^{(l)}H^{(l-1)} + b^{(l)}) \qquad (14.4)$$

where $H^{(l)}$ denotes the activations at layer l, $W^{(l)}$ and $b^{(l)}$ are the weight matrices and biases, and $\sigma$ is a nonlinear activation function, causal inference seeks to identify whether perturbations in X affect Y via a causal pathway. One approach to formalizing causality in deep networks is the use of causal feature selection, which attempts to identify a subset of features S such that:

$$P(Y|do(X_S = x_S)) = P(Y|X_S = x_S) \qquad (14.5)$$

for all $x_S$ in the support of $X_S$, ensuring that selected features S are causally relevant to Y. This can be achieved using conditional independence tests based on the back-door criterion:

$$P(Y|do(X)) = \sum_Z P(Y|X, Z)P(Z) \qquad (14.6)$$

where Z blocks all back-door paths between X and Y. In deep neural networks, this translates to modifying network architectures to impose structural constraints that enforce causal dependencies. A key tool in causal inference for deep learning is the concept of counterfactual reasoning. Given
an observed instance $X = x$ leading to output $Y = y$, a counterfactual query asks what would have happened had $X$ been different:

$$Y_{X \leftarrow x'} = f_Y(f_X^{-1}(x', U_X), U_Y) \qquad (14.7)$$

where $X \leftarrow x'$ denotes an intervention setting $X$ to $x'$, and $f_X^{-1}$ reconstructs the counterfactual world. Estimating counterfactuals in deep models requires generative approaches such as variational autoencoders (VAEs) and normalizing flows, which approximate the latent space $U$ from which counterfactual samples can be drawn. The fundamental theorem of causality states that if a probability distribution $P(X, Y)$ is generated from an SCM, then all counterfactual quantities are identifiable from the structural equations. Given a trained neural network with loss function $\mathcal{L}(\theta)$, causal constraints can be imposed by regularizing for causal invariance:

$$\mathcal{L}(\theta) + \lambda \sum_{i=1}^{n} |P(Y|X_i) - P(Y|do(X_i))| \qquad (14.8)$$

where $\lambda$ controls the strength of the causal penalty. This ensures that deep models learn representations aligned with causal mechanisms rather than spurious correlations. Another crucial aspect is the causal transportability of learned models, ensuring that a model trained in one environment generalizes to another with different distributions $P(X)$. This is characterized by domain adaptation techniques that minimize the divergence:

$$D(P(Y|do(X)), P'(Y|do(X))) \qquad (14.9)$$

where $P'$ is the distribution in the target domain, and $D$ is a divergence measure such as KL-divergence:

$$D_{\text{KL}}(P||Q) = \sum_{x} P(x) \log \frac{P(x)}{Q(x)} \qquad (14.10)$$

In deep networks, this is achieved using domain-invariant representations through adversarial training or contrastive learning. A critical issue in causal inference is the identifiability of causal effects from observational data. The fundamental problem of causal inference states that counterfactual outcomes cannot be directly observed:

$$Y_{X \leftarrow x} \neq Y_{X \leftarrow x'} \qquad (14.11)$$

for $x \neq x'$, meaning that estimating $P(Y|do(X))$ requires additional assumptions such as uncon- foundedness:

$$Y_{X \leftarrow x} \perp X|Z \qquad (14.12)$$

which allows estimation using propensity score matching:

$$P(Y|do(X)) = \sum_{Z} P(Y|X, Z)P(Z) \qquad (14.13)$$

Thus, deep causal inference seeks to disentangle causal structure within neural networks, ensuring that learned representations encode true causal effects rather than spurious correlations. This requires integrating structural causal models, counterfactual reasoning, domain adaptation, and invariant risk minimization, ensuring that deep models generalize to unseen domains while preserving causal mechanisms.

## 14.1 Structural Causal Model (SCM)

A **Structural Causal Model (SCM)** is a rigorous mathematical framework for representing and reasoning about causal relationships between variables. It formalizes causality using **structural**
equations and directed acyclic graphs (DAGs) to capture cause-and-effect mechanisms. Given a set of endogenous variables $V = \{V_1, V_2, ..., V_n\}$ and exogenous variables $U = \{U_1, U_2, ..., U_m\}$, an SCM is defined as a tuple $(V, U, F, P(U))$, where $F$ is a set of deterministic structural equations, and $P(U)$ is a probability distribution over exogenous variables. Each endogenous variable $V_i$ is determined by a function of its parents in the causal graph and the corresponding exogenous variables:

$$V_i = f_i(PA_i, U_i) \qquad (14.14)$$

where $PA_i$ represents the set of parent variables of $V_i$ in the causal graph. The causal relationships between variables are represented using a directed acyclic graph $G = (V, E)$, where each edge $V_j \to V_i$ signifies that $V_j$ is a direct cause of $V_i$. The absence of an edge between $V_j$ and $V_i$ indicates conditional independence given other parents. The structural equations define the causal mechanism, and their functional form encodes counterfactual relationships. Given an intervention $do(X = x)$, where the variable $X$ is forcibly set to $x$, the new SCM modifies the equation for $X$ while keeping all other equations unchanged:

$$SCM_{do(X=x)} = (V, U, F^{X=x}, P(U)) \qquad (14.15)$$

where $F^{X=x}$ represents the modified set of structural equations with $X$ replaced by the constant $x$. Causal effects are quantified using interventional distributions. The post-intervention distribution of an outcome $Y$ given an intervention on $X$ is computed using the **causal effect formula**:

$$P(Y | do(X = x)) = \sum_{V \setminus \{X, Y\}} P(V \setminus \{X\} | X = x)P(X = x) \qquad (14.16)$$

which integrates out all non-intervened variables. The backdoor criterion provides a criterion for identifying $P(Y | do(X))$ when a set of variables $Z$ blocks all backdoor paths from $X$ to $Y$, leading to the adjustment formula:

$$P(Y | do(X)) = \sum_{Z} P(Y | X, Z)P(Z) \qquad (14.17)$$

Counterfactuals are evaluated by computing potential outcomes. Given an observed world where $X = x$ and $Y = y$, the counterfactual query $Y_{X=x'}$ seeks the value $Y$ would have taken had $X$ been set to $x'$. This is formalized using the **three-step counterfactual algorithm**:

1. **Abduction:** Infer exogenous variables using the observed data:

$$U = f_X^{-1}(X) \qquad (14.18)$$

2. **Action:** Modify the SCM by replacing $X$ with $x'$:

$$SCM_{do(X=x')} \qquad (14.19)$$

3. **Prediction:** Solve for $Y_{X=x'}$ in the modified SCM:

$$Y_{X=x'} = f_Y(PA_Y, U_Y) \qquad (14.20)$$

The probability of causation (PC) quantifies the likelihood that $X$ caused $Y$ by comparing the counterfactual and factual outcomes:

$$PC = P(Y_{X=1} = 1 | X = 0, Y = 0) \qquad (14.21)$$

which requires structural knowledge of the system. Identification of causal effects depends on graphical conditions such as the **front-door criterion**, which allows inference of $P(Y | do(X))$ even when confounding is unobserved, provided an intermediate variable $M$ satisfies:

$$P(Y | do(X)) = \sum_{m} P(Y | m)P(m | do(X)) \qquad (14.22)$$
SCMs generalize probabilistic models by encoding mechanistic relationships rather than mere sta- tistical associations. The key distinction is that SCMs allow answering counterfactual queries, distinguishing correlation from causation, and supporting robust decision-making under interven- tions.

## 14.2 Counterfactual reasoning in causal inference for deep neural networks

Counterfactual reasoning in causal inference for deep neural networks is fundamentally grounded in the potential outcomes framework. Suppose we define an observed data distribution as $\mathcal{D} = \{(\mathbf{x}_i, t_i, y_i)\}_{i=1}^N$, where $\mathbf{x}_i \in \mathbb{R}^d$ represents the input features, $t_i \in \{0, 1\}$ denotes the binary treatment assignment, and $y_i \in \mathbb{R}$ represents the observed outcome. In the potential outcomes framework, each unit $i$ has two possible outcomes: $Y_i(1)$ when treated and $Y_i(0)$ when untreated. The observed outcome is expressed as

$$Y_i = t_i Y_i(1) + (1 - t_i) Y_i(0). \qquad (14.23)$$

However, the counterfactual outcome, which is the outcome corresponding to the treatment level not received, remains unobserved. The fundamental problem of causal inference lies in estimating this counterfactual quantity. The treatment effect for an individual $i$ is given by

$$\tau_i = Y_i(1) - Y_i(0), \qquad (14.24)$$

but due to the missing counterfactual, it is not directly observable. The average treatment effect (ATE) is expressed as

$$\mathbb{E}[\tau] = \mathbb{E}[Y(1) - Y(0)], \qquad (14.25)$$

which requires estimation of both the factual and counterfactual expectations. One approach to estimating counterfactuals in deep neural networks is using representation learning to balance treatment groups. A neural network is trained to learn a latent representation $\Phi(\mathbf{x})$ such that the distributions of treated and untreated units become similar:

$$\Phi(\mathbf{x}) = f_{\theta}(\mathbf{x}), \qquad (14.26)$$

where $f_{\theta}$ is a deep neural network parameterized by $\theta$. The objective function often incorporates a domain discrepancy measure such as the Integral Probability Metric (IPM):

$$\mathcal{L}_{\text{IPM}} = \sup_{h \in \mathcal{H}} \left| \mathbb{E}_{p(\Phi(\mathbf{x})|T=1)}[h(\Phi(\mathbf{x}))] - \mathbb{E}_{p(\Phi(\mathbf{x})|T=0)}[h(\Phi(\mathbf{x}))] \right|. \qquad (14.27)$$

By minimizing this term, the network ensures that the latent distributions for treated and control groups are similar, facilitating counterfactual inference. A common loss function for training the neural network includes both factual prediction loss and a regularization term for domain alignment:

$$\mathcal{L} = \sum_{i=1}^{N} \ell(Y_i, \hat{Y}_i) + \lambda \mathcal{L}_{\text{IPM}}. \qquad (14.28)$$

A counterfactual prediction $\hat{Y}_i(1 - t_i)$ is obtained by passing $\mathbf{x}_i$ through the network trained on both factual and imputed counterfactual distributions:

$$\hat{Y}_i(1) = g_{\theta}(\Phi(\mathbf{x}_i), 1), \quad \hat{Y}_i(0) = g_{\theta}(\Phi(\mathbf{x}_i), 0), \qquad (14.29)$$

where $g_{\theta}$ is the output layer of the neural network parameterized by $\theta$. To improve counterfactual estimation, techniques such as adversarial balancing loss and domain adaptation methods may be incorporated:

$$\mathcal{L}_{\text{adv}} = \mathbb{E}_{p(\Phi(\mathbf{x})|T=1)}[\log D(\Phi(\mathbf{x}))] + \mathbb{E}_{p(\Phi(\mathbf{x})|T=0)}[\log(1 - D(\Phi(\mathbf{x})))]. \qquad (14.30)$$
Here, $D(\Phi(\mathbf{x}))$ is a discriminator network that tries to distinguish between treated and untreated units in latent space, and its adversarial training ensures treatment group similarity. Thus, counterfactual reasoning in deep neural networks is achieved by leveraging representation learning, adversarial domain adaptation, and balanced predictive modeling to estimate missing potential outcomes.

## 14.3 Domain adaptation in causal inference within deep neural networks

Domain adaptation in causal inference within deep neural networks is fundamentally concerned with the transfer of knowledge from a source domain $\mathcal{S}$ to a target domain $\mathcal{T}$, where the distributions of the input features and the causal mechanisms governing the relationships among variables differ between domains. Given an input space $\mathcal{X}$, a covariate space $\mathcal{C}$, an outcome space $\mathcal{Y}$, and a treatment space $\mathcal{A}$, we consider the problem of estimating the causal effect of treatment $A \in \mathcal{A}$ on outcome $Y \in \mathcal{Y}$ under domain shift. Specifically, let $P_S(X, A, Y)$ and $P_T(X, A, Y)$ denote the joint distributions in the source and target domains, respectively. The key assumption in causal domain adaptation is that while the distribution of covariates $P_S(X)$ and $P_T(X)$ may differ, the underlying structural causal model (SCM) governing the treatment and outcome relationship remains invariant across domains. Mathematically, the invariance of causal mechanisms implies that the conditional distributions of treatment assignment and outcome given the covariates are domain-independent:

$$P_S(A | X) = P_T(A | X), \quad P_S(Y | X, A) = P_T(Y | X, A). \qquad (14.31)$$

However, direct inference of the causal effect in the target domain is hindered by the discrepancy between the marginal distributions $P_S(X)$ and $P_T(X)$, a phenomenon known as covariate shift. This necessitates a reweighting strategy to correct for the distributional shift. A common approach involves the use of importance weights:

$$w(X) = \frac{P_T(X)}{P_S(X)}, \qquad (14.32)$$

which can be estimated via density ratio estimation techniques such as kernel mean matching or adversarial learning. Given these weights, the expectation of any function $f(X, A, Y)$ in the target domain can be approximated using observations from the source domain:

$$\mathbb{E}_{\mathcal{T}}[f(X, A, Y)] \approx \mathbb{E}_{\mathcal{S}}[w(X)f(X, A, Y)]. \qquad (14.33)$$

A deep neural network-based approach to causal domain adaptation typically parameterizes the outcome model $h_{\theta}(X, A)$ as a neural network with parameters $\theta$, trained to minimize the weighted empirical loss:

$$\mathcal{L}(\theta) = \sum_{i=1}^{n_S} w(X_i)\ell(h_{\theta}(X_i, A_i), Y_i), \qquad (14.34)$$

where $\ell(\cdot, \cdot)$ is a suitable loss function such as mean squared error for continuous outcomes or cross-entropy loss for binary outcomes. Additionally, domain-adversarial neural networks (DANNs) introduce a domain discriminator $D_{\phi}(X)$, parameterized by $\phi$, trained to distinguish between source and target samples while the feature extractor $g_{\theta}(X)$ is trained to minimize the discriminator's ability to differentiate domains, effectively promoting invariant feature representations:

$$\min_{\theta} \max_{\phi} \sum_{i=1}^{n_S} \log D_{\phi}(g_{\theta}(X_i)) + \sum_{j=1}^{n_T} \log(1 - D_{\phi}(g_{\theta}(X_j))). \qquad (14.35)$$