## 10.3 Feedforward Neural Networks

Feedforward neural networks (FNNs) are mathematical constructs designed to approximate arbitrary mappings $f: \mathbb{R}^n \to \mathbb{R}^m$ by composing affine transformations and nonlinear activation functions. At their core, these networks consist of $L$ layers, where each layer $k$ transforms its input $\vec{a}_{k-1} \in \mathbb{R}^{m_{k-1}}$ into an output $\vec{a}_k \in \mathbb{R}^{m_k}$ via the operation

$$ \vec{a}_k = f_k(W_k \vec{a}_{k-1} + \vec{b}_k). \qquad (10.23) $$

Here, $W_k \in \mathbb{R}^{m_k \times m_{k-1}}$ represents the weight matrix, $\vec{b}_k \in \mathbb{R}^{m_k}$ is the bias vector, and $f_k: \mathbb{R}^{m_k} \to \mathbb{R}^{m_k}$ is a component-wise activation function. Formally, if we denote the input layer as $\vec{a}_0 = \vec{x}$, the final output of the network, $\vec{y} \in \mathbb{R}^m$, is given by $\vec{a}_L = f_L(W_L \vec{a}_{L-1} + \vec{b}_L)$. Each transformation in this sequence can be described as $\vec{z}_k = W_k \vec{a}_{k-1} + \vec{b}_k$, followed by the activation $\vec{a}_k = f_k(\vec{z}_k)$. The affine transformation $\vec{z}_k = W_k \vec{a}_{k-1} + \vec{b}_k$ encapsulates the linear combination of inputs with weights $W_k$ and the addition of biases $\vec{b}_k$. For any two layers $k$ and $k+1$, the overall transformation can be represented by

$$ \vec{z}_{k+1} = W_{k+1}(W_k \vec{a}_{k-1} + \vec{b}_k) + \vec{b}_{k+1}. \qquad (10.24) $$

Expanding this, we have

$$ \vec{z}_{k+1} = W_{k+1}W_k \vec{a}_{k-1} + W_{k+1}\vec{b}_k + \vec{b}_{k+1}. \qquad (10.25) $$

Without the nonlinearity introduced by $f_k$, the network reduces to a single affine transformation

$$ \vec{y} = W \vec{x} + \vec{b}, \qquad (10.26) $$

where $W = W_L W_{L-1} \cdots W_1$ and

$$ \vec{b} = W_L W_{L-1} \cdots W_2 \vec{b}_1 + \cdots + \vec{b}_L. \qquad (10.27) $$

Thus, the incorporation of nonlinear activation functions is critical, as it enables the network to approximate non-linear mappings. Activation functions $f_k$ are applied element-wise to the pre-activation vector $\vec{z}_k$. The choice of activation significantly affects the network's behavior and training. For example, the sigmoid activation $f(x) = \frac{1}{1+e^{-x}}$ compresses inputs into the range (0, 1) and has a derivative given by

$$ f'(x) = f(x)(1 - f(x)). \qquad (10.28) $$

The hyperbolic tangent activation $f(x) = \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$ maps inputs to (-1, 1) with a derivative

$$ f'(x) = 1 - \tanh^2(x). \qquad (10.29) $$

The ReLU activation $f(x) = \max(0, x)$, commonly used in modern networks, has a derivative

$$ f'(x) = \begin{cases} 1 & x > 0, \\ 0 & x \le 0. \end{cases} \qquad (10.30) $$

These derivatives are essential for gradient-based optimization. The objective of training a feedforward neural network is to minimize a loss function $\mathcal{L}$, which measures the discrepancy between the predicted outputs $\vec{y}_i$ and the true targets $\vec{t}_i$ over a dataset $\{(\vec{x}_i, \vec{t}_i)\}_{i=1}^N$. For regression problems, the mean squared error (MSE) is often used, given by

$$ \mathcal{L} = \frac{1}{N} \sum_{i=1}^{N} \| \vec{y}_i - \vec{t}_i \|^2 = \frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{m} (y_{i,j} - t_{i,j})^2. \qquad (10.31) $$

In classification tasks, the cross-entropy loss is widely employed, defined as

$$ \mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{m} t_{i,j} \log(y_{i,j}), \qquad (10.32) $$
where $t_{i,j}$ represents the one-hot encoded labels. The gradient of $\mathcal{L}$ with respect to the network parameters is computed using backpropagation, which applies the chain rule iteratively to propagate errors from the output layer to the input layer. During backpropagation, the error signal at the output layer is computed as

$$ \delta_L = \frac{\partial \mathcal{L}}{\partial \vec{z}_L} = \nabla_{\vec{y}} \mathcal{L} \odot f'_L(\vec{z}_L), \qquad (10.33) $$

where $\odot$ denotes the Hadamard product. For hidden layers, the error signal propagates backward as

$$ \delta_k = (W_{k+1}^T \delta_{k+1}) \odot f'_k(\vec{z}_k). \qquad (10.34) $$

The gradients of the loss with respect to the weights and biases are then given by

$$ \frac{\partial \mathcal{L}}{\partial W_k} = \delta_k \vec{a}_{k-1}^T, \quad \frac{\partial \mathcal{L}}{\partial \vec{b}_k} = \delta_k. \qquad (10.35) $$

These gradients are used to update the parameters through optimization algorithms like stochastic gradient descent (SGD), where

$$ W_k \leftarrow W_k - \eta \frac{\partial \mathcal{L}}{\partial W_k}, \quad \vec{b}_k \leftarrow \vec{b}_k - \eta \frac{\partial \mathcal{L}}{\partial \vec{b}_k}, \qquad (10.36) $$

with $\eta > 0$ as the learning rate. The universal approximation theorem rigorously establishes that a feedforward neural network with at least one hidden layer and sufficiently many neurons can approximate any continuous function $f: \mathbb{R}^n \to \mathbb{R}^m$ on a compact domain $D \subset \mathbb{R}^n$. Specifically, for any $\epsilon > 0$, there exists a network $\hat{f}$ such that $\|f(\vec{x}) - \hat{f}(\vec{x})\| < \epsilon$ for all $\vec{x} \in D$. This expressive capability arises because the composition of affine transformations and nonlinear activations allows the network to approximate highly complex functions by partitioning the input space into regions and assigning different functional behaviors to each.

In summary, feedforward neural networks are a powerful mathematical framework grounded in linear algebra, calculus, and optimization. Their capacity to model intricate mappings between input and output spaces has made them a cornerstone of machine learning, with rigorous mathematical principles underpinning their structure and training. The combination of affine transformations, nonlinear activations, and gradient-based optimization enables these networks to achieve unparalleled flexibility and performance in a wide range of applications.

## 10.4 Activation Functions

In the context of **neural networks**, activation functions serve as an essential component that enables the network to approximate complex, non-linear mappings. When a neural network processes input data, each neuron computes a weighted sum of the inputs and then applies an activation function $\sigma(z)$ to produce the output. Mathematically, let the input vector to the neuron be $\mathbf{x} = (x_1, x_2, \dots, x_n)$, and let the weight vector associated with these inputs be $\mathbf{w} = (w_1, w_2, \dots, w_n)$. The corresponding bias term is denoted as $b$. The net input $z$ to the activation function is then given by:

$$ z = \mathbf{w}^{\top} \mathbf{x} + b = \sum_{i=1}^{n} w_i x_i + b \qquad (10.37) $$

where $\mathbf{w}^{\top} \mathbf{x}$ represents the dot product of the weight vector and the input vector. The activation function $\sigma(z)$ is then applied to this net input to obtain the output of the neuron $a$:

$$ a = \sigma(z) = \sigma\left(\sum_{i=1}^{n} w_i x_i + b\right). \qquad (10.38) $$
The activation function introduces a non-linearity into the neuron's response, which is a crucial aspect of neural networks because, without it, the network would only be able to perform linear transformations of the input data, limiting its ability to approximate complex, real-world functions. The non-linearity introduced by $\sigma(z)$ is fundamental because it enables the network to capture intricate relationships between the input and output, making neural networks capable of solving problems that require hierarchical feature extraction, such as image classification, time-series forecasting, and language modeling. The importance of non-linearity is most clearly evident when considering the mathematical formulation of a multi-layer neural network. For a feed-forward neural network with $L$ layers, the output $\hat{y}$ of the network is given by the composition of successive affine transformations and activation functions. Let $\mathbf{x}$ denote the input vector, $W_k$ and $b_k$ be the weight matrix and bias vector for the $k$-th layer, and $\sigma_k$ be the activation function for the $k$-th layer. The output of the network is:

$$ \hat{y} = \sigma_L(W_L \sigma_{L-1}(W_{L-1} \dots \sigma_1(W_1 \mathbf{x} + b_1) + b_2) + \dots + b_L). \quad (10.39) $$

If $\sigma(z)$ were a linear function, say $\sigma(z) = c \cdot z$ for some constant $c$, the composition of such functions would still result in a linear function. Specifically, if each $\sigma_k$ were linear, the overall network function would simplify to a single linear transformation:

$$ \hat{y} = c_1 \cdot \mathbf{x} + c_2, \quad (10.40) $$

where $c_1$ and $c_2$ are constants dependent on the parameters of the network. In this case, the network would have no greater expressive power than a simple linear regression model, regardless of the number of layers. Thus, the non-linearity introduced by activation functions allows neural networks to approximate any continuous function, as guaranteed by the universal approximation theorem. This theorem states that a feed-forward neural network with at least one hidden layer and a sufficiently large number of neurons can approximate any continuous function $f(\mathbf{x})$, provided the activation function is non-linear and the network has enough capacity.

Next, consider the mathematical properties that the activation function $\sigma(z)$ must possess. First, it must be differentiable to allow the use of gradient-based optimization methods like backpropagation for training. Backpropagation relies on the chain rule of calculus to compute the gradients of the loss function $\mathcal{L}$ with respect to the parameters (weights and biases) of the network. Suppose $\mathcal{L} = \mathcal{L}(\hat{y}, \mathbf{y})$ is the loss function, where $\hat{y}$ is the predicted output of the network and $\mathbf{y}$ is the true label. During training, we compute the gradient of $\mathcal{L}$ with respect to the weights using the chain rule. Let $a_k = \sigma_k(z_k)$ represent the output of the activation function at layer $k$, where $z_k$ is the input to the activation function. The gradient of the loss with respect to the weights at layer $k$ is given by:

$$ \frac{\partial \mathcal{L}}{\partial W_k} = \frac{\partial \mathcal{L}}{\partial a_k} \frac{\partial a_k}{\partial z_k} \frac{\partial z_k}{\partial W_k}. \quad (10.41) $$

The term $\frac{\partial a_k}{\partial z_k}$ is the derivative of the activation function, which must exist and be well-defined for gradient-based optimization to work effectively. If the activation function is not differentiable, the backpropagation algorithm cannot compute the gradients, preventing the training process from proceeding.

Now consider the specific forms of activation functions commonly used in practice. The sigmoid activation function is one of the most well-known, defined as:

$$ \sigma(z) = \frac{1}{1 + e^{-z}}. \quad (10.42) $$

Its derivative is:

$$ \sigma'(z) = \sigma(z)(1 - \sigma(z)), \quad (10.43) $$
which can be derived by applying the chain rule to the expression for $\sigma(z)$. Although sigmoid is differentiable and smooth, it suffers from the *vanishing gradient problem*, especially for large positive or negative values of $z$. Specifically, as $z \to \infty$, $\sigma'(z) \to 0$, and similarly as $z \to -\infty$, $\sigma'(z) \to 0$. This results in very small gradients during backpropagation, making it difficult for the network to learn when the input values become extreme. To mitigate the vanishing gradient problem, the **hyperbolic tangent (tanh)** function is often used as an alternative. It is defined as:

$$ \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}, \qquad (10.44) $$

with derivative:

$$ \tanh'(z) = 1 - \tanh^2(z). \qquad (10.45) $$

The tanh function outputs values in the range (-1, 1), which helps to center the data around zero. While the tanh function overcomes some of the vanishing gradient issues associated with the sigmoid function, it still suffers from the problem for large $|z|$, where the gradients approach zero. The **Rectified Linear Unit (ReLU)** is another commonly used activation function. It is defined as:

$$ \mathrm{ReLU}(z) = \max(0, z), \qquad (10.46) $$

with derivative:

$$ \mathrm{ReLU}'(z) = \begin{cases} 1, & z > 0, \\ 0, & z \le 0. \end{cases} \qquad (10.47) $$

ReLU is particularly advantageous because it is computationally efficient, as it only requires a comparison to zero. Moreover, for positive values of $z$, the derivative is constant and equal to 1, which helps avoid the vanishing gradient problem. However, ReLU can suffer from the *dying ReLU problem*, where neurons output zero for all inputs if the weights are initialized poorly or if the learning rate is too high, leading to inactive neurons that do not contribute to the learning process. To address the dying ReLU problem, the **Leaky ReLU** activation function is introduced, defined as:

$$ \text{Leaky ReLU}(z) = \begin{cases} z, & z > 0, \\ \alpha z, & z \le 0, \end{cases} \qquad (10.48) $$

where $\alpha$ is a small constant, typically chosen to be 0.01. The derivative of the Leaky ReLU is:

$$ \text{Leaky ReLU}'(z) = \begin{cases} 1, & z > 0, \\ \alpha, & z \le 0. \end{cases} \qquad (10.49) $$

Leaky ReLU ensures that neurons do not become entirely inactive by allowing a small, non-zero gradient for negative values of $z$. Finally, for classification tasks, particularly when there are multiple classes, the **Softmax** activation function is often used in the output layer of the neural network. The Softmax function is defined as:

$$ \mathrm{Softmax}(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}}, \qquad (10.50) $$

where $z_i$ is the input to the $i$-th neuron in the output layer, and the denominator ensures that the outputs sum to 1, making them interpretable as probabilities. The Softmax function is typically used in multi-class classification problems, where the network must predict one class out of several possible categories.

In summary, activation functions are a vital component of neural networks, enabling them to learn intricate patterns in data, allowing for the successful application of neural networks to diverse tasks. Different activation functions—such as sigmoid, tanh, ReLU, Leaky ReLU, and Softmax—each offer distinct advantages and limitations, and their choice significantly impacts the performance and training dynamics of the neural network.
# 10.5 Loss Functions

In neural networks, the **loss function** is a crucial mathematical tool that quantifies the difference between the predicted output of the model and the true output or target. Let $\mathbf{x}_i$ be the input vector and $\mathbf{y}_i$ the corresponding target vector for the $i$-th training example. The network, parameterized by weights $\mathbf{W}$, generates a prediction denoted as $\hat{\mathbf{y}}_i = f(\mathbf{x}_i; \mathbf{W})$, where $f(\mathbf{x}_i; \mathbf{W})$ represents the model's output. The objective of training the neural network is to minimize the discrepancy between the predicted output $\hat{\mathbf{y}}_i$ and the true label $\mathbf{y}_i$ across all training examples, effectively learning the mapping function from inputs to outputs. A typical objective function is the **average loss** over a dataset of $N$ samples:

$$ \mathcal{L}(\mathbf{W}) = \frac{1}{N} \sum_{i=1}^{N} L(\mathbf{y}_i, \hat{\mathbf{y}}_i) \qquad (10.51) $$

where $L(\mathbf{y}_i, \hat{\mathbf{y}}_i)$ represents the loss function that computes the error between the true output $\mathbf{y}_i$ and the predicted output $\hat{\mathbf{y}}_i$ for each data point. To minimize this objective function, optimization algorithms such as **gradient descent** are used. The general update rule for the weights $\mathbf{W}$ is given by:

$$ \mathbf{W} \leftarrow \mathbf{W} - \eta \nabla_{\mathbf{W}} \mathcal{L}(\mathbf{W}) \qquad (10.52) $$

where $\eta$ is the **learning rate**, and $\nabla_{\mathbf{W}} \mathcal{L}(\mathbf{W})$ is the gradient of the loss function with respect to the weights. The gradient is computed using **backpropagation**, which applies the **chain rule** of calculus to propagate the error backward through the network, updating the parameters to minimize the loss. For this, we use the partial derivatives of the loss with respect to each layer's weights and biases, ensuring the error is distributed appropriately across all layers.

In summary, the loss function plays a central role in guiding the optimization process in neural network training by quantifying the error between the predicted and true outputs. Different loss functions are employed depending on the nature of the problem, with MSE being common for regression and cross-entropy used for classification. Regularization techniques such as L2 and L1 regularization are incorporated to prevent overfitting and ensure better generalization. Through optimization algorithms like gradient descent, the neural network parameters are iteratively updated based on the gradients of the loss function, with the ultimate goal of minimizing the loss across all training examples.

## 10.5.1 Loss Functions for Regression Tasks

### 10.5.1.1 Mean Squared Error (MSE/L2 Loss)

For regression tasks, the **Mean Squared Error (MSE)** loss is frequently used. This loss function quantifies the error as the average squared difference between the predicted and true values. The MSE for a dataset of $N$ examples is given by:

$$ L_{\text{MSE}} = \frac{1}{N} \sum_{i=1}^{N} (\mathbf{y}_i - \hat{\mathbf{y}}_i)^2 \qquad (10.53) $$

where $\hat{\mathbf{y}}_i = f(\mathbf{x}_i; \mathbf{W})$ is the network's predicted output for the $i$-th input $\mathbf{x}_i$. The gradient of the MSE with respect to the network's output $\hat{\mathbf{y}}_i$ is:

$$ \frac{\partial L_{\text{MSE}}}{\partial \hat{\mathbf{y}}_i} = 2(\hat{\mathbf{y}}_i - \mathbf{y}_i) \qquad (10.54) $$

This gradient guides the weight update in the direction that minimizes the squared error, leading to a better fit of the model to the training data.
<figure><img src="image_82.png" /><figcaption>Figure 10.1: Mean Squared Error (MSE/L2) Loss Function</figcaption></figure>

In neural network training, the optimization process often involves regularization techniques to prevent **overfitting**, especially in cases with high-dimensional data or deep networks. **L2 regularization** (also known as **Ridge regression**) is one common approach, which penalizes large weights by adding a term proportional to the squared L2 norm of the weights to the loss function. The regularized loss function becomes:

$$L_{\text{reg}} = L_{\text{MSE}} + \lambda \sum_{j=1}^{n} W_{j}^{2} \qquad (10.55)$$

where $\lambda$ is the regularization strength, and $W_j$ represents the parameters of the network. The gradient of the regularized loss with respect to the weights is:

$$\frac{\partial L_{\text{reg}}}{\partial W_j} = \frac{\partial L_{\text{MSE}}}{\partial W_j} + 2\lambda W_j \qquad (10.56)$$

This additional term discourages large values of the weights, reducing the complexity of the model and helping it generalize better to unseen data. Another form of regularization is **L1 regularization** (or **Lasso regression**), which promotes sparsity in the model by adding the L1 norm of the weights to the loss function. The L1 regularized loss function is:

$$L_{\text{reg}} = L_{\text{MSE}} + \lambda \sum_{j=1}^{n} |W_j| \qquad (10.57)$$
The gradient of this regularized loss function with respect to the weights is:

$$ \frac{\partial L_{\text{reg}}}{\partial W_j} = \frac{\partial L_{\text{MSE}}}{\partial W_j} + \lambda \text{sign}(W_j) \quad (10.58) $$

where $\text{sign}(W_j)$ is the sign function, which returns 1 for positive values of $W_j$, -1 for negative values, and 0 for $W_j = 0$. L1 regularization encourages the model to select only a small subset of features by forcing many of the weights to exactly zero, thus simplifying the model and promoting interpretability. The optimization process for neural networks can be viewed as solving a **non-convex optimization problem**, given the highly non-linear activation functions and the deep architectures typically used. In this context, **stochastic gradient descent (SGD)** is commonly employed to perform the optimization by updating the weights based on the gradient computed from a random mini-batch of the data. The update rule for SGD can be expressed as:

$$ \mathbf{W} \leftarrow \mathbf{W} - \eta \nabla_{\mathbf{W}} L_{\text{batch}} \quad (10.59) $$

where $\nabla_{\mathbf{W}} L_{\text{batch}}$ is the gradient of the loss function computed over the mini-batch, and $\eta$ is the learning rate. Due to the non-convexity of the objective function, SGD tends to converge to a **local minimum** or a **saddle point**, rather than the global minimum, especially in deep neural networks with many layers.

### 10.5.1.2 Mean Absolute Error (MAE/L1 Loss)

The Mean Absolute Error (MAE), equivalently termed L1 loss, is a fundamental statistical loss function defined as the expected value of the absolute deviation between a vector of true target values, $\mathbf{y} \in \mathbb{R}^N$, and a corresponding vector of predictions, $\hat{\mathbf{y}} \in \mathbb{R}^N$, over a dataset of $N$ independent and identically distributed (i.i.d.) observations. Its formal definition is given by the functional $L_{\text{MAE}}: \mathbb{R}^N \times \mathbb{R}^N \to \mathbb{R}_{\ge 0}$:

$$ L_{\text{MAE}}(\mathbf{y}, \hat{\mathbf{y}}) = \frac{1}{N} \sum_{i=1}^{N} |y_i - \hat{y}_i| = \mathbb{E}_{\text{data}}[|Y - \hat{Y}|]. \quad (10.60) $$

This formulation directly measures the average magnitude of errors without considering their direction, imposing a linear penalty on the residuals $r_i = y_i - \hat{y}_i$. This linearity stands in stark contrast to the quadratic penalty of the Mean Squared Error (MSE, or L2 loss),

$$ L_{\text{MSE}} = \frac{1}{N} \sum_{i=1}^{N} (y_i - \hat{y}_i)^2 \quad (10.61) $$

a distinction that fundamentally alters the estimator produced by its minimization and its robustness properties.

From a statistical learning perspective, the process of empirical risk minimization involves finding the model parameters $\boldsymbol{\theta}^*$ that minimize the expected loss. For MAE, this is:

$$ \boldsymbol{\theta}^* = \arg \min_{\boldsymbol{\theta}} \mathbb{E}_{(\mathbf{x}, y) \sim \mathcal{P}} [|y - f(\mathbf{x}; \boldsymbol{\theta})|] \approx \arg \min_{\boldsymbol{\theta}} \frac{1}{N} \sum_{i=1}^{N} |y_i - f(\mathbf{x}_i; \boldsymbol{\theta})|, \quad (10.62) $$

where $\mathcal{P}$ is the true data distribution. The solution to this optimization problem converges to an estimator of the conditional median of the target distribution,

$$ f(\mathbf{x}; \boldsymbol{\theta}^*) \approx \text{median}(Y | \mathbf{X} = \mathbf{x}) \quad (10.63) $$

This is proven by considering the derivative of the expected loss. For a fixed input $\mathbf{x}$, the point estimate $a$ that minimizes $\mathbb{E}[|Y - a| | \mathbf{X} = \mathbf{x}]$ is indeed the median. This contrasts with MSE,
<figure><img src="image_83.png" /><figcaption>Figure 10.2: Mean Absolute Error (MAE/L1) Loss Function</figcaption></figure>

which targets the conditional mean, $\mathbb{E}[Y|\mathbf{X} = \mathbf{x}]$, a property that makes MAE far more robust to outliers as the median is less sensitive to extreme values in the tail of the distribution than the mean.

The analytical properties of the MAE function present specific challenges for gradient-based optimization. The absolute value function, $g(z) = |z|$, is Lipschitz continuous but non-differentiable at $z = 0$. Consequently, the gradient of the MAE loss with respect to a prediction $\hat{y}_i$ must be expressed using the concept of a subgradient. The subdifferential $\partial g(z)$ at a point $z$ is the set of all values that serve as a valid substitute for the derivative. For $g(z) = |z|$, the subdifferential is:

$$ \partial|z| = \begin{cases} -1 & \text{if } z < 0 \\ [-1, 1] & \text{if } z = 0. \\ \{+1\} & \text{if } z > 0 \end{cases} \qquad (10.64) $$

Therefore, the subgradient of the MAE loss for a single data point with respect to the prediction is:

$$ \frac{\partial}{\partial \hat{y}_i} |y_i - \hat{y}_i| = \begin{cases} -1 & \text{if } \hat{y}_i < y_i \\ [-1, 1] & \text{if } \hat{y}_i = y_i \quad (\text{conventionally, 0 is chosen}). \\ \{+1\} & \text{if } \hat{y}_i > y_i \end{cases} \qquad (10.65) $$

The overall gradient of the total loss with respect to the model's parameters $\theta$ is then computed via the chain rule during backpropagation. If the model output is $\hat{\mathbf{y}} = f(\mathbf{X}; \theta)$, then:

$$ \nabla_{\theta} L_{\text{MAE}} = \frac{1}{N} \sum_{i=1}^{N} \left( \frac{\partial |y_i - \hat{y}_i|}{\partial \hat{y}_i} \right) \cdot \left( \frac{\partial \hat{y}_i}{\partial \theta} \right). \qquad (10.66) $$
This gradient has a constant magnitude of 1 for all misclassified samples, which ensures that no single outlier can dominate the parameter update step. However, it provides no second-order information (the Hessian is zero almost everywhere), and its discontinuous nature at the optimum can lead to oscillations and slower convergence rates compared to the smooth, quadratic basins of attraction found in MSE optimization, often necessitating the use of more sophisticated optimization techniques or a transition to a smooth analogue like the Huber loss near the minimum.

### 10.5.1.3 Huber Loss (Smooth Mean Absolute Error)

The Huber loss function, often termed Smooth Mean Absolute Error, is a piecewise-defined loss function employed in regression tasks to combine the beneficial properties of both the Mean Squared Error (MSE) and the Mean Absolute Error (MAE). Its primary motivation is to be robust to outliers in the data, like the MAE, while simultaneously maintaining differentiability at zero, a property of MSE that is crucial for stable optimization using gradient-based methods. The function achieves this by behaving quadratically for small errors, ensuring smoothness, and linearly for large errors, mitigating the excessive influence of outliers that is characteristic of the quadratic MSE. The transition point between these two regimes is governed by a hyperparameter, $\delta$ (delta), which defines the threshold at which the loss transitions from a quadratic to a linear behavior. The value of $\delta$ must be a positive real number, $\delta > 0$, and its specific value is often determined through hyperparameter tuning, though a common default is $\delta = 1.0$.

<figure><img src="image_84.png" /><figcaption>Figure 10.3: Huber Loss Function</figcaption></figure>

Mathematically, the Huber loss for a single prediction-error pair, where the residual is defined
as $r_i = y_i - f(x_i)$, is formally expressed as:

$$L_{\delta}(r_i) = \begin{cases} \frac{1}{2}r_i^2 & \text{for } |r_i| \le \delta \\ \delta|r_i| - \frac{1}{2}\delta^2 & \text{for } |r_i| > \delta \end{cases} \qquad (10.67)$$

This piecewise definition can be understood by analyzing its two distinct behavioral domains. For residuals whose absolute value is less than or equal to the threshold $\delta$, the loss is exactly equal to one-half of the squared error

$$L_{\delta}(r_i) = \frac{1}{2}r_i^2 \qquad (10.68)$$

The factor of $\frac{1}{2}$ is included for mathematical convenience, as it cancels out the factor of 2 arising from the derivative, simplifying the gradient computation. In this region, the function is convex and exhibits a quadratic growth, which provides a strong, continuous gradient signal for small errors, promoting efficient convergence during training.

For residuals that exceed the threshold $\delta$, meaning $|r_i| > \delta$, the loss function transitions to a linear growth rate. The expression in this regime is

$$L_{\delta}(r_i) = \delta|r_i| - \frac{1}{2}\delta^2 \qquad (10.69)$$

The linear term, $\delta|r_i|$, ensures that the loss increases at a constant rate for large errors, making it significantly more robust to outliers than the MSE, which would grow quadratically. The constant subtractive term, $-\frac{1}{2}\delta^2$, is not an arbitrary adjustment but is a necessary component to ensure the function is continuous and smooth at the point of transition, $|r_i| = \delta$. This term guarantees that the two pieces of the function meet at the boundary point.

The continuity and differentiability of the Huber loss function are its defining scientific features. The function is continuous at $|r_i| = \delta$, which is verified by evaluating both expressions at that point: $\frac{1}{2}\delta^2$ from the quadratic piece equals $\delta \cdot \delta - \frac{1}{2}\delta^2 = \frac{1}{2}\delta^2$ from the linear piece. Furthermore, the function is differentiable everywhere, including at the critical point $|r_i| = \delta$. The derivative of the loss with respect to the residual $r_i$ is given by:

$$\frac{\partial L_{\delta}}{\partial r_i} = \begin{cases} r_i & \text{for } |r_i| \le \delta \\ \delta \cdot \text{sign}(r_i) & \text{for } |r_i| > \delta \end{cases} \qquad (10.70)$$

At $|r_i| = \delta$, the derivative from the left is $\delta$, and the derivative from the right is $\delta \cdot \text{sign}(\delta) = \delta$, confirming the differentiability of the function. This continuous first derivative is essential for the stable application of gradient descent optimization algorithms, as it prevents the erratic behavior that can occur with the MAE loss, whose gradient is discontinuous at zero. The Hessian of the Huber loss also exists for $|r_i| < \delta$, where it is equal to 1, and for $|r_i| > \delta$, where it is equal to 0. This structure makes it a useful penalty function in advanced optimization contexts like proximal methods. In practice, the total cost for a dataset is computed as the mean of the individual Huber losses over all N data points:

$$J(\theta) = \frac{1}{N} \sum_{i=1}^{N} L_{\delta}(y_i - f(x_i, \theta)) \qquad (10.71)$$

The selection of $\delta$ thus represents a trade-off; a larger $\delta$ value means more data points are treated quadratically (MSE-like behavior), while a smaller $\delta$ value means more points are treated linearly (MAE-like behavior).
### 10.5.1.5 Python Code to generate Figure 10.2 illustrating Mean Absolute Error (MAE/L1) Loss Function

The Python code below produces the Figure 10.2 illustrating Mean Absolute Error (MAE/L1) Loss Function.

```python
import numpy as np
import matplotlib.pyplot as plt

# Residuals (y_true - y_pred)
a = np.linspace(-5, 5, 500)

# MAE / L1 Loss
mae_loss = np.abs(a)

# Plot
plt.figure(figsize=(8,6))
plt.plot(a, mae_loss, lw=2, color='green', label='MAE / L1 Loss |a|')

# Labels and title
plt.xlabel("Residual $a = y_{true} - y_{pred}$", fontsize=12)
plt.ylabel("Loss", fontsize=12)
plt.title("Mean Absolute Error (MAE / L1) Loss Function", fontsize=14)
plt.grid(True)
plt.legend()
plt.show()
```

### 10.5.1.6 Python Code to generate Figure 10.3 illustrating Huber Loss Function

The Python code below produces the Figure 10.3 illustrating Huber Loss Function.

```python
import numpy as np
import matplotlib.pyplot as plt

# Residuals (y_true - y_pred)
a = np.linspace(-5, 5, 500)

# Huber delta
delta = 1.0

# Compute Huber loss
huber_loss = np where(
    np abs(a) <= delta,
    0.5 * a**2,
    delta * (np abs(a) - 0.5 * delta)
)

# Plot
figsize=(8,6)
ax = fig, ax
ax plot(a, huber_loss, lw=2, color='blue', label='Huber Loss ($\delta$={delta}))

# Labels and formatting
```
22 plt.xlabel("Residual $a = y_{\{true\}} - y_{\{pred\}}$", fontsize=12)
23 plt.ylabel("Huber Loss", fontsize=12)
24 plt.title("Huber Loss Function", fontsize=14)
25 plt.grid(True)
26 plt.legend()
27 plt.show()

### 10.5.1.7 Python Code to generate Figure 10.4 comparing the Loss Functions: Huber, MSE, MAE

The Python code below produces the Figure 10.4 comparing the Loss Functions: Huber, MSE, MAE.

1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 # Residuals (y_true - y_pred)
5 a = np.linspace(-5, 5, 500)
6
7 # Huber delta
8 delta = 1.0
9
10 # Compute losses
11 mse_loss = 0.5 * a**2
12 mae_loss = np.abs(a)
13 huber_loss = np where(
14 np abs a) <= delta,
15 0.5 * a**2,
16 delta * (np abs a) - 0.5 * delta
17 )
18
19 # Plot
20 plt figure figsize=(8,6))
21 plt plot(a, mse_loss, lw=2, label='MSE (0.5*a^2)', color='red')
22 plt plot(a, mae_loss, lw=2, label='MAE (|a|)', color='green')
23 plt plot(a, huber_loss, lw=2, label=f'Huber (δ={delta})', color='blue')
24
25 # Labels and formatting
26 plt xlabel("Residual $a = y_{\{true\}} - y_{\{pred\}}$", fontsize=12)
27 plt ylabel("Loss", fontsize=12)
28 plt title("Comparison of Loss Functions: Huber, MSE, MAE", fontsize=14)
29 plt grid(True)
30 plt legend()
31 plt show()

### 10.5.2 Loss Functions for Classification Tasks

For classification tasks, the cross-entropy loss is often employed, as it is particularly well-suited to tasks where the output is a probability distribution over multiple classes. In the binary classification case, where the target label $y_i$ is either 0 or 1, the binary cross-entropy loss function is defined as:

$$L_{\text{CE}} = -\frac{1}{N} \sum_{i=1}^{N} [y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i)] \quad (10.72)$$
where $\hat{y}_i = f(\mathbf{x}_i; \mathbf{W})$ is the predicted probability that the $i$-th sample belongs to the positive class (i.e., class 1). For multiclass classification, where the target label $\mathbf{y}_i$ is a one-hot encoded vector representing the true class, the general form of the cross-entropy loss is:

$$L_{\text{CE}} = -\frac{1}{N} \sum_{i=1}^{N} \sum_{c=1}^{C} y_{i,c} \log(\hat{y}_{i,c}) \qquad (10.73)$$

where $C$ is the number of classes, and $\hat{y}_{i,c} = f(\mathbf{x}_i; \mathbf{W})$ is the predicted probability that the $i$-th sample belongs to class $c$. The gradient of the cross-entropy loss with respect to the predicted probabilities $\hat{\mathbf{y}}_i$ is:

$$\frac{\partial L_{\text{CE}}}{\partial \hat{y}_{i,c}} = \hat{y}_{i,c} - y_{i,c} \qquad (10.74)$$

This gradient facilitates the weight update by adjusting the model's parameters to reduce the difference between the predicted probabilities and the actual class labels.

### 10.5.2.1 Binary Cross-Entropy (Log Loss)

The Binary Cross-Entropy loss function is the cornerstone of optimization for binary classification models, deriving its form directly from the principles of maximum likelihood estimation for a Bernoulli distribution. Consider a dataset of $N$ independent and identically distributed observations, where the true labels are $y_i \in \{0, 1\}$ and the model's prediction is the parameter $\hat{y}_i = f(\mathbf{x}_i; \boldsymbol{\theta}) \in (0, 1)$, representing the estimated probability that $y_i = 1$. The probability of observing a single label $y_i$ given the input $\mathbf{x}_i$ is given by the Bernoulli probability mass function:

$$P(Y = y_i | \mathbf{x}_i) = \hat{y}_i^{y_i} (1 - \hat{y}_i)^{1-y_i} \qquad (10.75)$$

The likelihood function for the entire dataset, assuming conditional independence, is the product of these probabilities:

$$\mathcal{L}(\boldsymbol{\theta}) = \prod_{i=1}^{N} P(Y = y_i | \mathbf{x}_i) = \prod_{i=1}^{N} \hat{y}_i^{y_i} (1 - \hat{y}_i)^{1-y_i} \qquad (10.76)$$

The maximum likelihood estimate (MLE) for the parameters $\boldsymbol{\theta}$ is found by maximizing this likelihood. To simplify this optimization, we take the natural logarithm, obtaining the log-likelihood:

$$\ell(\boldsymbol{\theta}) = \log \mathcal{L}(\boldsymbol{\theta}) = \sum_{i=1}^{N} [y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i)] \qquad (10.77)$$

Since optimization algorithms are typically designed for minimization, we define the loss function as the negative log-likelihood, scaled by $1/N$ to represent an empirical expectation over the data distribution:

$$L_{\text{BCE}}(\mathbf{y}, \hat{\mathbf{y}}) = -\frac{1}{N} \sum_{i=1}^{N} [y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i)]. \qquad (10.78)$$

Minimizing $L_{\text{BCE}}$ is thus equivalent to maximizing the likelihood of the observed data under the assumed Bernoulli model, ensuring the learned parameters $\boldsymbol{\theta}$ yield a predicted probability distribution that is most consistent with the true labels.

From an information-theoretic perspective, Binary Cross-Entropy is the specific instance of the Kullback-Leibler (KL) Divergence between the empirical data distribution $P$ and the model's predictive distribution $Q$. For a single data point, the true distribution $P$ is a degenerate Bernoulli distribution with parameter $y_i$, having a probability mass of 1 at the true class. The model's distribution $Q$ is a Bernoulli distribution with parameter $\hat{y}_i$. The KL Divergence, which measures the
<figure><img src="image_85.png" /><figcaption>Figure 10.5: Binary Cross-Entropy (BCE) Loss Function</figcaption></figure>

information loss when $Q$ is used to approximate $P$, is:

$$D_{\text{KL}}(P \parallel Q) = \sum_{k \in \{0,1\}} P(k) \log \frac{P(k)}{Q(k)} = y_i \log \frac{y_i}{\hat{y}_i} + (1 - y_i) \log \frac{1 - y_i}{1 - \hat{y}_i} \quad (10.79)$$

This expression can be decomposed as

$$D_{\text{KL}}(P \parallel Q) = [y_i \log y_i + (1 - y_i) \log(1 - y_i)] - [y_i \log \hat{y}_i + (1 - y_i) \log(1 - \hat{y}_i)] \quad (10.80)$$

The first term is the negative entropy of the true distribution $H(P)$. Since $P$ is deterministic, $H(P) = 0$, as $0 \log 0$ is defined to be 0 by continuity and $1 \log 1 = 0$. Thus, the divergence simplifies to

$$D_{\text{KL}}(P \parallel Q) = -[y_i \log \hat{y}_i + (1 - y_i) \log(1 - \hat{y}_i)] \quad (10.81)$$

The Binary Cross-Entropy loss is then the expectation of this divergence over the dataset:

$$L_{\text{BCE}} = \frac{1}{N} \sum_{i=1}^{N} D_{\text{KL}}(P_i \parallel Q_i) = \mathbb{E}_{i \sim \text{Data}}[D_{\text{KL}}(P_i \parallel Q_i)] \quad (10.82)$$

The analytical properties of Binary Cross-Entropy are paramount for its use with gradient-based optimization.

The output of the neural network is typically produced by a sigmoid activation function $\sigma(z)$ applied to the final logit $z$, ensuring the prediction lies in the valid probability range $(0, 1)$:

$$\hat{y} = \sigma(z) = \frac{1}{1 + e^{-z}} \quad (10.83)$$
Binary Cross-Entropy Loss Surface

<figure><img src="image_86.png" /><figcaption>Figure 10.6: Binary Cross-Entropy Loss Surface</figcaption></figure>

The gradient of the loss with respect to the logit z exhibits a crucial simplification. We compute this gradient using the chain rule. First, the partial derivative of the loss for a single term with respect to the prediction ŷ is:

$$ \frac{\partial L_i}{\partial \hat{y}_i} = -\frac{y_i}{\hat{y}_i} + \frac{1 - y_i}{1 - \hat{y}_i}. \qquad (10.84) $$

The derivative of the sigmoid activation with respect to its input is well-known:

$$ \frac{\partial \hat{y}_i}{\partial z_i} = \frac{\partial \sigma(z_i)}{\partial z_i} = \sigma(z_i)(1 - \sigma(z_i)) = \hat{y}_i(1 - \hat{y}_i) \qquad (10.85) $$

Applying the chain rule, the gradient of the loss with respect to the logit is:

$$ \frac{\partial L_i}{\partial z_i} = \frac{\partial L_i}{\partial \hat{y}_i} \cdot \frac{\partial \hat{y}_i}{\partial z_i} = \left( -\frac{y_i}{\hat{y}_i} + \frac{1 - y_i}{1 - \hat{y}_i} \right) \cdot \hat{y}_i(1 - \hat{y}_i). \qquad (10.86) $$

Distributing the terms yields:

$$ \frac{\partial L_i}{\partial z_i} = -y_i(1 - \hat{y}_i) + (1 - y_i)\hat{y}_i = -y_i + y_i\hat{y}_i + \hat{y}_i - y_i\hat{y}_i = \hat{y}_i - y_i. \qquad (10.87) $$

This results in the elegant and stable gradient formula:

$$ \nabla_{z_i} L_{\text{BCE}} = \hat{y}_i - y_i \qquad (10.88) $$

This simplicity is a key advantage. The gradient is merely the error between the prediction and the true label. Its magnitude is proportional to the error, providing a strong, clear signal for weight
updates through backpropagation. The Hessian matrix, which contains the second derivatives, can be derived from this gradient. For a single data point, the second derivative with respect to the logit is

$$ \frac{\partial^2 L_i}{\partial z_i^2} = \frac{\partial \hat{y}_i}{\partial z_i} = \hat{y}_i(1 - \hat{y}_i) > 0 \qquad (10.89) $$

for $\hat{y}_i \in (0, 1)$, confirming the loss is convex with respect to the input logit $z_i$. This local convexity contributes to the stable convergence properties of the optimization process when using this loss function.

### 10.5.2.2 Categorical Cross-Entropy

Categorical Cross-Entropy is the generalization of binary cross-entropy to multi-class classification problems where each instance belongs to exactly one of $K > 2$ mutually exclusive classes. Its derivation is rooted in maximum likelihood estimation for the multinomial distribution. For a single data point, the true label is represented by a one-hot encoded vector $\mathbf{y}_i \in \mathbb{R}^K$, where $y_{i,k} = 1$ if the instance belongs to class $k$ and 0 otherwise. The model's output is a probability distribution over the $K$ classes, typically obtained using a softmax activation function on the final layer's logits $\mathbf{z}_i \in \mathbb{R}^K$. The softmax function is defined as

$$ \hat{y}_{i,k} = \sigma(\mathbf{z}_i)_k = \frac{e^{z_{i,k}}}{\sum_{j=1}^{K} e^{z_{i,j}}} \qquad (10.90) $$

which ensures $\hat{y}_{i,k} \in (0, 1)$ and $\sum_{k=1}^K \hat{y}_{i,k} = 1$. The probability of observing the true label $\mathbf{y}_i$ under this categorical model is given by the multinomial PMF for a single trial:

$$ P(\mathbf{Y} = \mathbf{y}_i | \mathbf{x}_i) = \prod_{k=1}^{K} \hat{y}_{i,k}^{y_{i,k}} \qquad (10.91) $$

The likelihood for the entire dataset of $N$ independent observations is

$$ \mathcal{L}(\boldsymbol{\theta}) = \prod_{i=1}^{N} \prod_{k=1}^{K} \hat{y}_{i,k}^{y_{i,k}} \qquad (10.92) $$

The log-likelihood is

$$ \ell(\boldsymbol{\theta}) = \sum_{i=1}^{N} \sum_{k=1}^{K} y_{i,k} \log \hat{y}_{i,k} \qquad (10.93) $$

The Categorical Cross-Entropy loss is consequently defined as the negative log-likelihood, scaled by the sample size:

$$ L_{\text{CCE}}(\mathbf{Y}, \hat{\mathbf{Y}}) = -\frac{1}{N} \sum_{i=1}^{N} \sum_{k=1}^{K} y_{i,k} \log \hat{y}_{i,k} \qquad (10.94) $$

Minimizing this loss is equivalent to maximizing the likelihood of the data, forcing the model's predicted distribution $\hat{\mathbf{y}}_i$ to converge towards the true data distribution, which is a point mass at the correct class.

The information-theoretic interpretation of Categorical Cross-Entropy is that it computes the expected Kullback-Leibler Divergence between the true distribution $P$ and the model's distribution $Q$ for each data point. For a single instance, the true distribution $P$ is a categorical distribution with all probability mass on the true class $k^*$, so $p_k = \delta_{k,k^*}$ where $\delta$ is the Kronecker delta. The model's distribution $Q$ is defined by $q_k = \hat{y}_{i,k}$. The KL Divergence is:

$$ D_{\text{KL}}(P \parallel Q) = \sum_{k=1}^{K} p_k \log \frac{p_k}{q_k} = 1 \cdot \log \frac{1}{\hat{y}_{i,k}^{y_{i,k}}} + \sum_{k \neq k^*} 0 \cdot \log \frac{0}{q_k} = - \log \hat{y}_{i,k}^{y_{i,k}} \qquad (10.95) $$
<figure><img src="image_87.png" /><figcaption>Figure 10.7: Categorical Cross-Entropy Loss</figcaption></figure>

The terms 0 log 0 are defined as 0 by continuity. This is identical to the inner sum $\sum_{k=1}^{K} y_{i,k} \log \hat{y}_{i,k}$ for a one-hot vector $\mathbf{y}_i$, as only the term for $k = k^*$ is non-zero. Therefore, the loss

$$L_{\text{CCE}} = \frac{1}{N} \sum_{i=1}^{N} D_{\text{KL}}(P_i \| Q_i) \qquad (10.96)$$

is the average KL Divergence across the dataset, measuring the total information loss when using the model's distribution $Q$ to approximate the true data distribution $P$.

The gradient of the Categorical Cross-Entropy loss with respect to the logits $\mathbf{z}_i$ is particularly elegant and central to its use in backpropagation. The loss for a single sample is

$$L_i = - \sum_{k=1}^{K} y_{i,k} \log \hat{y}_{i,k} \qquad (10.97)$$

To compute $\frac{\partial L_i}{\partial z_{i,j}}$, we must account for the softmax function, as each logit $z_{i,j}$ influences every output $\hat{y}_{i,k}$ due to the normalization in the denominator. The partial derivative of the softmax output for class $k$ with respect to the logit for class $j$ is:

$$\frac{\partial \hat{y}_{i,k}}{\partial z_{i,j}} = \hat{y}_{i,k}(\delta_{kj} - \hat{y}_{i,j}), \qquad (10.98)$$
where $\delta_{kj}$ is the Kronecker delta. Using the chain rule, the gradient of the loss with respect to a specific logit $z_{i,j}$ is:

$$ \frac{\partial L_i}{\partial z_{i,j}} = - \sum_{k=1}^{K} \frac{y_{i,k}}{\hat{y}_{i,k}} \cdot \frac{\partial \hat{y}_{i,k}}{\partial z_{i,j}} = - \sum_{k=1}^{K} \frac{y_{i,k}}{\hat{y}_{i,k}} \cdot \hat{y}_{i,k}(\delta_{kj} - \hat{y}_{i,j}) = - \sum_{k=1}^{K} y_{i,k}(\delta_{kj} - \hat{y}_{i,j}). \quad (10.99) $$

This sum can be separated:

$$ \frac{\partial L_i}{\partial z_{i,j}} = - \left[ \sum_{k=1}^{K} y_{i,k} \delta_{kj} - \sum_{k=1}^{K} y_{i,k} \hat{y}_{i,j} \right] = - \left[ y_{i,j} - \hat{y}_{i,j} \sum_{k=1}^{K} y_{i,k} \right]. \quad (10.100) $$

Since $\mathbf{y}_i$ is a one-hot vector, $\sum_{k=1}^{K} y_{i,k} = 1$. This leads to the profoundly simple and stable result:

$$ \nabla_{z_{i,j}} L_{\text{CCE}} = \frac{\partial L_i}{\partial z_{i,j}} = \hat{y}_{i,j} - y_{i,j}. \quad (10.101) $$

<figure><img src="image_88.png" /><figcaption>Figure 10.8: Categorical Cross-Entropy Loss</figcaption></figure>

This gradient

$$ \nabla_{\mathbf{z}_i} L_i = \hat{\mathbf{y}}_i - \mathbf{y}_i \quad (10.102) $$

is a vector whose j-th component is simply the difference between the predicted probability and the true label for class j. This elegant form ensures that the error signal propagated backwards during training is directly proportional to the discrepancy between the entire predicted distribution and the true distribution, providing a strong and clear direction for parameter updates. The Hessian of the loss with respect to the logits can be derived from this gradient and is related to the covariance of the predicted distribution, but the simplicity of the first-order gradient is its most critical property for efficient optimization in deep learning.

### 10.5.2.3 Sparse Categorical Cross-Entropy

Sparse Categorical Cross-Entropy is a computationally optimized variant of the standard Categorical Cross-Entropy loss function, designed for multi-class classification tasks where the true labels are provided as integer indices rather than one-hot encoded vectors. This formulation is mathematically equivalent to Categorical Cross-Entropy in its underlying probabilistic foundation but differs in its implementation, leading to significant gains in memory efficiency and computational speed, particularly when the number of classes K is very large. The fundamental principle remains maximum likelihood estimation for a categorical distribution. For a dataset of N samples, the true label for the i-th sample is given as an integer $t_i \in \{1, 2, ..., K\}$, which implicitly defines a
degenerate probability distribution $P_i$ where $p_k = \delta_{k,t_i}$ and $\delta$ is the Kronecker delta. The model, as in the non-sparse case, produces a probability distribution over the $K$ classes via the softmax function applied to a vector of logits $\mathbf{z}_i \in \mathbb{R}^K$:

$$ \hat{y}_{i,k} = \frac{e^{z_{i,k}}}{\sum_{j=1}^{K} e^{z_{i,j}}} \qquad (10.103) $$

The likelihood of observing the true label $t_i$ is $P(T = t_i|\mathbf{x}_i) = \hat{y}_{i,t_i}$, and the corresponding negative log-likelihood for a single sample is

$$ L_i = -\log P(T = t_i|\mathbf{x}_i) = -\log \hat{y}_{i,t_i} \qquad (10.104) $$

The full Sparse Categorical Cross-Entropy loss is then the empirical expectation of this negative log-likelihood:

$$ L_{\text{SCCE}} = -\frac{1}{N} \sum_{i=1}^{N} \log \hat{y}_{i,t_i}. \qquad (10.105) $$

This is computationally and notationally more efficient than the standard form

$$ -\frac{1}{N} \sum_{i=1}^{N} \sum_{k=1}^{K} y_{i,k} \log \hat{y}_{i,k} \qquad (10.106) $$

because it avoids the explicit construction and storage of the entire $N \times K$ matrix of one-hot encodings, requiring only the storage of an integer vector of length $N$ for the labels. The minimization of this loss function is identical to minimizing the KL Divergence

$$ D_{\text{KL}}(P_i \parallel Q_i) = -\log \hat{y}_{i,t_i} \qquad (10.107) $$

for each sample, where $Q_i$ is the model's distribution.

The gradient computation for Sparse Categorical Cross-Entropy, crucial for backpropagation, reveals its intimate connection to the standard formulation while highlighting its efficiency. The gradient of the loss for a single sample $L_i = -\log \hat{y}_{i,t_i}$ with respect to the logits $\mathbf{z}_i$ must be computed. This requires the partial derivative $\frac{\partial L_i}{\partial z_{i,j}}$ for any class $j$. The derivative of the loss with respect to the predicted probability of the true class is

$$ \frac{\partial L_i}{\partial \hat{y}_{i,t_i}} = -\frac{1}{\hat{y}_{i,t_i}} \qquad (10.108) $$

The connection to all logits is made through the Jacobian of the softmax function

$$ \frac{\partial \hat{y}_{i,k}}{\partial z_{i,j}} = \hat{y}_{i,k}(\delta_{kj} - \hat{y}_{i,j}) \qquad (10.109) $$

Applying the chain rule, the gradient for an arbitrary logit $z_{i,j}$ is:

$$ \frac{\partial L_i}{\partial z_{i,j}} = \frac{\partial L_i}{\partial \hat{y}_{i,t_i}} \cdot \frac{\partial \hat{y}_{i,t_i}}{\partial z_{i,j}} = \left(-\frac{1}{\hat{y}_{i,t_i}}\right) \cdot (\hat{y}_{i,t_i}(\delta_{t_i j} - \hat{y}_{i,j})) = -(\delta_{t_i j} - \hat{y}_{i,j}) = \hat{y}_{i,j} - \delta_{t_i j}. \qquad (10.110) $$

This results in the same elegantly simple gradient vector as the standard formulation:

$$ \nabla_{\mathbf{z}_i} L_i = \hat{y}_i - \mathbf{y}_i, \qquad (10.111) $$

where $\mathbf{y}_i$ is now the implicit one-hot vector corresponding to the integer label $t_i$, meaning its $j$-th component is $\delta_{t_i j}$.
<figure><img src="image_89.png" /><figcaption>Figure 10.9: Sparse Categorical Cross-Entropy Loss</figcaption></figure>

<figure><img src="image_90.png" /><figcaption>Figure 10.10: Surface Plot of Sparse Categorical Cross-Entropy Loss</figcaption></figure>

From an implementation perspective, this means the gradient for the logit corresponding to the true class $j = t_i$ is $\hat{y}_{i,t_i} - 1$, and for all other logits $j \neq t_i$, it is simply $\hat{y}_{i,j} - 0 = \hat{y}_{i,j}$. This result is identical to the gradient derived for the one-hot encoded case; however, the computational pathway to achieve it is more efficient. There is no need to materialize a full one-hot matrix, perform a large matrix multiplication, or sum over all *K* classes for each sample during the gradient calculation. The backward pass can be implemented by first computing the softmax probabilities $\hat{y}_i$, then copying this entire vector as the gradient $\nabla_{\hat{y}_i} L_i = \hat{y}_i$, and finally subtracting 1 from the single element at the index of the true label $t_i$. This operation is $O(K)$ per sample, but the constant factors and memory overhead are drastically lower than in the one-hot case, making Sparse Categorical Cross-Entropy the preferred choice in practice for problems with a large number of classes, such as
in language modeling or large-scale image classification.

### 10.5.2.4 Kullback-Leibler Divergence (KL Divergence)

The Kullback-Leibler (KL) Divergence, denoted $D_{\text{KL}}(P \parallel Q)$, is a fundamental measure of information theory that quantifies the difference between two probability distributions $P$ and $Q$ over the same random variable $X$. It is not a true metric, as it is not symmetric and does not satisfy the triangle inequality; rather, it is a divergence that measures the information loss when a distribution $Q$ is used to approximate the true distribution $P$. For discrete distributions, the KL Divergence is defined as

$$D_{\text{KL}}(P \parallel Q) = \sum_{x \in \mathcal{X}} P(x) \log \frac{P(x)}{Q(x)} \qquad (10.112)$$

where $\mathcal{X}$ is the sample space, and the sum is taken over all possible values of $X$. For continuous distributions, the definition involves an integral:

$$D_{\text{KL}}(P \parallel Q) = \int_{-\infty}^{\infty} p(x) \log \frac{p(x)}{q(x)} dx \qquad (10.113)$$

where $p$ and $q$ are the probability density functions of $P$ and $Q$, respectively. The logarithm is typically taken with base 2 (yielding units of bits) or base $e$ (yielding units of nats), and by convention, $0 \log \frac{0}{q(x)} = 0$ and $p(x) \log \frac{p(x)}{0} = \infty$ for $p(x) > 0$, reflecting the infinite penalty for assigning zero probability to an event that is possible under $P$. The divergence is always non-negative, $D_{\text{KL}}(P \parallel Q) \ge 0$, with equality if and only if $P = Q$ almost everywhere, a result known as Gibbs' inequality, which follows from the strict concavity of the logarithm function and Jensen's inequality.

For a fixed discrete probability distribution $P$:

$$P = [0.1, 0.2, 0.3, 0.4]$$

The plot of KL Divergence as a function of $Q_1$ shall be

The KL Divergence can be decomposed into two interpretable components: the cross-entropy between $P$ and $Q$ minus the entropy of $P$. Cross-entropy $H(P, Q)$ measures the average number of bits needed to encode events from $P$ using a code optimized for $Q$, defined as

$$H(P, Q) = - \sum_{x} P(x) \log Q(x) \qquad (10.114)$$

for discrete variables. The entropy of $P$

$$H(P) = - \sum_{x} P(x) \log P(x) \qquad (10.115)$$

measures the inherent uncertainty or the average number of bits needed to encode events from $P$ using an optimal code. Thus

$$D_{\text{KL}}(P \parallel Q) = H(P, Q) - H(P) \qquad (10.116)$$

This decomposition shows that the divergence represents the extra number of bits required to encode data from $P$ when using a code based on $Q$, rather than the true distribution $P$. In the context of machine learning, where $P$ is often the true data distribution and $Q$ is the model's distribution, minimizing the KL Divergence is equivalent to minimizing the cross-entropy, since $H(P)$ is fixed with respect to the model parameters. This is why cross-entropy loss is ubiquitously used in classification tasks.
<figure><img src="image_91.png" /><figcaption>Figure 10.11: Plot of KL Divergence</figcaption></figure>

For a fixed discrete probability distribution P :

$$
P = [0.1, 0.2, 0.3, 0.4]
$$

The 2D plot of KL Divergence as a function of Q₁ and Q₂ shall be

The mathematical properties of the KL Divergence are critical for its applications. It is con-
vex in both P and Q, meaning that for any two pairs of distributions (P₁, Q₁) and (P₂, Q₂), and
for 0 ≤ λ ≤ 1, we have

$$
D_{\mathrm{KL}}(\lambda P_1 + (1-\lambda)P_2 \parallel \lambda Q_1 + (1-\lambda)Q_2) \le \lambda D_{\mathrm{KL}}(P_1 \parallel Q_1) + (1-\lambda)D_{\mathrm{KL}}(P_2 \parallel Q_2) \quad (10.117)
$$

This convexity ensures that optimization problems involving KL Divergence have unique minima
under certain constraints. The divergence also factors elegantly for independent distributions. If
P(X, Y) = P_X(X)P_Y(Y) and Q(X, Y) = Q_X(X)Q_Y(Y), then

$$
D_{\mathrm{KL}}(P(X,Y) \parallel Q(X,Y)) = D_{\mathrm{KL}}(P_X \parallel Q_X) + D_{\mathrm{KL}}(P_Y \parallel Q_Y) \quad (10.118)
$$

Additivity extends to more complex factorizations, such as in variational inference, where it is used
to approximate posterior distributions. The gradient of the KL Divergence with respect to the
parameters of Q is essential for gradient-based optimization. For a parameterized distribution Q<sub>θ</sub>,
we have

$$
\nabla_{\theta} D_{\mathrm{KL}}(P \parallel Q_{\theta}) = -\nabla_{\theta} \mathbb{E}_{x \sim P}[\log Q_{\theta}(x)] \quad (10.119)
$$

as the entropy of *P* is independent of *θ*. This gradient is the same as that of the cross-entropy loss,
facilitating its use in training probabilistic models like variational autoencoders, where it acts as a
## 2D Surface Plot of KL Divergence

<figure><img src="image_92.png" /><figcaption>Figure 10.12: 2D Surface Plot of KL Divergence</figcaption></figure>

regularizer to push the approximate posterior $Q_{\theta}(z|x)$ toward a prior distribution $P(z)$, ensuring tractable inference and meaningful latent representations.

### 10.5.2.5 Python Code to generate Figure 10.5 illustrating Binary Cross-Entropy (BCE) Loss Function

The Python code below produces the Figure 10.5 illustrating Binary Cross-Entropy (BCE) Loss Function.

```python
import numpy as np
import matplotlib.pyplot as plt

# Predicted probability
p = np.linspace(0.001, 0.999, 500)  # avoid log(0)

# Binary cross-entropy loss for y=1 and y=0
bce_y1 = -np.log(p)          # true label = 1
bce_y0 = -np.log(1 - p)      # true label = 0

# Plot
plt.figure(figsize=(8,6))
plt.plot(p, bce_y1, label="True label = 1: -log(p)", lw=2)
plt.plot(p, bce_y0, label="True label = 0: -log(1-p)", lw=2)
```
```python
15
16
17
18
19
20
21
22
23
24
25
26

# Labels and title
plt.xlabel("Predicted Probability $p$", fontsize=12)
plt.ylabel("Binary Cross-Entropy Loss", fontsize=12)
plt.title("Binary Cross-Entropy (BCE) Loss Function", fontsize=14)

plt.grid(True)

plt.legend()

# Highlight typical probabilities

plt axvline(0.5, color='red', linewidth=1, label='p=0.5')

plt.legend()

plt.show()
```

### 10.5.2.6 Python Code to generate Figure 10.6 illustrating Binary Cross-Entropy Loss Surface

The Python code below produces the Figure 10.6 illustrating Binary Cross-Entropy Loss Surface.

```python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

# Predicted probability
p = np.linspace(0.001, 0.999, 100)

# True label
y = np.array([0, 1])

# Create meshgrid
P, Y = np.meshgrid(p, y)

# Compute Binary Cross-Entropy loss
BCE = - (Y * np.log(P) + (1 - Y) * np.log(1 - P))

# Plot 3D surface
fig = plt.figure(figsize=(10,6))
ax = fig.add_subplot(111, projection='3d')
surf = ax plot service (P, Y, BCE, cmap='viridis', edgecolor='k', alpha=0.9)

ax.set xlabel("Predicted Probability $p$", fontsize=12)
ax.set ylabel("True Label $y$", fontsize=12)
ax.set zlabel("Binary Cross-Entropy Loss", fontsize=12)
ax.set title("Binary Cross-Entropy Loss Surface", fontsize=14)

fig colorbar (surf, shrink=0.5, aspect=10, label='BCE Loss')

plt.show()
```

### 10.5.2.7 Python Code to generate Figure 10.7 illustrating Categorical Cross-Entropy Loss

The Python code below produces the Figure 10.7 illustrating Categorical Cross-Entropy Loss.

```python
1
2
3

import numpy as np
import matplotlib.pyplot as plt
```
```python
4 # Predicted probability for the true class
5 p_true = np.linspace(0.001, 1.0, 500) # avoid log(0)
6
7 # Categorical Cross-Entropy loss
8 cce_loss = -np.log(p_true)
9
10 # Plot
11 plt.figure(figsize=(8,6))
12 plt.plot(p_true, cce_loss, lw=2, color='blue', label="CCE Loss")
13
14 # Labels and formatting
15 plt.xlabel("Predicted Probability of True Class $p_{y}$", fontsize=12)
16 plt.ylabel("Categorical Cross-Entropy Loss", fontsize=12)
17 plt.title("Categorical Cross-Entropy Loss Function", fontsize=14)
18 plt.grid(True)
19
20 # Highlight typical probabilities
21 plt.axvline(0.5, color='red', linewidth=1, label='p=0.5')
22 plt axvline(0.9, color='green', linewidth=1, label='p=0.9')
23 plt.legend()
24
25 plt.show()
```

### 10.5.2.8 Python Code to generate Figure 10.8 illustrating Categorical Cross-Entropy Loss

The Python code below produces the Figure 10.8 illustrating Categorical Cross-Entropy Loss.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3 from mpl_toolkits.mplot3d import Axes3D
4
5 # Create a grid for p0 and p1 (p2 will be 1 - p0 - p1)
6 p0 = np.linspace(0.001, 0.998, 100)
7 p1 = np.linspace(0.001, 0.998, 100)
8 P0, P1 = np.meshgrid(p0, p1)
9
10 # Compute p2
11 P2 = 1 - P0 - P1
12
13 # Initialize CCE loss arrays for each true class
14 loss_class0 = np full_like(P0, np nan)
15 loss_class1 = np full_like(P0, np nan)
16 loss_class2 = np full_like(P0, np nan)
17
18 # Compute CCE loss where probabilities are valid (p2 > 0)
19 mask = P2 > 0
20 loss_class0 [mask] = -np log(P0 [mask])
21 loss_class1 [mask] = -np log(P1 [mask])
22 loss_class2 [mask] = -np log(P2 [mask])
23
24 # Plot 3D surfaces
25 fig = plt figure(figsize=(12,6))
26
27 # True class 0
```
```python
28 ax = fig.add_subplot(131, projection='3d')
29 surf0 = ax plot Cole P0, P1, loss_class0, cm = 'viridis', edgecolor = 'k', alpha = 0.9)
30 ax.set xlabel("Predicted p0")
31 ax. set ylabel("Predicted p1")
32 ax. set_zlabel("CCE Loss")
33 ax. set title("CCE Loss: True Class 0")
34 fig. colorbar(surf0, shrink = 0.5, aspect = 10)
35
36 # True class 1
37 ax = fig. add_subplot(132, projection='3d')
38 surf1 = ax. plot Cole P0, P1, loss_class1, cm = 'plasma', edgecolor='k', alpha = 0.9)
39 ax. set xlabel("Predicted p0")
40 ax. set ylabel("Predicted p1")
41 ax. set_zlabel("CCE Loss")
42 ax. set title("CCE Loss: True Class 1")
43 fig. colorbar(surf1, shrink = 0.5, aspect = 10)
44
45 # True class 2
46 ax = fig. add_subplot(133, projection='3d')
47 surf2 = ax. plot Cole P0, P1, loss_class2, cm = 'inferno', edgecolor='k', alpha = 0.9)
48 ax. set xlabel("Predicted p0")
49 ax. set ylabel("Predicted p1")
50 ax. set_zlabel("CCE Loss")
51 ax. set title("CCE Loss: True Class 2")
52 fig. colorbar(surf2, shrink = 0.5, aspect = 10)
53
54 plt. tight layout()
55 plt. show()
```

### 10.5.2.9 Python Code to generate Figure 10.9 illustrating Sparse Categorical Cross-Entropy Loss

The Python code below produces the Figure 10.9 illustrating Sparse Categorical Cross-Entropy Loss.

```python
1 import numpy as np
2 import matplotlib. pyplot as plt
3
4 # Number of classes
5 num_classes = 3
6
7 # Predicted probability for the true class (range 0 to 1)
8 p_true = np. linspace(0.001, 1.0, 500) # avoid log(0)
9
10 plt. figure(figure size=(8,6))
11
12 # Plot SCCE loss for each possible true class
13 for true_class in range(num_classes):
14     sparse_ce_loss = -np. log(p_true) # SCCE only depends on p_y for the
15     true_class
16     plt. plot(p_true, sparse_ce_loss, lw=2, label="True class = {
17         true_class}")
```
```python
16
17
# Labels and title
18
plt.xlabel("Predicted Probability of True Class $p_y$", fontsize=12)
19
plt.ylabel("Sparse Categorical Cross-Entropy Loss", fontsize=12)
20
plt.title(f"Sparse Categorical Cross-Entropy Loss Function ({num_classes}
) classes)", fontsize=14)
21
plt.grid(True)
22
plt.legend()
23
24
# Highlight key probabilities
25
plt axvline(0.5, color='red', linewidth=1, label='p=0.5')
26
plt axvline(0.9, color='green', linewidth=1, label='p=0.9')
27
plt.show()
```

### 10.5.2.10 Python Code to generate Figure 10.10 illustrating Surface Plot of Sparse Categorical Cross-Entropy Loss

The Python code below produces the Figure 10.10 illustrating Surface Plot of Sparse Categorical Cross-Entropy Loss.

```python
1
import numpy as np
2
import matplotlib.pyplot as plt
3
from mpl_toolkits.mplot3d import Axes3D
4
5
# Create a grid for p0 and p1
6
p0 = np.linspace(0.001, 0.998, 100)
7
p1 = np.linspace(0.001, 0.998, 100)
8
P0, P1 = np.meshgrid(p0, p1)
9
10
# Initialize arrays for p2 and SCCE loss
11
P2 = 1 - P0 - P1
12
loss_class0 = np FullLike(P0, np nan)
13
loss_class1 = np fullLike(P0, np nan)
14
loss_class2 = np fullLike(P0, np nan)
15
16
# Compute SCCE loss where probabilities are valid (p2 > 0)
17
mask = P2 > 0
18
loss_class0[mask] = -np log(P0[mask])
19
loss_class1[mask] = -np log(P1[mask])
20
loss_class2[mask] = -np log(P2[mask])
21
22
# Plot 3D surface for each true class
23
fig = plt figure(figsize=(12,6))
24
25
# True class 0
26
ax = fig.add_subplot(131, projection='3d')
27
surf0 = ax plot surface(P0, P1, loss_class0, colormap='viridis', edgecolor='k', alpha=0.9)
28
ax.set xlabel("Predicted p0")
29
ax.set ylabel("Predicted p1")
30
ax.set zlabel("Loss")
31
ax.set title("SCCE Loss: True Class 0")
32
fig colorbar(surf0, shrink=0.5, aspect=10)
33
34
# True class 1
```
```python
35 ax = fig.add_subplot(132, projection='3d')
36 surf1 = ax.plot_surface(P0, P1, loss_class1, cmap='plasma', edgecolor='k', alpha=0.9)
37 ax.set_xlabel("Predicted p0")
38 ax.set_ylabel("Predicted p1")
39 ax.set_zlabel("Loss")
40 ax.set_title("SCCE Loss: True Class 1")
41 fig.colorbar(surf1, shrink=0.5, aspect=10)
42
43 # True class 2
44 ax = fig.add_subplot(133, projection='3d')
45 surf2 = ax.plot(P0, P1, loss_class2, cmap='inferno', edgecolor='k', alpha=0.9)
46 ax.set_xlabel("Predicted p0")
47 ax.set_ylabel("Predicted p1")
48 ax.set_zlabel("Loss")
49 ax.set_title("SCCE Loss: True Class 2")
50 fig.colorbar(surf2, shrink=0.5, aspect=10)
51
52 plt.tight_layout()
53 plt.show()
```

### 10.5.2.11 Python Code to generate Figure 10.11 illustrating KL Divergence

The Python code below produces the Figure 10.11 illustrating KL Divergence.

```python
1 import numpy as np
2 import matplotlib.pyplot as plt
3
4 # Define discrete probability distribution P (fixed)
5 P = np.array([0.1, 0.2, 0.3, 0.4])
6
7 # Q will vary along one entry (e.g., Q[0] = q1), rest normalized
8 q1_values = np.linspace(0.01, 0.9, 500)
9 kl_divergence = []
10
11 for q1 in q1_values:
12     # Ensure Q is a valid distribution
13     Q = np.array([q1, 0.2, 0.3, 0.3 + (0.4 - q1)]) # sum = 1
14     kl = np.sum(P * np.log(P / Q))
15     kl_divergence.append(kl)
16
17 kl_divergence = np.array(kl_divergence)
18
19 # Plot
20 plt.figure(figsize=(8,6))
21 plt.plot(q1_values, kl_divergence, lw=2)
22 plt.xlabel("r"$Q_1$ value", fontsize=12)
23 plt.ylabel("KL Divergence $D_{KL}(P \ | \ Q)$", fontsize=12)
24 plt.title("KL Divergence as a function of $Q_1$", fontsize=14)
25 plt.grid(True)
26 plt.show()
```
### 10.5.2.12 Python Code to generate Figure 10.12 illustrating 2D Surface Plot of KL Divergence

The Python code below produces the Figure 10.12 illustrating 2D Surface Plot of KL Divergence.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

# Fixed discrete probability distribution P
P = np.array([0.1, 0.2, 0.3, 0.4])

# Create a grid for Q1 and Q2
q1 = np.linspace(0.01, 0.8, 100)
q2 = np.linspace(0.01, 0.8, 100)
Q1, Q2 = np.meshgrid(q1, q2)

# Compute KL divergence
KL = np.zeros_like(Q1)

for i in range(Q1.shape[0]):
    for j in range(Q1.shape[1]):
        q1_val = Q1[i, j]
        q2_val = Q2[i, j]
        q3_val = 0.3  # keep P[2] as reference or fix
        q4_val = 1 - (q1_val + q2_val + q3_val)  # ensure sum(Q) = 1
        # Avoid negative probabilities
        if q4_val <= 0:
            KL[i, j] = np.nan
        else:
            Q = np.array([q1_val, q2_val, q3_val, q4_val])
            KL[i, j] = np.sum(P * np.log(P / Q))

# Plot 3D surface
fig = plt.figure(figsize=(10,7))
ax = fig.add_subplot(111, projection='3d')
surf = ax.plot(Q1, Q2, KL, cmap='viridis', edgecolor='k', alpha
                    =0.9)

ax.set_xlabel(r"$Q_1$", fontsize=12)
ax.set_ylabel(r"$Q_2$", fontsize=12)
ax.set_zlabel(r"KL Divergence $D_{KL}$ (P \| Q)", fontsize=12)
ax.set_title("2D Surface Plot of KL Divergence", fontsize=14)
fig.colorbar(surf, shrink=0.5, aspect=10, label='KL Divergence')

plt.show()
```

## 10.5.3 Advanced and Specialized Loss Functions

### 10.5.3.1 Loss Functions for Generative Adversarial Networks (GANs)

#### 10.5.3.1.1 Wasserstein Loss (Earth Mover's Distance)

The Wasserstein Loss, also known as the Earth Mover's Distance (EMD) or Kantorovich-Rubinstein metric, is a fundamental function from optimal transport theory that provides a mathematically
rigorous and continuous measure of the distance between two probability distributions. Unlike the Kullback-Leibler (KL) or Jensen-Shannon (JS) divergences, which can be undefined or exhibit vanishing gradients when the distributions have disjoint supports—a common scenario in the low-dimensional manifolds of high-dimensional data like images—the Wasserstein Distance is defined even for non-overlapping distributions and yields a smooth, meaningful gradient that is essential for stable optimization in deep learning, particularly in Generative Adversarial Networks (GANs). For two probability distributions $P_r$ (real data) and $P_\theta$ (generated data) defined on a metric space $(\mathcal{X}, d)$, the p-th Wasserstein distance is defined as the minimum cost of transporting mass from $P_\theta$ to $P_r$, where the cost is taken to be the p-th power of the distance. For $p = 1$, which is most common in machine learning, it is given by:

$$W_1(P_r, P_\theta) = \inf_{\gamma \in \Pi(P_r, P_\theta)} \mathbb{E}_{(x,y) \sim \gamma} [d(x, y)]. \quad (10.120)$$

Here, $\Pi(P_r, P_\theta)$ denotes the set of all joint distributions $\gamma(x, y)$ whose marginal distributions are $P_r$ and $P_\theta$, i.e.,

$$\int \gamma(x, y) dy = P_r(x) \quad (10.121)$$

and

$$\int \gamma(x, y) dx = P_\theta(y) \quad (10.122)$$

This primal formulation directly captures the intuition of finding the most efficient transport plan $\gamma$ that moves the probability mass from $P_\theta$ to $P_r$, with the cost proportional to the distance $d(x, y)$. Computing the infimum over all possible couplings $\gamma$ is intractable for high-dimensional problems.

<figure><img src="image_93.png" /><figcaption>Figure 10.13: 1D Wasserstein Distance of two distributions having parameters: Normal($\mu = -1, \sigma = 1$) and Normal($\mu = 1.5, \sigma = 1$)</figcaption></figure>

However, a transformative result from duality theory, the Kantorovich-Rubinstein duality, provides a computationally feasible equivalent formulation. The duality states that:

$$W_1(P_r, P_\theta) = \sup_{\|f\|_L \le 1} \left[ \mathbb{E}_{x \sim P_r} [f(x)] - \mathbb{E}_{y \sim P_\theta} [f(y)] \right] \quad (10.123)$$
where the supremum is taken over all 1-Lipschitz functions $f: \mathcal{X} \to \mathbb{R}$, meaning functions that satisfy

$$|f(x) - f(y)| \le d(x, y) \qquad (10.124)$$

for all $x, y \in \mathcal{X}$. This dual form is the foundation of the Wasserstein Loss in Wasserstein GANs (WGANs). Instead of minimizing the distance directly, one maximizes the difference in expectations between the real and generated data over a family of Lipschitz-constrained functions, represented by a critic network $f_w$ with parameters $w$. The training objective becomes a minimax game:

$$\min_{\theta} \max_{\|f_w\|_L \le 1} \left[ \mathbb{E}_{x \sim P_r} [f_w(x)] - \mathbb{E}_{z \sim p(z)} [f_w(g_{\theta}(z))] \right] \qquad (10.125)$$

where $g_{\theta}$ is the generator mapping noise $z$ to data space. The critic $f_w$ is trained to maximize this expression, effectively estimating the Wasserstein distance, while the generator $g_{\theta}$ is trained to minimize it by making the generated distribution $P_{\theta}$ indistinguishable from $P_r$ under the critic's assessment. The critical mathematical detail is the enforcement of the 1-Lipschitz constraint on the

<figure><img src="image_94.png" /><figcaption>Figure 10.14: Multidimensional Wasserstein Distance of two distributions having parameters, Distribution A: $\mathcal{N}(\mu = [0, 0], \Sigma = I)$, Distribution B: $\mathcal{N}(\mu = [3, 3], \Sigma = I)$</figcaption></figure>

critic function $f_w$. Without this constraint, the maximization could diverge to infinity, rendering
the loss meaningless. Several techniques exist to enforce this constraint approximately. The original WGAN paper proposed weight clipping, constraining the weights $w$ of the critic to a compact space $[-c, c]$, which ensures the network is K-Lipschitz for some $K$ but can lead to pathological behavior like capacity underuse or gradient vanishing. A more effective and widely adopted method is the Gradient Penalty (WGAN-GP), which adds a regularization term to the loss that directly enforces the Lipschitz constraint on the interpolated data points. The WGAN-GP loss for the critic is:

$$L_{\text{critic}} = \mathbb{E}_{\tilde{x} \sim P_{\theta}}[f_{w}(\tilde{x})] - \mathbb{E}_{x \sim P_{r}}[f_{w}(x)] + \lambda \mathbb{E}_{\tilde{x} \sim P_{\tilde{x}}}[(\|\nabla_{\tilde{x}} f_{w}(\tilde{x})\|_{2} - 1)^{2}] \quad (10.126)$$

where $\hat{x}$ is a random point sampled along straight lines between points from $P_r$ and $P_\theta$, i.e.,

$$\hat{x} = \epsilon x + (1 - \epsilon) \tilde{x} \quad (10.127)$$

with $\epsilon \sim U[0, 1]$, $x \sim P_r$, $\tilde{x} \sim P_\theta$. This term penalizes the gradient norm of the critic at these points, deviating from 1, thus encouraging it to be 1-Lipschitz. The generator's loss remains

$$L_{\text{generator}} = -\mathbb{E}_{z \sim p(z)}[f_w(g_{\theta}(z))] \quad (10.128)$$

The differentiability and continuity of the Wasserstein Loss under this formulation provide stable, linear gradients that do not saturate, even when the critic is optimal, solving the key training instability problems of traditional GANs that use JS divergence. The value of the Wasserstein Loss itself also correlates well with the perceived quality of the generated samples, making it a useful diagnostic metric during training.

<figure><img src="image_95.png" /><figcaption>Figure 10.15: Comparison: Equal CDF, WGAN Critic, GAN Discriminator</figcaption></figure>

### 10.5.3.1.2 Least Squares GAN (LSGAN) Loss

The Least Squares Generative Adversarial Network (LSGAN) loss function is a modification of the standard Generative Adversarial Network (GAN) objective that replaces the binary cross-entropy loss with a least squares term, fundamentally altering the properties of the optimization landscape to improve training stability and mitigate the vanishing gradients problem inherent in the original
formulation. The core theoretical motivation stems from the observation that minimizing the Pearson $\chi^2$ divergence between the model distribution $P_\theta$ and the data distribution $P_r$ can be achieved through a least squares cost function, which provides smoother and more non-vanishing gradients, especially when updating the generator on samples that are on the correct side of the decision boundary but are still far from the real data manifold. In the standard GAN framework, the discriminator $D$ (often called the critic in this context) is trained to output a probability that a sample is real, using the sigmoid activation and binary cross-entropy loss. LSGAN re-frames this: the discriminator $D$ is now a function that assigns a real-valued score to inputs, and its objectives for real and fake data are defined by the least squares error. Specifically, for a real sample $x \sim P_r$, the desired discriminator output is $a$, and for a generated sample $G(z) \sim P_\theta$ where $z \sim p(z)$, the desired output is $b$, with $a$ and $b$ being hyperparameters typically set to $a = 1, b = 0$. The loss functions for the discriminator $D$ and generator $G$ are thus formulated as:

$$ \min_D L_{\text{LSGAN}}(D) = \frac{1}{2} \mathbb{E}_{x \sim P_r} [(D(x) - a)^2] + \frac{1}{2} \mathbb{E}_{z \sim p(z)} [(D(G(z)) - b)^2], \quad (10.129) $$

$$ \min_G L_{\text{LSGAN}}(G) = \frac{1}{2} \mathbb{E}_{z \sim p(z)} [(D(G(z)) - c)^2]. \quad (10.130) $$

Here, $c$ is another hyperparameter denoting the value that the generator wants the discriminator to believe for its generated samples; a common choice is $c = a = 1$, meaning the generator aims to make its samples indistinguishable from real data by pushing their discriminator scores towards 1. The choice of these targets $a, b$, and $c$ is not arbitrary; it has a theoretical basis in minimizing a specific $f$-divergence. The objective for the discriminator is to minimize the squared error of its predictions relative to the target labels $a$ (real) and $b$ (fake), effectively performing regression rather than classification. This formulation ensures that even if a generated sample is correctly classified as fake (i.e., $D(G(z))$ is close to $b$), it still incurs a significant loss if it is far from the decision boundary, providing a strong gradient signal to the generator to improve. Conversely, in the standard GAN, a generated sample that is confidently wrong (far on the wrong side of the decision boundary) results in a saturated sigmoid and a vanishingly small gradient. The mathematical optimality of the LSGAN can be analyzed by solving for the optimal discriminator $D^*$ for a fixed generator $G$. For any given generator, the discriminator objective is to minimize the expected squared error. Assuming the densities $p_r(x)$ for real data and $p_g(x)$ for generated data exist, the objective can be written as an integral over the input space:

$$ L(D) = \frac{1}{2} \int_{\mathcal{X}} [(D(x) - a)^2 p_r(x) + (D(x) - b)^2 p_g(x)] dx. \quad (10.131) $$

To find the function $D$ that minimizes this, we take the functional derivative with respect to $D(x)$ and set it to zero:

$$ \frac{\delta L}{\delta D(x)} = (D(x) - a)p_r(x) + (D(x) - b)p_g(x) = 0. \quad (10.132) $$

Solving for $D(x)$ yields the optimal discriminator:

$$ D^*(x) = \frac{ap_r(x) + bp_g(x)}{p_r(x) + p_g(x)}. \quad (10.133) $$

Substituting this optimal discriminator $D^*$ back into the generator's objective function reveals the quantity that the generator is ultimately minimizing. With $c$ as the generator's target, the generator loss becomes:

$$ L_G(G) = \frac{1}{2} \int_{\mathcal{X}} (D^*(x) - c)^2 p_g(x) dx \quad (10.134) $$

By plugging in the expression for $D^*(x)$, this loss can be shown to be proportional to the Pearson $\chi^2$ divergence between $p_r(x) + p_g(x)$ and $2p_g(x)$, plus a constant, for specific choices of $a, b$, and $c$. The most common setting $a = 1, b = 0, c = 1$ leads to the generator minimizing

$$ \frac{1}{2} \chi_{\text{Pearson}}^2 (p_r + p_g \parallel 2p_g) \quad (10.135) $$
<figure><img src="image_96.png" /><figcaption>Figure 10.16: Least Squares GAN (LSGAN) Loss Functions</figcaption></figure>

This divergence is smoother and better behaved than the Jensen-Shannon divergence targeted by the original GAN, leading to more stable training dynamics. The gradients for the generator are given by

$$ \nabla_{\theta_G} L_G = \mathbb{E}_{z \sim p(z)} [(D(G(z)) - c) \cdot \nabla_{\theta_G} D(G(z))] \quad (10.136) $$

Since $(D(G(z)) - c)$ is a linear term, it does not saturate, ensuring that the generator receives a usable gradient as long as the discriminator provides a meaningful score $D(G(z))$, which it does due to the regression task. This property is the primary advantage of LSGAN: it avoids the vanishing gradient problem that plagues standard GANs when the discriminator becomes too confident, thereby facilitating more reliable convergence and often producing higher quality generated samples.

### 10.5.3.1.3 Adversarial Loss (Standard GAN Loss)

The Adversarial Loss, as formulated in the original Generative Adversarial Network (GAN) framework, establishes a two-player minimax game between a generator network $G$ and a discriminator network $D$, where the solution is a Nash equilibrium. The generator $G$ maps samples from a prior noise distribution $p_z(z)$ (typically a multivariate Gaussian or uniform distribution) to the data space, synthesizing samples $G(z)$ intended to resemble real data. The discriminator $D$ is a binary classifier that outputs a probability $D(x)$ that a given input $x$ originates from the real data distribution $p_{\text{data}}(x)$ rather than the generator's distribution $p_g$. The objective function for this game is given by the value function $V(G, D)$:

$$ \min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}(x)}[\log D(x)] + \mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))] \quad (10.137) $$

This formulation derives from the principle of binary cross-entropy minimization. For a fixed
generator $G$, the optimal discriminator $D_G^*(x)$ is the function that maximizes this expression for every $x$. The argument inside the expectation for the discriminator is

$$p_{\text{data}}(x) \log D(x) + p_g(x) \log(1 - D(x)) \qquad (10.138)$$

To find the maximizing $D(x)$ for a fixed $x$, we take the functional derivative with respect to $D(x)$ and set it to zero, yielding the condition:

$$\frac{p_{\text{data}}(x)}{D(x)} - \frac{p_g(x)}{1 - D(x)} = 0 \qquad (10.139)$$

Solving this equation provides the optimal discriminator for a fixed generator:

$$D_G^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}. \qquad (10.140)$$

Substituting this optimal discriminator back into the value function $V(G, D)$ reveals the global loss function that the generator is ultimately minimizing:

$$V(G, D_G^*) = \mathbb{E}_{x \sim p_{\text{data}}} \left[ \log \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)} \right] + \mathbb{E}_{x \sim p_g} \left[ \log \frac{p_g(x)}{p_{\text{data}}(x) + p_g(x)} \right]. \qquad (10.141)$$

This expression can be rewritten in terms of the Kullback-Leibler (KL) divergence and the Jensen-Shannon (JS) divergence. Specifically, it equals:

$$V(G, D_G^*) = 2 \cdot D_{\text{JS}}(p_{\text{data}} \| p_g) - 2 \log 2, \qquad (10.142)$$

where the Jensen-Shannon divergence is defined as

$$D_{\text{JS}}(P \| Q) = \frac{1}{2} D_{\text{KL}}(P \| M) + \frac{1}{2} D_{\text{KL}}(Q \| M) \qquad (10.143)$$

with

$$M = \frac{P + Q}{2} \qquad (10.144)$$

Thus, the global minimum of the generator's loss is achieved if and only if $p_g = p_{\text{data}}$, at which point $D_G^*(x) = \frac{1}{2}$ for all $x$, and $V(G, D_G^*) = -\log 4$. The training process involves alternating gradient-based updates for $D$ and $G$. The discriminator update maximizes $V(D, G)$ by ascending its stochastic gradient:

$$\nabla_{\theta_d} \frac{1}{m} \sum_{i=1}^{m} [\log D(x^{(i)}) + \log(1 - D(G(z^{(i)})))] , \qquad (10.145)$$

where $\{x^{(i)}\}_{i=1}^m$ is a minibatch from the real data and $\{z^{(i)}\}_{i=1}^m$ is a minibatch from the noise prior. The generator update minimizes $V(G, D)$ by descending its stochastic gradient. However, in early training, when $D(G(z))$ is close to zero, the gradient of $\log(1 - D(G(z)))$ vanishes, providing no learning signal. To circumvent this, practitioners often maximize

$$\mathbb{E}_{z \sim p_z(z)}[\log D(G(z))] \qquad (10.146)$$

instead of minimizing

$$\mathbb{E}_{z \sim p_z(z)}[\log(1 - D(G(z)))] \qquad (10.147)$$

which provides stronger gradients early on, though it changes the theoretical interpretation. The gradient for this modified generator objective is:

$$\nabla_{\theta_g} \frac{1}{m} \sum_{i=1}^{m} \log D(G(z^{(i)})). \qquad (10.148)$$
<figure><img src="image_97.png" /><figcaption>Figure 10.17: Standard GAN (Adversarial) Loss Functions</figcaption></figure>

Despite the theoretical guarantee of convergence to the true data distribution, the training of standard GANs is notoriously unstable due to several factors: the simultaneous gradient descent on a non-convex game often leads to oscillatory behavior rather than convergence to the Nash equilibrium; the supports of $p_{\text{data}}$ and $p_g$ may be disjoint in high-dimensional spaces, causing the JS divergence to be constant and gradients to be zero; and mode collapse can occur, where the generator maps multiple noise inputs to the same output, failing to capture the full diversity of the data distribution. The adversarial loss thus represents a foundational but practically challenging objective that has spurred numerous variants designed to stabilize training and improve performance.

### 10.5.3.2 Loss Functions for Siamese and Metric Learning

#### 10.5.3.2.1 Contrastive Loss

Contrastive Loss is a distance-based loss function fundamental to metric learning and siamese network architectures, designed to learn an embedding space where similar data pairs are projected closer together while dissimilar pairs are pushed apart by a margin. The core objective is to train a parameterized embedding function $f_{\theta}(x)$ that maps input samples into a low-dimensional manifold where the Euclidean distance

$$d(f_{\theta}(x_i), f_{\theta}(x_j)) = \|f_{\theta}(x_i) - f_{\theta}(x_j)\|_2 \qquad (10.149)$$

directly reflects semantic similarity. For a training pair of samples $(x_i, x_j)$, a label $Y$ is defined where $Y = 0$ if the pair is similar (positive pair) and $Y = 1$ if dissimilar (negative pair). The loss
<figure><img src="image_98.png" /><figcaption>Figure 10.18: Contrastive Loss Functions</figcaption></figure>

function for a single pair is formulated as:

$$L(x_i, x_j, Y) = (1 - Y) \cdot \frac{1}{2} d^2(f_{\theta}(x_i), f_{\theta}(x_j)) + Y \cdot \frac{1}{2} \max(0, m - d(f_{\theta}(x_i), f_{\theta}(x_j)))^2. \quad (10.150)$$

Here, $m > 0$ is a hyperparameter defining the margin—the minimum distance that dissimilar pairs should maintain. The first term, $\frac{1}{2}d^2$, applies to positive pairs ($Y = 0$) and minimizes the squared Euclidean distance between their embeddings, directly pulling them together. The second term applies to negative pairs ($Y = 1$) and is active only when their distance is less than the margin $m$; it penalizes the pair by the squared difference between the margin and their current distance, effectively pushing them apart until their separation is at least $m$. This loss function encourages the embedding function to create a structured space where intra-class variance is minimized and inter-class variance is maximized, with the margin providing a buffer zone to prevent excessive contraction of the entire dataset into a single point.

The mathematical properties of Contrastive Loss reveal its optimization dynamics. The gradient with respect to the embedding vectors drives the learning process. For a positive pair ($Y = 0$), the gradient of the loss with respect to the distance $d$ is $\nabla_d L = d$, which is linear and pulls the embeddings together with a force proportional to their current separation. For a negative pair ($Y = 1$) where $d < m$, the gradient is

$$\nabla_d L = -(m - d) \qquad (10.151)$$

which pushes the embeddings apart with a force that diminishes as the distance approaches the margin. If $d \ge m$ for a negative pair, the loss is zero and no gradient is produced. The gradients
<figure><img src="image_99.png" /><figcaption>Figure 10.19: Contrastive Loss Landscapes</figcaption></figure>

with respect to the embeddings themselves are derived through the chain rule. For a positive pair, the gradient for an embedding vector $f_{\theta}(x_i)$ is:

$$ \nabla_{f_{\theta}(x_i)} L = (f_{\theta}(x_i) - f_{\theta}(x_j)) \quad (\text{for } Y = 0). \qquad (10.152) $$

For a negative pair with $d < m$, the gradient is:

$$ \nabla_{f_{\theta}(x_i)} L = - \left(1 - \frac{d}{m}\right) (f_{\theta}(x_j) - f_{\theta}(x_i)) \quad (\text{for } Y = 1). \qquad (10.153) $$

These gradients show that the update magnitude for positive pairs is linear in the distance, while for negative pairs it is sublinear, scaling with $(1 - d/m)$. This structure helps prevent catastrophic collapse where all embeddings converge to zero, as the repulsive force for negative pairs counteracts the attractive force for positive pairs. The margin $m$ is critical; it defines the scale of the embedding space and prevents the loss from becoming trivial (e.g., all distances shrinking to zero). The overall loss for a minibatch is typically the average over all pairs. In practice, forming pairs can lead to quadratic complexity, but mining hard negatives—focusing on negative pairs that are currently too close—can improve efficiency and performance. Contrastive Loss is a foundational objective that has been extended in numerous ways, such as in triplet loss and modern self-supervised methods like SimCLR, but its core principle of contrasting similar and dissimilar pairs remains a pillar of representation learning.

### 10.5.3.2.2 Triplet Loss

Triplet Loss is a metric learning objective designed to learn an embedding function $f_{\theta}: \mathcal{X} \to \mathbb{R}^d$ that projects input data into a Euclidean space where semantically similar points are clustered and dissimilar points are separated by a distance proportional to their semantic difference. It operates on triplets of samples: an anchor sample $x_a$, a positive sample $x_p$ (semantically similar to the anchor, e.g., same class), and a negative sample $x_n$ (semantically dissimilar to the anchor, e.g., different class). The fundamental aim is to ensure that the distance between the anchor and positive is less than the distance between the anchor and negative by at least a predefined margin $m > 0$. Mathematically, this constraint is expressed as

$$ \|f_{\theta}(x_a) - f_{\theta}(x_p)\|_2^2 + m < \|f_{\theta}(x_a) - f_{\theta}(x_n)\|_2^2 \qquad (10.154) $$
3D Contrastive Loss Landscapes

<figure><img src="image_100.png" /><figcaption>Contrastive Loss Surface (Dissimilar Pair y=0)</figcaption></figure>

<figure><img src="image_101.png" /><figcaption>Figure 10.20: 3D Contrastive Loss Landscapes</figcaption></figure>

Triplet Loss Function

<figure><img src="image_102.png" /><figcaption>Figure 10.21: Triplet Loss Function</figcaption></figure>

and the loss for a single triplet is defined as the amount by which this inequality is violated:

$$L(x_a, x_p, x_n) = \max (0, \|f_{\theta}(x_a) - f_{\theta}(x_p)\|_2^2 - \|f_{\theta}(x_a) - f_{\theta}(x_n)\|_2^2 + m). \quad (10.155)$$

This is a hinge-like loss that is zero if the embedding function successfully pushes the negative pair beyond the margin relative to the positive pair, and positive otherwise, providing a gradient to
correct the embedding. The margin $m$ acts as a hyperparameter controlling the minimum desired separation between positive and negative pairs, preventing the trivial solution where all embeddings collapse to a single point and enforcing a meaningful topology in the latent space. The optimization

Triplet Loss Landscape

<figure><img src="image_103.png" /><figcaption>Figure 10.22: Triplet Loss Landscape</figcaption></figure>

dynamics of Triplet Loss are governed by the gradients with respect to the embedding vectors. Let

$$d_{ap} = \|f_{\theta}(x_a) - f_{\theta}(x_p)\|_2 \qquad (10.156)$$

and

$$d_{an} = \|f_{\theta}(x_a) - f_{\theta}(x_n)\|_2 \qquad (10.157)$$

denote the Euclidean distances for the positive and negative pairs, respectively. The loss is active only when

$$d_{ap}^2 - d_{an}^2 + m > 0 \qquad (10.158)$$

In this case, the gradients are computed as follows. The gradient with respect to the anchor embedding $f_{\theta}(x_a)$ is:

$$\nabla_{f_{\theta}(x_a)} L = 2((f_{\theta}(x_a) - f_{\theta}(x_n)) - (f_{\theta}(x_a) - f_{\theta}(x_p))) = 2(f_{\theta}(x_p) - f_{\theta}(x_n)). \qquad (10.159)$$

The gradient with respect to the positive embedding $f_{\theta}(x_p)$ is:

$$\nabla_{f_{\theta}(x_p)} L = 2(f_{\theta}(x_a) - f_{\theta}(x_p)). \qquad (10.160)$$

The gradient with respect to the negative embedding $f_{\theta}(x_n)$ is:

$$\nabla_{f_{\theta}(x_n)} L = -2(f_{\theta}(x_a) - f_{\theta}(x_n)). \qquad (10.161)$$
These gradients reveal that, when active, the loss pulls the anchor and positive together while pushing the anchor and negative apart. The update magnitudes are proportional to the current distances, providing a self-stabilizing effect: large violations yield large gradients, while small violations yield fine adjustments. The overall loss for a dataset is the average over all valid triplets, but sampling all possible triplets is computationally prohibitive. Thus, hard negative mining is

<figure><img src="image_104.png" /><figcaption>Figure 10.23: Triplet Loss Contour Plot</figcaption></figure>

critical: selecting triplets where the negative is currently closer to the anchor than the positive i.e.

$$d_{an} < d_{ap} \qquad (10.162)$$

or semi-hard negatives where

$$d_{an} < d_{ap} + m \qquad (10.163)$$

but the loss is still positive. This focuses learning on the most informative triplets that violate the margin constraint, improving efficiency and convergence. The success of Triplet Loss hinges on the careful balance between attractive and repulsive forces, and it has been foundational for tasks like face recognition and image retrieval, where learning a discriminative embedding is paramount.

### 10.5.3.2.3 Center Loss

Center Loss is a supervised learning objective function designed to enhance the discriminative power of learned features by simultaneously promoting inter-class dispersion and intra-class compactness in the feature space of a deep neural network. It operates alongside a traditional softmax cross-entropy loss, which primarily encourages features to be separable by maximizing the decision
boundary between different classes but does not explicitly enforce features of the same class to form tight, distinct clusters. The Center Loss addresses this by introducing a penalty term that pulls the deep features of each sample toward the center of its corresponding class, thereby reducing the intra-class variations that can impede robust classification, especially under challenging conditions like pose changes or illumination variations in face recognition.

<figure><img src="image_105.png" /><figcaption>Figure 10.24: Center Loss Function</figcaption></figure>

Mathematically, for a training dataset with $N$ samples and $C$ classes, let $x_i \in \mathbb{R}^d$ be the deep feature of the $i$-th sample belonging to class $y_i$. The Center Loss maintains a learnable center vector $c_{y_i} \in \mathbb{R}^d$ for each class, which is updated iteratively during training to represent the mean feature of that class. The Center Loss term for a single sample is defined as the squared Euclidean distance between the feature vector $x_i$ and its corresponding class center $c_{y_i}$:

$$L_C = \frac{1}{2} \|x_i - c_{y_i}\|_2^2.$$

The complete loss function is then a weighted combination of the softmax cross-entropy loss $L_S$ and the Center Loss $L_C$:

$$L = L_S + \lambda L_C = - \log \frac{e^{W_{y_i}^T x_i + b_{y_i}}}{\sum_{j=1}^{C} e^{W_j^T x_i + b_j}} + \frac{\lambda}{2} \|x_i - c_{y_i}\|_2^2,$$

where $\lambda > 0$ is a hyperparameter that balances the two loss components, $W_j$ and $b_j$ are the weight vector and bias for class $j$ in the final fully connected layer, and the softmax loss ensures inter-class separability.
<figure><img src="image_106.png" /><figcaption>Figure 10.25: Center Loss Visualization: Features vs Class Centers</figcaption></figure>

The optimization of Center Loss involves updating both the network parameters $\theta$ and the class centers $c_j$ through backpropagation. The gradients of the total loss $L$ with respect to the feature vector $x_i$ and the center $c_{y_i}$ are critical for training. The gradient of $L$ with respect to $x_i$ is:

$$ \frac{\partial L}{\partial x_i} = \frac{\partial L_S}{\partial x_i} + \lambda \frac{\partial L_C}{\partial x_i} = \left( \frac{\partial L_S}{\partial x_i} \right) + \lambda(x_i - c_{y_i}), $$

where $\frac{\partial L_S}{\partial x_i}$ is the gradient from the softmax loss, which pushes features toward the decision boundary of their class, while the Center Loss gradient $\lambda(x_i - c_{y_i})$ pulls features toward their class center. The centers $c_j$ are updated based on the features of all samples in the batch belonging to class $j$, but to avoid large perturbations caused by misclassified samples, the update uses a scaling factor $\alpha \in [0, 1]$ and is computed as:

$$ \Delta c_j = \frac{\sum_{i=1}^{N} \delta(y_i = j)(c_j - x_i)}{1 + \sum_{i=1}^{N} \delta(y_i = j)}, $$

where $\delta(\cdot)$ is the indicator function, and the centers are updated via

$$ c_j \leftarrow c_j - \alpha \Delta c_j $$

This update gradually moves each center toward the mean of its class features without being overly sensitive to mini-batch noise.

The interplay between the softmax loss and Center Loss ensures that features not only are separable but also exhibit small intra-class variance, leading to a more robust and generalized feature
<figure><img src="image_107.png" /><figcaption>Figure 10.26: 3D Center Loss Surface</figcaption></figure>

representation. The hyperparameter $\lambda$ controls the trade-off: too small a $\lambda$ weakens the intra-class compactness effect, while too large a $\lambda$ may overwhelm the softmax loss, causing features to collapse into their centers and degrading classification performance. Center Loss has proven effective in tasks requiring high discriminative feature learning, such as face verification and fine-grained recognition, by producing features that are both highly discriminative and invariant to intra-class variations.

### 10.5.3.3 Loss Functions for Style Transfer and Super-Resolution

#### 10.5.3.3.1 Perceptual Loss

Perceptual Loss is a feature-based loss function that quantifies the discrepancy between two images not in raw pixel space but in a high-dimensional feature space extracted by a pre-trained deep convolutional neural network (typically a VGG network trained on ImageNet), thereby measuring perceptual similarity rather than strict pixel-to-pixel correspondence. This approach is grounded in the hypothesis that features learned by a network trained for a high-level task (e.g., image classification) encode semantic and perceptual information that is more aligned with human visual perception than low-level pixel errors. Given a target image $y$ and a generated image $\hat{y}$, let $\phi^j(x)$ denote the feature map activations from the $j$-th layer of a pre-trained network $\phi$ when processing an image $x$. The Perceptual Loss between $y$ and $\hat{y}$ is defined as the Euclidean distance between their feature representations at one or more layers:

$$L_{\text{perceptual}}(y, \hat{y}) = \sum_{j} \frac{1}{C_j H_j W_j} \|\phi^j(y) - \phi^j(\hat{y})\|_2^2,$$

where $C_j$, $H_j$, and $W_j$ are the number of channels, height, and width of the feature map at layer $j$, and the normalization by the spatial dimensions ensures the loss is scale-invariant across layers. By leveraging hierarchical features from multiple layers (e.g., ReLU layers from VGG-16), the loss captures both high-level semantic content (from deeper layers) and finer texture details (from shallower layers), enabling the generation of images that are perceptually realistic even if they exhibit slight geometric or photometric deviations from the target. The mathematical rigor of Perceptual Loss arises from its interpretation as a metric in the feature manifold of a pre-trained network. The feature extractor $\phi$ serves as a non-linear mapping from the image space $\mathcal{I}$ to a feature space $\mathcal{F}$, where distances correlate with perceptual differences. The loss assumes that $\phi$ has learned a sufficiently rich and disentangled representation where semantically similar images are mapped to nearby points. The gradient of the loss with respect to the generated image $\hat{y}$ is computed through the pre-trained network:

$$\nabla_{\hat{y}} L_{\text{perceptual}} = \sum_{j} \frac{2}{C_j H_j W_j} (\phi^j(y) - \phi^j(\hat{y})) \cdot J_{\phi^j}(\hat{y}),$$
<figure><img src="image_108.png" /><figcaption>Figure 10.27: Perceptual Loss Function</figcaption></figure>

where $J_{\phi^j}(\hat{y})$ is the Jacobian of the feature map $\phi^j$ at $\hat{y}$. Since $\phi$ is pre-trained and fixed, these gradients guide the optimization process to adjust $\hat{y}$ in directions that minimize feature-space distance, effectively transferring perceptual attributes from $y$ to $\hat{y}$. This is particularly useful for tasks like style transfer, super-resolution, and image synthesis, where maintaining perceptual fidelity is more critical than pixel-level accuracy. However, the loss depends heavily on the choice of network $\phi$ and the layers $j$; deeper layers emphasize semantic content, while shallower layers capture spatial details. Variants of Perceptual Loss, such as using the LPIPS (Learned Perceptual Image Patch Similarity) metric, further improve by learning the feature space distance from human judgments, but the core principle remains: leveraging deep features as a perceptual metric for image comparison.

### 10.5.3.3.2 Style Loss

Style Loss is a specialized function used primarily in neural style transfer to quantify the difference in stylistic appearance between two images by comparing the statistical properties of their feature representations, specifically the correlations between filter responses encoded by the Gram matrix of feature activations from a pre-trained deep convolutional network. Unlike perceptual loss, which focuses on content similarity by matching feature maps directly, style loss operates on the premise that the style of an image is captured by the inter-channel correlations within feature maps, which correspond to textures, patterns, and visual motifs independent of the spatial structure. Given a feature map activation tensor $\phi^j(x) \in \mathbb{R}^{C_j \times H_j \times W_j}$ extracted from layer $j$ of a pre-trained network (e.g., VGG-19) for an image $x$, the Gram matrix $G^j(x) \in \mathbb{R}^{C_j \times C_j}$ is computed as the inner product between the vectorized feature maps across spatial dimensions, effectively measuring the covariance
<figure><img src="image_109.png" /><figcaption>Figure 10.28: Pixel-wise MSE vs Perceptual Loss (Nonlinear Features)</figcaption></figure>

between channels:

$$G_{kl}^{j}(x) = \frac{1}{C_{j}H_{j}W_{j}} \sum_{h=1}^{H_{j}} \sum_{w=1}^{W_{j}} \phi_{k,h,w}^{j}(x) \cdot \phi_{l,h,w}^{j}(x),$$

where k, l index feature channels. This matrix is symmetric and positive semi-definite, and its elements represent the degree to which features k and l co-occur spatially in the image. The style loss between a target style image $y_s$ and a generated image $\hat{y}$ is then defined as the mean squared error between their Gram matrices across a set of layers $\mathcal{J}$:

$$L_{\text{style}}(y_s, \hat{y}) = \sum_{j \in \mathcal{J}} \frac{1}{C_j^2} \|G^j(y_s) - G^j(\hat{y})\|_F^2,$$

where $\| \cdot \|_F$ denotes the Frobenius norm, and the normalization by $C_j^2$ accounts for the number of elements in the Gram matrix. This loss encourages the generated image to exhibit similar texture statistics to the style image by matching the second-order moments of feature distributions, effectively transferring stylistic attributes such as brushstrokes, color palettes, and patterns without preserving the spatial content. The mathematical foundation of style loss relies on the interpretation of the Gram matrix as an estimator of the feature distribution's covariance structure. Minimizing the Frobenius norm between Gram matrices is equivalent to minimizing the Euclidean distance between the vectorized representations of the feature correlations, which aligns the stylistic properties of the generated image with those of the target style. The gradient of the style loss with respect to the generated image $\hat{y}$ is computed through the chain rule:

$$\nabla_{\hat{y}} L_{\text{style}} = \sum_{j \in \mathcal{J}} \frac{2}{C_j^2} (G^j(y_s) - G^j(\hat{y})) \cdot \nabla_{\hat{y}} G^j(\hat{y}),$$
<figure><img src="image_110.png" /><figcaption>Figure 10.29: Style Loss Function in Neural Style Transfer</figcaption></figure>

where $\nabla_{\hat{y}} G^j(\hat{y})$ involves the Jacobian of the Gram matrix computation. This gradient updates $\hat{y}$ to increase or decrease specific feature correlations, effectively synthesizing textures that mimic the style image. The choice of layers $\mathcal{J}$ is critical: shallow layers capture low-level textures (e.g., edges, colors), while deeper layers encode higher-order style patterns (e.g., complex structures). In practice, style loss is combined with content loss (e.g., perceptual loss) and often a total variation regularization term to produce visually coherent stylized images. The efficacy of style loss stems from its ability to disentangle style from content by leveraging the statistical independence of feature correlations from spatial arrangements, making it a cornerstone of artistic style transfer and texture synthesis in computer vision.

### 10.5.3.3.3 Total Variation (TV) Loss

Total Variation (TV) Loss is a regularization term employed in image processing and inverse problems to impose spatial smoothness on an image by penalizing excessive and spurious variations in pixel intensities, thereby reducing noise and undesired high-frequency artifacts while preserving essential edges and structures. It is grounded in the mathematical theory of functions of bounded variation, where the total variation of a function is defined as the integral of the absolute value of its gradient, capturing the total magnitude of oscillations and discontinuities. For a discrete 2D image $x \in \mathbb{R}^{H \times W}$ with height $H$ and width $W$, the anisotropic Total Variation Loss is commonly defined as the sum of the absolute differences between adjacent pixels along the horizontal and vertical directions:

$$L_{\text{TV}}(x) = \sum_{i=1}^{H-1} \sum_{j=1}^{W} |x_{i+1,j} - x_{i,j}| + \sum_{i=1}^{H} \sum_{j=1}^{W-1} |x_{i,j+1} - x_{i,j}|.$$
<figure><img src="image_111.png" /><figcaption>Figure 10.30: Style Loss vs Scaling of Generated Features</figcaption></figure>

<figure><img src="image_112.png" /><figcaption>Figure 10.31: Total Variation (TV) Loss</figcaption></figure>

This formulation measures the $l_1$-norm of the discrete gradients, promoting sparsity in the image gradient domain and encouraging piecewise constant regions with sharp boundaries. An alternative isotropic Total Variation Loss combines the horizontal and vertical differences using the Euclidean
norm at each pixel:

$$L_{\text{TV}}^{\text{iso}}(x) = \sum_{i=1}^{H-1} \sum_{j=1}^{W-1} \sqrt{(x_{i+1,j} - x_{i,j})^2 + (x_{i,j+1} - x_{i,j})^2},$$

which more accurately approximates the continuous total variation but is computationally more intensive due to the square root operation. The primary effect of TV Loss is to suppress small fluctuations caused by noise or artifacts while preserving significant edges, as the $l_1$-norm imposes a stronger penalty on small gradients than the $l_2$-norm used in quadratic smoothness terms. The

<figure><img src="image_113.png" /><figcaption>Figure 10.32: Total Variation Loss vs Noise Level</figcaption></figure>

mathematical properties of TV Loss arise from its convexity and non-differentiability at zero. The gradient of the anisotropic TV Loss with respect to the image $x$ is straightforward but requires care at points of non-differentiability. For a single pixel $x_{i,j}$, the partial derivative of $L_{\text{TV}}$ is:

$$\frac{\partial L_{\text{TV}}}{\partial x_{i,j}} = \text{sign}(x_{i,j} - x_{i-1,j}) + \text{sign}(x_{i,j} - x_{i+1,j}) + \text{sign}(x_{i,j} - x_{i,j-1}) + \text{sign}(x_{i,j} - x_{i,j+1}),$$

where the 'sign' function is defined in the subgradient sense at zero (typically returning 0 for a zero difference). This gradient effectively acts as a diffusion term that reduces variations between neighbors, flattening regions with small gradients while leaving large jumps unchanged. In optimization problems, such as image reconstruction or generation, TV Loss is often added as a regularization term with a weight $\lambda$ to the primary loss function:

$$L_{\text{total}}(x) = L_{\text{data}}(x) + \lambda L_{\text{TV}}(x),$$

where $L_{\text{data}}$ ensures fidelity to observed data (e.g., mean squared error for reconstruction). The parameter $\lambda$ controls the trade-off between smoothness and data fidelity; too large a $\lambda$ oversmooths
the image, erasing fine details, while too small a $\lambda$ fails to suppress noise. TV Loss is particularly effective in tasks like image denoising, inpainting, and style transfer, where it helps produce visually plausible results by eliminating unstructured noise and creating piecewise smooth outputs. Its use in deep learning, such as in generative models, stabilizes training by preventing high-frequency artifacts that can arise from unconstrained optimization, ensuring that generated images adhere to natural image statistics characterized by sparse gradients.

### 10.5.3.4 Loss Functions for Uncertainty Estimation and Bayesian Deep Learning

#### 10.5.3.4.1 Evidence Lower Bound (ELBO) Loss

The Evidence Lower Bound (ELBO) loss is the foundational objective function in variational inference, a method used to approximate intractable posterior distributions in probabilistic models, most prominently in Variational Autoencoders (VAEs). It derives from the principle of maximizing the log-likelihood of the observed data $\log p_{\theta}(x)$, which is computationally infeasible due to the integration over latent variables $z$. Instead, variational inference introduces an approximate posterior distribution $q_{\phi}(z|x)$, parameterized by $\phi$, and seeks to minimize the Kullback-Leibler (KL) divergence between $q_{\phi}(z|x)$ and the true posterior $p_{\theta}(z|x)$. This divergence is non-negative and satisfies:

$$ \log p_{\theta}(x) = D_{\text{KL}}(q_{\phi}(z|x) \parallel p_{\theta}(z|x)) + \mathcal{L}(\theta, \phi; x), $$

where $\mathcal{L}(\theta, \phi; x)$ is the ELBO, defined as:

$$ \mathcal{L}(\theta, \phi; x) = \mathbb{E}_{z \sim q_{\phi}(z|x)}[\log p_{\theta}(x|z)] - D_{\text{KL}}(q_{\phi}(z|x) \parallel p_{\theta}(z)). $$

The ELBO provides a lower bound on the log-evidence $\log p_{\theta}(x)$, as the KL term is always non-negative. Maximizing the ELBO is equivalent to minimizing $D_{\text{KL}}(q_{\phi}(z|x) \parallel p_{\theta}(z|x))$, thus making $q_{\phi}(z|x)$ a better approximation of the true posterior. The first term in the ELBO, $\mathbb{E}_{z \sim q_{\phi}(z|x)}[\log p_{\theta}(x|z)]$, is the reconstruction loss, which encourages the decoder $p_{\theta}(x|z)$ to generate data that matches the observed input $x$. The second term, $-D_{\text{KL}}(q_{\phi}(z|x) \parallel p_{\theta}(z))$, acts as a regularizer, penalizing deviations of the approximate posterior from the prior $p_{\theta}(z)$, typically chosen as a standard Gaussian $N(0, I)$. This KL divergence term can often be computed analytically, especially when both $q_{\phi}(z|x)$ and $p_{\theta}(z)$ are Gaussian, ensuring computational efficiency. The optimization of the ELBO involves gradient-based methods, but the expectation term $\mathbb{E}_{z \sim q_{\phi}(z|x)}[\log p_{\theta}(x|z)]$ poses a challenge due to the non-differentiability of sampling from $q_{\phi}(z|x)$. This is addressed by the reparameterization trick, which expresses the latent variable as a deterministic function of parameters and an auxiliary noise variable: $z = g_{\phi}(\epsilon, x)$ with $\epsilon \sim p(\epsilon)$. For a Gaussian $q_{\phi}(z|x) = N(\mu_{\phi}(x), \sigma_{\phi}^{2}(x))$, this becomes

$$ z = \mu_{\phi}(x) + \sigma_{\phi}(x) \odot \epsilon $$

where $\epsilon \sim N(0, I)$. This transformation allows gradients to flow through the stochastic layer, enabling unbiased Monte Carlo estimates of the gradient:

$$ \nabla_{\theta,\phi} \mathbb{E}_{z \sim q_{\phi}(z|x)}[\log p_{\theta}(x|z)] = \mathbb{E}_{\epsilon \sim p(\epsilon)}[\nabla_{\theta,\phi} \log p_{\theta}(x|g_{\phi}(\epsilon, x))]. $$

The KL divergence term $D_{\text{KL}}(q_{\phi}(z|x) \parallel p_{\theta}(z))$ for Gaussian distributions has a closed-form expression:

$$ D_{\text{KL}} = -\frac{1}{2} \sum_{j=1}^{J} (1 + \log \sigma_{j}^{2} - \mu_{j}^{2} - \sigma_{j}^{2}), $$

where $J$ is the dimensionality of $z$, and $\mu_j$ and $\sigma_j$ are the parameters of $q_{\phi}(z|x)$. This term encourages the latent codes to remain close to the prior, preventing overfitting and promoting a structured latent space. The ELBO loss is thus:

$$ L_{\text{ELBO}}(\theta, \phi; x) = -\mathcal{L}(\theta, \phi; x) = -\mathbb{E}_{z \sim q_{\phi}(z|x)}[\log p_{\theta}(x|z)] + D_{\text{KL}}(q_{\phi}(z|x) \parallel p_{\theta}(z)), $$
<figure><img src="image_114.png" /><figcaption>Figure 10.33: Evidence Lower Bound (ELBO) Loss during Training</figcaption></figure>

which is minimized during training. The balance between reconstruction accuracy and latent regularization is automatic but can be tuned with a weight parameter $\beta$ in variants like $\beta$-VAE, which uses $\beta D_{KL}$ to emphasize disentangled representations. The ELBO framework ensures tractable optimization while providing a rigorous bound on the data likelihood, making it a cornerstone of modern deep generative models.

### 10.5.3.4.2 Negative Log-Likelihood (NLL) Loss

Negative Log-Likelihood (NLL) Loss is a fundamental objective function in statistical estimation and machine learning that arises from the principle of maximum likelihood estimation (MLE), aiming to find the parameters of a model that make the observed data most probable. Given a parametric model $p_{\theta}(x)$ that defines the probability of data point $x$ under parameters $\theta$, and assuming independent and identically distributed (i.i.d.) data samples $\{x_i\}_{i=1}^N$, the likelihood function is

$$ \mathcal{L}(\theta) = \prod_{i=1}^{N} p_{\theta}(x_i) $$

Maximizing this likelihood is equivalent to minimizing the negative log-likelihood, due to the monotonicity of the logarithm function, leading to the NLL loss:

$$ L_{\text{NLL}}(\theta) = -\frac{1}{N} \sum_{i=1}^{N} \log p_{\theta}(x_i). $$

The normalization by $N$ expresses the loss as an empirical expectation over the data distribution: $\mathbb{E}_{x \sim p_{\text{data}}[-\log p_{\theta}(x)]}$, which is the cross-entropy between the true data distribution $p_{\text{data}}(x)$ and the
<figure><img src="image_115.png" /><figcaption>Figure 10.34: Negative Log-Likelihood (NLL) Loss during Training</figcaption></figure>

model distribution $p_{\theta}(x)$. Minimizing the NLL loss thus minimizes the Kullback-Leibler divergence $D_{\text{KL}}(p_{\text{data}} \parallel p_{\theta})$, as

$$ \mathbb{E}_{x \sim p_{\text{data}}} [-\log p_{\theta}(x)] = H(p_{\text{data}}) + D_{\text{KL}}(p_{\text{data}} \parallel p_{\theta}) $$

where $H(p_{\text{data}})$ is the entropy of the data and is constant with respect to $\theta$. This establishes NLL as a direct measure of how well $p_{\theta}$ approximates $p_{\text{data}}$.

In practice, for discrete outcomes such as classification, $p_{\theta}(x_i)$ is often a categorical distribution where the model outputs a probability vector over classes. For a true label $y_i$ (represented as a one-hot vector) and predicted probabilities $\hat{y}_i$, the NLL loss for that sample is $-\log \hat{y}_{i,y_i}$, which is the cross-entropy loss. For continuous data, such as regression, $p_{\theta}(x_i)$ is typically a parametric distribution like a Gaussian $N(\mu_{\theta}, \sigma_{\theta}^2)$, and the NLL loss becomes:

$$ -\log p_{\theta}(x_i) = \frac{1}{2} \log(2\pi\sigma_{\theta}^2) + \frac{(x_i - \mu_{\theta})^2}{2\sigma_{\theta}^2}, $$

which couples the mean and variance terms. In heteroscedastic regression, where the variance is also predicted, minimizing NLL allows the model to learn uncertainty estimates automatically. The gradient of the NLL loss with respect to the parameters $\theta$ is:

$$ \nabla_{\theta} L_{\text{NLL}} = -\frac{1}{N} \sum_{i=1}^{N} \nabla_{\theta} \log p_{\theta}(x_i), $$

which is the negative of the gradient of the log-likelihood. For exponential family distributions, this gradient has favorable properties, often leading to efficient optimization. The Hessian of the
NLL loss is related to the Fisher information matrix

$$I(\theta) = \mathbb{E}_{x \sim p_{\theta}} [\nabla_{\theta} \log p_{\theta}(x) \nabla_{\theta} \log p_{\theta}(x)^{\top}]$$

which characterizes the curvature of the likelihood surface and plays a key role in natural gradient methods. NLL loss is widely used in deep learning for generative modeling (e.g., VAEs, autoregressive models), classification, and probabilistic regression, providing a statistically principled way to train models that output full probability distributions rather than point estimates. Its minimization ensures consistency and efficiency under correct model specification, per the asymptotic properties of MLE.

### 10.5.3.5 Loss Functions for Domain Adaptation

#### 10.5.3.5.1 Domain Adversarial Loss

Domain Adversarial Loss is a strategic objective function employed in domain adaptation to learn feature representations that are invariant across a source domain $S$ and a target domain $T$, thereby facilitating knowledge transfer from a label-rich source to an unlabeled target. The core idea is to

<figure><img src="image_116.png" /><figcaption>Figure 10.35: Domain Adversarial Loss during Training</figcaption></figure>

train a feature extractor $G_f(\cdot; \theta_f)$ that produces embeddings which are indistinguishable between domains, as judged by a domain classifier $G_d(\cdot; \theta_d)$ that acts as an adversary. This induces a minimax game: the feature extractor aims to confuse the domain classifier, while the domain classifier strives to correctly identify the domain origin of features. Formally, let $x_i^s$ and $x_j^t$ be samples from the source and target distributions $P_s$ and $P_t$, respectively. The feature extractor maps input $x$ to a feature vector $f = G_f(x)$. The domain classifier $G_d$ takes $f$ as input and outputs
a probability $G_d(f) \in [0,1]$ that the feature originates from the source domain. The domain adversarial loss is then defined as the binary cross-entropy loss for the domain classifier:

$$L_{\text{dom}}(\theta_f, \theta_d) = -\frac{1}{n_s} \sum_{i=1}^{n_s} \log G_d(G_f(x_i^s)) - \frac{1}{n_t} \sum_{j=1}^{n_t} \log (1 - G_d(G_f(x_j^t))),$$

where $n_s$ and $n_t$ are the number of source and target samples in the batch. The parameters $\theta_d$ of the domain classifier are optimized to minimize this loss, thus improving domain discrimination. Conversely, the parameters $\theta_f$ of the feature extractor are optimized to maximize this loss, effectively making the features domain-invariant. This adversarial dynamic is encapsulated in the overall objective:

$$\min_{\theta_f} \max_{\theta_d} L_{\text{dom}}(\theta_f, \theta_d).$$

In practice, this is achieved by gradient reversal during backpropagation: when updating $\theta_f$, the gradient from $L_{\text{dom}}$ is multiplied by $-\lambda$ (a hyperparameter) before being propagated through the feature extractor, thus implementing the maximization step.

The mathematical rigor of domain adversarial training stems from its connection to minimizing the $H$-divergence between the source and target feature distributions. Let $Q_s$ and $Q_t$ be the distributions of features $G_f(x)$ for $x \sim P_s$ and $x \sim P_t$, respectively. The domain classifier $G_d$ aims to approximate the optimal discriminator between $Q_s$ and $Q_t$, which achieves an error rate related to the Jensen-Shannon divergence. By training the feature extractor to maximize the domain classifier's loss, it effectively minimizes the divergence between $Q_s$ and $Q_t$, leading to aligned feature distributions. The overall loss for the system often combines the domain adversarial loss with a task-specific loss (e.g., classification loss on the source domain):

$$L_{\text{total}} = L_{\text{task}}(\theta_f, \theta_y) - \lambda L_{\text{dom}}(\theta_f, \theta_d),$$

where $\theta_y$ denotes the parameters of a task-specific classifier (e.g., label predictor) that uses the features $f$, and $\lambda$ controls the trade-off between task performance and domain invariance. The gradient updates are:

$$\theta_d \leftarrow \theta_d + \eta_d \nabla_{\theta_d} L_{\text{dom}}, \theta_y \leftarrow \theta_y - \eta_y \nabla_{\theta_y} L_{\text{task}}, \theta_f \leftarrow \theta_f - \eta_f (\nabla_{\theta_f} L_{\text{task}} - \lambda \nabla_{\theta_f} L_{\text{dom}}),$$

where the negative sign for $\theta_f$ on $L_{\text{dom}}$ implements the gradient reversal. This framework ensures that the learned features are both discriminative for the task and invariant to domain shifts, enabling effective unsupervised domain adaptation. The success of domain adversarial loss hinges on balancing the adversary's strength; too weak a classifier fails to align distributions, while too strong a one can overwhelm the task loss and degrade feature quality.

### 10.5.3.5.2 Maximum Mean Discrepancy (MMD) Loss

Maximum Mean Discrepancy (MMD) Loss is a non-parametric, kernel-based measure for comparing two probability distributions by evaluating the distance between their means in a reproducing kernel Hilbert space (RKHS), providing a rigorous framework for testing whether two samples are drawn from the same distribution without requiring density estimation. Given two distributions $P$ and $Q$ over a space $\mathcal{X}$, and a feature map $\phi: \mathcal{X} \to \mathcal{H}$ that projects samples into an RKHS $\mathcal{H}$ with kernel

$$k(x, y) = \langle \phi(x), \phi(y) \rangle_{\mathcal{H}}$$

the MMD is defined as the distance between the mean embeddings $\mu_P = \mathbb{E}_{x \sim P}[\phi(x)]$ and $\mu_Q = \mathbb{E}_{y \sim Q}[\phi(y)]$:

$$\text{MMD}^2(P, Q) = \|\mu_P - \mu_Q\|_{\mathcal{H}}^2.$$
<figure><img src="image_117.png" /><figcaption>Figure 10.36: Maximum Mean Discrepancy (MMD) Loss during Training</figcaption></figure>

Using the kernel trick, this can be expressed entirely in terms of kernel evaluations:

$$MMD^2(P, Q) = \mathbb{E}_{x,x' \sim P}[k(x, x')] + \mathbb{E}_{y,y' \sim Q}[k(y, y')] - 2\mathbb{E}_{x \sim P, y \sim Q}[k(x, y)],$$

which avoids explicit computation of the feature maps. For empirical samples $X = \{x_i\}_{i=1}^m \sim P$ and $Y = \{y_j\}_{j=1}^n \sim Q$, an unbiased estimator of $MMD^2$ is:

$$MMD_u^2(X, Y) = \frac{1}{m(m-1)} \sum_{i \neq j} k(x_i, x_j) + \frac{1}{n(n-1)} \sum_{i \neq j} k(y_i, y_j) - \frac{2}{mn} \sum_{i,j} k(x_i, y_j),$$

where the sums exclude diagonal terms to avoid bias. The choice of kernel $k$ is critical; common choices include the Gaussian kernel

$$k(x, y) = \exp(-\|x - y\|^2/(2\sigma^2))$$

where the bandwidth $\sigma$ controls the sensitivity to multi-scale structures in the data. The MMD loss is zero if and only if $P = Q$ when the kernel is characteristic (e.g., Gaussian), meaning the kernel map is injective, ensuring that the mean embedding captures all information about the distribution.

The mathematical properties of MMD make it particularly suitable for training generative models and domain adaptation. Minimizing MMD between the model distribution $Q_\theta$ and the data distribution $P$ provides a loss function that encourages the generator to produce samples that match the statistical properties of the real data. The gradient of the MMD loss with respect to the model parameters $\theta$ is:

$$\nabla_{\theta} MMD^2(P, Q_{\theta}) = -2\mathbb{E}_{x \sim P, y \sim Q_{\theta}}[\nabla_{\theta} k(x, y)] + \mathbb{E}_{y, y' \sim Q_{\theta}}[\nabla_{\theta} k(y, y')],$$
which requires sampling from $Q_\theta$ and differentiating through the kernel function. This gradient can be computed efficiently using automatic differentiation. In practice, for large datasets, minibatch estimates are used, though care must be taken with the biased estimator that includes diagonal terms when gradients are involved. The loss function is differentiable almost everywhere (for smooth kernels) and provides stable gradients, unlike adversarial losses that can suffer from vanishing gradients or mode collapse. MMD has deep connections to integral probability metrics and is equivalent to the distance between distributions under the function class induced by the kernel. Its computational complexity is $O((m+n)^2)$, which can be prohibitive for large batches, but linear-time approximations exist, such as:

$$MMD_l^2(X, Y) = \frac{2}{m} \sum_{i=1}^{m/2} [k(x_{2i-1}, x_{2i}) + k(y_{2i-1}, y_{2i}) - k(x_{2i-1}, y_{2i}) - k(x_{2i}, y_{2i-1})],$$

which uses non-overlapping pairs for an unbiased estimate with $O(m+n)$ complexity. MMD is widely used in domain adaptation to align feature distributions between source and target domains, in generative models like MMD-GANs, and in two-sample hypothesis testing, providing a robust and statistically powerful measure of distributional similarity.

### 10.5.3.6 Loss Functions for Object Detection and Segmentation

#### 10.5.3.6.1 IoU Loss (Intersection over Union) and its variants (GIoU, DIoU, CIoU)

The fundamental Intersection over Union (IoU) metric for two axis-aligned bounding boxes, a ground truth box A defined by its top-left and bottom-right coordinates $(x_1^g, y_1^g, x_2^g, y_2^g)$ and a predicted box B with coordinates $(x_1^p, y_1^p, x_2^p, y_2^p)$, is calculated by first determining the coordinates of their intersection $I$:

$$x_1^i = \max(x_1^p, x_1^g), \quad y_1^i = \max(y_1^p, y_1^g), \quad x_2^i = \min(x_2^p, x_2^g), \quad y_2^i = \min(y_2^p, y_2^g). \qquad (10.164)$$

The area of intersection is

$$|I| = \max(0, x_2^i - x_1^i) \times \max(0, y_2^i - y_1^i) \qquad (10.165)$$

The area of the union is computed from the areas of the individual boxes: $|U| = |A| + |B| - |I|$, where

$$|A| = (x_2^g - x_1^g)(y_2^g - y_1^g) \qquad (10.166)$$

and

$$|B| = (x_2^p - x_1^p)(y_2^p - y_1^p) \qquad (10.167)$$

The IoU is then:

$$\text{IoU} = \frac{|I|}{|U|}. \qquad (10.168)$$

The IoU Loss is simply

$$\mathcal{L}_{IoU} = 1 - \text{IoU} \qquad (10.169)$$

The gradient of this loss with respect to a predicted coordinate, say $x_1^p$, is found through careful application of the chain rule. The derivative is non-zero only if the boxes intersect ($|I| > 0$):

$$\frac{\partial \mathcal{L}_{IoU}}{\partial x_1^p} = -\frac{1}{|U|^2} \left( |U| \cdot \frac{\partial |I|}{\partial x_1^p} - |I| \cdot \frac{\partial |U|}{\partial x_1^p} \right). \qquad (10.170)$$

Since

$$\frac{\partial |U|}{\partial x_1^p} = -\frac{\partial |B|}{\partial x_1^p} + \frac{\partial |I|}{\partial x_1^p} = (y_2^p - y_1^p) + \frac{\partial |I|}{\partial x_1^p} \qquad (10.171)$$
and $\frac{\partial |I|}{\partial x_1^p}$ is itself a piecewise function:

$$ \frac{\partial |I|}{\partial x_1^p} = \begin{cases} -(y_2^i - y_1^i) & \text{if } x_1^g < x_1^p < x_2^g \text{ and } |I| > 0 \\ 0 & \text{otherwise} \end{cases}. \qquad (10.172) $$

This complexity and the zero gradient for non-overlapping boxes motivated the development of improved variants. Generalized IoU (GIoU) Loss introduces a convex enclosure C, the smallest box containing both A and B, with coordinates:

$$ x_1^c = \min(x_1^p, x_1^g), \quad y_1^c = \min(y_1^p, y_1^g), \quad x_2^c = \max(x_2^p, x_2^g), \quad y_2^c = \max(y_2^p, y_2^g). \qquad (10.173) $$

The area of C is $|C| = (x_2^c - x_1^c)(y_2^c - y_1^c)$. GIoU is defined as:

$$ \mathrm{GIoU} = \mathrm{IoU} - \frac{|C \setminus U|}{|C|} = \mathrm{IoU} - \frac{|C| - |U|}{|C|}. \qquad (10.174) $$

The GIoU Loss is $\mathcal{L}_{GIoU} = 1 - \mathrm{GIoU}$. Its gradient provides a signal even for non-overlapping boxes. The derivative with respect to a coordinate incorporates the new term:

$$ \frac{\partial \mathcal{L}_{GIoU}}{\partial x_1^p} = \frac{\partial \mathcal{L}_{IoU}}{\partial x_1^p} - \frac{1}{|C|^2} \left( |C| \cdot \frac{\partial |U|}{\partial x_1^p} - (|C| - |U|) \cdot \frac{\partial |C|}{\partial x_1^p} \right). \qquad (10.175) $$

The derivative $\frac{\partial |C|}{\partial x_1^p}$ is non-zero only if $x_1^p$ is the left-most point:

$$ \frac{\partial |C|}{\partial x_1^p} = \begin{cases} -(y_2^c - y_1^c) & \text{if } x_1^p < x_1^g \\ 0 & \text{otherwise} \end{cases}. \qquad (10.176) $$

This guides the predicted box to expand towards the enclosure. Distance IoU (DIoU) Loss incorporates a penalty on the normalized distance between the box centers. Let the center points be

$$ b^g = \left( \frac{x_1^g + x_2^g}{2}, \frac{y_1^g + y_2^g}{2} \right) \qquad (10.177) $$

and

$$ b^p = \left( \frac{x_1^p + x_2^p}{2}, \frac{y_1^p + y_2^p}{2} \right) \qquad (10.178) $$

The Euclidean distance between centers is

$$ \rho = \|b^p - b^g\|_2 \qquad (10.179) $$

Let c be the diagonal length of the enclosure C:

$$ c = \sqrt{(x_2^c - x_1^c)^2 + (y_2^c - y_1^c)^2} \qquad (10.180) $$

DIoU is defined as:

$$ \mathrm{DIoU} = \mathrm{IoU} - \frac{\rho^2}{c^2}. \qquad (10.181) $$

The DIoU Loss is

$$ \mathcal{L}_{DIoU} = 1 - \mathrm{DIoU} \qquad (10.182) $$

Its gradient includes a term that directly minimizes the center distance:

$$ \frac{\partial \mathcal{L}_{DIoU}}{\partial x_1^p} = \frac{\partial \mathcal{L}_{IoU}}{\partial x_1^p} + \frac{2}{c^2} \left( \frac{\rho}{c} \cdot \frac{\partial \rho}{\partial x_1^p} - \frac{\rho^2}{c} \cdot \frac{\partial c}{\partial x_1^p} \right). \qquad (10.183) $$
The derivative of the center distance $\rho$ with respect to the coordinate $x_1^p$ is:

$$ \frac{\partial \rho}{\partial x_1^p} = \frac{1}{2\rho} \cdot 2(b_x^p - b_x^g) \cdot \frac{1}{2} = \frac{b_x^p - b_x^g}{2\rho}, \qquad (10.184) $$

assuming the prediction affects the center point $b_x^p$. The derivative of the diagonal length $c$ is:

$$ \frac{\partial c}{\partial x_1^p} = \begin{cases} -\frac{(x_2^c - x_1^c)}{c} & \text{if } x_1^p \text{ is the left-most point} \\ 0 & \text{otherwise} \end{cases}. \qquad (10.185) $$

Complete IoU (CIoU) Loss builds upon DIoU by adding an aspect ratio penalty term. It introduces a measure $v$ to quantify the aspect ratio consistency:

$$ v = \frac{4}{\pi^2} \left( \arctan \left( \frac{w^g}{h^g} \right) - \arctan \left( \frac{w^p}{h^p} \right) \right)^2, \qquad (10.186) $$

where $w^g, h^g$ and $w^p, h^p$ are the width and height of the ground truth and predicted boxes, respectively. A trade-off parameter $\alpha$ is defined as

$$ \alpha = \frac{v}{(1 - \text{IoU}) + v} \qquad (10.187) $$

CIoU is then:

$$ \text{CIoU} = \text{IoU} - \frac{\rho^2}{c^2} - \alpha v. \qquad (10.188) $$

The CIoU Loss is

$$ \mathcal{L}_{\text{CIoU}} = 1 - \text{CIoU} \qquad (10.189) $$

Its gradient is the most complex, requiring the derivative of the aspect ratio term $v$ with respect to the box dimensions. The gradient with respect to the predicted width $w^p$ is:

$$ \frac{\partial v}{\partial w^p} = \frac{8}{\pi^2} \left( \arctan \left( \frac{w^g}{h^g} \right) - \arctan \left( \frac{w^p}{h^p} \right) \right) \cdot \left( -\frac{1}{(1 + (w^p/h^p)^2)} \cdot \frac{1}{h^p} \right). \qquad (10.190) $$

Similarly, the gradient with respect to the height $h^p$ is:

$$ \frac{\partial v}{\partial h^p} = \frac{8}{\pi^2} \left( \arctan \left( \frac{w^g}{h^g} \right) - \arctan \left( \frac{w^p}{h^p} \right) \right) \cdot \left( \frac{1}{(1 + (w^p/h^p)^2)} \cdot \frac{w^p}{(h^p)^2} \right). \qquad (10.191) $$

These gradients are then propagated back to the coordinate parameters through the relationships

$$ w^p = x_2^p - x_1^p \qquad (10.192) $$

and

$$ h^p = y_2^p - y_1^p \qquad (10.193) $$

for example

$$ \frac{\partial v}{\partial x_1^p} = -\frac{\partial v}{\partial w^p} \qquad (10.194) $$

This comprehensive penalty ensures convergence not only in overlap and center alignment but also in the shape of the predicted bounding box, leading to superior regression accuracy. The careful derivation of these gradients is essential for their implementation in deep learning frameworks using automatic differentiation.
<figure><img src="image_118.png" /><figcaption>Figure 10.37: IoU Loss and Variants (GIoU Loss, DIoU Loss, CIoU Loss)</figcaption></figure>

### 10.5.3.6.2 Dice Loss

Dice Loss is a region-based loss function originating from the Sørensen-Dice coefficient, a statistical metric used to gauge the similarity between two sets. In the context of image segmentation, it measures the overlap between the predicted segmentation mask $P$ and the ground truth mask $G$, making it particularly effective for tasks characterized by severe class imbalance, such as medical image segmentation where the object of interest often occupies a small fraction of the image. The Dice coefficient $DSC$ for binary segmentation is defined as:

$$DSC(P, G) = \frac{2|P \cap G|}{|P| + |G|} = \frac{2 \sum_{i=1}^{N} p_i g_i}{\sum_{i=1}^{N} p_i + \sum_{i=1}^{N} g_i}, \qquad (10.195)$$

where $N$ is the total number of pixels, $p_i \in [0, 1]$ is the predicted probability that pixel $i$ belongs to the foreground class, and $g_i \in \{0, 1\}$ is the corresponding ground truth label. The Dice Loss is then formulated as

$$\mathcal{L}_{\text{Dice}} = 1 - DSC \qquad (10.196)$$

which directly minimizes the dissimilarity between the predicted and true regions. To ensure numerical stability and avoid division by zero, especially when both masks are empty, a smooth term $\epsilon$ (typically 1) is added to both the numerator and denominator:

$$\mathcal{L}_{\text{Dice}} = 1 - \frac{2 \sum_{i=1}^{N} p_i g_i + \epsilon}{\sum_{i=1}^{N} p_i + \sum_{i=1}^{N} g_i + \epsilon}. \qquad (10.197)$$

This loss function is differentiable and can be optimized using gradient-based methods. The gradient of the Dice Loss with respect to the predicted probability $p_j$ at a specific pixel $j$ is crucial for
<figure><img src="image_119.png" /><figcaption>Figure 10.38: Dice Loss during Training</figcaption></figure>

backpropagation and is given by:

$$ \frac{\partial \mathcal{L}_{\text{Dice}}}{\partial p_j} = - \frac{2g_j(\sum_i p_i + \sum_i g_i) - 2 \sum_i p_i g_i}{(\sum_i p_i + \sum_i g_i)^2}. \qquad (10.198) $$

This gradient indicates that the update for each pixel is influenced by the global statistics of the entire mask—specifically, the sums over all pixels—rather than just the local error. This property makes Dice Loss inherently global and particularly sensitive to the overall structure of the segmentations, providing a strong gradient signal even when the foreground region is small.

The mathematical formulation of Dice Loss can be extended to multi-class segmentation problems using a one-hot encoded representation for C classes. For each class c, the loss is computed separately and then averaged, leading to:

$$ \mathcal{L}_{\text{Dice}}^{\text{multi}} = \frac{1}{C} \sum_{c=1}^{C} \left[ 1 - \frac{2 \sum_{i=1}^{N} p_{i,c} g_{i,c} + \epsilon}{\sum_{i=1}^{N} p_{i,c} + \sum_{i=1}^{N} g_{i,c} + \epsilon} \right], \qquad (10.199) $$

where $p_{i,c}$ is the predicted probability that pixel i belongs to class c, and $g_{i,c}$ is the corresponding ground truth indicator. In practice, Dice Loss is often combined with other loss functions, such as cross-entropy, to form a composite objective that leverages both region-based and distribution-based optimization. The combined loss

$$ \mathcal{L} = \mathcal{L}_{\text{CE}} + \lambda \mathcal{L}_{\text{Dice}} \qquad (10.200) $$

helps to balance the emphasis on pixel-wise accuracy and overall region overlap, often leading to improved performance and more stable convergence. The hyperparameter $\lambda$ controls the contribution of the Dice term. From a computational perspective, the sums over all pixels make the loss
efficient to compute using tensor operations, and its differentiability allows for seamless integration into deep learning frameworks. The Dice coefficient's equivalence to the F1 score, a harmonic mean of precision and recall, underscores its suitability for scenarios where both false positives and false negatives are critical, as it naturally balances these two metrics without requiring explicit weighting.

### 10.5.3.6.3 Focal Loss

Focal Loss is a dynamically scaled cross-entropy loss function designed to address the extreme foreground-background class imbalance prevalent in one-stage object detection models, such as RetinaNet, by systematically reducing the relative loss contribution from well-classified examples and focusing training on hard, misclassified examples. The loss is a modification of the standard binary cross-entropy (BCE) loss for binary classification, defined for a true label $y \in \{0, 1\}$ and a model-estimated probability $p \in [0, 1]$ for the class with label $y = 1$. To notational convenience, we define $p_t$ as the model's estimated probability for the true class:

$$p_t = \begin{cases} p & \text{if } y = 1 \\ 1 - p & \text{if } y = 0 \end{cases} \qquad (10.201)$$

The standard binary cross-entropy loss is then

$$L_{CE}(p_t) = -\log(p_t) \qquad (10.202)$$

The fundamental innovation of Focal Loss is the introduction of a modulating factor $(1 - p_t)^\gamma$, where $\gamma \ge 0$ is a tunable focusing parameter, and a weighting factor $\alpha_t \in [0, 1]$ for the class $y = 1$, with the weight for $y = 0$ being $1 - \alpha_t$. The complete formulation of Focal Loss is

$$L_{FL}(p_t) = -\alpha_t(1 - p_t)^\gamma \log(p_t) \qquad (10.203)$$

The modulating factor $(1 - p_t)^\gamma$ is the core component that dictates the dynamic scaling; for a well-classified example where $p_t \to 1$, this factor approaches 0, thus down-weighting the loss contribution for that example. Conversely, for a misclassified example where $p_t \to 0$, the modulating factor approaches 1, and the loss is largely unaffected. The rate of this down-weighting is controlled by $\gamma$; higher values of $\gamma$ (e.g., $\gamma = 2$) further suppress the loss from easy examples, intensifying the focus on hard negatives. The $\alpha_t$ factor is a static weight used to address class imbalance by increasing the importance of the rare class (e.g., foreground) in the loss; it can be set inversely proportional to the class frequency. The mathematical properties of Focal Loss are critical to its efficacy. The loss function is continuous and differentiable almost everywhere, which is essential for gradient-based optimization. The gradient of Focal Loss with respect to the model's output logit $z$ (where $p = \sigma(z)$) reveals its behavior during training. The gradient is given by:

$$\frac{\partial L_{FL}}{\partial z} = \frac{\partial L_{FL}}{\partial p} \cdot \frac{\partial p}{\partial z} = \left( -\alpha_t \gamma (1 - p_t)^\gamma \frac{\log(p_t)}{p_t} + \alpha_t (1 - p_t)^\gamma \frac{1}{p_t} \right) \cdot p(1 - p). \qquad (10.204)$$

This can be simplified, considering the definition of $p_t$, to:

$$\frac{\partial L_{FL}}{\partial z} = \alpha_t y(1 - p_t)^\gamma (\gamma p_t \log(p_t) - p_t + 1) - \alpha_t (1 - y) p_t^\gamma (\gamma (1 - p_t) \log(1 - p_t) - (1 - p_t) + 1). \qquad (10.205)$$

A more intuitive understanding emerges when considering the case of a positive example ($y = 1$), where the gradient magnitude is proportional to $(1 - p_t)^\gamma$. This shows that the gradient signal diminishes for easy examples (high $p_t$) and remains significant for hard examples (low $p_t$), directly addressing the class imbalance issue where the vast number of easy negative examples would otherwise dominate the gradient. The loss is convex for a given value of $p_t$, but the dynamic scaling
<figure><img src="image_120.png" /><figcaption>Figure 10.39: Focal Loss vs Predicted Probability</figcaption></figure>

<figure><img src="image_121.png" /><figcaption>Figure 10.40: Focal Loss vs Standard Binary Cross-Entropy</figcaption></figure>

makes the overall optimization landscape more complex, requiring careful tuning of the learning rate. The hyperparameter $\gamma$ smoothly interpolates the loss behavior; when $\gamma = 0$, Focal Loss is equivalent to $\alpha$-balanced cross-entropy, and as $\gamma$ increases, the effect of the modulating factor becomes more pronounced. The $\alpha$ parameter helps to counter the implicit weighting introduced by $\gamma$, which can suppress the loss for the rare class if its examples are initially easy; setting $\alpha$ higher for the rare class ensures it continues to receive sufficient learning signal. In practice, the
combined effect of $\alpha$ and $\gamma$ drastically improves the precision-recall trade-off on imbalanced datasets, allowing one-stage detectors to surpass the performance of more complex two-stage architectures by efficiently harnessing a much larger and more diverse set of candidate proposals during training, whose gradients are no longer drowned out by a sea of trivial background examples.

### 10.5.3.7 Loss Functions for Knowledge Distillation

### 10.5.3.8 Loss Functions for Reinforcement Learning

#### 10.5.3.8.1 Hinge Loss

The Hinge Loss function is a convex relaxation of the 0-1 loss primarily employed for maximum-margin classification, most notably in Support Vector Machines (SVMs) but also applicable to neural networks. Its objective is not to produce probability estimates but to find a decision boundary

<figure><img src="image_122.png" /><figcaption>Figure 10.41: Hinge Loss vs Prediction Score</figcaption></figure>

that maximizes the geometric margin between classes, thereby enhancing the model's generalization capability. For a binary classification task with true labels $y_i \in \{-1, +1\}$ and a model that produces real-valued scores $f(\mathbf{x}_i) = \mathbf{w}^\top \mathbf{x}_i + b$ (where $\mathbf{w}$ is the weight vector and $b$ is the bias), the Hinge Loss for a single data point is defined as

$$L_{\text{hinge}}(y_i, f(\mathbf{x}_i)) = \max(0, 1 - y_i f(\mathbf{x}_i)) \qquad (10.206)$$

This piecewise function is zero if $y_i f(\mathbf{x}_i) \ge 1$, meaning the prediction is correct and lies beyond the margin boundary, and increases linearly for $y_i f(\mathbf{x}_i) < 1$. The term $y_i f(\mathbf{x}_i)$ is known as the functional margin; a positive value indicates a correct classification, and its magnitude represents the confidence of the prediction. The loss explicitly penalizes predictions that are either incorrect
<figure><img src="image_123.png" /><figcaption>Figure 10.42: Comparison of the Standard Hinge Loss and the Squared Hinge Loss</figcaption></figure>

or correct but within the margin boundary (where $0 < y_i f(\mathbf{x}_i) < 1$), with the penalty proportional to the degree of margin violation. The aggregate loss over a dataset of $N$ samples is typically given by

$$L = \frac{1}{N} \sum_{i=1}^{N} \max(0, 1 - y_i f(\mathbf{x}_i)) + \lambda \|\mathbf{w}\|_2^2 \quad (10.207)$$

where the $L_2$ regularization term $\lambda \|\mathbf{w}\|_2^2$ is not merely a technical addition but is fundamental to the maximum-margin principle, directly controlling the trade-off between maximizing the margin and minimizing classification error.

From an optimization perspective, the Hinge Loss is convex and piecewise linear, which allows for efficient training using subgradient methods. Its subgradient with respect to the model's parameters $\mathbf{w}$ is central to learning. For a single data point, the subdifferential $\partial_{\mathbf{w}} L_{\text{hinge}}(y_i, f(\mathbf{x}_i))$ is:

$$\partial_{\mathbf{w}} L_{\text{hinge}} = \begin{cases} \{0\} & \text{if } y_i f(\mathbf{x}_i) > 1 \\ \{-y_i \mathbf{x}_i\} & \text{if } y_i f(\mathbf{x}_i) < 1. \\ \{-\alpha y_i \mathbf{x}_i | \alpha \in [0, 1]\} & \text{if } y_i f(\mathbf{x}_i) = 1 \end{cases} \quad (10.208)$$

In practice, at the point of non-differentiability $y_i f(\mathbf{x}_i) = 1$, a common convention is to choose a subgradient of 0, treating the point as part of the flat region. The overall subgradient of the regularized objective $L$ with respect to $\mathbf{w}$ is therefore

$$\nabla_{\mathbf{w}} L = \frac{1}{N} \sum_{i=1}^{N} \partial_{\mathbf{w}} L_{\text{hinge}}(y_i, f(\mathbf{x}_i)) + 2\lambda \mathbf{w} \quad (10.209)$$

This leads to a simple and intuitive update rule: for samples that violate the margin ($y_i f(\mathbf{x}_i) < 1$), the update is

$$\mathbf{w} \leftarrow \mathbf{w} + \eta(y_i \mathbf{x}_i - 2\lambda\mathbf{w}) \quad (10.210)$$

(if using a subgradient of $-y_i \mathbf{x}_i$), where $\eta$ is the learning rate, which pushes the decision boundary to correctly classify the sample, while the regularization term shrinks the weights. Samples outside the margin do not contribute to the weight update. This sparsity in the gradient contribution is a key property, as it means the final classifier is only defined by a subset of critical training examples—the support vectors—which are the samples that lie on or within the margin ($y_i f(\mathbf{x}_i) \le 1$).
The Hinge Loss can be generalized to multi-class classification, known as the Crammer-Singer loss or multi-class Hinge Loss. For a problem with K classes, where the true label for a sample $\mathbf{x}_i$ is an integer $t_i \in \{1, \dots, K\}$ and the model produces a score vector $\mathbf{f}(\mathbf{x}_i) \in \mathbb{R}^K$, the loss is defined as

$$L_{\text{multi-hinge}} = \max \left( 0, 1 + \max_{j \neq t_i} f_j(\mathbf{x}_i) - f_{t_i}(\mathbf{x}_i) \right) \quad (10.211)$$

This formulation penalizes the model if the score for the correct class $f_{t_i}$ is not greater by at least a margin of 1 than the maximum score among all incorrect classes. The subgradient for this loss involves the incorrect class $j^*$ that achieves the maximum score

$$j^* = \arg \max_{j \neq t_i} f_j(\mathbf{x}_i) \quad (10.212)$$

leading to updates that increase the score for the correct class and decrease the score for this most offending incorrect class. The Hinge Loss's primary strength is its focus on the geometric structure of the problem, promoting robust classifiers with good generalization bounds derived from statistical learning theory, such as the Vapnik-Chervonenkis (VC) dimension and Rademacher complexity, which directly relate to the concept of the margin. Its main limitation is its lack of inherent probabilistic calibration, as it outputs a decision function rather than a probability, though methods like Platt scaling can be applied post-hoc to address this.

### 10.5.3.8.2 Value Loss

Value Loss, in the context of reinforcement learning (RL) and particularly actor-critic methods, is an objective function designed to train the value function estimator $V_\phi(s)$, parameterized by $\phi$, to accurately predict the expected cumulative return from a given state $s$. The value function represents the expected return when starting from state $s$ and following policy $\pi$:

$$V^{\pi}(s) = \mathbb{E}_{\pi} \left[ \sum_{t=0}^{\infty} \gamma^t r_t \mid s_0 = s \right] \quad (10.213)$$

where $\gamma \in [0, 1]$ is the discount factor and $r_t$ is the reward at time $t$. The Value Loss is typically defined as the mean squared error (MSE) between the predicted value $V_\phi(s)$ and a target value estimate $V_{\text{target}}(s)$, which is derived from empirical returns or bootstrapped estimates. In the Monte Carlo approach, the target is the actual return

$$G_t = \sum_{k=0}^{\infty} \gamma^k r_{t+k} \quad (10.214)$$

leading to the loss

$$\mathcal{L}(\phi) = \frac{1}{2} \mathbb{E} \left[ (G_t - V_{\phi}(s_t))^2 \right] \quad (10.215)$$

However, due to the high variance of $G_t$, temporal difference (TD) methods are preferred, where the target is the bootstrapped estimate

$$V_{\text{target}}(s_t) = r_t + \gamma V_{\phi_{\text{old}}}(s_{t+1}) \quad (10.216)$$

using parameters $\phi_{\text{old}}$ from a previous iteration to enhance stability. The loss becomes:

$$\mathcal{L}(\phi) = \frac{1}{2} \mathbb{E}_{s_t \sim \rho^{\pi}} \left[ (r_t + \gamma V_{\phi_{\text{old}}}(s_{t+1}) - V_{\phi}(s_t))^2 \right], \quad (10.217)$$

where $\rho^\pi$ is the state distribution under policy $\pi$. In advanced algorithms like Proximal Policy Optimization (PPO) and Generalized Advantage Estimation (GAE), the value loss is minimized
<figure><img src="image_124.png" /><figcaption>Figure 10.43: Value Loss (MSE) for Different Targets</figcaption></figure>

concurrently with the policy loss to reduce variance and accelerate learning. The gradient of the value loss with respect to the parameters $\phi$ is:

$$ \nabla_{\phi} \mathcal{L}(\phi) = -\mathbb{E}[(r_t + \gamma V_{\phi \text{old}}(s_{t+1}) - V_{\phi}(s_t)) \nabla_{\phi} V_{\phi}(s_t)], \quad (10.218) $$

which adjusts $\phi$ to minimize the TD error

$$ \delta_t = r_t + \gamma V_{\phi \text{old}}(s_{t+1}) - V_{\phi}(s_t) \quad (10.219) $$

To prevent overfitting and ensure training stability, the value loss is often clipped or regularized. For instance, in PPO, the value function is trained via multiple epochs on the same data, and the loss may include a clipping term or an entropy bonus. The objective becomes:

$$ \mathcal{L}^{\text{CLIP}}(\phi) = \frac{1}{2} \mathbb{E} \left[ \max \left( (V_{\phi}(s_t) - V_{\text{target}}(s_t))^2, \text{clip}(V_{\phi}(s_t) - V_{\text{old}}(s_t), -\epsilon, \epsilon) + V_{\text{old}}(s_t) - V_{\text{target}}(s_t) \right)^2 \right], \quad (10.220) $$

though this is less common than the policy clipping. Alternatively, a squared error loss with a penalty on the divergence from the previous value function can be used:

$$ \mathcal{L}(\phi) = \frac{1}{2} \mathbb{E} \left[ (V_{\text{target}}(s_t) - V_{\phi}(s_t))^2 + \beta \cdot D_{\text{KL}}(V_{\text{old}} \| V_{\phi}) \right], \quad (10.221) $$

where $\beta$ is a hyperparameter. The target $V_{\text{target}}$ can be computed using GAE, which provides a low-variance estimate of the advantage function $A^{\pi}(s, a)$ by exponentially averaging TD errors of different steps:

$$ V_{\text{target}}(s_t) = V_{\phi \text{old}}(s_t) + \hat{A}_t, \quad (10.222) $$
<figure><img src="image_125.png" /><figcaption>Figure 10.44: Value Loss (MSE) vs Huber Loss</figcaption></figure>

where

$$ \hat{A}_t = \sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l} \qquad (10.223) $$

is the GAE estimate, with $\lambda \in [0, 1]$ controlling the bias-variance trade-off. The value loss is then:

$$ \mathcal{L}(\phi) = \frac{1}{2} \mathbb{E} \left[ \left( V_{\phi_{\text{old}}}(s_t) + \hat{A}_t - V_{\phi}(s_t) \right)^2 \right]. \qquad (10.224) $$

This formulation ensures that the value function learns to approximate the true value function $V^{\pi}(s)$, which is critical for accurate advantage estimation and effective policy updates. The optimization of the value loss is typically performed using stochastic gradient descent, with gradients computed over minibatches of trajectories to ensure computational efficiency and convergence. The value function's accuracy directly influences the policy gradient estimates, as the advantage $\hat{A}_t$ is used to scale the policy updates, making the value loss a cornerstone of stable and efficient reinforcement learning.

### 10.5.3.8.3 Policy Loss

Policy Loss is the objective function used to optimize the policy $\pi_{\theta}(a|s)$ in reinforcement learning, where $\theta$ represents the parameters of a neural network that maps states $s$ to a probability distribution over actions $a$. The fundamental goal is to maximize the expected cumulative reward

$$ J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}} \left[ \sum_{t=0}^{\infty} \gamma^t r(s_t, a_t) \right] \qquad (10.225) $$

where $\tau = (s_0, a_0, s_1, a_1, \dots)$ is a trajectory and $\gamma$ is the discount factor. The policy gradient theorem provides the foundation for the policy loss by giving the gradient of this objective:

$$ \nabla_{\theta} J(\theta) = \mathbb{E}_{\tau \sim \pi_{\theta}} \left[ \sum_{t=0}^{\infty} \nabla_{\theta} \log \pi_{\theta}(a_t|s_t) A^{\pi_{\theta}}(s_t, a_t) \right] \qquad (10.226) $$

where

$$ A^{\pi_{\theta}}(s_t, a_t) = Q^{\pi_{\theta}}(s_t, a_t) - V^{\pi_{\theta}}(s_t) \qquad (10.227) $$
is the advantage function, measuring how much better action $a_t$ is compared to the average action in state $s_t$. The basic policy loss is then formulated as the negative of the objective to be minimized:

$$ \mathcal{L}_{\text{PG}}(\theta) = -\mathbb{E}[\log \pi_{\theta}(a_t|s_t)A_t] \qquad (10.228) $$

where $A_t$ is an estimate of the advantage. However, this vanilla policy gradient can lead to unstable updates with high variance. To address stability issues, Proximal Policy Optimization (PPO)

<figure><img src="image_126.png" /><figcaption>Figure 10.45: Policy Loss vs Action Probability</figcaption></figure>

introduces a clipped surrogate objective that prevents excessively large policy updates. Let

$$ r_t(\theta) = \frac{\pi_{\theta}(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)} \qquad (10.229) $$

be the probability ratio between the new and old policies. The unclipped objective is

$$ \mathcal{L}^{\text{CPI}}(\theta) = \mathbb{E}[r_t(\theta)A_t] \qquad (10.230) $$

where CPI denotes conservative policy iteration. PPO modifies this by clipping the probability ratio to remain near 1, forming the primary policy loss:

$$ \mathcal{L}^{\text{CLIP}}(\theta) = \mathbb{E}[\min(r_t(\theta)A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)A_t)], \qquad (10.231) $$

where $\epsilon$ is a hyperparameter, typically 0.1 or 0.2. The clip function restricts $r_t(\theta)$ to the interval $[1-\epsilon, 1+\epsilon]$, ensuring the policy update does not change too radically. The min operator ensures the objective is a lower bound (pessimistic clip) on the unclipped objective, preventing the policy from being improved excessively when the advantage is positive and from being degraded too much when
<figure><img src="image_127.png" /><figcaption>Figure 10.46: PPO Clipped Surrogate Objective</figcaption></figure>

the advantage is negative. The gradient of this loss with respect to $\theta$ requires careful computation of the derivatives of the probability ratio. For a positive advantage $A_t > 0$, the gradient is

$$ \nabla_{\theta} \mathcal{L} = \mathbb{E}[\nabla_{\theta} \min(r_t(\theta)A_t, (1+\epsilon)A_t)] \quad (10.232) $$

which becomes $\nabla_{\theta}r_t(\theta)A_t$ if $r_t(\theta) \le 1 + \epsilon$, and 0 otherwise. For a negative advantage $A_t < 0$, the gradient is

$$ \nabla_{\theta} \mathcal{L} = \mathbb{E}[\nabla_{\theta} \min(r_t(\theta)A_t, (1-\epsilon)A_t)] \quad (10.233) $$

which becomes $\nabla_{\theta}r_t(\theta)A_t$ if $r_t(\theta) \ge 1 - \epsilon$, and 0 otherwise. This clipping mechanism effectively removes the incentive for moving $r_t(\theta)$ outside of the interval $[1 - \epsilon, 1 + \epsilon]$. An alternative PPO objective combines the clipped surrogate with a penalty on the KL divergence between the new and old policies:

$$ \mathcal{L}^{\text{KLPEN}}(\theta) = \mathbb{E}[r_t(\theta)A_t - \beta \cdot D_{\text{KL}}(\pi_{\theta_{\text{old}}}(\cdot|s_t)\|\pi_{\theta}(\cdot|s_t))] \quad (10.234) $$

where $\beta$ is an adaptive coefficient. The KL divergence for categorical policies is

$$ D_{\text{KL}}(p \| q) = \sum_{i} p_i \log \frac{p_i}{q_i} \quad (10.235) $$

and its gradient encourages the new policy to not deviate too far from the old policy in terms of action distribution. The complete PPO objective often includes a value function error term and an entropy bonus for exploration, leading to the total loss:

$$ \mathcal{L}^{\text{total}}(\theta, \phi) = \mathcal{L}^{\text{CLIP}}(\theta) + c_1 \mathcal{L}^{\text{VF}}(\phi) - c_2 \mathcal{S}[\pi_{\theta}](s_t) \quad (10.236) $$
<figure><img src="image_128.png" /><figcaption>Figure 10.47: PPO Loss Comparison</figcaption></figure>

where $c_1, c_2$ are coefficients, $\mathcal{L}^{\text{VF}}$ is the value loss, and $\mathcal{S}$ is the entropy. The policy gradient can be computed using the likelihood ratio trick:

$$ \nabla_{\theta} \log \pi_{\theta}(a_t|s_t) = \frac{\nabla_{\theta} \pi_{\theta}(a_t|s_t)}{\pi_{\theta}(a_t|s_t)} \qquad (10.237) $$

allowing for efficient estimation from trajectories. For continuous action spaces, where the policy is often a Gaussian distribution

$$ \pi_{\theta}(\cdot|s) = \mathcal{N}(\mu_{\theta}(s), \sigma_{\theta}^{2}(s)) \qquad (10.238) $$

the log probability is

$$ \log \pi_{\theta}(a|s) = -\frac{1}{2} \left( \frac{a - \mu_{\theta}(s)}{\sigma_{\theta}(s)} \right)^2 - \log \sigma_{\theta}(s) - \frac{1}{2} \log(2\pi) \qquad (10.239) $$

and its gradient with respect to $\theta$ involves the derivatives of the mean and standard deviation networks. The policy loss thus provides a mathematically grounded mechanism for incrementally improving the policy while maintaining training stability through constraints on the policy change.

### 10.5.3.8.4 Entropy Regularization

Entropy Regularization is a technique employed in reinforcement learning, particularly in policy optimization methods like Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), to encourage exploration by penalizing low entropy in the policy distribution, thereby preventing premature convergence to suboptimal deterministic policies. The entropy $\mathcal{H}$ of a discrete probability distribution $\pi(a|s)$ is defined as

$$ \mathcal{H}(\pi(\cdot|s)) = -\sum_{a \in \mathcal{A}} \pi(a|s) \log \pi(a|s) \qquad (10.240) $$

which measures the uncertainty or randomness in the action selection; maximum entropy is achieved by the uniform distribution where all actions are equally likely, while minimum entropy (zero) corresponds to a deterministic policy. In the context of the maximum entropy reinforcement learning framework, the standard objective of maximizing the expected cumulative reward $\mathbb{E}[\sum_t \gamma^t r(s_t, a_t)]$ is augmented with an entropy term to form a modified objective

$$ J(\theta) = \mathbb{E}[\sum_t \gamma^t (r(s_t, a_t) + \alpha \mathcal{H}(\pi_{\theta}(\cdot|s_t)))] \qquad (10.241) $$
<figure><img src="image_129.png" /><figcaption>Figure 10.48: PPO Clipped Surrogate Loss with Entropy Regularization</figcaption></figure>

where $\alpha > 0$ is a temperature parameter that controls the relative importance of entropy versus reward. The policy loss function then incorporates entropy regularization as a negative term to be maximized (or a positive term to be minimized when framed as a loss), leading to

$$ \mathcal{L}_{\text{policy}}(\theta) = -\mathbb{E}[\log \pi_{\theta}(a_t|s_t)A_t - \alpha \mathcal{H}(\pi_{\theta}(\cdot|s_t))] \quad (10.242) $$

where $A_t$ is the advantage estimate. The gradient of the entropy term with respect to the policy parameters $\theta$ is

$$ \nabla_{\theta}\mathcal{H}(\pi_{\theta}) = -\sum_{a}(\log \pi_{\theta}(a|s) + 1)\nabla_{\theta}\pi_{\theta}(a|s) \quad (10.243) $$

which can be computed efficiently using automatic differentiation. For continuous action spaces where the policy is parameterized as a Gaussian distribution

$$ \pi_{\theta}(\cdot|s) = \mathcal{N}(\mu_{\theta}(s), \Sigma_{\theta}(s)) \quad (10.244) $$

the entropy has a closed-form expression. For a multivariate Gaussian with diagonal covariance

$$ \Sigma = \text{diag}(\sigma_1^2, \dots, \sigma_n^2) \quad (10.245) $$

the entropy is

$$ \mathcal{H}(\mathcal{N}(\mu, \Sigma)) = \frac{n}{2}\log(2\pi e) + \frac{1}{2}\log\det(\Sigma) = \frac{n}{2}(1 + \log(2\pi)) + \sum_{i=1}^{n}\log\sigma_i \quad (10.246) $$

The gradient of this entropy with respect to the standard deviation parameters $\sigma_i$ is straightforward:

$$ \nabla_{\sigma_i} \mathcal{H} = \frac{1}{\sigma_i} \quad (10.247) $$
<figure><img src="image_130.png" /><figcaption>Figure 10.49: Comparison: PPO Unclipped, PPO Clipped, PPO + Entropy</figcaption></figure>

<figure><img src="image_131.png" /><figcaption>Figure 10.50: Soft Actor-Critic (SAC) Policy Loss with Entropy Regularization</figcaption></figure>

which provides a clear signal to increase the standard deviations, thereby increasing exploration. In practice, the temperature parameter $\alpha$ can be treated as a hyperparameter or adapted dynamically during training to maintain a target entropy level $\bar{H}$, often set as the negative dimension of the action space

$$ \bar{H} = - \dim(\mathcal{A}) \qquad (10.248) $$
The loss for the temperature parameter is

$$ \mathcal{L}(\alpha) = \mathbb{E}[-\alpha \log \pi_{\theta}(a_t|s_t) - \alpha \bar{\mathcal{H}}] \qquad (10.249) $$

and its gradient

$$ \nabla_{\alpha} \mathcal{L} = \mathbb{E}[-\log \pi_{\theta}(a_t|s_t) - \bar{\mathcal{H}}] \qquad (10.250) $$

adjusts $\alpha$ to balance the entropy penalty. The inclusion of entropy regularization alters the optimal policy to a softmax form:

$$ \pi^*(a|s) \propto \exp(Q_{\text{soft}}(s, a)/\alpha) \qquad (10.251) $$

where $Q_{\text{soft}}$ is the soft Q-function that includes entropy bonuses. This results in a policy that not only seeks high rewards but also maintains stochasticity, which improves exploration and robustness by preventing the policy from becoming too confident too early. The mathematical effect of entropy regularization is to smooth the optimization landscape, making gradient estimates more reliable and reducing the likelihood of getting trapped in local optima, which is crucial for solving complex environments with sparse rewards.

### 10.5.3.9 Loss Functions for Sparse and Structured Outputs

#### 10.5.3.9.1 Kullback-Leibler (KL) Divergence Loss (as a Sparsity Constraint)

The utilization of Kullback-Leibler (KL) Divergence as a sparsity constraint is a Bayesian regularization technique that forces the empirical distribution of activations within a neural network layer, typically a hidden layer of an autoencoder, to approximate a pre-specified sparse prior distribution. This method induces a form of representational sparsity, where only a small fraction of units are significantly active for any given input, thereby learning efficient, distributed codes. The core mathematical object is the KL divergence between two Bernoulli distributions: a target sparse prior with a very low mean activation probability $\rho$ (e.g., $\rho = 0.05$), and the empirical average activation $\hat{\rho}_j$ of a particular hidden unit $j$ over a batch of $m$ training examples. For a single unit, the average activation is

$$ \hat{\rho}_j = \frac{1}{m} \sum_{i=1}^{m} a_j^{(i)} \qquad (10.252) $$

where

$$ a_j^{(i)} = f(\mathbf{w}_j^T \mathbf{x}^{(i)} + b_j) \qquad (10.253) $$

is the activation of unit $j$ for input $\mathbf{x}^{(i)}$, and $f$ is a sigmoidal activation function. The KL divergence for a Bernoulli distribution, which measures the inefficiency of assuming the distribution $\hat{\rho}_j$ when the true distribution is $\rho$, is given by:

$$ D_{\text{KL}}(\rho \parallel \hat{\rho}_j) = \rho \log \frac{\rho}{\hat{\rho}_j} + (1 - \rho) \log \frac{1 - \rho}{1 - \hat{\rho}_j} \qquad (10.254) $$

This function has the desired properties of a sparsity penalty: it is zero if and only if $\hat{\rho}_j = \rho$, and it increases asymmetrically as $\hat{\rho}_j$ deviates from $\rho$. The asymmetry is crucial; it penalizes units that are too active ($\hat{\rho}_j > \rho$) more heavily than units that are not active enough ($\hat{\rho}_j < \rho$), although the term for $\hat{\rho}_j < \rho$ still provides a pulling force towards the target. The total sparsity penalty for a hidden layer with $s$ units is the sum of the KL divergences over all units: $\sum_{j=1}^s D_{\text{KL}}(\rho \parallel \hat{\rho}_j)$. This penalty term is then added to the primary objective function, such as the reconstruction error of an autoencoder, scaled by a sparsity regularization parameter $\beta$. The complete loss function for a sparse autoencoder becomes:

$$ J(\mathbf{W}, \mathbf{b}) = \frac{1}{m} \sum_{i=1}^{m} \left( \frac{1}{2} \|\mathbf{x}^{(i)} - \hat{\mathbf{x}}^{(i)}\|^2 \right) + \beta \sum_{j=1}^{s} D_{\text{KL}}(\rho \parallel \hat{\rho}_j) \qquad (10.255) $$
<figure><img src="image_132.png" /><figcaption>Figure 10.51: KL Divergence Loss as Sparsity Constraint</figcaption></figure>

The gradient of this composite loss with respect to the network's parameters must be computed for backpropagation. The gradient of the KL penalty term with respect to the average activation $\hat{\rho}_j$ is:

$$ \frac{\partial}{\partial \hat{\rho}_j} \sum_{j=1}^{s} D_{\text{KL}}(\rho \| \hat{\rho}_j) = -\frac{\rho}{\hat{\rho}_j} + \frac{1-\rho}{1-\hat{\rho}_j}. \qquad (10.256) $$

This gradient indicates the direction to push $\hat{\rho}_j$ to minimize the penalty. Since $\hat{\rho}_j$ is not a direct parameter but a function of the activations $a_j^{(i)}$, the chain rule is applied. The gradient of the total loss $J$ with respect to the activation $a_j^{(i)}$ of unit $j$ for a single example $i$ receives an additional contribution from the sparsity penalty:

$$ \frac{\partial J}{\partial a_j^{(i)}} = \frac{\partial J_{\text{recon}}}{\partial a_j^{(i)}} + \beta \left( -\frac{\rho}{\hat{\rho}_j} + \frac{1-\rho}{1-\hat{\rho}_j} \right) \cdot \frac{\partial \hat{\rho}_j}{\partial a_j^{(i)}}. \qquad (10.257) $$

Given that

$$ \hat{\rho}_j = \frac{1}{m} \sum_{i=1}^{m} a_j^{(i)} \qquad (10.258) $$

the derivative

$$ \frac{\partial \hat{\rho}_j}{\partial a_j^{(i)}} = \frac{1}{m} \qquad (10.259) $$

Therefore, the sparsity-induced gradient for a single training example is effectively

$$ \frac{\beta}{m} \left( -\frac{\rho}{\hat{\rho}_j} + \frac{1-\rho}{1-\hat{\rho}_j} \right) \qquad (10.260) $$
<figure><img src="image_133.png" /><figcaption>Figure 10.52: Total KL Sparsity Loss for 10 Hidden Units ($\rho = 0.05$)</figcaption></figure>

This term is then propagated backwards through the activation function to update the weights **W** and biases **b**. For a sigmoid activation $a = \sigma(z)$, the gradient with respect to the pre-activation $z_j^{(i)}$ is

$$ \frac{\partial J}{\partial z_j^{(i)}} = \frac{\partial J}{\partial a_j^{(i)}} \cdot a_j^{(i)} (1 - a_j^{(i)}) \qquad (10.261) $$

This mechanism ensures that the learning process not only minimizes reconstruction error but also adjusts the parameters to drive the average activations of the hidden units towards the desired sparse target $\rho$, resulting in a model that discovers statistically significant features in the data without over-representing them.

### 10.5.3.9.2 Structured Support Vector Machine (SSVM) Loss

The Structured Support Vector Machine (SSVM) loss, also known as the max-margin Markov network loss or structured hinge loss, is a convex objective function for training models that predict complex, interdependent output structures $\mathbf{y} \in \mathcal{Y}$ from an input $\mathbf{x}$, where $\mathcal{Y}$ is a discrete but exponentially large space (e.g., all possible sequences, sparse trees, or segmentations). The fundamental principle is a generalization of the binary SVM margin principle to structured outputs: the score

$$ s(\mathbf{x}, \mathbf{y}; \mathbf{w}) = \mathbf{w}^{\top} \Psi(\mathbf{x}, \mathbf{y}) \qquad (10.262) $$

of the correct output structure $\mathbf{y}_i$ should be greater than the score of any other structure $\mathbf{y}$ by a margin that scales with the loss incurred by predicting $\mathbf{y}$ instead of $\mathbf{y}_i$. Here, $\Psi(\mathbf{x}, \mathbf{y})$ is a joint
<figure><img src="image_134.png" /><figcaption>Figure 10.53: Total KL Sparsity Loss for 10 Hidden Units with Random Activations ($\rho = 0.05$)</figcaption></figure>

feature map that encodes the compatibility between input $\mathbf{x}$ and output $\mathbf{y}$, and $\mathbf{w}$ are the model parameters. The structured hinge loss for a single training example $(\mathbf{x}_i, \mathbf{y}_i)$ is formulated as:

$$L_i(\mathbf{w}) = \max_{\mathbf{y} \in \mathcal{Y}} [\Delta(\mathbf{y}_i, \mathbf{y}) + \mathbf{w}^\top \Psi(\mathbf{x}_i, \mathbf{y}) - \mathbf{w}^\top \Psi(\mathbf{x}_i, \mathbf{y}_i)], \quad (10.263)$$

where $\Delta(\mathbf{y}_i, \mathbf{y}) \ge 0$ is a task-specific cost function that quantifies the discrepancy between the true structure $\mathbf{y}_i$ and a candidate structure $\mathbf{y}$; it is zero if $\mathbf{y} = \mathbf{y}_i$ and positive otherwise. The maximization over $\mathbf{y} \in \mathcal{Y}$, often called the "loss-augmented inference" problem, finds the most violating incorrect structure—the one that has a high score according to the current model $\mathbf{w}$ and also incurs a high cost $\Delta$. The complete objective function for the SSVM, including $L_2$ regularization, is:

$$\min_{\mathbf{w}} \frac{\lambda}{2} \|\mathbf{w}\|^2 + \frac{1}{N} \sum_{i=1}^{N} \max_{\mathbf{y} \in \mathcal{Y}} [\Delta(\mathbf{y}_i, \mathbf{y}) + \mathbf{w}^\top \Psi(\mathbf{x}_i, \mathbf{y}) - \mathbf{w}^\top \Psi(\mathbf{x}_i, \mathbf{y}_i)]. \quad (10.264)$$

This objective is convex in $\mathbf{w}$ because it is a sum of a convex quadratic function and maxima of affine functions of $\mathbf{w}$. The optimization of this loss is typically performed using subgradient methods. The subgradient of the loss $L_i(\mathbf{w})$ with respect to $\mathbf{w}$ is given by the difference between the feature vector of the maximally violating structure and the feature vector of the true structure:

$$\partial_{\mathbf{w}} L_i(\mathbf{w}) = \Psi(\mathbf{x}_i, \mathbf{y}^*) - \Psi(\mathbf{x}_i, \mathbf{y}_i), \quad (10.265)$$

where

$$\mathbf{y}^* = \arg \max_{\mathbf{y} \in \mathcal{Y}} [\Delta(\mathbf{y}_i, \mathbf{y}) + \mathbf{w}^\top \Psi(\mathbf{x}_i, \mathbf{y})] \quad (10.266)$$
<figure><img src="image_135.png" /><figcaption>Figure 10.54: Structured SVM (SSVM) Hinge Loss</figcaption></figure>

This subgradient indicates the direction to adjust **w** to increase the score gap between the true structure and the most offending incorrect structure. The critical computational challenge is solving this loss-augmented inference problem efficiently. For many structured problems where the cost function Δ decomposes additively over the output structure (e.g., Hamming loss over sequences), and the feature map Ψ factorizes according to a graphical model (e.g., a chain or tree), dynamic programming algorithms like the Viterbi algorithm or belief propagation can compute **y*** exactly and efficiently. The overall training algorithm involves iterating over examples, solving for **y*** for each, and updating the weights via

$$ \mathbf{w} \leftarrow \mathbf{w} - \eta_t (\lambda \mathbf{w} + \Psi(\mathbf{x}_i, \mathbf{y}^*) - \Psi(\mathbf{x}_i, \mathbf{y}_i)) \quad (10.267) $$

where $\eta_t$ is a learning rate. A common alternative is the margin-rescaled formulation presented above; another is the slack-rescaled formulation, which scales the margin by the cost:

$$ L_i(\mathbf{w}) = \max_{\mathbf{y} \in \mathcal{Y}} \left[ \Delta(\mathbf{y}_i, \mathbf{y}) \cdot \left( 1 + \mathbf{w}^\top \Psi(\mathbf{x}_i, \mathbf{y}) - \mathbf{w}^\top \Psi(\mathbf{x}_i, \mathbf{y}_i) \right) \right] \quad (10.268) $$

The SSVM framework provides a principled, margin-based approach to structured prediction with strong theoretical guarantees, but its efficacy hinges on the tractability of the loss-augmented inference problem and the design of a meaningful feature map Ψ(**x**, **y**) that captures the input-output relationships.

### 10.5.3.9.3 Conditional Random Field (CRF) Loss

The Conditional Random Field (CRF) loss function is derived from the principle of maximum likelihood estimation for conditional probabilistic graphical models, specifically designed for structured
Structured SVM Loss Surface ($\Delta=1.0$)

<figure><img src="image_136.png" /><figcaption>Figure 10.55: Structured SVM Loss Surface ($\Delta = 1.0$)</figcaption></figure>

prediction tasks where the output variables $\mathbf{y} = (y_1, y_2, \dots, y_T)$ exhibit mutual dependencies given an input sequence $\mathbf{x}$. A CRF directly models the conditional probability $P(\mathbf{y} | \mathbf{x})$ without making independence assumptions among the output variables $y_t$, unlike generative models like Hidden Markov Models. For a linear-chain CRF, which is common in sequence labeling, the probability distribution is defined by a Gibbs distribution:

$$P(\mathbf{y} | \mathbf{x}; \mathbf{w}) = \frac{1}{Z(\mathbf{x}; \mathbf{w})} \exp \left( \sum_{t=1}^{T} \mathbf{w}_{u}^{\top} \phi_{u}(x_{t}, y_{t}) + \sum_{t=1}^{T-1} \mathbf{w}_{p}^{\top} \phi_{p}(y_{t}, y_{t+1}) \right), \quad (10.269)$$

where $\phi_u(x_t, y_t)$ is a feature vector capturing the association between input $x_t$ and label $y_t$ at position $t$ (unary potential), $\phi_p(y_t, y_{t+1})$ is a feature vector capturing the transition between adjacent labels $y_t$ and $y_{t+1}$ (pairwise potential), $\mathbf{w} = (\mathbf{w}_u, \mathbf{w}_p)$ are the model parameters to be learned, and $Z(\mathbf{x}; \mathbf{w})$ is the partition function ensuring normalization:

$$Z(\mathbf{x}; \mathbf{w}) = \sum_{\mathbf{y}' \in \mathcal{Y}^T} \exp \left( \sum_{t=1}^{T} \mathbf{w}_{u}^{\top} \phi_{u}(x_{t}, y'_{t}) + \sum_{t=1}^{T-1} \mathbf{w}_{p}^{\top} \phi_{p}(y'_{t}, y'_{t+1}) \right). \quad (10.270)$$
<figure><img src="image_137.png" /><figcaption>Figure 10.56: Structured SVM Loss Contour ($\Delta = 1.0$)</figcaption></figure>

The summation over all possible label sequences $\mathbf{y}' \in \mathcal{Y}^T$ makes direct computation intractable for long sequences, but the chain structure allows efficient calculation using the forward-backward algorithm. The CRF loss for a training dataset $\{(\mathbf{x}^{(i)}, \mathbf{y}^{(i)})\}_{i=1}^N$ is the negative conditional log-likelihood:

$$ \mathcal{L}(\mathbf{w}) = - \sum_{i=1}^{N} \log P(\mathbf{y}^{(i)} | \mathbf{x}^{(i)}; \mathbf{w}) = \sum_{i=1}^{N} \left[ \log Z(\mathbf{x}^{(i)}; \mathbf{w}) - \left( \sum_{t=1}^{T} \mathbf{w}_{u}^{\top} \phi_{u}(x_{t}^{(i)}, y_{t}^{(i)}) + \sum_{t=1}^{T-1} \mathbf{w}_{p}^{\top} \phi_{p}(y_{t}^{(i)}, y_{t+1}^{(i)}) \right) \right] \quad (10.271) $$

Minimizing this loss maximizes the likelihood of the true label sequences under the model. The gradient of the loss with respect to the parameters reveals a crucial contrast between the empirical counts and the model's expected counts. The gradient for the unary parameters $\mathbf{w}_u$ is:

$$ \nabla_{\mathbf{w}_u} \mathcal{L}(\mathbf{w}) = \sum_{i=1}^{N} \left[ \sum_{t=1}^{T} \phi_u(x_t^{(i)}, y_t^{(i)}) - \sum_{t=1}^{T} \sum_{y'} P(y_t = y' | \mathbf{x}^{(i)}; \mathbf{w}) \phi_u(x_t^{(i)}, y') \right] \quad (10.272) $$

and for the pairwise parameters $\mathbf{w}_p$:

$$ \nabla_{\mathbf{w}_p} \mathcal{L}(\mathbf{w}) = \sum_{i=1}^{N} \left[ \sum_{t=1}^{T-1} \phi_p(y_t^{(i)}, y_{t+1}^{(i)}) - \sum_{t=1}^{T-1} \sum_{y', y''} P(y_t = y', y_{t+1} = y'' | \mathbf{x}^{(i)}; \mathbf{w}) \phi_p(y', y'') \right] \quad (10.273) $$

The first term in each gradient is the empirical feature count from the training data, and the second term is the expected feature count under the current model distribution $P(\mathbf{y} | \mathbf{x}^{(i)}; \mathbf{w})$. These marginal probabilities $P(y_t | \mathbf{x})$ and $P(y_t, y_{t+1} | \mathbf{x})$ are computed efficiently using the forward-backward algorithm, which leverages the Markov property to perform dynamic programming. The
<figure><img src="image_138.png" /><figcaption>Figure 10.57: Simplified CRF Loss vs Correct Label Score</figcaption></figure>

forward algorithm calculates

$$ \alpha_t(y) = P(y_t = y, x_{1:t}) \qquad (10.274) $$

the probability of being in state y at time t given the observations up to t, recursively:

$$ \alpha_1(y) = \exp(\mathbf{w}_u^\top \phi_u(x_1, y)), \quad \alpha_t(y) = \sum_{y'} \alpha_{t-1}(y') \exp(\mathbf{w}_u^\top \phi_u(x_t, y) + \mathbf{w}_p^\top \phi_p(y', y)) \qquad (10.275) $$

The backward algorithm similarly calculates

$$ \beta_t(y) = P(x_{t+1:T} | y_t = y) \qquad (10.276) $$

Therefore we have

$$ \beta_T(y) = 1, \quad \beta_t(y) = \sum_{y'} \beta_{t+1}(y') \exp(\mathbf{w}_u^\top \phi_u(x_{t+1}, y') + \mathbf{w}_p^\top \phi_p(y, y')) \qquad (10.277) $$

The partition function is obtained as

$$ Z(\mathbf{x}; \mathbf{w}) = \sum_{y} \alpha_{T}(y) \qquad (10.278) $$

The marginal probability for a single label is

$$ P(y_t = y | \mathbf{x}) = \frac{\alpha_t(y) \beta_t(y)}{Z(\mathbf{x})} \qquad (10.279) $$
CRF Loss Surface (Single Timestep, 2 Labels)

<figure><img src="image_139.png" /><figcaption>Figure 10.58: CRF Loss Surface (Single Timestep, 2 Labels)</figcaption></figure>

and for two consecutive labels is

$$P(y_t = y, y_{t+1} = y' | \mathbf{x}) = \frac{\alpha_t(y) \exp(\mathbf{w}_u^\top \phi_u(x_{t+1}, y') + \mathbf{w}_p^\top \phi_p(y, y')) \beta_{t+1}(y')}{Z(\mathbf{x})} \quad (10.280)$$

Optimization of the CRF loss is typically performed using iterative gradient-based methods like L-BFGS or stochastic gradient descent. The convexity of the negative log-likelihood ensures convergence to a global optimum, and $L_2$ regularization is often added to the loss to prevent overfitting:

$$\mathcal{L}_{\text{reg}}(\mathbf{w}) = \mathcal{L}(\mathbf{w}) + \frac{\lambda}{2} \|\mathbf{w}\|^2 \quad (10.281)$$

The CRF loss effectively captures dependencies between output labels, making it superior to independent classifiers for structured tasks like named entity recognition and part-of-speech tagging.

### 10.5.3.9.4 Connectionist Temporal Classification (CTC) Loss

Connectionist Temporal Classification (CTC) loss is an objective function designed for sequence-to-sequence learning tasks where the alignment between the input sequence

$$\mathbf{x} = (x_1, \dots, x_T) \quad (10.282)$$

and the target sequence

$$l = (l_1, \dots, l_U) \quad (10.283)$$
<figure><img src="image_140.png" /><figcaption>Figure 10.59: CRF Loss Contour (Single Timestep, 2 Labels)</figcaption></figure>

is unknown and $U \le T$, commonly encountered in speech recognition and handwriting recognition. The fundamental challenge is that the same target label sequence can be generated by multiple, variable-length alignments between the input and output. CTC addresses this by introducing a latent alignment path

$$ \pi = (\pi_1, \dots, \pi_T) \qquad (10.284) $$

where each $\pi_t$ belongs to an extended alphabet

$$ L' = L \cup \{\text{blank}\} \qquad (10.285) $$

which includes the original labels $L$ plus a special blank symbol (denoted here as "-"). The blank symbol represents no label emission and is crucial for handling repetitions and variable timing. The conditional probability of a path $\pi$ given the input sequence is factorized as a product of independent probabilities at each time step, which are output by a recurrent neural network with a softmax layer over $L'$:

$$ P(\pi | \mathbf{x}) = \prod_{t=1}^{T} y_{\pi_t}^{t} \qquad (10.286) $$

where $y_k^t$ is the probability of emitting symbol $k \in L'$ at time $t$.

A many-to-one mapping function $\mathcal{B}$, known as the CTC collapse function, is defined to remove all blank symbols and then merge consecutive duplicate symbols. For example,

$$ \mathcal{B}(a, -, a, b, b, -) = \mathcal{B}(-, -, a, a, -, b) = aab \qquad (10.287) $$
<figure><img src="image_141.png" /><figcaption>Figure 10.60: CTC Loss vs Probability of Correct Sequence</figcaption></figure>

The total probability of a given label sequence **l** is the sum of the probabilities of all paths that map to it under **B**:

$$P(\mathbf{l} | \mathbf{x}) = \sum_{\pi \in \mathcal{B}^{-1}(\mathbf{l})} P(\pi | \mathbf{x}) = \sum_{\pi \in \mathcal{B}^{-1}(\mathbf{l})} \prod_{t=1}^{T} y_{\pi_t}^t \qquad (10.288)$$

The CTC loss function for a training example (x, **l**) is the negative log-likelihood of this probability:

$$\mathcal{L}_{\text{CTC}}(\mathbf{l}, \mathbf{x}) = -\ln P(\mathbf{l} | \mathbf{x}) = -\ln \left( \sum_{\pi \in \mathcal{B}^{-1}(\mathbf{l})} \prod_{t=1}^{T} y_{\pi_t}^t \right) \qquad (10.289)$$

The central computational problem is efficiently calculating the sum over the exponentially large number of paths in $\mathcal{B}^{-1}(\mathbf{l})$. This is solved using a dynamic programming algorithm analogous to the forward-backward algorithm in Hidden Markov Models, known as the CTC forward-backward algorithm. The algorithm operates on a modified label sequence **l'** of length $2|\mathbf{l}| + 1$, which is created by inserting blank symbols at the beginning, end, and between every pair of labels in **l**. For example, if **l** = (a, b), then **l'** = (−, a, −, b, −). This ensures that valid paths can transition between labels only via a blank. Let the forward variable $\alpha_t(s)$ be the total probability of all valid paths of length $t$ that map to the first $s/2$ symbols of **l** (where $s$ is an index in **l'**). The recurrence relation for $\alpha_t(s)$ is complex due to the rules imposed by **B**: a path can only extend by staying on the same label, moving to the next label, or inserting a blank between non-repeating labels. The recurrence is:

$$\alpha_t(s) = y_{l'_s}^t \sum_{i=s-2}^{s} \alpha_{t-1}(i) \cdot \mathbf{1}_{\text{valid transition from } l'_i \text{ to } l'_s}, \qquad (10.290)$$
<figure><img src="image_142.png" /><figcaption>Figure 10.61: Approximate CTC Loss vs Per-Timestep Probability for Different Sequence Lengths</figcaption></figure>

where the indicator function 1 allows the transition if it conforms to the collapse rules (e.g., skipping a blank or a non-repeating label). The initial conditions are $\alpha_1(1) = y_1^-$ and $\alpha_1(2) = y_1^+$, with all others zero. The total probability $P(1 | \mathbf{x})$ is then the sum of the forward variables at the final time step for the last two nodes of $l'$:

$$P(1 | \mathbf{x}) = \alpha_T(|l'|) + \alpha_T(|l'| - 1) \qquad (10.291)$$

Similarly, a backward variable $\beta_t(s)$ is defined as the probability of all paths starting from time $t$ and position $s$ in $l'$ to the end, and it is computed backwards from $T$ to 1. The gradient of the CTC loss with respect to the network outputs $y_k^t$ is essential for backpropagation. It can be expressed using the forward and backward variables:

$$\frac{\partial \mathcal{L}_{\text{CTC}}}{\partial y_k^t} = - \frac{1}{P(1 | \mathbf{x})} \frac{\partial P(1 | \mathbf{x})}{\partial y_k^t} \qquad (10.292)$$

The derivative $\frac{\partial P(1|\mathbf{x})}{\partial y_k^t}$ is the sum of the probabilities of all paths that go through the symbol $k$ at time $t$, which is computed as $\sum_{s \in \text{Sets}(k)} \alpha_t(s)\beta_t(s)$, where $\text{Sets}(k)$ is the set of indices $s$ in $l'$ for which $l'_s = k$. This yields the final gradient:

$$\frac{\partial \mathcal{L}_{\text{CTC}}}{\partial y_k^t} = - \frac{1}{P(1 | \mathbf{x})} \sum_{s \in \text{Sets}(k)} \alpha_t(s) \beta_t(s) \qquad (10.293)$$

This gradient is then propagated back through the softmax layer to update the network weights. The CTC loss elegantly handles the alignment problem without pre-segmented data, making it a cornerstone of modern sequence-to-sequence models. Its main limitation is the conditional independence assumption between output frames, which more advanced models like RNN-Transducers aim to address.
CTC Loss Surface vs Sequence Length and Per-Timestep Probability

<figure><img src="image_143.png" /><figcaption>Figure 10.62: CTC Loss Surface vs Sequence Length and Per-Timestep Probability</figcaption></figure>

10.5.3.9.5 Maximum Margin Markov Networks Loss

Maximum Margin Markov Networks (M³N) loss is a structured prediction objective that integrates the max-margin principle of Support Vector Machines (SVMs) with the probabilistic graphical model framework of Markov networks, providing a convex optimization formulation for learning discriminative models over interdependent output variables. The core idea is to ensure that the score assigned to the correct output structure $\mathbf{y}_i$ is greater than the score of any other structure $\mathbf{y}$ by a margin that scales with the dissimilarity between them, as measured by a task-specific loss function $\Delta(\mathbf{y}_i, \mathbf{y})$. The score of a structure is defined linearly as

$$s(\mathbf{x}, \mathbf{y}; \mathbf{w}) = \mathbf{w}^{\top} \Psi(\mathbf{x}, \mathbf{y}) \qquad (10.294)$$

where $\Psi(\mathbf{x}, \mathbf{y})$ is a joint feature mapping that decomposes over the cliques of the Markov network. For a pairwise Markov network, this decomposition is

$$\mathbf{w}^{\top} \Psi(\mathbf{x}, \mathbf{y}) = \sum_{t=1}^{T} \mathbf{w}_{u}^{\top} \phi_{u}(x_{t}, y_{t}) + \sum_{(t, t') \in \mathcal{E}} \mathbf{w}_{p}^{\top} \phi_{p}(y_{t}, y_{t'}) \qquad (10.295)$$

where $\phi_u$ and $\phi_p$ are feature functions for nodes and edges respectively, and $\mathcal{E}$ is the set of edges in the graph. The M³N loss for a single example $(\mathbf{x}_i, \mathbf{y}_i)$ is formulated as the structured hinge loss:

$$L_i(\mathbf{w}) = \max_{\mathbf{y} \in \mathcal{Y}} [\Delta(\mathbf{y}_i, \mathbf{y}) + \mathbf{w}^{\top} \Psi(\mathbf{x}_i, \mathbf{y}) - \mathbf{w}^{\top} \Psi(\mathbf{x}_i, \mathbf{y}_i)] \qquad (10.296)$$
<figure><img src="image_144.png" /><figcaption>Figure 10.63: CTC Loss Contour vs Sequence Length and Per-Timestep Probability</figcaption></figure>

<figure><img src="image_145.png" /><figcaption>Figure 10.64: Maximum Margin Markov Networks (M³N) Loss</figcaption></figure>
This loss is zero if the margin constraint

$$ \mathbf{w}^{\top}\Psi(\mathbf{x}_i, \mathbf{y}_i) - \mathbf{w}^{\top}\Psi(\mathbf{x}_i, \mathbf{y}) \geq \Delta(\mathbf{y}_i, \mathbf{y}) \quad (10.297) $$

holds for all $\mathbf{y} \in \mathcal{Y}$, and positive otherwise. The complete regularized objective function is:

$$ \min_{\mathbf{w}, \xi \ge 0} \frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{i=1}^{N} \xi_i \quad \text{subject to} \quad \mathbf{w}^{\top}\Psi(\mathbf{x}_i, \mathbf{y}_i) - \mathbf{w}^{\top}\Psi(\mathbf{x}_i, \mathbf{y}) \ge \Delta(\mathbf{y}_i, \mathbf{y}) - \xi_i, \quad \forall i, \forall \mathbf{y} \in \mathcal{Y} \quad (10.298) $$

where $\xi_i$ are slack variables allowing soft margins and $C$ is a regularization parameter. This for-

<figure><img src="image_146.png" /><figcaption>Figure 10.65: M³N Loss Surface ($\Delta = 1.0$)</figcaption></figure>

mulation has an exponentially large number of constraints, but it can be optimized using the cutting-plane algorithm, which iteratively adds the most violated constraint for each training example. The most violated constraint for example $i$ is found by solving the loss-augmented inference problem:

$$ \mathbf{y}_i^* = \arg \max_{\mathbf{y} \in \mathcal{Y}} [\Delta(\mathbf{y}_i, \mathbf{y}) + \mathbf{w}^{\top}\Psi(\mathbf{x}_i, \mathbf{y})] \quad (10.299) $$

The key computational requirement is that this maximization must be tractable. When the loss function $\Delta$ decomposes additively over the graph structure (e.g., Hamming loss), and the feature mapping $\Psi$ decomposes over the same cliques, dynamic programming or graph cuts can be used for efficient exact inference.
The dual formulation of the M³N objective provides insight into the representation of the solution. The dual problem involves maximizing a concave objective over dual variables $\alpha_{i,y} \ge 0$ for each example $i$ and each structure $y$:

$$ \max_{\alpha \ge 0} \sum_{i,y} \alpha_{i,y} \Delta(\mathbf{y}_i, \mathbf{y}) - \frac{1}{2} \left\| \sum_{i,y} \alpha_{i,y} (\Psi(\mathbf{x}_i, \mathbf{y}_i) - \Psi(\mathbf{x}_i, \mathbf{y})) \right\|^2 \quad (10.300) $$

subject to $\sum_y \alpha_{i,y} = C$, $\forall i$. The optimal weight vector is given by

$$ \mathbf{w}^* = \sum_{i,y} \alpha_{i,y} (\Psi(\mathbf{x}_i, \mathbf{y}_i) - \Psi(\mathbf{x}_i, \mathbf{y})) \quad (10.301) $$

The Karush-Kuhn-Tucker (KKT) conditions for optimality require that $\alpha_{i,y} > 0$ only for structures $y$ that achieve the maximum in the loss-augmented inference problem for example $i$, meaning the solution is sparse and only supported by a small set of "active" structures. The gradient of the primal loss $L_i(\mathbf{w})$ with respect to $\mathbf{w}$ is a subgradient:

$$ \partial_{\mathbf{w}} L_i(\mathbf{w}) = \Psi(\mathbf{x}_i, \mathbf{y}_i^*) - \Psi(\mathbf{x}_i, \mathbf{y}) \quad (10.302) $$

which can be used in subgradient descent optimization. The M³N framework generalizes multiclass SVMs and provides a principled way to incorporate structural dependencies into max-margin learning, with strong theoretical guarantees on generalization error. However, its practical applicability

<figure><img src="image_147.png" /><figcaption>Figure 10.66: Contour Plot of M³N Loss ($\Delta = 1.0$)</figcaption></figure>

depends critically on the ability to solve the loss-augmented inference problem efficiently, which
restricts its use to structures where exact inference is feasible, such as sequences, trees, or graphs with low treewidth. The margin rescaling approach in M³N can be contrasted with slack rescaling, which uses constraints of the form

$$ \mathbf{w}^{\top}(\Psi(\mathbf{x}_i, \mathbf{y}_i) - \Psi(\mathbf{x}_i, \mathbf{y})) \geq 1 - \xi_i / \Delta(\mathbf{y}_i, \mathbf{y}) \quad (10.303) $$

but margin rescaling is more commonly used due to its more straightforward integration with efficient inference algorithms.

### 10.5.3.9.6 Listwise Ranking Losses

Listwise ranking losses are a class of objective functions designed for learning to rank by directly modeling the probability distribution over the entire permutation of items in a list, thereby optimizing the global ordering rather than focusing on independent pairwise comparisons. The fundamental principle is to define a mapping from the predicted scores of items, assigned by a model $f(x_i; \theta)$, to a probability distribution over all possible permutations of those items, and then to minimize the discrepancy between this predicted distribution and the idealized distribution derived from the ground truth ranking. The ground truth is typically represented as a set of relevance scores $\mathbf{y} = (y_1, y_2, \dots, y_n)$ for a list of $n$ items associated with a query. The ListNet framework, a prominent listwise approach, utilizes the Plackett-Luce model to define the probability of a particular permutation $\pi$ of the indices $\{1, 2, \dots, n\}$. According to this model, the probability of a permutation $\pi$ is given by the product of sequential conditional probabilities:

$$ P(\pi|\mathbf{s}) = \prod_{j=1}^{n} \frac{\exp(s_{\pi(j)})}{\sum_{k=j}^{n} \exp(s_{\pi(k)})} \quad (10.304) $$

where $\mathbf{s} = (s_1, \dots, s_n)$ are the predicted scores from the model, $s_i = f(x_i; \theta)$, and $\pi(j)$ denotes the item placed at the $j$-th position in the permutation. This model captures the notion that the probability of an item being ranked first is proportional to the exponential of its score, and subsequent positions are determined by repeatedly applying the same logic to the remaining items. The ground truth distribution is often not a full permutation but a set of relevance labels. To handle this, the top-one probability is used as a surrogate. The top-one probability for item $i$ is the probability that it is ranked first under the Plackett-Luce model:

$$ P_{\text{top-one}}(i|\mathbf{s}) = \frac{\exp(s_i)}{\sum_{k=1}^{n} \exp(s_k)} \quad (10.305) $$

Similarly, the ground truth top-one probability is defined using the actual relevance scores:

$$ P_{\text{top-one}}(i|\mathbf{y}) = \frac{\phi(y_i)}{\sum_{k=1}^{n} \phi(y_k)} \quad (10.306) $$

where $\phi$ is a transformation function, typically chosen as an exponential function $\phi(y) = \exp(y)$ to maintain a softmax form, or a linear function for simplicity. The loss function for a single query is then defined as the cross-entropy between the two top-one probability distributions over the $n$ items:

$$ L(\theta) = - \sum_{i=1}^{n} P_{\text{top-one}}(i|\mathbf{y}) \log P_{\text{top-one}}(i|\mathbf{s}) = - \sum_{i=1}^{n} \frac{\phi(y_i)}{\sum_{k=1}^{n} \phi(y_k)} \log \left( \frac{\exp(s_i)}{\sum_{k=1}^{n} \exp(s_k)} \right) \quad (10.307) $$

If $\phi$ is the exponential function, this simplifies to:

$$ L(\theta) = - \sum_{i=1}^{n} \frac{\exp(y_i)}{\sum_{k=1}^{n} \exp(y_k)} \left( s_i - \log \sum_{k=1}^{n} \exp(s_k) \right) = \log \sum_{k=1}^{n} \exp(s_k) - \sum_{i=1}^{n} \frac{\exp(y_i)}{\sum_{k=1}^{n} \exp(y_k)} s_i \quad (10.308) $$
<figure><img src="image_148.png" /><figcaption>Figure 10.67: Listwise Ranking Losses (ListNet vs ListMLE)</figcaption></figure>

The gradient of this loss with respect to the predicted score $s_j$ of a single item is:

$$ \frac{\partial L}{\partial s_j} = \frac{\exp(s_j)}{\sum_{k=1}^{n} \exp(s_k)} - \frac{\exp(y_j)}{\sum_{k=1}^{n} \exp(y_k)} = P_{\text{top-one}}(j|\mathbf{s}) - P_{\text{top-one}}(j|\mathbf{y}) \quad (10.309) $$

This gradient has an elegant interpretation: it is the difference between the predicted probability that item $j$ is the most relevant and the ground truth probability. This gradient is then backpropagated through the scoring function $f(x_j; \theta)$ to update the model parameters $\theta$. The loss function is convex in the scores $\mathbf{s}$, and the optimization process effectively pushes the predicted score distribution to match the ground truth relevance distribution. A more direct but computationally intensive listwise loss is the likelihood loss, which aims to maximize the probability of the optimal permutation. If the ground truth is a full ordering $\pi_g$, the loss is the negative log-likelihood under the Plackett-Luce model:

$$ L(\theta) = -\log P(\pi_g|\mathbf{s}) = -\sum_{j=1}^{n} \left( s_{\pi_g(j)} - \log \sum_{k=j}^{n} \exp(s_{\pi_g(k)}) \right) \quad (10.310) $$

The gradient of this loss for a score $s_i$ involves a sum over all positions in the permutation where the item appears and all items that were ranked below it, making it more complex than the top-one approximation. Listwise losses are considered superior to pairwise and pointwise losses in information retrieval because they directly optimize the entire list structure, which is the ultimate goal of ranking. However, their computational cost is higher due to the need to compute the normalization term over all items for each query, and they require meaningful ground truth relevance scores rather than just binary preferences. The mathematical rigor of listwise approaches lies in their direct formulation as probabilistic models of permutation, providing a principled foundation for learning ranking functions that perform well on metrics like Normalized Discounted Cumulative Gain (NDCG).

### 10.5.3.9.7 Pairwise Ranking Losses
Listwise Ranking Losses (3D Surfaces)

<figure><img src="image_149.png" /><figcaption>Figure 10.68: Listwise Ranking Losses (3D Surfaces)</figcaption></figure>

<figure><img src="image_150.png" /><figcaption>Figure 10.69: Listwise Ranking Loss Contours</figcaption></figure>

Pairwise ranking losses constitute a fundamental family of objective functions in metric learning and information retrieval, operating on the principle of comparing pairs of items to learn a representation or scoring function that imposes a desired ordinal structure on the data. The core objective is to ensure that for a given anchor instance $x_a$, the distance to a positive (similar) instance $x_p$ is smaller than the distance to a negative (dissimilar) instance $x_n$ by at least a specified margin. Let $f_\theta(x)$ be an embedding function parameterized by $\theta$ that maps an input $x$ to a vector in $\mathbb{R}^d$, and let $d(f_\theta(x_i), f_\theta(x_j))$ be a distance metric, typically the Euclidean distance $\|f_\theta(x_i) - f_\theta(x_j)\|_2$. The generalized pairwise hinge loss, or margin ranking loss, for a triplet $(x_a, x_p, x_n)$ is defined as:

$$L_{\text{pairwise}}(x_a, x_p, x_n; \theta) = \max(0, m + d(f_{\theta}(x_a), f_{\theta}(x_p)) - d(f_{\theta}(x_a), f_{\theta}(x_n))), \quad (10.311)$$

where $m > 0$ is the margin hyperparameter. This loss is zero if the negative instance is already more distant from the anchor than the positive instance by at least the margin $m$, and increases
<figure><img src="image_151.png" /><figcaption>Figure 10.70: Pairwise Ranking Losses</figcaption></figure>

linearly with the violation otherwise. The gradient of this loss with respect to the embedding vectors drives the learning process. When the loss is active

$$m + d_{ap} - d_{an} > 0 \qquad (10.312)$$

The gradients are:

$$\frac{\partial L}{\partial f_{\theta}(x_a)} = 2 \left( \frac{f_{\theta}(x_a) - f_{\theta}(x_p)}{d_{ap}} - \frac{f_{\theta}(x_a) - f_{\theta}(x_n)}{d_{an}} \right), \qquad (10.313)$$

$$\frac{\partial L}{\partial f_{\theta}(x_p)} = -2 \frac{f_{\theta}(x_a) - f_{\theta}(x_p)}{d_{ap}}, \qquad (10.314)$$

$$\frac{\partial L}{\partial f_{\theta}(x_n)} = 2 \frac{f_{\theta}(x_a) - f_{\theta}(x_n)}{d_{an}}, \qquad (10.315)$$

where

$$d_{ap} = d(f_{\theta}(x_a), f_{\theta}(x_p)) \qquad (10.316)$$

and

$$d_{an} = d(f_{\theta}(x_a), f_{\theta}(x_n)) \qquad (10.317)$$

These gradients pull the anchor and positive together while pushing the anchor and negative apart, with forces inversely proportional to their current distances, providing a stable optimization landscape. A more probabilistic formulation is provided by the Bayesian Personalized Ranking (BPR) loss, which is derived from a maximum posterior estimator for implicit feedback data. The BPR objective maximizes the probability that a user prefers a positive item $x_p$ over a negative item $x_n$. The probability is modeled by a sigmoid function over the difference of their scores:

$$P(x_p > x_n) = \sigma(\hat{s}(x_p) - \hat{s}(x_n)) \qquad (10.318)$$
<figure><img src="image_152.png" /><figcaption>Figure 10.71: Pairwise Ranking Loss Landscapes</figcaption></figure>

where $\hat{s}(x)$ is a scoring function. The BPR loss for a triplet $(u, x_p, x_n)$ is the negative log-likelihood:

$$L_{\text{BPR}} = -\ln \sigma(\hat{s}(x_p) - \hat{s}(x_n)) = -\ln \frac{1}{1 + e^{-(\hat{s}(x_p) - \hat{s}(x_n))}} = \ln (1 + e^{-(\hat{s}(x_p) - \hat{s}(x_n))}). \quad (10.319)$$

This is a smooth approximation of the hinge loss. Its gradient with respect to the score difference is:

$$\frac{\partial L_{\text{BPR}}}{\partial(\hat{s}(x_p) - \hat{s}(x_n))} = -\frac{e^{-(\hat{s}(x_p) - \hat{s}(x_n))}}{1 + e^{-(\hat{s}(x_p) - \hat{s}(x_n))}} = -\sigma(\hat{s}(x_n) - \hat{s}(x_p)), \quad (10.320)$$

which provides a well-defined, non-zero gradient for all inputs. The gradients with respect to the model parameters $\theta$ are then computed via the chain rule based on the specific form of $\hat{s}(x)$. For

3D Surfaces of Pairwise Ranking Losses

<figure><img src="image_153.png" /><figcaption>Figure 10.72: 3D Surfaces of Pairwise Ranking Losses</figcaption></figure>
a scoring function based on Euclidean distance in an embedding space, where

$$ \hat{s}(x) = -\|f_{\theta}(u) - f_{\theta}(x)\|_{2} \qquad (10.321) $$

the BPR loss encourages the distance between the user and the positive item to be less than the distance to the negative item. The gradient computations become analogous to the margin ranking loss but with a weighting factor from the sigmoid derivative. The overall objective for a dataset is the sum over all valid triplets $(u, x_p, x_n)$, which can be sampled from the observed interactions. The optimization of pairwise ranking losses is efficient because it scales with the number of pairs rather than the full list, but it requires careful sampling of informative triplets. Hard negative mining, which selects negative instances that are currently close to the anchor, is critical for efficient learning and high model performance. The mathematical rigor of pairwise losses lies in their direct optimization of the relative order between items, which aligns with ranking metrics more closely than pointwise approaches, while remaining computationally more tractable than full listwise methods.

### 10.5.3.9.8 Contrastive Divergence (CD) Loss

Contrastive Divergence (CD) loss is an approximate maximum likelihood learning algorithm for energy-based probabilistic models, most notably Restricted Boltzmann Machines (RBMs), which provides an efficient method to estimate the gradient of the log-likelihood function without requiring intractable computations over the entire model distribution. The core challenge in training an RBM

<figure><img src="image_154.png" /><figcaption>Figure 10.73: Contrastive Divergence (CD) Loss vs Weight</figcaption></figure>

with parameters $\theta = (\mathbf{W}, \mathbf{a}, \mathbf{b})$—where $\mathbf{W}$ is the weight matrix between visible units $\mathbf{v}$ and hidden units $\mathbf{h}$.