# Additive Orthant Loss for Deep Face Recognition
## 2 Background
### Some Known Loss Functions  
The softmax function [22] has been mainly used as a loss function, which is given by:  
\[\begin{split}\mathcal{L}_{\text{Softmax}}&=-\frac{1} {N}\sum_{i=1}^{N}\log\frac{e^{\mathbf{w}_{y_{i}}^{T}\mathbf{x}_{i}}}{e^{ \mathbf{w}_{y_{i}}^{T}\mathbf{x}_{i}}+\sum_{j\
eq y_{i}}e^{\mathbf{w}_{j}^{T} \mathbf{x}_{i}}}\\ &=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{e^{\mathbf{y}_{i}\cos{( \theta_{y_{i}})}}}{e^{\mathbf{y}_{ij}\cos{(\theta_{y_{i}})}}+\sum_{j\
eq y_{i} }e^{\mathbf{\hat{\varepsilon}}^{j}\cos{(\theta_{j})}}},\end{split} \tag{1}\]  
where \(\mathbf{w}_{y_{i}}\) and \(\mathbf{w}_{j}\) are the \(y_{i}\)th and the \(j\)th column of \(\mathbf{W}\), respectively, \(\mathbf{x}_{i}\) is a feature obtained by inputting the \(i\)th image into DCNN, and \(N\) is the batch size. In (1), \(\mathbf{w}_{y_{i}}^{T}\mathbf{x}_{i}=s_{y_{i}}\cos{(\theta_{y_{i}})}\) and \(\mathbf{w}_{j}^{T}\mathbf{x}_{i}=s_{j}\cos{(\theta_{j})}\). While \(\mathcal{L}_{\text{Softmax}}\) is known in [23] to have good capability to improve the separability between classes (inter-class separability), it has insufficient capability to compress the feature space of each class (intra-class compactness).  
In order to improve the intra-class compactness, many softmax-based loss functions have been studied. The Center loss [25] is given by  
\[\mathcal{L}_{\text{Center}}=\mathcal{L}_{\text{Softmax}}+\frac{\lambda}{2} \sum_{i=1}^{N}\lVert\mathbf{x}_{i}-\mathbf{c}_{y_{i}}\rVert^{2}, \tag{2}\]  
where \(\mathbf{c}_{y_{i}}\) is the center position of the \(y_{i}\)th class, \(\lambda\) is a hyperparameter for balancing the two terms, and \(\lVert\cdot\rVert\) is the \(l_{2}\)-norm of a vector. \(\mathcal{L}_{\text{Center}}\) improves the intra-class compactness by penalizing the distances between features and their corresponding class centers.  
Figure 1: Deep face recognition (FR) system model.  
The SphereFace [28] is formulated as:  
\[\mathcal{L}_{\text{Sphere}}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{e^{\|\mathbf{x}_{i }\|\psi(\theta_{y_{i}})}}{e^{\|\mathbf{x}_{i}\|\psi(\theta_{y_{i}})}+\sum_{j\
eq y _{i}}e^{\|\mathbf{x}_{i}\|\cos{(\theta_{j})}}},\]  
where \(\psi(\theta_{y_{i}})=(-1)^{k}\cos(m_{p}\theta_{y_{i}})-2k\), \(\theta_{y_{i}}\in\left[\frac{k\pi}{m_{p}},\frac{(k+1)\pi}{m_{p}}\right]\) and \(k\in[0,m_{p}-1]\), in which \(m_{p}\) is a multiplicative angular margin. \(\mathcal{L}_{\text{Sphere}}\) uses an multiplicative angular margin to improve the intra-class compactness on a hypersphere manifold.  
The ArcFace [33] has been proposed by:  
\[\mathcal{L}_{\text{ArcFace}}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{e^{s\cos{( \theta_{y_{i}}+m_{\theta})}}}{e^{s\cos{(\theta_{y_{i}}+m_{\theta})}}+\sum_{j \
eq y_{i}}e^{s\cos{(\theta_{j})}}},\]  
where \(m_{\theta}\) is an additive angular margin, and \(s\) is a scaling constant. \(\mathcal{L}_{\text{ArcFace}}\) improves the intra-class compactness using an additive angular margin on a hypersphere manifold.  
The MagFace [35] is given by:  
\[\mathcal{L}_{\text{MagFace}}=\frac{1}{N}\sum_{i=1}^{N}\bigl{(}\mathcal{L}_{ \text{Mag}}+\lambda_{g}g(\|\mathbf{x}_{i}\|)\bigr{)},\]  
where \(\mathcal{L}_{\text{Mag}}=-\log\frac{e^{s\cos{(\theta_{y_{i}}+m_{g}(\|\mathbf{ x}_{i}\|))}}}{e^{s\cos{(\theta_{y_{i}}+m_{g}(\|\mathbf{x}_{i}\|))}}+\sum_{j \
eq y_{i}}e^{s\cos{(\theta_{j})}}}\), in which \(m_{g}(\|\mathbf{x}_{i}\|)\) is a strictly increasing function, \(g(\|\mathbf{x}_{i}\|)\) is a strictly decreasing function and \(\lambda_{g}\) is a hyperparameter that controls the trade-off between \(\mathcal{L}_{\text{Mag}}\) and \(g(\|\mathbf{x}_{i}\|)\). \(\mathcal{L}_{\text{MagFace}}\) uses an adaptive margin \(m_{g}(\|\mathbf{x}_{i}\|)\) and regularization \(g(\|\mathbf{x}_{i}\|)\) to improve the intra-class compactness.