# Additive Orthant Loss for Deep Face Recognition
## 3 Orthant Loss  
In this paper, we propose an additive orthant loss, termed the Orthant loss, to improve the inter-class separability and the intra-class compactness. The Orthant loss learns the distributions of features away from the origin and compresses the feature spaces by attempting to make the features of the same identity have the same signs. The Orthant loss is proposed by:  
\[\mathcal{L}_{\text{Orth}}=\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{l}a\left[\frac {1}{r}\log\left(1+e^{-r(\tilde{x}_{i,k}\mathcal{Q}_{1}(w_{y_{i},k})-m_{c})} \right)\right]^{2}, \tag{3}\]  
where \(\tilde{x}_{i,k}\) is the \(k\)th element of a normalized feature \(\tilde{\mathbf{x}}_{i}=\frac{\mathbf{x}_{i}}{\|\mathbf{x}_{i}\|},w_{y_{i},k}\) is the \(k\)th element of \(\mathbf{w}_{y_{i}}\) and \(\mathcal{Q}_{1}(w)\) is the \(1\)-bit quantization function, i.e., \(\mathcal{Q}_{1}(w)=+1\) if \(w>0\) and \(-1\) otherwise. In addition, \(m_{c}\) is an additive margin, \(a\) is a hyperparamter to adjust the size of the loss, \(r\) is a hyperparameter to control the slope of the rescaled softplus function, \(l\) is the feature dimension and \(N\) is the batch size. The proposed algorithm of \(\mathcal{L}_{\text{Orth}}\) is given in Algorithm 1. The reasoning behind the design of Orthant loss is to make each feature element have the same sign as the weight vector of each class by making \(\tilde{x}_{i,k}\mathcal{Q}_{1}(w_{y_{i},k})>0\) using \(1\)-bit quantization and element-wise operation, which improves the intra-class compactness. In addition, the Orthant loss makes the magnitude of each feature element larger than \(m_{c}\) to improve the inter-class separability by using the additive margin and the rescaled softplus function. Since \(\mathcal{L}_{\text{Orth}}\) only considers the feature and weight vectors of target identity \(y_{i}\) in (3), we combine \(\mathcal{L}_{\text{Orth}}\) with a softmax-based loss \(\mathcal{L}_{\text{Soft}}\), taking into account non-target identities \(j\
eq y_{i}\), which gives a final loss:  
\[\mathcal{L}_{\text{Final}}=\mathcal{L}_{\text{Soft}}+\mathcal{L}_{\text{Orth}}. \tag{4}\]```
0: Feature vectors \(\mathbf{x}_{i}\), Weight matrix \(\mathbf{W}=(\mathbf{w}_{1},\cdots,\mathbf{w}_{M})\), Ground-Truth ID \(y_{i}\), Hyper-parameters \(a,r\) and \(m_{c}\). 1: Normalize the feature vectors by \(\bar{\mathbf{x}}_{i}=\frac{\mathbf{x}_{i}}{\|\mathbf{x}_{i}\|}\)
2: Clone and detach the weight matrix \(\mathbf{W}\) to prevent gradient computation by \(\mathbf{W}=\mathbf{W}\).clone().detach()
3: Quantize the chosen weight vectors corresponding to \(y_{i}\) by \(\bar{\mathbf{w}}_{y_{i}}=\left[Q_{1}(w_{y_{i},1}),Q_{1}(w_{y_{i},2}),\cdots,Q_ {1}(w_{y_{i},l})\right]\)
4: Calculate the element-wise magnitudes including margin \(m_{c}\) by \(\mathbf{u}_{i}=\bar{\mathbf{w}}_{y_{i}}\bigodot\bar{\mathbf{x}}_{i}-m_{c}\), where \(\bigodot\) is the element-wise product of two vectors. 5: Calculate the proposed loss in (3) using the rescaled softplus function for a batch by \(\mathcal{L}_{\text{Orth}}=\frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{l}a\left[\frac {1}{r}\log\left(1+e^{-ru_{i,k}}\right)\right]^{2}\)
6: Loss score \(\mathcal{L}_{\text{Orth}}\)
```

**Algorithm 1** The pseudo-code of Orthant loss \(\mathcal{L}_{\text{Orth}}\)

In what follows, we describe the features of the Orthant loss. **Rescaled softplus function. ** To learn the distributions of features to be well separated, we use the rescaled softplus function [43]\(f_{s}(x)=\frac{1}{r}\log\left(1+e^{-rx}\right)\), where \(r\) controls the slope of \(f_{s}(x)\). As \(r\) goes to infinity, \(f_{s}(x)\) converges approximately into the hinge function [44]\(f_{h}(x)=\max\left(0,-x\right)\). In \(f_{h}(x)\), if \(x\geq 0\), learning no longer proceeds as the gradient becomes 0. On the other hand, in \(f_{s}(x)\), even if \(x\geq 0\), learning is possible so that \(x\) is farther away from zero since the gradient of \(f_{s}(x)\) is not zero. Therefore, we use \(f_{s}(x)\) to improve the inter-class separability by distinguishing features around the origin.