# Additive Orthant Loss for Deep Face Recognition
## 3 Orthant Loss
**Additive center separating margin. ** Figure 2a shows that a softmax-based loss function [28; 29; 30; 31; 32; 33; 34; 35] learns the distribution of features centered on \(\mathbf{w}_{1}\) and \(\mathbf{w}_{2}\) to improve the intra-class compactness using an angular margin \(m_{\theta}\), respectively. However, since features with small magnitudes can easily move to the feature spaces of other classes, the softmax-based loss function using an angular margin has difficulty in distinguishing these features correctly. To overcome this problem, this paper uses an additive margin \(m_{c}\) to improve the inter-class separability by pushing the features away from the origin, as illustrated in Figure 2b. In the Center loss [25], which also uses an additive term, both ambiguous and discriminative features can be gathered at the center of each class without considering the feature magnitudes. On the other hand, both ambiguous and discriminative features move away from the origin in the Orthant loss, which improves the inter-class separability by increasing their magnitudes. **One-bit quantization and element-wise operation. ** To improve the intra-class compactness, we quantize each weight vector of the last fully connected layer and perform element-wise product on the normalized feature vectors. When the weight vectors are not quantized, the signs of some elements of normalized feature vectors can be easily changed if the magnitudes of the corresponding elements of the weight vectors are small. Therefore, we train the DCNN model so that the signs of feature and weight vectors are identical by multiplying the normalized feature by the 1-bit quantized weight vector. Then, we can improve the intra-class compactness through the element-wise sign consistency between feature and quantized weight vectors. Despite the low computational cost, the 1-bit quantization may reduce the recognition accuracy due to the inaccurate gradient update [45]. Since 1-bit quantization has zero gradients on positive and negative sides, weight vectors of the last fully connected layer are updated in a wrong direction using a surrogate gradient function such as hard hyperbolic tangent (tanh) function. To maintain the accuracy while reducing the computational cost, we exclude the 1-bit quantized weight vectors from the gradient calculation. In other words, the weight vectors are updated using the softmax-based loss \(\mathcal{L}_{\text{Soft}}\) without the Orthant loss \(\mathcal{L}_{\text{Orth}}\) in the back-propagation process. Note that the Orthant loss \(\mathcal{L}_{\text{Orth}}\) is combined to a softmax-based loss when updating the set of parameters \(\mathcal{W}\) of the DCNN. **Squared loss function. ** As \(r\) increases, the gradient for a misclassified feature in the rescaled softplus function \(f_{s}(x)\) is approximately \(-1\). Since \(f_{s}(x)\) has the gradient of \(-1\) for both easy and hard features to distinguish, features do not quickly converge. To improve the convergence speed, we introduce a squared loss to give a larger weight to a hard feature. In many studies [44,46], the squared hinge function has been used to give a larger weight to misclassified features. Inspired by this idea, we introduce a squared softplus function to expect fast convergence of misclassified features. In the learning process, 1-bit quantization and element-wise operation of the Orthant loss make feature vectors have the same sign as the weight vectors of the corresponding class. However, feature vectors may converge to an orthant of the corresponding weight vector before the distances between weight vectors are sufficiently large, which fixes the sign of the weight vectors prematurely. In other words, since the speed at which feature vectors converge to the corresponding weight vector is faster than the speed at which the weight vectors move away from each other, the sign of the weight vectors may be fixed before the distance between the weight vectors increases.