# Double Additive Margin Softmax Loss for Face Recognition
## 2 Preliminaries  
In order to clearly understand the proposed DAM-Softmax loss, we will briefly review the classical softmax loss and AM-softmax loss. The classical softmax loss is formulated by  
\[L_{1}=-\frac{1}{N}\sum_{i=1}^{N}log\frac{e^{\mathbf{w}_{g_{i}}^{\top}\mathbf{ f}_{i}}}{\sum_{c=1}^{C}e^{\mathbf{w}_{c}^{\top}\mathbf{f}_{i}}} \tag{1}\]  
where \(\mathbf{w}_{c}\) ( \(c=1,\ldots,C\), \(C\) is the number of classes) denotes the weight vector of the last fully connected classifier layer, \(\mathbf{f}_{i}\) is the learned deep feature input vector of the last fully connected classifier layer corresponding to the original input \(\mathbf{x}_{i}\) with the label \(y_{i}\), and \(N\) is the number of training samples in a minibatch. The inner product, \(\mathbf{w}_{c}^{\top}\mathbf{f}_{i}\), between \(\mathbf{w}_{c}\) and \(\mathbf{f}_{i}\) can be also factorized into \(\|\mathbf{w}_{c}\|\|\mathbf{f}_{i}\|cos(\theta_{c})\) where \(\theta_{c}\) is the angle between \(\mathbf{w}_{c}\) and \(\mathbf{f}_{i}\), the loss can thus be rewritten as  
\[L_{2}=-\frac{1}{N}\sum_{i=1}^{N}log\frac{e^{\|\mathbf{w}_{g_{i}}\|\|\mathbf{f }_{i}\|cos(\theta_{g_{i}})}}{\sum_{c=1}^{C}e^{\|\mathbf{w}_{c}\|\|\mathbf{f}_{ i}\|cos(\theta_{c})}} \tag{2}\]  
The A-Softmax is a new loss function derived from the classical Softmax loss which proposed to impose a constraint to make \(\|\mathbf{w}_{c}\|=1\) and generalize the modified softmax loss to angular softmax (A-Softmax) loss by replacing \(\|\mathbf{f}_{i}\|cos(\theta_{g_{i}})\) with \(\|\mathbf{f}_{i}\|\psi(\theta_{g_{i}})\),\[L_{3}=-\frac{1}{N}\sum_{i=1}^{N}log\frac{e^{\|\mathbf{f}_{i}\|\psi(\theta_{y_{i}})} }{e^{\|\mathbf{f}_{i}\|\Psi(\theta_{y_{i}})}+\sum_{c=1,c\
eq y_{i}}^{C}e^{\| \mathbf{f}_{c}\|cos(\theta_{c})}} \tag{3}\]  
where the authors proposed to define \(\psi(\theta)\) as \(\psi(\theta)=(-1)^{k}cos(m\theta)-2k\), \(\theta\in[\frac{k\pi}{m},\frac{(k+1)\pi}{m}]\) and \(k\in[0,m-1]\) for removing the restriction which \(\theta\) must be in the range of \([0,\frac{\pi}{m}]\).  
In the AM-Softmax loss, the authors suggested to introduce an additive margin to its decision boundary by defining \(\psi(\theta)\) as \(cos(\theta)-m\). In addition, both the deep feature vector \(\mathbf{f}_{i}\) and weight vectors \(\mathbf{w}_{c}\) are normalized during the implementation. Thus, The AM-Softmax loss is given by  
\[\begin{split} L_{4}&=-\frac{1}{N}\sum_{i=1}^{N}log \frac{e^{s\cdot(cos(\theta_{y_{i}})-m)}}{e^{s\cdot(cos(\theta_{y_{i}})-m)}+ \sum_{c=1,c\
eq y_{i}}^{C}e^{s\cdot cos(\theta_{c})}}\\ &=-\frac{1}{N}\sum_{i=1}^{N}log\frac{e^{s\cdot(\mathbf{w}_{y_{i} }^{\top}\mathbf{f}_{i}-m)}}{e^{s\cdot(\mathbf{w}_{y_{i}}^{\top}\mathbf{f}_{i}- m)}+\sum_{c=1,c\
eq y_{i}}^{C}e^{s\cdot\mathbf{w}_{c}^{\top}\mathbf{f}_{i}}}\end{split} \tag{4}\]  
where \(s\) is a hyper-parameter for scaling the cosine values.  
In order to simultaneously enlarge the between-class angular margin and compress the within-class angular variation, It is clear to learn that both the A-Softmax loss and AM-Softmax loss share a common idea to generalize the original softmax loss to angular softmax loss by introducing an integer \(m\) to quantitatively control the decision boundary. specifically, In binary class case, a learned feature \(\mathbf{f}\) from class 1 is given and \(\theta_{i}\) is the angle between \(\mathbf{f}\) and \(\mathbf{w}_{i}\). A-softmax loss requires \(cos(m\theta_{1})>cos(\theta_{2})\) to correctly classify \(\mathbf{f}\). AM-softmax loss instead proposes \(cos(\theta_{1})-m>cos(\theta_{2})\) to correctly classify \(\mathbf{f}\). Both of them explicitly enforce the intra-class compactness to achieve more discriminative deep features by imposing an intra-class angular margin in the multiplicative manner and in the additive manner, respectively. Compared with the A-Softmax loss, AM-Softmax loss is simpler which is simpler and reaches better performance. In addition, It is much easier to implement because the computation of the gradient for back-propagation is no longer required.