## 3 Inserting Scalable BN Layer to Improve the Classification Performance of Softmax Loss Function  
Through the observation of Softmax and cross-entropy loss, we found that the calculation of cross-entropy loss mainly depends on the corresponding probability of the network output of the correct category through Softmax. The Softmax function is as follows:  
\[p_{i}=\frac{e^{z_{i}}}{e^{z_{i}}+\sum_{j=1,j\
eq i}^{N}e^{z_{j}}} \tag{5}\]  
where \(z_{i}=W\cdot x\) is the \(i\)-d output of the last linear layer of the convolutional neural network, \(W\) is the linear layer's weight, \(x\) is the output feature from the layer before the linear layer. \(p_{i}\) is the result of \(z_{i}\) after Softmax calculation, it converts the output of the linear layer into a probabilistic representation. After observing the formula of Softmax, we found that if we want to increase the prediction probability of the image's correct category, one of the simplest methods is to multiply the value of linear layer weight by a constant value greater than 1. However, this way of improving the accuracy of the final prediction cannot enhance the feature extraction ability of the convolutional neural network, and is useless for improving classification performance. The key of the problem is to let the linear layer's input feature be more effective. In our experiment, we will display the contrast diagram. Let us take a classification problem with three categories as an example. One image sample belongs to the third category pass through the neural network and its final feature vector is [1,1,2]. Then, the probability that it belongs to the correct category calculated by Softmax is \(\mathrm{e}^{2}/\Big{(}\mathrm{e}^{1}+\mathrm{e}^{1}+\mathrm{e}^{2}\Big{)}=0.6769\). If the final linear layer weight \(W\) is simply multiplied by 2, then the final output feature of the image will become [2,2,4], and the prediction probability increases to \(\mathrm{e}^{4}/\Big{(}\mathrm{e}^{2}+\mathrm{e}^{2}+\mathrm{e}^{4}\Big{)}=0.8481\). Without improving the ability of neural networks to extract features, the prediction accuracy of the image was improved by 17.12%. This will undoubtedly affect the effectiveness of updating network weights by back propagation through the cross-entropy loss, that is because the gradient disappears, the loss function of the network lost its ability to update the parameters in the network. It will also be demonstrated in the following experiments. The updating gradient is calculated by taking the derivative of the final output vector of the network with cross-entropy loss, which can be illustrated by \(\partial L/\partial Z_{j}\):  
\[\frac{\partial L}{\partial Z_{j}}=-\sum_{i=1}^{N}y_{i}\frac{\partial\log(p_{i })}{\partial Z_{j}}=-\sum_{i=1}^{N}\frac{y_{i}}{p_{i}}\frac{\partial p_{i}}{ \partial Z_{j}} \tag{6}\]  
where \(p\) is the output of \(z\) after the softmax function, \(p=\mathrm{softmax}(z)\) the solution of this can be divided into two cases, one is \(i=j\) and another is \(i\
eq j\):\[\frac{\partial L}{\partial Z_{j}}=-\frac{y_{j}}{p_{j}}p_{j}\Big{(}1-p_{j}\Big{)}- \sum_{i\
eq j}\frac{y_{j}}{p_{j}}\Big{(}-p_{i}p_{j}\Big{)}=p_{j}-y_{j} \tag{7}\]  
In the vector form:  
\[\frac{\partial L}{\partial z}=\mathbf{p}-\mathbf{y} \tag{8}\]  
It can be seen that when the output prediction vector of the neural network is closer to the correct label, the gradient update amplitude of the back propagation will be lower and lower, resulting in the phenomenon of gradient vanishing problem, which will greatly affect the effectiveness of feature extraction. In order to maintain the ability of gradient updating in the training process of neural network, researchers also made some explorations, such as normalizing the weight of linear layer and the input feature, which also alleviated this problem to some extent, but the effect is less obvious in general data sets.