# Improving Adversarial Robustness of CNNs via Maximum Margin
## 3 Proposed Method
### At-Svm
#### 3.3.1 Overview  
As Figure 2 shows, our method is mainly inserting an auxiliary SVM classifier in the penultimate layer of the network, which is parallel to the last layer. The SVM layer contains as many neurons as the number of classes and is purposed to obtain the maximum margin decision boundary in this feature space. The SVM auxiliary classifier treats the front layer of the original network as a feature extractor and uses a separate optimizer for training. In each epoch of training, the SVM auxiliary classifier and the original network are trained successively, and a mask on which examples will be trained is produced by the SVM auxiliary classifier. The SVM auxiliary classifier is discarded during inference. The specific implementation will be introduced in the algorithm.  
Figure 2: Framework of our proposed AT-SVM.  
In CNNs, the last FC layer is linear, so we regard the front layers as a feature extractor. Using \(\phi\) to denote the extracted feature, we can train an SVM auxiliary classifier using Equation (10).  
\[\min\sum_{i=1}^{m}\sum_{j=1}^{n}\max\left(0,1-y_{j}^{(i)}\left(\mathbf{\omega}_{j}^{T }\phi^{(i)}+b_{j}^{T}\right)\right)+C^{\prime}\|\mathbf{\omega}_{j}\|_{2}^{2} \tag{10}\]  
where the first item is the hinge loss represented by \(L_{hinge}\) below. However, there is a problem in training the network with Equation (10) that the feature space \(\mathbf{\phi}\) is constantly changing. Accordingly, maximizing the margins in Equation (10) can be trivially attained by scaling up the space \(\mathbf{\phi}\). This problem can be solved by using the SVM auxiliary layers, because \(C^{\prime}\|\mathbf{\omega}_{j}\|_{2}^{2}\) only helps training the SVM auxiliary and indirectly affects \(L_{hinge}\). The training of the main network is only using \(L_{hinge}\), which constrains the feature extractor to separate input examples of different classes in the feature space. The auxiliary classifier is affected by both \(L_{hinge}\) and \(C^{\prime}\|\mathbf{\omega}_{j}\|_{2}^{2}\), and continuously obtains a better classification boundary. The change of the boundary will also affect the value of \(L_{hinge}\). The boundary and \(L_{hinge}\) will be converged under the continuous alternating training of the original network and the auxiliary classifier.  
The distance from \(\phi^{(i)}\) to the binary decision boundary of class \(j\) is:  
\[d_{j}(\phi^{(i)})=\frac{\mathbf{\omega}_{j}^{T}\phi^{(i)}+b_{j}}{\left\|\mathbf{\omega} _{j}^{T}\right\|} \tag{11}\]  
For a clean example \(\mathbf{x}^{(i)}\), we denote its adversarial example as \(\hat{\mathbf{x}}^{(i)}\). Only a large margin in SVM auxiliary cannot guarantee robustness because the extracted feature \(\hat{\phi}^{(i)}\) of \(\hat{\mathbf{x}}^{(i)}\) may have large distance with \(\phi^{(i)}\). So we will constrain the movement of the adversarial examples relative to the clean examples in the margin direction, as shown in Equation (12).  
\[d_{j}(\phi^{(i)})-d_{j}(\hat{\phi}^{(i)})=\frac{\mathbf{\omega}_{j}^{T}\left(\phi ^{(i)}-\hat{\phi}^{(i)}\right)}{\left\|\mathbf{\omega}_{j}^{T}\right\|_{2}} \tag{12}\]  
After normalization, the following objective function can be obtained, as shown in Equation (13).  
\[L_{margin}=\sum_{i}\sum_{j}\max\left(\frac{d_{j}(\phi^{(i)})-d_{j}(\hat{\phi}^ {(i)})}{d_{j}(\phi^{(i)})},0\right) \tag{13}\]  
In each epoch, we will first train the SVM auxiliary classifier using Equation (10). Then compute \(L_{margin}\), \(L_{hinge}\) and add them to the loss of the original network to train.