# Feature Mining: A Novel Training Strategy for Convolutional Neural Network
## 4 Results and Discussion
### Image Classification  
#### 4.1.1 CIFAR Classification  
The CIFAR100 dataset [24] consists of 60,000 32 \(\times\) 32 color images of 100 classes, each with 600 images including 500 training images and 100 test images. For ResNet [4] and Wide ResNet [22], the mini-batch size was set to 128, and the models were trained for300 epochs with an initial learning rate of 0.1 decayed by a factor of 0.1 at Epochs 150 and 225. We changed the batch size to 64 when training with DenseNet [21]. As mentioned in [23], we changed the initial learning rate to 0.5 and decreased it by a factor of 0.1 at Epochs 150 and 225 when training PyramidNet [23]. **Comparison with the Dropout methods:** We adopt ResNet-56, ResNet-110, ResNet-164, DenseNet-100-12, Wide ResNet-28-10, and PyramidNet-110-270 as the baselines to evaluate Feature Mining's generalization for different layers and structures of networks. We tested Feature Mining with the last layer and the last two layers. Dropblock [17] was applied to the output of the first two groups. Dropout [15] and SpatialDropout [16] were applied to the output of the penultimate group. Table 1 shows that Feature Mining outperformed other baselines consistently. In particular, Feature Mining improved the top-1 accuracy of the cross-entropy loss from 74.98% to 78.31% (independent-samples \(t\) test, \(p\)-value = 0.000177 < 0.001) of ResNet-164 under the CIFAR100 dataset. Our method, only used in the last layer, was better than the performance of the three Dropout methods, and we considered the reason to be that Dropout and its variants strengthen the part of the neurons' learning by discarding other parts of the neurons, which may cause information loss during the training phase. Feature Mining can make the network simultaneously learn the different regions of the feature without information loss, so that the network can achieve better performance. We can also find that the network performance of Feature Mining used in the last two layers was much better than that only used in the last layer, which indicates that the local feature in the front layer is also beneficial to the network training. **Comparison with self-distillation methods:** We also compared our method with self-distillation techniques such as by your own teacher (BYOT) [18] and classwise knowledge distillation (CS-KD) [19]. BYOT improved the performance of the network by making the shallow network learn the knowledge of the last layer. Therefore, we took the output of the last layer in ResNet-110 as the teachers of all the front layers and provided the final results after ensembling for a fair comparison. Besides, CS-KD improved the generalization of the same kind of images from the perspective of intra-class distillation. Note that we only chose ResNet-110 as the baseline model due to the limitation of computing resources, but it can be seen from the results in Table 1 that the improvement of our method in a deeper network (ResNet-110) was more obvious than that in a shallow network (ResNet-164). Therefore, the performance of our method on ResNet-110 was able to prove its advantage because of its generalization. As shown in Table 2, Feature Mining showed better top-1 accuracy on ResNet-110 compared with BYOT and CS-KD. Note that BYOT added extra overhead in the training, while the time of our method was negligible. Besides, we combined Feature Mining with these two methods and improved the top-1 accuracy of ResNet-110 from 76.68% to 77.34% (independent-samples \(t\) test, \(p\)-value = 0.046 < 0.05) of BYOT and from 75.92% to 77.68%