# Feature Mining: A Novel Training Strategy for Convolutional Neural Network
## 2 Related Works  
**Methods for reducing information loss in CNNs:** The pooling layer decreases the size of the activation maps to reduce the computational requirements of the CNN, which would inevitably incur information loss. Many strategies try to alleviate this problem by reforming the structure of the pooling layer. Random shifting [29] embeds random shifting in the downsampling layers during the training process to generate more robust features in the network. S3 Pool [30] replaces the regular downsampling by a more general stochastic version and can be seen as performing implicit data augmentation by introducing distortions in the feature maps. Inspired by the human visual system, detail-preservingpooling [31] magnifies spatial changes and preserves important structural detail to make the CNN focus on local spatial changes. Based on softmax normalization, SoftPool [32] preserves the descriptive activation features, while remaining computationally and memory efficient. Note that although our approach also attempts to reduce information loss and strengthen local feature learning as the above methods, we did not change the structure of the pooling layer, but reused the segmented features for efficient learning.  
**Dropout methods:** The Dropout methods aim to improve the generalization of neural networks. Dropout [15] injects noise into the feature space by randomly zeroing the activation function to avoid overfitting. SpatialDropout [16] randomly discards all the channels rather than individual activations to improve the generalization of the CNN. DropBlock [17] randomly drops some contiguous regions of a feature map for better network performance. Besides, References [33; 34; 35; 36] also proposed variants of Dropout. These Dropout methods enhance local feature learning by discarding other parts of the feature, but Feature Mining divides the feature into different parts and reuses all of them.  
**Data augmentation methods:** Data augmentation is an effective strategy for network training. Random cropping and horizontal flipping [1] are the most commonly used data augmentation techniques. By randomly removing contiguous sections of the input images, Cutout [10] improves the robustness of the network. Random erasing [11] randomly selects a rectangle region in an image and erases its pixels with random values. Input Mixup [12] creates virtual training examples by linearly interpolating two input data and the corresponding one-hot labels. Inspired by Cutout and Mixup, CutMix [37] cuts patches and pastes them among the training images. Feature Mining is complementary to the above data augmentation methods such as Mixup because it operates on the feature level without changing the data processing.  
**Self-distillation methods:** Knowledge distillation [14] is proposed for model compression. Unlike traditional knowledge distillation methods that require teacher and student models, self-distillation distills knowledge itself. Data distortion [38] transfers knowledge between different augmented versions of the same training data. Be your own teacher [18] improves the efficiency of network training by squeezing the knowledge in the deeper portion of the networks into the shallow ones. Classwise self-knowledge distillation [19] improves the generalization of the same kind of images from the perspective of intra-class distillation. Both self-distillation and Feature Mining could improve the utilization of the internal features. Experiments showed that our method outperformed existing self-distillation methods and can be superimposed on self-distillation.