## 5 Discussion
### Experiments on Overfitting Mechanism  
According to the derivation about late-stop and dropout, this part of the experiment mainly focuses on the impact of overfitting on feature clustering. First of all, we output the distance variance of features. For two features \(f_{i}\) and \(f_{j}\) both belong to one class \(CLASS_{k}\), the distance between them is calculated as follows:  
\[dis_{ij}^{k}=\left\|f_{i}-f_{j}\right\|_{Euclidean},\ \ \ i,j\in CLASS_{k},i\
eq j \tag{17}\]  
where \(\left\|\bullet\right\|_{Euclidean}\) is the Euclidean distance. Because we pay more attention to the degree of fluctuation of feature clustering in the CIL process, we present the mean standard deviation of this distance at each training epoch as follows:  
\[ \begin{array}{l}dis_{ave}^{e}=\frac{1}{N_{CLASS}}\times\\ \sum\limits_{k\in N_{CLASS}}\left(\frac{1}{N_{CLASS_{k}}}\sum\limits_{i,j\in N _{CLASS_{k}}}\left(\frac{1}{N_{CLASS_{k}}(N_{CLASS_{k}}-1)}\sum\limits_{i,j\in N _{CLASS_{k}}}dis_{i,j}^{k}\right)\right)\end{array}  \tag{18}\]  
where \(N_{CLASS}\) is the number of classes, and \(N_{CLASS_{k}}\) is the number of the instances belongs to \(CLASS_{k}\). Then, we obtain Figure 10, which shows the difference between _dis_ with and without dropout in different training stages. Firstly, based on the two accuracy change trends (ACC with dropout and ACC without dropout) in Figure 10, the network without dropout fits faster than the network with dropout. Then, from the two standard deviation (std) trends (cluster STD with dropout and cluster STD without dropout), it is observed that the standard deviation without  
Figure 10: Mean variance of feature clustering distance in feature space. dropout consistently remains lower than that with dropout. This observation validates that removing the dropout structure contributes to enhancing the stability of feature clustering. Furthermore, as the accuracy increases rapidly, the standard deviation also exhibits a rapid increase. Subsequently, std will gradually decrease as training progresses, indicating that that \"late-stop\" can also help to increase the stability of feature clustering. We additionally employ the T-SNE method to visualize the difference in feature distributions between the network with and without dropout at the 80 th epoch with the same hyperparameters, as shown in Figure 11.  
The scatter points in six colors represent features of six classes. Compared to (b), it is evident that in (a) the distance between features belonging to the same classes is smaller. This observation confirms that the features extracted by the network without dropout have a higher clustering level, which is expected to enhance the performance of CIL. The feature extractor in our paper utilizes the scattering feature, resulting in more stable extracted features. Consequently, the optimization of the network has a greater impact on the fully connected layers after the feature extractor. Since the magnitude of the feature vector and the classifier weights in our paper are all reset to the same value, separately, the calculation of fully connected layers can be considered as the projection of the extracted feature onto the classifier weights. Therefore, we analyze the projection of features onto the classifier weight as depicted in Figure 12.  
Figure 11: Feature distribution of different network structures. Taking class c as an example, the black arrow represents the classifier weight vector, while the blue arrows represent the feature vectors of class c. The vectors in other colors are the features of other classes. The scatter points of different colors on the black arrow represent the projections of the endpoints of different feature vectors on the weight vector. Set the direction of the black arrow as the coordinate axis, the starting point of all vectors (black point) as the origin, and the positions of the features' projection points are the corresponding outputs of the c-th classifier. Because of the projection, the multi-label classification task becomes a binary classification task.