## 4 Experimental Results
### Visualization of the Visual Representation  
To explore the effectiveness of the proposed semi-supervised learning framework, we visualized the visual representation produced by the last layer (after the flatten layer) of the backbone using t-SNE [60]. With 40 test samples per class as input, we first extracted the deep visual representation. Next, we utilized t-SNE to reduce the dimension of visual representation from 1280D to 2D. Finally, the 2D representation was plotted on a 2D plane. All experiments were based on MsvNetLite. In semi-supervised learning, 144,000 unlabeled samples were used for self-supervised learning and only 192 labeled samples for fine-tuning. In supervised learning, only 192 labeled samples were used for training. The parameters of t-SNE were set as follows: \(perplexity=100\), \(iterations=500\), and \(initialization=pca\).  
The t-SNE visualization results are shown in Figure 9. Different colors and numbers stand for different class labels corresponding to Figure 8. For intuitive evaluation, we visualized the reduced embeddings of six situations: (a) the backbone output of MsvNetLite after fine-tuned using the SGDM optimizer in semi-supervised learning, (b) the backbone output of MsvNetLite initialized with ImageNet pre-trained weights, (c) the backbone output of MsvNetLite with random initialization, (d) the backbone output of MsvNetLite after self-supervised stage in semi-supervised learning, (e) the backbone output of MsvNetLite after supervised learning, and (f) the backbone output of MsvNetLite after fine-tuned using the Adam optimizer in semi-supervised learning.  
In Figure 9a, after semi-supervised learning, the embeddings seem to cluster, and 24 well-separated clusters can be seen. In Figure 9b, the backbone is only trained on the ImageNet and never sees the FeatureNet dataset. Although the embeddings look indistinguishable and are not clearly separated, some of them appear to cluster together. This presents an intuitive explanation for knowledge transferring. Figure 9c shows the representation from the backbone without training, in which the embeddings look chaotic and random. In Figure 9d, multiple distinct clusters can be seen after self-supervised learning, which indicates that self-supervised learning produces meaningful visual representation even without the use of labeled data. However, we can still observe that some class overlapping exists. These results demonstrate that the self-supervised learning helps network in class separation and learn discriminative visual representation even without labeled data. Figure 9e depicts the representation from the backbone trained with supervised learning. We can observe that the embeddings produced by the supervised backbone are more indistinguishable and not separated well in comparison to Figure 9a. Therefore, the proposed semi-supervised learning produces high-quality representation and better class separation, validating its effectiveness in improving recognition performance. Figure 9f presents a fail case where the Adam optimizer is used for fine-tuning. We can see that the majority of the embeddings become random and indistinguishable, which may indicate that the knowledge from self-supervised learning is lost. In Section 4.5.5, there is a detailed discussion about why the Adam optimizer is unable to fine-tune.