## 2 The Proposed DR Detection System
### The Proposed Method
#### 2.2.1 Pre-Training Using Self-Supervised Learning  
The approach of using self-supervised and semi-supervised learning in this study was inspired by the work of exploring pre-training and the semi-supervised and self-training methods to utilize unlabeled data of the text corpus for improving performance on the text classification task [40]. The results of [PERSON] et al. show the obvious improvement in the performance using both pre-training and self-training [40]. In addition, these authors also found that if there was a text corpus consisting of a large-scale dataset in the domain, then it was not necessary for their model to be pre-trained on the general open-domain corpus. For example, the size of an unlabeled in-domain dataset is hundreds of times larger than its labeled dataset. However, in practical research, collecting millions of pieces of medical data is almost impossible. In this study, therefore, self-supervised learning was applied for conducting the pre-training process to learn presentations from unlabeled fundus image data, and is presented as module '1' in Figure 3. After pre-training, the learned parameters were used to initiate the encoder in module '2', which represents the target model of performing DR detection trained or fine-tuned on the labeled data using supervised learning.  
Contrastive self-supervised learning model was used in the pre-training task. The object for comparison is the similarity between two vectors or two groups of vectors on a hypersphere [41]. The similarity can be measured by the dot product of L2 normalization of the vectors, or the cosine similarity, defined as  
\[S_{(z_{i}\cdot z_{j})}=\frac{z_{i}^{\mathrm{T}}z_{j}}{\left|z_{i}\right|_{2} \left|z_{j}\right|_{2}}, \tag{1}\]  
Figure 3: The framework of the proposed DR detection system.  
The vectors \(z_{i}\) and \(z_{j}\) are projections of the representation (or embedding) projected out by the encoder of the network. Cosine similarity, which is defined based on the cosine of the angle between vectors, is a measure of similarity between vectors. An image can be transformed into two different versions through rotation, flipping, resizing, etc. These two transformed images from the same image are called a pair of positive samples, while other different data samples from the same batch are deemed as the negative samples for one positive sample. In theory, the similarity between positive samples should be higher than between negative samples [41]. The loss function is defined as  
\[L_{i}=-\log(\frac{\exp(S_{(z_{i},z_{j}^{+})}/\tau)}{\sum_{j=0}^{K}\exp(S_{(z_{ i},z_{j}^{+})}/\tau)}, \tag{2}\]  
where the above loss function is a form of the contrastive loss function called InfoNCE that uses similarity measured by dot product [41]. Here, \(\tau\) is a temperature hyperparameter for controlling the alignment and uniformity to find a balance. The alignment property guarantees that the distance between positive vectors should be close in the unit hypersphere, while the uniformity property causes the feature vectors to be distributed uniformly on the unit hypersphere. The model was optimized by minimizing this loss function.