## 3 Proposed Method
### SSL for Weight Initialization  
Recently, SSL has received considerable attention and has enabled much progress in the field of natural images. Inspired by these studies, we use BYOL for the initialization of the weights. BYOL is a new algorithm for the SSL of image representations [46]. Its target is to learn a good representation \(y_{\theta}\) that can then be used for other downstream tasks. This algorithm is composed of two networks: an online network and a target network. The two networks interact with and learn from each other. The online network is defined by a set of weights \(\theta\) and is composed of three stages: an encoder \(y_{\theta}\), a projector \(g_{\theta}\), and a predictor \(q_{\theta}\). The target network is similarly defined by a set of weights \(\xi\) but is composed of only two stages: an encoder \(y_{\xi}\) and a projector \(g_{\xi}\). As shown in Figure 4, the target network provides the regression targets to train the online network, and its parameters \(\xi\) are exponential moving averages of the online parameters \(\theta\). It achieves this goal by minimizing the L2 loss between the two representations \(\overline{q_{\theta}}(z_{\theta})\) and \(\overline{z}_{\xi}^{\prime}\). During backpropagation, the target network is subjected to a stop-gradient operation. More specifically, after each training step, the following update is performed:  
\[\xi\gets m\xi+(1-m)\theta \tag{1}\]  
where \(m\) is a target decay rate such that \(m\in[0,1]\), \(\theta\) represents online parameters, and \(\xi\) are exponential moving averages of the online parameters \(\theta\). Let image \(x\) be sampled uniformly from a set of images \(D\), and let \(\mathcal{T}\) and \(\mathcal{T}^{\prime}\) be two distributions of image augmentations. First, by applying image augmentations \(t\sim\mathcal{T}\) and \(t^{\prime}\sim\mathcal{T}^{\prime}\), we can obtain two augmented views \(v\) and \(v^{\prime}\), respectively. Next, the online network outputs a representation \(y_{\theta}\) and a projection \(z_{\theta}\) from the first augmented view \(v\), and the target network outputs the target representation \(y^{\prime}_{\xi}\) and the target projection \(z^{\prime}_{\xi}\) from the second augmented view \(v^{\prime}\). Then, we can obtain two representations \(q_{\theta}(z_{\theta})\) and \(z^{\prime}_{\xi}\). Since we need to compute the distance between these two representations, L2 normalization is applied to both \(q_{\theta}(z_{\theta})\) and \(z^{\prime}_{\epsilon}\). After L2 normalization, we obtain the two corresponding representations \(\overline{q_{\theta}}(z_{\theta})\) and \(\overline{z}_{\xi}^{\prime}\). The goal of our network is to push the prediction of the online network, \(\overline{q_{\theta}}(z_{\theta})\), closer to the target network's projection, \(\overline{z}_{\xi}^{\prime}\). Thus, the loss function can be defined as follows:  
\[L_{\theta,\xi}\triangleq\|\overline{q_{\theta}}(z_{\theta})-\overline{z}_{\xi }^{\prime}\|_{2}^{2}=2-2\cdot\frac{\langle q_{\theta}(z_{\theta}),z^{\prime}_ {\xi}\rangle}{\|q_{\theta}(z_{\theta})\|_{2}\cdot\|z^{\prime}_{\xi}\|_{2}} \tag{2}\]  
Figure 4: Architecture of the SSL algorithm BYOL. BYOL attempts to construct image representations that are invariant with respect to image augmentation. It encourages the representations of an image \(x\) and its transformed counterparts \(v\) and \(v^{\prime}\) to be similar. where \(\theta\) and \(\xi\) denote the parameters of the online and target networks, respectively, \(z_{\theta}\) represents the input data, and \(L_{\theta,\xi}\) denotes the loss function. During the training phase, we can also feed \(v^{\prime}\) into the online network and \(v\) into the target network to make fuller use of the data, and this loss can be symmetrized, yielding \(\bar{L}_{\theta,\xi}\). In each step of training, we perform stochastic optimization to minimize \(L_{\theta,\xi}^{Total}=L_{\theta,\xi}+\bar{L}_{\theta,\xi}\) with respect to the online network \(\theta\) only.