# A Study on Small-Scale Snake Image Classification Based on Improved SimCLR
## 2 Materials and Methods
### Related Network
#### 2.3.4 Optimization of the Training Strategy  
The parameter update was performed using the AdamW optimizer, with the initial learning rate set to \(10^{-4}\) and the weight decay coefficient set to \(10^{-4}\). AdamW decouples the weight decay from the gradient update, effectively mitigating the risk of overfitting in the traditional Adam optimizer [33]. Concurrently, CosineAnnealing Learning Rate Scheduling is intended to facilitate the model's escape from local optimal solutions and promote convergence to a more optimal generalization space [34].  
These hyperparameters were selected based on prior empirical findings from the contrastive learning literature, such as SimCLR [27], which demonstrated that a learning rate of \(10^{-4}\) and a temperature of 0.07 in the contrastive loss yield robust convergence across medium-scale datasets. The temperature parameter in the contrastive loss controls the concentration level of the similarity distribution, and 0.07 has been shown to provide a stable balance between gradient magnitude and optimization dynamics. A gradient clipping threshold of 1.0 was also applied to stabilize the training and further reduce overfitting risk under small-sample conditions.  
1. Cosine annealing  
The cosine annealing algorithm is an optimization strategy that dynamically adjusts the learning rate, and its core idea is to simulate the periodic decay process of the learning rate through the cosine function, given the initial learning rate \(\eta_{\text{max}}\) and the cycle length \(T_{max}\) at the t-th iteration is given by the learning rate formula:  
\[\eta_{t}=\eta_{min}+\frac{1}{2}(\eta_{max}-\eta_{min})\bigg{(}1+cos\bigg{(} \frac{t\;modT_{max}}{T_{max}}\pi\bigg{)}\bigg{)} \tag{10}\]  
where \(\eta_{\text{min}}\) is the minimum learning rate, which decreases smoothly from \(\eta_{\text{max}}\) to \(\eta_{\text{min}}\) along the cosine curve in each cycle \(T_{max}\), followed by an immediate restart to the initial value to start a new round of optimization. The periodic restart mechanism allows the optimization process to jump out of the current local minima and explore a more optimal convergence region, mitigating the risk of the gradient descent falling into a flat loss surface.  
2. AdamW  
The AdamW optimizer is an improved version of the Adam optimizer, which mainly addresses the misuse of weight decay in adaptive learning rate methods and is more suitable for Transformer architectures. In Adam, weight decay is implementationally equivalent to L2 regularization and is handled directly by merging the gradient update formulas:  
\[\theta_{t+1}=\theta_{t}-\eta*\frac{\hat{m}_{t}}{\sqrt{\hat{\theta}_{t}}+\epsilon }-\eta*\lambda\theta_{t} \tag{11}\]where \(\eta\) is the learning rate, and \(\lambda\) is the weight decay coefficient. This coupling leads to weight decay being influenced by the adaptive learning rate, destroying its regularization nature.  
AdamW explicitly separates weight decay from parameter updating, independent of the adaptive learning rate mechanism, as follows:  
\[\theta_{t+1}=\theta_{t}-\eta*(\frac{\hat{m}_{t}}{\sqrt{\hat{\upsilon}_{t}}+ \epsilon}+\lambda\theta_{t}) \tag{12}\]  
The weight decay term is no longer scaled by the adaptive learning rate and always acts on the parameters at a fixed scale. This is more in line with the original design intent of weight decay and improves the effectiveness of regularisation.