## 6 Conclusions and Future Works  
Recently, many contrastive learning methods have been proposed to fine-tune BERT in the STS downstream task, which learns from the contrast between similar and dissimilar sentences. However, the effect of the order of difficulty level of triplets during training has not been investigated in them. In this work, our aim was to study the effect of using both contrastive learning and curriculum learning, simultaneously. For this purpose, we proposed a curriculum contrastive learning model by transferring self-taught knowledge for fine-tuning BERT, which first sorts the triplets based on curriculum order by transferring BERT self-taught knowledge, and then learns based on a contrastive objective. We developed SelfCCL-BERT and SelfCCL-SEBERT models based on BERT and SBERT architectures. We evaluated our models on standard STS and transfer learning tasks.  
Our models, SelfCCL-BERT and SelfCCL-SBERT, surpasses baseline model SBERT, SBERT-base-nli-v2, and SOTA SimCSE model in terms of average Spearman's rank correlation on STS tasks. Furthermore, our models, SelfCCL-BERT and SelfCCL-SBERT, outperforms SOTA SimCSE on transfer learning tasks. Although our models fell behind SOTA SupMPN model on the STS tasks, it still has good performance compared to other baselines and requires low computational power compared to SOTA SupMPN, according to Table 3.  
Moreover, as explained in Section 5.1, SelfCCL-SBERT requires less computational power than SelfCCL-BERT, while it achieves competitive performance compared to SelfCCL-BERT on both STS and transfer learning tasks.  
Besides, the analysis of correlation between human-annotated scores and cosine similarity of sentence embeddings generated by our models on SICK dataset shows that our SelfCCL models learn better representation space compared to the baseline models.  
In summary, the empirical results show that the use of curriculum learning together with contrastive learning leads to a relatively small improvement in the efficiency of the model for measuring semantic similarity between texts.  
The challenge that can be discussed here is the conflict between contradiction and similarity concepts. As explained in the third experiment, two sentences can be semantically contradictory but conceptually completely similar. As a future research direction, this challenge can be investigated for the STS task.  
For future studies on curriculum learning, we plan to investigate curriculum data augmentation. Curriculum data augmentation involves gradually increasing the noise in the data to generate new data. For example, the synonym replacement method involves gradually increasing the number of words that are replaced with similar words using WordNet or contextual word embeddings, e.g., BERT.  
Conceptualization, methodology, software, validation, formal analysis, investigation, resources, data curation, writing--original draft preparation, writing--review and editing, visualization, S.D.; supervision, M.F.A. All authors have read and agreed to the published version of the manuscript.  
This study was supported by the Scientific and Technological Research Council of Turkey (TUBITAK) Grant No: 120E100.  
Not applicable.  
Our pre-trained models and training data are publicly available at: [[https://github.com/SoDehghan/SelfCCL](https://github.com/SoDehghan/SelfCCL)]([https://github.com/SoDehghan/SelfCCL](https://github.com/SoDehghan/SelfCCL)) (accessed on 13 December 2022).  
The authors declare no conflict of interest.