## 1 Introduction  
The advent of BERT [1], a pre-trained conceptualized language model, was a paradigm shift in Natural Language Processing (NLP) particularly because of the introduction of the pre-training/fine-tuning mechanism [2, 3]. That is, after pre-training in a self-supervised way on a tremendous amount of textual data, the BERT model can be rapidly fine-tuned on a certain downstream task with small labeled-data and little time, because the general linguistic patterns have already been learned during the pre-training phase [3]. One of the essential downstream tasks in NLP, for which BERT needs to be fine-tuned, is Semantic Textual Similarity (STS), which quantifies the closeness of semantic meaning of given texts that have the same meaning, but are not necessarily lexically similar. STS is indispensable for many NLP applications including information retrieval, text classification, machine translation, sentiment analysis, text summarization, market intelligence, named entity recognition, etc. For example, in information retrieval, STS is used to measure the relevancy between the retrieved document and the user query; in sentiment analysis of tweets, STS is used to estimate similarity scores between tweets; in question answering, STS is used to find correct answers that have no lexical overlap but are still semantically similar; in market intelligence, STS is used to select the best wording for questions to use when creating surveys, as the questions can be expressed in such a way that the information gathered remains valuable and relevant over time. For example, consider these two questions: \"How old are you?\" and \"What year were you born?\". Both have the same meaning, but the latter will still be useful several years from now. Although BERT has been very successful in some NLP tasks, sentence embeddings derived from BERT are susceptible to collapse without fine-tuning. That is, they are mapped in a small space and almost all pairs of sentences have a cosine similarity in the range of {0.6, 1} [4]. This problem arises from the fact that the frequency of words distorts the embedding space [4, 5]. To overcome this problem, a thread of research has thus been working on applying contrastive learning techniques under the pre-train-then-fine-tune paradigm of BERT [4, 6, 7, 8, 9, 10, 11, 12, 13, 14]. Contrastive learning pulls similar samples closer together, but pushes out dissimilar samples [15]. Basically, contrastive learning is inspired by the way humans learn about the world around them, using the principle of contrasting samples against each other. The idea behind it is similar to the \"match the correct image\" game for children, where they learn by comparing similar and dissimilar pictures. The issue here, however, is the difficulty level of the game. For example, as shown in Figure 1, we can match easy or hard pictures. The learning process begins with simple examples of a task and then gradually increases the difficulty of the task; this is called curriculum learning, and is based on the human learning strategy from easy to hard material. Based on the above descriptions, this work aims to answer the following question: Can applying curriculum learning to contrastive learning increase the performance of contrastive learning? To answer this question, we propose Curriculum Contrastive Learning by Transferring Self-taught Knowledge for Fine-Tuning BERT. For simplicity, we abbreviate the foregoing as SelfCCL. Our model learns in the meaningful order from easy to hard triplets based on the contrastive objective in a self-taught manner. That is, SelfCCL transfers self-taught knowledge to sort input triplets by difficulty. In essence, self-taught learning is a subcategory of transfer learning, which is the ability of transferring the knowledge and skills learned in a previous task to a new task [16]. SelfCCL first uses unlabeled contrastive triplets \((x_{1},x_{i}^{+},x_{i}^{-})\) for triplet mining and scoring triplets to easy, semi-hard and hard through BERT itself.