Improving Contrastive Learning of Sentence Embeddings from AI Feedback
Year: 2023

Abstract:
Contrastive learning has become a popular approach in natural language processing, particularly for the learning of sentence embeddings. However, the discrete nature of natural language makes it difficult to ensure the quality of positive and negative sample pairs generated through data augmentation methods. Although supervised contrastive learning can produce more accurate sample pairs with human feedback labels, it still lacks fine-grained training signals. In this paper, we propose to improve Contrastive Learning of sentence embeddings from AI Feedback (CLAIF). Our method utilizes AI feedback from large pre-trained language models (LLMs) to construct sample pairs with fine-grained sample similarity scores to improve contrastive learning. Besides, we combine human feedback and AI feedback to provide better supervision signals for supervised contrastive learning of sentence embeddings. Experimental results show that our method achieves state-of-the-art performance on several semantic textual similarity (STS) and transfer learning tasks compared to other unsupervised and supervised contrastive learning methods. 1

heading: 1 Introduction
text: Learning sentence embeddings with rich semantics is very important for many natural language processing tasks, such as semantic matching and information retrieval. Recently, pre-trained language models (Devlin et al., 2019; Liu et al., 2019; Qiu et al., 2020) provide a convenient way to get sentence embeddings. However, sentence embeddings directly generated by pre-trained language models show poor performance on semantic textual similarity (STS) tasks due to the representation degeneration problem (Gao et al., 2019). Therefore,
1We will release our code and data at https://github. com/xiami2019/CLAIF
†Corresponding author.
finding ways to further improve pre-trained models to produce better sentence embeddings becomes an crucial and fundamental challenge in natural language processing.
Given the shortage of labeled data for sentence embedding learning, recent studies mainly focus on unsupervised methods, such as utilizing contrastive learning methods(Yan et al., 2021; Gao et al., 2021; Chuang et al., 2022). Contrastive learning can be classified into two categories (Khosla et al., 2020): supervised contrastive learning and unsupervised contrastive learning, depending on whether additional label information is utilized to construct positive and negative sample pairs. However, the quality of positive and negative sample pairs in unsupervised contrastive learning can be difficult to ensure. Recent studies also show that data augmentation strategies in unsupervised contrastive learning may introduce some bias like length information (Wu et al., 2022) and improper negatives (Zhou et al., 2022a). While supervised contrastive learning methods can produce more accurate sample pairs by utilizing label information, such as using supervised datasets from natural language inference (Gao et al., 2021), it can only provide coarse-grained labels and lack fine-grained supervision signals. We aruge that these limitations of current contrastive learning methods restrict further performance enhancement of sentence embeddings.
With the emergence of large pre-trained language models (LLMs) (Brown et al., 2020; Sun et al., 2021; Ouyang et al., 2022; Zhang et al., 2022), researchers hope powerful LLMs can help human train other AI models (Bai et al., 2022). One way is to use LLMs to generate datasets using for zero-shot learning (Schick and Schütze, 2021; Ye et al., 2022; Meng et al., 2022). These methods all use predefined labels and task descriptions to generate training inputs, instead of utilizing AI feedback as supervision signals. Therefore, these
11122
method are not suitable for tasks whose labels are continuous values and may lead to lack of diversity in training samples. Inspired by these studies, we hope to exploit the capability of LLMs to address shortcomings in contrastive learning of sentence embeddings.
We propose to improve Contrastive Learning of sentence embeddings from AI Feedback (CLAIF). Specifically, we design a two-step sample pair generation method to produce high quality sentence pairs and fine-grained semantic similarity scores using AI feedback from GPT-3, as shown in Figure 1. In the first step, we mask some words in a sentence with different mask rates and then use GPT-3 to generate new sentences based on the remaining information in the masked sentence. Then we combine the generated sentences and the original sentence to construct sentence pairs. In this way, we can use the mask rate to control the amount of sharing information between two sentences in a pair, which will produce sentence pairs with different semantic similarities. In the second step, we utilize GPT-3 to generate semantic similarity scores for sentence pairs. These scores are the AI feedback on sample similarity. Since the semantic change caused by reconstructing a masked sentence is difficult to measure, we leverage the
linguistic knowledge of LLMs to generate the semantic similarity score. The diversity of AI feedback similarity scores ensured by the sentence pair generation process in the first step. At last we use our generated sample pairs and similarity scores to train the model for sentence embeddings.
In addition to using AI feedback alone, we also combine human feedback and AI feedback by introducing AI feedback into supervised contrastive learning of sentence embeddings which needs human feedback labels to generate positive sample pairs. We use the AI feedback similarity score for the positive sample pair as a soft label to replace the one-hot label in InfoNCE loss (He et al., 2020). We term our loss Soft InfoNCE. This process can be referred to as contrastive learning of sentence embeddings from human and AI feedback (CLHAIF).
We conduct extensive experiments to show the effectiveness of our method. Sentence embeddings learned with CLAIF and CLHAIF achieve state-ofthe-art performance on standard semantic textual similarity tasks and outperform strong baselines on transfer learning tasks. We also find that CLAIF results in significant improvements to the crossencoder architecture for the sentence-pair modeling task.
Our main contributions are as follows:
• We propose to improve contrastive learning of sentence embeddings from AI feedback (CLAIF) and achieve state-of-the-art performance on several semantic textual similarity tasks and transfer learning tasks.
• We construct a semantic textual similarity dataset with high quality sentence pairs and fine-grained AI feedback similarity scores using large pre-trained language models.
• We propose a method to incorporate human feedback and AI feedback to provide better supervision for contrastive learning of sentence embeddings.
• Experimental results show the scalability of CLAIF, which is cheaper and more efficient than collecting data from human feedback.

heading: 2 Understanding Contrastive Learning from Different Feedback
text: In this section, we categorize contrastive learning methods into four categories according to their feedback sources. We summarize the details of contrastive learning from different feedback in Table 1, including their feedback types, sample pairs construction methods and representative loss functions.

heading: 2.1 Contrastive Learning from Zero Feedback
text: Traditional contrastive learning is used for selfsupervised representation learning (Hadsell et al.,
2006; He et al., 2020). These methods construct positive and negative sample pairs using data augmentation strategies without any human feedback. For example, in natural language processing, Gao et al. (2021) construct positive sample pairs by doing the dropout operation twice for the same sentence and negative pairs by combining with another sentences. We refer to these methods as Contrastive Learning from Zero Feedback (CLZF). The most common loss function for CLZF is InfoNCE (van den Oord et al., 2018). Chen et al. (2020) propose NT-Xent loss, which can be seen as a variant of InfoNCE. However, due to the discrete nature of natural language, it is hard to find effective and unbiased data augmentation strategies to construct high quality sample pairs.

heading: 2.2 Contrastive Learning from Human Feedback
text: Recently, Khosla et al. (2020) propose to use label information to construct positive sample pairs. In sentence embeddings, Gao et al. (2021) use premise-hypothesis pairs with entailment relationship from natural language inference (NLI) datasets as positive sample pairs and still use InfoNCE for training. Since these methods leverage label information from human, we refer to them as Contrastive Learning from Human Feedback (CLHF). With the help of label information, some new losses can be used in CLHF, like SupCon (Khosla et al., 2020) and KNN-Contrastive (Zhou et al., 2022b). Although CLHF can construct more accurate sam-
ple pairs, it still lacks fine-grained supervision signals. For example, in InfoNCE, all positive pairs have a label of 1. But there are also differences in the similarity between different positive sample pairs.

heading: 2.3 Contrastive Learning from AI Feedback
text: Measuring the similarity of sample pairs in contrastive learning is a laborious task. However, thanks to emergence of LLMs, we can use LLMs to measure the similarity of sample pairs and use the AI feedback as our training signals. We refer to this approach as Contrastive Learning from AI Feedback (CLAIF). CLAIF does not need to explicitly construct positive and negative sample pairs because each sample pair has a fine-grained label. We use mean squared error (MSE) loss for the training of CLAIF in this work.

heading: 2.4 Contrastive Learning from Human and
text: AI Feedback
Besides contrastive learning from AI feedback, we propose to combine human and AI feedback to produce better supervision signals when they are both available. We call this category contrastive learning from human and AI feedback (CLHAIF) and we propose a soft InfoNCE loss for the training of CLHAIF. We hope to use fine-grained AI feedback to refine the coarse-grained signals in current CLHF methods.

heading: 3 Methodology
text: In this section, we first introduce our method to generate sample pairs and the training process of CLAIF. In order to obtain high quality sentence pairs with diverse and fine-grained similarity scores, we propose a two-step sample pair generation method: Sentence Pair Generation and Semantic Similarity Labeling. The generation process is shown in Figure 1. We use these sample pairs to train language models like BERT and RoBERTa. Then we introduce CLHAIF, which combines human and AI feedback in contrastive learning of sentences embeddings.

heading: 3.1 Sentence Pair Generation
text: We use unpaired sentences from the training set of STS Benchmark (Cer et al., 2017) as our original sentences. As shown in Figure 1, we first mask some words of the original sentence "a man is playing a flute." with different mask rates using the
<mask> token, in order to delete some information in the original sentence. The more words that are masked, the less information is left. We use the depth of color to indicate the degree of information sharing between two sentences in Figure 1. Then we write a task description prompt to steer GPT-3 to generate new sentences based on masked sentences. We provide our task descriptions in Appendix B. To increase the diversity of generated sentences, we merge adjacent <mask> tokens in 50% of masked sentences into one <mask> token. Then we combine the original sentence with each generated sentence to construct sentence pairs.

heading: 3.2 Semantic Similarity Labeling
text: In this step, we label the semantic similarity score for each sentence pair using AI feedback from GPT3. The similarity score ranges from 0 to 1, where a score of 1 means that the semantic of the two sentences are exactly the same, and a score of 0 means that the semantic of the two sentences are completely different. We write a task description prompt to steer GPT-3 to generate a similarity score between 0 and 1 for each sample pair generated in step 1. The first step ensures the diversity of semantic similarity scores. As illustrated in Figure 2, the generated scores are diverse and distributed in the value range from 0 to 1.

heading: 3.3 Training on Generated Pairs
text: With the generated sample pairs, we train a language model as the sentence encoder to get better sentence embeddings. Given diverse sentence pairs which have fine-grained similarity scores, we do not need to explicitly construct positive and negative sample pairs. Therefore, we directly use the mean squared error (MSE) loss to fit the cosine similarity of each sentence pair to its AI feedback
similarity score:
L = 1 N
N∑
i=1
[ cos ( hi,h ′ i ) − yi ]2 (1)
where N is the batch size, hi and h′i are two sentence embeddings of the i-th sentence pair (xi, x′i) encoded by the model, yi is the corresponding similarity score and cos means the calculation of cosine similarity. During inference, we use the cosine similarity of two sentence embeddings as their semantic similarity score.

heading: 3.4 Combining Human Feedback and AI Feedback
text: In this section, we mainly study the cooperation of human and AI models to provide better training signals for contrastive learning, which we called CLHAIF. Reimers and Gurevych (2019) use supervised NLI datasets to learn sentence embeddings. Gao et al. (2021) construct positive and hard negative sample pairs for contrastive learning leveraging label information of NLI datasets, achieving significant improvements. However, as we mentioned in Section 2.2, CLHF does not distinguish between different positive sample pairs and assigns label of 1 for all positive pairs. In this way, all positive sample pairs are pulled together with the same extent in contrastive learning, ignoring differences in similarity between different positive pairs. Therefore, we use AI feedback to refine these coarse-grained supervision signals.
At first, we use the semantic similarity labeling step in Section 3.2 to generate AI feedback similarity scores for sentence pairs constructed from supervised NLI datasets: SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018). Following Gao et al. (2021), we construct sample pairs using the label information. For the i-th sample of the NLI dataset, we can obtain two sentence pairs (xi, x + i ) and (xi, x − i ), where xi is the premise, x + i and x−i are entailment and contradiction hypothesis. (xi, x+i ) is the positive pair and (xi, x − i ) is the hard negative pair.
In order to incorporate AI feedback, we propose soft InfoNCE loss by replacing the one-hot label
with the AI feedback score as the soft label:
L = − 1 N
N∑
i=1
li (2)
li = yi log ecos(hi,h
+ i )/τ
∑N j=1 ( ecos(hi,h + j )/τ + ecos(hi,h − j )/τ )
where N is the batch size, hi, h+i and h − i are sentence embeddings of xi, x+i and x − i , yi is the AI feedback similarity score for the positive pair (xi, x + i ) and τ is the temperature parameter.

heading: 4 Experiments

heading: 4.1 Evaluation Datasets
text: We conduct extensive experiments on seven semantic textual similarity (STS) tasks and seven transfer learning tasks. The STS tasks include STS 2012-2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS Benchmark (Cer et al., 2017) and SICK-Relatedness (Marelli et al., 2014). The transfer learning tasks include MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000) and MRPC (Dolan and Brockett, 2005).
Following Gao et al. (2021), for STS tasks, we calculate the Spearman’s correlation between the cosine similarity of sentence embeddings and the golden similarity scores from STS datasets. For transfer learning tasks, we train a logistic regression classifier based on fixed sentence embeddings and follow the default settings of SentEval (Conneau and Kiela, 2018). We use the same evaluation script as Gao et al. (2021) to calculate metrics.

heading: 4.2 Baselines
text: We compare our method with some strong baselines among three types of sentence embedding methods: Post-processing methods: These methods adopt some post-processing operations to enhance sentence embeddings which do not need to further train the backbone model. We use BERT-whitening (Su et al., 2021), BERT-flow (Li et al., 2020) and prompt based BERT (Jiang et al., 2022) as baselines. Training methods: These methods use additional data to further train the backbone model for better sentence embeddings. We use SBERT (Reimers and Gurevych, 2019), ConSERT (Yan et al., 2021),
SimCSE (Gao et al., 2021), DiffCSE (Chuang et al., 2022) and PromptBERT (Jiang et al., 2022) as baselines. Dataset-generation based methods: Some studies generate datasets from LLMs for sentence embedding learning. We use Dino (Schick and Schütze, 2021) as our baseline. Dino generates sentence pairs based on three discrete similarity labels using GPT2-XL. For a fair comparison, we re-implement Dino using GPT-3 in our experiments.

heading: 4.3 Implementation Details
text: Choice of large pre-trained language models: In our experiments, we get all AI feedback from textdavinci-003, which is the latest version of GPT-3. We access text-davinci-003 through the OpenAI API. Sample pair generation: We use nine mask rates for each original sentence in sentence pair generation: 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8. For CLAIF, we use unpaired sentences from the training set of STS-B as original sentences to construct sentence pairs from scratch and randomly sample two other sentences for each original sentence to construct two sentence pairs with a similarity score of 0. For CLHAIF, following previous studies (Gao et al., 2021; Jiang et al., 2022), we use the SNLI and MNLI datasets to construct sentence pairs and add a AI feedback similarity score for each sentence pair. We only use the AI feedback scores for positive pairs in our experiments of CLHAIF. Besides, to demonstrate the scalability of CLAIF, we use sentence pairs constructed from STS-B and from NLI datasets for the training of CLAIF, which we called CLAIFscaled. We list statistics of some datasets used for different methods in Table 2. Training details: We use the base version of the pre-trained language model BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) as our backbone models. We use the development set of STS-B as our validation set. In CLAIF, we use the mean
pooling strategy to get sentence embeddings for BERT and RoBERTa. For CLHAIF, we take the same pooling strategy as the corresponding baseline. Other implementation details are in Appendix A.

heading: 4.4 Main Results
text: Semantic Textual Similarity We compare CLAIF with methods which do not use additional labeled datasets for training, including CLZF methods and dataset generation methods. The results of CLAIF on STS tasks are shown in Table 4. We observe that CLAIF achieves the best performance on the four datasets STS15, STS16, STS-B, SICK-R and get the highest averaged Spearman’s correlation on seven STS datasets. And in the comparison with dataset generation methods, CLAIF outperforms Dino by 3.37 and 2.75 points on BERT and RoBERTa. Therefore, we believe that CLAIF is more effective for the learning of sentence embeddings than CLZF methods.
We implement CLHAIF by incorporating AI feedback into supervised SimCSE and supervised PromptBERT/PromptRoBERTa. We compare CLHAIF with other methods that use additional labeled datasets for training. As shown in Table 5, incorporating AI feedback improves results of
CLHF methods like supervised SimCSE on six STS datasets except STS12.
Transfer Tasks In addition to STS tasks, we also evaluate several transfer learning tasks from SentEval. Experimental results show that sentence embeddings learned with CLAIF and CLHAIF also achieve better or comparable performance compared to baselines. We present the average results for seven transfer tasks in Table 3 and detailed results in Appendix C.

heading: 4.5 Scalability of CLAIF
text: In this section we discuss the scalability of CLAIF. The results of CLAIFscaled in Table 5 show that using more data to scale CLAIF can bring significant improvements. CLAIFscaled greatly outputforms CLAIF by 2.74 points on BERT-base (79.63 → 82.37 ) and even outputforms or performs on par with CLHF and CLHAIF methods. We believe that using more data can further improve the performance of CLAIF. Since collecting data from AI
feedback is more cheaper than from human feedback, we argue that CLAIF has great potential in practical applications.

heading: 4.6 Sentence-Pair Modeling
text: In this section, we evaluate CLAIF on the sentencepair modeling task. Cross-encoders usually outperform bi-encoders in information retrieval. However, we observe in Liu et al. (2022) that the crossencoder does not show its superior on sentence-pair modeling. We contribute this to the lack of finegrained training signals. We train a cross-encoder with CLAIF. Experimental results in Table 11 show that, with the help of AI feedback, CLAIFcross brings significant improvements for cross-encoders on the sentence-pair modeling task compared to the previous model Trans-Encoder (Liu et al., 2022). More training details are in Appendix D.

heading: 4.7 Human Evaluation
text: In this section, we conduct human evaluation to measure the quality of generated sentences and similarity scores. We measure whether the generated sentences are fluent and whether the similarity scores are consistent with the real semantic similarities. To help human judge the consistency, we generate a natural language explanation for each generated similarity score using GPT-3. We invite 4 experts to participate in our human evaluation. Then we random pick 100 samples from the dataset used in CLAIF and assign 25 samples to each expert. In the evaluation, 92 percent of generated sentences are considered fluent and 90 percent of generated scores are considered consistent by the expert, which means our method can generate high quality sentence pairs and similarity scores.

heading: 5 Related Work
text: Recent studies about sentence embeddings mainly focuse on using additional data to further train pre-trained language models. Yan et al. (2021) and Gao et al. (2021) propose different data aug-
mentation strategies for contrastive learning and achieve significant improvements using unlabeled data. Chuang et al. (2022) use equivariant contrastive learning for learning better representations. Zhou et al. (2022a) and Wu et al. (2022) address the bias caused by construction processes of negative and positive samples. Jiang et al. (2022) use different prompt templates to produce positive pairs for contrastive learning. Opitz and Frank (2022) use various semantic sentence features to construct fine-grained labels for sentence embedding training.
Impressed by the powerful capabilities of LLMs (Brown et al., 2020; Ouyang et al., 2022), researchers pay more attention to using AI feedback from LLMs for zero-shot and few-shot learning. Li et al. (2023); Li and Qiu (2023) use AI feedback from language models to enhance In-context Learning and Chain-of-Thoughts. Ye et al. (2022) and Meng et al. (2022) generate datasets by taking labels and prompts as the input of LLMs and then let LLMs generate training samples. Schick and Schütze (2021) design a dataset generation method for STS tasks. They construct three natural language instructions based on three discrete similarity scores and then use these instructions to steer LLMs to construct sentence pairs. However, it is hard to use natural language to describe various similarity scores, since the similarity score is a continuous variable with values ranging from 0 to 1.

heading: 6 Conclusion
text: In this paper, we first formalize four types of contrastive learning: contrastive learning from zero feedback (CLZF), contrastive learning from human feedback (CLHF), contrastive learning from AI feedback (CLAIF) and contrastive learning from human and AI feedback (CLHAIF). Then we improve contrastive learning of sentence embeddings from AI feedback and combine human feedback with AI feedback to produce better supervision
signals. Experimental results show that CLAIF and CLHAIF can bring substantial improvements for sentence embedding learning. We hope that learning from AI feedback can shed new lights for representation learning and contrastive learning.
Limitations
To inspire future work, we conclude some limitations of our work as follows:
• While our method achieves promising performance on sentence embedding related tasks like STS, the performance on other natural language processing tasks are still need to investigate.
• The AI feedback in our experiments comes from GPT-3, which requires a fee to use.
• We do not explore the effect of different task description prompts on the quality of generated sample pairs, which may influence the performance of CLAIF.
• In CLHAIF, we only use the AI feedback for positive sample pairs. How to utilize AI feedback for negative sample pairs remains to be studied.

heading: Acknowledgement
text: We would like to thank all anonymous reviewers for their valuable advice. This work was supported by the National Natural Science Foundation of China (No. 62236004 and No. 62022027).

heading: B Task Descriptions
text: We use three task description prompts in our experiments. For sentence pair generation in Section 3.1, our two prompts are: "Replace all <mask> tokens in ’<maskedsentence>’ to make a new sentence. The new sentence is:" and "Write two sentences that mean the same thing. Sentence 1: ’<sentence1>’ Sentence 2:". For semantic similarity labeling in Section 3.2, our prompt is: "The similarity score for two sentences is in the range from 0.0 to 1.0, 0.0 means completely different and 1.0 means almost the same. Now given two sentences ’<sentence1>’ and ’<sentence2>’, please give a similarity score for these two sentences: The similarity score for these two sentences is".

heading: C Transfer Learning Tasks
text: We list the detailed performance comparison of CLAIF and CLHAIF in Table 7 and Table 8. Experimental results show that CLAIF achieves the best performance on RoBERTa-base and comparable performance on BERT-base. CLHAIF also achieves better results compared to the baselines. Using more data to scale CLAIF also brings performance improvements on transfer learning tasks as shown in Tabel 8.

heading: D Sentence-Pair Modeling
text: In sentence-pair modeling task, cross-encoders can be used to directly encode the sequence of two sentences and then predict a similarity score. Previous studies (Thakur et al., 2021; Liu et al., 2022; Lu et al., 2022) show that cross-encoders usually outperform bi-encoders. We find that CLAIF can significantly improve the performance of crossencoders on sentence-pair modeling task, with the help of fine-grained AI feedback scores.
We use the binary cross-entropy (BCE) loss to train cross-encoders initialized from BERT and RoBERTa:
L = − 1 N
N∑
i=1
li (3)
li = yi log σ(ŷi) + (1− yi) log (1− σ(ŷi))
where N is the batch size, ŷi is the predicted score of the i-th sentence pair, yi is the AI feedback similarity score and σ is the sigmoid function.

heading: E Cost for Data Generation
text: According to our billings, we spent about $100 to generate data for CLAIF and about $720 for the scaled dataset.

heading: F Generated Examples
text: We present some generated sample pairs used in CLAIF in Table 9 and some generated similarity
scores for sample pairs constructed from NLI in Table 10.

heading: G Comparison with Text-Ada-Embedding-002
text: Recently, OpenAI has released a powerful embedding model named text-ada-embedding-002, we compare the performance of it on STS tasks with CLAIF here. The results show that CLAIF-scaled achieves better performance on STS tasks than textada-embedding-002.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations.
7 A2. Did you discuss any potential risks of your work? Our work is about representation learning and contrastive learning, which are general methods and do not have potential risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract, 1 Introduction section.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Implementation Details section in Appendix.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Implementation Details section in Appendix.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Because we only use the public datasets and open source code in this work.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Our use of these public datasets and open source code is exactly what it was intended to be.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We use publicly available datasets that are commonly used by researchers. And our work mainly focuses on representation learning.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? We use publicly available datasets that are commonly used by researchers and we cite the paper of the open code and datasets we used, where the detailed documentation can be found.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4 Experiments section.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments?

heading: 4 Experiments section.
text: 7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We use well-known language models BERT-base and RoBERTa-base.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4 Experiments Section and Implementation Details section in Appendix.
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We fix the random seed in all our experiments.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4 Experiments Section and Implementation Details section in Appendix.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Limitations
To inspire future work, we conclude some limitations of our work as follows:
• While our method achieves promising performance on sentence embedding related tasks like STS, the performance on other natural language processing tasks are still need to investigate. • The AI feedback in our experiments comes from GPT-3, which requires a fee to use. • We do not explore the effect of different task description prompts on the quality of generated sample pairs, which may influence the performance of CLAIF. • In CLHAIF, we only use the AI feedback for positive sample pairs. How to utilize AI feedback for negative sample pairs remains to be studied