Improving Sentence Embeddings with Automatic Generation of Training Data Using Few-shot Examples (2024)

Abstract: Decoder-based large language models (LLMs) have shown high performance on many tasks in natural language processing. This is also true for sentence embedding learning, where a decoder-based model, PromptEOL, has achieved the best performance on semantic textual similarity (STS) tasks. However, PromptEOL requires a manually annotated natural language inference (NLI) dataset for finetuning. We aim to improve sentence embeddings without using large manually annotated datasets by automatically generating an NLI dataset with an LLM and using it for finetuning of PromptEOL. To achieve this, we explore methods of data generation suitable for sentence embedding learning in this study. Specifically, we will focus on automatic dataset generation through few-shot learning and explore the appropriate methods to leverage fewshot examples. Experimental results on the STS tasks demonstrate that our approach outperforms existing models in settings without large manually annotated datasets.

heading: 1 Introduction
text: Sentence embeddings are widely studied as they can be used for many tasks such as text search, entailment recognition, and information extraction (Reimers and Gurevych, 2019; Tsukagoshi et al., 2021; Gao et al., 2021; Jiang et al., 2022; Raffel et al., 2022). Among these, methods based on decoder-based large language models (LLMs) have shown high performance in recent years. For example, SGPT (Muennighoff, 2022), which uses decoder-based LLMs to generate embeddings, and PromptEOL (Jiang et al., 2023), which generates sentence embeddings using a prompt-based method focusing on a single word, have been proposed. PromptEOL achieves the highest performance in STS in a setting using manually annotated data. However, when not using manually annotated NLI datasets, its performance is much lower.
Since the advent of high-performance decoderbased LLMs like GPT-4,1 many efforts have been made to use data generated by decoder-based LLMs as a substitute for training data in various tasks, and their effectiveness has been reported (Meng et al., 2022; Ye et al., 2022a,b). Similarly, for sentence embedding learning, there are approaches such as GenSE (Chen et al., 2022), which automatically generates datasets using LLMs to augment sentence embedding datasets, and STSDino (Schick and Schütze, 2021), which is an automatically generated dataset for training sentence embedding models using LLMs. However, there has not been sufficient investigation on how to generate data using LLMs for sentence embedding learning. It is known that when generating datasets automatically via few-shot learning, the generated datasets are heavily dependent on the few-shot examples (Zhao et al., 2021), and if all the data is generated by using the same few-shot examples, the diversity of the generated datasets may be limited.
In this study, we explore how few-shot examples should be leveraged to automatically generate training data to obtain better sentence embeddings in a framework where NLI datasets generated by an LLM are used for fine-tuning of PromptEOL. Specifically, we examine how the quality of the final sentence embeddings varies when the number of few-shot examples used to generate training data is varied or when multiple sets of few-shot examples are used, and we reveal the optimal way to leverage few-shot examples. Our contributions are two-fold. First, we explored the optimal ways to leverage few-shot examples when using LLMs to generate NLI datasets for sentence embedding learning. Second, we achieved the highest score in the STS tasks in a setting without large manually annotated datasets.
1https://openai.com/index/gpt-4-research/
519

heading: 2 Related Work
text: This section introduces PromptRoBERTa (Jiang et al., 2022) and PrompEOL (Jiang et al., 2023), which successfully generate high-performance sentence embeddings by devising prompts.
PromptRoBERTa PromptRoBERTa introduces a new contrastive learning method to improve sentence embedding performance of RoBERTa. Specifically, it takes a sentence like “I have a dog.” as input and transforms it using templates as follows: “This sentence: "I have a dog." means [MASK].” and “The sentence: "I have a dog." means [MASK].” . By using the embeddings of the “[MASK]”, it can represent the same sentence from diverse perspectives using different templates, resulting in reasonable positive pairs of sentence embeddings. By learning to bring these positive pairs of sentence embeddings closer together, PromptRoBERTa significantly reduces the performance gap between supervised and unsupervised settings, achieving better sentence embedding performance compared to traditional methods.
PromptEOL PromptEOL introduces a constraint called the “one-word limitation” and inputs the target sentence into LLMs along with a template. For example, to obtain the embedding of the sentence “I have a dog.”, it inputs the prompt “This sentence: "I have a dog." means in one word: "” into a decoder-based LLM. The hidden vector after “in one word: "” is then used as the sentence embedding. Since the decoder-based LLM is pretrained on the next-token prediction task, it can obtain a sentence embedding that captures the meaning of the whole sentence by using the prompt to predict a word that paraphrases the entire sentence. Although PromptEOL demonstrates relatively high performance in an unsupervised setting, it can produce higher quality embeddings through supervised learning. PromptEOL achieved the best performance in the STS tasks by fine-tuning the LLM via contrastive learning on NLI datasets similar to supervised SimCSE (Gao et al., 2021).

heading: 3 Automatic NLI Dataset Generation
text: In this study, we explore how to automatically construct datasets for sentence embedding learning using LLMs. In this section, we explain the process of generating NLI datasets with LLMs.

heading: 3.1 Existing NLI Datasets
text: NLI datasets are widely used in various sentence embedding models, including SimCSE (Gao et al., 2021) and PromptEOL (Jiang et al., 2023). They contain sentence pairs comprising a premise and a hypothesis, which is labeled with either “entailment,” “neutral,” or “contradiction.” The prominent NLI datasets include the Stanford NLI (SNLI) corpus (Bowman et al., 2015), the Multi-Genre NLI (MNLI) corpus (Williams et al., 2018), and the Cross-Lingual NLI (XNLI) corpus (Conneau et al., 2018), which contain approximately 579,000, 433,000, and 112,500 sentence pairs, respectively. Following Jiang et al. (2023), we use a dataset that combines the SNLI and MNLI corpora, and refer to it as the manual NLI dataset.

heading: 3.2 Automatic Generation Procedure
text: To automatically build NLI datasets, we generate hypothesis sentences from premise sentences. Specifically, we replace [premise] in each of the following prompts with a premise sentence and then feed the prompt to the LLM.
Prompt for entailment Write one sentence that is logically entailed by
[premise] in the form of a statement beginning
with "Answer: ". Answer: "
Prompt for contradiction Write one sentence that logically contradicts
[premise] in the form of a statement beginning
with "Answer: ". Answer: "
Next, we take the tokens generated between “Answer:"” and the next “"” as the generated hypothesis sentence.
We further improve the quality of the generated hypothesis sentences by applying few-shot learning (Brown et al., 2020). Specifically, we extract a few sentence pairs from the manual NLI dataset and add them as few-shot examples. The number of examples is around 20 at most, which is a feasible amount even if created manually from scratch.

heading: 4 Experiments
text: We first evaluate automatically generated NLI datasets using NLI classifiers. Next, we evaluate sentence embedding models fine-tuned with automatically generated NLI datasets. We explore how to use few-shot examples specifically for sentence
embedding learning. After conducting these experiments, we compare the best-performing method from our exploration with existing methods.

heading: 4.1 Evaluation of NLI Dataset
text: We evaluated the quality of the automatically generated NLI datasets using an NLI classifier. This allows us to assess the quality of NLI datasets generated by LLMs automatically.
Generation Method In our method, we generate hypotheses according to premises as input. Therefore, for the source premise sentences, we randomly extracted one million sentences from Wikipedia, following the unsupervised fine-tuning dataset of SimCSE (Gao et al., 2021). To reduce potential biases from the difference between the manual NLI datasets and sentences from Wikipedia, we used sentences with token counts between 4 and 32 to approximate the distribution of the manual NLI dataset. The frequency distribution of the token count is shown in Appendix A. We used LLaMA2-7B-Chat (Touvron et al., 2023) as the LLM.
Evaluation Method We used DeBERTa (He et al., 2021) trained on the MNLI corpus.2 For each sentence pair in the generated dataset, we performed a three-way classification of entailment, neutral, or contradiction. We then calculated the agreement ratio between the classification result and the assigned labels. For the manual NLI dataset and a zero-shot generated dataset, we randomly selected 3,000 sentence pairs for both entailment and contradiction, totaling 6,000 pairs, and calculated the agreement ratios for these pairs. For the fewshot generated datasets, to mitigate randomness due to the few-shot examples, we first created 10 sets of different examples for both entailment and contradiction. Then, each set was given 1,000 different premise sentences to create pairs, resulting in 20,000 pairs for evaluation.
2https://huggingface.co/microsoft/ deberta-v2-xxlarge-mnli
Experimental Results Table 1 lists the agreement ratios for each dataset. The ratio improved as the number of few-shot examples increased. In the 5-shot setting, the ratio of contradiction is comparable to that of the manual NLI dataset, and in the 20-shot setting, the ratio for both entailment and contradiction reached levels comparable to those of the manual NLI dataset. These results suggest that the automatically generated NLI datasets with 5-shot or 20-shot learning were reasonably high quality. We provide examples of datasets obtained with 0-shot and 20-shot learning in Appendix B.

heading: 4.2 Explore How to Use Few-shot Examples
text: We evaluated sentence embedding models finetuned with the automatically generated NLI datasets using the STS tasks.3 The STS task is to evaluate whether a model could correctly estimate the semantic similarity of sentence pairs. Specifically, we calculated the semantic similarity via the model and tested its closeness to a human evaluation. Following previous studies (Reimers and Gurevych, 2019; Gao et al., 2021; Jiang et al., 2023), the sentence embedding quality was evaluated in terms of Spearman’s rank correlation coefficient between the cosine similarity of sentence embeddings and the human ratings.
Experimental Setup We fine-tuned LLaMA-27B (Touvron et al., 2023) with NLI datasets. Following Jiang et al. (2023), we used the same seven STS datasets for evaluation: STS 2012– 2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016), STS-B (Cer et al., 2017), and SICK-R (Marelli et al., 2014). To investigate the relationship between dataset size and performance, we trained our model with different numbers of examples. The number of examples in the datasets is 4,000×2n (n = 0, 1, . . . , 6). For fine-tuning with PromptEOL, we used NLI datasets generated with 0-shot, 1-shot, 5-shot, 20-shot, 1-shot×5 (five combined 1-shot datasets), 5-shot×4 (four combined 5-shot datasets) setups and the manual NLI dataset. We used the same hyperparameters as PromptEOL (Jiang et al., 2023), with a batch size of 256 during training, 10% of the total steps for warm-up, and a learning rate of 5e-4. During training, we calculated Spearman’s rank correlation coefficient on the STS-B develop-
3Evaluations were also conducted on downstream tasks of SentEval (Conneau and Kiela, 2018), but as reported in Jiang et al. (2023), the effectiveness of fine-tuning with the NLI dataset could not be confirmed. We provide the results of downstream tasks in Appendix C.
ment set every (number of data / 4000) step and used the model with the highest score for the final evaluation. To minimize randomness from fewshot examples, we generated multiple NLI datasets: 10 for 1-shot, 5 for 1-shot×5 and 5-shot, 4 for 5- shot×4 and 20-shot, and 3 for zero-shot. We report their average scores for the final evaluation.
Experimental Results Figure 1 shows the results. Comparing the zero-shot and few-shot results, the few-shot performances outperformed the zero-shot performance regardless of the amount of data size, thus confirming the effectiveness of few-shot learning. Comparing 1-shot, 5-shot, and 20-shot, there there was no improvement in scores as the number of shots increased. This indicates that merely increasing the number of shots does not necessarily lead to better performance. Although there was little performance difference between 5-shot and 1-shot×5, 5-shot×4 consistently outperformed 20-shot, regardless of data size. According to Section 4.1, although the quality of the generated dataset with 1-shot learning is not sufficient, the generated dataset with 5-shot learning has sufficiently high quality. This suggests that distributing few-shot examples can improve performance, but only when the data quality exceeds a certain threshold. 5-shot×4 successfully introduces diversity while maintaining sufficient quality, and this balance between diversity and quality appears to be crucial for enhancing the effectiveness of sentence embeddings generated from NLI datasets.

heading: 4.3 Comparison with Existing Methods
text: To evaluate the performance of models trained on automatically generated NLI datasets, we compared the following five models: 1) PromptEOL without fine-tuning, 2) PromptEOL fine-tuned with the generated dataset using 0-shot learning, 3) PromptEOL fine-tuned with the generated dataset
using 5-shot×4 learning, 4) PromptEOL finetuned with the manual NLI dataset, 5) Unsupervised PromptRoBERTa (Jiang et al., 2022), which achieved the highest performance without using manually annotated large-scale datasets. For unsupervised PromptRoBERTa, we used the premise sentences to automatically generate NLI datasets, which are used for training. For PromptRoBERTa and experiments using manually annotated datasets, we conducted experiments three times with different random seeds, and we reported their average scores as the final score. Other experimental settings and evaluation methods were the same as in Section 4.2.
Experimental Results Figure 2 shows the results. Overall, the models trained with automatically generated datasets consistently outperformed unsupervised methods. Specifically, the 5-shot×4 setting achieved the highest score of 82.71. Comparing the performance of PromptEOL without fine-tuning and PromptEOL fine-tuned with the automatically generated dataset using zero-shot learning, the fine-tuned model consistently outperformed. This indicates that fine-tuning with the generated NLI dataset is effective when no manually created examples are available. Moreover, our models outperformed PromptRoBERTa, indicating that our model achieved the best performance without using large manually annotated datasets.
Compared to the model fine-tuned with the manual dataset, the performance of the 5-shot×4 setting was 2–3 points lower. This indicates that there is still a gap between the 5-shot×4 dataset and the manual dataset, suggesting room for improvement. Despite this gap, there was an approximately 10- point performance improvement compared to the model without fine-tuning, confirming the effectiveness of the automatically generated dataset. We
provide the detailed experimental results in Appendix D.

heading: 5 Conclusion and Future Work
text: In this study, we explore optimal ways to leverage few-shot examples when using LLMs to generate NLI datasets for sentence embedding learning. Through experiments, we found that the performance could be enhanced by dividing the few-shot examples, as seen with the 5-shot×4 setting, since it improves dataset diversity. Furthermore, models trained with automatically generated NLI datasets outperformed existing unsupervised methods.
In future work, we will explore more sophisticated ways to generate a diverse and high-quality dataset. For example, instead of just dividing fewshot examples, a set of various overlapping fewshot examples could be generated and used in fewshot learning. It is also future work to apply our data generation procedure, which generates data by dividing few-shot examples, to data generation other than NLI datasets for sentence embedding learning.
Limitations
There are three major limitations in this study. Firstly, we only conducted experiments using LLaMA-2-7B as the LLM for both the automatic generation of the NLI dataset and the generation of sentence embeddings. It is known that the quality of generated sentences improves as the number of parameters in the LLM increases. In this study as well, it may be possible to obtain higher quality NLI datasets and sentence embedding models by using a model larger than LLaMA-2-7B. Since this method is expected to be applicable to many LLMs without depending on a specific LLM, to demonstrate the model-independent usefulness of our observations, we need to conduct experiments using various LLMs, such as the GPT series and OPT (Zhang et al., 2022).
Secondly, we followed the previous research, PromptEOL, and conducted evaluations using SentEval. However, it is not enough to comprehensively assess the quality of sentence embeddings to evaluate only with the STS tasks and SentEval. It is necessary to use various benchmarks, such as MTEB (Muennighoff et al., 2023), which evaluate sentence embeddings from multiple perspectives.
Thirdly, the experiments were conducted only in English. It is potentially applicable to many
languages to generate datasets automatically because it does not require large, manually-annotated datasets, but our experiments were conducted only in English. To demonstrate the usefulness of our observation for multiple languages and improve cross-lingual/multi-lingual sentence embeddings, it is helpful to conduct experiments in languages other than English.

heading: Acknowledgements
text: This work was partly supported by JSPS KAKENHI Grant Number 24H007271.

heading: A Frequency Distribution of Token Counts in the Manual NLI Dataset
text: Figure 3 shows the frequency distribution of the token counts in the manual NLI dataset. The counts for most sentences are distributed between 1 and 100, with about 83.0% of them having counts between 4 and 32. Accordingly, we used sentences within that range in this work.

heading: B Examples of Automatically Generated NLI Datasets
text: Tables 2 and 3 provide examples of NLI data that were automatically generated with 0-shot and 20- shot learning, respectively, for the same premise sentences.
We observed that some sentences generated in the zero-shot setting are predicted as neutral, but sentences close to entailment and contradiction can also be generated. By shifting the period or using negation, diverse entailment and contradiction sentences can be created. The sentences generated with 20-shot learning tended to strongly refer to the premise sentence, indicating higher precision in generating both entailment and contradiction sentences. Additionally, these sentences tended to be shorter than those generated with 0-shot learning.

heading: C Evaluation of Transfer Tasks
text: To evaluate the effectiveness of the generated sentence embeddings in transfer tasks, we conducted evaluations with transfer tasks from SentEval (Conneau and Kiela, 2018). These tasks use sentence embeddings as input and train a linear classifier. Specifically, the embeddings generated from each sentence are used as features to train linear classifiers, such as logistic regression. The classification task performance is assessed with the trained classifier, and the accuracy and other related metrics are measured to quantitatively evaluate the effectiveness of the sentence representations.
Table 4 summarizes the results. As reported in Jiang et al. (2023), we could not confirm any performance improvement on the transfer tasks via fine-tuning; however, the scores for both the zero-shot and few-shot settings were comparable
to those with training on the manual NLI dataset. Tables 5 and 6 show detailed scores for each experiment. In the proposed method, it is confirmed that the score is low and unstable when the data size is small, but it stabilizes as the data size increases.

heading: D Detailed STS Scores
text: Table 7 shows the performances of the STS tasks for each model with 256,000 examples. Additionally, Tables 8 and 9 show detailed performances of the STS tasks for each experiment. It is evident that good sentence embedding models have been created without any extreme highs or lows for any dataset. Furthermore, the 1-shot setting tends to have a larger variance, while the variance tends to decrease as the number of shots increases. This confirms the validity of increasing the number of trials as the number of shots increases.

heading: Future Work
text: In future work, we will explore more sophisticated ways to generate a diverse and high-quality dataset. For example, instead of just dividing fewshot examples, a set of various overlapping fewshot examples could be generated and used in fewshot learning. It is also future work to apply our data generation procedure, which generates data by dividing few-shot examples, to data generation other than NLI datasets for sentence embedding learning. Limitations
There are three major limitations in this study. Firstly, we only conducted experiments using LLaMA-2-7B as the LLM for both the automatic generation of the NLI dataset and the generation of sentence embeddings. It is known that the quality of generated sentences improves as the number of parameters in the LLM increases. In this study as well, it may be possible to obtain higher quality NLI datasets and sentence embedding models by using a model larger than LLaMA-2-7B. Since this method is expected to be applicable to many LLMs without depending on a specific LLM, to demonstrate the model-independent usefulness of our observations, we need to conduct experiments using various LLMs, such as the GPT series and OPT (Zhang et al. , 2022). Secondly, we followed the previous research, PromptEOL, and conducted evaluations using SentEval. However, it is not enough to comprehensively assess the quality of sentence embeddings to evaluate only with the STS tasks and SentEval. It is necessary to use various benchmarks, such as MTEB (Muennighoff et al. , 2023), which evaluate sentence embeddings from multiple perspectives. Thirdly, the experiments were conducted only in English. It is potentially applicable to many
languages to generate datasets automatically because it does not require large, manually-annotated datasets, but our experiments were conducted only in English. To demonstrate the usefulness of our observation for multiple languages and improve cross-lingual/multi-lingual sentence embeddings, it is helpful to conduct experiments in languages other than English