Meta-Task Prompting Elicits Embeddings from Large Language Models (2024)

Abstract: We introduce a new unsupervised text embedding method, Meta-Task Prompting with Explicit One-Word Limitation (MetaEOL), for generating high-quality sentence embeddings from Large Language Models (LLMs) without the need for model fine-tuning. Leveraging meta-task prompting, MetaEOL guides LLMs to produce embeddings through a series of carefully designed prompts that address multiple representational aspects. Our comprehensive experiments demonstrate that embeddings averaged from various meta-tasks are versatile embeddings that yield competitive performance on Semantic Textual Similarity (STS) benchmarks and excel in downstream tasks, surpassing contrastive-trained models. Our findings suggest a new scaling law, offering a versatile and resource-efficient approach for embedding generation across diverse scenarios.1

heading: 1 Introduction
text: The advent of Large Language Models (LLMs) such as GPT-3 (Brown et al., 2020) and LLaMA (Touvron et al., 2023a) has marked a significant milestone in the field of natural language processing (NLP), introducing promising unsupervised methods for various NLP tasks by leveraging task-related instructions or prompts (Qin et al., 2023; Zhong et al., 2023; Zhao et al., 2023). These tasks also include the generation of sentence embeddings, which aims to produce sentence representations that can be applied across a wide range of scenarios. They have been applied to intrinsic tasks like Semantic Textual Similarity (STS) (Agirre et al., 2012a; Cer et al., 2017b), to downstream tasks including information retrieval (Mitra et al., 2017; Izacard et al., 2021), and to sentiment classification (Ke et al., 2020) and beyond. By employ-
*Corresponding to: Yibin Lei (e-mail: y.lei@uva.nl) and Chongyang Tao (e-mail: chotao@microsoft.com).
1Our code is publicly available at https://github.com/ Yibin-Lei/MetaEOL.
ing specific prompts (Jiang et al., 2023b, 2022a), researchers have begun to explore the potential of extracting meaningful sentence embeddings directly from the hidden states of LLMs without the need for explicit training. These prompt-based approaches generate embeddings without the need for any fine-tuning or in-context learning, which is a substantial improvement over approaches that require extensive fine-tuning to achieve high performance.
Initial efforts in this domain, as highlighted by works like (Jiang et al., 2023b, 2022a; Liu et al., 2023a), have focused on unsupervised techniques that extract sentence representations directly from LLMs. These methods typically involve using fill-in-the-blanks prompts, such as This sentence: "[TEXT]" means in one word:" (Jiang et al., 2023b), to embed a sentence into a single token representation by using the output hidden state of the last token as the sentence’s embedding. While they
10141
perform well, these approaches also reveal the inherent challenges of this task: embeddings may be overly simplistic or misaligned with the intended semantic nuances of the sentences.
In a pilot experiment illustrated in Figure 1, we demonstrate that a previous prompt-based method (Jiang et al., 2023b) can struggle to capture a sentence’s meaning, especially when the usage of the sentence is associated with multiple aspects. When probing the probability distribution for the next token during decoding, which reflects the embedding quality of the last token2, the highest probabilities are mostly distributed to frequent stop words. Although the general movie topic appears, other meaningful aspects like sentiments are missing.
A straightforward solution to mitigate this issue is to provide LLMs with task-specific instructions. This approach involves instructing the model with prompts explicitly designed for a particular task, thereby tailoring the embeddings to better suit the specific requirements of that task. However, considering the wide range of distinct tasks that an embedding may be used for (Mishra et al., 2022; Wang et al., 2022; Chung et al., 2022), this would be impractical. Furthermore, while task-specific embeddings are effective for their corresponding tasks, they may fail to generalize well across different tasks.
Inspired by the principles of the usage-based theory of language acquisition (Tomasello, 2009), which asserts that the essence of meaning is rooted in the practical utilization of language, our approach aims to generate broad embeddings through the use of meta-task prompting, inspired by metatask prompted training (Sanh et al., 2022) and hyper-prompt (He et al., 2022) techniques. By defining a suite of meta-tasks, each tailored to a distinct application context, MetaEOL prompts LLMs to consider multiple representational tokens from a variety of perspectives. This multifaceted approach enables the extraction of more diverse and nuanced contextualized token embeddings that collectively form a comprehensive sentence embedding.
Extensive experiments empirically show that: (i) Simply averaging embeddings from different metatasks without any training leads to general embeddings that are competitive to contrastive-trained models on STS tasks and can achieve the best av-
2The decoding probabilities are derived by comparing the similarity between the output hidden state of the last token and the token embeddings of the whole vocabulary.
erage result on several downstream tasks. (ii) Incrementally integrating more meta-tasks (ranging from one to four) yields consistent improvements across STS tasks, showcasing high generalities, and highlighting the significant impact of meta-task integration on overall performance. (iii) The final layer is not always the most effective for STS tasks and with a simple proportional layer selection strategy, we achieve the best results with a 70B model, which points to a potential scaling law.

heading: 2 Related Work
text: Sentence Embeddings. Sentence embeddings aim to encode the semantic content of sentences into fixed-sized vector representations. Recent developments in contrastive learning have proven to be highly effective for generating sentence embeddings, under both unsupervised and supervised settings (Gao et al., 2021; Jiang et al., 2022a; Chuang et al., 2022; Wu et al., 2022). For instance, SimCSE (Gao et al., 2021) utilizes different dropout masks as a form of noise to create positive pairs in an unsupervised fashion, while in a supervised setting, models like Sentence-BERT (Reimers and Gurevych, 2019) leverage natural language inference (NLI) datasets to construct positive and negative pairs. Additionally, Su et al. (2023) and Asai et al. (2023) show that training with a large amount of tasks with annotated instructions can enable the model to generate embeddings tailored to different downstream tasks. In contrast, our approach MetaEOL demonstrates the potential of utilizing LLMs directly to generate instruction-followed embeddings without the need for any additional training.
Large Language Models for Text Representation. Recent studies have explored the use of LLMs for enhancing text embeddings through data augmentation techniques (Cheng et al., 2023; Zhang et al., 2023). Notably, Sentence-T5 (Ni et al., 2022a) and GTR (Ni et al., 2022b) employ contrastive learning on models with billions of parameters. More recently, research has focused on converting an LLM directly into a text encoder without any training. Liu et al. (2023b) represents sentences through the distribution of possible text continuations, comparing the distributional similarity between sentences. This method, although effective, necessitates the generation of 20 trajectories, each up to 20 tokens in length, making it computationally intensive. Jiang et al. (2022a) in-
corporates in-context learning (Dong et al., 2023) to enhance sentence embeddings. While proven effective, it also reveals that the produced embeddings are task-specific and struggle with generalization across various downstream tasks, in addition to being highly sensitive to the choice of demonstrations. Contemporaneous works investigate the potential of LLMs either by repeating the input texts (Springer et al., 2024) or enabling bidirectional attention (BehnamGhader et al., 2024) to address the lack of backward dependency in LLMs. Additionally, the potential of LLMs under supervised training settings has also been studied in recent works (Ma et al., 2023; Li and Li, 2024b,a; Wang et al., 2024; Li et al., 2024; Lee et al., 2024).
Multitask Prompts. Studies have demonstrated that models fine-tuned using multi-task prompts and datasets can serve as general-purpose models with strong capabilities in generalizing to new tasks (Sanh et al., 2022; Wei et al., 2022; Chung et al., 2022; Wang et al., 2022; Mishra et al., 2022). Our approach MetaEOL aligns with this concept, showcasing that multi meta-task prompts can similarly generate general-purpose embeddings, remarkably without necessitating any training.

heading: 3 Method
text: In this section, we begin by reviewing two kinds of previous prompting methods for deriving sentence representation from masked and causal language models, respectively (Section 3.1). Subsequently, we describe our proposed method, i.e., Meta-Task Prompting with Explicit One-Word Limitation
(MetaEOL) in detail (Section 3.2). Lastly, we describe meta-tasks involved in this paper (Section 3.3).

heading: 3.1 Previous Language Model Prompting

heading: 3.1.1 Masked Language Model
text: Masked language models, e.g., BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), use a mask prediction task to capture contextual information for a certain token. To align with this point, PromptBERT (Jiang et al., 2022a) formulates the sentence embedding extraction as a similar task and employs the following template,

heading: This sentence : “[TEXT]” means [MASK] .
text: for prompting. Here, [TEXT] and [MASK] indicate the placeholder for the input sentence and the mask token. The last layer’s hidden vector of [MASK] token is directly used as the sentence representation.
Jiang et al. (2022a) empirically show that such a simple prompting method can achieve decent performance, and equipping it with a contrastive loss for large-scale continued training leads to further enhancements for embedding quality. However, it is worth noting that extra training is resourceintensive, especially for today’s LLMs. To enhance clarity, we provide results both with and without training on BERT and RoBERTa in the following experiments.

heading: 3.1.2 Causal Language Model
text: Others have investigated directly extracting sentence representation from large Causal Language Models (CLMs), e.g., OPT (Zhang et al., 2022)
or LLAMA (Touvron et al., 2023a), without additional training. Inspired by Jiang et al. (2022a), PromptEOL (Jiang et al., 2023b) employs a similar prompting template as follows,

heading: This sentence: “ [TEXT] ” means in one word:“
text: where the last layer’s hidden vector for the last token ““” is extracted as the sentence representation. A constraint of “in one word” is applied to avoid the model’s tendency to generate long sentences such that the last token fails to capture the overall information.
However, the obtained embedding highly relies on the single prompt, which confines the inference process and can result in non-comprehensive features. For example, as shown in Figure 1, for a negative review of a movie, the resulting embedding does not capture critical aspects such as sentiment.

heading: 3.2 Meta-Task Prompting
text: To overcome the issues raised above, we propose Meta-Task Prompting with Explicit One-Word Limitation (MetaEOL). A meta-task is associated with a potential broad usage scenario for the corresponding sentence representation. As shown in Figure 2, we directly prompt casual LLMs with the goals of multiple meta tasks, aiming to obtain the representations under various broad intents.
Specifically, we produce task-oriented prompts by decorating the original prompting template used for causal LLMs (Section 3.1.2) with the corresponding task description. For example, given a meta-task where representations are extracted for Text Classification (TC), we extend the template with task-oriented context to define the behavior during inference. As shown in the template of Meta Task-1 in Figure 2, a detailed task description text is placed at the beginning of the prompt, instructing the LLM to categorize the excerpt into a broad category. Then, an instruction with a constraint of “in one word” is followed to ensure models aggregate the information of the whole sentence into the embedding of the last token. The placeholder [TEXT] will be substituted with the original sentence to produce the final task-oriented prompt. The resulting task-specific prompt will serve as input to LLMs. Subsequently, we extract the hidden vector of the last token ““” as the sentence representation, following the pattern outlined in Section 3.1.
It is worth noting that given various meta-tasks, distinct templates will be employed, leading to multiple different sentence embeddings. Our hypothe-
sis is that each embedding captures a distinct representation customized for a specific feature viewpoint (meta-task). In this paper, we empirically show that simply averaging different embedding derived from multiple meta-tasks can achieve superior performance for both intrinsic and downstream evaluation benchmarks.

heading: 3.3 Types of Meta-Tasks
text: In this paper, we conduct experiments on the following four distinct meta-tasks, i.e., Text Classification (TC), Sentiment Analysis (SA), Paraphrase Identification (PI), and Information Extraction (IE), aiming to capture information from different angles. For example, intuitively, the TC task primarily emphasizes topic-level information, whereas the IE task concentrates on surface-level signals.
For each meta-task, we straightforwardly leverage ChatGPT-4 as a template generator to produce multiple templates. The instruction we used to prompt the ChatGPT-4 is provided in Appendix A.1.
Note that introducing more meta-tasks is trivial, because it only requires adding more task names to the generator. Here, we choose the above four meta-tasks as a testbed to assess scalability. More specifically, in Section 5.2, we show that incrementally adding more meta-tasks to our workflow results in consistently better performance.

heading: 4 Experiments

heading: 4.1 Settings
text: Dataset. Suggested by prior works (Reimers and Gurevych, 2019; Gao et al., 2021; Jiang et al., 2022b) that an important objective of sentence embeddings is to cluster semantically similar sentences, we evaluate MetaEOL on seven semantic textual similarity (STS) datasets, utilizing the SentEval toolkit (Conneau and Kiela, 2018). The STS datasets consist of STS 2012-2016 (Agirre et al., 2012b, 2013, 2014, 2015, 2016), STS-B (Cer et al., 2017a), and SICK-R (Marelli et al., 2014). Each sentence pair in the STS datasets is annotated with a score from 0 to 5 indicating the pairwise semantic similarity. The Spearman correlation (scaled by 100x) between the model-predicted similarity scores and human-annotated similarity scores is used as the metric. We employ cosine similarity to measure the similarity between sentence embeddings. The Spearman correlation is computed under the “all” setting.
Baselines. The baselines we consider can be categorized into two types – models with contrastive training and without contrastive training: (i) Models with Contrastive Training: We compare MetaEOL with SOTA unsupervised contrastivetrained models, namely SimCSE (Gao et al., 2021) and PromptBERT (Jiang et al., 2022a). The models are trained on 106 sentences randomly sampled from Wikipedia. Results based on BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) models are reported. Contemporaneous LLM-based approach LLM2Vec (BehnamGhader et al., 2024) is also included for comparison. LLM2Vec comprises three stages: bidirectional attention enabling, masked next token prediction training, and unsupervised contrastive training (similar to SimCSE) to transform an LLM into a text encoder. Considering (ii) Models without Contrastive Training: We compare MetaEOL with (1) average pooling methods, where average pooling is applied to the output hidden states of all tokens in a sentence to obtain the sentence embedding. We report results with BERT, the encoder of ST5 (Ni et al., 2022a), LLAMA2 (Touvron et al., 2023b) and Mis-
tral (Jiang et al., 2023a) models; and (2) Promptbased methods, which include BERT Prompt that employs the same prompt strategy as PromptBERT but does not incorporate contrastive training, PromptEOL and the contemporaneous Echo embeddings (Springer et al., 2024). Echo embeddings repeat the input once and extract embeddings from the second occurrence. All methods mentioned above rely on the output from the final layer to obtain the sentence embedding.
Implementation Details. We apply MetaEOL to LLAMA2-7B, LLAMA3-8B, LLAMA2-13B, and Mistral-7B models, using meta-tasks consisting of Text Classification (TC), Sentiment Analysis (SA), Paraphrase Identification (PI), and Information Extraction (IE). These tasks are distinct and collectively consider diverse aspects of a sentence. For each of these meta-tasks, we utilize GPT-4 to create two unique task prompts, resulting in a total of eight task prompts.3 MetaEOL rely on the output from the final layer to obtain the sentence embedding. We simply average the resulting em-
3The details of these eight task prompts are presented in Appendix A.3.
beddings of task prompts from different meta-tasks to obtain the final embedding.

heading: 4.2 Main Results
text: The results of MetaEOL on STS tasks are shown in Table 1, with notable performance by MetaEOL which requires no training. Among models that do not require training, prompt-based methods exhibit superior results compared to average pooling methods, especially with the LLAMA and Mistral models. Across various models including LLAMA27B/13B, LLAMA3-8B, and Mistral-7B, MetaEOL demonstrates competitive performance compared to contrastive-trained models such as SimCSEBERT and SimCSE-Roberta, albeit with a slight lag behind PromptBERT. Furthermore, MetaEOL significantly outperforms PromptEOL and Echo embeddings across various test models, demonstrating a consistent improvement. Notably, the LLAMA2-13B model using MetaEOL shows an average improvement of 6.73% over PromptEOL, underscoring the efficacy of MetaEOL. Compared to LLM2Vec which requires two-stage training, MetaEOL is competitive when using LLAMA27B, without the need for any training.

heading: 4.3 Qualitative Example
text: We further show the top-10 tokens predicted by different task prompts in Table 2. The example illustrates that PromptEOL creates sentence embeddings focusing on stop-word tokens (such as a, this, the, it), which convey minimal information. In contrast, the four meta-tasks of MetaEOL demonstrably shift the behavior of the embeddings, leading to the prediction of tokens that are distinct and imbued with substantive meaning.
Specifically, Text Classification steers the embeddings toward tokens that are indicative of specific topics, such as Culture. Sentiment Analysis is inclined to produce embeddings close to sentiment-
related words. Paraphrase Identification yields embeddings that capture a spectrum of synonyms, enriching the sentence with varied linguistic expressions like intelligent, short, and clever. Information Extraction modifies the embeddings towards tokens that represent key facts or elements within the sentence.

heading: 5 Analysis
text: In this section, we thoroughly analyze MetaEOL using the LLAMA2-7B model.

heading: 5.1 Ablation Study
text: We evaluate the effectiveness of key components of MetaEOL in Table 3. First, to ensure the improvement observed with MetaEOL is not merely due to involving more prompts, we create seven paraphrased versions of the PromptEOL prompt, resulting in a total of eight prompts.4 We then average the embeddings from these eight prompts to form the final sentence embedding. We find merely duplicating PromptEOL prompts (w. 7 paraphrase) does not improve PromptEOL but results in a significant decline. Additionally, we im-
4The seven paraphrased prompts are presented in Appendix A.2
plement MetaEOL exclusively on each meta-task (TC/SA/PI/IE only). We find that tasks requiring a detailed comprehension of sentences (PI and IE) yield superior performance compared to those requiring a broader understanding, even surpassing PromptEOL. MetaEOL, which combines the embeddings from these meta-tasks, outperforms all individual meta-tasks, confirming the complementarity of the meta-tasks and the effectiveness of combining embeddings from diverse meta-tasks. We finally find that averaging the embeddings from different meta-tasks yields better results than either concatenating them or max pooling them across each dimension.

heading: 5.2 Influence of Number of Tasks
text: We investigate the influence of the number of tasks as presented in Table 4. We find increasing the number of tasks leads to a consistent improvement in performance on average and nearly every individual STS task. This further verifies the complementarity of the meta-tasks and underscores the importance of utilizing various diverse meta-tasks.

heading: 5.3 Influence of Number of Prompts
text: Here, we investigate the impact of the number of prompts in Figure 3. We concentrate on Sentiment Analysis as the meta-task and utilize GPT-4 to generate three additional Sentiment Analysis prompts besides the two we used in MetaEOL. This results in a total of five distinct prompts, specifically tailored for Product Review Rating, Emotion Detection, Sentiment Polarity Detection, Sentiment Intensity and Emotion Detection, and Aspect-Based Sentiment Analysis, respectively.5 We systematically computed the average performance across all combinations of these five prompts, conditioned on a fixed number of prompts.
As Figure 3 shows, increasing the number of prompts within a particular task type facilitates more nuanced embeddings, thereby leading to better STS results. We opt for two prompts for each
5The details of these five instructions are in Appendix A.4.
meta-task for MetaEOL to optimize both performance and computational efficiency.

heading: 5.4 Prompt Sensitivity Analysis
text: To test the sensitivity of MetaEOL, we specifically focus on the influence of (i) Tiny perturbations on the task prompt; and (ii) Variations of the major prompting instruction in Appendix A.1. 5.4.1 Sensitivity to Tiny Prompt
Perturbations We apply the synonym replacement operation in Wei and Zou (2019) to replace 10% of words in the sentiment analysis task prompt of MetaEOL with their synonyms. We craft an additional 4 perturbed prompts. The synonym replacements are sourced directly from WordNet without filtering, which often results in unnatural substitutions, as shown in the Appendix A.5. To provide context for the sensitivity, we included results from SimCSEBERT-Base with varying random seeds in Jiang et al. (2022a) as a reference.
The results in Table 5 show that even with unnatural substitutions, our MetaEOL-LLAMA2-7B still exhibits a standard deviation of ±0.67 on STS Avg., which is in line with the variance observed in SimCSE-BERT-Base (±0.86), suggesting that
our method’s sensitivity to prompt perturbations is comparable to that of existing approaches to random seeds.

heading: 5.4.2 Sensitivity to Variations of the Major
text: Prompting Instruction
We vary the example task prompt in the major prompting instruction in Appendix A.1 with task prompts from the four meta-tasks used in our MetaEOL. As the major prompting instruction is used to prompt ChatGPT-4 to generate task prompts, changing it will lead to a completely different set of task prompts.
Overall the results show MetaEOL with LLAMA2-7B can beat tuned SimCSE-BERT-Base on average, and is worse but still comparable to SimCSE-Bert-Base in terms of standard deviation, suggesting that MetaEOL’s sensitivity to major prompting instruction’s variations is also comparable to that of existing approaches to random seeds.

heading: 5.5 Influence of Output Layers
text: We check the impact of output layers for LLAMA2 and Mistral-7B models, using PromptEOL and MetaEOL. Figure 4 shows that the last layer is not always the most effective for STS tasks, which is consistent with the findings in Li and Li (2024b).
It is highlighted that the third-to-last layers (indexed as -3) across all four configurations perform similarly well, which suggests that this layer can be considered as a point of convergence in terms of optimal performance for these models.
MetaEOL outperforms PromptEOL across all layers and configurations. Interestingly, PromptEOL tends to show more variability in performance across different layers compared to MetaEOL. This suggests that the MetaEOL approach potentially stabilizes the representational quality across layers.

heading: 5.6 Scaling LLMs
text: Previous study (Jiang et al., 2023b) show scaling model sizes does not lead to performance improvement on STS tasks. In this section, we investigate the impact of model size on the performance of
MetaEOL. For the sake of computational resources, we load models with 4-bit precision.
Informed by the insights observed from Section 5.5, which suggested that for 7B models, the layer index -3 can be considered optimal, as evidenced by its performance in both PromptEOL and MetaEOL. We, therefore, propose a simple proportional layer selection strategy, opting for layers -3 of 32, -4 of 40, and -8 of 80 as the output layers for the LLAMA2-7B, LLAMA2-13B, and LLAMA2-70B models respectively. This approach aligns with the model sizes, which correlates to 10% from the final layer.
The results in Table 7 show that using the final layer for sentence embedding generation, which is indicated by layer index -1, does not yield improved performance with increased model size. Contrastingly, the application of our proportional layer strategy reveals a different trend. Specifically, the LLAMA2-70B model, which utilizes the -8 layer, demonstrates superior performance, suggesting that larger models might benefit more significantly from selecting a proportionate layer rather than the last layer for sentence embedding. This observation could point to a potential scaling law, where larger models require a different, non-final layer to maximize performance effectively.
Method Params MR CR SUBJ MPQA SST TREC MRPC Avg.
Fine-tuning on supervised datasets SimCSE-RoBERTa 123M 84.92 92.00 94.11 89.82 91.27 88.80 75.65 88.08 ST5-Enc 4.8B 90.83 94.44 96.33 91.68 94.84 95.40 77.91 91.63
Without fine-tuning MRPrompt-LLAMA2 7B 91.82 92.88 97.07 91.60 96.54 95.80 74.61 91.47 CRPrompt-LLAMA2 7B 91.17 93.27 96.62 91.75 96.60 95.80 73.22 91.20 SUBJPrompt-LLAMA2 7B 91.88 93.17 96.96 91.09 95.66 96.00 76.41 91.60 MPQAPrompt-LLAMA2 7B 91.10 93.04 96.30 91.82 95.72 96.00 75.42 91.34 SSTPrompt-LLAMA2 7B 91.82 92.88 97.07 91.60 96.54 95.80 74.61 91.47 TRECPrompt-LLAMA2 7B 88.97 92.19 96.23 91.45 94.18 96.80 74.72 90.65 MRPCPrompt-LLAMA2 7B 90.33 93.32 96.36 91.45 94.67 96.00 75.13 91.04
Avg. on task-specific prompting (i.e., diagonal): 91.76

heading: 5.7 Transfer Learning Tasks
text: We conclude our analysis by assessing the performance of MetaEOL on transfer learning tasks. Following prior works (Gao et al., 2021; Ni et al., 2022a), we utilize the standard transfer learning tasks provided by SentEval. The tasks consist of MR (Pang and Lee, 2005), CR (Hu and Liu, 2004), SUBJ (Pang and Lee, 2004), MPQA (Wiebe et al., 2005), SST-2 (Socher et al., 2013), TREC (Voorhees and Tice, 2000), and MRPC (Dolan and Brockett, 2005). For each task, logistic regression classifiers are trained using the created sentence embeddings as input features. The test accuracy on each task is used as the metric. Additionally, we include two supervised contrastivetrained models (SimCSE and ST5-Enc) for reference. Notably, ST5-Enc, a model with a 4.8B parameter count, is extensively trained on natural language inference (NLI) data and two billion question-answer pairs.
To investigate the ability of task-specific prompts to modify embedding behavior, we have crafted task prompts tailored to each SentEval task.6 As an example, for the Movie Review (MR) dataset, we designed a prompt structured as: In this task, you’re given a movie review, and you need to classify its sentiment into positive or negative. For this task, this sentence: "input sentence" means in one word:", referred to as MRPrompt in Ta-
6The details of the task prompts are in Appendix A.6.
ble 8. These task-specific prompts significantly improve the corresponding task performance, always better than PromptEOL and heavily supervised contrastive-trained ST5-Enc, verifying that LLAMA2-7B can follow the prompt to generate tailored embeddings without any training. This indicates that carefully designed prompts can effectively steer the pre-trained embeddings to align with various NLP tasks, thus providing a more resource-efficient alternative to the traditional finetuning paradigm.
Moreover, although without being explicitly customized for these tasks, MetaEOL achieves the highest average result, even outperforming heavily trained ST5-Enc. This suggests that the integration of the four meta-tasks in MetaEOL can cultivate generalized embeddings that perform admirably across different tasks.

heading: 6 Conclusion
text: In this paper, we introduce MetaEOL, a new approach for deriving high-quality sentence embeddings from LLMs without requiring any training. By leveraging a diverse set of meta-task prompts, MetaEOL effectively captures multiple representations of sentences from distinct perspectives. We show simply averaging these meta-task derived embeddings leads to generalized general-purpose embeddings, which work remarkably well across STS datasets and transfer learning tasks.

heading: Limitations
text: We note two limitations in our work: computational overhead and restricted evaluation benchmarks. As MetaEOL requires feeding multiple prompts to LLMs to generate several embeddings, the computational cost will be higher than that of previous methods. Our results indicate that increasing the number of tasks leads to performance improvements, but it also worsens the efficiency issue. If the number of prompts is increased, the efficiency of our approach would further decrease. Nonetheless, in contexts where sentences are consistently reused, such as when embeddings are stored for downstream classification or retrieval tasks, the issue becomes less significant. Our evaluation is currently confined to sentence-level tasks in English only. As LLMs continue to advance, exploring the performance of MetaEOL in multilingual contexts and its applicability to document retrieval (Zhuang et al., 2024) presents an intriguing avenue for future research.

heading: Acknowledgement
text: This research was supported by the Hybrid Intelligence Center, a 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, https://hybrid-intelligence-centre.nl, and project VI.Vidi.223.166 of the NWO Talent Programme which is (partly) financed by the Dutch Research Council (NWO).

heading: A Appendix
text: A.1 Instruction to Prompt ChatGPT4 for Template Generation We insert a blank line between paragraphs to enhance readability.
Obtaining the representation of sentences is a fundamental task in natural language processing.
The representation can not only be used to compute the semantic similarity between different sentences but also to be directly used for downstream tasks, like Text Categorization, Sentiment Analysis, Summarization, Style Transfer, Text Simplification, and Sentence Composition.
A common way to obtain the representation is to use the format "This sentence "input sentence" means in one word:"" and use the hidden states of the last token as the representation of the sentence. However, we want a versatile representation that covers various aspects of the sentence by adding task instructions before the format. For instance: "In this task, you’re given a review from Amazon. Your task is to generate a rating for the product on a scale of 1-5 based on the review. The rating means 1: extremely poor, 2: poor, 3: neutral, 4: good, 5: extremely good. For this task, this sentence : "input sentence" means in one word:"" is used to obtain the representation of the sentence conditioned on the given task.
Can you help me write task instructions that can cover different aspects of the sentence such that the representation is versatile to both similarity tasks and downstream tasks?
Please write two instructions for each of the Text Classification, Sentiment Analysis, Paraphrase Identification, and Information Extraction tasks.

heading: A.2 Paraphrased Prompts of PromptEOL
text: 1. This sentence : "input sentence" can be rephrased to one word:" 2. This sentence : "input sentence" can be expressed as one word:" 3. This sentence : "input sentence" implies in one word:" 4. This sentence : "input sentence" indicates in one word:" 5. The meaning of this sentence : "input sentence" can be conveyed in another word:" 6. This sentence : "input sentence" can be restated as one word:" 7. This sentence : "input sentence" can be reformulated as one word:"

heading: A.3 Prompts of MetaEOL

heading: Text Classification
text: General Category Identification: In this task, you’re presented with a text excerpt. Your task is to categorize the excerpt into a broad category such as ’Education’, ’Technology’, ’Health’, ’Business’, ’Environment’, ’Politics’, or ’Culture’. These categories help in organizing content for better accessibility and targeting. For this task, this sentence : "input sentence" should be classified under one general category in one word:" Opinion vs. Fact Discrimination: In this task, you’re given a statement and you need to determine whether it’s presenting an ’Opinion’ or a ’Fact’. This distinction is vital for information verification, educational purposes, and content analysis. For this task, this sentence : "input sentence" discriminates between opinion and fact in one word:"

heading: Sentiment Analysis
text: Product Review Rating: In this task, you’re given a review from an online platform. Your task is to generate a rating for the product based on the review on a scale of 1-5, where 1 means ’extremely negative’ and 5 means ’extremely positive’. For this task, this sentence : "input sentence" reflects the sentiment in one word:" Emotion Detection: In this task, you’re reading a personal diary entry. Your task is to identify the predominant emotion expressed, such as joy, sadness, anger, fear, or love. For this task, this sentence : "input sentence" conveys the emotion in one word:"
Paraphrase Identification Similarity Check: In this task, you’re presented with two sentences. Your task is to assess whether the sentences convey the same meaning. Use ’identical’, ’similar’, ’different’, or ’unrelated’ to describe the relationship. To enhance the performance of this task, this sentence : "input sentence" means in one word:" Contextual Synonym Detection: In this task, you’re given a sentence and a phrase. Your task is to determine if the phrase can be a contextual synonym within the given sentence. Options include ’yes’, ’no’, or ’partially’. To enhance the performance of this task, this sentence : "input sentence" means in one word:"

heading: Information Extraction
text: Key Fact Identification: In this task, you’re examining a news article. Your task is to extract the most critical fact from the article. For this task, this sentence : "input sentence" encapsulates the key fact in one word:" Entity and Relation Extraction: In this task, you’re reviewing a scientific abstract. Your task is to identify the main entities (e.g., proteins, diseases) and their relations (e.g., causes, treats). For this task, this sentence : "input sentence" highlights the primary entity or relation in one word:"

heading: A.4 Prompts of the Sentiment Analysis Meta-Task

heading: Sentiment Analysis Meta-Task
text: Product Review Rating: In this task, you’re given a review from an online platform. Your task is to generate a rating for the product based on the review on a scale of 1-5, where 1 means ’extremely negative’ and 5 means ’extremely positive’. For this task, this sentence : "input sentence" reflects the sentiment in one word:" Emotion Detection: In this task, you’re reading a personal diary entry. Your task is to identify the predominant emotion expressed, such as joy, sadness, anger, fear, or love. For this task, this sentence : "input sentence" conveys the emotion in one word:" Sentiment Polarity Detection: In this task, you’re analyzing customer feedback from various platforms. Your task is to identify the overall sentiment polarity of the feedback. The sentiment polarity means: 1 for very negative, 2 for negative, 3 for neutral, 4 for positive, and 5 for very positive. Based on this guidance, this sentence : "input sentence" represents in one word:" Sentiment Intensity and Emotion Detection: In this task, your objective is to gauge the intensity and type of emotion conveyed in a piece of text, such as a social media post or a product review. This involves not just identifying whether the sentiment is positive or negative, but also understanding the strength of that sentiment and the specific emotions involved (e.g., joy, anger, sadness, surprise). For this task, this sentence : "input sentence" conveys an emotion that is best described in one word as:" Aspect-based Sentiment Analysis: In this task, you’re given a review of a product or service. Your task is to assess the sentiment toward specific aspects of the product or service mentioned in the review. For each mentioned aspect (e.g., quality, price, customer service), classify the sentiment as: 1 for very negative, 2 for negative, 3 for neutral, 4 for positive, and 5 for very positive. Based on this instruction, this sentence : "input sentence" signifies in one word:"

heading: A.5 Sentiment Analysis Task Prompts with Tiny Perturbatios

heading: Original
text: In this task, you’re given a review from an online platform. Your task is to generate a rating for the product based on the review on a scale of 1-5, where 1 means ’extremely negative’ and 5 means ’extremely positive’. For this task, this sentence : "input sentence" reflects the sentiment in one word:"

heading: Perturbed
text: 1. In this task, you’re given a reappraisal from an online chopine. Your task is to generate a rating for the product based on the reappraisal on a scale of 1-5, where 1 think of ’extremely negative’ and 5 think of ’extremely positive’. For this task, this sentence : "input sentence" reflects the sentiment in one word:" 2. In this task, you’re given a review from an online chopine. Your task is to generate a rating for the product based on the review on a scale of 1-5, where 1 means ’extremely damaging’ and 5 means ’extremely plus’. For this task, this sentence : "input sentence" reflects the sentiment in one word:" 3. In this job, you’re given a brush up from an online platform. Your job is to generate a rating for the product based on the brush up on a scale of 1-5, where 1 means ’highly negative’ and 5 means ’highly positive’. For this task, this sentence : "input sentence" reflects the sentiment in one word:" 4. In this task, you’re reach a refresh from an online platform. Your task is to generate a rating for the product based on the refresh on a scale of 1-5, where 1 means ’highly negative’ and 5 means ’highly positive’. For this task, this sentence : "input sentence" reflects the sentiment in one word:"

heading: A.6 Task-Specific Prompts on Transfer Tasks

heading: MR/SST
text: In this task, you’re given a movie review, and you need to classify its sentiment into positive or negative. For this task, this sentence : "input sentence" means in one word:"
CR In this task, you’re given a customer review of a product sold online, and you need to classify its sentiment into positive or negative. For this task, this sentence : "input sentence" means in one word:"

heading: SUBJ
text: In this task, you’re analyzing movie reviews to determine their level of subjectivity. A subjective review is filled with personal opinions, feelings, and preferences of the reviewer, often expressing likes or dislikes and personal experiences. An objective review, on the other hand, sticks to factual information, such as plot details or actor performances, without revealing the reviewer’s personal stance. For this task, this sentence : "input sentence" means in one word:"

heading: MPQA
text: In this task, you are given a description of a entity or event expressed in data such as blogs, newswire, and editorials. You need to classify its sentiment into positive or negative. For this task, this sentence : "input sentence" means in one word:"

heading: TREC
text: In this task, you are given a question. You need to detect which category better describes the question. A question belongs to the description category if it asks about description and abstract concepts. Entity questions are about entities such as animals, colors, sports, etc. Abbreviation questions ask about abbreviations and expressions abbreviated. Questions regarding human beings, description of a person, and a group or organization of persons are categorized as Human. Quantity questions are asking about numeric values and Location questions ask about locations, cities, and countries. Answer with "Description", "Entity", "Abbreviation", "Person", "Quantity", and "Location". For this task, this sentence : "input sentence" means in one word:"

heading: MRPC
text: In this task, you are given two sentences(Sentence1 and Sentence2). Answer "Yes" if these sentences are a paraphrase of one another, otherwise answer "No". For this task, this sentence : "input sentence" means in one word:"

heading: Future Work
text: , 2024) presents an intriguing avenue for future research