LLMEmbed: Rethinking Lightweight LLM’s Genuine Function in Text Classification (2024)

Abstract: With the booming of Large Language Models (LLMs), prompt-learning has become a promising method mainly researched in various research areas. Recently, many attempts based on prompt-learning have been made to improve the performance of text classification. However, most of these methods are based on heuristic Chain-of-Thought (CoT), and tend to be more complex but less efficient. In this paper, we rethink the LLM-based text classification methodology, propose a simple and effective transfer learning strategy, namely LLMEmbed, to address this classical but challenging task. To illustrate, we first study how to properly extract and fuse the text embeddings via various lightweight LLMs at different network depths to improve their robustness and discrimination, then adapt such embeddings to train the classifier. We perform extensive experiments on publicly available datasets, and the results show that LLMEmbed achieves strong performance while enjoys low training overhead using lightweight LLM backbones compared to recent methods based on larger LLMs, i.e. GPT-3, and sophisticated prompt-based strategies. Our LLMEmbed achieves adequate accuracy on publicly available benchmarks without any fine-tuning while merely use 4% model parameters, 1.8% electricity consumption and 1.5% runtime compared to its counterparts. Code is available at: https://github.com/ ChunLiu-cs/LLMEmbed-ACL2024.

heading: 1 Introduction
text: Recently Large Language Models (LLMs) have shown remarkable abilities on various NLP applications, such as GPT (Brown et al., 2020; OpenAI, 2023), PaLM (Chowdhery et al., 2023) and LLaMA (Touvron et al., 2023a,b), offering services to users through dialogue. Since LLMs refer to large-scale pre-trained language models (PLMs) that undergo *Equal contribution. †Corresponding author.
extensive training on massive textual corpora to understand the complexity and relationships within language, LLMs exhibit amazing emergent abilities in comprehension and reasoning (Wei et al., 2022a). This phenomenon further promotes research of prompt learning for LLMs (White et al., 2023), such as Chain-of-Thought (CoT) (Wei et al., 2022b) and Tree-of-Thoughts (ToT) (Yao et al., 2023).
Prompt-learning (Liu et al., 2023) is closely connected to the training and inference of LLMs. Instead of adapting the pre-trained language models (PLMs) to address downstream tasks, promptlearning directly adapts LLMs to cloze-style prediction, autoregressive modeling, or sequence to sequence generation, leading to promising performances on various tasks (Ding et al., 2022). Its major advantage of is that, given a suite of appropriate prompts, LLMs can be used to solve a great number of tasks (Brown et al., 2020; Sun et al., 2021) with no necessity of training from the scratch. However, such a paradigm is deeply involved with prompt engineering to find the most appropriate prompts to improve the overall performance.
Based on the sophisticated prompts, LLMs are guided to generate results and have achieved counterpart, or even stronger performance comparable to supervised baselines in various downstream NLP tasks such as natural language inference (Sheng et al., 2023), question answering (Trivedi et al., 2023), information extraction (Josifoski et al., 2023), named entity recognition (Wang et al., 2023) and relation extraction (Wadhwa et al., 2023). Nevertheless, as a generative model, LLMs still underperform discriminative models in text classification. Recently Sun et al. introduced the so-called clue and reasoning prompting (CARP) (Sun et al., 2023) to guide GPT-3 through text classification and got the SOTA performances on 5 widely-used text classification benchmarks.
For specific scenario customization of LLMs
7994
(Zhang et al., 2024) or considerations regarding user data privacy security (Tan et al., 2024), there is still a need for the localized deployment of LLMs. However, deploying LLMs is not always easily available as it requires enormous amounts of computation for training and inference (Xia et al., 2023). Moreover, it is worthy noting that only lightweight LLMs (e.g. LLaMA2-7B) are open-source (compared to GPT-3 with 175B parameter scale, 7B is lightweight) by far. Subsequently, most users can merely deploy the open-source lightweight LLMs. As the parameter scale of open-source lightweight LLMs is far smaller than online LLMs, the emergent abilities of lightweight LLMs fall behind (Wei et al., 2022a).
In this paper, we rethink if the prompt-based paradigm remain effective to lightweight LLMs for text classification, and propose a novel and effective paradigm, namely LLMEmbed to improve the overall training efficiency and generalized performance. Specifically, we fully investigate the effectiveness of prompt-based methodology in text classification, and realize a low-cost and easy-touse transfer learning framework via the use of LLM embeddings. We perform extensive experiments
on publicly available datasets and fairly compare LLMEmbed with recent baselines. It is observed that the the inference of prompt-based paradigms is complex and highly costly. Moreover, their performance is not as promising as CARP (Sun et al., 2023) claimed when lightweight LLMs are used, some of the generated results even deviates from the inputs, i.e. hallucination (Zhang et al., 2023; Huang et al., 2023). For comparison, our LLMEmbed enjoys a highly robust performance and very low training overhead in all scenarios, which further highlights the usefulness of LLMEmbed.
In summary, our contributions in this paper are listed as follows:
• To the best of our knowledge, we are the first to adapt lightweight LLM’s semantic embeddings for text classification. We propose a simple but effective paradigm, namely LLMEmbed, based on lightweight LLMs to address the text classification task. Our paradigm achieves SOTA results compared with promptbased methods with the same lightweight LLM backbone, and comparable performance to methods using large-scale LLMs.
• Our LLMEmbed paradigm directly constructs the mapping from input texts to output classification results. Therefore, there is no need for users to design sophisticated prompts to align inputs and outputs, i.e. there exists no hallucination. Compared to existing works, LLMEmbed is budget-saving since no extra token overhead is required.
• Our LLMEmbed is more flexible, scalable and efficient compared to prompt-based methods. LLMEmbed can combine the embeddings of lightweight LLMs with discriminative models (e.g. RoBERTa and BERT), or employ other representation learning methods to improve the classification performance. Moreover, LLMEmbed can perform classification in a high-speed parallel manner by feeding a large batch of text datapoints as input, whereas prompt-based paradigms cannot.

heading: 2 Related Work

heading: 2.1 Lightweight Large Language Models
text: Scaling up the parameters of a language model has demonstrated the effectiveness on various NLP tasks (Brown et al., 2020; Chowdhery et al., 2023). Especially, the breakthrough achieved by ChatGPT/GPT-4 (OpenAI, 2023) has made LLMs a promising approach to understanding language and generating contents. However, considering specific scenario customization of LLMs (Zhang et al., 2024), user data privacy security (Tan et al., 2024), and limited computational resource (Xia et al., 2023), open-source lightweight LLMs also aroused researchers’ interests, such as LLaMA27B (Touvron et al., 2023b), OPT-6.7B (Zhang et al., 2022a), Pythia-6.9B (Biderman et al., 2023).

heading: 2.2 Prompt-learning
text: Prompt works as the input to guide LLMs to generate contents satisfying users’ expectations. With the recent booming of generative LLMs, the prompt engineering (Liu et al., 2023), which focuses on designing effective prompts, has been a promising research topic. In-Context Learning (ICL) is a kind of typical prompt-learning which lists examples of the dataset as demonstrations to the LLM without adjusting the LLM’s network architecture (Brown et al., 2020; Schick and Schütze, 2021b). Besides the demonstrations, Instruction-Following introduces task-describing instructions with the desired responses into the prompt, and then fine-tunes
LLMs on the instructional data (Yi et al., 2019; Wei et al., 2021; Mishra et al., 2022; Ouyang et al., 2022; Wang et al., 2022b). Impersonation is another technique which make LLMs pretend to be a domain expert when answering a domain-specific question (Salewski et al., 2023).
From the viewpoint of human being’s thinking process when solving complicated tasks, Wei et al. proposes Chain-of-Thought (CoT) prompting to make LLMs decompose the problem into intermediate steps and solve each before giving the final answer (Wei et al., 2022b). Inspired by CoT, extensions such as zero-shot variants (Kojima et al., 2022) and Auto-CoT (Zhang et al., 2022b) have been introduced. Self-consistency samples multiple reasoning paths and selects the most consistent answer via a vote (Wang et al., 2022a). Least-toMost decomposes a given complex problem and solves subproblems iteratively to get the final answer (Zhou et al., 2022). ReAct allows the language model to interact with external environments such as Wikipedia to incorporate knowledge to inference (Yao et al., 2022). Self-refine makes the LLM generate an initial output and then iteratively provide feedback on the previous output, which is used to revise the output (Madaan et al., 2023). Tree of Thoughts (ToT) generalize CoT to maintain a tree of thoughts containing multiple different steps, enabling the LLM to self-evaluate the best way (Yao et al., 2023).

heading: 2.3 Prompt-based Text Classification
text: Based on the designed prompt, language models are guided to generate answers as the classification results. Schick et al. reformulated input texts into cloze-style phrases and generated the labels (Schick and Schütze, 2021a). Han et al. designed sub-prompts and composed them into final prompts based on logic rules, to guide the language model to generate results (Han et al., 2022). Liu et al. designed prompt based on the test sample’s semantically similar examples (Liu et al., 2022), and Shi et al. used k-nearest neighbor (KNN) algorithm to retrieve similar examples as demonstrations of ICL prompts (Shi et al., 2022).
Further combining ICL prompts, CoT and KNN algorithm, Sun et al. proposed a method called clue and reasoning prompting (CARP) (Sun et al., 2023) to guide GPT-3 through text classification and got the SOTA performances on 5 widely-used benchmarks. First, they used texts and labels of training dataset to generate clues and reasoning
of text classification by GPT-3. Next, they used PLM such as SimCSE (Gao et al., 2021) to compute the sentence level representation of training dataset, and then got the most semantically similar demonstrations to the test sample based on KNN algorithm. Finally, based on the reasoning process of these demonstrations, GPT-3 performs reasoning on the test sample and generates the classification result with many rounds of voting.

heading: 3 Methodology
text: As mentioned in the Section 1, improving the performance of locally deployed lightweight LLMs is urgently important, considering specific scenario customization of LLMs (Zhang et al., 2024), user data privacy security (Tan et al., 2024), and limited computational resource (Xia et al., 2023). However, the emergent abilities of lightweight LLMs fall behind online LLMs due to their far smaller parameter scale (Wei et al., 2022a), which subsequently limits the effectiveness of lightweight LLMs in prompt-based text classification methods. Therefore, we present the so-called LLMEmbed, which directly employ the lightweight LLMs to extract text embeddings, and improve the overall performance of text classification.
In this paper, we extract and fuse embeddings extracted by multiple backbones at different depths
to improve the robustness and generalization. Let us take f(·|m, dm) to denote the embedding extraction where m is the model and dm is the network depth. g(·|θg) refers to the classifier head. D = {xi, yi}Ni=1 is the dataset where xi is the datapoint, yi is the label and N is the total number of datapoints. D is divided into Dtrain and Dtest for training and evaluation.
During training, we first randomly sample a batch of datapoints {xi, yi}Nbi=1 ∈ Dtrain where Nb refers to the batchsize, then feed them into f(·|m, dm) to extract the semantic embeddings {ϕ(m,dm)i }Nbi=1.
ϕ (m,dm) i = f(xi|m, dm), (1)
where m ∈ {llama2, roberta, and bert} refers to the backbone selected, dm denotes the embedding is extracted from the last dm-th block of model m. Specifically, dm ∈ {1...5} for llama2 where the embedding of each block is the mean of this block’s all token embeddings, dm = 1 for roberta and bert where the final sentence semantic is used for embedding fusion.
Following we fuse all available embeddings via the operator v(·) to get the final semantic embedding ψi for the i-th datapoint.
ψi = v({ϕ(m,dm)i }), (2)
i , ϕ (bert,1) i are 4096-d, 1024-d, 1024-d, respectively.
The fusion operators investigated in this paper are listed in Table 1. It can be seen that average pooling, max pooling, co-occurrence pooling and concatenation are considered. Due to its complexity, here we take co-occurrence pooling as example to demonstrate the fusion process in detail.
Co-occurence pooling. Given the embedding set {ϕm,dmi }Nbi=1 for each datapoint xi, we first use linear mappings to project them into the same latent space and align them to the same length 1 × K, thus making ϕm,dmi ∈ R1×K ∀m, d. Then they are stacked to formulate the ϕ̂i ∈ RH×K where H = ∑ m dm. Following, we calculate their cooccurrence statistics ψi ∈ RK×K .
ψi = PN(ϕ̂i ⊗ ϕ̂ T i , σ), (3)
s. t. PN(X; σ) = 1− e−σX 1 + e−σX = tanh(2σX)
where PN(·) is a power normalization function used to balance the power distribution of cooccurrences, σ is the hyper-parameter to control the slope of PN(·) function, and the empirical range of value is [0.1, 0.5] in this paper.
Finally, all fused embeddings are fed into the classifier head to get the predictions ỹi = g(ψi|θg), and CrossEntropy loss is performed to update its
parameters θg.
argmin θg
L = − ∑
i
yi log ỹi. (4)
Overall, our LLMEmbed pipeline can be regarded as a typical transfer learning framework, which focuses on difference fusion strategies on LLM-based embeddings. As the embedding extraction process is fast, our model can be trained and adapted to the target domains with very high efficiency.

heading: 4 Experiment

heading: 4.1 Setups
text: We perform experiments on five widely-used text classification benchmarks, namely SST-2 1, MR2, AGNews3, R8 and R524. The detailed experimental settings follow CARP (Sun et al., 2023). SST-2 (Socher et al., 2013) is sampled from snippets of Rotten Tomatoes HTML files. The amounts of train set and test set are 67349 and 872 respectively, and the max length of words is 56.
1https://nlp.stanford.edu/sentiment/ 2http://www.cs.cornell.edu/people/pabo/movie-review-data/ 3http://groups.di.unipi.it/∼gulli/AG_corpus_of_news_articles.html 4https://www.cs.umb.edu/∼smimarog/textmining/datasets/
MR (Maas et al., 2011) contains movie reviews representing positive or negative sentiment. The corpus has 40000 training data and 10000 testing data, of which the max length of words is 2470.
AGNews (Zhang et al., 2015) consists of 4 types of news articles from the AG’s corpus. The dataset contains 120000 training and 7600 testing examples, and the max length of words is 177.
R8 and R52 are two subsections of Reuters collection, containing 8 and 52 classifications. The R8 dataset is composed of 5485 documents for training and 2189 documents for testing, of which the max length of words is 964. The R52 dataset is composed of 6532 training and 2568 testing documents, of which the max length of words is 1039.
Moreover, we solve these text classification tasks on Nvidia A100-40G platform, and employ LLaMA2-7B (Touvron et al., 2023b) as the lightweight LLM backbone throughout all of our main experiments. The batch size we set for each dataset is 1024. The classifier is trained for 100 epochs with initial learning rate 1el−4. Our LLMEmbed is compared with recent PLM- and prompt-based baselines w.r.t. performance, efficiency and budget.

heading: 4.2 Performance Analysis
text: For the conventional PLM method, we employ the widely used BERT and RoBERTa to extract embeddings and adapt them to the downstream text classification tasks. Though the PLM method demon-
strates satisfactory performance, there is still scope for enhancement, such as the RoBERTa hardly converges on R52 with only 42.17% accuracy.
For the prompt-based method, we initially leverage the Input-Output (IO) prompting (Brown et al., 2020), which provides input-output pairs as demonstrations to guide the LLM in generating results. Additionally, We implement the SOTA prompt method, CARP (Sun et al., 2023). Under the detailed guidance of CARP, the overall performance of prompt paradigm improve significantly (the average accuracy improve 21.63%). However, due to the far small parameter scale of LLaMA2(7B) compared to GPT-3(175B), the emergent abilities are somewhat limited. We note that the CARP also hallucinates, and if the prompts exceeds the LLaMA2’s input length limitation, CARP will probably generate irrelevant content.
For the LLMEmbed paradigm, we employed fusion strategies as mentioned in Table 1. As shown in Table 2, it is evident that average pooling of LLM’s embeddings, further concatenated with different models’ embeddings is most effective. Average pooling of LLM’s embeddings: We observe that the overall performance of average pooling surpasses max pooling by 0.1%∼0.2% and only-concatenating embeddings by 3%∼10%. The average pooling retains more information than max pooling. The only-concatenation degrades the performance due to the too large space of embeddings. Concatenating with different models’ embeddings: We find that after average pooling of LLM’s embeddings, concatenating it with different models’ embeddings will further improve the performance, e.g. Avg + Cat of three models outper-
forms two models. This is due to fusing embeddings of generative LLM and discriminative models can complement each other in semantic space. Co-occurrence pooling: This pooling itself may be not the most effective method, but it extract the high-order representation. We concatenate this representation further and get the SOTA performance based on the LLaMA2 backbone. The SOTA LLMEmbed outperforms CARP by 7.34%, 11.55%, 10.65%, 23.12% and 22.63% for SST2, MR, AGNews, R8 and R52, respectively.
Furthermore, we also compare local lightweight LLM-based LLMEmbed with online LLM-based CARP. Despite LLaMA2’s significant parameter scale disadvantage compared to GPT-3, the LLMEmbed achieves a comparable performance to the GPT-3 CARP by employing fusion strategies, even outperforms over SST-2, MR, AGNews, R8.

heading: 4.3 Efficiency
text: We have measured each process’s time cost of employing the prompt paradigm and LLMEmbed paradigm for text classification, and listed the results in Table 3. Overall, the proposed LLMEmbed paradigm is significantly more efficient than the prompt paradigm.
As shown in Figure 3, the downstream model reaches convergence at around 50 epochs. For SST-2, LLMEmbed’s time cost is only 0.46% (00:22:11/79:31:09) of the prompt-based paradigm. Similarly, the ratio of time cost is 3.51% for MR, 0.48% for AGNews, 0.66% for R8 and 0.70% for R52.
The reasons for the surge in LLMEmbed-based efficiency are: 1) The lightweight LLM only needs
to extract the representation of the input texts, without spending time generating answers. 2) LLMEmbed only processes the original input texts without any additional sophisticated prompt words. So the length of input is much smaller than the promptbased method, resulting in much less computation. 3) LLMEmbed can conduct the text classification in a parallel manner by feeding a batch of texts to the lightweight LLM, whereas prompt-based paradigm can’t. We note that if feed a batch of prompts to guide the lightweight LLM to generate results, prompts of this batch will effect each other, leading to a low performance.
As shown in Table 3, LLMEmbed costs the most time when solving the MR task. Since the length of each text in MR is generally much longer than other datasets, solving MR task requires larger GPU memory, resulting in limited parallel processing. So, the lightweight LLM spends the most time extracting the representation of MR.

heading: 4.4 Budget of Users
text: We first compare the electricity consumption between prompt paradigm and LLMEmbed paradigm. For the prompt-based CARP, the power of generating clues and reasoning is about 170W , KNN with representation extracting is about 250W , and generating result is about 190W . So, referring to the time cost listed in Table 3, the total electricity consumption of SST-2, MR, AGNews, R8 and R52 is 13.77kWh, 33.65kWh, 41.42kWh, 7.44kWh and 8.37kWh respectively.
For LLMEmbed paradigm, the power of representation extracting is about 240W . When training or testing the downstream model, the power is 35W ∼ 55W , which is almost equivalent to the
power of a GPU booting up without running any program. We take the average value 45W as the power of these processes. As shown in Figure 3, the downstream model converges at around 50 epochs. Combined with the time cost in Table 3, the total electricity consumption of SST-2, MR, AGNews, R8 and R52 is 0.06kWh, 1.51kWh, 0.22kWh, 0.06kWh and 0.07kWh respectively.
The detailed results of electricity consumption on five benchmarks have been listed in Table 4. For SST-2, LLMEmbed’s electricity consumption is merely 0.44% (0.06kWh/13.77kWh) of the prompt-based paradigm. Similarly, the ratio of electricity consumption is 4.49% for MR, 0.53% for AGNews, 0.81% for R8 and 0.84% for R52. Therefore, it can be seen that LLMEmbed paradigm is greatly more energy-efficient than prompt paradigms.
Following we compare the budget of users between the local LLMEmbed and the online promptbased LLMs,(e.g. GPT5, as CARP does). The pricing of GPT is about $0.002 per 1k tokens6. We tokenize all the input and output of CARP, and calculate the total sum of these token-based budget. For the local LLMEmbed, its budget is purely decided by the electricity consumption calculated above. Taking the electricity tariffs of Beijing which is $0.065/kWh as an example, we can easily work out the realistic electricity bill.
All the results have been listed in Table 5. Overall, the budget of local LLMEmbed is much smaller than the online prompt-based LLM. The ratio of budget is 0.01% for SST-2, 0.02% for MR, 0.01% for AGNews, 0.01% for R8 and 0.005% for R52.
The LLMEmbed paradigm extracts the representations from the original input text, thus obtaining the semantic space through the powerful language comprehension capabilities of LLMs. In this rich semantic space, we can improve the performance of LLMEmbed by employing various semantic representation optimization strategies in text classification. Furthermore, we can adapt LLMEmbed to many other downstream NLP tasks by constructing a proper mapping from the semantic space to the output. Our LLMEmbed is expected to be beneficial for generative LLMs in handling other tasks such as information extraction, script event prediction.
5InstructGPT-3 (text-davinci-003, 175B) 6https://openai.com/pricing

heading: 5 Conclusion
text: In this paper, we propose a concise and effective LLM-based paradigm, namely LLMEmbed, to address the text classification task. This novel LLMEmbed paradigm achieves the state-of-theart performance when solving text classification with lightweight LLMs, even comparable to the performance of LLMs (e.g. GPT-3) with sophisticated prompt-based strategies. The LLMEmbed paradigm is flexible and scalable in which the text embeddings obtained from different lightweight LLMs can be fused to improve the overall performance. Moreover, LLMEmbed is far more efficient and budget saving than the prompt-based paradigms. Lastly, we expect that our novel LLMEmbed paradigm can be beneficial for using generative LLMs to handle other down-stream discriminative tasks.
Limitations
Despite the promising results, LLMEmbed still suffers the following limitations: 1) LLMEmbed needs optimization to further improve the performance in long-text classification tasks if no finetuning is performed on the LLM; 2) The explainability of LLMEmbed may be weaker than the prompt-based methods. Our future works will combine the chain-of-thoughts prompts with LLMEmbed to address this issue.

heading: Acknowledgements
text: This research was supported by the National Nature Science Foundation of China (No. 62106282) and Beijing Nova Program (No. 20220484139).

heading: Future Work
text: Our future works will combine the chain-of-thoughts prompts with LLMEmbed to address this issue