LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification

Authors: zhen  qin
junru  wu
jiaming  shen
tianqi  liu
xuanhui  wang
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 17:19:16

Abstract: We introduce LAMPO, a novel paradigm that leverages Large Language Models (LLMs) for solving few-shot multi-class ordinal classification tasks. Unlike conventional methods, which concatenate all demonstration examples with the test instance and prompt LLMs to produce the pointwise prediction, our framework uses the LLM as a preference machine that makes a relative comparative decision between the test instance and each demonstration. A self-supervised method is then introduced to aggregate these binary comparisons into the final ordinal decision. LAMPO addresses several limitations inherent in previous methods, including context length constraints, ordering biases, and challenges associated with absolute point-wise estimation. Extensive experiments on seven public datasets demonstrate LAMPO's remarkably competitive performance across a diverse spectrum of applications (e.g., movie review analysis and hate speech detection). Notably, in certain applications, the improvement can be substantial, exceeding 20% in an absolute term. Moreover, we believe LAMPO represents an interesting addition to the non-parametric application layered on top of LLMs, as it supports black-box LLMs without necessitating the outputting of LLM's internal states (e.g., embeddings), as seen in previous approaches.

Main paper: 
 introduction: 
 In-Context Learning (ICL) with few-shot demonstrations, also known as few-shot prompting, is a prominent approach for multi-class classification using Large Language Models (LLMs) (Brown et al., 2020) . ICL's remarkable performance and training-free nature, combined with off-the-shelf LLMs (e.g., GPT-4 (OpenAI, 2023) and PaLM2 (Google et al., 2023) ), has significantly reduced the model adaption costs for new tasks and facilitated the concept of language-model-as-a-service (Sun et al., 2022) . Despite these benefits, recent studies indicate that few-shot prompting can sometimes produce unreliable predictions and maybe not always outperform zero-shot prompting (Zhang et al., 2023) . To understand these phenomena, extensive research has been conducted, leading to the following insightful observations:Restrictive Context Length Limit. Despite extensive research efforts (Ratner et al., 2023; Press et al., 2021) , the input context length limitation remains a major challenge in applying LLMs in many practical scenarios, especially given limited access to advanced commercial systems such as Reid et al. (2024) . This constraint limits the number of demonstrations that can be included in a prompt, adversely affecting LLM performance, especially in challenging tasks with a large label space or long example texts. Also note that supporting long contexts does not mean the models necessarily perform well (Li et al., 2024) due to issues such as ordering bias, discussed next. Destructive Ordering Bias. Recent studies (Liu et al., 2023; Lu et al., 2022) have identified a pronounced ordering bias in input prompts for LLMs. For standard few-shot multi-class classification tasks, the arrangement of demonstrations within the prompt can drastically affect the LLM performance, ranging from near chance to state-of-the-art levels. Addressing this issue is inherently complex due to the exponentially increasing combinations of demonstration orders as the number of examples grows.Challenges of Pointwise Estimation. In many applications, pointwise estimation is notably more complex compared to comparative estimation. For instance, resolving pointwise relevance estimation addresses relevance ranking, but not conversely, a fundamental observation underpinning the learning-to-rank domain (Liu, 2009) . Similarly, in fine-grained sentiment classification, LLMs may struggle to differentiate subtle sentiment nuances, such as between "very positive" and "positive", especially when given limited demonstrations in unfamiliar domains, or facing instructions not covered in their pre-training or instruction fine-tuning phases.This study focuses on ordinal classification (Gutiérrez et al., 2016) , an important subset of multi-class classification problem where the labels possess a natural order. Ordinal classification has extensive applications such as customer reviews analysis (Zhang et al., 2022) , relevance rating in web search (Thomas et al., 2023) , and opinion monitoring in social media (Barbieri et al., 2020) . We propose to use instruction-following LLMs as preference machines to make pairwise comparisons between the test instance and each demonstration. In this paradigm, LLMs make comparative, rather than absolute, judgments in the ordinal label space, and we propose an unsupervised aggregation method to convert these comparison outcomes into the final ordinal prediction.We present LAMPO, a novel framework that leverages LArge language Models as Preference machines for Ordinal classification. LAMPO implicitly addresses the above discussed ICL methods' limitations from three perspectives. First, it avoids packing all demonstrations in a single prompt and thus enables LLMs to leverage an arbitrary number of demonstrations. Second, it includes only two examples (one demonstration and one test instance) in each prompt and treats all prompts independently during the aggregation stage, which substantially mitigates the demonstration ordering bias in LLMs. Third, LAMPO employs the LLM as a preference machine and thus transforms the difficult pointwise estimation problem into a more manageable pairwise comparison problem.The advantages of LAMPO are demonstrated by its competitive performance across seven diverse datasets, compatibility with both open-sourced (i.e., Flan-T5) and black-box LLMs (i.e., PaLM2), and reliance solely on binary generative decisions. This broad versatility makes it adaptable to various black-box LLMs, including those API-based LLMs with restricted access to internal states (e.g., embeddings) (Zhao et al., 2021) . However, it is essential to acknowledge that our paradigm entails an increased number of LLM API calls (though with shorter sequences) and is only applicable to instruction-following LLMs, which we elaborate upon in Section 6.Our contributions are summarized as follows:• We introduce the LAMPO framework for the important ordinal classification problem.This framework effectively addresses several issues in the existing ICL paradigms. • We present a pipeline that operates without the need for internal logits or embeddings from LLMs, and does not rely on an additional development dataset. • We conduct extensive experiments on seven challenging datasets, covering the analysis of positivity, aspect-based sentiment, hatefulness, irony, and offensiveness of texts. LAMPO consistently demonstrates competitive performance when employed with various LLMs, where the improvements over state-of-the-art ICL approaches can be substantial. 
preliminary: 
 In this section, we first introduce the original ICL method for few-shot ordinal classification with basic notations. Then, we discuss two representative ICL methods, which will shed lights on the difference between our work and the existing literature.ICL is a paradigm that allows LLMs to learn tasks given only a few demonstration examples.The standard m-way multi-class classification aims to assign the test input text x to one candidate answer y in the label space Y = {Y 0 , ..., Y m-1 }. This paper focuses on ordinal classification where there exists a natural order of the labels. Take the sentiment analysis task as an example, Y 0 denotes "very negative" and Y m-1 denotes "very positive".A large language model M takes the candidate answer with the maximum score as the prediction conditioning a task instruction I and a demonstration set C which includes k demonstration examples for each class1 . Namely, we have C = {(x i , y i )| mk i=1 }, where (x i , y i ) denotes one demonstration and there are a total of mk ones. The likelihood of a candidate answer Y j could be represented by a scoring function f of the whole input sequence with the LLM M: P(Y j |x) f M (Y j , I, C, x).(1)The final predicted label ŷ is the candidate answer with the highest probability:EQUATIONThe scoring function f estimates how possible the current answer is given the demonstration and the query text. For black-box generation-only LLMs, the most probable label string is expected to be generated directly.Existing work (Zhao et al., 2021; Fei et al., 2023) found that ICL is sensitive to the quality and ordering of demonstrations, leading to the calibration issue where the model biases towards a certain answer regardless of the input x. To mitigate this issue, contextual calibration (CC) (Zhao et al., 2021) first estimates the bias towards each answer by asking for the prediction probabilities pc f of a content-free (cf) input such as "N/A", and then uses them to correct the prediction probabilities px ([P(Y 1 |x), ..., P(Y m |x)] T ) asEQUATIONThere are two major limitations of CC. First, CC requires LLM's internal states (i.e., probability logits) thus is not applicable to many black-box LLMs. Second, CC implicitly assumes the context-free input "N/A" can catch the deviation from a uniform prediction of the labels. This assumption might be less problematic in simpler classification scenarios, like binary sentiment analysis (positive vs negative). However, it does not hold for many real-world applications. For instance, in fine-grained classification tasks that include a "neutral" label, the "N/A" input likely has a non-trivial probability mass, leading to questionable calibration. Similarly, "N/A" assigns a probability mass to labels such as "non-hate" in hatefulness detection, "non-offensive" in offensiveness detection, etc. Thus, the practical utility of CC in such complex scenarios is limited.Lu et al. ( 2022) studied the impact of demonstration ordering given a fixed demonstration set and presented a GlobalE method to identify prompts of specific demonstration orderings that prevent the extremely unbalanced prediction issue2 . Specifically, GlobalE first generates multiple sets of candidate contexts by sampling different orderings of the demonstrations. Then, for each candidate contexts set C m , it constructs a probing set by sampling from the LLM:(x i , y i ) ∼ P M (•|C m ).See more details of probing set construction in Appendix B. After that, GlobalE computes the predicted label ŷi for sampled data point under each C m as follows: ŷi = arg maxEQUATIONFinally, it ranks all candidate contexts based on the category label entropy of the predictions of the probing set, and uses the top-ranked context for actual inference. This method is which one is more positive?"X i X or LLM or X X F(x,x i ) = 1 F(x,x i ) = 0 X i X F(x,x i ) = -1 s(F(x,x i ), l(y i ))s(F(x,x 1 ), l(y 1 )) s(F(x,x mk ), l(y mk ))....X i X iYm-1: Very Postive effective under the mild assumption that the probing set should have a uniform distribution over the label space, since they were sampled using sampled C m s, which should not have strong biases towards any labels globally.GlobalE does not depend on LLM internal states and generally outperforms basic ICL in our experiments. However, it can still be limited by restrictive context length and difficulties of pointwise estimation. Also, given the combinatorial nature of possible orderings, candidate contexts C m can only be sampled and are unlikely optimal. 
scope: 
 To address the ambiguity surrounding the term "ICL" (Dong et al., 2022) , it is necessary to delineate the scope of this research. We focus on the practical "true" k-shot learning (Perez et al., 2021) for multi-class classification with off-the-shelf LLMs, where at most k demonstrations are available for each class. This is distinct from previous studies that can retrieve a fixed number of demonstrations from a potentially large labeled dataset (Liu et al., 2021) .Furthermore, this work does not assume the presence of a labeled development set for tuning hyperparameters, as such an assumption would violate the aforementioned setting. This better aligns with the standard setting from the original ICL work (Brown et al., 2020) and several key subsequent ICL research (Zhao et al., 2021; Lu et al., 2022) discussed above.The comparison with methods involving test example dependent demonstration retrieval and LLM fine-tuning is beyond the scope of this paper. 
lampo: 
 We now describe our new paradigm LAMPO for few-shot ordinal classification, with the overall diagram shown in Figure 1 . LAMPO consists of two steps: (1) a scoring step that uses the LLM to compute the comparative score between each test instance with each demonstration, and (2) a decision making step that converts those scores into ordinal labels by using offline learned thresholds without an additional labeled development set.We score each test instance x as follows:EQUATIONThe fundamental computation unit of LAMPO is denoted as F(x, x i ), which involves LLM calls to compare input x with each demonstration x i within the ordinal label space. For example, in hatefulness detection, the prompt can be "Given Passage A: {x} and Passage B:{x i }, which one is more hateful?". For the exact prompts used in each dataset, please refer to Appendix C.When computing F(x, x i ), we can significantly mitigate the position bias by making two LLM calls with swapped orderings of x and x i in the prompts, which is not feasible for existing ICL methods with combinatorial orderings. The resulting value of F(x, x i ) can assume one of three outcomes: F(x, x i ) = 1 indicates x is preferred (both LLM calls prefer x), F(x, x i ) = -1 signifies a preference for x i (both LLM calls prefer x i ), and F(x, x i ) = 0 denotes a tie (the two LLM calls yield conflicting or inconclusive results).The local score s for comparing x and x i is derived from the comparison result F(x, x i ) and label y i . In line with established practices in ordinal classification literature, we use an integer number to represent each label Y, namely, l(Y j ) = j. This approach simplifies our scoring process as follows:EQUATIONFor example, if x wins over x i , it gets a local score of l(y i ) plus 1 point. These local scores are added together as the final score S(x) following Eq. 5.Given the score S(x), we need to convert it back into the ordinal label space, which can be reduced to the problem of identifying m -1 thresholds T = {T 1 , ..., T m-1 }, so that:EQUATIONNote that if there exists a labeled development set, the search of thresholds would be easy. However, it will violate the "true" few-shot setting. We thus propose the following strategies to find T:Expected Thresholds. Intuitively, imagine we have a test example with label value Y j , we know the scores of comparing it with each demonstration from Eq. 6 if everything is noise-free. We can easily derive the expected score of an example of any label value Y j :EQUATIONThis computation does not require any LLM calls since we just derive the expected score given the demonstration labels. Then the expected threshold T j is simply (S j + S j-1 )/2, and the other thresholds can be derived similarly. However, this is under the assumptions that the demonstration labels are noise free and pairwise comparisons are unbiased globally, which is hardly true in practice. Still, we can view this as a global prior of the thresholds.Self-supervised Thresholds. Systematic bias can arise from both the demonstrations and LLM itself and the expected thresholds may not be optimal. We are thus interested in decision thresholds that are adaptive to the current demonstrations and LLM. Inspired by Lu et al. (2022) , we leverage the probing set to rank candidate thresholds based on the global entropy of predicted labels. While we believe applying the probing set to our problem is a contribution, the probing set construction method itself is not our contribution, which is deferred to Appendix B. Different from Lu et al. (2022) where their candidate space is combinatorial in terms of the number of demonstrations, our search space is combinatorial in terms of the number of integer thresholds, which is usually small. Note that the LLM calls that compare each example in the probing set and demonstrations are only required once before the threshold search. The major caveat of this self-supervised approach is potentially large variance due to the limited number of demonstrations. We can treat these "self-supervised thresholds" as the empirical observation.The Mixture. Given the pros and cons of the above two methods, we propose to use a mixture of above two strategies, which can be treated as a simple combination of prior and empirical thresholds. Since we do not assume a development set, we simply use the average of their corresponding thresholds without tuning the weights for all experiments.Let's consider a 5-shot 3-way ordinal classification with labels {negative (0), neutral (1), positive (2)}. The expected thresholds would be {10, 20}, which can be derived as follows: a negative example is expected to have a score 5 in a noise-free setting (it will tie with the 5 negative demonstrations, and lose to the other 5 neutral and 5 positive demonstrations, thus 0 + 0 + 5), a neutral example with score 15(5 + 5 + 5), and a positive example with score 25(5 + 10 + 10), thus the expected thresholds are 10 = (5 + 15)/2 and 20 = (15 + 25)/2.Furthermore, let's assume the self-supervised thresholds are {12, 21}, then our final thresholds would be {11, 20.5}. Now given a test instance x, we do comparisons using LLM with each of the 15 demonstrations and get a score S(x) = 16 according to Eq. 5, then the final predicted label of x will be "neutral". 
datasets: 
 We use 7 challenging ordinal classification datasets, i.e., Twitter, SST-5, Hate, Yelp-5, Lap14, Offensive, and Irony. We follow the publicly hosted setting3 where each dataset provides 3 seeds of demonstrations in each configuration, i.e., 5-shot and 10-shot, which are used to compute mean and standard deviation of the concerned metrics on the test set. The same publicly hosted setting is used for all compared methods for a fair and reproducible comparison. The datasets are summarized in Table 3 in Appendix A, including their tasks, metrics, and label spaces.For few-shot LLM-based methods, we compare LAMPO with ICL, CC, and GlobalE, which were discussed in Section 2. For few-shot LLM-based methods, we use the black-box PaLM2-S model (text-Bison4 ) and the white-box Flan-T5-XXL (Chung et al., 2022) . We use the more general generation mode, that does not request any internal states of LLMs, for ICL, GlobalE, and LAMPO. We use the scoring mode to get the required internal logit values of label texts with Flan-T5-XXL for CC.Note that due to potential data leakage issues of modern LLMs (Zhou et al., 2023) (see more discussions in Section 6.4), we will focus on comparing different methods under the same LLM, and only briefly compare trends of performance across different LLMs.This section describes our experimental results, which are shown in Table 1 and Table 2 . We discuss datasets in groups due to their diverse settings and behaviors.Results on Twitter. LAMPO exhibits the best performance on PaLM2-S and Flan-T5-XXL. Contrastingly, CC is ineffective due to the presence of a "neutral" class that violates its calibration assumptions. Furthermore, it's worth noting that LAMPO is the sole method capable of handling 10-shot scenarios on this dataset using the concerned popular LLMs. The slightly lower performance in this 10-shot setting compared to the 5-shot setting across both LLMs highlights the importance of the demonstrations' quality, as other factors, including ordering bias, have been largely mitigated. Results on SST-5. LAMPO works well on this highly challenging dataset with 5-level labels, outperforming ICL and GlobalE with both LLMs. 10-shot also consistently outperforms 5-shot with LAMPO, while it is infeasible to perform 10-shot with other methods. CC with 5-shot performs well with Flan-T5-XXL, as it avoids generation errors on this challenging (for Flan-T5-XXL) dataset. See more discussions in the summary of results below.Results on Yelp-5. Results on Yelp-5 mainly shows the capability of LAMPO -no other methods are feasible given the long example texts. Also, 10-shot works better than 5-shot on both LLMs for LAMPO. LAMPO allows us to find that there still exists a gap between LAMPO and supervised methods, so the community is suggested to focus on challenging datasets like Yelp-5 for future work.Results on Lap14. LAMPO works well on this dataset, with the best performance on both PaLM2-S and Flan-T5-XXL, showing LAMPO is compatible with the challenging aspect-based sentiment analysis task. CC works poorly since there is a "neutral" class.We get very strong results of LAMPO on PaLM2-S, where the absolute performance gain can be larger than 20%. LAMPO also generates the best performance with Flan-T5-XXL, though the gap is smaller. We hypothesize this is because PaLM2-S was not exposed to similar datasets, especially with regard to their label space, during its pre-training and instruction fine-tuning. In contrast, LAMPO handles a more general "hateful" concept that PaLM2-S is familiar with, regardless of whether it has encountered the exact label space before or not. CC again works poorly because the probability mass will be assigned to "non-hate" for the "N/A" input.Results on Offensive and Irony. LAMPO is competitive and robust for these two datasets on both LLMs. One noticeable benefit of LAMPO is its robustness across different shotsthe performance of other methods on Offensive actually drops with 10 shots, but LAMPO is robust and generally improves performance with more shots. CC does not work as the probability mass will be concentrated on "non-offensive" and "non irony".On the seven tasks across a broad range of domains we find that:(1) LAMPO is very competitive and works the best in most cases. Importantly, LAMPO is the most robust method, while other methods have their respective failure patterns. In general, LAMPO shows better comparative performance with respect to other methods on PaLM2-S than Flan-T5-XXL, indicating that more advanced LLMs tend to do comparisons better across various domains.(2) LAMPO enables popular LLMs the capability to perform competitive ordinal classification in various settings, including black-box or white-box LLMs, number of shots, and on tasks where the LLM may not have strong prior knowledge of the label space.(3) The calibration assumption of CC is too strong and this impairs its robustness. Despite its strong performance on SST-5 with Flan-T5-XXL, it can not be used with black-box LLMs and it hurts performance on most datasets due to violation of its calibration assumption.(4) GlobalE outperforms the ICL baseline in most cases, but it is generally less effective than LAMPO, possibly because the ordering bias is intrinsically difficult to address due to the combinatorial nature of possible orderings.6 Discussion and Limitation 6.1 When do we expect LAMPO to be most valuable?LAMPO shows competitive performance across different domains. One most prominent improvement LAMPO got is on the hatefulness detection dataset, where we hypothesize that PaLM2-S does not have a strong prior understanding of the task, especially concerning the "non-hate" label. However, LAMPO mainly leverages more general knowledge such as "which passage is more hateful?" and does not use the actual label text until the very last decision making step. Thus, we believe LAMPO is particularly valuable when the LLM lacks a strong prior understanding of the task's label space. 
restriction to instruction-following llms: 
 Earlier research before the ubiquity of instruction-following LLMs on ICL mainly depends on text continuation. On the other hand, as hinted by the prompts in Appendix C, the comparison prompts used in LAMPO are instruction-following and are likely only applicable on instruction-following LLMs. This is one limitation of our framework. However, we argue the limitation may not be significant as instruction-following LLMs are becoming the norm.We acknowledge that LAMPO does involve an increased number of LLM API calls compared to standard ICL methods. However, we note that: (1) The number of calls only linearly scales with the number of demonstrations, which is usually not big in the concerned "few-shot" setting. On the other hand, LAMPO's new capability allows for the potential utilization of an unlimited number of demonstrations, making the pursuit of more efficient methods a promising future research direction. (2) Each prompt is shorter, so each API call is faster and less expensive than traditional ICL. (3) All comparison API calls can be paralleled, thus the latency can be lower than traditional ICL, given sufficient parallelism.It is possible that certain LLMs saw some datasets during their pre-training or instruction fine-tuning, which is a major concern of fair evaluation in the era of LLMs (Zhou et al., 2023) . We mainly focused on comparing different methods using the same LLMs so the comparisons are fair. In fact, as most existing LLM pre-training and instruction tuning methods adopt the pointwise strategy, a potential data leakage will benefit them more, compared to our pairwise framework. Thus, the improvements achieved by LAMPO over these baselines further demonstrate its effectiveness.As noted, all datasets used in this paper are publicly available, and we did not change any of their demonstrations, labels, or any other content. The exact prompt templates we used are shown in Appendix C. 
related work: 
 In-Context Learning. We have discussed several popular ICL-based methods in Section 2, which is a prominent family of approaches that leverages LLM for few-shot multi-class classification. Another emerging paradigm, which has gained popularity recently, involves retrieving distinct demonstrations for each test instance (Rubin et al., 2021) . These demonstrations are sourced from a potentially vast annotated demonstration pool, which is very different from the "true" few-shot setting studied in this work.Comparisons using LLMs. We note that the "pairwise" or "comparative" paradigm is a classic concept in various domains. LLMs can serve as evaluators to compare preferences of generative model's outputs (Kocmi & Federmann, 2023; Vu et al., 2024; Yan et al., 2023) , such as to determine which summary is better in text summarization, with many important applications such as language model alignment (Liu et al., 2024) . In contrast, our approach compares inputs to derive an output. For web search ranking, given a query, pairs of candidate documents can be compared using LLMs in terms of their relevance to the query (Qin et al., 2024) . This task significantly differs from ours, and it involves a quadratic increase in API calls with the number of documents under each query. In summary, existing work only share the very general idea of doing pairwise comparisons with very different problem setups where natural pairs are readily available. In contrast, our work pioneers the exploration of LLMs as preference machines for the important few-shot ordinal classification task and demonstrates LLMs' comparative capability in broader dimensions (i.e., traditional sentiment, hatefulness, offensiveness, irony, and aspect-based sentiment)."Non-parametric" application of LLMs. Very recently, Xu et al. (2023) have shown a "nearest neigbor" style application of LLMs. We posit that our paradigm offers a compelling addition to this non-parametric perspective of LLM utilization, where existing research (Xu et al., 2023) hinges upon extracting embeddings that are not generally accessible, and neglects to explore the preference machine perspective of LLMs. We show the descriptions of each dataset used in our evaluation in Table 3 . We provide more details on the construction and use of the "probing set", with most contents below adopted from Lu et al. (2022) . The key idea is to build a collection of examples that has a roughly uniform label distribution. Then given a candidate in the search space (a set of thresholds in LAMPO and a specific ordering of demonstrations in GlobalE), the predictions of the examples can be generated, and the best candidate should produce label predictions closest to the uniform distribution, measured by entropy.To construct such a "probing set" without access to any additional data, Lu et al. (2022) proposes to directly sample from an LLM. Concretely, given a set of training / demonstration examples S = (x i , y i ), i = 1, ..., n, we define a transformation T, mapping each example into natural language space. We use a simple transformation function T such that T(x i , y i ) = input : x i type : y i . This transforms each example into a standard format sentence, which linearises each element in the set into the natural language space defined as S = {t i }, i = 1, ..., n.We then define a full permutation function group of n training examples, F = { f m }, m = 1, ..., n!, where each function f m takes S as input and outputs C m : the concatenation of a unique permutation. For each prompt candidate C m , we then sample from the large language model M to obtain the probing sequence g m ∼ P M (•|C m ). We stop decoding from the language model upon generating the special end-of-sentence token defined by a template, or reach the generation length limit. The probing set construction method is illustrated in Figure 2 , where the objective is to generate a probing set that shares a similar distribution to the training examples.We run this sampling process for sampled prompt ordering permutations and extract probing examples from them, then gather extracted examples together to form the probing set D. Although the probing set contains predicted label for each sentence, there is no guarantee on the validity of these labels. Therefore, we discard them from the probing set as we are only interested in sampling probes from the language model corresponding to the input distribution.There's ambiguity for "k-shot" in the literature. In this paper "k-shot" refers to k demonstrations per class, as inLu et al. (2022);Zhang et al. (2023).Lu et al. (2022) also proposed a LocalE method that depends on internal states of LLMs and under-performs GlobalE in most cases. Thus, we omit this method in the paper.https://github.com/DAMO-NLP-SG/LLM-Sentiment/tree/master/datahttps://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/text 
 
 Appendix:  


Decision: The paper introduces a method to use LLMs for solving few-shot multi-class ordinal classification tasks. Compared with conventional ICL, the proposed method aggregates the relative comparative decisions between the test instance and each demonstration. The cost becomes larger while it overcomes some limitations such as ordering biases. One out of four reviewers tends to reject the submission because of the lack of comparisons with more techniques for aggregating the results of the pairwise comparisons, and LLMs with larger context window. The experimental settings can also be improved as suggested.

Reviews: review: paper_summary: paper_summary: This article presents the LAMPO approach to performing sentiment analysis. 
This approach is "LLM free" which means one can use any LLM they want since the strategy focuses on a way to compare scores extracted in two different ways, then the approach compares the scores and performs a decision process to extract the sentiment level.
strength_weakness: strength_weakness: One can regret the approache is not compared to fine-tuned approach in order to have a benchmark.
questions: questions: What are the performances of classical approaches against yours?
You discuss the ecological impact of your model. Could you elaborate in terms of energy and CO2 consumption?
paper_summary: paper_summary: This paper proposes LAMPO, a method that uses LLMs to conduct few-shot ordinal classification by prompting the LLM to perform pairwise comparisons between the test example to be labeled and the few-shot examples available. They explore an array of ordinal classification tasks (e.g., SST, Yelp, etc.) using two LLMs (Palm2-Bison, Flan-T5-XXL).

Overall, the idea itself is simple and the results on the surface are promising, but there are aspects of the method and experimental methodology that could be further improved.
strength_weakness: strength_weakness: **The paper could explore more techniques for aggregating the results of the pairwise comparisons.** Currently, the pairwise comparisons are aggregated into a score number, and two methods are explored for obtaining class thresholds. However, the paper could be strengthened by exploring other ways of performing this aggregation, especially given the unique nature of pairwise data (e.g., one could cast this as an optimization problem, where the label that violates the smallest number of pairwise comparisons is set as the predicted class).

**One could argue that the central issue this paper tries to address—how to perform ICL when you are severely constrained in terms of input length—is less and less of a problem as newer LLMs support ever-larger context windows.** Repeating these experiments with more recent LLMs than the ones in the paper (e.g., Gemini, GPT-4, Llama-3, etc.) and showing the benefits remain when longer context windows are available would strengthen the paper’s central point.

**There are aspects of the experimental methodology that could be improved.**

- It is not clear whether it is fair to the baselines that if they cannot fit a 10-shot prompt in their context window, then the result should be “NA”; in this case, a practitioner would likely pare down the number of examples until it fits in that context window (e.g., using 7-shot, or a variable number by class). This is especially true of the Yelp-5 experiment: could you not do at least 1-shot in this case, for the baselines?
- In some cases the error bars around the results suggest that LAMPO might not be better than the baselines (e.g., PaLM SST-5, Lap14). It would be appropriate to discuss this, and you should highlight all models in a row that perform statistically equivalently.
questions: questions: Why do some of the baselines have such high variance in their results, e.g., 10-shot Offensive for Flan-T5? Is this because of some really poor runs?
paper_summary: paper_summary: This work studies multi-class ordinal classification (i.e., classification where there is an inherent ordering between the classes) using LLM prompting with demonstrations (i.e., in-context learning). Simply concatenating demonstrations for in-context learning has challenges, including (1) context length limitation, (2) ordering bias, and (3) pointwise estimation. The paper proposes LAMPO to address these challenges. LAMPO can be viewed as a prompting technique where each demonstration is compared against the test example separately, and the LLM judge is only required to make pairwise comparisons. Given $m$ demonstrations, the LLM judge performs $m$ comparisons (or $2m$ for symmetry) and then the outcomes are aggregated to obtain a final predicted class. The work considers a novel method to map from combined output into the final class. Experiments are conducted on a range of standard datasets (e.g., Twitter, SST-5, Yelp-5, etc), and the results demonstrate the effectiveness of LAMPO.
strength_weakness: strength_weakness: 1. The method requires many calls to LLMs (e.g., $2m$ call for each test example with $m$ demonstration).
2. The method could become less relevant as the context length of LLMs are becoming longer.
questions: questions: Have you considered applying LAMPO (in your few-shot learning manner) to evaluate open-ended text generation tasks?
paper_summary: paper_summary: This paper introduces LAMPO, a new method for ordinal classification (e.g., sentiment analysis) using LLMs. The main idea is to use the LLM as a preference machine, making pairwise comparisons between a test instance with each demonstration example. This approach addresses limitations of prior ICL methods, such as context length constraints and ordering biases. LAMPO aggregates these pairwise comparisons into a final ordinal prediction using a self-supervised method. Experiments on several datasets show LAMPO's good performance with improvement over state-of-the-art ICL approaches.
strength_weakness: strength_weakness: - The blackbox LLM model used in the paper are not SOTA ones, making it less convincing on whether the proposed approach can still outperform other ICL approaches.

- The paper acknowledges that LAMPO requires more LLM API calls than traditional ICL. While it argues that the calls are shorter and can be parallelized, the overall cost implications is still a major concern given the pair-wise comparison.

- The evaluation is primarily focused on empirical performance. More in-depth analysis is recommended to understand why LAMPO works well and under what conditions it might fail.

Comments: title: response
Feedbacks
We appreciate your constructive feedback
Rebuttal response
comment: Although my assessment remains the same after reviewing it, I appreciate you taking the time to respond.
Thank you for your response. I've read it and my assessment remains the same.
Hello,
thanks for your answers and precisions.
Best,
We again thank the reviewer for their constructive feedback to help us improving the paper.

Regarding the last point: We appreciate your explanation and agree it is a great suggestion. Yes, 1-shot (5+1 examples in the input prompt) is the headroom for Yelp-5 on the LLMs we use. Note that we follow the publicly hosted benchmark which provides up to 10-shot for each dataset. We will try to add ablation studies to go beyond 10-shots and update here or in the next version, which needs nontrivial efforts including gathering and sampling additional training demonstrations and performing inference. We feel the benefit of LAMPO may be more prominent as position bias can be more difficult to mitigate (due to its combinatorial nature) for existing baselines such as GlobalE. We really appreciate the suggestion.
Thank you for your response!

RE: Further exploration. Thank you for clarifying this point; however, I still think the paper in its current form could benefit from further exploration (as opposed to deferring it as future work).

RE: Context length. I definitely agree that there may be other aspects at play here, and the paper could definitely be strengthened by exploring these and showing that such positional biases have an impact on this task (or citing relevant work that explores this).

RE: 1-shot Yelp. Thank you for the additional experiments!
- Following up on this, is 1-shot the most that you can do given context length limits of the LLMs?
- To clarify, my point was broader that just Yelp: It was that, to be fair to the baselines, se should at least do k-shot for the highest k possible given context length limits (1-shot being the minimum).