LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classification
Authors: Zhen Qin
Junru Wu
Jiaming Shen
Tianqi Liu
Xuanhui Wang
Venue: COLM (2024)
Keywords: ordinal classification, preference machines
TL;DR: A new paradigm that leverages Large Language Models for solving few-shot multi-class ordinal classification tasks.

Abstract: We introduce LAMPO, a novel paradigm that leverages Large Language Models (LLMs) for solving few-shot multi-class ordinal classification tasks. Unlike conventional methods, which concatenate all demonstration examples with the test instance and prompt LLMs to produce the pointwise prediction, our framework uses the LLM as a preference machine that makes a relative comparative decision between the test instance and each demonstration. A self-supervised method is then introduced to aggregate these binary comparisons into the final ordinal decision. LAMPO addresses several limitations inherent in previous methods, including context length constraints, ordering biases, and challenges associated with absolute point-wise estimation. Extensive experiments on seven public datasets demonstrate LAMPO's remarkably competitive performance across a diverse spectrum of applications (e.g., movie review analysis and hate speech detection). Notably, in certain applications, the improvement can be substantial, exceeding 20% in an absolute term. Moreover, we believe LAMPO represents an interesting addition to the non-parametric application layered on top of LLMs, as it supports black-box LLMs without necessitating the outputting of LLM's internal states (e.g., embeddings), as seen in previous approaches.

Published as a conference paper at COLM 2024
Large Language Models as Preference Machines for Few-shot Ordinal Classification
LAMPO:
Zhen Qin, Junru Wu, Jiaming Shen, Tianqi Liu, Xuanhui Wang Google (zhenqin, jmshen, tianqi liu, xuanhui | Jegoogle.com
Abstract
We introduce LAMPOO, novel paradigm that leverages Large Language Models (LLMs) for solving few-shot multi-class ordinal classification tasks. Unlike conventional methods, which concatenate all demonstration ex- amples with the test instance and prompt LLMs to produce the pointwise prediction, our framework uses the LLM preference machine that makes a relative comparative decision between the test instance and each demonstra- tion. A self-supervised method is then introduced to aggregate these binary comparisons into the final ordinal decision. LAMPO addresses several lim- itations inherent in previous methods, including context length constraints, ordering biases, and challenges associated with absolute point-wise es- timation. Extensive experiments on seven public datasets demonstrate LAMPO's remarkably competitive performance across diverse spectrum applications (e-g., movie review analysis and hate speech detection). Notably, in certain applications, the improvement can be substantial, ex- ceeding 20% in an absolute term. Moreover, we believe LAMPO represents an interesting addition to the non-parametric application layered on top of LLMs, as it supports black-box LLMs without necessitating the outputting of LLM's internal states (e.g., embeddings), seen in previous approaches.
1 Introduction
In-Context Learning (ICL) with few-shot demonstrations, also known as few-shot prompt- ing, is a prominent approach for multi-class classification using Large Language Models (LLMs) (Brown et al., 2020). ICL's remarkable performance and training-free nature, com- bined with off-the-shelf LLMs (e.g., GPTT4 (OpenAI, 2023) and PaLM2 (Google et al., 2023)), has significantly reduced the model adaption costs for new tasks and facilitated the concept of language-model-ant -osee--------eeviice (Sun et al., 2022). Despite these benefits, recent studies indicate that few-shot prompting can sometimes produce unreliable predictions and maybe not always outperform zero-shot prompting (Zhang et al., 2023). To understand these phenomena, extensive research has been conducted, leading to the following insightful observations:
Restrictive Context Length Limit. Despite extensive research efforts (Ratner et al., 2023; Press et al., 2021), the input context length limitation remains major challenge in applying LLMs in many practical scenarios, especially given limited access to advanced commercial systems such as Reid et al. (2024). This constraint limits the number of demonstrations that can be included in a prompt, adversely affecting LLM performance, especially in challenging tasks with a large label space or long example texts. Also note that supporting long contexts does not mean the models necessarily perform well (Li et al., 2024) due to issues such as ordering bias, discussed next.
Destructive Ordering Bias. Recent studies (Liu et al., 2023; Lu et al., 2022) have identified pronounced ordering bias in input prompts for LLMs. For standard few-shot multi-class classification tasks, the arrangement of demonstrations within the prompt can drastically affect the LLM performance, ranging from near chance to otate-of-the-art levels. Address- ing this issue is inherently complex due to the exponentially increasing combinations of demonstration orders the number of examples grows.

---

Published as a conference paper at COLM 2024
Challenges of Pointwise Estimation. In many applications, pointwise estimation is tably more complex compared to comparative estimation. For instance, resolving pointwise relevance estimation addresses relevance ranking, but not conversely, a fundamental ob- servation underpinning the learning- soonngttt-ankk domain (Liu, 2009). Similarly, in fine-grained sentiment classification, LLMs may struggle to differentiate subtle sentiment nuances, such as between "very positive" and "positive", especially when given limited demonstrations in unfamiliar domains, or facing instructions not covered in their pre-training or instruction fine-tuning phases.
This study focuses on ordinal classification (Gutiérrez et al., 2016), an important subset of multi-class classification problem where the labels possess a natural order. Ordinal classification has extensive applications such as customer reviews analysis (Zhang et al., 2022), relevance rating in web search (Thomas et al., 2023), and opinion monitoring in social media (Barbieri et al., 2020). We propose to use instruction-follovinn machines to make pairwise comparisons between the test instance and each demonstration. In this paradigm, LLMs make comparative, rather than absolute, judgments in the ordinal label space, and propose unsupervised aggregation method to convert these comparison outcomes into the final ordinal prediction.
LLMs as preference
We present LAMPO, novel framework that leverages LArge language Models Preference machines for Ordinal classification. LAMPO implicitly addresses the above discussed ICL methods' limitations from three perspectives. First, it avoids packing all demonstrations in a single prompt and thus enables LLMs to leverage an arbitrary number of demonstrations. Second, includes only two examples (one demonstration and one test instance) in each prompt and treats all prompts independently during the aggregation stage, which substan- tially mitigates the demonstration ordering bias in LLMs. Third, LAMPO employs the LLM as preference machine and thus transforms the difficult pointwise estimation problem into a more manageable pairwise comparison problem.
The advantages of LAMPO are demonstrated by its competitive performance across seven diverse datasets, compatibility with both open-sourced (i.e., Flan-T5) and black-box LLMs (i.e., PaLM2), and reliance solely on binary generative decisions. This broad versatility makes it adaptable to various black-box LLMs, including those API-based LLMs with restricted access to internal states (e.g., embeddings) (Zhao et al., 2021). However, it is essential to acknowledge that our paradigm entails an increased number of LLM API calls (though with shorter sequences) and is only applicable to instruction-followinng LLMs, which we elaborate upon Section 6.
Our contributions are summarized as follows:
We introduce the LAMPO framework for the important ordinal classification problem.
This framework effectively addresses several issues in the existing ICL paradigms.
We present pipeline that operates without the need for internal logits or embeddings from LLMs, and does not rely on additional development dataset.
We conduct extensive experiments on seven challenging datasets, covering the analysis of positivity, aspect-based sentiment, hatefulness, irony, and offensiveness of texts. LAMPO consistently demonstrates competitive performance when employed with various LLMs, where the improvements over state-of-the-art ICL approaches can be substantial.
2 Preliminary
In this section, we first introduce the original ICL method for few-shot ordinal classification with basic notations. Then, we discuss two representative ICL methods, which will shed lights on the difference between our work and the existing literature.
2.1 In-Context Learning (ICL) for Few-shot Ordinal Classification
ICL is paradigm that allows LLMs to learn tasks given only few demonstration examples. The standard m-way multi-class classification aims to assign the test input text x to one candidate answer y in the label space Y (Yo, ..., Ym 1)- This paper focuses on ordinal
2

---

Published as a conference paper at COLM 2024
classification where there exists natural order of the labels. Take the sentiment analysis task as an example, Yg denotes "very negative" and Ym-1 denotes "very positive".
A large language model M takes the candidate answer with the maximum score the prediction conditioning task instruction and demonstration set C which includes demonstration examples for each class Namely, we have C = {(x, Vi) ( denotes one demonstration and there are total of mkk ones. The likelihood of candidate answer Y, could be represented by scoring function, f of the whole input sequence with the LLM M:
[i]], } where (x, yi)
P(Y)))) A fM(Y,,
(1)
The final predicted label 9 is the candidate answer with the highest probability:
y = arg max m)
(2)
YjcY
The scoring function f estimates how possible the current answer given the demonstration and the query text. For black-box generation-only LLMs, the most probable label string is expected to be generated directly.
2.2 Contextual Calibration (CC)
Existing work (Zhao et al., 2021; Fei et al., 2023) found that ICL sensitive the quality and ordering of demonstrations, leading to the calibration issue where the model biases towards a certain answer regardless of the input x. To mitigate this issue, contextual calibration (CC) (Zhao et al., 2021) first estimates the bias towards each answer by asking for the prediction probabilities Pef of a content-free (cf) input such as "N/A", and then uses them P(Y to correct the prediction probabilities px ( [)) as
pr = (Pefag(Pe))) Px.
(3)
There are two major limitations of CC. First, CC requires LLM's internal states (i.e., proba- bility logits) thus is not applicable to many black-box LLMs. Second, CC implicitly assumes the context-free input "N can catch the deviation from uniform prediction of the labels. This assumption might be less problematic in simpler classification scenarios, like binary sentiment analysis (positive vs negative). However, it does not hold for many real-world applications. For instance, fine-grained classification tasks that include "neutral" label, the "N/A" A" input likely has non-trivial probability mass, leading to questionable calibra- tion. Similarly, "N/A" assigns a probability mass to labels such as "non-hate" in hatefulness detection, "non-offensive" in offensiveness detection, etc. Thus, the practical utility of CC in such complex scenarios is limited.
2.3 Global Entropy (GlobalE)
Lu al. (2022) studied the impact of demonstration ordering given a fixed demonstration set and presented GlobalE method to identify prompts of specific demonstration orderings that prevent the extremely unbalanced prediction issue?. Specifically, GlobalE first generates multiple sets of candidate contexts by sampling different orderings of the demonstrations. Then, for each candidate contexts set Cm, it constructs probing set by sampling from the See more details of probing set construction in Appendix B. After that, GlobalE computes the predicted label Ji for sampled data point under each Cn as follows:
LLM: (x/,y/) y/) ~
f arg max fM (Y x)) Y₁εY Finally, it ranks all candidate contexts based on the category label entropy of the predictions of the probing set, and uses the top-ranked context for actual inference. This method is
Thheee's ambiguity for "k-shot" in the literature. In this paper "k-shot" refers k demonstrations per class, as Lu et al. (2022); Zhang et al. (2023).
2Lu et al. (2022) also proposed a LocalE method that depends on internal states of L.MMs and under-performs GlobalE in most cases. Thus, we omit this method in the paper.
3

---

Published as a conference paper at COLM 2024
Imputs
Scorting
Decision Making
Output Label
Teat Iratanco
"Gheen
and
Expected
Vey Froetue
"Elion
and
more
Yar Pootivn
: #FFoeeg) Hed0
Ski
Final Thresho de To (T+ Tea
Y; Neutal
Y Negativo
Bellauperried Thresholde
FOCK]
FOXDU
Very Nangiition
Probing Set
Figure 1: An illustration of our LAMPO m-way sentiment classification task. The scoring step compares test instance with every demonstration using LLM. The decision making step converts the score to ordinal label by using thresholds learned offline.
effective under the mild assumption that the probing set should have a uniform distribution over the label space, since they were sampled using sampled Cms,, which should not have strong biases towards any labels globally.
GlobalE does not depend on LLM internal states and generally outperforms basic ICL in our experiments. However, it can still be limited by restrictive context length and difficulties of pointwise estimation. Also, given the combinatorial nature of possible orderings, candidate contexts Cm can only be sampled and are unlikely optimal.
Scope
To address the ambiguity surrounding the term "ICL" (Dong et al., 2022), it is necessary to delineate the scope of this research. We focus on the practical "true" k-shot learning (Perez et al., 2021) for multi-class classification with off-the-shelf LLMs, where at most k demon- strations are available for each class. This is distinct from previous studies that can retrieve a fixed number of demonstrations from a potentially large labeled dataset (Liu et al., 2021).
Furthermore, this work does not assume the presence of labeled development set for tuning hyperparameters, as such an assumption would violate the aforementioned setting. This better aligns with the standard setting from the original ICL work (Brown et al., 2020) and several key subsequent ICL. research (Zhao et al., 2021; Lu et al., 2022) discussed above. The comparison with methods involving test example dependent demonstration retrieval and LLM fine-tuning is beyond the scope of this paper.
4 LAMPO
We now describe new paradigm LAMPO for few-shot ordinal classification, with the overall diagram shown in Figure LAMPO consists of two steps: (1) scoring step that uses the L.LM to compute the comparative score between each test instance with each demonstration, and (2) a decision making step that converts those scores into ordinal labels by using offline learned thresholds without additional labeled development set.
4.1 Scoring
We score each test instance x as follows:
S(x)
Σ xFF(xxx;),)(())) (XW
The fundamental computation unit of LAMPO is denoted F(x, xx), which involves LLM calls to compare input x with each demonstration x; within the ordinal label space. For example, in hatefulness detection, the prompt can be "Given Passage A: (x) and Passage B:
4

---

Published as a conference paper at COLM 2024
{ which one is more hateful?". For the exact prompts used in each dataset, please refer to Appendix C.
When computing F(x,,;), we significantly mitigate the position bias by making two LLM calls with swapped orderings of x and x; in the prompts, which is not feasible for existing ICL methods with combinatorial orderings. The resulting value of F(x,x;) can assume one of three outcomes: x(x,x;) = indicates x is preferred (both LLM calls prefer x), F(x, x;) -1 signifies preference for x; (both LLM calls prefer x;), and F(x,x;) denotes a tie (the two LLM calls yield conflicting or inconclusive results).
The local score for comparing x and x; is derived from the comparison result x(x,x;) and label yi- In line with established practices in ordinal classification literature, we use an integer number to represent each label Y, namely, 1(Y)) = j. This approach simplifies our scoring process follows:
s
((y;) F(x,x;;.
(6)
For example, if x wins over X,, it gets a local score of (yr)) plus point. These local scores are added together the final score 5xx) following Eq. 5.
4.2 Decision Making
Given the score 5(x), we need to convert back into the ordinal label space, which can be reduced to the problem of identifying m thresholds T = {Ti..., Tm-1), so that:
Mm-1
if S(x) > Tm-1
Ym 2
if Tm 2 < S(x) < Tm-1
f
Yo if S(x) < Ti
Note that if there exists labeled development set, the search of thresholds would be easy. However, it will violate the "true" few-shot setting. We thus propose the following strategies to find T:
Expected Thresholds. Intuitively, imagine we have a test example with label value Yj, we know the scores of comparing it with each demonstration from Eq. 6 if everything is noise-free. We can easily derive the expected score of an example of any label value Y;
5₁= I(y:) + E(x, (1(i) -1)
L(xwo)le (1(1)) + 1) E (x.,))lury
(8)
This computation does not require any LLM calls since we just derive the expected score given the demonstration labels. Then the expected threshold T₁ is simply (S, S and the other thresholds can be derived similarly. However, this is under the assumptions that the demonstration labels are noise free and pairwise comparisons are unbiased globally, which is hardly true in practice. Still, we can view this global prior of the thresholds.
Self-supervised Thresholds. Systematic bias can arise from both the demonstrations and LLM itself and the expected thresholds may not be optimal. We are thus interested in decision thresholds that are adaptive to the current demonstrations and LLM. Inspired by Lu et al. (2022), we leverage the probing set rank candidate thresholds based on the global entropy of predicted labels. While we believe applying the probing set to our problem contribution, the probing set construction method itself is not our contribution, which is deferred to Appendix B. Different from Lu et al. (2022) where their candidate space is combinatorial in terms of the number of demonstrations, our search space is combinatorial in terms of the number of integer thresholds, w which is usually small. Note that the LLM calls that compare each example in the probing set and demonstrations are only required once before the threshold search. The major caveat of this self-supervised approach is potentially large variance due to the limited number of demonstrations. We can treat these "self-supervised thresholds" as the empirical observation.
5

---

Published as a conference paper at COLM 2024
The Mixture. Given the pros and cons of the above two methods, we propose to mixture of above two strategies, which can be treated as a simple combination of prior and empirical thresholds. Since we do not assume development set, simply use the average of their corresponding thresholds without tuning the weights for all experiments.
4.3 A Running Example
Let's consider 5-shot 3-way ordinal classification with labels (negative (0), neutral (1), positive (2)). The expected thresholds would be {10, 20}, which can be derived as follows: a negative example expected have a score in noise-free setting (it will tie with the negative demonstrations, and lose to the other 5 neutral and positive demonstrations, thus 0+0+ 5), a neutral example with score 15(5 + 5 + 5), and a positive example with score 25(5 + 10 + 10), thus the expected thresholds are 10 (5 + 15)/2 and 20 = (15 + 25)
Furthermore, let's assume the self-supervised thresholds are {12, 21}, then our final thresh- olds would be 20.5). Now given a test instance x, we do comparisons using LLM with each of the 15 demonstrations and get a score ((x) = 16 according to Eq. 5, then the final predicted label of x will be "neutral".
Experiments
5.1 Datasets
We use 7 challenging ordinal classification datasets, i.e., Twitter, SST-5, Hate, Yelp-5, Lap14, Offensive, and Irony. We follow the publicly hosted setting² where each dataset provides seeds of demonstrations in each configuration, i.e., 5-shot and 10-shot, which are used to compute mean and standard deviation of the concerned metrics on the test set. The same publicly hosted selting is used for all compared methods for a fair and reproducible comparison. The datasets are summarized in Table 3 in Appendix A, including their tasks, metrics, and label spaces.
5.2 Baselines and Model Configurations
For few-shot LLM-based methods, we compare LAMPO with ICI, CC, and GlobalE, which were discussed in Section 2. For few-shot LLM-based methods, we use the black-box PaLM2- S model (text-Bison') and the white-box Flan-15-XXL (Chung et al., 2022). We use the more general generation mode, that does not request any internal states of LLMs, for ICL, GlobalE, and LAMPO. We use the scoring mode to get the required internal logit values of label texts with FlaX-T5-XXL for CC.
Note that due to potential data leakage issues of modern LLMs (Zhou et al., 2023) (see more discussions in Section 6.4), we will focus on comparing different methods under the same LLM, and only briefly compare trends of performance across different L.MMs.
Experimental Results
This section describes our experimental results, which are shown in Table and Table 2. We discuss datasets in groups due to their diverse settings and behaviors.
Results on Twitter. LAMPO exhibits the best performance on PaLM2-S and Flxn-T5-XXL. Contrastingly, CC is ineffective due to the presence of "neutral" class that violates its calibration assumptions. Furthermore, it's worth noting that LAMPO is the sole method capable of handling 10-shot scenarios on this dataset using the concerned popular LLMs. The slightly lower performance this 10-shot setting compared to the 5-shot setting across both LLMs highlights the importance of the demonstrations' quality, as other factors, including ordering bias, have been largely mitigated.
9https://github.com http:///daaugooogo.c/////r//rssr

---

Published as a conference paper at COLM 2024
Table 1: Experimental results PaLM2-S models. "NA" means "Not Applicable" for ICL and GlobalE denotes infeasible experiments due to limited sequence length. CC marked as "NA" as it requires outputting logits of each label, which is not available for black-box LLMs such as PaLM2-S, The best performing method is bolded for each row.
Few-shof LLM-based Methods with PaLM2-8
Dataset
Metric
Config
ICT (Permn at, 2000)
(Zhoo at
GlobalF (ln 2001)
LAMPO (Ours)
Twitter
Acc
10-shot
5-shot
65.13
NA
NA
NA
65.00
NA
66.87a52
65.870.52
5-shot
54.8
NA
54.53a50
55.41,47
SST-5
Acc
10-shot
NA
NA
NA
56.00
Yelp-5
Acc
5-shot
NA
NA
NA
50.67
10-shot
NA
NA
51.130.19
5-shot
75.07 00
NA
76.53,
78.13
Lapl4
10-shot
NA
NA
NA
78.730.66
5-shot 10-shot
48.81241
NA
49.942.67
68.88).38
Hate
macro fl
43.18
NA
46.22
71,431.9
Offensive
macro fl
5-shot 10-shol
76.99
80.232.18
NA
NA
76.90,.5
82,38
81.10p.47
80.82
Irony
fi(ronyy)
5-shot 10-shot
78.93295
NA
79.602
89,611,.13
78.40, sM
NA
80.65 .6
89.53
Table 2: Experimental results Flan-T5-XXL models. "NA" means "Not Applicable" for ICL, CC, and GlobalE denotes infeasible experiments due to limited sequence length. The best performing method is bolded for each row.
Few-shot LLM--based Methods with Flan-T5-XXL
Dataset
Metric
Config
ICL [Brow at.
CC st
GlobalE (La at.
LAMPO (Ours)
5-shot 10-shot
59.00 NA
50.53 NA
63.67 NA
64.55,21
Twitter
Ace
63.88
5-shot
33.13321
53.4
37.07 244
38.73
$81-5
Acc
10-shot
NA
NA
NA
39,47196
Yelp-5
Acc
5-shot
NA
NA
NA
32.93:22
10-shot
NA
NA
NA
33.07
Lap14
Acc
5-shot
76.27
71.93u25
77.23
77.47a52
10-shot
NA
NA
NA
76.930m2
5-shot
54.53,
30.68p.61
58.84476
61.57
Hate
macro.f1
10-shot
48.77936
30.5%
54.43,3.5
61.732.37
Offensive
macro
10-shot
5-shot
60.612741
80.63
60..60)325
49.80
60.3327.19
80.71nss
78.42
78 26.266.0
Irony
fllirony)
10-shot
5-shot
86.72g.37
87.85p.17
64.432.39
63.63m.s
88.11g61
87 37.66118
86.83,10
86 86.37
Results on SST-5. LAMPO works well on this highly challenging dataset with 5-level la- bels, outperforming ICL and GlobalE with both LLMs. 10-shot also consistently outperforms 5-shot with LAMPO, while infeasible to perform 10-shot with other methods. CC with 5-shot performs well with F5an-T5-XXL, as it avoids generation errors this challenging (for Flan-T5-XXL) dataset. See more discussions the summary of results below.
Results on Yelp-5. Results on Yelp-5 mainly shows the capability of LAMPO no other methods are feasible given the long example texts. Also, 10-shot works better than 5-shot on both LLMs for LAMPO. LAMPO allows us to find that there still exists a gap between LAMPO and supervised methods, so the community is suggested to focus on challenging datasets like Yelp-5 for future work.
Results on Lap14. LAMPO works well on this dataset, with the best performance on both PaLM2-S and F5xX-,5-XXL, showing LAMPO is compatible with the challenging aspect-based sentiment analysis task. CC works poorly since there is a "neutral" class.
Results on Hate. We get very strong results of LAMPO on PaLM2-S, where the absolute performance gain be larger than 20%. LAMPO also generates the best performance with Flan-T5-XXL, though the gap is smaller. We hypothesize this because PaLM2-S was not exposed to similar datasets, especially with regard to their label space, during
7

---

Published as a conference paper at COLM 2024
its pre-training and instruction fine-tuning. In contrast, LAMPO handles more general "hateful" concept that PaLM2-S familiar with, regardless of whether it has encountered the exact label space before not. CC again works poorly because the probability mass will be assigned to "non-hate" for the "N/A" A" input.
Results on Offensive and Irony. LAMPO is competitive and robust for these two datasets both LLMs. One noticeable benefit of LAMPO is its robustness across different shots the performance of other methods on Offensive actually drops with 10 shots, but LAMPO is robust and generally improves performance with more shots. CC does not work the probability mass will be concentrated on "non-offensive" and "non moon.rrony".
Summary of Results.
On the seven tasks across a broad range of domains we find that:
(1) LAMPO very competitive and works the best in most cases. Importantly, LAMPO is the most robust method, while other methods have their respective failure patterns. In general, LAMPO shows better comparative performance with respect to other methods on PaLM2-S than Flan-T5-XXL, indicating that more advanced LLMs tend to do comparisons better across various domains.
(2) LAMPO enables popular LLMs the capability to perform competitive ordinal classifica- tion in various settings, including black-box or white-box LLMs, number of shots, and on tasks where the LLM may not have strong prior knowledge of the label space.
(3) The calibration assumption of CC is too strong and this impairs its robustness. Despite its strong performance on SST-5 with Flan-T5-XXL, it can not be used with black-box LLMs and it hurts performance on most datasets due to violation of its calibration assumption.
(4) GlobalE outperforms the ICL baseline in most cases, but it is generally less effective than LAMPO, possibly because the ordering bias is intrinsically difficult to address due to the combinatorial nature of possible orderings.
Discussion and Limitation
6.1 When do we expect LAMPO to be most valuable?
LAMPO shows competitive performance across different domains. One most prominent improvement LAMPO got is the hatefulness detection dataset, where we hypothesize that PaLM2-S does not have a strong prior understanding of the task, especially concerning the "non-hate" label. However, LAMPO mainly leverages more general knowledge such as "which passage is more hateful?" and does not use the actual label text until the very last decision making step. Thus, we believe LAMPO is particularly valuable when the LLM lacks a strong prior understanding of the task's label space.
6.2 Restriction to Instruction-followin LLMs
Earlier research before the ubiquity of instruction-followiin LLMs on ICL mainly depends on text continuation. On the other hand, as hinted by the prompts in Appendix Ć, the com- parison prompts used in LAMPO are instruction-folloiing and are likely only applicable instruction-followiin LLMs. This is one limitation of our framework. However, we argue the limitation may be significant instruction-following LLMs are becoming the norm.
6.3 Cost
We acknowledge that LAMPO does involve an increased number of LLM API calls compared to standard ICL methods. However, we note that: (1) The number of calls only linearly scales with the number of demonstrations, which is usually not big in the concerned "hew-shot" setting. On the other hand, LAMPO's new capability allows for the potential utilization of an unlimited number of demonstrations, making the pursuit of more efficient methods promising future research direction. (2) Each prompt is shorter, so each API call is faster
8

---

Published as a conference paper at COLM 2024
and less expensive than traditional ICL. (3) All comparison API calls can be paralleled, thus the latency can be lower than traditional ICL, given sufficient parallelism.
Data Leakage
possible that certain LLMs saw some datasets during their pre-training or instruction fine-tuning, which is major concern of fair evaluation in the era of LLMs (Zhou et al., 2023). We mainly focused on comparing different methods using the same LLMs so the comparisons are fair. In fact, most existing LLM pre-training and instruction tuning methods adopt the pointwise strategy, a potential data leakage will benefit them more, compared to our pairwise framework. Thus, the improvements achieved by LAMPO over these baselines further demonstrate its effectiveness.
6.5 Reproducibility
As noted, all datasets used in this paper are publicly available, and we did not change any of their demonstrations, labels, or any other content. The exact prompt templates we used are shown in Appendix C.
Related Work
In-Context Learning. We have discussed several popular ICL-based methods in Section which is prominent family of approaches that leverages LLM for few-shot multi-class classification. Another emerging paradigm, which has gained popularity recently, involves retrieving distinct demonstrations for each test instance (Rubin et al., 2021). These demon- strations are sourced from potentially vast annotated demonstration pool, which is very different from the "true" few-shot setting studied in this work.
Comparisons using LLMs. We note that the "pairwise" or "comparative" paradigm is classic concept in various domains. LLMs serve evaluators to compare preferences of generative model's outputs (Kocmi & Federmann, 2023; Vu et al., 2024; Yan et al., 2023), such as to determine which summary is better in text summarization, with many important applications such as language model aligmment (Liu et al., 2024). In contrast, our approach compares inputs derive an output. For web search ranking, given a query, pairs of candidate documents can be compared using LLMS in terms of their relevance to the query (Qin et al., 2024). This task significantly differs from ours, and it involves quadratic increase in API calls with the number of documents under each query. In summary, existing work only share the very general idea of doing pairwise comparisons with very different problem setups where natural pairs are readily available. In contrast, our work pioneers the exploration of LLMs as preference machines for the important few-shot ordinal classification task and demonstrates LLMs' comparative capability in broader dimensions (i.e., traditional sentiment, hatefulness, offensiveness, irony, and aspect-based sentiment).
"Non-parametrice application of LLMs. Very recently, Xu et al. (2023) have shown "near- est neigbor" style application of LLMs. We posit that our paradigm offers a compelling addition this non-parametric perspective of LLM utilization, where existing research (Xu et al., 2023) hinges upon extracting embeddings that are not generally accessible, and neglects to explore the preference machine perspective of LLMs.
Conclusion
This paper introduces a simple yet novel paradigm, LAMPO, for the important k-shot multi-class ordinal classification problem with LLMs. LAMPO effectively addresses several inherent limitations of traditional ICL methods, and demonstrates strong performance on 7 publicly available datasets covering diverse topics. Moreover, LAMPO is versatile, compatible with both black-box and white-box LLMs, and capable of accommodating an arbitrary number of demonstrations.

---

Published as a conference paper at COLM 2024
References
Francesco Barbieri, Jose Camacho-Collados, Leonardo Neves, and Luis Espinosa-Anke. Tweeteval: Unified benchmark and comparative evaluation for tweet classification. ar Xiv preprint ar Xiw::0010.224555 12421,, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. arXio preprint ar arXiv:2210.114166 2022.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. ar arxiv preprint ar wwxi::2301.02..0 2022.
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. Mitigating label biases for in-context learning. Xrv preprint ar Xrx::23305.199 199448 2023.
Google, Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez
Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng. Colin Cherry, Christopher A. Shoqueete-Choo Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaLM technical report, 2023.
Pedro Antonio Gutiérrez, María Pérez-Ortiz, Javier Sänchez-Monederr, Francisco Fernández-Navarro and César Hervás-Martinez Ordinal regression methods: Sur- vey and experimental study. IEEE Transactions on Knowledge and Data Engineering, 28(1): 127-146, 2016.
Tom Kocmi and Christian Federmann. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint ar xrXiv:2302.14520, 2023.
Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context Ilms struggle with long in-context learning, 2024.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint ar arXio:2101.00 0068044 2021.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint ar Xiv: 031722, 2023.
10

---

Published as a conference paper at COLM 2024
Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al. Lipo: Listwise preference opti- mization through maaning-to-ranlee Xiv preprint XrXi:22402.01878 2024.
Tie-Yan Liu. Learning to rank for information retrieval. Found. Trends Inf. Retr., 2009.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, 2022.
OpenAl. GPT-4 technical report. Xiv preprint arXiv:2303.08774, 2023.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. Advances in neural information processing systems, 34:11054-11070, 2021.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference Learning Representations, 2021.
Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, et al. Large language models are effective text rankers with pairwise ranking prompting. Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2024.
Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. Parallel context windows for large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6383-6402, 2023.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint ar xrx::2403.005330 2024.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint ar arXiv:2112.08633, 2021.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for wawpaaage--mddll----seeeeeiiiiie In International Conference on Machine Learning, Pp. 20841-20855. PMLR, 2022.
Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. Large language models can accurately predict searcher preferences. arXiv preprint ar arXio:2309.1062) 2023.
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. Foundational autoraters: Taming large language models for better automatic evaluation. ar Xive preprint ar arXio:2407.108) 2024.
Benfeng Xu, Quan Wang, Zhendong Mao, Yajuan Lyu, Qiaoqiao She, and Yongdong Zhang. knn prompting: Beyond-context learning with calibration-free nearest neighbor inference. In The Eleventh International Conference on Learning Representations, 2023.
Jing Nathan Yan, Tianqi Liu, Justin T Chiu, Jiaming Shen, Zhen Qin, Yue Yu, Yao Zhao, Charu Lakshmanan, Yair Kurzion, Alexander M Rush, et al. On what basis? predicting text preference via structured comparative reasoning. ar Xiv preprint ar xrXi::2311.08390 2023.
Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. A survey on aspect-based sentiment analysis: Tasks, methods, and challenges. IEEE Transactions on Knowledge and Data Engineering, 2022.
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. Sentiment analysis in the of large language models: A reality check. arxiv preprint ar arXi::2305.1000 2023.
11

---

Published as a conference paper at COLM 2024
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance language models. In International Conference on Machine Learning, pp. 12697-12706. PMLR, 2021.
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your Ilm an evaluation benchmark cheater. arviv preprint ar xrx::3311.01164, 2023.
A Dataset Descriptions
We show the descriptions of each dataset used in our evaluation in Table
Table 3: A description of the datasets used in evaluations. The labels are ordered in the ordinal space
Dataset
Label Space
Metric
Task
negative, neutral, positive
Accuracy
Sentiment andlysis social pusts.
S
very negative, negative, neutral, positive, very positive
Accuracy
Sentiment analysis on movie reviews.
Yelp-5
very negative, negative, neutral, positive, very positive
Accuracy
Sentiment analysis customee reviews businesoes.
Lap14
negative, neutral, positive
Aceuracy
Aspect based sentiment analysis.
Hate
non hate, hate
Macro Fl
Hate speech detection.
Otfensive
nut-oflensive, offersive
Macro
Offensive language identification.
Irony
non .rony, Ireny
of ofirony
Irony detection.
B
The Probing Set
We provide more details on the construction and use of the "probing set", with most contents below adopted from Lu et al. (2022). The key idea is to build collection of examples that has a roughly uniform label distribution. Then given a candidate in the search space (a set of thresholds in LAMPO and specific ordering of demonstrations in GlobalE), the predictions of the examples can be generated, and the best candidate should produce label predictions closest to the uniform distribution, measured by entropy.
To construct such a "probing set" without access to any additional data, Lu et al. (2022) proposes to directly sample from an LLM. Concretely, given set of training demonstration examples 5 = (x, i;), 1,..., MI we define transformation T, mapping each example into natural language space. We use a simple transformation function T such that T(x,
input: xytype: y/- This transforms each example into standard format sentence, which linearises each element in the set into the natural language space defined as 5 [t;),i= 1,.., n.
We then define full permutation function group of n training examples, F {ff }
1,..,, nt, where each function fm takes 5' as input and outputs Cm: the concatenation of unique permutation. For each prompt candidate Cm, we then sample from the large language model M to obtain the probing sequence 8m ~ PM We stop decoding from the language model upon generating the special end-of-sentence token defined by template, or reach the generation length limit. The probing set construction method is illustrated in Figure 2, where the objective is generate probing set that shares a similar distribution to the training examples.
We run this sampling process for sampled prompt ordering permutations and extract probing examples from them, then gather extracted examples together to form the probing set D. Although the probing set contains predicted label for each sentence, there is no guarantee the validity of these labels. Therefore, discard them from the probing set as we are only interested in sampling probes from the language model corresponding to the input distribution.
12

---

Published as a conference paper at COLM 2024
Train
Train
Train
Train
Gonsration I
Tratn fxentence. Inbelt your
(Review: the creling
Probing Set (sentenec, label)
reeping
unhereaaly parmol
Seniment repetve
Train 3
Train
Train 2
Train
Review: Ramnes muttpple codings
PLM
Sentmert posttive
usivenily mullipie
Train (Prompd
Review: has way of seeping into conscivasaess Sentinene posttive
Greeration
Train
Train
Train
Train
[Review: Iice movie
(dice novie,
3
Scalinete posilive
Figure 2: An illustration of probing set construction from Lu et al. (2022). "PLM" is the same as "LLM" in the context of this paper.
Once we have constructed probing set for given set set of demonstrations, we can now use that probing set to identify the best possible prompt order as in al. (2022) and thresholds in LAMPO. In this work, and the same probing set is used for LAMPO and GlobalE for fair comparison.
PaLM2-S to generate 50 examples in each configuration,
C Prompt Templates
We show the prompt template used by LAMPO on each dataset. In general, we tried to avoid the usage of specific words such "Tweet" "Review", but use more generic word "Passage". Also, we did not heavily tune the prompts on each dataset, some datasets even share the same prompt template. These demonstrate the generality and robustness of the LAMPO paradigm, and better performance should be achieved with better prompt engineering.
C.1 Twitter Dataset
Given two Passages, compare their sentiments with labels from l'negative', 'neutral', 'positive'].
Passage A: {item;}
Passage B: {item2)
Which Passage is more positive in terms of its sentiment?
Output Passage A Passage B:
C.2 SST-5 and Yelp-5 Datasets
Given two Passages, compare their sentiments with labels from ['very negative', 'negative', 'neutral', 'positive', 'very positive'].
Passage A: {item}
Passage B: {itemz)
Which Passage is more positive in terms of its sentiment?
Output Passage A Passage B:
13

---

Published as a conference paper at COLM 2024
C.3 Lap14 Dataset
Given two Passages, compare their sentiments towards their respective aspects with labels from ['negative', 'neutral', 'positive'].
Passage A: (item; (sentiment towards (aspect;]] }),
Passage B: { (sentiment towards {aspect2))
Which Passage is more positive in terms of its sentiment towards its aspect?
Output Passage A or Passage B:
C.4 Hate Dataset
Given two Passages, compare their sentiments with labels from l'non-hate', /hate').
Passage A: {item1}
Passage B: {item}}
Which Passage more hateful in terms of its sentiment?
Output Passage A Passage B:
C.5
Offensive Dataset
Given two Passages, compare their sentiments with labels from ('non-offensive', 'offensive').
Passage A: {item]}
Passage B: {item?}
Which Passage more offensive in terms its sentiment?
Output Passage A or Passage B:
C.6 Irony Dataset
Given two Passages, compare their irony with labels from I'non iion''rony', 'irony').
Passage A: [item]}
Passage B: {items}
Which Passage is more ironic in terms of its sentiment?
Output Passage A Passage B:
14


Reviews:
[{"review_id": "zKkMLY7C7p", "reviewer": "Reviewer_jhug", "summary": "This article presents the LAMPO approach to performing sentiment analysis. \nThis approach is \"LLM free\" which means one can use any LLM they want since the strategy focuses on a way to compare scores extracted in two different ways, then the approach compares the scores and performs a decision process to extract the sentiment level.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback and advocate of our paper. Here we discuss some questions/concerns you raised.\n\n- One can regret the approache is not compared to fine-tuned approach in order to have a benchmark.\n\nThank you for your suggestion. We agree a fine-tuned approach can serve as a good reference point. We note that this paper focused on fair comparative performance of different methods using the same LLMs. Comparison with fine-tuned models may have caveats such as potential data leakage. Nevertheless, we report fine-tuned numbers here using a T5 large model trained on each datasets’ respective training data (not available to LLMs method, despite potential data leakage). Below are the summarized numbers (note the best numbers can be from different LLMs and/or baselines):\n\ndataset, best baseline, best LAMPO, Fine-tuned\n\ntwitter, 65.13, 66.87, 67.73\n\nSST-5, 54.8, 56, 56.80\n\nYelp-5, NA, 51.13, 65.60\n\nLap14, 77.23, 78.73, 78.60\n\nHate, 58.84, 71.43, 46.94\n\nOffensive, 82.38, 81.10, 80.76\n\nIrony, 88.11, 89.61, 79.44\n\nWe believe the results are appealing - on Lap14, Hate, Offensive, and Irony, LAMPO can outperform fine-tuned models. On twitter, SST-5, LAMPO can fill the gap by a large margin. Yelp-5 appears difficult for the LLMs we experimented with but LAMPO is the only one that can produce some reasonable results.  \nWe plan to add these to the appendix for the future version and discuss the potential caveats. We appreciate your suggestions.\n\n\n- You discuss the ecological impact of your model. Could you elaborate in terms of energy and CO2 consumption?\n\nThis is a great question. Overall we believe that, although LAMPO proposes a paradigm shift for ordinal classification, it is still comparable to existing methods in terms of how LLMs are used, so it should not lead to noticeably more energy cost or CO2 consumption.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The article is well presented and well written; the approach is well described.\nThe experiences sufficiently present the approach's effectiveness against classical prompting LLM approaches.", "reasons_to_reject": "One can regret the approache is not compared to fine-tuned approach in order to have a benchmark.", "questions_to_authors": "What are the performances of classical approaches against yours?\nYou discuss the ecological impact of your model. Could you elaborate in terms of energy and CO2 consumption?", "extra_scores": {}}, {"review_id": "5R6ZLbpCEJ", "reviewer": "Reviewer_9SYA", "summary": "This paper proposes LAMPO, a method that uses LLMs to conduct few-shot ordinal classification by prompting the LLM to perform pairwise comparisons between the test example to be labeled and the few-shot examples available. They explore an array of ordinal classification tasks (e.g., SST, Yelp, etc.) using two LLMs (Palm2-Bison, Flan-T5-XXL).\n\nOverall, the idea itself is simple and the results on the surface are promising, but there are aspects of the method and experimental methodology that could be further improved.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your careful review and constructive feedback to help us improve our paper.\n\n- The paper could explore more techniques for aggregating the results of the pairwise comparisons. \n\nWe appreciate your suggestion. As LAMPO is the first work along few-shot comparison-based ordinal classification, there are several future directions. Your suggestion is promising. Other directions may include intelligent selection of pairs and joint pairwise comparison in one LLM call. \n\n- One could argue that the central issue this paper tries to address—how to perform ICL when you are severely constrained in terms of input length—is less and less of a problem as newer LLMs support ever-larger context windows. \n\nThis is a thoughtful point. We note that context length constraint is *one of* the major issues LAMPO wants to address. It is possible that long-context LLMs *may* have *more* severe position biases or still not optimal to make (the difficult) pointwise decision. These aspects may need further studies. \n\n- Questions about \"NA\" and error bars.\n\nWe used “NA” mainly to show applicability of LAMPO, and did not mean to be unfair for baselines in terms of performance (they are missing, but not low). Per your suggestion, we performed 1-shot experiments on the Yelp-5 dataset on PaLM2:\n\nICL: 41.47_{1.20}\n\nCC: NA\n\nGlobalE: 43.20_{1.47}\n\nLAMPO: 47.53_{1.40} \n\nThe gap is noticeable, possibly because it is more difficult for the LLM to handle several long documents (for 1-shot 5 way, there are 6 long documents in the prompt). We will add more discussion and appreciate your suggestion.\n\nFor error bars, we note that they are the variance from 3 runs and may not be appropriate for significance tests. We thus perform significance tests on the instance prediction level on PaLM2-S against the competitive GlobalE. On a two-tailed t-test at the p < 0.05 level, LAMPO is significantly better than GlobalE on all settings except for Offensive 10-shot.\n\n- Why do some of the baselines have such high variance in their results?\n\nYes, we found that pointwise ICL can be fragile based on demonstrations. Our hypothesis is that some demonstrations can be harmful for performance. On the other hand, in LAMPO, each demonstration is compared separately, so a harmful demonstration only contributes 1/kn. We did not include this since we did not perform comprehensive study on this perspective, and will discuss more.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**The method is simple, and provides an alternative to in-context learning for ordinal classification.** Because the method only works using pairwise comparisons, this means that input length limits are in general not a problem, which is not the case with in-context learning (ICL).", "reasons_to_reject": "**The paper could explore more techniques for aggregating the results of the pairwise comparisons.** Currently, the pairwise comparisons are aggregated into a score number, and two methods are explored for obtaining class thresholds. However, the paper could be strengthened by exploring other ways of performing this aggregation, especially given the unique nature of pairwise data (e.g., one could cast this as an optimization problem, where the label that violates the smallest number of pairwise comparisons is set as the predicted class).\n\n**One could argue that the central issue this paper tries to address—how to perform ICL when you are severely constrained in terms of input length—is less and less of a problem as newer LLMs support ever-larger context windows.** Repeating these experiments with more recent LLMs than the ones in the paper (e.g., Gemini, GPT-4, Llama-3, etc.) and showing the benefits remain when longer context windows are available would strengthen the paper’s central point.\n\n**There are aspects of the experimental methodology that could be improved.**\n\n- It is not clear whether it is fair to the baselines that if they cannot fit a 10-shot prompt in their context window, then the result should be “NA”; in this case, a practitioner would likely pare down the number of examples until it fits in that context window (e.g., using 7-shot, or a variable number by class). This is especially true of the Yelp-5 experiment: could you not do at least 1-shot in this case, for the baselines?\n- In some cases the error bars around the results suggest that LAMPO might not be better than the baselines (e.g., PaLM SST-5, Lap14). It would be appropriate to discuss this, and you should highlight all models in a row that perform statistically equivalently.", "questions_to_authors": "Why do some of the baselines have such high variance in their results, e.g., 10-shot Offensive for Flan-T5? Is this because of some really poor runs?", "extra_scores": {}}, {"review_id": "f8KAaDVMa3", "reviewer": "Reviewer_cX41", "summary": "This work studies multi-class ordinal classification (i.e., classification where there is an inherent ordering between the classes) using LLM prompting with demonstrations (i.e., in-context learning). Simply concatenating demonstrations for in-context learning has challenges, including (1) context length limitation, (2) ordering bias, and (3) pointwise estimation. The paper proposes LAMPO to address these challenges. LAMPO can be viewed as a prompting technique where each demonstration is compared against the test example separately, and the LLM judge is only required to make pairwise comparisons. Given $m$ demonstrations, the LLM judge performs $m$ comparisons (or $2m$ for symmetry) and then the outcomes are aggregated to obtain a final predicted class. The work considers a novel method to map from combined output into the final class. Experiments are conducted on a range of standard datasets (e.g., Twitter, SST-5, Yelp-5, etc), and the results demonstrate the effectiveness of LAMPO.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback and advocate of our paper. Here we discuss some questions/concerns you raised.\n\n- The method requires many calls to LLMs (e.g., 2𝑚  call for each test example with 𝑚 demonstration).\n\nYes, the number of API calls increasing is a limitation of LAMPO at its current form and we fully acknowledge it (also in the paper). As LAMPO is the first work along the concerned direction (few-shot comparison-based ordinal classification), we did not focus on cost optimization. Interesting future work can include smarter pair selection and debiasing to save half of the calls.\n\n\n- The method could become less relevant as the context length of LLMs are becoming longer.\n\nWe agree with you. We believe that LAMPO may stay relevant for some time since, even if long context LLMs may address the input length constraint, it may not address the other constraints (position bias, difficulty of pointwise estimation). In fact, it is unclear if the position bias will be more severe or not with longer context given the “lost in the middle” behavior commonly found in LLMs. Long context LLMs such as Gemini-1.5 are promising but further research might be needed to address possible pitfalls raised from such models before they become the default choice. \n\n- Have you considered applying LAMPO (in your few-shot learning manner) to evaluate open-ended text generation tasks?\n\nThis is an interesting idea for an important problem, and can be a promising future research direction. We believe that the general idea of LAMPO (preference + decision making) can be applied to a wide range of ordinal classification / ranking problems, which is ubiquitous in practice. For example, as you suggested, we may want to measure the safety (not-safe vs safe, or more fine-grained safety levels than the two options) of a text generation, and may follow LAMPO to improve the prediction accuracy given some  labeled instances. These can be impactful topics to explore.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. LAMPO is a simple and novel prompting idea, which can mitigate the three challenges\n2. The results illustrate the effectiveness of the proposed approach on several datasets\n3. The approach is applicable to black-box LLMs", "reasons_to_reject": "1. The method requires many calls to LLMs (e.g., $2m$ call for each test example with $m$ demonstration).\n2. The method could become less relevant as the context length of LLMs are becoming longer.", "questions_to_authors": "Have you considered applying LAMPO (in your few-shot learning manner) to evaluate open-ended text generation tasks?", "extra_scores": {}}, {"review_id": "Tec3TWBhGB", "reviewer": "Reviewer_pYRD", "summary": "This paper introduces LAMPO, a new method for ordinal classification (e.g., sentiment analysis) using LLMs. The main idea is to use the LLM as a preference machine, making pairwise comparisons between a test instance with each demonstration example. This approach addresses limitations of prior ICL methods, such as context length constraints and ordering biases. LAMPO aggregates these pairwise comparisons into a final ordinal prediction using a self-supervised method. Experiments on several datasets show LAMPO's good performance with improvement over state-of-the-art ICL approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback and advocate of our paper. Here we discuss some questions/concerns you raised.\n\n- The blackbox LLM model used in the paper are not SOTA ones, making it less convincing on whether the proposed approach can still outperform other ICL approaches.\n\nWe acknowledge this comment. As discussed in the paper,  the two LLMs chosen are representative in terms of black-box vs white box, and vary in their sizes. Performance comparison is focused on comparing different methods using the same LLMs. PaLM2 is still popular at the time of writing the paper and available as a commercial API. Benchmarking on more LLMs is a meaningful effort.\n\n\n- The paper acknowledges that LAMPO requires more LLM API calls than traditional ICL. While it argues that the calls are shorter and can be parallelized, the overall cost implications is still a major concern given the pair-wise comparison.\n\nYes, the number of API calls increases. As LAMPO is the first work along the concerned direction, we did not focus on cost optimization. Interesting future work can include smarter pair selection and we are excited about the possibilities.\n\n\n- The evaluation is primarily focused on empirical performance. More in-depth analysis is recommended to understand why LAMPO works well and under what conditions it might fail.\n\nThank you for your suggestion. We discussed one key observation in section 6.1, where we observed that existing ICL methods can be brittle to label strings. As discussed in the paper, the major strength of LAMPO is its robustness and applicability across different settings. On the benchmark we tested, we did not find clear failure patterns of LAMPO.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- LAMPO presents a new approach for ordinal classification with LLMs, using them as preference machines rather than for direct point-wise prediction. This approach is innovative and could inspire further research in this direction.\n\n- The paper demonstrates LAMPO's performance across diverse datasets, outperforming existing ICL methods in many cases. The significant improvements in some tasks highlight the potential impact of this approach.\n\n- LAMPO is compatible with both open-source and black-box LLMs, making it widely applicable. It also doesn't require access to internal LLM states, which increases its flexibility.", "reasons_to_reject": "- The blackbox LLM model used in the paper are not SOTA ones, making it less convincing on whether the proposed approach can still outperform other ICL approaches.\n\n- The paper acknowledges that LAMPO requires more LLM API calls than traditional ICL. While it argues that the calls are shorter and can be parallelized, the overall cost implications is still a major concern given the pair-wise comparison.\n\n- The evaluation is primarily focused on empirical performance. More in-depth analysis is recommended to understand why LAMPO works well and under what conditions it might fail.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a method to use LLMs for solving few-shot multi-class ordinal classification tasks. Compared with conventional ICL, the proposed method aggregates the relative comparative decisions between the test instance and each demonstration. The cost becomes larger while it overcomes some limitations such as ordering biases. One out of four reviewers tends to reject the submission because of the lack of comparisons with more techniques for aggregating the results of the pairwise comparisons, and LLMs with larger context window. The experimental settings can also be improved as suggested.
Decision: Accept