Predicting Text Preference Via Structured Comparative Reasoning (2024)

Abstract: Comparative reasoning plays a crucial role in predicting text preferences; however, large language models (LLMs) often demonstrate inconsistencies in their reasoning, leading to incorrect preference predictions. While approaches like Chain-of-Thought improve accuracy in many settings, they struggle to consistently distinguish the similarities and differences of complex texts. We introduce SC, a model that prompts LLMs to predict text preferences by generating structured intermediate comparisons. SC begins by proposing aspects for comparison, followed by generating textual comparisons under each aspect. We select consistent comparisons with a pairwise comparator that ensures each comparison of a given aspect clearly distinguishes differences between texts, significantly reducing hallucination and improving consistency. Our empirical studies across various NLP tasks, including summarization, retrieval, and automatic rating, demonstrate that SC’s enhanced performance in text preference prediction is significant.

heading: 1 Introduction
text: Comparative reasoning is crucial for predicting text preferences, as deciding the best out of a set of texts requires careful examination of the similarities and differences across the documents. Hence, comparative reasoning has been especially useful in NLP tasks such as text summarization (Yang et al., 2023; Lee et al., 2023), search ranking (Qin et al., 2023), and automatic evaluation (Adlakha et al., 2023), where text preference prediction is a key step.
However, as corpora grow more dense and complex across domains, accurate comparative reasoning becomes increasingly challenging. Existing approaches rely on pretraining or fine-tuning models (Yu et al., 2023a; Iso et al., 2022) at the
∗ Work done during the internship at Google. E-mail: jy858@cornell.edu.
cost of massive human annotation and computation. With the emergence of large language models (LLMs) (OpenAI, 2023; Touvron et al., 2023a; Anil et al., 2023; Jiang et al., 2023a), prompting approaches like Chain-of-Thought (CoT) (Wei et al., 2022) offer a promising solution for enhancing comparative reasoning. These approaches leverage LLMs’ exceptional language generation capabilities without incurring significant overhead.
Nonetheless, LLMs exhibit arbitrary and erroneous outputs when prompted for comparative reasoning (Adlakha et al., 2023). Specifically, LLMs demonstrate inconsistency in their comparative analyses of texts. Figure 1 (bottom left) provides an example of logically inconsistent LLM reasoning using zero-shot CoT prompting. The LLM’s generated explanation initially describes a property as common to the text pair (highlighted in green), but later implies that the same property is a strength of just one of the documents (highlighted in yellow). This inconsistency in the LLM’s comparative analysis leads to an incorrect prediction.
To address these challenges, we present SC2, a StruCtured Comparative reasoning model that constructs an intermediate structured representation contrasting two text corpora for more accurate text preference prediction, as illustrated in Figure 1. First, SC2 proposes a set of aspects from text pairs to guide the comparison step. Second, SC2 generates textual comparisons for every aspect. We refer to aspects and comparisons together as intermediate structured representations. To improve the consistency of reasoning (e.g., a contrastive comparison of a aspect should not overlap with a common comparison), SC2 adopts approximate inference: SC2 samples multiple responses in generative process and uses a pairwise comparator to select the most consistent intermediate structured representation for final preference prediction.
We demonstrate the effectiveness of SC2 in improving text preference prediction across various
10040
tasks including text summarization (Stiennon et al., 2020), document retrieval (Soboroff et al., 2018), and helpfulness and harmlessness detection (Bai et al., 2022) with average 2.5 and 7.0 points gain over the top and bottom baselines, respectively.
Our analysis further confirms the effectiveness of SC2 without incurring expensive LLM usage, and ablation studies emphasize the importance of the pairwise comparator. Our extensive human evaluations also indicate that SC2 aids in interpretation and assists users in making decisions.

heading: 2 Related Work
text: Prompting Large Language Models LLMs have recently advanced the state-of-the-art performance across many NLP tasks (Anil et al., 2023; OpenAI, 2023; Chowdhery et al., 2022; Touvron et al., 2023a,b; Jiang et al., 2023a). These LLMs have demonstrated the capability to provide chain-of-thought explanations that elucidate their reasoning processes (Wei et al., 2022; Kojima et al., 2022). However, the chain-of-thoughts
generated by LLMs are often arbitrary or contradictory (Wang et al., 2022; Turpin et al., 2023; Chen et al., 2023; Dhuliawala et al., 2023), unfaithful to the facts (Lyu et al., 2023, 2024) or lacking robustness to rephrased questions. To mitigate these issues, several works aim to leverage consistencybased (Wang et al., 2023; Zhou et al., 2022), or verification-based approaches (Ling et al., 2023; Lyu et al., 2024) to improve the reasoning capacity of LLMs, yet the benefit of such additional techniques are still ambivalent (Huang et al., 2023). Furthermore, all these advanced techniques still concentrate on processing raw-text inputs, thereby overlooking the integration of structural information. Moreover, they lack implementations of explicit consistency constraints, which is crucial for maintaining logical coherence in generated outputs.
Comparative Reasoning and Summarization Comparative reasoning involves comparing and contrasting different documents (Yu et al., 2023a), which has applications for a broad range of NLP tasks including text ranking (Jiang et al., 2023b;
Qin et al., 2023), reward modeling (Ouyang et al., 2022; Lee et al., 2023; ?) and automatic text generation evaluation (Liu et al., 2023a). Initial explorations focused on mining comparative content from text corpora (Jindal and Liu, 2006; Li et al., 2010). More recent studies have developed models for generating comparative text, including generating arguments for answering comparative questions (Chekalina et al., 2021; Amplayo et al., 2021) and summarizing comparative opinions (Iso et al., 2022). Additionally, Zhong et al. (2022, 2023) prompt LLM to describe the differences between two text distributions in natural language and Dunlap et al. (2023) further extends to discover differences given a set of images from ImageNet.
One challenge of directly prompting LLMs for comparative reasoning is that the input text often contains a mixture of diverse patterns. As such, it is crucial to incorporate fine-grained aspects to guide LLMs for generating more comprehensive summarizations (Sun et al., 2023; Xu et al., 2023; Wu et al., 2023; Yu et al., 2023b). Early works (Lin and Hovy, 2000; Titov and McDonald, 2008) used clustering or topic modeling to identify aspects in documents. Lekhtman et al. (2021) fine-tune a pretrained language model for aspect extraction, which relies on manual labeling of comparative data. On the other hand, Goyal et al. (2022); Yang et al. (2023) leverage LLMs to perform summarization with the fixed aspects provided by humans. Differently, we leverage LLMs to automatically discover aspects to guide comparative reasoning, which provides a flexible way to incorporate fine-grained task-relevant signals while requiring minimal labeling efforts.

heading: 3 Methods
text: Our model, SC2, produces comparative reasoning for text preference prediction that applies to densely written texts, generalizes to multiple domains, and ensures consistency. In this section, we give the generative process and inference procedure for SC2. Our primary focus is ensuring the comparisons consistently distinguish similarities and differences between texts.

heading: 3.1 Generative Process
text: The generative process has three steps. First, given a text pair, SC2 simplifies the task by delineating a set of aspects, as depicted in Figure 1. These aspects, consisting of concise phrases, enable the structured comparison between the texts. Second,
SC2 produces concise comparisons, aspect-focused comparative statements that clearly express how the texts are similar and different. In this paper, we implement this explicit consistency mechanism: similarities identified as shared between the text pair should not overlap with what’s unique to each of them. Given the aspects and comparisons, the final step predicts which text is preferred.
Formally, for a text pair problem, we denote the text pair as← and→, along with a query. SC2 has three components: Aspects a = {a1, a2, . . . , an}, comparisons c = {c1, c2, . . . , cn}, and text preferences y ∈ {←,→}. The comparison c has three columns: {c←, c→i , c∩i }, c→i and c←i refers to properties exclusive to→ and← respectively, and c∩i to properties shared by both texts.
SC2 follows the following generative process: First, it generates the aspects conditioned on the text using an aspect model, P (a). Second, comparisons for each aspect are generated from the comparison model
P (c|a) ∝ ∏
i
l(ci)× P (ci|c<i, a)
where the function l : C → R+ evaluates the consistency of ci. A higher value of l(ci) indicates a greater degree of consistency. Finally, preference model P (y|c, a) produces the preference label y. Parameterization We use LLMs with specific prompts to parameterize each model. With LLMs generating reliable scalar values of consistency is unreliable (Imani et al., 2023; Liu et al., 2023b). Instead of directly regressing a consistency score, we rely on pairwise comparisons, which have been observed to be more reliable (Qin et al., 2023). We define a pairwise comparator l′(c, c′) = 1(l(c) ≥ l(c′))1, which takes a pair of comparisons (c, c′) and determines the more consistent one.
To facilitate this, we recruit experts to develop few-shot prompts that demonstrate a direct comparison of two structured representations based on consistency within itself. We guide our annotators to assess pairs (c, c′) against consistency criteria, emphasizing that elements of the comparison should ideally exhibit no overlap. Detailed instructions are attached in the Appendix.

heading: 3.2 Tournament-based Inference
text: Given the generative model, the goal of inference is to produce aspects and comparisons that are high
1We break the tie randomly.
probability under the model and consistent. We take a step-wise approach, choosing aspects, comparisons, and then finally predicting preferences.
When choosing aspects, we follow prior work by employing a variety of sampling strategies to obtain near-optimal aspects a∗ from P (a) (Wang et al., 2023; Amplayo et al., 2021). We provide more details on these strategies in Section 4.
Given aspects a∗, our next goal is to find comparisons that are likely under the comparison model argmaxc P (c|a∗) = argmaxc l(c) · P (c|a∗). There are two challenges with this objective: First, the set of possible comparisons is intractably large. Second, the consistency function l(c) is unreliable. We approach the first challenge by sampling a set C of high probability comparisons from P (c|a), and the second challenge by selecting the most consistent comparison by applying the pairwise consistency comparator l′(c, c′) in a binary reduction. Formally, we select the most consistent comparison by optimizing
c∗ = argmax c∈C
∑
c′∈C\{c} l′(c, c′).
Naively, this optimization problem above requires O(|C|2) pairwise comparisons to optimize exactly.
To reduce the number of pairwise comparisons, we utilize a tournament approach that performs O(|C|) comparisons. The tournament approach utilizes a binary reduction: Each step of the binary reduction takes a pair of comparisons and eliminates the less logically consistent one into the successive rounds. We illustrate the tournament
approach in Figure 2. The naive and tournament approaches are equivalent if transitivity holds in the consistency comparator l′(c, c′). In practice, transitivity does not always hold with LLM parameterizations, resulting in the tournament approach trading off accuracy for efficiency.
Finally, with structured intermediate representation (a∗, c∗), SC2 decides between ←,→ which one is preferred by taking argmaxP (y|a∗, c∗).

heading: 4 Experimental Setup
text: Aspect Model We experiment with two models for generating aspects: the online aspect model and the offline aspect model. Both models use PaLM2-L to obtain aspects.
The online aspect model dynamically generates aspects using the CoT paradigm (Adlakha et al., 2023) to deduce aspects based on text inputs and applies self-consistency (Wang et al., 2023) to select the most agreeable aspect for each text pair. However, this model is costly due to the extensive use of LLM API calls for every pair of tasks.
The offline aspect model extracts aspects from a collection of text corpora, adapting the concept from Pang et al. (2021) but employing LLMs. Specifically, this model prompts an LLM to extract aspects from each text within the collected corpora (50 pairs of texts for each task in this paper). It then prompts an LLM to refine and consolidate all generated aspects. Finally, we identify five fixed aspects as to use directly for any text pair of one task. This aspect model significantly reduces costs by allowing offline refinement of aspects. Refined aspects are fixed, thus they can be directly used without any additional expense.
In our experimental studies, we report only the best results for both baselines and in this section. To understand the impact of the aspect model, we detail its effects in our analysis section.
Comparison Model We use PaLM-2-L as the major LLM backbone of comparison model of SC2 to produce intermediate structured representations.
Preference Model For the final text preference prediction model, we experiment with two other LLM backbones differing in their model capacity. We aim to prove that the intermediate structured representations produced by SC2 with PaLM-2-L can help any backbone LLMs to predict text preference more accurately, regardless of their capacity. Specifically, we have used OpenAI’s GPT-3.5, and
GPT-4 (OpenAI, 2023) in our experiments.
Prompting Templates Prompts used in different models can be found in our Appendix. Note that we do not tailor the preference model’s prompts; instead, we adapted the templates from Rafailov et al. (2023) for a fair comparison across baselines2.
Hyperparameters As SC2 searches for the best comparisons during the inference stage, as a result, we have a hyperparameter |C|, referring to the number of samples generated by the comparison model. |C| is an important parameter that might affect the quality of the intermediate structured representation produced by . For the reported results in this section, we set |C| = 8. We study the influence of this hyperparameter in Sec 6.
Baselines For evaluation, we consider several baselines, primarily focused on the LLM-based prompting approaches. Below is a detailed overview of these baselines: (1) Direct Prompting (DP): This method directly prompts LLMs to predict text preference. (2) DP w/Aspects: This approach is a variation of DP. The difference is that DP w/Aspects incorporates aspects generated by the aspect models. (3) CoT-0-shot: This baseline utilizes a standard CoT-0-shot template for task preference prediction (with "let’s think step by step"). More details of the prompt template are available in the appendix. (4) CoT-1-shot: In addition to zero-shot prompting, we also carry out experiments using a 1-shot example within the CoT paradigm. For that purpose, we craft our 1-shot examples across different datasets. (5) CoT-SelfCon: This baseline integrates selfconsistency to CoT-0-shot baseline aiming to remove the arbitrariness.
Specifically, CoT-SelfCon first samples multiple responses from an LLM using the same prompt and text pair input. Subsequently, CoT-SelfCon aggregates all responses to identify the most frequent answer. In our experimental studies, we set the number of sample responses to 8 and use a majority vote to determine the desired response, randomly selecting a response in the event of a tie.
Datasets (1) TL;DR (Stiennon et al., 2020): We use OpenAI’s filtered Reddit and CNN/Daily Mail TL;DR dataset. OpenAI also created a preference
2In the original DPO paper (Rafailov et al., 2023), the authors did not use the Anthropic-Harmless dataset, we adapted their templated for Harmless datasets.
dataset from this, where labelers rated two generated summaries per post. For the CNN/Daily Mail part, for a given piece of news, we extracted two graded summaries and used the overall score to create the label. More details can be found in the original paper. (2) RLAIF-HH (Bai et al., 2022): The RLAIFHH from Anthropic dataset comprises dialogues from interactions between crowdworkers and large language models. In these exchanges, workers either seek assistance or provoke potentially harmful responses from the AI. The responses are then labeled based on their helpfulness or harmfulness. (3) TREC News (Soboroff et al., 2018): The TREC News dataset contains query-document pairs focused on ad-hoc ranking and filtering tasks from the late 1980s to early 2000s. We modify the dataset as follows for preference prediction: for a given query, we extract two document answers to construct the triplet and use the relevance score provided by the original dataset to decide which document is more preferred.
Dataset Sampling As datasets that have been used in the past are in large volumes, we only sampled a small ratio of them due to the cost of running all experiments. We sample roughly 250-300 data points from each dataset uniformly. More details of the sampled dataset can be found in Table 1
Metrics We report the accuracy of all approaches in our experiment ( Correctly Predicted InstancesAll Instances ) to measure the performance.

heading: 5 Results
text: Experimental results in Table 2 demonstrate SC2’s strong performance across all evaluation domains, with average gains of ∼ 2.5 and ∼ 7.0 points over the top and bottom baselines respectively. This confirms the benefits of structured comparative reasoning for enhanced text preference prediction. Using structured intermediate representations produced
by SC2, the preference prediction model better handles these comparative reasoning difficulties.
Moreover, we observe the input length as an additional factor impacting performance. For instance, the TREC News dataset comprises considerably longer texts than other corpora. Here, the DP method lags behind SC2 by over 9 points, compared to the average 7 point deficit across baselines. Though input length serves as an imperfect proxy for complexity, the results also signaled the potential benefit of using our method for longer inputs.
We also want to point out that SC2 could be further improved by coupling with some of the existing general prompting techniques, for example, self-consistency (Wang et al., 2023) and selfverification (Madaan et al., 2023).

heading: 6 Analysis
text: To further understand the benefit of using SC2 to produce an intermediate structured representation, in this section, we conduct ablation studies and indepth analysis. We also implement a user study to explore the potential of using SC2 to inform human beings’ decisions.

heading: 6.1 Effectiveness of Pairwise Comparator
text: To calibrate the effective gain arising from the pairwise comparator l′, we first compare variants of SC2 with the comparators and those with different hyperparameter configurations of SC2. We use different intermediate structured representations
produced by variants of SC2 to predict the text preference. Results are shown in Figure 3.
With |C| = 1, where there is effectively no pairwise comparator l′, the performance of the preference model was found to be comparable to baseline results shown in Table 2. This suggests that inconsistent structured representations could potentially degrade the performance of the preference model. An increase in accuracy was observed with larger values of |C|, indicating the benefits of pairwise comparator. However, this improvement plateaued when |C| exceeded 8, hinting at a potential ceiling effect for our approach, irrespective of further increases in |C|.

heading: 6.2 Impact of Different Aspect Models
text: To understand the effect of different aspect models, we conduct ablation studies comparing the baseline that used aspects and SC2 with aspect models proposed in our experimental study.
Table 3 presents the results. It shows that SC2
with the offline aspect model consistently outperforms or performs as well as SC2 with the online aspect model. However, for the DP w/Aspects baseline, neither the online nor the offline aspect model demonstrates superiority over the other. This indicates that SC2 does not require online LLM calls which dynamically generate aspects and can effectively utilize the offline aspect model to obtain aspects for the given task at pretty low cost.
6.3 Cost Analysis of SC2 and Few-shot CoT-SelfCon
As discussed in our experimental study, CoTSelfCon has no pairwise comparator components, resulting in lower LLM usage. On the other hand, in our primary experimental studies, we utilize PaLM-2-L to create intermediate structured representations and other LLM for preference prediction to avoid potential overfitting. In contrast, the CoTSelfCon baseline consistently employs the same LLM (GPT-4) all the way.
To ensure a fair comparison and eliminate biases that might arise from using different LLMs and # total LLM calls, we only use GPT-4 for both SC2 and CoT-SelfCon in this analysis. We use a fixed number of total LLM calls, including the generation of intermediate structured representations and
the final preference prediction. We limit our experiments to a single dataset with 100 samples and average the results over 5 rounds for the cost consideration. The results are shown in Table 4. Our analysis indicates that with the same # total LLM calls and the same LLM backbone, SC2 predicts preference consistently more accurately.
Furthermore, we evaluate against few-shot CoTSelfCon, commonly regarded as a strong baseline. Given that SC2 is in a zero-shot setting in our experiments, for a fair comparison, we compare few-shot SC2 with few-shot CoT-SelfCon, varying # LLM calls and # few-shot examples. We limit # few-shot examples to 5. This makes sure the context length is within the LLM’s length window.
Results are shown in Table 5. When the total LLM calls are low, CoT-SelfCon maintains a slight advantage over SC2. However, as the number of LLM calls increases, SC2 consistently outperforms few-shot CoT-SelfCon with the margin widening. This trend is attributed to the necessity for pairwise comparators to produce logically consistent intermediate-structured representations, leading to more accurate predictions.

heading: 6.4 Efficiency of Tournament Approach
text: We study the efficiency and effectiveness of the tournament approach w.r.t. other inference methods. Random Selection refers to the process of randomly selecting one sample from C during the inference stage, while Exact Search involves running all possible comparisons, which takes O(n2). We measure the cost using the total input length and the number of LLM calls, as this is common practice for the actual cost calculation in commer-
cial Large Language Models (LLMs). We used the same dataset from the previous subsection.
We find a significant gap between the Random Selection approach and the other two approaches as shown in Table 6. Although Exact Search yields the best results, it requires 4 times the token length and 49 more LLM calls, potentially leading to a substantial increase in cost.

heading: 6.5 Human Evaluation
text: We conduct additional human evaluations to see how the intermediate structured representations produced by SC2 inform human decision-making.
Annoators We recruit our annotators from an internal pool. Demographic and geographic characteristics of the annotator population are not accessible to our researchers. Information can be used to identify annotators that are fully anonymized. Consent forms have to be signed by annotators to take part in this study.
Study Design In consideration of ethical standards and the requirement to avoid directly testing annotators, we structure our human evaluation as follows: Annotators are presented with a query alongside a pair of text options, denoted as (←,→). They determine which text, either← or →, is preferable. They have three options: ← is
better,→ is better, and tie. Following their initial decision, annotators are then shown the intermediate structured representations generated by different variants of SC2. They decide if this additional information leads them to reconsider their initial choice and provide reasons for any change in their decision. This evaluation process uses two variants of SC2: |C| = 1 and |C| = 8 respectively. For ethical considerations, we only experiment with RLAIF-helpfulness and TL;DR-Reddit, ensuring the content is not harmful or violent manually. We instantiate 100 data points for each dataset and assign each question to three annotators. We collect 96 and 98 questions with useful responses from all three annotators for RLAIF-helpful and TL;DRReddit respectively.
Metrics We use the ground truth to gather the scores: we assign 1 for any correct answer, 0 for a tie, and −1 for any other incorrect answers. Findings As shown in Figure 4, with the aid of more consistent intermediate structured representations (|C| = 8), annotators are inclined to revise and flip their previous wrong answers to correct ones. This suggests that the intermediate structured representation may facilitate better decision-making among human evaluators. However, we also observe that intermediate structured representations without using a pairwise comparator (|C| = 1) could mislead annotators, deterring them from selecting the correct preference. This amplifies the importance of the pairwise comparator to ensure consistency.
We also look into quantitative justifications provided by annotators. Most annotators stated that intermediate structured representations helped them better understand two texts. One mentioned, "the table gives the concise comparison", while another pointed out, "this [table]
helped me to understand better the implications of the two answers, and I changed my mind after reading [the table]". Besides, we also observe complaints about the intermediate structured representations being hallucinatory and not factual. The issue is more noticeable in cases where the structured representation is produced by SC2 without a pairwise comparator. This suggests that enforcing a pairwise comparator might mitigate the arbitrariness of LLM’s output for better consistency, but still poses the risk of presenting hallucinated results to annoators.

heading: 7 Conclusion
text: This paper presents SC2, a structured comparative reasoning model for improving text preference prediction. SC2 constructs intermediate structured representations to explicitly contrast text pairs, incorporating a consistency comparator to enhance accuracy and coherence. Comprehensive experiments across text summarization, retrieval, and response rating tasks demonstrated that SC2 significantly improves consistency and achieves state-of-the-art performance. Analyses confirm the effectiveness of SC2’s structured reasoning approach and consistency enforcement. Our human evaluations show that SC2 interpretations can assist users in making informed decisions.

heading: 8 Limitations
text: This work has several limitations that provide opportunities for future investigation. First, the evaluation was conducted on a sample set of datasets that, while spanning diverse domains, might not fully characterize the breadth of real-world textual comparison needs. Expanding SC2’s testing to larger, multilingual corpora is essential to assess its full potential and limitations beyond English. Furthermore, there are likely upper bounds on SC2’s effectiveness imposed by the reasoning capacity of the underlying language model backbone. As more advanced LLMs emerge, exploring their integration could help quantify this ceiling effect. On a technical level, in this paper, measuring consistency relies on approximate metrics, so developing more rigorous evaluation schemes could better highlight SC2’s benefits. We also do not include other prompting techniques that have been well-studied in the community, which we leave for future work.

heading: 9 Ethical Considerations
text: This research paper might risk potential biases that could arise from textual comparisons, particularly around sensitive attributes. SC2 is trained on established corpora like Wikipedia and books that may inherently contain societal biases. While a full analysis of these biases is beyond the scope here, we acknowledge the risk that SC2 may inherit problematic biases from its training data. Applying recent advancements in language bias detection to SC2 could help quantify and mitigate these risks. We are interested in exploring this as part of future work. Furthermore, this research focused solely on English; extending to other languages is an important direction that would require non-trivial adaptation. Overall, while showing promise, SC2 has significant scope for improvement as limitations around evaluation, multilingual capabilities, consistency measurement, bias, and applied usage are addressed through future work.

heading: Acknowledgement
text: We thank Pengcheng Yin, Ethan Liang, Wenting Zhao, Celine Lee, Woojeong Kim, Jack Morris, and Junxiong Wang for their valuable suggestions and feedback. J.NY and AMR are supported by NSF CAREER #2037519, NSF III:#1901030, and NSF #2229873.

heading: Future Work
text: We are interested in exploring this as part of future work. Furthermore, this research focused solely on English; extending to other languages is an important direction that would require non-trivial adaptation. Overall, while showing promise, SC2 has significant scope for improvement as limitations around evaluation, multilingual capabilities, consistency measurement, bias, and applied usage are addressed through future work