QQSUM: A Novel Task and Model of Quantitative Query-Focused Summarization for Review-based Product Question Answering
Authors: An Quang Tang, Xiuzhen Zhang, Minh Ngoc Dinh, Zhuang Li
Accepted at: acl25

QQSUM: A Novel Task and Model of Quantitative Query-Focused Summarization for Review-based Product Question Answering

An Quang Tang and Xiuzhen Zhang * and Minh Ngoc Dinh and Zhuang Li
RMIT University, Australia
s3695273@rmit.edu.vn, xiuzhen.zhang@rmit.edu.au
minh.dinh4@rmit.edu.vn, zhuang.li@rmit.edu.au

Abstract

Review-based Product Question Answering (PQA) allows e-commerce platforms to automatically address customer queries by leveraging insights from user reviews. However, existing PQA systems generate answers with only a single perspective, failing to capture the diversity of customer opinions. In this paper we introduce a novel task Quantitative Query-Focused Summarization (QQSUM), which aims to summarize diverse customer opinions into representative Key Points (KPs) and quantify their prevalence to effectively answer user queries. While Retrieval-Augmented Generation (RAG) shows promise for PQA, its generated answers still fall short of capturing the full diversity of viewpoints. To tackle this challenge, our model QQSUM-RAG, which extends RAG, employs few-shot learning to jointly train a KP-oriented retriever and a KP summary generator, enabling KP-based summaries that capture diverse and representative opinions. Experimental results demonstrate that QQSUM-RAG achieves superior performance compared to state-of-the-art RAG baselines in both textual quality and quantification accuracy of opinions. Our source code is available at: https://github.com/antangrocket1312/QQSUM

1 Introduction

With the rapid expansion of e-commerce, consumers increasingly rely on product reviews to inform their purchasing decisions. Automatic review-based product question answering (PQA) systems have emerged, leveraging user reviews to provide immediate responses on e-commerce Q&A platforms (McAuley and Yang, 2016; Gupta et al., 2019). However, current PQA systems face a key limitation: they typically generate a single answer (Gupta et al., 2019), overlooking the fact that many subjective e-commerce queries require answers that reflect diverse viewpoints. For example, when comparing camera lenses (Figure 1), some shoppers prioritize versatility and affordability, while others focus on image quality and speed. Recent PQA approaches aim to improve answer quality using retrieval-augmented generation (RAG). These systems first retrieve reviews relevant to the query and then use them as context for large language models (LLMs) to generate answers. Yet, LLMs often struggle to present multifaceted perspectives (Sorensen et al., 2024), leading to answers that primarily reflect dominant opinions from the retrieved reviews (Deng et al., 2020, 2023).

Separately, opinion summarization has made progress through Key Point Analysis (KPA), which summarizes reviews into concise, representative statements called key points (KPs) while also quantifying their prevalence (Bar-Haim et al., 2020a,b, 2021; Tang et al., 2024a,b). However, these KPA methods focus on general summarization rather than answering specific queries. For tasks like product comparison, summarization must incorporate only query-focused KPs, making general KPA approaches insufficient for PQA.

![Comparison of conventional Q&A and QQSUM. More details of QQSUM output are in Table 12.](page_1012_1342_388_180.png)

Figure 1: Comparison of conventional Q&A and QQSUM. More details of QQSUM output are in Table 12.

In this paper, we introduce a novel task Quantitative Query-Focused Summarization (QQSUM), which generates comprehensive answers containing diverse KPs along with their quantified relative importance (Figure 1). Our solution, QQSUM-RAG, extends the RAG framework by integrating
KP-oriented retrieval and summarization. Specifically, QQSUM-RAG retrieves query-relevant reviews, clusters them by distinct opinions, and summarizes representative KPs from each cluster. This approach provides broader coverage of key insights, overcoming the single-perspective limitation of conventional RAG-based systems.

A key challenge in implementing this approach is scarcity of training data for such a specialized task. To address this, we develop a co-training strategy that jointly optimizes the retriever and LLM through shared supervision signals, enhancing the alignment between retrieved opinion clusters and generated KPs. This strategy enables robust performance of QQSUM-RAG even with limited training examples. To support few-shot learning, we carefully curated a dataset of queries with KPs and their prevalence quantification, through human-LLM collaboration. Empirical results show that QQSUM-RAG significantly outperforms RAG baselines based on in-context learning and quantitative summarization.

Our main contributions are:

• We introduce a novel task QQSUM. Unlike traditional PQA, QQSUM generates answers that capture diverse customer opinions with their prevalence, addressing queries that require multiple viewpoints.

• We propose QQSUM-RAG, a RAG-based framework with KP-oriented retrieval and summarization. The framework is optimized through a co-training strategy that improves alignment between retrieved opinion clusters and generated KPs in few-shot learning setting. Our experiments show that QQSUM-RAG significantly outperforms baselines with up to 2.11 times improvement in textual similarity with ground-truth KPs and up to 67.12% improvement in quantification performance over state-of-the-art KPA system for reviews (Tang et al., 2024b).

2 Related Work

2.1 Review-based PQA

Unlike domain-specific QA tasks such as biomedical or legal QA focusing on factual answers, review-based PQA seeks to provide answers of consumers’ subjective opinions about a product. While extractive PQA approaches retrieve relevant review snippets as answers (Chen et al., 2019a; Yu et al., 2012), it fails to provide precise responses since the review might not be specifically written for answering the given question. Recently, inspired by the advances of seq-2-seq models, abstractive, i.e., generation-based, approaches can generate natural-language answers from reviews (Chen et al., 2019c; Gao et al., 2019). However, these approaches frequently suffer from hallucinations and factual inconsistencies, sometimes generating random answers that misrepresent or contradict the prevalent opinions (Deng et al., 2020, 2023). Existing review-based PQA framework then cannot capture nor quantify faithfully the diverse opinions of reviews in its answer.

2.2 Key Point Analysis

Developed initially to summarize arguments (Bar-Haim et al., 2020a,b), KPA was later adapted for summarization of reviews (Bar-Haim et al., 2021; Tang et al., 2024a,b). While Bar-Haim et al. (2021) integrates sentiment analysis and collective key point mining to select and match KPs from broader domain with comments, Tang et al. (2024a) integrates aspect-based sentiment analysis (ABSA) into extracting and matching of KPs to comments for more unique KPs and precise quantification. More recent abstractive KPA studies apply abstractive summarization to paraphrase and generate KPs from comments (sentences) (Kapadnis et al., 2021; Li et al., 2023; Tang et al., 2024b). Overall, whether extractive or abstractive approaches, KPA can only produce KPs for general and high-level opinions without catering to specific queries.

2.3 Textual Summarization

Document summarization aims to produce concise textual summaries capturing the salient information in source documents. While extractive review summarization approaches use surface features to rank and extract salient opinions for summarization (Mihalcea and Tarau, 2004; Angelidis and Lapata, 2018; Zhao and Chaturvedi, 2020), abstractive techniques use sequence-to-sequence models (Chu and Liu, 2019; Suhara et al., 2020; Bražinskas et al., 2020b,a; Zhang et al., 2020a) to generate review-like summaries containing only the most prevalent opinions. Recently, prompted opinion summarization leveraging Large Language Models (LLMs) was applied to generate fluent and concise review summaries (Bhaskar et al., 2023). However, existing studies lack focus on presenting and quantifying the diverse opinions in reviews.
3 Quantitative Query-Focused Summarization

3.1 Task Formulation
Let q denote a query, i.e., community question, and \( R_e = \{ r_j \}_{j=1}^{|R_e|} \) denotes a set of review comments on a product e. QQSUM aims to retrieve relevant comments \( D \) to answer q and generate a KP-based summary S quantifying viewpoints presented in \( D \). We formulate \( S = \{ kp_1, \ldots, kp_n \} \) as a bullet-point summary containing multiple KPs, where each bullet-point represents a KP ¹ and its prevalence (Bar-Haim et al., 2021). For instance, with the bullet-point “23 comments praise that the headphone is very comfortable for long hours”, the KP is “Comfortable for long hours”, and the prevalence count is 23. Each key point \( kp_i \), is matched to a subset of supporting comments \( C_i = \{ c_1, c_2, \ldots \} \) (where \( c_i \in D \)), with prevalence being measured as \( |C_i| \).

3.2 The QQSUM-RAG Framework
Figure 2 illustrates the architecture of QQSUM-RAG. QQSUM-RAG is based on the retrieval-augmented generation (RAG) paradigm and consists of 2 stages: KP-Oriented Retrieval and KP Summary Generation. It utilizes a Retriever to retrieve and cluster query-relevant comments into groups, and the LLM to generate the final KP summary based on the comment clusters. Importantly, the retriever and LLM can be jointly trained with shared supervision signals to ensure comment clusters retrieved match KPs generated.

The following general loss function describes every training step of QQSUM-RAG, whose parameters are updated at the cluster-KP level rather than the query level:

\[
\mathcal{L} = (1 - d) \cdot (\mathcal{L}_{clus} + \text{gold\_score}) + d \cdot \mathcal{L}_{gen}
\]

where \( \mathcal{L}_{clus} \) is the retrieval loss for each comment cluster, \( \mathcal{L}_{gen} \) is the LLM’s generation loss computed for the KP generated from the respective cluster, and d is a damping factor to balance between the two. Notably, gold_score represents the Perplexity Distillation loss (Izard et al., 2023), which transforms the supervisory signals of the LLM to improve the Retriever. The intuition is that within a cluster, comments that better contribute to helping the LLM generate the KP with lower perplexity should be ranked higher.

¹unique and non-overlapping opinion at high level

3.2.1 KP-Oriented Retrieval
Given a query q, the Retriever should retrieve relevant review comments \( R_q \) that emphasize opinions focused on q. We utilize a shared encoder E that can encode both the input query q and each review comment \( r_j \in R_e \). Comments are ranked by the similarity score \( s(x, r_j) = \mathbf{E}_c(x)^T \mathbf{E}_d(r_j) \) that is calculated by taking the dot product of the embeddings of the query x and the comment \( r_j \). Only comments with \( s(x, r_j) \geq 1 \) is selected for \( R_q \).²

Different from standard RAG where generation is based on the direct retrieval result, to ensure diverse and representative opinions for generation, we enhance the Retriever with the clustering objective to produce distinctive comment groups that conceptually match KPs for generation.

KP-Oriented Retrieval Loss  Starting with an empty list of clusters C, and iterate through every comment in \( R_q \), for every comment, we further iterate through every existing cluster \( c_i \in C \) and calculate its average cosine similarity score to all comments of the cluster. Finally, we add the comment to any clusters with average cosine similarity score above a threshold (\( \lambda = 1.2 \)),³ otherwise, a new cluster is created. Importantly, a comment can be mapped to multiple clusters. We empirically showed that our proposed clustering algorithm is more effective than HDBSCAN (McInnes et al., 2017) and K-Means through an ablation experiment in Appendix K.

To train the retriever for KP-oriented retrieval, we align predicted comment clusters C with annotated clusters P, where P groups comments matched to the same KP (annotation details in §3.3). The centroid embedding of a cluster is the mean embedding of its comments \( r_k \): \( \bar{\mathbf{E}}_c(c_i) = \frac{1}{M} \sum_{k=1}^M \mathbf{E}(r_k) \). Because a cluster \( c_i \in C \) may contain mixed opinions represented by multiple clusters from P, we map each \( c_i \) to the mean embedding of \( P_{match} \subset P \): \( \bar{\mathbf{E}}_c(P_{match}) = \frac{1}{M} \sum_{j=1}^M \bar{\mathbf{E}}_c(p_j) \), where the semantic similarity between \( c_i \) and every \( p_j \) is \( sim(c_i, p_j) = \bar{\mathbf{E}}_c(c_i)^T \bar{\mathbf{E}}_c(p_j) \geq threshold \). The training objective minimizes the mean-squared-error (MSE) loss between each comment \( r_k \) in \( c_i \) and the average center of the most similar clusters \( P_{match} \).

\[
\mathcal{L}_{clus} = \frac{1}{|c_i|} \sum_{k=1}^{|c_i|} \| \bar{\mathbf{E}}_c(P_{match}) - \mathbf{E}(r_k) \|_2^2.
\]

²the similarity threshold 1 is set empirically
³set empirically based on cluster quality
3.2.2 KP Summary Generation
A key limitation of previous KPA studies is that KPs may contain redundant opinions, due to that review comments, possibly containing multiple opinions, are mapped to individual KPs locally (Bar-Haim et al., 2021; Tang et al., 2024b). To address this limitation, we propose to generate KPs at the global level, where the goal is to generate an overall KP-based summary without redundancy. Our main idea is that generated KPs are used as the context for the LLM to better reason and generate the next KP, which should be a unique, non-overlapping opinion statement.

Prompting Strategies Following OpenAI’s prompt engineering guidelines4, we format query-relevant comment clusters from the Retriever into a structured prompt with four parts (detailed in Listing 3, Appendix F): 1) Context and input structure, 2) Task definition and output requirements, 3) Summarization steps for identifying representative KPs per cluster and generating the final KP-based summary, and 4) Commonsense quantification rules to prioritize clusters by size and prevent overlapping KPs. To minimize ambiguity and hallucination, we encode predicted clusters C as JSON objects and assign each a unique ID, requiring the LLM to label generated KPs accordingly.

Next-KP-Generation Training During training, generating multiple KPs in a summary lacks alignment with \( \mathcal{L}_{clus} \), which is computed per comment cluster. To address this, we introduce a Next-KP-Generation objective, inspired by Next-Token Prediction in LMs (Brown et al., 2020), to enhance the generation of salient, non-overlapping KPs. This approach fine-tunes the LLM to iteratively generate KPs within the summary. Specifically, let the final KP-based summary \( S = \{kp_1, \ldots, kp_i, \ldots, kp_n\} \), each \( kp_i \) is generated with preceding KPs \( \{kp_1, \ldots, kp_{i-1}\} \) as the context, prompting the LLM to iteratively complete \( S \). The generation loss for each \( kp_i \) of \( c_i \in C \) is computed as the negative log-likelihood (NLL) against the reference KP, annotated for the most similar \( p_i \in P \) identified during retrieval,

\[
\mathcal{L}_{gen} = -\frac{1}{T} \sum_{t=1}^{T} \log P(x_t|x_{<t})
\]

where \( P(x_t|x_{<t}) \) represents the probability assigned by the model to the correct token \( x_t \), given the preceding tokens \( x_{<t} \).

3.3 Human-LLM Key Point Annotation
From Section 3.2, to train our QQSUM-RAG framework in the few-shot setting, annotation of KPs for queries and relevant comments are necessary. Prior KPA studies only include annotations matching comments to KPs without queries (Bar-Haim et al., 2020a,b). No datasets exist for matching comments to KPs in PQA.

Figure 2: The training architecture of the QQSUM-RAG framework.

Figure 3: Illustration of the human-LLM collaborative annotation pipeline for AMAZONKP.

4https://platform.openai.com/docs/guides/prompt-engineering
<table>
  <tr>
    <th>Statistic</th>
    <th>Train</th>
    <th>Test</th>
  </tr>
  <tr>
    <td># Product Categories</td>
    <td>17</td>
    <td>17</td>
  </tr>
  <tr>
    <td># Instances (queries) Per Category</td>
    <td>2</td>
    <td>148</td>
  </tr>
  <tr>
    <td>Total Instances</td>
    <td>34</td>
    <td>2516</td>
  </tr>
  <tr>
    <td># Reviews Per Query</td>
    <td>71.18</td>
    <td>72.70</td>
  </tr>
  <tr>
    <td># Review Comments Per Query</td>
    <td>452.03</td>
    <td>431.62</td>
  </tr>
  <tr>
    <td># KPs Per Query</td>
    <td>7.53</td>
    <td>6.45</td>
  </tr>
  <tr>
    <td># KPs Per Query (Stage 1)</td>
    <td>6.36</td>
    <td>7.96</td>
  </tr>
  <tr>
    <td># Relevant Comments Per Query (Stage 2)</td>
    <td>24.50</td>
    <td>--</td>
  </tr>
  <tr>
    <td># Comments (Prevalence) per KP (Stage 2)</td>
    <td>6.37</td>
    <td>--</td>
  </tr>
  <tr>
    <td>Summary Length (Stage 3)</td>
    <td>101.29</td>
    <td>--</td>
  </tr>
</table>

Table 1: Core statistics of the AMAZONKP dataset.

We leverage the popular PQA dataset AmazonQ&A (Gupta et al., 2019) for our QQSUM task, focusing on only answerable, subjective (non-factual) questions that have multiple answers. Out of 17 product categories (e.g., Electronics, Video Games), we only include businesses with 50-100 reviews, and sampling top 150 questions per category based on answer count. For ease of reference we name this curated dataset AMAZONKP. Details on question classification for AMAZONKP are in Appendix A, and their taxonomy in Appendix B. Notably, the dominance of “Scenario-based” questions underscore the importance of QQSUM for generating KP summary to answer user questions on preferences and scenarios.

Manually summarizing and quantifying opinions from comments is laborious and time-consuming, if not impossible. Research shows LLM’s strong annotation capabilities (He et al., 2024), and so we design a three-stage human-LLM collaborative annotation pipeline, shown in Figure 3.

Stage 1: KP Extraction from Gold Community Answers Given a query \( q_i \), the AmazonQ&A dataset provides multiple answers, i.e. responses, from online users \( A_i = \{a_1, a_2, ...\} \), serving as ideal approximation of gold opinions. However, these responses can contain overlapping opinions. We therefore zero-shot prompted GPT4-o-mini to extract distinctive and non-overlapping KPs from \( A_i \). Empirical validation with human annotators confirms that the extracted KPs are of high quality, with 90% of community answers were represented by KPs, while 87.5% of the extracted KPs are verified as valid (precision). Further details are in Appendices C and D.

Stage 2: LLM-based and Manual Comment-KP Matching Based on the annotation process in the literature (Bar-Haim et al., 2020a), we further integrate LLMs to reduce human effort and time. Using KPs extracted from gold answers (Stage 1), we prompt GPT4-o-mini to annotate pairwise matches between comments and KPs from all available reviews of the product. LLM-matched pairs are then validated by three Amazon Mechanical Turk (MTurk) workers. Finally, comments from validated pairs are grouped by similar KPs, with KP prevalence determined by the number of matching comments. Further details on KP Matching annotations are provided in Apppendix E.

Stage 3: KP-based Summary We utilize KPs and their prevalence counts, discovered for every query, to manually compose a bullet-point KP-based summary, where each bullet point corresponds to a KP and is annotated as “\( |kp_i| \) comments say that \( kp_i \)”.
The number of pairwise comment-KP matching annotations required per query can be up to 2K-3.5K. For training, to control annotation costs, we conducted Stages 1, 2 and 3 annotations on a small subset of 34 instances for few-shot training of QQSUM-RAG, randomly selecting two queries per product category for supervised labeling. For evaluating the KP-based summary, the remaining examples with only Stage 1 annotations serve as the test set. The core statistics of AMAZONKP are shown in Table 1.

4 Experiments

We employ Atlas (Izacard et al., 2023), a pre-trained efficient RAG model, as our backbone model for QQSUM-RAG. We utilized Contriever (Izacard et al., 2022) as the retriever while replacing the original language model with open-source LLMs (e.g., Vicuna-7B 5, Mistral-7B 6) for generation. For computational feasibility, we apply Low-Rank Adaptation (LoRA) (Hu et al., 2021), which adds trainable parameters while freezing the model’s original weights.

4.1 Baselines

We benchmark QQSUM-RAG against 3 RAG baselines.

(Retriever + LLM)\(_{\text{co-train}}\) We few-shot trained Atlas (Izacard et al., 2023), with the standard RAG architecture and Retriever-LLM generator co-training, for the QQSUM task. The retriever retrieves relevant comments, while letting the LLM implicitly infer KPs’ matching comments and their

5https://huggingface.co/lmsys/vicuna-7b-v1.5
6https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2
quantities during KP summary generation. For training, we aggregated matching comments across KPs, per query, as the retrieval ground truth.

Frozen Retriever + Prompt LLM To assess in-context learning (ICL) for QQSUM, we use a frozen retriever and Vicuna-7B, Mistral-7B, and GPT-4-Turbo as the LLM for ICL. Few-shot training instances are concatenated with test instances, with the number of few-shot examples optimized for context length and cost: 4-shot for Mistral-7B and GPT-4-Turbo, and 2-shot for Vicuna-7B.

Frozen Retriever + KPA We replace the LLM of a standard RAG with existing KPA review summarization systems to adapt KPA to the QQSUM task. Comments were first retrieved by a frozen retriever and then RKPA-Base (Bar-Haim et al., 2021) utilizes a quality ranking model (Gretz et al., 2020) to extract KP candidates before matching comments to KPs using a KP Matching model (Bar-Haim et al., 2020b) at threshold \( t_{match} = 0.99 \). PAKPA (Tang et al., 2024b) clusters comments by aspect and sentiment before generating aspect-oriented KPs.

All experiments were conducted at the KP level, focusing on KPs in the summary outputs of QQSUM-RAG and baselines for fair comparison. We post-process the output KP-based summary into KPs as JSON objects, where each object covers the KP information of a bullet point in the summary.7 The baselines were implemented using either the PyTorch module or the Huggingface transformers framework, and were trained on a NVIDIA GeForce RTX 4090 GPU.

4.2 Evaluation Dimensions

We conducted experiments on the test set of AMAZONKP (\S3.3), consisting of questions from 17 product categories. For reasonable cost, we sample 8 questions from each category for evaluation.

4.2.1 KP Textual Quality

Automatic Evaluation Extracted KPs from gold community answers in AmazonKP (Stage 1 of \S 3.3) serves as the reference KPs for this automatic evaluation. We first perform a lexical comparison between KPs in the generated summary and the ground truth by computing the highest Rouge (Lin, 2004) score between generated and reference KPs for each query and then average the maxima. Then, following Li et al. (2023), we calculate soft-Precision/Recall/F1 (denoted by sP, sR and sF1, respectively), which measure the semantic similarity between individual generated KP and the reference KP. While sP finds the reference KP with the highest similarity score for each generated KP, sR is vice-versa, and (sF1) is the harmonic mean between sP and sR.

\[
sP = \frac{1}{n} \times \sum_{\alpha \in \mathcal{A}} \max_{\beta \in \mathcal{B}} f(\alpha_i, \beta_j)
\]
(4)

\[
sR = \frac{1}{m} \times \sum_{\beta \in \mathcal{B}} \max_{\alpha \in \mathcal{A}} f(\alpha_i, \beta_j)
\]
(5)

Additionally, inspired by sP/sR/sF1 of Li et al. (2023), we further propose RD to identify KP redundancy. For each generated KP in the summary for a query, RD finds the neighborhood KP with the highest similarity score.

\[
RD = \frac{1}{n} \times \sum_{\alpha \in \mathcal{A}} \max_{\theta \neq \alpha \in \mathcal{A}} f(\alpha_i, \theta_j)
\]
(6)

where \( f \) computes similarities between two individual key points, \( \mathcal{A}, \mathcal{B} \) is the set of generated and reference KPs and \( n = |\mathcal{A}| \) and \( m = |\mathcal{B}| \), respectively. We use state-of-the-art semantic similarity metrics BLEURT (Sellam et al., 2020) and BERTScore (Zhang et al., 2020b), along with LLM-based metric G-EVAL-4 (Liu et al., 2023) as \( f_{max} \). Note that G-EVAL scores are scaled from 1-5 to 0-1 for comparability and its evaluation prompt was also customized to fit our evaluation (Appendix G).

Human Evaluation We manually evaluated the information quality of generated KPs in the summary considering 7 different dimensions utilized in previous KPA studies (Kapadnis et al., 2021; Tang et al., 2024b), including REDUNDANCY, COVERAGE, FAITHFULNESS VALIDITY, SENTIMENT, INFORMATIVENESS and SINGLE ASPECT. Details of these dimensions are in Appendix H.

We conducted pairwise comparisons of KPs from different systems using Amazon Mechanical Turk (MTurk). Given a dimension for evaluation, each comparison involved choosing the better one from two summaries, each taken from a different system. Using the Bradley-Terry model Friedman et al. (2021), we calculated rankings from these comparisons among the models. For an example of
<table>
  <tr>
    <th></th>
    <th>P@5</th>
    <th>P@10</th>
    <th>P@20</th>
    <th>P@all</th>
  </tr>
  <tr>
    <th colspan="5">QQSUM-RAG (Ours)</th>
  </tr>
  <tr>
    <td>Contriever (Izacard et al., 2022)</td>
    <td>0.668</td>
    <td>0.633</td>
    <td>0.601</td>
    <td>0.535</td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td>0.567</td>
    <td>0.527</td>
    <td>0.526</td>
    <td>0.367</td>
  </tr>
  <tr>
    <td>+ Vicuna</td>
    <td>0.590</td>
    <td>0.538</td>
    <td>0.500</td>
    <td>0.440</td>
  </tr>
  <tr>
    <td>all-MiniLM-L12-v2 (Wang et al., 2021)</td>
    <td>0.569</td>
    <td>0.507</td>
    <td>0.468</td>
    <td>0.362</td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td>0.544</td>
    <td>0.511</td>
    <td>0.459</td>
    <td>0.345</td>
  </tr>
  <tr>
    <td>+ Vicuna</td>
    <td>0.444</td>
    <td>0.467</td>
    <td>0.442</td>
    <td>0.328</td>
  </tr>
  <tr>
    <td>(Retriever + LLM)<sub>co-train</sub> (Izacard et al., 2023)</td>
    <td>0.531</td>
    <td>0.530</td>
    <td>0.515</td>
    <td>0.350</td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td>0.552</td>
    <td>0.512</td>
    <td>0.454</td>
    <td>0.339</td>
  </tr>
  <tr>
    <th colspan="5">frozen Retriever + prompt LLM</th>
  </tr>
  <tr>
    <td>Contriever</td>
    <td>0.494</td>
    <td>0.447</td>
    <td>0.404</td>
    <td>0.325</td>
  </tr>
  <tr>
    <td>all-MiniLM-L12-v2</td>
    <td>0.479</td>
    <td>0.446</td>
    <td>0.452</td>
    <td>0.315</td>
  </tr>
  <tr>
    <td>BM25</td>
    <td>0.469</td>
    <td>0.432</td>
    <td>0.387</td>
    <td>0.283</td>
  </tr>
</table>

Table 2: Performance of retrieval models.

an annotation, see Appendix I. Note that for reasonable cost, we sample and select only the popular question (with the highest average KP prevalence ), each from 5 common categories \(^8\) of AMAZONKP.

4.2.2 KP Quantification Performance

We evaluate the KP quantification performance of different systems for KP-comment matching and factual alignment.

KP-comment matching We first assess the accuracy of the KP comment matching by extending Bar-Haim et al. (2021) to measure both precision (correctness of predicted matches) and recall (coverage of ground-truth matches). For each system, we compute precision and recall by prompting gpt-4-o-mini to annotate pairwise match/non-match between generated KPs and retrieved comments \(R_q\). Additionally, leveraging annotated comment-KP pairs, we introduce QuantErr, which measures the mean absolute error between predicted and actual KP prevalence count. Empirical validation shows gpt-4-o-mini annotations highly correlated with MTurk workers’ judgement (Pearson’s \(r = 0.647\)) (Appendix J).

KP-comment factual alignment: We further employed AlignScore (Zha et al., 2023) for automatic evaluation of factual alignment between generated KPs and their corresponding comments.

4.3 Results

4.3.1 The Retrieval Model

The retriever is important for retrieving comments relevant to queries and so we first evaluated the performance of different backbone retrieval models.

For this we prompted gpt-4-o-mini to annotate the relevance of retrieved comments to queries. Table 2 reports the retrieval Precision@k (P@k), measured at different levels of top-k-ranked retrieved comments ([5, 10, 20, all]), using 3 different retrieval models: Contriever, all-MiniLM-L12-v2 and BM25. Note that BM25 is not a neural encoder and therefore can only be evaluated in the frozen Retriever + Prompt LLM setting.

Overall, the trained retriever of QQSUM-RAG, as being co-trained with the LLM and extended for KP-oriented retrieval, outperform all baselines. Notably, co-training with stronger LM can also contribute up to 45.78% improvement, as the supervision signal from more query-focused KP generation helps train the Retriever to rank documents more accurately. Contriever stood out as the best performer regardless of the LM selection. Hereafter we base all upcoming experiments with Contriever as the retrieval model.

4.3.2 KP Quality

KPs produced by different systems in terms of textual quality, semantic quality and redundancy are reported in Table 3. Scores of all systems are low in general, as opinions in product reviews may not cover all opinions from community answers to questions. From Table 3, QQSUM-RAG outperforms other systems in all quality dimensions. It shows 2.11 times improvement in textual similarity with reference KPs (0.256 vs. 0.121 in ROUGE-1), 0.23 point absolute improvement in semantic similarity (0.39 vs. 0.16 in BERTScore) and 0.14 point absolute reduction in Redundancy (0.37 vs. 0.51 using BERTScore for semantic similarity).

The high quality of KPs in QQSUM-RAG can be attributed to the KP-oriented retrieval of QQSUM-RAG. Notably, although (Retriever + LLM)<sub>co-train</sub> shares the same backbone model and co-training design with QQSUM-RAG, the lack of (1) opinion-level clustering of retrieved comments and (2) limited modeling capability of LLMs makes this model unable to produce KPs as diverse, unique and representative as QQSUM-RAG. The weak reasoning capability of LLMs for diverse opinion summarization is further exposed in the frozen Retriever + prompt LLMs setting, where LLMs even with strong modelling capability like GPT-4-Turbo struggle to elaborate diverse and distinctive KPs from hundreds of comments.

It is worth noting that Mistral~7B broadly exhibits higher performance than Vicuna~7B across
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="4">ROUGE</th>
    <th colspan="4">BERTScore</th>
    <th colspan="4">BLEURT</th>
    <th colspan="4">G-Eval-4</th>
  </tr>
  <tr>
    <th>R-1</th><th>R-2</th><th>R-L</th><th>sP</th>
    <th>sR</th><th>sF1</th><th>RDj</th>
    <th>sP</th><th>sR</th><th>sF1</th><th>RDj</th>
    <th>sP</th><th>sR</th><th>sF1</th><th>RDj</th>
  </tr>
  <tr>
    <th>QQSUM–RAG (Ours)</th>
    <td colspan="4">0.256 0.061 0.220 0.39</td>
    <td colspan="4">0.29 0.33 0.37 0.51</td>
    <td colspan="4">0.41 0.46 0.49 0.88</td>
    <td colspan="4">0.82 0.85 0.36</td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td colspan="4">0.222 0.078 0.204 0.38</td>
    <td colspan="4">0.26 0.31 0.53 0.49</td>
    <td colspan="4">0.39 0.44 0.54 0.87</td>
    <td colspan="4">0.81 0.84 0.36</td>
  </tr>
  <tr>
    <th>(Retriever + LLM)<sub>co-train</sub> (Izacard et al., 2023)</th>
    <td colspan="4">0.209 0.057 0.194 0.37</td>
    <td colspan="4">0.28 0.32 0.43 0.49</td>
    <td colspan="4">0.40 0.44 0.55 0.81</td>
    <td colspan="4">0.82 0.81 0.41</td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td colspan="4">0.174 0.041 0.161 0.37</td>
    <td colspan="4">0.26 0.31 0.48 0.48</td>
    <td colspan="4">0.38 0.42 0.58 0.78</td>
    <td colspan="4">0.78 0.78 0.41</td>
  </tr>
  <tr>
    <th>Frozen Retriever + prompt LLM</th>
    <td colspan="4">0.210 0.055 0.191 0.33</td>
    <td colspan="4">0.26 0.29 0.51 0.46</td>
    <td colspan="4">0.38 0.42 0.55 0.79</td>
    <td colspan="4">0.80 0.79 0.41</td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td colspan="4">0.164 0.059 0.154 0.22</td>
    <td colspan="4">0.20 0.21 0.48 0.46</td>
    <td colspan="4">0.31 0.37 0.59 0.73</td>
    <td colspan="4">0.73 0.73 0.41</td>
  </tr>
  <tr>
    <td>+ GPT-4-Turbo</td>
    <td colspan="4">0.197 0.048 0.174 0.32</td>
    <td colspan="4">0.25 0.28 0.44 0.45</td>
    <td colspan="4">0.38 0.41 0.54 0.77</td>
    <td colspan="4">0.77 0.77 0.38</td>
  </tr>
  <tr>
    <th>Frozen Retriever + KPA</th>
    <td colspan="4">0.179 0.027 0.162 0.34</td>
    <td colspan="4">0.28 0.31 0.46 0.47</td>
    <td colspan="4">0.41 0.44 0.54 0.79</td>
    <td colspan="4">0.80 0.80 0.36</td>
  </tr>
  <tr>
    <td>+ PAKPA (Tang et al., 2024b)</td>
    <td colspan="4">0.121 0.016 0.106 0.16</td>
    <td colspan="4">0.14 0.14 0.50 0.43</td>
    <td colspan="4">0.36 0.39 0.61 0.69</td>
    <td colspan="4">0.70 0.69 0.51</td>
  </tr>
  <tr>
    <td>+ RKPA-Base (Bar-Haim et al., 2021)</td>
    <td colspan="4">0.121 0.016 0.106 0.16</td>
    <td colspan="4">0.14 0.14 0.50 0.43</td>
    <td colspan="4">0.36 0.39 0.61 0.69</td>
    <td colspan="4">0.70 0.69 0.51</td>
  </tr>
</table>

Table 3: KP summary textual quality. sP, sR and sF1 refer to Soft-Precision, Soft-Recall, and Soft-F1 respectively based on set-level evaluation method against reference KPs in gold answer.

<table>
  <tr>
    <th></th>
    <th>CV</th><th>FF</th><th>RD</th><th>VL</th><th>SN</th><th>IN</th><th>SA</th>
  </tr>
  <tr>
    <th>QQSUM-RAG (Ours)</th>
    <td>28.44</td><td>26.56</td><td>25.34</td><td>35.23</td><td>31.11</td><td>25.9</td><td>24.8</td>
  </tr>
  <tr>
    <th>(Retriever + LLM)<sub>co-train</sub> (Izacard et al., 2023)</th>
    <td>11.06</td><td>11.17</td><td>14.2</td><td>9.99</td><td>9.54</td><td>13.49</td><td>17.52</td>
  </tr>
  <tr>
    <th>Frozen Retriever + prompt LLM (GPT-4-Turbo)</th>
    <td>15.12</td><td>12.84</td><td>15.7</td><td>10.36</td><td>14.6</td><td>12.59</td><td>10.79</td>
  </tr>
  <tr>
    <th>Frozen Retriever + PAKPA (Tang et al., 2024b)</th>
    <td>9.94</td><td>12.47</td><td>13.25</td><td>7.7</td><td>8.87</td><td>13.04</td><td>9.34</td>
  </tr>
  <tr>
    <th>Frozen Retriever + RKPA-Base (Bar-Haim et al., 2021)</th>
    <td>16.20</td><td>22.28</td><td>15.75</td><td>22.91</td><td>20.75</td><td>21.02</td><td>18.77</td>
  </tr>
</table>

Table 4: Human evaluation of KP information quality by different dimensions. Reported are the Bradley Terry scores of 7 dimensions, from left to right, COVERAGE, FAITHFULNESS and REDUNDANCY, VALIDITY, SENTIMENT, INFORMATIVENESS, SINGLEASPECT. For reasonable cost, we only conducted manual evaluation on Mistral - the best LM configuration of QQSUM-RAG and (Retriever + LLM)<sub>co-train</sub>, selected from Table 3.

all systems based on LLM generation and in all KP quality measurement (up to 15.32%), largely due to its stronger modeling capability.

Frozen Retriever + KPA baselines, despite their high performance for review summarization, is ineffective for QQSUM. Not surprisingly PAKPA, which generates KPs based on aspect-sentiment, broadly shows better performance than RKPA-Base, an extractive KPA system. It is possible that multiple query-relevant opinions on the same aspect are expected to answer a user query, thus leading to the weak performance of PAKPA.

Our manual evaluation of KP information quality further validates the above findings, as shown by the Bradley Terry scores in Table 4. Overall, QQSUM-RAG achieves up to 4.58 times improvements on all 7 dimensions, and are notably higher on COVERAGE (CV) (2.86 times), VALIDITY (VL) (2.38 times), and SENTIMENT (SN) (3.5 times).

4.3.3 KP Quantification

Table 5 presents the quantification performance for different systems. F1, combining Recall and Precision, measures the overall performance of KP-comment matching for all systems. QuantErr (lower the better) directly measures KP quantification errors. Overall, QQSUM-RAG shows the best performance in terms of both F1 (0.792 vs. 0.154) and QuantErr (4.24 vs. 30.13).

Comparing QQSUM-RAG against the Retriever+LLM generation systems, namely (Retriever + LLM)<sub>co-train</sub> and Frozen Retriever + prompt LLM, we can see that, without clustering comments, LLMs perform comment-KP matching and KP quantification, showing extremely low Recall (0.185–0.249), in contrast to the high Recall of QQSUM-RAG (0.684-0.869). This can be attributed to two main factors: (1) LLMs tend to hallucinate when generating KPs from a large set of retrieved comments, and (2) their limited context window restricts their ability to effectively match comments to KPs.

Comparing QQSUM-RAG against Retriever + KPA systems, our model shows up to 67.12% improvement in quantification performance over state-of-the-art KPA system for reviews (PAKPA) (Tang et al., 2024b), with a 36.53% reduction in QuantErr. Note that Frozen Retriever + PAKPA achieves the highest matching precision due to aspect-level opinion quantification. However, it has low recall, possibly because it relies on aspect-based sentiment analysis of comments, which can fail to identify implicit opinions not explicitly including aspects.

As shown in Table 4, results for KP-Comment Factual Alignment show that QQSUM-RAG and
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="4">KP-Comment Matching</th>
    <th colspan="2">KP-Comment Factual Alignment</th>
  </tr>
  <tr>
    <th>P</th>
    <th>R</th>
    <th>F1</th>
    <th>QuantErr.</th>
    <th>AlignScore</th>
  </tr>
  <tr>
    <td><b>QSUM-RAG (Ours)</b></td>
    <td>0.694</td>
    <td><b>0.869</b></td>
    <td><b>0.792</b></td>
    <td><b>0.424</b></td>
    <td><b>0.749</b></td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td>0.538</td>
    <td>0.684</td>
    <td>0.602</td>
    <td>0.783</td>
    <td>0.630</td>
  </tr>
  <tr>
    <td>(Retriever + LLM)<sub>co-train</sub> (Izacard et al., 2021)</td>
    <td>0.567</td>
    <td>0.249</td>
    <td>0.346</td>
    <td>18.10</td>
    <td>0.653</td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td>0.442</td>
    <td>0.094</td>
    <td>0.154</td>
    <td>30.13</td>
    <td>0.394</td>
  </tr>
  <tr>
    <td><b>Frozen Retriever + prompt LLM</b></td>
    <td>0.746</td>
    <td>0.200</td>
    <td>0.313</td>
    <td>16.63</td>
    <td>0.673</td>
  </tr>
  <tr>
    <td>+ GPT-4-Turbo</td>
    <td>0.498</td>
    <td>0.214</td>
    <td>0.300</td>
    <td>19.14</td>
    <td>0.624</td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td>0.439</td>
    <td>0.185</td>
    <td>0.260</td>
    <td>21.52</td>
    <td>0.531</td>
  </tr>
  <tr>
    <td><b>Frozen Retriever + KPA</b></td>
    <td>0.762</td>
    <td>0.520</td>
    <td>0.619</td>
    <td>06.68</td>
    <td>0.749</td>
  </tr>
  <tr>
    <td>+ PAKPA (Tang et al., 2024b)</td>
    <td>0.371</td>
    <td>0.314</td>
    <td>0.340</td>
    <td>15.62</td>
    <td>0.354</td>
  </tr>
  <tr>
    <td>+ RKPA-Base (Bar-Haim et al., 2021)</td>
    <td>0.371</td>
    <td>0.314</td>
    <td>0.340</td>
    <td>15.62</td>
    <td>0.354</td>
  </tr>
</table>

Table 5: Performance for KP-Comment matching and factual alignment

Frozen Retriever + KPA (PAKPA) achieve high factual correctness in KP generation, outperforming other systems (0.749 vs. 0.354). This result highlights that QSUM-RAG generates KPs grounded in the retrieved comments, and similarly PAKPA generates KPs grounded in aspects.

4.4 Ablation Study

We evaluate the contribution of Next-KP-Generation in QSUM-RAG, with results in Tables 10 and 11 (Appendix L). In particular, we configure a variant QSUM-RAG Single-KP that replaces Next-KP-Generation with KP generation for each comment cluster. Not including previously generated KPs as context, QSUM-RAG Single-KP struggles to capture the truly representative opinion of the cluster, likely generating KPs with overlapping opinions, especially for comments containing multiple opinions. Note that while its KP quality underperforms RAG baselines, its KP Quantification performance remain superior, largely attributed to KP-oriented Retrieval.

4.5 Case studies

We conducted case studies to evaluate the redundancy and specificity of generated KPs for a query comparing camera lenses, presented in Table 14 (Appendix M). Overall, QSUM-RAG stands out for generating KPs with minimal redundancy, high informativeness, and alignment with the query. First, QSUM-RAG reduces redundancy by effectively capturing distinct product features relevant to the user’s needs (e.g., faster aperture), whereas (Retriever + LLM)<sub>co-train</sub>, GPT-4-Turbo Prompt LLM, and PAKPA tend to generate repetitive and generic statements, such as “The 24-70mm f/2.8 is a better lens overall.” Furthermore, QSUM-RAG expands feature coverage, capturing details such as Vibration Reduction (VR) technology, which several baselines fail to mention.

4.6 Error Analysis

Our analysis on a KP summary of QSUM-RAG reports two systematic error patterns, as shown in Table 13. First, a KP can be falsely matched to comments expressing similar opinions but on different targets. For instance, the comment “For a lens that is overall a rather mixed bag ... it is very expensive.” was matched to KP “The 24-120mm F4 lens has a longer zoom range and is more affordable than the 24-70mm F2.8.”. Since the comment lacks an explicit product reference, it remains unclear whether it critiques the 24-120mm F4 or the 24-70mm F2.8. The second type of errors stems from the sentence-level quantification, where input review sentences often contain co-occurring multi-aspect opinions, making it difficult for the Retriever to isolate distinct aspects into separate clusters.

5 Conclusion

In this paper, we studied a new task Quantitative Query-focused Summarization, namely QSUM, for capturing and quantifying diverse opinions from online reviews for PQA. We propose QSUM-RAG, a few-shot summarization model based on retrieval-augmented generation where summary is generated by LLMs from groups of user opinions relevant to a query. QSUM-RAG addresses the issue of existing RAG frameworks for providing only random or major opinion in the answer. By extending the retriever with opinion-based clustering of relevant comments, our model ensures capturing more diverse and representative opinions in the summary, along with accurate quantification. Experimental results show that our solution greatly enhances both the quality and quantitative performance of key point generation in summaries.
Acknowledgement

This research is supported in part by the Australian Research Council Discovery Project DP200101441.

Limitations

We evaluated the textual quality of generated KPs only on AmazonQ&A, as it is the only (to our best knowledge) public dataset with abundance of online community answers written by online users usable as ground truth for our automatic evaluation.

Since we are leveraging answers from AmazonQ&A to summarize and quantify the prevalence of query-relevant opinions from reviews regarding a query, an inevitable limitation is that key points extracted from the Q&A answers might not fully in line with viewpoints in reviews to answer questions. Similarly, opinions in product reviews also may not sufficiently cover all opinions in community answers.

Ethics Statement

We have applied ethical research standards in our organization for data collection and processing throughout our work.

The AmazonQ&A dataset used in our experiments was publicly crowdsourced and released for the research publication for the review-based product question answering task (Gupta et al., 2019). The dataset was published following their ethical standard, after removing all personal information. The answers to questions do not contain contents that are harmful to readers.

We ensured fair compensation for crowd annotators on Amazon Mechanical Turk. We setup and conducted fair payment to workers on their annotation tasks/assignments according to our organization’s standards, with an estimation of the difficulty and expected time required per task based on our own experience. Especially, we also made bonus rewards to annotators who exerted high-quality annotations in their assignments.

References

Stefanos Angelidis and Mirella Lapata. 2018. Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3675–3686, Brussels, Belgium. Association for Computational Linguistics.

Roy Bar-Haim, Lilach Eden, Roni Friedman, Yoav Kantor, Dan Lahav, and Noam Slonim. 2020a. From arguments to key points: Towards automatic argument summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4029–4039, Online. Association for Computational Linguistics.

Roy Bar-Haim, Lilach Eden, Yoav Kantor, Roni Friedman, and Noam Slonim. 2021. Every bite is an experience: Key Point Analysis of business reviews. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3376–3386, Online. Association for Computational Linguistics.

Roy Bar-Haim, Yoav Kantor, Lilach Eden, Roni Friedman, Dan Lahav, and Noam Slonim. 2020b. Quantitative argument summarization and beyond: Cross-domain key point analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 39–49, Online. Association for Computational Linguistics.

Adithya Bhaskar, Alex Fabbri, and Greg Durrett. 2023. Prompted opinion summarization with gpt-3.5. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9282–9300.

Ralph Allan Bradley and Milton E. Terry. 1952. RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS: THE METHOD OF PAIRED COMPARISONS. Biometrika, 39(3-4):324–345.

Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020a. Few-shot learning for opinion summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4119–4135, Online. Association for Computational Linguistics.

Arthur Bražinskas, Mirella Lapata, and Ivan Titov. 2020b. Unsupervised opinion summarization as copycat-review generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5151–5169, Online. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
Kathy Charmaz. 2015. Grounded theory. Qualitative psychology: A practical guide to research methods, 3:53–84.

Long Chen, Ziyu Guan, Wei Zhao, Wanqing Zhao, Xiaopeng Wang, Zhou Zhao, and Huan Sun. 2019a. Answer identification from product reviews for user questions by multi-task attentive networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 45–52.

Qibin Chen, Junyang Lin, Yichang Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang. 2019b. Towards knowledge-based personalized product description generation in e-commerce. In KDD 2019, pages 3040–3050.

Shiqian Chen, Chenliang Li, Feng Ji, Wei Zhou, and Haiqing Chen. 2019c. Driven answer generation for product-related questions in e-commerce. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 411–419.

Eric Chu and Peter Liu. 2019. Meansum: A neural model for unsupervised multi-document abstractive summarization. In International Conference on Machine Learning, pages 1223–1232. PMLR.

Yang Deng, Wenxuan Zhang, and Wai Lam. 2020. Opinion-aware answer generation for review-driven question answering in e-commerce. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 255–264.

Yang Deng, Wenxuan Zhang, Qian Yu, and Wai Lam. 2023. Product question answering in E-commerce: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11951–11964, Toronto, Canada. Association for Computational Linguistics.

Roni Friedman, Lena Dankin, Yufang Hou, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2021. Overview of the 2021 key point analysis shared task. In Proceedings of the 8th Workshop on Argument Mining, pages 154–164, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Shen Gao, Zhaochun Ren, Yihong Zhao, Dongyan Zhao, Dawei Yin, and Rui Yan. 2019. Product-aware answer generation in e-commerce question-answering. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pages 429–437.

Shai Gretz, Roni Friedman, Ed Cohen-Karlil, Assaf Toledo, Dan Lahav, Ranit Aharonov, and Noam Slonim. 2020. A large-scale dataset for argument quality ranking: Construction and analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7805–7813.

Mansi Gupta, Nitish Kulkarni, Raghveer Chanda, Anirudha Rayasam, and Zachary C Lipton. 2019. Amazonqa: A review-based question answering task. arXiv preprint arXiv:1908.04364.

Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2024. AnnoLLM: Making large language models to be better crowdsourced annotators. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pages 165–190, Mexico City, Mexico. Association for Computational Linguistics.

Edward Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning.

Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1–43.

Manav Kapadnis, Sohan Patnaik, Siba Panigrahi, Varun Madhavan, and Abhilash Nandy. 2021. Team enigma at ArgMining-EMNLP 2021: Leveraging pre-trained language models for key point matching. In Proceedings of the 8th Workshop on Argument Mining, pages 200–205, Punta Cana, Dominican Republic. Association for Computational Linguistics.

J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174.

Hao Li, Viktor Schlegel, Riza Batista-Navarro, and Goran Nenadic. 2023. Do you hear the people sing? key point analysis via iterative clustering and abstractive summarisation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14064–14080, Toronto, Canada. Association for Computational Linguistics.

Piji Li, Zihao Wang, Lidong Bing, and Wai Lam. 2019. Persona-aware tips generation? In WWW 2019, pages 1006–1016.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Linguistics.
Julian McAuley and Alex Yang. 2016. Addressing complex and subjective product-related queries with customer reviews. In Proceedings of the 25th International Conference on World Wide Web, pages 625–635.

Leland McInnes, John Healy, Steve Astels, et al. 2017. hdbscan: Hierarchical density based clustering. J. Open Source Softw., 2(11):205.

Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404–411, Barcelona, Spain. Association for Computational Linguistics.

Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics.

Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, et al. 2024. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.05070.

Yoshihiko Suhara, Xiaolan Wang, Stefanos Angelidis, and Wang-Chiew Tan. 2020. OpinionDigest: A simple framework for opinion summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5789–5798, Online. Association for Computational Linguistics.

An Tang, Xiuzhen Zhang, and Minh Dinh. 2024a. Aspect-based key point analysis for quantitative summarization of reviews. In 18th Conference of the European Chapter of the Association for Computational Linguistics.

An Tang, Xiuzhen Zhang, Minh Dinh, and Erik Cambria. 2024b. Prompted aspect key point analysis for quantitative review summarization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10691–10708, Bangkok, Thailand. Association for Computational Linguistics.

Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021. MiniLMv2: Multi-head self-attention relation distillation for compressing pretrained transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2140–2151, Online. Association for Computational Linguistics.

Jianxing Yu, Zheng-Jun Zha, and Tat-Seng Chua. 2012. Answering opinion questions on products by exploiting hierarchical organization of consumer reviews. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 391–401, Jeju Island, Korea. Association for Computational Linguistics.

Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348, Toronto, Canada. Association for Computational Linguistics.

Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020a. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328–11339. PMLR.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

Chao Zhao and Snigdha Chaturvedi. 2020. Weakly-supervised opinion summarization by leveraging external information. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9644–9651.

A Opinionated Question Classification for AMAZONKP Dataset

Existing online product-related questions can be categorized into two groups: subjective (opinionated) or objective (factual). While subjective questions asks about positive/negative feeling or stance (e.g., whether a product is “good” or “bad”), objective questions confirms the actual product details (e.g., products properties, specific use-cases). In E-Commerce, questions are often subjective, i.e., asking for former buyer’s opinion, where different customers often have certain preferences over product aspects or information needs (Chen et al., 2019b; Li et al., 2019), leading to various expectations for the provided answers.

We extract subjective, i.e., opinionated, question from AmazonQ&A by prompting the Mistral-7B open-source LLM to analyze the question and its associated answers, published by the online community. In this case, leveraging answers helps to understand the nature of the questions, thereby better reasoning whether the question is seeking for subjective information from users. We present the few-shot prompt for classifying opinionated, i.e., subjective, questions from AmazonQ&A in Listing 1.
Listing 1: Few-shot prompt (2 examples) for prompting Mistral-7B on opinionated question classification.
You will be provided with a question and multiple answers to that question, delimited by triple quotes.
The question was taken from a Question Answering dataset of product reviews, and can either be an opinionated or factual question.

You were tasked to classify whether the given question is an opinionated or factual question.
Factual questions ask for objective data, specifications, or information that can be definitively answered based on product facts , manual, user experience, or specifications. Factual question tends to covers unique and consistent opinions/fact in its answers.
Opinionated questions are subjective and seek insights that are based on personal use, feelings, preferences, judgments, or evaluations about a product. Opinionated question has multiple and diverse opinions in its answers.

Formally, you should perform the following steps:
1. Identify unique opinions from the answers of the given question
2. Based on the question content and the amount of opinions in the question’s answer, identify the question's type.

Note that you must briefly explain why the question is opinionated or factual before giving the final decision.

Below are some few–shot examples:

Questions: How well does it work with wireless charging
Answers: ['Unfortunately with this case installed it will not hold the phone vertically.', 'I use the case with the official wireless charger and have had no problems at all.', 'Works great. Not a fan of the dimensions.']
Type: 'Opinionated Question'

Questions: Are the shelves removeable?
Answers: ['yes, they are removeable.', 'Yes they are, you can arrange them for the size of the shot glass.']
Type: 'Factual Question'

B Qualitative Data Analysis of Opinionated Questions’ Categories in AMAZONKP

We further studied the utility of the QQSUM task and our by conducting qualitative data analysis to categorize possible opinionated question’s type in AMAZONKP. Based on the grounded theory methodology (Charmaz, 2015), our analysis employ human-LLM collaborative annotation to iteratively code the fine-grained categories from opinionated questions. We sampled a subset of 100 questions from AMAZONKP for data coding and interpretation. On the subset, we start by prompting ChatGPT to identify potential categories of opinionated questions, including the categories’ name and their definitions (Step 1). Importantly, the data coding process involves human validation, in which we iteratively a human annotator iteratively evaluate the representative of generated categories while interacting with ChatGPT, and manually refine the categories where possible 9 (Step 2). Then, we prompted a gpt-4-o-mini to annotate the labels of entire questions in the subset, before asking human annotator again to validate the representative and suitability of the candidate categories on questions. Categories with abnormal distribution, e.g., 5 times higher than others, or with high unmatching cases will be passed back to Step 2 for another iterative analysis cycles.

As a result, our analysis reported 5 categories commonly representative of question in AMAZONKP, namely, Performance, Quality, Recommendation, Comparative and Controversial, with each the stating clearly the purpose of the users asking the questions and expected answers. Finally, We prompted gpt-4-o-mini to annotate such categories on AMAZONKP’s opinionated questions, and reported their taxonomy and statistics in Table 6. Notably, the dominance of “Scenario-based” questions underscore the importance of QQSUM for generating KP summary to answer user questions on preferences and scenarios.

C Human Validation of GPT4’s Key Point Extraction from Gold Community Answer of AmazonQ&A

In this experiment, we empirically validate gpt-4-o-mini’s performance and credibility in extracting KPs from gold community answers for AmazonKP (Stage 1 of §3.3). Specifically, to maintain reasonable cost, we sampled a question,

9On categories requiring more fine-grained categorization, we further conduct another analysis cycle on the particular coarse-grained category, by selecting questions and answers from the specific category for analysis.
<table>
  <tr>
    <th>Category</th>
    <th>Description</th>
    <th>Example</th>
    <th># Query</th>
  </tr>
  <tr>
    <td>Performance</td>
    <td>Ask how well a product performs or functions in general.</td>
    <td>How well does it work on carpet?</td>
    <td>376</td>
  </tr>
  <tr>
    <td>Quality</td>
    <td>Ask about the overall or aspect-specific quality of the product.</td>
    <td>Is this product worth the money?</td>
    <td>265</td>
  </tr>
  <tr>
    <td>Scenario-based</td>
    <td>Ask whether a product fits specific use cases, sizes, or other products.</td>
    <td>Does this item really stop the glare at night even in rain or snow?</td>
    <td>1402</td>
  </tr>
  <tr>
    <td>Recommendation</td>
    <td>Ask for suggestions tailored to specific issues or use cases.</td>
    <td>What do you use to spray this stuff on your lawn?</td>
    <td>156</td>
  </tr>
  <tr>
    <td>Comparative</td>
    <td>Seeks opinions about the relative advantages or disadvantages of a product compared to others.</td>
    <td>Would a wired keyboard/mouse be better than wireless?</td>
    <td>227</td>
  </tr>
  <tr>
    <td>Controversial</td>
    <td>Reflect dissatisfaction or complaint about a product, likely to provoke debate or controversy.</td>
    <td>Why does this need adjustment screws? If I have to align the laser then what's the point?</td>
    <td>124</td>
  </tr>
</table>

Table 6: A taxonomy of opinion questions AmazonKP

i.e., queries, from 5 common product categories of AmazonKP \(^{10}\), totaling 5 questions, and hired workers to annotate whether the extracted KPs matches original gold community answers of the sampled questions, which is inspired by the KP Matching evaluation of Bar-Haim et al. (2021). More specifically, for a given query, we asked workers to perform pairwise annotation between extracted KPs and the query’s respective community answers. While *Precision* calculates the fraction of KPs matched to at least one gold answer, i.e., out of all extracted KPs how many are correctly mapped, *Recall* shows the fractions of gold answers matched to at least one KP, i.e., out of all answers how many are covered by KPs. We then macro-averaged Precision/Recall computed for every question to obtain the final values.

For human annotation, we employed 3 MTurk crowd workers on every answer-KP pair, selecting only those with an 80% or higher approval rate and at least 10 approved tasks. Following Bar-Haim et al. (2021), we exclude annotators with Annotator-\( \kappa < 0 \) for quality control. This score averages all pairwise Cohen’s Kappa (Landis and Koch, 1977) for a given annotator, for any annotator sharing at least 50 judgments with at least 2 other annotators. For labelling correct matches, we applied a strict threshold, in which 100% votes (3 out of 3) of the annotators had to agree that the match was correct. Otherwise, it is incorrect.

Table 7 presents the fraction of extracted KPs matched to at least one gold answer (Precision) and vice versa (Recall). Overall, the experiment confirms that the extracted KPs are of high quality, with 90.0% of community answers were represented with KPs (recall), while 87.5% of the extracted KPs are verified as valid (precision).

<table>
  <tr>
    <th>Precision</th>
    <td>87.5%</td>
  </tr>
  <tr>
    <th>Recall</th>
    <td>90.0%</td>
  </tr>
  <tr>
    <th># Matched Answers Per KP</th>
    <td>2.39</td>
  </tr>
  <tr>
    <th># Matched KPs Per Answer</th>
    <td>2.61</td>
  </tr>
</table>

Table 7: Performance validation of gpt-4-o-mini’s KP extraction from gold community answer. While precision calculates the fraction of KPs matched to at least one gold answer, recall shows the fractions of gold answers matched to at least one KP.

Below are the match annotation guidelines for (extracted KP, gold answer) pairs:

In this task you are presented with a question on a product, a key point extracted from community answers answering the question, and a community answer for answering the query of that product.

You will be asked to answer the following question:"Does the key point match, i.e., represent an opinion in the community answer?"

A community answer might express opinions on multiple aspects. A key point matches a community answer if it captures the gist of the answer, or is directly supported by a point made in the community answer.

The options are:

• Not At All

• Somewhat Not Well

• Somewhat Well

• Very Well
D Prompt for Key Point Extraction from Gold Community Answer of AmazonQ&A

We present the few-shot prompts for extracting key points (KPs) from gold online community answers of AmazonKP in Listing 2.

E Annotation Details of KP Matching for AMAZONKP Dataset

We offer GPT-4-o-mini with 4 options for labelling the matching status of given comment-KP pairs. Pairs annotated as Very Well or Somewhat Well by LLM then becomes candidate matching pairs, which will be further validated by human annotation for their correctness. For human annotation, we employed 3 MTurk crowd workers per comment-KP pair, selecting only those with an 80% or higher approval rate and at least 10 approved tasks. Following Bar-Haim et al. (2021), we exclude annotators with Annotator-\( \kappa < 0 \) for quality control. This score averages all pairwise Cohen’s Kappa (Landis and Koch, 1977) for a given annotator, for any annotator sharing at least 50 judgments with at least 2 other annotators. For labelling correct matches, at least 60% of the annotators had to agree that the match is correct; otherwise, it is incorrect. Comments from final matching pairs, after confirmed by human, will then be grouped by similar KPs, where the amount of matching comments per KP is the prevalence of the respective KP.

Below are the matching prompt for LLM and the annotation guidelines for workers validating (sentence, KP) pairs:

In this task, you are presented with a question on a product, a key point taken from the summary answering the question, and a sentence taken from a review of that product.

You will be asked to answer the following question: "Does the key point match, i.e, represent an opinion in the review sentence?"

A review sentence might express opinions on multiple aspects. A key point matches a sentence if it captures the gist of the sentence, or is directly supported by a point made in the sentence.

The options are:

• Not At All
• Somewhat Not Well
• Somewhat Well
• Very Well

F Prompts for KP Summary Generation of QQSUM-RAG

We present the instruction-finetuning prompts for KP Summary Generation of QQSUM-RAG in Listing 3.

G Prompts for G-EVAL Evaluation

For implementation of G-EVAL in our KP quality evaluation dimension (\S4.2), we specifically customize the model’s original prompt for evaluating summary’s relevance and redundancy. While the relevance evaluation prompt is customized for evaluating sP/sF/sF1 (Li et al., 2023) between individual generated KPs and the reference KPs, redundancy is customized for evaluating \( RD \) among generated KPs. We presented our relevance evaluation prompt in Listing 4 and the redundancy evaluation prompt in Listing 5

H Dimensions of KP Quality Evaluation

This section provides detailed descriptions of tasks and dimensions involved in our manual evaluation of the KP textual quality. Annotators were asked to perform a pairwise comparison between two sets of KPs, each taken from a different model, generated for a specific reviewed business entity considering a specific dimension. The annotators must answer a comparative question with respect to the evaluating dimension. (e.g., *Which of the two summaries captures better . . .*). For each dimension, following Friedman et al. (2021), we calculate the ranking using the Bradley-Terry model (Bradley and Terry, 1952), which predicts the probability of a given participant winning a paired comparison, based on previous paired comparison results of multiple participants, and thus allows ranking them.

• VALIDITY: The key point in the summary should be an understandable, well-written sentence representing an opinion of the users towards the question. This would filter out sentences such as “*It’s rare these days to find that!*”.

• SENTIMENT: The key point in the summary should have a clear sentiment towards the product being questioned (either positive or
Listing 2: One-shot prompt (1 example) for prompting GPT-4-o-mini on KP Extraction from community answers.
You will be provided with an opinionated question and multiple answers to that question, delimited by triple quotes.
An opinionated question seek insights of user opinions that are based on personal use, feelings, preferences, judgments, or evaluations about a product, and was taken from a Question Answering dataset of product reviews.

You were tasked to extract a list of unique and concise key points from the list of answers to given opinionated question.
Key points are short and high quality sentences that expresses the main claims/viewpoints of users answering the opinionated question
Note that the final extracted list of key points must directly relevant and can answer the input opinionated question.

Formally, you should perform the following steps:
1. In every answer from the list, extract all possible key point candidates.
2. From the extracted list of key point candidates, generate a list of only general and non–overlapping key points that are relevant and can answer the input opinionated question.

Below are some few–shot examples:

Questions: Can I use these for running/working out? Do they handle sweat?
Answers: [I have seen other people using these for running/working out. These are very comfortable in your ears for long hours. As long you clean them after working out, you should be fine. These are built to last a long time., 'I use them in the gym and on the stair climber machine. They are fine. Not sure about running but would think they would work ok.', "I don't know if I'll be any help, but I'll tell you about my experience nevertheless. I used it everyday in the gym & while I go for work on my bike inside my helmet. In both cases, the sweat doesn't seem to have any effect. Even during long rides, and when it rained heavily, the IE80 held up fine. The only issue you will have to worry about is the cable.
Though the cables are good quality, rough usage may affect the balance in volume levels between the two channels.
Though this doesn't affect the clarity, the balance can be disturbed. After a year of really rough usage, the IE80 right volume was 1–2% lower than the left [I got mine replaced for free soon after]. But, this is an issue which affects every IEM, and nothing to sweat over, since we can replace the cables if necessary. So if you don't give it a hard time, it should hold up fine.[I can't even count the times it has fallen down or swung down and taken a hit against the gym equipment, or when my phone/DAP slipped and yanked the cable]"]
Key Points: ['Comfortable for long hours', 'Built to last a long time', 'Suitable for gym and stair climber machine', 'Sweat resistant during workouts', 'Cables may be affected by rough usage']

Listing 3: Prompt for instruction-finetuning QQSUM-RAG’s LLM for KP Summary Generation. Please refer to our released code for full prompts.
You will be provided with a question and a JSON list of relevant review comments, delimited by triple quotes.
The question asks the opinions of user reviews about a product, and can be answered by the list of comment clusters in the provided JSON list. Each element in the JSON has been has been clustered to represent a common opinion answering the question, accompanied by the quantity.

You were tasked to generate a quantitative summary that covers all opinions captured in the JSON list in answering the questions.

Perform the following actions to solve this task:
– For every element in the JSON list, find the key point that represent the common opinion across the comments of the cluster
– Generate a long–form quantitative summary including all extracted key points and the cluster size, following the below template:
'While answering about [Question]:
+ [Cluster size] of comments believe that [Key Point 1]
+ [Cluster size] of comments believe that [Key Point 2]
...'
Below are fundamental rules:
+ Larger cluster means higher support for the key point and with a bigger cluster size, the quantity must be higher
+ Only use number to report the cluster size for each key point, avoiding vague terms (e.g., some, most)
+ Ensure that each key point extracted from a cluster is distinctive and doesn't redundantly cover aspects mentioned in larger clusters
Listing 4: Zero-shot prompt for G-EVAL relevancy evaluation between generated KPs and reference KPs, supporting sP/sR/sF1 calculation.

You will be given one key point, short salient sentence, written to describe user opinion on a product.

Your task is to rate the summary on one metric.

Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:

Relevance (1–5) – selection of important content from the source. The summary should include only important information from the source document. Annotators were instructed to penalize summaries which contained redundancies and excess information.

Evaluation Steps:

1. Read the key point and the source key point carefully.
2. Compare the key point to the source key point and identify the main points.
3. Assess how well the key point covers the main points of the source key point, and how much irrelevant or redundant information it contains.
4. Assign a relevance score from 1 to 5.

Listing 5: Zero-shot prompt for G-EVAL redundancy evaluation of generated KPs, supporting \( RD \) calculation.

You will be given one key point, short salient sentence, written to describe user opinion on a product.

Your task is to rate the summary on one metric.

Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:

Redundancy (1–5) – overlapping opinion with the source. The summary should not include semantically similar opinion with the source document. Annotators were instructed to penalize summaries which contained overlapping opinion with the source.

Evaluation Steps:

1. Read the key point and the source key point carefully.
2. Compare the key point to the source key point and identify the main points.
3. Assess how much redundant opinion and information the key point covers that overlap with the source key point
4. Assign a redundancy score from 1 to 5.
negative). This would exclude sentences like “I came for a company event”.

• INFORMATIVENESS: The key point in the summary should discuss should discuss some aspects of the reviewed product and contain useful information. Any key point that is too specific or only expresses sentiment cannot be considered a good candidate.

• SINGLEASPECT: The key point in the summary should not discuss multiple aspects (e.g., “Decent price, respectable portions, good flavor”).

• REDUNDANT: Each KP should express a distinct aspect. In other words, there should be no overlap between the key points.

• COVERAGE: The summary, containing the set of key points, should cover a wide diversity of opinions relevant and representative to the question.

• FAITHFULNESS: The key point in the summary should express reasonable and meaningful opinions relevant to the question raised on the product without hallucination. No conjecture or unfounded claims should arise.

I Pairwise KP Quality Comparison Annotation Guidelines

Below are the two summaries for a product question in Tools_and_Home_Improvement, generated by two different summarization frameworks. Each summary contains several key points (i.e., salient points) generated summarizing the user opinions on different aspects. You are tasked to select which summary you think is better according to the below criteria.

Question: Does this tester accurately test AA Lithium? The power drop off curve is so steep. It seems unlikely...but I am hoping!.

Criteria: REDUNDANCY. Each key point in the summary should express a distinct aspect. In other words, there should be no overlap between the key points.

Summary A: [’the tester accurately tests various types of batteries, including AA Lithium, and provides accurate readings’, ’there is uncertainty about the accuracy of the percentage of charge remaining for AA Lithium batteries’, ’the tester does not test a specific version of AA Lithium battery (L91)’, ’the tester is big and cumbersome, but effective in testing batteries under load’, ’the tester requires four AA batteries to operate’, ’the tester tests batteries by putting a load on them, making the readings more accurate’, ’the tester tests batteries quickly, with a test taking only 3-4 seconds for a AA battery’, ’the tester is expensive but worth the investment due to its accuracy and ability to save money by testing old batteries’, ’the tester tests batteries of various sizes, including AA, AAA, C’]

Summary B: [’I have compared the testers results to battery powered devices and found it does give you the true useful state of a battery. ’, ’Now that I found this tester I am happy, because it tests a battery the way a battery should be tested.’, ’That model also tests 6v lithium 2CR5 used in some older cameras, which the current tester does not since the times have moved on.’]

The options are:

• Summary A

• Summary B

J GPT4’s Comment-KP Matching Annotation against Human Judgement

To validate gpt-4-o-mini’s annotation performance and credibility, we conduct an experiment to measure LLM annotation judgement, as utilized for the KP-comment matching evaluation in our main experiment, in agreement with human (gold) preference. We sampled a subset of 5 queries from the test set in our main experiment and hired workers to annotate the correctness of comment-KP pairs produced as the results of our framework’s quantification outcome. Note that these sampled pairs are part of the our main test set and have already been annotated for LLM’s labels in our main experiment. For human annotation, we employed 6 MTurk crowd workers on every comment-KP pair, selecting only those with an 80% or higher approval rate and at least 10 approved tasks. Following Bar-Haim et al. (2021), we exclude annotators with Annotator-\( \kappa < 0 \) for quality control. This score averages all pairwise Cohen’s Kappa (Landis and Koch, 1977) for a given annotator, for any annotator sharing at least 50 judgments with at least 5 other annotators. For labelling correct matches, at least 60% of the annotators had to agree that the match is correct, otherwise, it is incorrect. In this experiment, we measured the accuracy, and conducted a Pearson correlation (\( r \)) test of gpt-4-o-mini’s
annotation performance against human judgement, with results reported in Table 8. For \( r \) test, we set the null hypothesis as gpt-4-o-mini’s and Mturk annotated labels are independent.

From Table 8, we saw significant small p-value, which indicates strong evidence against the null hypothesis. Importantly, we also recorded Spearman’s rank correlation coefficient to be relatively closed to 1. This implies that there is a statistically significant positive correlation between gpt-4-o-mini and Mturk annotated labels, which substantiates our decision of using gpt-4-o-mini for comment-KP matching evaluation.

<table>
  <tr>
    <th>Pearson correlation (\( r \))</th>
    <td>0.647</td>
  </tr>
  <tr>
    <th>p_value</th>
    <td>5.342e-16</td>
  </tr>
  <tr>
    <th>Accuracy</th>
    <td>0.807</td>
  </tr>
</table>

Table 8: Performance valuation of GPT4’s comment-KP matching annotation against human judgement

Below are the match annotation guidelines for (sentence, KP) pairs:

In this task, you are presented with a question on a product, a key point taken from the summary answering the question, and a sentence taken from a review of that product.

You will be asked to answer the following question: "Does the key point match, i.e, represent an opinion in the review sentence?"

A review sentence might express opinions on multiple aspects. A key point matches a sentence if it captures the gist of the sentence, or is directly supported by a point made in the sentence.

The options are:

• Not At All

• Somewhat Not Well

• Somewhat Well

• Very Well

K Clustering Algorithm of KP-Oriented Retrieval in QQSUM-RAG

To validate other clustering techniques, we have developed an additional baseline that employs either HDBSCAN (McInnes et al., 2017) or K-Means clustering algorithm for grouping similar comments by the Retriever, following our main experimental setup and configuration in Section 4.2.2.

Better than K-Means, HDBSCAN can automatically detect the number of clusters without pre-defined parameters and is used in a previous KPA work (Li et al., 2023). We compare the factual alignment of KP-comment pairs (measured by AlignScore) across clustering methods in Table 9, using our best model configuration (Contriever + Mistral):

While both HDBSCAN and K-Means perform reasonably, they are consistently outperformed by our specialized clustering approach. More specifically, although HDBSCAN or K-Means achieves relatively comparable matching Precision with our clustering algorithm, our algorithm can capture comments more sufficiently (much higher Recall) than HDBSCAN and K-Means. This is mostly because our algorithm contains more tuneable clustering parameters and operations that are specifically optimized for the QQSUM problem.

L Ablation Study: Single-KP Generation vs KP Summary Generation in QQSUM-RAG

We conducted an ablation study to evaluate the impact of KP Summary Generation on QQSUM-RAG, with KP quality and KP-comment matching and factual consistency performance presented in Table 10 and 11 respectively. To this end, we configure QQSUM-RAG Single-kp, a variant that generates one KP at a time for each comment cluster formed by KP-oriented Retrieval.

Overall, not including previously generated KPs as context, QQSUM-RAG Single-kp struggles to capture the truly representative opinion of the cluster, likely generating KPs with overlapping opinions, especially for comments containing multiple opinions.

M Example output of QQSUM-RAG and Baselines

We report the example output of query-relevant comment clusters and KP summary produced by QQSUM-RAG in Table 12 and 13, and further compare top 5 key points, extracted from the summary of QQSUM-RAG and the baselines in Table 14. Overall, QQSUM-RAG stands out for generating KPs with minimal redundancy, higher informativeness, and better alignment with the query.
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="5">KP-Comment Matching</th>
    <th colspan="3">KP-Comment Factual Alignment</th>
  </tr>
  <tr>
    <th>P</th>
    <th>R</th>
    <th>F1</th>
    <th>QuantErr.</th>
    <th>AlignScore</th>
    <th>AlignScore</th>
    <th>(cluster-level)</th>
    <th>(retrieval-level)</th>
  </tr>
  <tr>
    <td>Our proposed clustering algorithm</td>
    <td><b>0.694</b></td>
    <td><b>0.869</b></td>
    <td><b>0.792</b></td>
    <td><b>04.24</b></td>
    <td><b>0.749</b></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>HDBSCAN clustering algorithm</td>
    <td>0.682</td>
    <td>0.507</td>
    <td>0.582</td>
    <td>11.47</td>
    <td>0.718</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>K-Means clustering algorithm (n_clusters = 3)</td>
    <td>0.677</td>
    <td>0.424</td>
    <td>0.522</td>
    <td>15.50</td>
    <td>0.681</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 9: KP-Comment matching performance and factual consistency of generated summary between different clustering methods applied for KP-oriented Retrieval of QSUM-RAG. The experiment was conducted with the Mistral configuration for QSUM-RAG, proven to have superior performance than Vicuna from Table 3.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="6">ROUGE</th>
    <th colspan="6">BERTScore</th>
    <th colspan="6">BLEURT</th>
    <th colspan="6">G-Eval-4</th>
  </tr>
  <tr>
    <th>R-1</th>
    <th>R-2</th>
    <th>R-L</th>
    <th>sP</th>
    <th>sR</th>
    <th>sF1</th>
    <th>RD<sub>1</sub></th>
    <th>Rel</th>
    <th>sP</th>
    <th>sR</th>
    <th>sF1</th>
    <th>RD<sub>1</sub></th>
    <th>Rel</th>
    <th>sP</th>
    <th>sR</th>
    <th>sF1</th>
    <th>RD<sub>1</sub></th>
    <th>Rel</th>
  </tr>
  <tr>
    <td>QSUM-RAG (Ours)</td>
    <td>0.256</td>
    <td>0.061</td>
    <td>0.220</td>
    <td>0.39</td>
    <td>0.29</td>
    <td>0.33</td>
    <td>0.37</td>
    <td>0.27</td>
    <td>0.51</td>
    <td>0.41</td>
    <td>0.46</td>
    <td>0.49</td>
    <td>0.45</td>
    <td>4.52</td>
    <td>4.29</td>
    <td>4.40</td>
    <td>2.43</td>
    <td>4.05</td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td>0.222</td>
    <td><b>0.078</b></td>
    <td>0.204</td>
    <td>0.38</td>
    <td>0.26</td>
    <td>0.31</td>
    <td>0.53</td>
    <td>0.25</td>
    <td>0.49</td>
    <td>0.39</td>
    <td>0.44</td>
    <td>0.54</td>
    <td>0.41</td>
    <td>4.47</td>
    <td>4.25</td>
    <td>4.36</td>
    <td>2.45</td>
    <td>3.68</td>
  </tr>
  <tr>
    <td>QSUM-RAG<sub>single-KP</sub></td>
    <td>0.191</td>
    <td>0.035</td>
    <td>0.160</td>
    <td>0.29</td>
    <td>0.22</td>
    <td>0.25</td>
    <td>0.48</td>
    <td>0.22</td>
    <td>0.48</td>
    <td>0.39</td>
    <td>0.43</td>
    <td>0.62</td>
    <td>0.39</td>
    <td>4.21</td>
    <td>4.22</td>
    <td>4.22</td>
    <td>2.51</td>
    <td>3.14</td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td>0.171</td>
    <td>0.043</td>
    <td>0.154</td>
    <td>0.22</td>
    <td>0.17</td>
    <td>0.19</td>
    <td>0.57</td>
    <td>0.20</td>
    <td>0.48</td>
    <td>0.38</td>
    <td>0.42</td>
    <td>0.66</td>
    <td>0.36</td>
    <td>4.10</td>
    <td>4.12</td>
    <td>4.11</td>
    <td>2.60</td>
    <td>2.87</td>
  </tr>
  <tr>
    <td>+ Vicuna</td>
    <td>0.171</td>
    <td>0.043</td>
    <td>0.154</td>
    <td>0.22</td>
    <td>0.17</td>
    <td>0.19</td>
    <td>0.57</td>
    <td>0.20</td>
    <td>0.48</td>
    <td>0.38</td>
    <td>0.42</td>
    <td>0.66</td>
    <td>0.36</td>
    <td>4.10</td>
    <td>4.12</td>
    <td>4.11</td>
    <td>2.60</td>
    <td>2.87</td>
  </tr>
</table>

Table 10: KP-level textual quality evaluation of generated summary between full implementation of QSUM-RAG and without (w/o) KP Summary Generation. sP, sR and sF1 refer to Soft-Precision, Soft-Recall, and Soft-F1 respectively based on set-level evaluation method against reference KPs in gold answer. G-EVAL-4 asks GPT-4 to score a summary from 1-5.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="5">KP-Comment Matching</th>
    <th colspan="2">KP-Comment Factual Consistency</th>
  </tr>
  <tr>
    <th>P</th>
    <th>R</th>
    <th>F1</th>
    <th>QuantErr.</th>
    <th>AlignScore</th>
    <th>AlignScore</th>
    <th>(cluster-level)</th>
    <th>(retrieval-level)</th>
  </tr>
  <tr>
    <td>QSUM-RAG (Ours)</td>
    <td><b>0.694</b></td>
    <td><b>0.869</b></td>
    <td><b>0.792</b></td>
    <td><b>04.24</b></td>
    <td><b>0.749</b></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td>0.538</td>
    <td>0.684</td>
    <td>0.602</td>
    <td>07.83</td>
    <td>0.630</td>
    <td>0.690</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>QSUM-RAG<sub>single-KP</sub></td>
    <td>0.640</td>
    <td>0.520</td>
    <td>0.574</td>
    <td>17.84</td>
    <td>0.682</td>
    <td>0.741</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>+ Mistral</td>
    <td>0.598</td>
    <td>0.471</td>
    <td>0.527</td>
    <td>22.63</td>
    <td>0.601</td>
    <td>0.660</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 11: KP-Comment matching performance and factual consistency of generated summary between full implementation of QSUM-RAG and without (w/o) KP Summary Generation.
Query: How does this Nikon 24-120mm F4 lens compared with the 24-70mm F2.8 as a general walk around lense?

<table>
  <tr>
    <th>Query-Relevant Comment Clusters</th>
    <td>
      <b>Cluster1:</b><br>
      • I like the 24-70 better but this lens is a good all around and compact optic for everyday shooting.<br>
      • As has been said many times before: "the best lens is the one you will use", and I know I wouldn not use the 24-70mm F2.8 because it's too heavy and bulky to take on backpacking/camping trips and when traveling abroad.<br>
      • This is the one lens which could replace 24-70 / 2.8, 70-200 2.8 VR II (up to some extent) for "everyday" use.<br>
      • …<br>
      <b>Cluster2:</b><br>
      • I have an upcoming stay in Spain, and I'm seriously considering taking this lens instead of my AF-S 24-70 because of its size and zoom range.<br>
      • My only complaint is the price tag: for a lens that is overall a rather mixed bag (depending on what you're looking for you might be very happy with it, or very disappointed) it is very expensive.<br>
      • The 24-120 has good reach, good image quality, not heavy, not that expensive for what it can do (constant f/4 in a zoom is very respectable) and it's also the only usable medium-telephoto FX zoom from Nikon with the VR technology.<br>
      • For a 5x zoom to be able to compete with a 3x zoom costing over $500 more(the Nikkor 24-70mm F2.8) should only mean that the 5x zoom is a remarkable lens.<br>
      • …<br>
      <b>Cluster3:</b><br>
      • For one thing, 24 70 is know to have better quality than this one.<br>
      • The range from 70 to 120 is not as important as a better overall quality.<br>
      • This is probably not the best lens to use for portraits because it's just not fast enough (f-stop), but for travel, chasing you kids around, or any other every day shooting this lens is perfect.<br>
      • The biggest pro for the 24-70mm is the extra 1 stop of light, slightly quicker autofocus speed, and of course the corresponding softer bokeh due to the 1 stop aperture opening.<br>
      • …
    </td>
  </tr>
</table>

<table>
  <tr>
    <th>KP Summary</th>
    <td>
      While comparing the Nikon 24-120mm F4 lens with the 24-70mm F2.8 lens as a general walk-around lens:<br>
      + 135 of comments believe that the Nikon 24-120mm F4 lens is relatively lightweight and compact, making it easy to carry around and use for extended periods of time.<br>
      + 11 of comments suggest that the 24-120mm F4 lens has a longer zoom range and is more affordable than the 24-70mm F2.8.<br>
      + 9 of comments prefer the 24-70mm F2.8 for its better image quality and faster aperture.<br>
      …
    </td>
  </tr>
</table>

Table 12: Example output of query-relevant comment clusters and KP summary produced by QQSUM-RAG, given a query, i.e., question, from AmazonQ&A. Comment clusters to a particular KP are marked in the same color as the corresponding bullet in the summary. The relevant opinion in each comment that directly support the corresponding KP is italicized.

<table>
  <tr>
    <th>Query: How does this Nikon 24-120mm F4 lens compared with the 24-70mm F2.8 as a general walk around lense?</th>
    <th>Key Point</th>
    <th>Prevalence</th>
    <th>Matching Comments</th>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>The Nikon 24-120mm F4 lens is relatively lightweight and compact, making it easy to carry around and use for everyday shooting.</td>
    <td>135</td>
    <td>I like the 24-70 better but this lens is a good all around and compact optic for everyday shooting.<br><br>As has been said many times before: "the best lens is the one you will use", and I know I wouldn not use the 24-70mm F2.8 because it's too heavy and bulky to take on backpacking/camping trips and when traveling abroad.</td>
  </tr>
  <tr>
    <td>The 24-120mm F4 lens has a longer zoom range and is more affordable than the 24-70mm F2.8.</td>
    <td>11</td>
    <td>I have an upcoming stay in Spain, and I'm seriously considering taking this lens instead of my AF-S 24-70 because of its size and zoom range.<br><br>My only complaint is the price tag: for a lens that is overall a rather mixed bag (depending on what you're looking for you might be very happy with it, or very disappointed) it is very expensive.</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>Prefer the 24-70mm F2.8 for its better image quality, faster aperture and better for wide shot.</td>
    <td>9</td>
    <td><i>For one thing, 24 70 is know to have better quality than this one.</i><br><br><i>The range from 70 to 120 is not as important as a better overall quality.</i></td>
  </tr>
</table>

Table 13: Top 3 key points mentioned in the KP summary produced by QQSUM-RAG for answering a query from AMAZONKP. For each key point, we show the prevalence, i.e., number of matching comments (with similar aspects of the same cluster), and two top matching comments. The relevant opinion in each comment that directly support the corresponding KP is italicized.
<table>
  <tr>
    <th>Query: How does this Nikon 24-120mm F4 lens compared with the 24-70mm F2.8 as a general walk around lense?</th>
    <th>QQSUM-RAG</th>
    <th>(Retriever+ LLM)<sub>co-trained</sub></th>
    <th>Contriver + GPT-4-Turbo</th>
    <th>Contriver + PAKPA</th>
    <th>Contriver + RKPA-Base</th>
  </tr>
  <tr>
    <td>The Nikon 24-120mm F4 lens is relatively lightweight and compact, making it easy to carry around and use for everyday shooting</td>
    <td>The 24-120mm f/4 offers more reach and versatility than the 24-70mm f/2.8.</td>
    <td>The 24-120mm lens offers good versatility and value for general use</td>
    <td>The 24-120 lens is preferred over the Nikkor 24-70mm F2.8, due to its lighter weight.</td>
    <td>The 24-120 is finally at a stage where you can carry it around on your FX camera and have no regrets.</td>
  </tr>
  <tr>
    <td>The 24-120mm F4 lens has a longer zoom range and is more affordable than the 24-70mm F2.8.</td>
    <td><b>The 24-120mm f/4 is lighter</b> and more affordable than the 24-70mm f/2.8.</td>
    <td>The 24-70mm lens has superior image quality and performance</td>
    <td><b>Best 4+ star walk-around lens on the market.</b></td>
    <td>If you want a 4+ star walk-around lens that covers a great range, this is the best on the market.</td>
  </tr>
  <tr>
    <td>Prefer the 24-70mm F2.8 for its better image quality, faster aperture and better for wide shot.</td>
    <td><b>The 24-70mm f/2.8 is a better lens overall.</b></td>
    <td>the 24-70mm lens is preferred for its optical superiority.</td>
    <td>The 24-70mm lens is highly recommended for wide shots.</td>
    <td>The 24-70mm lens is more expensive but buy it if you need to shoot wide.</td>
  </tr>
  <tr>
    <td>The 24-120mm F4 lens has good image quality, with sharpness and contrast that is comparable to the 24-70mm f/2.8</td>
    <td><b>The 24-120mm f/4 is too heavy.</b></td>
    <td><b>The 24-120mm lens is a more practical choice for everyday use.</b></td>
    <td>The Nikon 24-120 lens has good contrast compared to the Nikon 24-70 lens.</td>
    <td>I briefly considered the 24-70, but the extra reach, vibration reduction, and lower price point sold me on this lens.</td>
  </tr>
  <tr>
    <td>The 24-120mm F4 lens has good Vibration Reduction (VR) technology that helps to reduce camera shake when taking handheld shots.</td>
    <td>The 24-120mm f/4 has image stabilization, which is a significant advantage for handheld shots.</td>
    <td>The 24-120mm f/4 has image stabilization for handheld shots.</td>
    <td>N/A</td>
    <td>N/A</td>
  </tr>
</table>

Table 14: Top 5 key points, extracted from the summary of QQSUM-RAG and the baselines, ranked by their prevalence on an example query from AMAZONKP. Overlapping opinions across KPs are highlighted <span style="color:red">red</span>. KPs lacking of informativeness are highlighted <span style="color:yellow">yellow</span>

Reviewer paper summaries: This paper introduces Quantitative Query-Focused Summarization task to generate answers that capture diverse customer opinions and quantify their prevalence, addressing limitations of existing PQA systems. The proposed QQSUM-RAG model extends the Retrieval-Augmented Generation framework, combining a Key Point (KP)-oriented retriever and summary generator. Experiments show QQSUM-RAG outperforms RAG baselines in text quality and opinion quantification.

The paper introduces new summarization task specifically designed for review-based Product Question Answering (PQA) that is supposed to capture a diverse set of customer feedback. Their term the task as Quantitative Query-
Focused Summarization (QQSUM). The goal of QQSUM is to summarize diverse customer opinions into Key Points (KPs), quantifying their prevalence to effectively answer user queries. To achieve this, the authors propose QQSUM-RAG, an extension that employs few-shot learning and co-trains a KP-oriented retriever and Largel Language Model (LLM) based summarizer. Experimental results shows that QQSUM-RAG substantially outperforms baseline models in both summary quality and KP-Comment matching. The paper also provides satisfactory error analysis and ablation studies.
Summary of strengths: The co-training approach effectively optimizes the retriever and LLM by aligning opinion clusters with generated KPs under shared supervision, enabling strong performance and adaptability in few-shot learning with limited data. The authors constructed the AMAZONKP dataset by enhancing the Amazon Q&A dataset with additional data, including KP extraction, KP-comment pair matching, and summaries.

1. Introduces a new summarization task on review based product question answering that considers diverse customer opinions and represnting them as key points in order to improve question answering. 

2. The study also proposes a new state-of-the-art joint model for the proposed task that consists of a retrieval system for comment retrieval and LLM for summarization. In order to capture diverse set of user opinion, the retrieval system retrieves a set of comments and uses them to find KP and then use these comments to generate a summarization using an LLM. Then the model is jointly trained based on a loss that penalizes both the retrieval system and the LLM. 

3. The study also provides a new dataset for the associated task.
Summary of weaknesses: 1. The model's performance partially depends on the choice of retriever and generator, as different language models may lead to varying generation outcomes. If the authors could test the effects of different retrievers and generators, the experimental results would be more convincing.
2. The authors aimed to generate diverse summaries, but the experimental evaluation only assessed KP accuracy and factual consistency. It would be better if the authors could include evaluations on the diversity of the generated summaries.

1. I am found the inter annotator agreement score reporting a bit confusing. Was the Kappa set at 0.67 for annotators based on 5 way match and 50 judgements? I would like some clarification around the IAA.

2. I think the author address this in the error analysis, but I would like a discussion around it. The annotated clusters are e annotated clusters $p_j ∈ P$ are matched to predicted clusters $c_i ∈ C$. I have two queries around this 
* The selection of $p_j based solely on maximal similarity and this may lead to neglecting penalizing certain subtle clustering mistakes. Specifically, it may inadequately penalize clusters that mix different KPs if each KP is semantically close enough to another KP's cluster centroid. 

* Similarly, If the annotated clusters are dense and numerous, may the algorithm always find an annotated cluster that's relatively close to a predicted cluster, but might not contain necessary relevant information but still might reduce the loss.
Comments, suggestions and typos: See above.

Overall the paper is well written and I am happy with most of the reporting. I am also more than happy to adjust my scores once my queries in the weakness section are addressed.
Overall assessment scores: 2.5

4