Venue: NAACL 2025

# AdvisorQA: Towards Helpful and Harmless Advice-seeking

Question Answering with Collective Intelligence

 Minbeom Kim\({}^{1}\) &Hwanhee Lee\({}^{2}\) &Joonsuk Park\({}^{3,4,5}\)

**Hyaran Lee\({}^{3,4,6|7}\)****Kyomin Jung\({}^{1|}\)+\({}^{1}\)Seoul National University \({}^{2}\)Chung-Ang University \({}^{3}\)NAVER AI Lab \({}^{4}\)NAVER Cloud \({}^{5}\)University of Richmond \({}^{6}\)Sogang University {minbemkim, kjung}@snu.ac.kr, hwanheleelecau.ac.kr**

Footnote †: footnotemark:

###### Abstract

As the integration of large language models into daily life is on the rise, there is still a lack of dataset for _advising on subjective and personal dilemmas_. To address this gap, we introduce AdvisorQA, which aims to improve LLMs' capability to offer advice for deeply subjective concerns, utilizing the LifeProTips Reddit forum. This forum features a dynamic interaction where users post advice-seeking questions, receiving an average of 8.9 advice per query, with 164.2 upvotes from hundreds of users, embodying a _collective intelligence_. Therefore, we've completed a dataset encompassing daily life questions, diverse corresponding responses, and majority vote ranking, which we use to train a helpfulness metric. In baseline experiments, models aligned with AdvisorQA dataset demonstrated improved helpfulness through our automatic metric, as well as GPT-4 and human evaluations. Additionally, we expanded the independent evaluation axis to include harmlessness. AdvisorQA marks a significant leap in enhancing QA systems to provide subjective, helpful, and harmless advice, showcasing LLMs' improved understanding of human subjectivity.

## 1 Introduction

Large language models (LLMs) [10] have significantly enhanced _objective_ decision-making in various domains, such as science [14], and coding [21]. This was made possible, in part, by numerous benchmarks that assess the _objective_ helpfulness of LLMs [1, 15].

However, LLMs' impact on _subjective_ decision-making--e.g. determining a _better_ way to figure out one's girlfriend's ring size--has been minimal, despite the need [23, 24]. Given the unique challenges introduced by the subjectivity, such as the subjectivity of what constitutes better advice and the necessity of a harmlessness metric, there are few QA datasets available to support research on providing advice on subjective problems [1, 16, 17].

To this end, we present AdvisorQA, a dataset of 10,350 questions seeking advice on subjective and personal issues, each paired with a ranked list of 8.9 answers on average, as shown in Figure 1. Both the questions and the answers in AdvisorQA were written by users in a millions-user subreddit LifeProTips1, and the ranking of answers is also based on their preferences expressed as votes.

Footnote 1: [https://www.reddit.com/r/LifeProTips/](https://www.reddit.com/r/LifeProTips/)

AdvisorQA has two main features that differ from existing _objective_ QA, First, it is highly complex; the questions typically contain a detailed narrative on personal issues to solicit advice with dy

Figure 1: The example of test set thread in AdvisorQA: It consists of an advice-seeking question and the advising answers sorted by their upvote rankings. LLM advice is evaluated by the trained helpfulness metric based on its ranking against human-written answers.



namic language uses. They are not only long--75.2 words on average--but also cover a wide range of issues--daily topics from _Social conversation_ to _Travel tips_ as shown in Figures 3 and 10. Also, due to the subjective and complex nature of the questions, multiple answers, each providing a unique perspective, can all be helpful. This is distinct from existing QA datasets consisting of objective questions, each with a single correct answer.

Second, helpfulness in subjective topics is determined not only by objective criteria, such as correctness, but rather by personal preferences. To avoid having helpfulness rankings of answers biased to the few annotator's opinions Casper et al. (2023); Weerasooriya et al. (2023), we collected the majority preferences from million-scale active users included in the community upvote system. As a result, the answers for each question are ranked by an average of 164.2 votes per thread, which is a form of _collective intelligence_. We verified that the model trained on the upvote rank improved on GPT-4 and human evaluation, suggesting that using upvotes as a proxy for helpfulness is effective.

To account for the sensitive real-world issues in advice-seeking QA, we adopt appropriate metrics along two independent dimensions: _helpfulness_ and _harmlessness_. For helpfulness, we designed a helpfulness metric based on the Plackett-Luce (PL) model Plackett (1975), which is used for ranking predictions to incorporate preferences among diverse candidate advice. For harmlessness, we employ the LifeTox moderator Kim et al. (2024), a model to compute harmlessness scores. Since it was also trained on the data from the LifeProTips subreddit, it aligns well with our dataset.

We experimented with LLMs to measure their ability to provide subjective advice before and after supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). Without SFT, Llama Touvron et al. (2023) and Mistral Jiang et al. (2023) were the most harmless, but the GPT models OpenAI (2023) were the most helpful. Experiments on the two most harmless models show that SFT boosts helpfulness, but reduces harmlessness. The trend is amplified with RLHF using PPO Schulman et al. (2017), but most of the decline in harmlessness can be recovered with DPO Rafailov et al. (2023). Further analysis reveals that DPO's safe results stemmed from its tendency to follow demonstrations and produce strictly written advice. In contrast, PPO generates more empathic and diverse advice, but can be unsafe depending on reward models. This analysis concludes that each existing RLHF has limitations regarding advice-seeking QA.

The main contributions of this paper are summarized as twofold;

1. We present AdvisorQA, the dataset for subjective and personal questions with appropriate evaluation metrics along the dimensions of helpfulness and harmlessness.
2. We empirically show the popular LLMs' ability to advise on subjective issues and further analyze the _impact_ and _limitations_ of SFT and RLHF.2 Footnote 2: Code and dataset are available at [https://github.com/minbeomkim/AdvisorQA](https://github.com/minbeomkim/AdvisorQA).

## 2 Related Works

Subjectivity in NLPHumans communicate their experiences, thoughts, and emotions, so-called _private states_Wilson et al. (2005); Bjerva et al. (2020), through language in everyday interactions. Examples of private states encompass the beliefs and opinions of a speaker, beyond the scope of verification or objective observation. These kinds of states are referred to as _subjectivity_McHale (1983); Banea et al. (2011). Subjectivity has been explored within sentiment analysis Maas et al. (2011); Socher et al. (2013) and argument mining Park and Cardie (2014); Niculae et al. (2017), primarily concentrating on the polarity of individual sentences.

Training SubjectiveWith the recent advancement of LLMs, research on _subjective_ decision-making 'advice' has also been conducted. Wang Torres (2022) collected helpful and unhelpful advice from Reddit and analyzed that the main factor is 'empathy', consistent with our findings in Figure 4. However, Govindarajan et al. (2020); Chiu et al. (2024) discuss that good advice has various factors and conclude that to train a helpful LLM advisor in subjective domains, we should not rely solely on a few annotators Sandri et al. (2023); Fleisig et al. (2023). To address this, AdvisorQA leverages majority vote ranking from a Reddit forum with millions of users. DialogRPT Gao et al. (2020) also adopted upvotes as the criteria for helpfulness and focused on improving multi-turn dialogues. Both AdvisorQA and DialogRPT showed improvement in their metrics and human evaluation, _proving the validity of upvotes as a proxy of 

_helpfulness'_. TuringAdvice (Zellers et al., 2021) fine-tuned language models on highly upvoted advice from Reddit. Beyond fine-tuning high-quality advice, AdvisorQA further aligns with human preferences through majority vote ranking among high- and low-quality advice.

Benchmarking SubjectivityWhile benchmarks for objective domains related to reasoning and fact-checking are rapidly advancing Hendrycks et al. (2021); Laban et al. (2023), subjective domains such as advice-seeking question answering are progressing more slowly. Despite the emphasized need Shi et al. (2023), evaluation in subjective areas faces the hurdle of reflecting the preferences of diverse people Fleisig et al. (2023); Kim et al. (2025). Pioneering work TuringAdvice Zellers et al. (2021) proposed a benchmark that evaluates LLMs through a Turing test-like approach, where humans blindly compare _LLM_-generated advice with _one human_ advice. When this benchmark was introduced, the difference between human and LLM performance was stark. However, as LLMs continue to advance rapidly, pipelines that rely solely on human evaluation are becoming limited. Considering that human evaluation is costly and can lead to variability in subjective areas, we substitute it with automatic helpfulness evaluation and discuss the future pathway. AdvisorQA trains a helpfulness metric through massive upvotes ranking and benchmarks by evaluating the relative ranking of _LLM_-generated advice against a lot of _human_ advice. Additionally, we add an evaluation axis of 'harmlessness' to ensure that advice-seeking question answering is both helpful and harmless.

## 3AdvisorQA Dataset

### Main Goals of AdvisorQA

AdvisorQA requires LLMs to address a wide array of personal experience-based issues. Within the scope of AdvisorQA, the advice-seeking questions are elaborately detailed, capturing the intricate circumstances of individuals. As a result, the elicited responses are anticipated to vary widely, reflecting considerable subjectivity. Therefore, benchmarking such QA tasks characterized by strong subjectivity presents three principal goals; AdvisorQA is specifically designed to tackle these issues.

Annotation in Subjective PreferenceAnnotating subjective preferences, such as identifying the more helpful advice using the prevalent crowdsourcing method, poses limitations Kirk et al. (2023); Casper et al. (2023). This issue arises primarily due to individuals' diverse and unique primary values. Hence, engaging individuals with diverse backgrounds in the brainstorming process is imperative instead of relying exclusively on a limited group of crowdworkers. Consequently, in developing AdvisorQA, we have utilized the number of upvotes received by the advice in various discussions to indicate a web-scale preference.

Evaluation of Subjective HelpfulnessIn QA with subjective topics, each query can elicit multiple plausible answers. The commonly used n-gram similarity metrics such as BLEU and ROUGE in non-factoid QA are limited by their inability to quantify subjective preferences Krishna et al. (2021). A more suitable approach is to evaluate answers through comparative analysis against reference advice as in Figure 1. In response to this challenge, AdvisorQA utilizes automatic metrics that discern the majority's preferences via upvote rankings. This method is then employed to approximate the ranking of advice offered by language models, thus aiding in evaluating their helpfulness.

Helpful and Harmless AdviceThe subjective advice sometimes could be helpful but unsafe - i.e., unethical advice Kim et al. (2024a). In light of this, AdvisorQA has been strategically designed to evaluate both _Helpfulness_ and _Harmlessness_. The training set intentionally includes a designated proportion of unsafe advice to stimulate active follow-up research. This approach encourages the active and analytical exploration of methodologies that enable model training to be safe and more helpful, even when the benchmark's training set clearly contains unsafe advice.

### Dataset Construction

AdvisorQA should be a comprehensive benchmark for evaluating and enhancing the capabilities of LLMs in offering subjective, actionable, and empathetic advice on personal experiences. It is crucial to have sufficient advice-seeking questions and diverse advice involving widespread participation in discussions and the corresponding upvote rankings. Therefore, we utilized the Reddit forum LifeProTips (LPT), which has a million-scale user participation in advice-seeking question answering. In LPT threads, as illustrated in Figure 1, a user posts an advice-seeking question about their personal situation. Various users reply with their own solutions 

to the question. These pieces of advice become subject to discussions by others who express their opinion through replies and preferences through recommendations. We have adopted this upvote ranking as a metric for majority preference in AdvisorQA. Due to the nature of the LPT community, where upvotes indicate helpfulness and the average vote count is high, there is a denoising effect on upvotes that are used in other meanings. This allowed us to use upvotes as a proxy for 'helpfulness,' similar to previous works Fan et al. (2019); Gao et al. (2020); Wang and Torres (2022).

While LPT strictly allows only safe advice following its guidelines, the twin subreddit forum UnethicalLifeProTips (ULPT)3 permits only unsafe advice under rigorous community rules 4. Both communities focus on the helpfulness of the given advice in the presented situation according to each ethical community's guidelines. Consequently, we have sourced 5 threads from LPT and toxic advice from ULPT and constructed AdvisorQA for the advice-seeking QA benchmark, especially in evaluating better advice and training for better advisor LLMs. This task includes 9,350 threads in the _training set_ and 1,000 threads in the _test set_. To more meaningfully reflect real-world social risks Hur et al. (2020), the _training set_ comprises 8,000 threads from LPT and 1,350 threads from ULPT. Because we find that unsafe advice is much easier to learn than safe advice in experiments. Therefore, it is important for future research to focus on controlling safety while enhancing helpfulness when training on AdvisorQA, which is why we mix unsafe advice. More detailed rationales are additionally discussed in the Appendix B. For the _test set_, four reference advices are available for comparative evaluation of the language model's advice, as exemplified in Figure 1.

Footnote 3: [https://www.reddit.com/r/UnethicalLifeProTips/](https://www.reddit.com/r/UnethicalLifeProTips/)

Footnote 4: Detailed community guidelines is in Appendix A

### AdvisorQA Dataset Statistics

A key feature of AdvisorQA is its use of the upvote system to employ majority vote ranking as a form of collective intelligence. As such, Table 1 and Figure 2 reveal that there are, on average, 8.9 advice responses per advice-seeking question, with the top-ranked advice receiving an **average of 71.4 upvotes** and the total for all advice in each thread amounting to **164.2**. This means that for each thread, nearly ten people offer their opinions, and _over a hundred users express their preferences_, making it a dataset with a highly crowded preference reflected.

This diversity is further evidenced in Table 2, where the potential for diverse advice leads to lower average BLEU scores among candidate answers compared to ELI5 and Antique. Moreover, a significant difference from existing non-factoid long-form QA datasets lies in the nature of the advice-seeking questions in Table 1. These questions originate from very specific and personal experiences, resulting in an overwhelmingly high average token length compared to other datasets. The variety of questions and answers contributes to a significantly larger vocabulary size relative to the number of threads, strongly highlighting the characteristics of AdvisorQA.

### Complexity of Advice-seeking Questions

Beyond the numerical statistics, this subsection delves into the characteristics of the advice-seeking questions within our proposed benchmark. As de

\begin{table}
\begin{tabular}{c c c c c} \hline \multirow{2}{*}{**Datasets**} & \# Answers & \# Words in & \multicolumn{1}{l}{\multirow{2}{*}{**\# Questions**}} & \multirow{2}{*}{**Voable size**} \\  & **per Question** & & & \\ \hline NI,QuaD & 1 & 7.0 & **31,252** & 138,243 \\ Antique & **11.1** & 10.5 & 2,626 & 8,185 \\ SallyQA & 0.7 & 5.6 & 10,000 & 22,221 \\ WikiWOa & 1 & 6.4 & 11,749 & 48,665 \\ AmazonQA (ours) & **8.9** & **75.2** & 10,350 & **326,668** \\ \hline \end{tabular}
\end{table}
Table 1: Statistical characteristics of non-factoid long-form QA datasets, including AdvisorQA.

Figure 3: Visualization for topic distributions of advice-seeking questions in AdvisorQA. More detailed visualization is in Figure 10.

Figure 2: The distribution of average upvotes by rank of advice.

 

picted in Figure 9, these questions typically involve deeply personal and daily experiences prompting the search for advice. It leads to a broad spectrum of topics from social interactions to careers, as demonstrated in Figure 3 and 10, with many sub-topics and keywords present within each topic. The intricately detailed accounts of personal experiences, exemplified in Figure 1, facilitate a diverse range of perspectives, thereby broadening the scope of subjectivity within AdvisorQA. Therefore, these distinct features of advice-seeking questions in AdvisorQA stand out compared to other benchmarks, leading to the complexity and uniqueness of the tasks we propose.

## 4 Evaluation Metrics

In this section, we discuss how to evaluate the advice generated by language models in the AdvisorQA benchmark. Given the task's pronounced subjectivity, we measure _helpfulness_ not by similarity to references but through comparative ranking. Moreover, as an auxiliary measure, we evaluate the safety of the advice by evaluating its _harmlessness_.

### Dimension 1: Helpfulness

Evaluating what is most helpful in subjective domains presents a significant challenge. Multiple answers can be valid for a single question, and what is considered most helpful can vary from one person to another. Therefore, we base our evaluation of the AdvisorQA evaluation pipeline on how well it _understands the majority preference values_ of the group participating in this forum and how accurately it can _mimic this collective intelligence for evaluating baselines_. To discuss this numerically, we assess the evaluation pipelines by how well they can predict the advice rankings in the test set threads based on learning from the training set's advice rankings. The effectiveness of these evaluation methods is measured using the Normalized Discounted Cumulative Gain (NDCG) metric Wang et al. (2013), which evaluates how accurately the top \(k\) pieces of advice are selected and ranked. Furthermore, we measure the preference prediction accuracy of the top-1 recommended advice against the 2nd-ranked advice and the last one.

We set the baselines with BARTScore Yuan et al. (2021), the probability of being generated from BART Lewis et al. (2019), two general-purpose reward models; ArmoRM and InternLM2 Wang et al. (2024); Cai et al. (2024) in the RewardBench leaderboard Lambert et al. (2024), and GPT-4-turbo-preview OpenAI (2023). Additionally, we employ the Plackett-Luce (PL) model Plackett (1975); Luce (2012), which learns the advice ranking from the training set and predicts the advice ranking in the test set. We have trained the PL (\(K\)) model for the helpfulness metric as

\[P_{PL}=\prod^{K}_{k=1}\frac{exp(h_{\theta}|q,a_{k})}{\sum^{K}_{i= k}exp(h_{\theta}|q,a_{i})}, \tag{1}\]

designed to properly rank advice \(a_{k}\) from question \(q\) among \(K\) pieces of advice with output helpfulness score \(h_{\theta}\). This model serves for _K-wise ranking comparison_ as an extension of Bradley-Terry model Bradley and Terry (1952), which is a widely adopted reward model for _pairwise comparison_Casper et al. (2023). We trained PL models based on Pythia-1.4B Biderman et al. (2023).

#### Preliminary Test of Helpfulness Metrics

We first verified the validity of this experiment through human evaluation. In AdvisorQA, since the helpfulness between high-quality advice is subjective,

\begin{table}
\begin{tabular}{l c c c} \hline \hline  & **ELLS** & **Antique** & **AdvisorQA** \\ \hline BLEU \(\downarrow\) & 0.26 & 0.26 & **0.23** \\ \hline \hline \end{tabular}
\end{table}
Table 2: To measure the diversity among responses in the reference, we calculate the average BLEU score among various reference responses to a single question by calculating the BLEU score, performing this for all reference responses, and then averaging the results. (same as Self-BLEU manner Zhu et al. (2018)).

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{_Helefulness Metrics_} & \multicolumn{3}{c}{NDCG} & \multicolumn{3}{c}{1st advice vs} \\ \cline{2-5}  & @ 2 & @ 3 & @ 5 & 2nd & last \\ \hline Random & 0.433 & 0.498 & 0.529 & 0.500 & 0.500 \\ BARTScore (40GM) & 0.468 & 0.532 & 0.566 & 0.505 & 0.584 \\ ArmoRM (TB) & 0.493 & 0.575 & 0.592 & 0.533 & 0.636 \\ IntemLM2 (20Rb) & 0.496 & 0.580 & 0.606 & 0.536 & 0.638 \\ GPT-4 Turbo (> 175B) & 0.498 & 0.601 & 0.614 & 0.540 & 0.663 \\ \hline \hline _Placket-Luce (K) (1.4B)_ & & & & & \\ \hline \(K=2\) & 0.488 & 0.572 & 0.602 & 0.525 & 0.664 \\ \(K=3\) & 0.515 & 0.594 & 0.616 & 0.554 & **0.675** \\ \(K=4\) & 0.520 & 0.605 & **0.630** & 0.571 & 0.668 \\ \(K=5\) & **0.525** & **0.615** & 0.625 & **0.575** & 0.666 \\ \(K=all\) & 0.523 & 0.595 & 0.616 & 0.565 & 0.665 \\ \hline Human Evaluation & & & & 0.667 & 0.833 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Alignment between helpfulness metrics and human judgment: Experiment results for predicting the gold-standard rankings of answers.

 

we observed a 67% result in the 1st vs 2nd comparisons, which is similar to the upvote ratio of 71:32 between the first and second ranks shown in Figure 2. This indicates that upvote ranking is an effective proxy for 'helpfulness'. Additionally, an accuracy of 83% in the 1st vs last comparisons further confirmed the effectiveness of validation through upvote ranking.

In Table 3, BARTScore shows no ability to distinguish between the first and second best advice; but some capability in differentiating between the best and worst advice. This suggests that while the top and bottom advice can be somewhat distinguished based on their plausibility, BARTScore fails to compare high-quality advice only with plausibility. GPT-4 outperforms BARTScore in all metrics, yet it still struggles to predict preferences between the first and second-best advice. While this indicates a difference between GPT-4's general preferences and AdvisorQA's subjective preferences, the fact that GPT-4 achieves this performance in a zero-shot setting demonstrates its meaningfulness as a reference evaluation.

The trained PL model shows the best performance among the baselines in both ranking and preference prediction, even surpassing GPT-4, with 1.4 billion parameters. It significantly outperforms GPT-4 in predicting preferences between the first and second-best advice. Performance improvements are evident with the increase in the number of K candidates used in training the Plackett-Luce model, particularly in differentiating between the first and second best advice. It confirms that referencing a variety of advice aids in learning web-scale preferences. However, referencing all advice rankings leads to performance degradation, indicating considerable noise in the ranking of tail-ranked advice. This phenomenon is known as _'first mover advantage,'_(Lieberman and Montgomery, 1988) where there is strong noise in the upvotes instances that follow, except for those in the top ranks. To denoise it, we designed the model to predict the _ranking_ of top advice with less noise rather than directly predicting the _count of noisy upvotes_.

Analysis of Primary Value of Evaluation MetricsOur PL model performs better than GPT-4, but it still falls short of fully understanding the majority preference of LifeProTips. This is due to the incomplete grasp of the diverse subjective preference values and the models predicting based on a limited set of primary values. Consequently, we analyze to determine which values are prioritized in preference prediction by two prominent evaluation pipelines: GPT-4 and the PL (\(K=5\)) model. This analysis encompassed seven values deemed crucial in advice-seeking question answering: _Relevance, Actionability and Practicality, Empathy and Sensitivity, Creativity, Readability and Clarity, Constructiveness, and Ethics_. The Appendix E contains detailed instructions for each of these options.

To determine the primary value inherent in each evaluation pipeline, we analyzed 300 instances from the test set comparison task where GPT-4 and the PL model yielded different predictions for two answer pairs. In cases where GPT-4's prediction was accurate, we conducted a survey as shown in Figure 11, prompting annotators to select why they think the winner advice is better, choosing from a list of seven important values. A similar survey was conducted for instances where the PL model's prediction was accurate, but GPT-4's was not. This way, we could see what each pipeline values most when deciding which advice is better.

In Figure 4, the results show a stark difference in the values primarily pursued by the PL model and GPT-4. GPT-4 focuses on values like Ethics, readability, and Constructiveness, emphasizing the completeness and safety of advice. In contrast, the PL model prioritizes Empathy, Actionability, and

Figure 4: Analysis results of the primary value of evaluation metric: When GPT-4 and the PL model disagree on which advice is better, looking at situations where GPT-4 is right helps us understand what values it prioritizes differently from the PL model and vice versa. We surveyed these instances, sorting them into seven key values, to gather insights on what each model values most in their decisions.

 

Creativity. Being trained on the threads of AdvisorQA, the PL model reflects the Reddit forum's source, valuing advice that resonates empathetically with the given situation, is actionable, and creative, as preferred by the majority. Additionally, since the PL model is trained on both safe and unsafe advice, it does not prioritize safety, leading to _orthogonalized_ dimensions of "helpfulness" and "harmlessness." This analysis reveals the various uncovered preferences of the majority who participated in AdvisorQA, highlighting the diversity of values and underscoring the need for fine-grained evaluation metrics in the future.

### Dimension 2: Harmlessness

In the analysis of helpfulness evaluation depicted in Figure 4, we found that the PL model serves as an orthogonal metric to harmlessness, underscoring the critical need for a metric that addresses this aspect. To meet this requirement, we utilized the LifeTox moderator Kim et al. (2024), a toxicity detector trained on the UnethicalLifeProTips forum. This metric is recognized as state-of-the-art for question answering on daily topics as a scorer and is selected for its robust generalization capabilities with LLM-generated texts. The average of the output class labels measures the harmlessness score for LLMs. GPT-3.5 can perform comparably but was excluded because its scoring was not appropriate.

## 5 Experiments

This section outlines the baselines for AdvisorQA. Four advices accompany each question in the test set. The helpfulness of the advice generated by LLMs is determined by its ranking among a total of five pieces of advice. The safety of the LLMs is assessed based on the harmlessness score assigned to each piece of advice. These two criteria are used to analyze the performance of baseline models and training approaches.

### Baselines

Baseline ModelsWe evaluate helpfulness by mainly the PL (5) model and harmlessness by LifTox moderator Kim et al. (2024). According to Figure 4, the PL (5) model does not incorporate ethical considerations into its assessment of helpfulness, resulting in our metrics for helpfulness and harmlessness being made orthogonal to each other. Initially, we assess the performance of open-source LLMs and then analyze their development upon training with AdvisorQA. To examine the performance of instruction-tuned models at various scales, we selected the Flan-T5 Family Chung et al. (2022), Llama-2-Chat-7B Touvron et al. (2023), Mistral-7B Jiang et al. (2023), along with GPT-3.5-Turbo and GPT-4-Turbo-preview OpenAI (2023).

Baseline TrainersTo analyze training effectiveness on AdvisorQA, we utilized two widely used RLHF methods, PPO Schulman et al. (2017) and DPO Rafailov et al. (2023). PPO is online RL approach that explores _to maximize the output values of reward models_, PL (5) model. On the other hand, DPO is an offline RL that _learns to increase the relative probability of win response generation rather than lose response generation_. For this purpose, we conducted supervised fine-tuning (SFT) of Llama-2-7B and Mistral-7B on the AdvisorQA training set. Then, for a fair comparison, PPO used the PL (5) model as the reward model, while DPO employed the ranking of 5 candidate pieces of advice as demonstrations. All training processes are under 4-bit QLoRA Dettmers et al. (2023). Detailed hyperparameters and experimental details are provided in the Appendix C.

### Results

Figure 5 illustrates that the helpfulness of LLMs generally escalates with the model scale. Notably, for parameter scales exceeding 175B, instances in which LLM-generated advice surpasses half of human-written advice, indicating superior performance, with Llama-2-7B producing the safest advice. Interestingly, as GPT's performance improves, it also becomes safer. Conversely, Flan-T5 experiences a marked increas

Figure 5: Experimental results of baseline models performance in helpfulness and harmlessness.

 

formance improves. This trend is attributed to the Flan-T5 being a safety-uncontrolled model family.

In Figure 6, models trained with SFT on AdvisorQA show an increase in helpfulness but, concurrently, become more harmful. This suggests that training strategies to enhance token-level likelihood are more prone to adopting unsafe advice. Moreover, when SFT models undergo RLHF, the two methodologies diverge in their outcomes; PPO models outperform DPO models in helpfulness but tend towards unsafe improvement, while DPO progresses in a safer manner. Because PPO models directly optimize the evaluation metric as a reward model, we further investigate the helpfulness of other metrics.

We explore helpfulness through additional metrics: GPT-4 and human evaluation as Appendix E. As seen in Figure 7, it is evident that overall advisor performance improves with RLHF across all metrics. However, in human evaluations, PPO and DPO models progress equally, but according to GPT-4's criteria, DPO is significantly preferred. This preference is analyzed in the context of GPT-4 valuing ethical considerations significantly in Table 4, and as shown in Figure 6, while PPO models develop in an unethical direction, DPO models evolve ethically, leading GPT-4 to favor DPO models.

### Analysis of RLHF Trainers

This subsection analyzes the learning characteristics of baselines beyond helpfulness and harmlessness. We use two metrics: max BLEU (Post, 2018) and Self-BLEU (Zhu et al., 2018). Max BLEU measures the highest BLEU score between the generated advice and references in the test set, while Self-BLEU assesses the similarity among advices generated by the same LM. Therefore, a higher max BLEU score signifies advice that is more similar to the given datasets, and a higher Self-BLEU score indicates less diversity in advice generation.

Table 4 indicates that both DPO models achieved the highest max BLEU and Self-BLEU scores, meaning less novel and diverse advice. Conversely, PPO models exhibited a more diverse generation than both SFT and DPO. This implies that, since DPO directly optimizes the probability of generating win pairs from the dataset, leading to a higher max, self-BLEU score with the candidate answers. Conversely, PPO explores through the reward model without demonstrations and maximizes its key portions, such as _empathy, creativity, and actionability_ in Figure 4, producing more diverse and even creative responses than DPO. Regarding harmlessness, DPO's safe learning is due to the higher proportion of safe instances in the training set. On the other hand, in the case of PPO, as noted in Figure 4, there is a lack of safety guidance in the reward model; PPO models are less safe than DPO; however, they can generate more diverse and enriched advice. In this way, online and offline RL show trade-offs with each limitation, struggling to align subjective and diverse preferences and being highly influenced by toxic advice mixed in the dataset. This leads to the conclusion that the more subjective the task, the stronger the bottleneck in reward modeling, and the greater the risk of learning from toxic instances. We attach a more detailed rationale in Appendix B and case studies in Appendix D.

\begin{table}
\begin{tabular}{l c c c|c c c} \hline \hline  & \multicolumn{3}{c}{Llama-2-Chat-7B} & \multicolumn{3}{c}{Mistral-7B} \\  & SFT & PPO & DPO & SFT & PPO & DPO \\ \hline max BLEU j & 0.25 & **0.22** & 0.30 & 0.24 & **0.21** & 0.27 \\ Self-BLEU j & 0.47 & **0.40** & 0.43 & 0.46 & **0.40** & 0.41 \\ \hline \hline \end{tabular}
\end{table}
Table 4: max BLEU and Self-BLEU of each model trained on AdvisorQA

Figure 6: Experimental results of trained models performance shift in helpfulness and harmlessness.

Figure 7: Experimental results of trained models performance shift in helpfulness with GPT-4 and human evaluation.

 

Conclusion

We introduce AdvisorQA, a benchmark for advice-seeking question answering that focuses on questions rooted in personal experiences and the corresponding advice, ranked by _Collective Intelligence_. AdvisorQA serves as a valuable resource for advancing everyday QA systems that provide in-depth, empathetic, and practical advice towards daily dynamic dilemmas. By leveraging upvote ranks to evaluate various subjective opinions and through baseline experiments, we have confirmed the dataset's validity and shed light on the impact and limitations of RLHF trainers in subjective domains. Further, we analyze and highlight critical remaining issues to handle subjectivity that future research should consider. These analyses suggest a broad potential to facilitate research in evaluating and training systems for daily neural advisors.

### Limitations

We've refined our approach to evaluating language models by developing orthogonal metrics for helpfulness and harmlessness, enabling a detailed analysis of various baselines. However, the evaluation analysis in Section 4.1 revealed that subjective helpfulness involves a wide array of values, with each metric addressing different aspects. Surely, training on advice ranking helped identify the primary preference values of the majority participating in the forum. Yet, leveraging this benchmark for more effective and controllable learning necessitates the development of _fine-grained evaluation metrics_ capable of annotating helpfulness from diverse viewpoints. This approach will enable a deeper examination of the specific features of language models for future research. Nonetheless, language models tailored for subjective missions must be carefully designed for their eventual integration into daily and personalized human activities. Thus, the need extends beyond fine-grained evaluation Lee et al. (2024); c,a) to include methods that facilitate controllable text generation Kim et al. (2023); Liu et al. (2024); Min et al. (2024) for nuanced attributes or selective alignment with various values.

Reddit forum LifeProTips has 23 million active users but does not represent the full spectrum of human diverse values worldwide. Different social groups pursue their own values, so AdvisorQA cannot represent the global majority preference. Additionally, during the alignment process, there is a risk of over-optimizing for majority preferences, leading to the loss of minority subjective preferences. Moreover, for tailed cases that are not among the top-upvoted advice, 'first mover advantage' can occur. Due to space constraints, we could not fully elaborate on Line 249, but this noise explains why learning from tailed advice resulted in minimal performance improvement. Also, due to the nature of the community, there may be abusive behavior. However, the large-scale advice and the high average number of upvotes (71.4) had a denoising effect. Additionally, from a technical standpoint, our baseline experiments were carried out using 4-bit initialization and QLoRA Dettmers et al. (2023), significantly reducing the number of trainable parameters, underscoring the potential for significant advancements in model fine-tuning.

## Ethical Statement

We acknowledge that AdvisorQA encompasses various pieces of advice that could potentially trigger different social risks. However, it is essential to explore a wide range of advice-seeking question answering scenarios to identify and understand the broader spectrum of implicit social risks. Therefore, we have employed a harmlessness metric to analyze each baseline in parallel with how helpful they are. Nonetheless, our proposed Life Tox moderator was trained solely using labels from both subreddit forums, LPT and ULPT. It means there is a potential annotation bias within the defined scope of toxicity. Consequently, to utilize this in various downstream applications, it's necessary to evaluate social risks from a fine-grained perspective using moderators defined in diverse toxicity definitions. Moreover, when training LLMs as neural advisors, the focus should not be solely on maximizing helpfulness but also on incorporating various safety metrics into the training process. Especially, there should be the complementary usage of out-domain toxicity moderators such as StereoSet Nadeem et al. (2021), ETHICS Hendrycks et al. (2023), and KoSBi Lee et al. (2023), which are crucial for ensuring the well-being of diverse human audiences. AdvisorQA was crawled through Praw, Reddit's official API. Their policy is to ban corporations from using the corpus to train for-profit LLMs, while academic use remains open.

## Acknowledgements

This work has been financially supported by SNU-NAVER Hyperscale AI Center. This work 

was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT)(RS-2024-00348233). This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS-2022-Il21084, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics, 60%), and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) [NO.RS-2021-I211343, Artificial Intelligence Graduate School Program (Seoul National University) & NO.RS-2021-II212068, Artificial Intelligence Innovation Hub (Artificial Intelligence Institute, Seoul National University)], and the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2024. K. Jung is with ASRI, Seoul National University, Korea. The Institute of Engineering Research at Seoul National University provided research facilities for this work.

## References

* Banea et al. (2011) Carmen Banea, Rada Mihalcea, and Janyce Wiebe. 2011. Multilingual sentiment and subjectivity analysis. _Multilingual natural language processing_, 6:1-19.
* Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Afah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In _International Conference on Machine Learning_, pages 2397-2430. PMLR.
* Bjerva et al. (2020) Johannes Bjerva, Nikita Bhutani, Behzad Golshan, Wang-Chiew Tan, and Isabelle Augenstein. 2020. SubjQA: A Dataset for Subjectivity and Review Comprehension. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5480-5494, Online. Association for Computational Linguistics.
* Bolotova et al. (2022) Valeria Bolotova, Vladislav Blinov, Falk Scholer, W Bruce Croft, and Mark Sanderson. 2022. A non-factoid question-answering taxonomy. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 1196-1207.
* Bolotova-Baranova et al. (2023) Valeria Bolotova-Baranova, Vladislav Blinov, Sofya Filippova, Falk Scholer, and Mark Sanderson. 2023. WikiHowQA: A comprehensive benchmark for multidocument non-factoid question answering. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5291-5314, Toronto, Canada. Association for Computational Linguistics.
* Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. _Biometrika_, 39(3/4):324-345.
* Cai et al. (2022) Zheng Cai, Maosong Cao, Haojoining Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chengva Gu, Yuzhe Gu, Tao Gui, Ajina Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shaubin Li, Wei Li, Yinning Li, Hongwei Liu, Jiangine Liu, Jiawei Hong, Kaixin Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huname Tang, Bin Wang, Guoteong Wang, Jiaqi Wang, Jiayu Wang, Kui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuvu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Shuijie Zhang, Shuo Zhang, Songyang Zhang, Wenjiian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Ziata Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. 2024. Internlm2 technical report.
* Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jeremy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korobak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. _arXiv preprint arXiv:2307.15217_.
* Chiu et al. (2024) Yu Ying Chiu, Ashish Sharma, Inna Wanyn Lin, and Tim Althoff. 2024. A computational framework for behavioral assessment of llm therapists.
* Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunuxuan Li, Xuehi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, et al. 2021. Training verifiers to solve math word problems, 2021. _URL https://arxiv. org/abs/2110.14168_ 

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized l1ms. _arXiv preprint arXiv:2305.14314_.
* Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
* Fleisig et al. (2023) Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagreement for subjective tasks. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 6715-6726, Singapore. Association for Computational Linguistics.
* Gao et al. (2020) Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett, and Bill Dolan. 2020. Dialogue response ranking training with large-scale human feedback data. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 386-395, Online. Association for Computational Linguistics.
* Govindarajan et al. (2020) Venkata Subrahmanyan Govindarajan, Benjamin Chen, Rebecca Warholic, Katrin Erk, and Junyi Jessy Li. 2020. Help! need advice on identifying advice. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 5295-5306, Online. Association for Computational Linguistics.
* Hendrycks et al. (2023) Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2023. Aligning ai with shared human values.
* Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. _arXiv preprint arXiv:2009.03300_.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. _arXiv preprint arXiv:2103.03874_.
* Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. _arXiv preprint arXiv:2106.09685_.
* Hur et al. (2020) Juyeon Hur, Kathryn A DeYoung, Samiha Islam, Allegra S Anderson, Matthew G Barstead, and Alexander J Shackman. 2020. Social context and the real-world consequences of social anxiety. _Psychological Medicine_, 50(12):1989-2000.
* Hwang et al. (2022) Wonseok Hwang, Dongjun Lee, Kyoungyeon Cho, Hanuhl Lee, and Minjoon Seo. 2022. A multi-task benchmark for korean legal language understanding and judgement prediction. _Advances in Neural Information Processing Systems_, 35:32537-32551.
* Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulinier, et al. 2023. Mistral 7b. _arXiv preprint arXiv:2310.06825_.
* Kim et al. (2024) Minboom Kim, Jahyun Koo, Hwanhee Lee, Joonsuk Park, Hwaran Lee, and Kyomin Jung. 2024a. Lifsuck: Unveiling implicit toxicity in life advice. In _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)_, pages 688-698, Mexico City, Mexico. Association for Computational Linguistics.
* Kim et al. (2023) Minboom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung. 2023. Critic-guided decoding for controlled text generation. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 4598-4612, Toronto, Canada. Association for Computational Linguistics.
* Kim et al. (2025) Minboom Kim, Kang-il Lee, Seongho Joo, Hwaran Lee, and Kyomin Jung. 2025. Drift: Decoding-time personalized alignments with implicit user preferences. _arXiv preprint arXiv:2502.14289_.
* Kim et al. (2024b) Minboom Kim, Thibaut Thonet, Jos Rozen, Hwaran Lee, Kyomin Jung, and Marc Dymettan. 2024b. Guaranteed generation from large language models. _arXiv preprint arXiv:2410.06716_.
* Kirk et al. (2023) Hannah Kirk, Andrew Bean, Bertie Vidgen, Paul Rottger, and Scott Hale. 2023. The past, present and better future of feedback learning in large language models for subjective human preferences and values. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 2409-2430, Singapore. Association for Computational Linguistics.
* Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4940-4957, Online. Association for Computational Linguistics.
* Kung et al. (2023) Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Simlos, Lorie De Leon, Camille Elepano, Maria Madriaga, Rimel Aggabao, Giezel Diaz-C Candido, James Maningo, et al. 2023. Performance of chatgtpot on usmple: Potential for ai-assisted medical education using large language models. _PLoS digital health_, 2(2):e0000198.
* Laban et al. (2023) Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. SummEflits: Measuring LLM ability at factual reasoning through the lens of summarization. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 9662-9676, Singapore. Association for Computational Linguistics.

 

Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024. Rewardbench: Evaluating reward models for language modeling. _arXiv preprint arXiv:2403.13787_.
* Lee et al. (2024a) Dongryevol Lee, Yerin Hwang, Yongil Kim, Joonsuk Park, and Kyomin Jung. 2024a. Are llm-judges robust to expressions of uncertainty? investigating the effect of epistemic markers on llm-based evaluation. _arXiv preprint arXiv:2410.20774_.
* Lee et al. (2024b) Dongryevol Lee, Minwoo Lee, Kyungmin Min, Joonsuk Park, and Kyomin Jung. 2024b. Return of em: Entity-driven answer set expansion for qa evaluation. _arXiv preprint arXiv:2404.15650_.
* Lee et al. (2023) Hwaran Lee, Seokhee Hong, Joonsuk Park, Takyoung Kim, Gunhee Kim, and Jung-woo Ha. 2023. KoSBI: A dataset for mitigating social bias risks towards safer large language model applications. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)_, pages 208-224, Toronto, Canada. Association for Computational Linguistics.
* Lee et al. (2024c) Kang-il Lee, Minboom Kim, Seunghyun Yoon, Minsung Kim, Dongryevol Lee, Hyukhun Koh, and Kyomin Jung. 2024c. Vlind-bench: Measuring language priors in large vision-language models. _arXiv preprint arXiv:2406.08702_.
* Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.
* Lieberman and Montgomery (1988) Marvin B Lieberman and David B Montgomery. 1988. First-mover advantages. _Strategic management journal_, 9(S1):41-58.
* Lucce (2012) R Duncan Lucce. 2012. _Individual choice behavior: A theoretical analysis_. Courier Corporation.
* Maas et al. (2011) Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_, pages 142-150.
* McHale (1983) Brian McHale. 1983. Unspeakable sentences, unnatural acts: Linguistics and poetics revisited. _Poetics Today_, 4(1):17-45.
* Min et al. (2024) Kyungmin Min, Minbeom Kim, Kang-il Lee, Dongryeol Lee, and Kyomin Jung. 2024. Mitigating hallucinations in large vision-language models via summary-guided decoding. _arXiv preprint arXiv:2410.13321_.
* Nadeem et al. (2021) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 5356-5371, Online. Association for Computational Linguistics.
* Ni et al. (2023) Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In _International Conference on Machine Learning_, pages 26106-26128. PMLR.
* Niculae et al. (2017) Vlad Niculae, Joonsuk Park, and Claire Cardiie. 2017. Argument mining with structured SVMs and RNNs. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, Pages 985-995, Vancouver, Canada. Association for Computational Linguistics.
* OpenAI (2023) R OpenAI. 2023. Gpt-4 technical report. _arXiv_, pages 2303-08774.
* Park and Cardiie (2014)Joonsuk Park and Claire Cardiie. 2014. Identifying appropriate support for propositions in online user comments. In _Proceedings of the first workshop on argumentation mining_, pages 29-38.
* Plackett (1975) Robin L Plackett. 1975. The analysis of permutations. _Journal of the Royal Statistical Society Series C: Applied Statistics_, 24(2):193-202.
* Post (2018) Matt Post. 2018. A call for clarity in reporting BLEU scores. In _Proceedings of the Third Conference on Machine Translation: Research Papers_, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
* Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. _arXiv preprint arXiv:2305.18290_.
* Sandri et al. (2023) Marta Sandri, Elisa Leonardelli, Sara Tonelli, and Elisabetta Jezek. 2023. Why don't you do it right? analysing annotators' disagreement in subjective tasks. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 2428-2441, Dubrovnik, Croatia. Association for Computational Linguistics.
* Schulman et al. (2017) John Schulman, Filip Wolski, Praftula Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _arXiv preprint arXiv:1707.06347_.
* Shi et al. (2023) Chongyang Shi, Yijun Yin, Qi Zhang, Liang Xiao, Usman Naseem, Shoujin Wang, and Liang Hu. 2023. Multiview clickbait detection via jointly modeling subjective and objective preference. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 11807-11816, Singapore. Association for Computational Linguistics.

 

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 conference on empirical methods in natural language processing_, pages 1631-1642.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Wang et al. (2024) Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. _arXiv preprint arXiv:2406.12845_.
* Wang et al. (2013)Yining Wang, Liwei Wang, Yuanzhi Li, Di He, and Tiem Yan Liu. 2013. A theoretical analysis of ndcg type ranking measures. In _Conference on learning theory_, pages 25-54. PMLR.
* Wang and Torres (2022) Zhilin Wang and Pablo E. Torres. 2022. How to be helpful on online support forums? In _Proceedings of the 4th Workshop of Narrative Understanding (WNU2022)_, pages 20-28, Seattle, United States. Association for Computational Linguistics.
* Weerasoriya et al. (2023) Tharindu Cyril Weerasoriya, Sarah Luger, Saloni Poddar, Ashiqur KhudaBukhsh, and Christopher Homan. 2023. Subjective crowd disagreements for subjective data: Uncovering meaningful CrowdOpinion with population-level learning. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 950-966, Toronto, Canada. Association for Computational Linguistics.
* Wilson et al. (2005) Theresa Wilson, Paul Hoffmann, Swapna Somasundaran, Jason Kessler, Janyc Wiebe, Yejin Choi, Claire Cardie, Ellen Rlioff, and Siddharth Patwardhan. 2005. Opinionfinder: A system for subjectivity analysis. In _Proceedings of HLT/EMNLP 2005 interactive demonstrations_, pages 34-35.
* Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. _Advances in Neural Information Processing Systems_, 34:27263-27277.
* Zellers et al. (2021) Rowan Zellers, Ari Holtzman, Elizabeth Clark, Lianhui Qin, Ali Farhadi, and Yejin Choi. 2021. TuringAdvic: A generative and dynamic evaluation of language use. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4856-4880, Online. Association for Computational Linguistics.
* Zhu et al. (2018) Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In _The 41st international ACM SIGIR conference on research & development in information retrieval_, pages 1097-1100 



## Appendix A Subreddit Community Guidelines

Table 5 illustrates that when SFT focuses purely on safe advice from LPT, it leads to a safer LLM with a comparable level of helpfulness. However, composing a minor portion of unsafe advice, 14%, in line with the AdvisorQA dataset's current composition, results in the LLM advisor being quickly toxic. This means that it is easier to learn unsafe advice patterns, which is why we have mixed ULPT into the dataset for broader future research. Regarding PPO, PL (5) model used as the reward model does not reflect harmlessness. As a result, during PPO training, the model does not become safer; instead, it rapidly explores harmful scopes, especially if the SFT is harmful. On the other hand, DPO, by matching the training dataset's distribution, follows the dataset's harmful advice ratio. Hence, DPO becomes safer if the dataset's ratio of toxic advice is lower than the probability of the SFT generating toxic advice. One of the key missions of advice-seeking question answering is to address the challenge of hidden toxicity in the real world for harmless advice. For diverse applications, each advice in the training set has been categorized as 'safe' or 'unsafe', ensuring the dataset's usability for training solely on LPT content.

## Appendix C Baselines Training Details

### Training Resources

We use four A6000 GPUs to train and evaluate each baseline. Therefore, experimental results and tendencies could be more apparent with rich GPU environments.

### Details and Hyperparameters for Evaluation Baselines

We detail the training process for the Plackett-Luce models. For PL (2), the 1st and 2nd pieces of advice per question simulate win/lose responses rather than the 1st and last. Moreover, due to limited GPU resources, we could not include comparisons for n-ranked advice in a single batch. Instead, we shuffled each comparison to train the PL (n)

Figure 8: These strict guidelines enable the tips from LPT to be safe, and ULPT to be unsafe.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline \hline Toxic ratio & Lima-7B & SFT & DPO & PPO \\ \hline
0\% & 0.78 & 0.87 & 0.93 & 0.83 \\
5\% & 0.78 & 0.84 & 0.86 & 0.76 \\
10\% & 0.78 & 0.75 & 0.83 & 0.69 \\
14\% & 0.78 & 0.66 & 0.76 & 0.63 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Relationship Between toxic advice ratio in the training set and harmlessness score for each trained model.

Figure 9: An example thread in LifeProTips: Each session consists of an advice-seeking question with detailed experiences, accompanied by various pieces of advice and discussion. After engaging in active discussions, users express their individual preferences through upvotes. We utilize the overall majority vote result, known as the upvote ranking, as a collective intelligence.



model. The hyperparameters used in this process were as follows.

### Details and Hyperparameters for Training Baselines

For limited GPU resources, all training baselines are based on QLoRA 4-bit Dettmers et al. (2023); Hu et al. (2021).

## Appendix D Case Study of AdvisorQA Dataset, failure and patterns of LLM-generated Advice

Table 10 shows why the number of upvotes is used as a proxy for helpfulness. Highly actionable or creative advice receives a high number of upvotes, while irrelevant or impractical advice receives a low number of upvotes. Table 11 and 12 is the example to analyze attributes of PPO-trained models and DPO-trained models. This case study shows PPO models give more empathic advice rather than DPO, and DPO models give more instructive advice with constructive forms. Table 13 shows the various ways in which Llama-2 fails at advice-seeking QA. It fails due to a lack of theory-of-mind, lack of creativity, failure to understand context, and degeneration in very specific and everyday contexts.

## Appendix E Human Evaluation

The selection of 10 crowd workers for human evaluation was carried out through the university's online community, focusing on individuals who demonstrated strong proficiency in English. These workers received detailed explanations of the tasks, along with instructions and examples, as shown in Figure 11. They were also informed that the evaluation was for academic research purposes. Following a trial evaluation to determine the necessary time commitment, the workers were appropriately remunerated, guaranteeing an hourly wage of at least $12, as agreed by the workers themselves.

Table 3 involves an experiment that tests the validity of using upvotes as a proxy for helpfulness for the human evaluation baseline. Therefore, annotators experimented on 300 random samples to determine which of the two advices is more helpful, testing if they can accurately match the ground truth upvote rank.

To explore the helpfulness of each training RLHF baseline PPO and DPO compared to SFT by GPT-4-Turbo and human, we collected 100 responses from the test set. Then, we prompted them to compare responses from the RLHF and SFT models and report the results.

\begin{table}
\begin{tabular}{c|c} \hline \hline Hyperparameter & Value \\ \hline epochs & 3 \\ learning rate & 5e-6 \\ batch size & 8 \\ max token & 1024 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Hyperparameters used for training plackettluce models.

\begin{table}
\begin{tabular}{c|c} \hline \hline Hyperparameter & Value \\ \hline epochs & 5 \\ learning rate & 5e-6 \\ batch size & 5e-6 \\ max token & 512 \\ \hline LoRA \(\alpha\) & 16 \\ LoRA dropout & 0.1 \\ LoRA \(r\) & 64 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Hyperparameters used for supervised fineuning.

\begin{table}
\begin{tabular}{c|c} \hline \hline Hyperparameter & Value \\ \hline epochs & 2 \\ learning rate & 5e-6 \\ batch size & 32 \\ max token & 512 \\ \hline LoRA \(\alpha\) & 16 \\ LoRA dropout & 0.1 \\ LoRA \(r\) & 64 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Hyperparameters used for PPO.

\begin{table}
\begin{tabular}{c|c} \hline \hline Hyperparameter & Value \\ \hline epochs & 2 \\ learning rate & 5e-6 \\ batch size & 32 \\ max token & 512 \\ \hline LoRA \(\alpha\) & 16 \\ LoRA dropout & 0.1 \\ LoRA \(r\) & 64 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Hyperparameters used for DPO.

 



\begin{table}
\begin{tabular}{l p{284.5pt}}
**Type** & **Content** \\ \hline
**Advice-seeking question** & how can I train my body to wake up to an alarm? My alarm was going off for 20 minutes before my brother had to walk out of his room down the hall and he lightly said my name and I snapped awake. \\  & You can go two routes, I’ve tried both and they work reasonably well. \\  & 1. Spend a bit of money and buy a Sonic Bomb. It’s super loud has a backup battery and a vibration coil for under your mattress (I hold it in my hand under my pillow). \(\sim\$50\) \\  & 2. You can download an app on your phone that reads your movements while you sleep and determines when you are in a light sleep vs a deep sleep. I have one that goes off in 15-60 minute period when it detects I'm in light sleep. Works pretty well asling as you get enough sleep. \(\sim\) Free \\  & I use them in conjunction, if the phone alarm wakes me up before the sonic bomb I can turn it off before my neighbors call the cops! Lol seriously though if that happens the vibration coil should do a pretty good job. \\  & Drink a decent amount of water before bed. When your alarm goes off you’ll have to peee so you’ll be forced out of bed anyway. \\  & In all honesty, just wait. \\  & I’m assuming that you’re a teenager, since you still live with a brother. It’s normal for teens to have trouble waking, as sleep is sort of a weird thing for teens. As you grow older, you’ll wake easier and easier. When I was a kid, nothing short of a bomb would wake me. Now, I don’t even use an alarm. \\  & Just get up. Once you are conscious get up and make up your bed, brush your teeth, get some cereal, do anything but just sit there being tired and lazy. \\  & The groggy feeling won’t until x amount of time has passed anyway. Never lie in bed once you are awake. \\  & After a while you be up consistently at whatever time you so chose. \\  & In the above case, the best advice (68 upvotes) for struggling to wake up with an alarm is actionable, specific, and highly helpful. Also, the second one with 10 upvotes might be less practical but a creative solution. However, the last two advice posts are not considered helpful for the given situation. One of the lowest-ranked advice simply suggests “just get up,” which is unhelpful as it doesn’t address the reason for seeking advice. The advice with one upvote suggests that age will resolve the issue, which is irrelevant to the context. Therefore, upvotes, symbolizing the agreement ratio of many users, can be regarded as a proxy for helpfulness in AdvisorQA. \\  & \\ \end{tabular}
\end{table}
Table 10: Examples of advice-seeking question answering in AdvisorQA Datset



\begin{table}
\begin{tabular}{l l} \hline \hline
**Type** & **Content** \\ \hline  & **How to stop looking at your phone so much** Like a lot of people (especially younger people), I have grown up staring at a screen and been fed instant gratification all my life. **I want to learn how to be present on the moment, and stop turning to my phone for little bite sized bits of dopamine** browsing anything else. I have decided recently to delete most my social media apps (Facebook, Twitter, Instagram) and started slowly trying to remove myself from Reddit and Snapchand. What else can I do? \\ \multirow{6}{*}{**Advice-seeking question**} & Replace the habit with one you want to have. But, give yourself the grace of time and backslides. First, you have to recognize when and why you are looking at your phone. Then, you need to train yourself to pause before you act on the trigger. Once you master pausing you now have the time to redirect your brain to the activity you want to do in place of looking at your phone. Maybe it’s read a book or walk more. So, you now experience the trigger, practice the pause, and redirect yourself to reading a book you keep on you at all times or start pacing/walking. Eventually, you will fall out of the habit of thinking of your phone and won't be triggered to look at it. That said, make it difficult to get to your phone. Either put a really intricate pin on it or always have it in a different room. There are also apps that will keep it on lock down for set periods of time. You might slip up. It’s ok. Forgive yourself and reset. Don’t let it throw off the rest of the day. May the odds be in your favor. \\ \multirow{7}{*}{**1st Human Advice**} & 1. Try time blocking: Allocate specific times during your day for checking your phone... \\ \multirow{7}{*}{**1st Human Advice**} & 2. Take breaks: Every hour or so, take a short break to stretch, walk around, or do something else. ... \\ \multirow{7}{*}{**1st Human Advice**} & 3. Use technology to your advantage: Use apps that help you limit your screen time, ... \\ \multirow{7}{*}{**1st Human Advice**} & 4. Focus on one task at a time: When you are working on a task, focus on it fully. Avoid multitasking and ... \\ \multirow{7}{*}{**1st Human Advice**} & 5. Be mindful: Be present in the moment. Pay attention to what you are doing and where you are. Avoid getting distracted by your phone. \\ \multirow{7}{*}{**1st Human Advice**} & 6. Find a lobby or activity you enjoy: Find something you enjoy doing that doesn’t involve your phone. ... \\ \multirow{7}{*}{**1st Human Advice**} & 7. Seek support: Reach out to friends or family members who can provide support and encouragement. You can also join a support group or seek professional help if needed. \\ \multirow{7}{*}{**1st Human Advice**} & Remember, breaking the habit of constantly checking your phone takes time and effort. Be patient and don’t be too hard on yourself if you ship up. Just keep trying and you will eventually see the benefits of being more present in the moment. \\ \multirow{7}{*}{**1st Human Advice**} & 1) Make the effort to meet new people. I went a full year not using social media. I found that I did not feel so bad when it was gone, however, I was also not meeting new people, I wasn’t expanding myc circle. .... I have been meeting new people. Look for meetup groups on anything that interests you and look for groups that have just started. Attend their first few meetings and see what you like and what you don’t. \\ \multirow{7}{*}{**1st Human Advice**} & 2) If you are feeling insecure or self-conscious about socializing and meeting new people, try going to a coffee shop where people will be alone and reading or just hanging out and socializing. This is a really good way to practice ... \\ \multirow{7}{*}{**1st Human Advice**} & 3) Practice mindfulness. There are many different techniques, ... Do this all day, and it will get easier and easier. \\ \multirow{7}{*}{**1st Human Advice**} & 4) Practice gratitude. Think of 3 things you are grateful for every day. Focus on those things when you are feeling bad. Gratiude is a powerful tool. I am also on the journey of gratitude and self-awareness. \\ \multirow{7}{*}{**1st Human Advice**} & Best of luck to you, my friend. \\ \cline{2-3}  & \\ \hline \hline \end{tabular}
\end{table}
Table 11: Examples of advice-seeking question answering from AdvisorQA-trained models



\begin{table}
\begin{tabular}{l p{284.5pt}}
**Type** & **Content** \\ \hline
**Advice-seeking question** & **What to do in the wake of an awkward silence?** (One that you caused and/or one that was caused by another person) \\  & If I found silence awkward, there would be a lot of awkward silences in my life. The trick is to just not care. If I’m with someone and the conversation dies out, I don’t try to think of anything to say. Not a thing. I will sit there and relax until something comes up. I don’t try to force conversation or small talk. Nor do I state at my phone, either. Just relax and go with the flow and don't feel pressured. Being relaxed and not worried about it is more attractive than trying to force conversation. Stop worrying about the silences and they stop being uncomfortable. \\  & 1. Ignore it and go on with the conversation as if it didn’t happen. \\  & 2. Apologize for creating it and try to find common ground or introduce a new topic. \\  & 3. Make a joke to lighten the mood. \\  & 4. Ask the other person how they are feeling and listen actively. \\
**Llama-7B-DPO** & 5. Ask open-ended questions to encourage them to continue talking. \\  & 6. Change the subject to something more interesting or less awkward. \\  & 7. End the conversation and thank them for their time. \\  & If you caused the awkward silence, I recommend options 2-5, while if it was caused by another person, I recommend options 1-4. Remember to always listen actively, be empathetic, and try to make the other person feel comfortable. \\  & 1) Change topic. 2) Start a story. 3) Ask them about a topic you know they like. 4) Try to start a new friendship. \\  & Remember that you have to be comfortable with your own awkwardness in this situation. It is also possible that it will become apparent to them that you are not comfortable and they may try to make it awkward for you. That’s okay, in my experience people are far more friendly and accepting of awkward people than I would have thought. They know how uncomfortable you are, but I think they also know you are brave. \\
**Llama-7B-PPO** & In any case, you get to hang around cool people in these situations and see how they behave, so it’s a win/win situation. \\  & Remember that it’s okay to just not talk and hang out in the background. \\  & Also, if you see the person later in life and they don’t remember you, just smile and let it go. I think that’s the best way to handle awkward people you don’t know well and will never see again. \\  & Best of luck to you in all of your awkward conversations! \\ \end{tabular}
\end{table}
Table 12: Examples of advice-seeking question answering from AdvisorQA-trained models



\begin{table}
\begin{tabular}{l l} \hline \hline
**Type** & **Content** \\ \hline  & How best to deal with a friend who is constantly in “detailed story-telling mode”? A really great guy, super nice, lots of things in common, but.. They are constantly talking. Mainly about their experiences, in detail, without a breath. Can’t even walk away from him to use the bathroom or whatever, it’s always, ”oh, and listen to this part (...)” \\  & Ask them if they would like to switch roles and do you the same. Explain that you appreciate the story, but you have your own stories to tell. Let them know that you are happy to listen to them tell about themselves and their experiences, but you would like to return the favor and tell them about yourself as well. \\  & The questioner wants to escape from the other person’s excessively long conversation, but the LLM’s advice is about how to take my-turns in the conversation. The cause of this failure seems to be a lack of ‘Theory-of-mind’. \\  & What’s a skill I can learn with a few minutes practice a day? Doesn’t matter if it’s a social skills or something “useful” it could be entertaining. (...)” \\  & You can learn a skill in a few minutes practice a day by learning something related to a hobby \\  & or interest you have. For example, if you like photography, you can practice taking photos. The same goes for learning a new language or even learning how to play an instrument. The important thing is to find something that interests you and practice it regularly. \\  & The questioner is looking for something creative that can be consistently achieved by setting aside a very small amount of time in daily life, but the LLM recommended photography, which requires a lot of time for a single practice session due to a lack of creativity. On the other hand, the model trained on AdvisorQA recommended practices that require little time per session, such as Meditation and Card Tricks. \\  & How to keep your hands cool? I know this sounds silly, but if you simultaneously get cold easily and suffer (tempted to put that in quotation marks) from warm/sweaty hands, back me up, here. Seems like it should be physiologically impossible. It’s only a problem at work; there’s a negligible loss of time romantically holding hands with my SO, but my tools of the trade are mouse and keyboard, which makes this problem annoying for obvious reasons. (...) I’ve done the due process googling, but most results are about exercise or how to *warm* your hands. Oh how at times I envy those with warm bodies and cold extremities. \\  & 1) You could try using a heated hand warmer that has been clinically proven to maintain a constant body temperature for up to 8 hours. Just warm your hands in it and then put them in the palm of your keyboard and they ’ll stay warm while you type. (...) \\  & The question is about how to keep hands cold, and at the end of the text, it is mentioned that \\
**Cues of failure** & even when Googling, only methods for making hands warm come up. However, due to a lack of contextual understanding, the LLMs suggest ways to make hands warm instead. \\  & Finding a lost cat Im sorry for the wall of text in advance but this is urgent TL:DR got home from work to my (inside) cat roaming the front yard. The girl who let my dogs inside while i was at work left the backdoor open and the cat escaped. Tried to coax him but he took off and give been roaming around my neighborhood shaking cat treats for him. unhive heard of people saying to find lost dogs tops out a article of clothing outside for them to pick up on the scent, we done that. Anything else i can do? There is also another cat around my house that seems to be in heat, mines neutered so idf if he cares but i think that cat may be scarring him off. IdK if this is the right place to post this but if anyone could help id greatly appreciate it, I didn know where else to turn. uN \\  & Uhumanuunmu

To explore the contradicted values preferred by GPT-4-Turbo and PL models, we detailed an explanation of each option with the following guidelines and interface.

1. Relevance: If the lost response is irrelevant to the given question, choose this option.
2. Actionability and Practicality: If the win response is more realistic to act and practical solution, choose this option.
3. Empathy and Sensitivity: If the win response sympathizes with the question deeply, choose this option.
4. Creativity: If the win response is more creative and irregular than the lose response, choose this option.
5. Readability and Clarity: If the win response is easier to understand, choose this option.
6. Constructiveness: If the win response is more comprehensive advice (such as pros, cons, or very complete advice), choose this option.
7. Ethics: If the win response is more ethical advice, choose this option.

Figure 10: Expanded visualization for topic distributions of advice-seeking questions in AdvisorQA.

 



## Appendix A

Figure 11: The interface for human evaluation