A Graph per Persona: Reasoning about Subjective Natural Language Descriptions (2024)

Abstract: Reasoning about subjective natural language descriptions, such as opinions and preferences, is a challenging topic that largely remains unsolved to date. In particular, state-of-the-art large language models (LLMs) perform disappointingly in this task, show strong biases, and do not meet the interpretability requirements often needed in these kinds of applications. We propose a novel approach for reasoning about subjective knowledge that integrates potential and implicit meanings and explicitly models the relational nature of the information. We apply supervised graph learning, offer explanations for the model’s reasoning, and show that our model performs well across all 15 topics of OpinionQA, outperforming several prominent LLMs. Our detailed analysis further shows its unique advantages and the complementary nature it offers in comparison to LLMs1.

heading: 1 Introduction
text: Subjective knowledge such as personal opinions and preferences represents a considerable challenge for automated reasoning. In fact, on the recently proposed OpinionQA benchmark (Santurkar et al., 2023), a collection of 15 subsets of survey questions by the PEW Research Center2, even the stateof-the-art large language models (LLMs) reach surprisingly low scores and reveal certain biases (Santurkar et al., 2023; Hwang et al., 2023). As LLMs are incorporated into applications aimed at assisting individuals in daily tasks and decision-making (OpenAI, 2023; Google, 2022; Ye et al., 2024), it is imperative that they can personalize their outputs for individual users.
One of the inherent problems with reasoning with subjective knowledge is its implicit nature.
1Project page: https://github.com/eujhwang/graph-per-persona
2https://www.pewresearch.org/; See Appendix H for details about the 15 subsets.
Rather than explicitly specifying their preferences and opinions, users may express these opinions indirectly through continuous interactions. Other properties that affect opinions and preferences may be external to the discourse, such as demographic information and cultural background (Suriyakumar et al., 2023). Finally, we observe that various aspects of a problem are usually related, and the models often have to combine various pieces of information.
To test LLMs’ ability to learn personal opinions, the OpinionQA datasets present models with the dialogue history containing a participant’s responses to survey questions (e.g., Do you use a password manager to help keep track of your online passwords?), as well as their demographic information (e.g., Age: 50-64, Political affiliation: Republican). The model is then tasked with answering a set of multiple-choice questions pertaining to the opinions (e.g., Yes/No).
Current state-of-the-art LLMs still perform poorly on OpinionQA (Santurkar et al., 2023). In particular, models often ignore the survey history and over-rely on demographic information, which may lead to perpetuating societal biases (Hwang et al., 2023). Moreover, English LLMs struggle
1928
with questions from cross-national surveys (Durmus et al., 2023), given that they are trained on English web text coming primarily from users in the US. Current solutions focus on improving the reasoning by filtering the information that is available to the model when making a certain judgment (Hwang et al., 2023; Do et al., 2023), but there is still considerable room for improvement. Further, observe that general-purpose LLMs may not best suit this task by design, specifically alignment, since sycophancy (i.e., models adjusting their responses to align with a human user’s perspective, even when that perspective lacks objective correctness) is considered an undesirable behavior (Sun et al., 2024).
We propose an alternative approach to reasoning about subjective descriptions, inspired by traditional techniques modeling the relational nature of complex conceptual knowledge in semantic networks (Lehmann, 1992). Our framework, depicted in Figure 1, creates one opinion graph per individual, explicitly modeling relationships between their opinions on various topics (green). Due to the often intricate and implicit nature of opinions, we complete the graph with derived knowledge generated by an LLM (yellow). Finally, we add auxiliary nodes for the answer choices (blue, rose) and apply supervised graph learning to determine the opinions that are most relevant to the given question.
Our approach outperforms prominent LLMs across most of the 15 OpinionQA subsets. We ablate and evaluate our approach in detail. Most importantly, our analysis shows that our answers often complement those of the LLMs, which offers interesting future research potential. Finally, the graph neural network allows for extracting the attention flow over the graph nodes and hence naturally delivers an explanation for its reasoning. While the explanations are not perfect, they are useful for analyzing the reasoning steps and hint at future research questions.3

heading: 2 Related Work
text: Reasoning about Subjective Descriptions. Simpler forms of reasoning over subjective text have been studied in NLP for a long time in tasks such as sentiment prediction or user-item recommendation (Gao et al., 2023; He et al., 2017; Li et al., 2021). More complex tasks, predicting an opinion based
3We will make the code available upon publication.
on other opinions, have been considered recently with the study of personalized question answering over surveys (Santurkar et al., 2023; Durmus et al., 2023). Overall, LLMs have been shown to be underperforming (Santurkar et al., 2023; Ziems et al., 2023). Among their many findings, we point out the importance of curated personal opinions for personalized prediction (Hwang et al., 2023; Do et al., 2023). Understanding the model’s ability to reason about human opinions is crucial to ensure safer alignment with a user’s ethical principles, moral beliefs, and culture-specific values. We build upon the previous works by focusing on opinion data and employing graph learning to select opinions relevant to the task at hand.
Importance of Implicit Information. Most popular reasoning benchmarks focus on reasoning on objective knowledge. Additional factual context has been shown to improve LM reasoning in these setups (e.g., Akyürek et al., 2024). In the subjective context, we draw inspiration from the early work of Hobbs et al. (1988), who showed that explicit representations of meaning can help text understanding. More recently, Hoyle et al. (2023) showed the importance of having explicit representations of implicit content with LLMs. We adopt this finding into our graph-based reasoning framework, which is an alternative to the popular chain-of-though reasoning paradigm (Wei et al., 2023; Yao et al., 2023; Besta et al., 2024), in which the LLM is reasoning in natural language. These methods often overly rely on demographic information when reasoning over human opinions, even in the presence of related opinions (Hwang et al., 2023).
Relational Reasoning. “Relational reasoning, or the ability to consider relationships between multiple mental representations, is directly linked to the capacity to think logically and solve problems in novel situations” in humans (Cattell, 1971; Crone et al., 2009). Motivated by this, graphs have been employed in NLP models to represent knowledge, primarily for reasoning about objective knowledge. (Jung et al., 2020; Xu et al., 2019; Das et al., 2021). To simulate step-by-step reasoning, Jung et al. (2020) and Das et al. (2021) particularly integrate reasoning paths in the models. We use the graph-based reasoning model from Jung et al. (2020).

heading: 3 Our Approach
text: Overview, Figure 2. Given a user’s answers to previous opinion questions, our goal is to predict the answer to a multiple-choice question about an unstated opinion. We exploit the relational nature (entailment information) of personal opinions and create a graph for each person, containing their known opinions as nodes and, additionally, potential implicit meanings and relations between them; Sec 3.1. We encode the graph and consider a supervised learning problem where the given question and possible answer nodes are added to the graph as auxiliary nodes, and the graph model is biased to learn paths (think of sequences of attention values between graph nodes) between them; Sec 3.2. Lastly, we extract the highest-ranked paths (i.e., the nodes most relevant to the task) to predict an answer; Sec 3.3.
Notation. We consider a given set of multiplechoice questions answered by a specific person: {(qi, ai, Ci)} containing questions qi, corresponding answer choices Ci, and the chosen answers ai ∈ Ci. The question answering task is similarly given as a tuple (q, a, C) not part of the above set, where a denotes the correct answer.

heading: 3.1 A Graph per Persona
text: I Given Opinions. We follow Hwang et al. (2023) and use the Wizard-Vicuna-30B model (Luo et al., 2023; Chiang et al., 2023) to convert each question-answer pair into a declarative sentence
(e.g., I do not use a password manager to help keep track of my online passwords.). We obtain a set O = {(qi, ai)} representing the answers of a given survey participant and a set T = {(q, c) | c ∈ C} representing the task.
II Generating Implications. We use WizardVicuna-30B to generate implications from the explicitly given opinions (see Appendix D for the prompt). For example, from the given statement: “I do not use Instagram”, we can infer that the person may not be interested in sharing photos or videos on social media.
Since we observed some of the generated implications to be irrelevant in the context of the given opinion (see examples in Appendix E), we filter them as follows. We calculate the cosine similarity between the given opinion and each implication, and implications with a cosine similarity below a pre-defined threshold tsim are discarded (we used tsim = 0.8, based on preliminary experiments).
III Graph Construction. We construct a multirelational graph G = (V,R, E) per person. The opinions O and implications I represent this set V of belief nodes, and we add the task encoding, i.e., V = O ∪ I ∪ T . For brevity, we often call all in O ∪ I opinions, although the implications are only potential derivations.
To capture an entailment relationship between opinions, we decide to represent the opinions as a graph structure. Since we generate multiple implications for each opinion, the graph should be dense
by design. However, we are still missing more detailed knowledge about the exact nature of the connections (i.e., about the type or strength of the individual relations between two nodes). We consider the setR of relation types to contain one type for opinion-opinion, opinion-implication, implicationopinion, and implication-implication edges, respectively, and define the set E of edges to contain all corresponding tuples (vi, vj , r) ∈ V×V×R. That is, we have two edges between each pair of nodes, one in each direction; for a uni-directional relationship, we add two tuples differing in the type.
Nevertheless, the implications are considered to be consequences of the opinions, and we assume additional such entailment relations to hold between other beliefs of the person. To model this information explicitly, we considerR to contain an additional entailment relation type. We compute these entailment edges using an LM, as described next.
We use a state-of-the-art model for natural language inference (NLI), T5-base (Raffel et al., 2020) to predict the probability pij for the graph edges to represent entailment.4 We also use these predictions to filter out noise in terms of relationships, in that we consider the predicted entailment score to tell us about relatedness between our additional implications (and opinions) and filter out all edges below a pre-selected threshold tentail (in our experiments, we chose tentail = 0.1 based on manual observation). The final graph is thus no longer fully connected, but still dense enough for the model to broadly explore the space.
We fine-tuned the NLI model using the implications generated previously, since the model may lack prior knowledge about the specific domain under consideration (e.g., this was the case for the data we experimented with). More specifically, we considered each pair of opinion and the corresponding generated implication as a positive example, and constructed a single negative example for each positive example by pairing an opinion with a randomly chosen implication that was generated for another answer choice for the same question.

heading: 3.2 Reasoning over the Graph
text: Initial Graph Representation. For embedding the graph nodes, we apply a sentence embedding MS : V → RdS (we used Sentence Trans-
4We also experimented with Flan-T5. T5 and Flan-T5 turned out to have similar performance in understanding the entailment relationship between subjective opinions.
former5); a unique identifier for opinion nodes op : V →∈ Rd, which maps implications to the identifier of the opinion they were generated for; and a (binary) node type identifier typ : V → Rd, which distinguishes opinion and implication nodes. We create an embedding as follows, for each vi ∈ V:
h0i = Wv[MS(vi)||op(vi)||typ(vi)],
where Wv represents a linear transformation. The edge representations unify all relationships we have between a given pair of nodes vi and vj as follows: e0ij = Wee ′ ij , We is a linear transformation, and e′ij a one-hot vector with one flag per r ∈ R. That flag is set to 1 if (vi, vj , r) ∈ E ; for the entailment relation, we set it to 1 if pij > 0.5, according to the predicted entailment probability.
Graph Learning using Graph Attention Flows. The goal in graph representation learning is to compute node representations htj iteratively, for each layer t, by aggregating the embeddings ht−1i of the incoming neighbor nodes vi ∈
#»N j , i.e., (vi, vj) ∈ E . The graph attention network (GAT) (Veličković et al., 2017) specifically applies attention to weigh the neighbors,6 and there are versions taking relation types into account (Salehi and Davulcu, 2019). To emphasize the flow of information over the graphs, we follow works which compute the training loss including attention values of all model layers, in that the first layer computation models the first task reasoning step, from the question node to an opinion node; the second layer models a step from one to another opinion node; etc. until an answer node is reached (Jung et al., 2020; Xu et al., 2019). The model explores such paths in parallel. The goal is to obtain attention values ãti as a representation of the importance the answer choices have in the context of the opinion nodes. Formally, at each layer t (for readability we drop many superscripts ·t): • We first compute node embeddings using GAT:
ht+1j = σ
 ∑
i∈ # »Nj
aijWk(h t j + e t ij)
 
aij = softmaxi∈ # »Nj (e t+1 ij )
et+1ij = σ((Wn(h t j + e t ij)) · (Wmhti)⊺)
5BAAI/bge-base-en-v1.5 6Observe that this can be seen as transformer architecture with a strong structural prior, in that attention for node pairs that are not connected by an edge are always 0.
where σ denotes leaky-ReLU and, for simplicity, j and vj are used interchangeably. • To bias the computation towards the question answering task under consideration, we incorporate a representation q of the target question, a sentence embedding acquired by Sentence Transformer. In case we want to consider demographic features, we embed them similarly, obtaining an embedding d:
ĥt+1j = h t+1 j +Wqq(+Wdd).
• Instead of directly taking GAT’s attention values as node importance scores, Jung et al. (2020) normalize them in the context of their neighbors and incorporate the values from previous steps. Note that initial scores ã0i then have to be given, we compute:
ã0i = h 0 i · (Wqq+Wdd)
To obtain normalized attention values ãt+1ij for each neighbor vi, we weigh the edge from vi to vj in the context of vi’s outgoing neighbors
←−Ni and compute that impact γij similar as above:
ãt+1ij = γ t+1 ij ã t i γt+1ij = softmaxj∈←−Ni(ê t+1 ij ) êt+1ij = σ((Wn′(ĥ t+1 i + e t+1 ij )) · (Wm′ĥt+1j )⊺)
Note that here the t+ 1-step’s node embedding impacts the node score. We obtain the final value by aggregating the incoming edges’ (at vj) values. Thus, a high score for the target node means it has a large influence on its neighbors.
ãtj = ∑
i∈ # »Nj
ãtij
Training Objective. We apply supervised learning as proposed by Jung et al. (2020); Xu et al. (2019), by focusing on the attention scores computed for the target answer node vtarget across all layers t ∈ T . Note that we cannot use other opinion nodes relevant to the task for supervision because
our data does not contain such ground truth information.
L = T∑
t=1
− log ãttarget

heading: 3.3 Extracting Relevant Opinions
text: To determine the answer, we extract paths in the graphs with the highest attention scores up to a depth T , considering each to contain opinions most relevant to the task; we chose T = 3. We collect these paths using a beam search, starting at t = 0 and consider the k nodes vi with highest values ãti and iteratively select the k neighbors with highest ãt+1i for each of them. We stop at t = T , drop all paths that do not end in an answer node, and score each remaining path P as follows:
sP = |P |
√√√√ |P |∏
t=0
ãtP (t),
where |P | denotes the length of P , and P (t) the index of the t-th node in P .
Then we obtain a score Ansc per answer choice c, by aggregating the top-k scores of the paths P ∈ P top-kc leading to that answer; we used k = 5. Lastly, we select the highest one as the final answer.
Ansc = ∑
P∈P top-kc
sP
Ansfinal = max({Ansc})
We chose this prediction mechanism based on the top-k paths to include alternative sets (i.e., paths) of opinions into the prediction; we will also focus on the opinions in all top-k paths in our evaluation.

heading: 4 Evaluation
text: Settings. To test the model’s personalization and reasoning ability, we use OpinionQA datasets and train and test the models in a question-answering (QA) setup. In terms of baselines, we consider BERT (Devlin et al., 2018), Mistral-7B (Jiang
et al., 2023), text-davinci-003 (GPT-3), gpt-3.5turbo (GPT-3.5), and ChOiRe (Do et al., 2023). The LLMs are used in a zero-shot setting. We use accuracy as the primary performance metric.
For the BERT and LLM baselines, we use the top K of the user’s prior opinions, based on embedding similarity to the given question, using OpenAI’s text-embedding-ada-002 model. This follows the setup introduced by (Hwang et al., 2023), which was shown to be better than providing all known user opinions. In our approach, we use all of the previous opinions of a person and build a graph to reason about the opinions that have the most relevance and implications for the target opinion. Prompts we used for LLM baselines and more details about the model and hyperparameters used for experiments are given in Appendix A.
Overall Performance, Tables 1, 2, 3. At first glance, our models compete well with the LLMs. In particular, they show consistently good/best performance with and without demographic information. Among the LLMs this is only the case for GPT-3. We posit that the GPT3+ models trained on considerably larger datasets might have a better understanding of opinions.
We observe notable differences especially on Guns, Biomedical-food, and Misinformation. Comparing our models with and without implications, we observe that including implications significantly improves performance on most topics, particularly on Sexual Harassment, Misinformation, and Global Attitudes. Similarly, the entailment information further shows rather consistent performance improvements. Since the main table considers subsets of the data as they were used in previous works (Hwang et al., 2023; Do et al., 2023), we check what happens if we increase the number of survey participants whose answers we consider to 500, see Table 3. Interestingly, the positive impact of our proposed architecture gets clearer.
Note that, in the setting with demographics, we do not consider the entailment version of our model since the entailment probabilities are computed for the original textual nodes but the demographic information, incorporated at each node, will likely change the nature of this relation.
Reasoning Examples, Figure 3. We start the analysis by showing an example, which also demonstrates the challenging nature of the problem. The figure shows the top-5 paths found leading to a correct answer prediction in GOO. Over-
all, we see that the fully-connected nature of the graph makes it possible to derive the answer directly based on a few relevant opinions (i.e., the paths are rather short). While these selected opinions seem all rather similar at first glance, observe that especially the derived, potential implicit meanings The person may . . . point out interesting, often rather subtle aspects (e.g., possible political opinions, values more generally, or consequences on future plans). A more detailed error analysis is presented later, other examples are given in the appendix.
Analyzing Predicted Relevant Opinions, Table 4. To give an impression of the nature of the explanations, we present statistics about the node types in the best paths, which reveal that they equally rely on explicit and implicit knowledge. The overall number of about three relevant opinions on average, plus two derived ones, seems reasonable. Observe that, when explicitly asked to explain its reasoning, Mistral op gives a similar number of opinions. Table 9 in the appendix further shows the similarity in predicted relevant opinions in terms of overlap between different models and model variants on the correct predicted paths where both models agree. We see that the overlap between models can be rather low, which shows the need for making this information explicit and thus verifiable. These numbers also highlight that, in our model, adding entailment information can have more im-
pact on the explanations than adding demographics. This underlines the power of this kind of implicit semantic and relational knowledge.
Moreover, we conducted a human evaluation comparing the outputs generated by our op+imp+entail model and Mistral-7B op through
Amazon MTurk. We randomly selected 30 examples, two per topic, and each example was evaluated by three annotators. Annotators were asked to determine whether the target opinion could be inferred from a set of opinions chosen by our model (yes/no), along with a brief explanation. Based on the latter, we manually filtered out 13% noise. Overall 83% of our examples were deemed reasonable. Mistral-7B achieved a rating of 87%. However, note that the LLM was given the top-8 most similar opinions to the target question. Thus, finding relevant ones among those is much easier, and the scores are not directly comparable.
In what follows, we analyze the predicted answers in detail and show that both GOO and LLMs have unique advantages. Thus our work presents a promising, novel method to complement LLMs.
Comparing Individual Predictions, Table 5, Appendix B. We compute the agreement in correct
and incorrect predictions between Mistral-7B and GOO. The numbers on a per-topic basis show that the trend is rather consistent and well reflected in the corresponding averages, 34/18/21/27%. This shows that the models may complement each other: When we combine the three cases where either of the models provides the correct answer, we can significantly improve the individual models’ performance and obtain 73% accuracy.
We further show the agreement rates between the model variants, (e.g., GOO with and without entailment information) in Appendix B. Overall, we see that the agreement in both correct and incorrect predictions is considerably higher for variants of the same model than for different model families, both are around 40-50% across topics. First, this can be considered as verification that GOO is reasoning consistently in that adding information does not completely change the nature of the predictions. Interestingly, this is even the case where we compare the versions with(out) demographics for our model, but also for Mistral-7B in Table 7. Hence this also shows that combining different reasoning approaches (or model families) can be a promising direction to explore in the future.
Comparing Predictions on the Level of Individual Persons, Figure 4. The figure illustrates the distribution of how the model performs on a per-person basis, compared to Mistral-7B. We selected three topics where our model performs better than/similarly to/worse than Mistral-7B. The distributions from our model are generally less skewed, meaning that it shows more equal performance across individuals. In Mistral-7B, we observe that while the model achieves very high
performance for certain people (Biomedical-Food and Privacy), resulting in an overall performance increase, there are also more individuals for which it’s performing worse than our model. This experiment gives a more detailed view of how our model, or maybe even supervised learning more generally, could complement LLMs, to mitigate biases due to the potentially highly biased pre-training data.
Comparing Majority Predictions across Demographic Groups, Table 6. Here, we zoom out from the level of individual persons and consider the majority prediction of groups (i.e., all people in the dataset, and for groups with different political affiliations). Specifically, we compare them to the majority prediction from GOO and the LLM for those groups. There are interesting trends. First of all, the numbers are overall considerably higher for GOO, which makes it seem that the supervised approach allows the model to capture commonalities for certain populations, while this seems not the case for the LLM. Moreover, GOO does similarly well on all groups, even though the data itself is slightly biased (# rep./dem./ind.: 774/1075/683). On the other hand, the LLM, also here, shows clear bias (towards dem. opinions), even when given
extra demographics. Overall, incorporating demographic information seems to generally enhance the models’ ability to capture majority opinions.
Common Errors, Appendix C. We manually checked wrong predictions and corresponding explanations, see examples in the appendix. Amongst others, we noticed that including demographic information can overly strengthen a particular node and wrongly influence the selection of subsequent path nodes. Overall, we observe that the inclusion of demographics needs more careful consideration and study in future work. Furthermore, the diverse and nuanced context our graphs provide occasionally leads the model to irrelevant conclusions.

heading: 5 Conclusions
text: We propose a novel approach for reasoning about subjective natural language descriptions. Our approach represents a person’s opinions in a graph which also includes generated implications, explicitly modeling the relationships between various statements. Given a question about a previously unstated opinion, we apply supervised graph learning to find reasoning paths from the existing knowledge to one of the candidate answers. Our model outperforms several prominent language models across all 15 topics of OpinionQA, while also offering explanations for its predictions. Detailed analysis further shows our model’s unique advantages and the complementary nature it offers, in comparison to LLMs. Altogether, our work proposes a promising research direction to address this challenging problem and opens up interesting future research.
Limitations
From a data perspective, our work showed that we need better methods to integrate demographic or other additionally given information (i.e., beyond opinions), which is left as a challenging question for future research. We further note that our work, similar to the related works on the topic, focuses on the somewhat restricted survey scenario, where all users are captured in terms of one set of descriptions. If the latter varied (e.g., by having freeform answers), our supervised learning problem would become considerably harder. Our analysis has also clearly demonstrated that the implicit knowledge added using an LLM is often sensible, and manual checks are critical. Moreover, our approach is somewhat complex in that we need to apply an LLM during training for obtaining the
derived knowledge; it is very efficient for inference though. For the LLM comparison, we applied a single prompt format as it was used in related works due to limited resources; ideally, we would average across a range of prompt templates. Finally, we point out that today’s research (ours but also the related works) is far from being applicable in practice which, in turn, shows the critical need for this kind of research.
Ethics Statement
Data The dataset used in our work, OpinionQA is publicly available at Pew Research Center’s website. The dataset includes subjective opinions from humans and may contain offensive content to some people.
Data Collection We use Amazon Mechanical Turk to evaluate the quality of the opinions selected by our model and Mistral-7B. To ensure the quality of evaluation, we required that workers were located in English-speaking countries (e.g. US, UK, Canada, Australia, and New Zealand), and had an acceptance rate of at least 98% on 1,000 prior HITs. We paid $0.20 for the evaluation task. The annotators were compensated with an average hourly wage of $13, which is comparable to the US minimum wage. We did not collect any personal information from annotators.
Models The large language models we used for the experiments are trained on a large-scale web corpus and some of them utilize human feedback. This may also bring some bias when predicting user answers. With LLMs, users can select information that adheres to their system of beliefs and to amplify potentially biased and unethical views. Such an echo chamber (Del Vicario et al., 2016) can eventually cause harm by reinforcing undesirable or polarized a user’s views. Our model based on a graph neural network mitigates these biases by focusing on the entailment relationship between opinions.

heading: Acknowledgements
text: This work was funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chairs program, an NSERC discovery grant, a research gift from AI2, and DARPA Machine Common Sense.

heading: A More Details about Evaluation Settings
text: Dataset To test the model’s personalization and reasoning ability, we use the OpinionQA dataset and train and test the model under a questionanswering (QA) setup. OpinionQA dataset contains 15 topics ranging from guns, global attitudes, and political views, and each topic contains an average of 100 questions and 5K users. Due to limited resources, we follow previous works (Hwang et al., 2023; Do et al., 2023) that use sampled data, in which the data includes 100 users per topic and each user has their past opinions up to 16 and 30 opinions to evaluate the model’s personalization and reasoning capabilities. Then, we use 35 users per topic to test the model’s abilities, ensuring the same test set used in Hwang et al. (2023), and the rest are used as training. The final dataset results in a total of 525 users and 45K QA pairs. In our setting, we treat political ideology information as a part of user demographics.
Baselines We compare our model performance with BERT (Devlin et al., 2018), Mistral-7B (Jiang et al., 2023), text-davinci-003 (GPT-3), gpt-3.5-turbo (GPT-3.5), and ChOiRe (Do et al., 2023). BERT is a transformer-based language model, which can be finetuned for a wide range of tasks, including question answering and natural language inference. In our task, input to the BERT model is: [USER user id][DEMO]demographics[SEP][OPINION]topk opinions[SEP]question and the model is trained to predict the user’s answer for a given question. Mistral-7B (Jiang et al., 2023) is a large language model that improves generation quality and facilitates inference using grouped-query attention and sliding window attention. Mistral-7B performs on par with LLaMA2-13B and LLaMA-34B (Touvron et al., 2023), across diverse tasks, including reasoning. LLaMA1 and LLaMA2 are transformer-based language models that were trained on trillions of tokens from exclusively publicly available data. ChOiRe (Do et al., 2023) is an approach with a chain of opinion reasoning.
They propose a 4-step framework that filters out irrelevant information in demographics or user opinions to answer an input question.
Metric For accuracy evaluation, we simply calculate the accuracy of the predicted answer choice to the gold answer choice in the dataset.
Hyperparameters We use 5 implications for each opinion. The number of GAT layers was set to 3. When selecting top-k paths, we set K to 5. The learning rate is set to 0.00005, the number of epochs is set to 30, and the batch size is set to 1 due to a varying number of opinions for each user. We used GPU A40 for all our experiments and our model took 2-3 hours. Our models ran three times with different seed numbers and we report the average of them with their standard deviations.
Prompts used for LLM baselines Figure 5 shows the prompt used to predict a person’s opinion using LLMs.

heading: B Additional Results: Comparing Predictions
text: Table 7 and 8 show agreement rates in individual predictions among the same model variants (e.g Mistral-7B optop-8, Mistral-7B optop-8+demo)

heading: C An example of common errors
text: Figure 6 shows a common error when incorporating demographics.

heading: D Prompt for generating implications
text: To generate implications for opinions, we use the following prompt:
USER: For a question: <question> with the following answer choices: [<choice1>, <choice2>, <choice3>], a person chose <choice1> as the answer. What does this imply? Generate implications in up to 5 sentences. 1. <implication1> 2. <implication2> 3. <implication3> 4. <implication4> 5. <implication5> ASSISTANT:

heading: E Examples of irrelevant implications

heading: F Prediction Distribution on More Users
text: Figure 8 presents the distribution of how the model performs on 100 users. We observe a similar trend to the distributions with 35 users.

heading: G Amazon MTurk for human evaluation
text: For human evaluation, we instruct annotators as follows:
You will be given a survey question, a person’s answer choice for the question, and their past opinions. Evaluate whether the selected opinions are reasonable to address the person’s answer choice for a given question.
Next, we present Figure 9 to annotators. Annotators are asked to evaluate the quality of selected opinions with a short explanation of why. We conduct two rounds of evaluation (our model and Mistral-7B) to avoid annotators being biased by looking at the responses from another model variant.

heading: H Reference for 15 Subsets from Pew Research Survey used in OpinionQA
text: We use the OpinionQA dataset derived from 15 annual Pew American Trends Panel (ATP) survey subsets. We list the full datasets we used below.
• American Trends Panel Wave 26: Guns (Center, Washington, D.C. June 22, 2017)
• American Trends Panel Wave 27: Automation and driverless vehicles (Center, Washington, D.C. Oct 4, 2017)
• American Trends Panel Wave 29: Views on gender (Center, Washington, D.C. Dec 5, 2017,W)
• American Trends Panel Wave 32: Community types, Sexual harassment (Center, Washington, D.C. MAY 22, 2018)
• American Trends Panel Wave 34: Biomedical and food issues (Center, Washington, D.C. Nov 19, 2018,W,W)
• American Trends Panel Wave 36: Gender and leadership (Center, Washington, D.C. SEPTEMBER 20, 2018)
• American Trends Panel Wave 41: Views of America in 2050 (Center, Washington, D.C. Dec. 10 – Dec. 23, 2018)
• American Trends Panel Wave 42: Trust in science (Center, Washington, D.C. AUG 2, 2019,W)
• American Trends Panel Wave 43: Race in America (Center, Washington, D.C. MAY 8, 2019,W)
• American Trends Panel Wave 45: Misinformation (Center, Washington, D.C. DEC 12, 2019,W)
• American Trends Panel Wave 49: Privacy and surveillance (Center, Washington, D.C. NOV 15, 2019,W)
• American Trends Panel Wave 50: American families (Center, Washington, D.C. NOV 6, 2019,W)
• American Trends Panel Wave 54: Economic inequality (Center, Washington, D.C. JAN 9, 2020,W)
• American Trends Panel Wave 82: 2021 Global Attitudes Project U.S. survey (Center, Washington, D.C. NOV 1, 2021a,W,W,W,W,W,W,W,W,W,W)
• American Trends Panel Wave 92: Political typology (Center, July 8 – July 18, 2021)

heading: Future Work
text: Altogether, our work proposes a promising research direction to address this challenging problem and opens up interesting future research. Limitations
From a data perspective, our work showed that we need better methods to integrate demographic or other additionally given information (i. e. , beyond opinions), which is left as a challenging question for future research. We further note that our work, similar to the related works on the topic, focuses on the somewhat restricted survey scenario, where all users are captured in terms of one set of descriptions. If the latter varied (e. g. , by having freeform answers), our supervised learning problem would become considerably harder. Our analysis has also clearly demonstrated that the implicit knowledge added using an LLM is often sensible, and manual checks are critical. Moreover, our approach is somewhat complex in that we need to apply an LLM during training for obtaining the
derived knowledge; it is very efficient for inference though. For the LLM comparison, we applied a single prompt format as it was used in related works due to limited resources; ideally, we would average across a range of prompt templates. Finally, we point out that today’s research (ours but also the related works) is far from being applicable in practice which, in turn, shows the critical need for this kind of research