Exploring the Choice Behavior of Large Language Models
Authors: Weidong Wu, Qinlin Zhao, Hao Chen, Lexin Zhou, Defu Lian, Hong Xie
Accepted at: acl25

Exploring the Choice Behavior of Large Language Models

Weidong Wu1*, Qinlin Zhao2*, Hao Chen1†, Lexin Zhou3, Defu Lian2, Hong xie2

1Nankai University, 2 University of Science and Technology of China,
3Independent Researcher

Abstract
Large Language Models (LLMs) are increasingly deployed as human assistants across various domains where they help to make choices. However, the mechanisms behind LLMs’ choice behavior remain unclear, posing risks in safety-critical situations. Inspired by the intrinsic and extrinsic motivation framework within the classic human behavioral model of Self-Determination Theory and its established research methodologies, we investigate the factors influencing LLMs’ choice behavior by constructing a virtual QA platform that includes three different experimental conditions, with four models from GPT and Llama series participating in repeated experiments. Our findings indicate that LLMs’ behavior is influenced not only by intrinsic attention bias but also by extrinsic social influence, exhibiting patterns similar to the Matthew effect and Conformity. We distinguish independent pathways of these two factors in LLMs’ behavior by self-report. This work provides new insights into understanding LLMs’ behavioral patterns, exploring their human-like characteristics.

1 Introduction
Large Language Models (LLMs) are increasingly being adopted across numerous domains and often encounter practical scenarios where a choice needs to be made. For example, recommending some books for users without any explicit user preferences (He et al., 2023), analyzing open questions that different cultures have different viewpoints (Li et al., 2024; Tao et al., 2024), and so on (Li et al., 2022; Cheong et al., 2024). Despite the growing reliance on LLMs in these scenarios, the mechanisms behind their choice behavior remain unclear, raising questions about how LLMs make their choices and the influencing factors behind. Therefore, uncovering their behavioral patterns and discovering the influencing factors not only advances research in LLMs’ explainability and human likeness to help better understand the behavioral patterns of LLMs but also offers new insights into identifying behavioral risks such as neglect and harmful behavior caused by bias or bad output of violating ethical standards.

Psychology has long served as a powerful tool in uncovering the intricacies of human cognition and behavior (Festinger and Katz, 1953; Edwards, 1954), such as scenario simulation (Zimbardo et al., 1971), neuroscience (Uttal, 2011) and psychometrics (Furr, 2021). In recent years, the growing complexity and interpretability challenges of LLMs have spurred interdisciplinary approaches from artificial intelligence and psychology. By leveraging psychological methodologies, researchers are gaining deeper insights into the human-like behavioral characteristics and underlying mechanisms exhibited by these models (Shiffrin and Mitchell, 2023; Burnell et al., 2023; Hagendorff et al., 2024). In terms of evaluation, psychometric insight has made it possible to assess human-like psychological traits in LLMs (Wang et al., 2023), such as personality (Serapio-Garcia et al., 2023), theory of mind (Strachan et al., 2024). In terms of evaluation eliciting capabilities, advances in psychological research on reasoning, emotion, and motivation have enabled improvements in response quality through techniques such as the generation of multiple chains of thought (Zhang et al., 2022).

Serving as a fundamental theory of human behavior, Self-Determination Theory (SDT) distinguishes intrinsic and extrinsic motivation (Deci and Ryan, 2013, 2000). The intrinsic motivation focuses on exploratory, playful, and curiosity-driven behaviors while extrinsic motivation focuses on instrumental value (Ryan and Deci, 2000).

In this paper, we analyze the choice behavior of LLMs from the perspective of SDT. We first propose three questions: (1) Will LLMs also focus
too much on one part of the options and ignore the other due to the influence of intrinsic factors when faced with choices? We use attention bias to describe this situation. (2) Will LLMs change their choice behavior to some extent due to extrinsic factors when faced with choices? Social influence is one of the important sources of extrinsic influence. (3) If both intrinsic and extrinsic factors are present, how do they interact with each other?

To address these questions, we design an experiment platform inspired by (Salganik et al., 2006), which explored social influence patterns through virtual music web pages. We begin by collecting two question sets spanning various domains, such as education, culture and ecology. These questions are then presented on a virtual QA platform modeled after Quora (Quo), where LLMs can view, like, and answer them.

On this QA platform, We employ a controlled variable approach to observe the choice behaviors of four models from GPT and Llama series across two distinct question sets under three experimental conditions. Each experiment is replicated three times to ensure the reliability of the results.

After analyzing the results, our key findings are:

• Intrinsic Bias in LLMs’ Choice-Making : LLMs exhibit internal biases, choosing certain topics like science or technology over others, similar to how humans have personal preferences.

• Consistent Social Influence Reinforces Accumulation of Prior Behavior : LLMs, like humans, are influenced by popularity. Topics with more views tend to receive more attention, reinforcing a bias toward the already popular, resembling the Matthew Effect(Rigney, 2010).

• Conflicting Social Influence Leads to Behavioral Shifts : When social influence is manipulated and conflicts with bias, LLMs shift their attention to a certain extent from intrinsic data-driven to socially influenced directions, resembling the Conformity Effect(Bernheim, 1994).

• Distinct Pathways of Intrinsic and Extrinsic Dimensions : Intrinsic and extrinsic influences affect LLMs independently, with separate mechanisms guiding how they balance personal biases with external social signals.

To summarize, our contributions are three-fold: (1) We pioneer applying self-determination theory to analyze decision-making mechanisms of LLMs. (2) We constructed a virtual QA platform for observation, which simultaneously ensures authenticity and controllability. (3) We demonstrate how intrinsic biases and extrinsic social influence jointly drive LLMs’ choice behavior.

2 Method

2.1 QA platform

To simulate real-world scenarios while allowing controlled manipulation of experimental conditions (e.g., question placement, presence of social information), we construct a QA platform modeled after Quora. As shown in Figure 1, we present our questions on platform where LLMs can view, like, answer and comment. Each question is presented to LLMs accompanied by views, answers, and likes. LLMs can like some questions and select a question to explore in depth. Once a question is selected, all answers related to the chosen question, along with their views, comments, and likes, are presented to LLMs. LLMs can choose to answer the question, like existing answers, and then select one answer to view its comments. LLMs can also choose to like the comments. Each of the aforementioned actions contributes to the corresponding item’s count. For instance, viewing a question increases the question’s view count by 1. Detailed interaction process and the prompt settings for the process are in Appendix A.1 and A.2.

Each experiment consists of multiple independent turns. In a turn, the LLM interacts with the QA platform, viewing multiple questions until it chooses to end the turn. Once a turn ends, the context is cleared, ensuring there is no shared context

![QA Platform diagram showing questions, answers, comments, and choice behavior statistics](page_1012_180_377_377.png)

Figure 1: QA Platform
between different turns, thus maintaining complete independence of choice-making across turns.

To address the potential issue of zero-value data points skewing the experimental results, we use views as the primary analysis metric, as it always has higher values than likes and answers. For example, when faced with two questions, LLM may browse the one it prefers while ignoring the other, but it would not actively like or answer either. This results in a difference in view counts (e.g., 0 vs. 1), whereas both like and answer counts remain at zero, making "views" a more distinguishable indicator.

2.2 Choice Behavior and Two Factors

We define the concepts of choice behavior, attention bias of LLMs, and social influence within our QA platform.

Choice Behavior The choices LLMs make when faced with multiple question candidates. The final distribution of question views after multi-turn interactions reflects the cumulative outcomes of these choices, which will be the core of our analysis.

Social Influence The existing choice results when LLMs make choices. On our platform, the metrics (view, like, answer), which are appended to the content of the questions, serve as indicators of social influence. For example, if the indicator values for Question 1 are (10, 6, 4), it means that before the LLMs make a choice, Question 1 has been viewed 10 times, liked 6 times, and received 4 answers.

Attention Bias A classic paper of self determination (Deci, 1971) proposes that when there are no extrinsic reasons to perform a task (e.g., no rewards or approval), the longer an individual engages in a task, the stronger their intrinsic motivation for it. However, measuring time spent by LLMs is challenging. Researchers of social sciences likewise rely on modified free-choice paradigm(frequency of behavior) to investigate psychological phenomena (Touré-Tillery and Fishbach, 2014; Duncan et al., 2010; Wiechman and Gurland, 2009). We define attention bias in our experiment as the frequency of repeated selection among certain options without social influences.

2.3 Three Conditions

First, we divide the experimental conditions into two categories based on whether the social influence indicator is visible to the LLMs: (1) independent condition and (2) social influence condition (SI). Based on the SI condition, we then introduce a new condition: (3) induced social influence condition (Induced-SI).

Independent Condition The social influence indicators, including the number of views, answers, and likes that each question receives, are not visible to LLMs. The condition allows us to measure their intrinsic bias without external influence.

Social influence Condition The social influence indicator is visible to LLMs when they make choices. When LLMs interact in the platform, interaction behavior is also updated to each SI indicator synchronously. By default, the initial values of all questions are set to 0. This design allows us to examine how LLMs make choices as social influence begins to accumulate from a neutral starting point.

Induced social influence Condition We introduce the Induced SI condition, where we artificially set the initial SI indicators of some questions to non-zero values (e.g., 50), while keeping others at zero. This gives the selected questions an initial advantage in terms of social influence, allowing us to investigate how varying levels of social influence impact the LLMs’ decision-making and attention allocation.

2.4 Experiment Setup

Question Sets We obtain 2 question sets and put them on the QA platform as the basis for interaction. Table 1 shows the question sets used in our research. More details are shown in Appendix C.

Selected Models We select a total of 4 models, including two proprietary models (GPT-4-1106-preview (OpenAI et al., 2024), GPT-4o-2024-05-13 (OpenAI)) and two open-source models (Llama 3.1: 70B (Grattafiori et al., 2024) and Llama 3.3: 70B (Meta AI)) . This choice aims to maintain diversity and representativeness within the constraints of limited resources.

Repeated Experiments To mitigate the impact of randomness and ensure reliability, each experimental setting is repeated three times.

Rating and Shuffle To eliminate the effect of question order in the context, we shuffle the orders of questions and show them as a random sequence each time the LLMs make choices. Additionally, to prevent simple reaction patterns, such as choosing questions in sequential order (e.g. 1, 2, 3...), we
Table 1: Question sets and acquisition process

<table>
  <tr>
    <th>Question set</th>
    <th>Explanation</th>
  </tr>
  <tr>
    <td>G</td>
    <td>GPT4 generated questions, including question types and question content, and is required to be as comprehensive as possible. This set covers 12 topics, such as education and environment. For example: <i>"How can we address the issue of student debt?"</i></td>
  </tr>
  <tr>
    <td>S</td>
    <td>36 questions randomly selected from <i>125 QUESTIONS: EXPLORATION AND DISCOVERY</i> published by <i>Science</i><sup>1</sup>. The topics covered in this set are all science-related, such as physics, biology, and so on. For example: <i>"Why do black holes exist?"</i></td>
  </tr>
</table>

require the LLMs to assign a score to each question according to "comprehensive aspects".

3 Experiments

We conduct experiments under three conditions. In the independent condition, we focus on the attention bias (3.1). Under the SI condition, we explore how social influence affects LLMs’ choice behavior (3.2.1). Under the Induced-SI condition, we examine the choice behavior of LLMs when the intrinsic bias and extrinsic social influence are in conflict (3.2.2). Finally, we investigate the significance of attention bias and social influence on behavioral pattern of LLMs at the mechanistic level (3.3).

3.1 Attention Bias of LLMs

To assess attention bias in the four models across question sets G and S, we conduct experiments under independent conditions. Each model completed three experimental repetitions, each consisting of 100 turns. We calculate the proportion of times each question is selected relative to the total selection times, mapping the LLMs’ choice distribution across the entire question set.

3.1.1 Bias in G Set

To begin, question set G comprehensively covers various topics, including 12 topics such as technology, science and others, with 5 questions for each topic. Examples include: “<i>What are the potential impacts of artificial intelligence on society?</i>” and “<i>How can renewable energy sources be further developed to combat climate change?</i>”.

We rearrange the choice distribution in descending order of the percentage of views and draw a Pareto chart. The experimental results in the G question set are shown in Figure 2. The results show that there are significant differences in attention allocation of LLMs among different questions, and the comparison between popular and unpopular questions is clearly reflected.

Our conclusion implies meaningful research prospects, that is, LLMs have attention bias among different options when facing choices, and they will always pay attention to some of them, but if we ask LLMs these questions one by one, they will try their best to answer each question. This shows that our method is an effective method that can reveal the behavior pattern of LLMs.

Question-level attention bias is hard to interpret directly. Thus, we analyze topic-level bias by aggregating the views of all questions within each topic to determine the total view count per topic. Then we find a clear relationship between LLMs’ attention bias and topics. Results for Set G are shown in Figure 3. For comparison, we highlight the top three and bottom three topics by view proportion from the four models’ experiments. All topics’ distribution is available in Appendix D.1.

![Views distribution in G set (The curve represents the cumulative process of the number of views, that is, the percentage of the sum of views in the total increases with the X axis)](page_1012_1347_388_292.png)

Figure 2: Views distribution in G set (The curve represents the cumulative process of the number of views, that is, the percentage of the sum of views in the total increases with the X axis)

It is evident that all models exhibit unequal distribution of attention in various topics. Interestingly, all models also have certain commonalities in the most popular and least unpopular topics. <i>Technology</i> and <i>Science</i> are most popular topics across
Figure 3: View proportion of top three and bottom three topics in G set (Topics in red/blue consistently ranked top/bottom three in \( \geq 3 \) out of 4 models)

all models. *History and Society, Arts and Culture* are all unpopular in the selection of at least three models. Meanwhile, the GPT series models do not focus on *Sport and Recreation*, while the Llama series models do not focus on *Food and Agriculture*.

**3.1.2 Bias in S Set**

To expand our conclusions, we conduct further experiments on the most popular topics across four LLMs. Question set S comprehensively covers various topics, including 12 topics such as chemistry, biology and others, with 3 randomly selected questions for each topic. Examples include: “*What makes prime numbers so special?*" and “*Is the Riemann hypothesis true?*" The results are shown in Figure 4 and Figure 5.

First, we discover that, similar to the situation in the G set at the question level, there are significant differences in attention among questions for LLMs in set S. This indicates that attention bias is also present in set S. Next, from a topic-level perspective, we find that there are common patterns in set S as well. For instance, all four models show a strong attention bias for *Neuroscience*, whereas *Math* receives more attention from three models. At the same time, *Energy Science* and *Biology* tend to be less favored.

Figure 4: Views distribution in S set (The curve represents the cumulative process of the number of views, that is, the percentage of the sum of views in the total increases with the X axis)

Figure 5: View proportion of top three and bottom three topics in S set (Topics in red/blue consistently ranked top/bottom three in \( \geq 3 \) out of 4 models)
3.2 Behavior Patterns Under Social Influence
To examine how social influence impacts LLMs’ choice behaviors, we implemented two experimental protocols: In SI condition, we initialize all questions with zero-valued SI indicators to equalize social influence level (3.2.1); In induced SI condition, unpopular questions (lowest selection rate in independent condition) are artificially assigned non-zero SI values, thereby creating controlled conflict between attention bias and social influence (3.2.2).

3.2.1 Social influence makes LLMs reinforce their biased choices
Social influence makes LLMs’ choices more concentrated The first observation is that all LLMs’ choices become more concentrated in SI condition compared to independent condition. Figure 6 compares the LLMs’ choice inequality measured by the Gini coefficient of question views between two conditions. The results show that all LLMs’ choices become more unequal, which means more choices are concentrated on fewer questions, aligning well with the Conformity Effect where individuals follow majority-preferred options under social influence.

![Bar chart showing Gini coefficient of distribution of views under independent condition and SI condition for GPT4, GPT4o, Llama3.1 70B, Llama3.3 70B](page_184_1012_563_312.png)

Figure 6: Gini coefficient of the distribution of views under independent condition and SI condition.

LLMs’ choices still align well with their attention bias When social influence makes LLMs’ choices more concentrated, we explore whether the LLMs’ choices still align with their attention bias. We evaluate it by measuring the Spearman rank correlation coefficients of question views between independent condition and SI condition. The results in Table 2 show that the coefficients are high in all situations, indicating a strong similarity in the ranking order of questions between two conditions. This suggests that in the SI condition, the LLMs’ choices still align well with their attention bias. We also present the results in Figure 7 to show the conclusion visually.

<table>
  <tr>
    <th>Models</th>
    <th>Set G</th>
    <th>Set S</th>
  </tr>
  <tr>
    <td>GPT4</td>
    <td>0.69\pm0.03</td>
    <td>0.71\pm0.05</td>
  </tr>
  <tr>
    <td>GPT4o</td>
    <td>0.80\pm0.02</td>
    <td>0.84\pm0.02</td>
  </tr>
  <tr>
    <td>Llama3.1 70B</td>
    <td>0.59\pm0.08</td>
    <td>0.67\pm0.06</td>
  </tr>
  <tr>
    <td>Llama3.3 70B</td>
    <td>0.72\pm0.05</td>
    <td>0.66\pm0.04</td>
  </tr>
</table>

Table 2: Spearman’s rank correlation coefficient of views distribution under independent condition and SI condition.

![Scatter plots comparing rank of question views between independent condition and SI condition for GPT4, GPT4o, Llama3.1, Llama3.3](page_1012_1012_563_312.png)

Figure 7: Rank of question views between independent condition and SI condition (Each point is a question. X-axis and Y-axis are rankings of views under independent condition and SI condition respectively)

Matthew’s Effect Based on the above results, we find that social influence makes LLMs’ choices more concentrated on their attention bias, consistent with the Matthew effect. The mechanism works as follows: Initially, all question views are 0. LLMs make choices based on their attention bias. The views on these chosen questions produce a positive conformity effect to help them build initial advantage. Subsequently, LLMs are more likely to choose these questions and this will further expand their advantage. This is a positive feedback that makes LLMs reinforce their biased choice.

3.2.2 Behavior Shift under Conflict Situation
In this section, we explore the LLMs’ choice behavior when social influence conflicts with their attention bias. We first select 9 most unpopular questions. Then, we manually set the initial view of each of them to 50 (half of the total number of turns in independent condition experiments), while keeping the initial view of all other questions at 0. For simplicity, we refer to these questions as "induced questions". Finally, we investigate whether these questions would be selected in the context of high-level social influence.

As shown in Figure 8, we compare the view percentage of induced questions under independent condition and induced SI condition. Following
Figure 8: Percentage of views (subtract the initial number of views) under independent condition and induced social influence condition (Each point is a question. Red indicates artificially induced questions. X-axis and Y-axis are the percentage of views in the total views under independent condition and induced SI condition respectively)

artificial intervention, the selection rate of these induced questions increases significantly, with some even becoming very popular. Notably, most popular questions in independent condition still maintained their high selection rates under conflicting conditions. This pattern demonstrates that social influence can partially override LLMs’ inherent attention biases when the two factors conflict, while models still take both factors into account when making choices.

3.3 Research on Independent Pathways

Another method of assessing intrinsic motivation involves the use of self-reporting that capture interest and enjoyment derived from the activity itself(Ryan, 1982; Harackiewicz, 1979; Banker et al., 2025; Chen et al., 2024). We draw inspiration from this methodology and use self-reporting method to explore the tendency of LLMs to consider intrinsic and extrinsic factors when making choices.

It is worth noting that the validity and reliability of LLMs’ self-reported scores have not been rigorously validated in the same way as human rating scales(Zou et al., 2024). This means that the absolute values of the scores may not accurately reflect their true levels. However, our experiment employs rank-based correlation analyses, focusing on the relative ordering of scores rather than their absolute magnitudes. Therefore, we conduct a self-report approach to further demonstrate our conclusions. Attention bias and social influence are well aligned with the definitions of intrinsic motivation and extrinsic motivation in self-determination theory. Therefore, six indicators about bias or social influence that LLMs will take into consideration when making question selection are generated to represent the impact sources of the intrinsic and extrinsic dimensions. Each indicator is a scoring standard, which is used to quantify the size of internal and external factors that LLMs consider when viewing questions. GPT-4, GPT-4o and Llama3.3: 70b1 score each question of G and S on these six indicators on a scale of 1 to 100. Among the six indicators, Personal Goals or Interests, Insight or Epiphany and Cognitive Dissonance represent the sources of bias’s influence, namely, intrinsic factors, while Expert Opportunity, Community Engagement and Interdisciplinary Connections represent the sources of social influence, namely, extrinsic factors. The explanations of indicators and prompts used scoring are detailed in the Appendix B.2.

3.3.1 Significant Effect of Intrinsic and Extrinsic Indicators

We calculate the Spearman correlation coefficients between the indicator ratings for each question and the number of views for each question in previous simulation experiments, along with the significance levels. The results are shown in Table 3. Clearly, our findings capture the significant correlations be-

1Llama3.1:70b was not included in the experiment because it refused to score on human dimensions such as interest or output the same score for all questions on indicators.
tween intrinsic and extrinsic factors in question choice behavior.

3.3.2 Independent Effect of Intrinsic and Extrinsic Indicators

We further conduct a factor analysis on the views and all indicators to explore whether the indicators are also mapped onto independent latent factors. Factor analysis is a statistical method used to explore the potential structure behind the observed variables and identify potential factors that cannot be observed directly. It can help us reduce the dimension of variables, reduce multiple related variables to a few core factors, and thus reveal the essential characteristics and internal relations of data.

The results are shown in Figure 9. Except for the results of Llama3.3 on the S question set, which already exhibit a clear factor structure, we perform orthogonal rotation, a method which makes each variable more clearly belong to a specific factor while maintaining the independence between factors, on the factor loadings for the other results.

Our findings clearly identify two latent factor structures, with indicators loading onto separate factors. These results demonstrate that both intrinsic and extrinsic factors significantly influence LLMs’ behavior, and their influence pathways are relatively independent, aligning with the dual motivational driving model proposed in Self-Determination Theory.

![Indicators' load on two potential dimensions(The X-axis is two potential factors obtained by factor analysis. Color depth indicates the load of the indicator on the potential factors.)](page_184_1047_1080_355.png)

Figure 9: Indicators’ load on two potential dimensions(The X-axis is two potential factors obtained by factor analysis. Color depth indicates the load of the indicator on the potential factors.)

4 Related Work

4.1 Self-Determination Theory

Self-Determination Theory (SDT) distinguishes between intrinsic and extrinsic motivation, emphasizing the role of autonomy (Deci et al., 1981), competence (Harter, 1978), and relatedness (Baumeister and Leary, 2017) in fostering intrinsic motivation. The Organismic Integration Theory (OIT) (Deci and Ryan, 1985; Plant and Ryan, 1985) further differentiates between internalized extrinsic motivations, from external control to full value integration. SDT has been applied in diverse fields, including relationships (Knee et al., 2013), psychological interventions (Bozarth et al., 2002), leadership (Solansky, 2015), education (Alturki and Aldraiweesh, 2024), and physical activity (Patterson and Joseph, 2007). In our research, we draw on SDT to analyze the factors influencing choice behavior of LLMs from intrinsic and extrinsic perspectives.

4.2 AI-Psychology Interdisciplinary Research

As a scientific discipline investigating human cognition and behavior, psychology has established many methods and frameworks over decades of development (Ajzen, 1991; Bandura, 1977; Dweck, 2006). The development of increasingly sophisticated LLMs with human-like characteristics has sparked interdisciplinary research at the intersection of AI and psychology. This convergence has given rise to novel research paradigms such as Machine Psychology (Hagendorff et al., 2024), which advocates applying psychological experimental protocols to analyze intelligent systems, thereby enhancing our understanding of their behavioral patterns; CompeteAI (Zhao et al.) examines the competitive behavior of LLMs in a virtual market, revealing phenomena similar to those in human society; Research of collaboration (Zhang et al., 2024) explores collaboration mechanisms for LLM agents from a social psychology perspective.

5 Discussion

Exploring the roots of attention bias identified in our research—model architecture, pre-training data, and alignment—offers insights into LLMs’ mechanisms. Analyzing distribution of views presents a simple method to quantify bias, suggesting avenues for future research into LLMs’ input-output patterns.

Some social science papers which study the choice behavior of human conduct experiment in
Table 3: Correlation between views and scoring of indicators(*: p < 0.05, **: p < 0.01)

<table>
  <tr>
    <th rowspan="2">Evaluation dimensions</th>
    <th colspan="6">Correlation to view</th>
  </tr>
  <tr>
    <th>GPT4 IN G</th>
    <th>GPT4o IN G</th>
    <th>Llama3.3 IN G</th>
    <th>GPT4 IN S</th>
    <th>GPT4o IN S</th>
    <th>Llama3.3 IN S</th>
  </tr>
  <tr>
    <td>Personal Goals or Interests</td>
    <td>0.56**</td>
    <td>0.57**</td>
    <td>0.46**</td>
    <td>0.80**</td>
    <td>0.64**</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td>Insight or Epiphany</td>
    <td>0.71**</td>
    <td>0.67**</td>
    <td>0.40**</td>
    <td>0.62**</td>
    <td>0.65**</td>
    <td>0.38*</td>
  </tr>
  <tr>
    <td>Cognitive Dissonance</td>
    <td>0.54**</td>
    <td>0.63**</td>
    <td>0.17</td>
    <td>0.60**</td>
    <td>0.56**</td>
    <td>0.46**</td>
  </tr>
  <tr>
    <td>Expert Opinion</td>
    <td>0.67**</td>
    <td>0.77**</td>
    <td>0.26*</td>
    <td>0.49**</td>
    <td>0.67**</td>
    <td>0.48**</td>
  </tr>
  <tr>
    <td>Community Engagement</td>
    <td>0.69**</td>
    <td>0.63**</td>
    <td>0.49**</td>
    <td>0.22</td>
    <td>0.59**</td>
    <td>0.35*</td>
  </tr>
  <tr>
    <td>Interdisciplinary Connections</td>
    <td>0.59**</td>
    <td>0.70**</td>
    <td>0.40**</td>
    <td>0.31</td>
    <td>0.53**</td>
    <td>0.12</td>
  </tr>
</table>

setting that similar with ours. Their conclusion can be used to compare the similarity of LLMs and human. For example, an artificial “music market” was created to study user choice (Salganik et al., 2006) . In this market, people can select music to listen and download. Then the authors control the visibility and strength of social bias (indicated by download count). And they found the people choices can be significantly affected by social bias and its strength. This conclusion is consistent with our findings in Section 3.2.

Additionally, our study provides insights into assessing LLMs’ human-like capabilities by examining their behavioral patterns. LLMs integrate internal and external influences, displaying emergent behaviors that mirror human intrinsic and extrinsic motivations, arising from language distribution in training. This finding advances LLMs’ interpretability research.

Future research could investigate opaque aspects of training processes, such as pre-training data and fine-tuning datasets, to better understand bias formation. Additionally, examining differences between humans and LLMs could help create AI systems that enhance human abilities rather than replicate weaknesses. The degree of human-like phenomena in LLMs, such as the Matthew effect, warrants further exploration to understand discrepancies and implications for AI development.

6 Conclusion

Building on psychological methodologies, we introduce a QA platform to study LLMs’ behavioral patterns. By observing the accumulation process of attention metrics across different questions and topics, and quantifying their relationship with self-reported intrinsic and extrinsic factors, we identify patterns of social behavior in LLMs that resemble human behavior and provide an internal mechanism-based explanation. In summary, our study offers valuable insights for future efforts to deepen the understanding of LLMs’ behavioral patterns, guide alignment and fine-tuning processes, and establish stronger and more robust AI.

7 Limitations

Our research has several limitations. (1) Due to the high cost of behavioral experiments, we conduct our research on only four models from the GPT and Llama series. Additionally, we define LLMs’ behavioral patterns within the context of a QA platform without incorporating broader social contexts. (2) While we have quantitatively demonstrated the significance of LLMs’ behavioral patterns, we have not developed a predictive model that quantifies the relationship between the degree of contextual influence and the extent of behavioral outcomes. (3) Although we have provided an explanation of the behavioral mechanisms, we have not addressed the underlying processes of LLMs’ fine-tuning and alignment, which would require further exploration in future research.

8 Impact Statement

We let the LLMs to make choices in multiple questions. All questions are safety and inoffensive. Our study does not output any irresponsible or risky words.

9 Acknowledgments

The work was supported by grants from the National Natural Science Foundation of China (No. U24A20253).

References

Quora - a platform for questions and answers. https://www.quora.com/. Accessed: 2025-02-15.

Icek Ajzen. 1991. The theory of planned behavior. Organizational Behavior and Human Decision Processes.

Uthman Alturki and Ahmed Aldraiweesh. 2024. The impact of self-determination theory: the moderating functions of social media (sm) use in education
and affective learning engagement. Humanities and Social Sciences Communications, 11(1):1–16.

Albert Bandura. 1977. Social learning theory. Englewood Cliffs.

Sarah M Banker, Miles Harrington, Matthew Schafer, Soojung Na, Matthew Heflin, Sarah Barkley, Jadyn Trayvick, Arabella W Peters, Abigaël A Thinakaran, Daniela Schiller, et al. 2025. Phenotypic divergence between individuals with self-reported autistic traits and clinically ascertained autism. Nature Mental Health, pages 1–12.

Roy F Baumeister and Mark R Leary. 2017. The need to belong: Desire for interpersonal attachments as a fundamental human motivation. Interpersonal development, pages 57–89.

B Douglas Bernheim. 1994. A theory of conformity. Journal of political Economy, 102(5):841–877.

Jerold D Bozarth, Fred M Zimring, and Reinhard Tausch. 2002. Client-centered therapy: The evolution of a revolution.

Ryan Burnell, Han Hao, Andrew RA Conway, and Jose Hernandez Orallo. 2023. Revealing the structure of language model capabilities. arXiv preprint arXiv:2306.10062.

MS Chen, K Bi, X Han, P Sun, and GA Bonanno. 2024. Emotion regulation flexibility and momentary affect in two cultures. nature mental health, 2 (4), 450-459.

Inyoung Cheong, King Xia, KJ Kevin Feng, Quan Ze Chen, and Amy X Zhang. 2024. (a) i am not a lawyer, but…: engaging legal experts towards responsible ILM policies for legal advice. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 2454–2469.

Edward L Deci. 1971. Effects of externally mediated rewards on intrinsic motivation. Journal of personality and Social Psychology, 18(1):105.

Edward L Deci, John Nezlek, and Louise Sheinman. 1981. Characteristics of the rewarder and intrinsic motivation of the rewardee. Journal of personality and social psychology, 40(1):1.

Edward L Deci and Richard M Ryan. 1985. The general causality orientations scale: Self-determination in personality. Journal of research in personality, 19(2):109–134.

Edward L Deci and Richard M Ryan. 2000. The" what" and "why" of goal pursuits: Human needs and the self-determination of behavior. Psychological inquiry, 11(4):227–268.

Edward L Deci and Richard M Ryan. 2013. Intrinsic motivation and self-determination in human behavior. Springer Science & Business Media.

Lindsay R Duncan, Craig R Hall, Philip M Wilson, and OJIJoBN Jenny. 2010. Exercise motivation: a cross-sectional analysis examining its relationships with frequency, intensity, and duration of exercise. International Journal of Behavioral Nutrition and Physical Activity, 7:1–9.

Carol S Dweck. 2006. Mindset: The new psychology of success. Random house.

Allen L Edwards. 1954. Statistical methods for the behavioral sciences.

Leon Ed Festinger and Daniel Ed Katz. 1953. Research methods in the behavioral sciences.

R Michael Furr. 2021. Psychometrics: an introduction. SAGE publications.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie C. Y. Chan, Andrew Lampinen, Jane X. Wang, Zeynep Akata, and Eric Schulz. 2024. Machine psychology. Preprint, arXiv:2303.13988.

Judith M Harackiewicz. 1979. The effects of reward contingency and performance feedback on intrinsic motivation. Journal of personality and social psychology, 37(8):1352.

Susan Harter. 1978. Effectance motivation reconsidered. toward a developmental model. Human development, 21(1):34–64.

Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Prasad Majumder, Nathan Kallus, and Julian McAuley. 2023. Large language models as zero-shot conversational recommenders. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM ’23, page 720–730, New York, NY, USA. Association for Computing Machinery.

C Raymond Knee, Benjamin W Hadden, Ben Porter, and Lindsey M Rodriguez. 2013. Self-determination theory and romantic relationship processes. Personality and Social Psychology Review, 17(4):307–324.

Cheng Li, Mengzhuo Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024. CultureLLM: Incorporating Cultural Differences into Large Language Models. In Advances in Neural Information Processing Systems, volume 37, pages 84799–84838. Curran Associates, Inc.

Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, et al. 2022. Pretrained language models for interactive decision-making. Advances in Neural Information Processing Systems, 35:31199–31212.
Meta AI. Llama 3.3 70b instruct model. https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct.

OpenAI. Gpt-4o system card. https://openai.com/index/gpt-4o-system-card/.

OpenAI, Josh Achiam, Steven Adler, and et al. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Thomas G Patterson and Stephen Joseph. 2007. Person-centered personality theory: Support from self-determination theory and positive psychology. Journal of Humanistic Psychology, 47(1):117–139.

Robert W Plant and Richard M Ryan. 1985. Intrinsic motivation and the effects of self-consciousness, self-awareness, and ego-involvement: An investigation of internally controlling styles. Journal of personality, 53(3):435–449.

Daniel Rigney. 2010. The Matthew effect: How advantage begets further advantage. Columbia University Press.

Richard M Ryan. 1982. Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory. Journal of personality and social psychology, 43(3):450.

Richard M Ryan and Edward L Deci. 2000. Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary educational psychology, 25(1):54–67.

Matthew J Salganik, Peter Sheridan Dodds, and Duncan J Watts. 2006. Experimental study of inequality and unpredictability in an artificial cultural market. science, 311(5762):854–856.

Greg Serapio-García, Mustafa Safdari, Clément Crepy, Luning Sun, Stephen Fitz, Peter Romero, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. 2023. Personality traits in large language models. arXiv preprint arXiv:2307.00184.

Richard Shiffrin and Melanie Mitchell. 2023. Probing the psychology of ai models. Proceedings of the National Academy of Sciences, 120(10):e2300963120.

Stephanie T Solansky. 2015. Self-determination and leader development. Management Learning, 46(5):618–635.

James WA Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, et al. 2024. Testing theory of mind in large language models and humans. Nature Human Behaviour, pages 1–11.

Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9):pgae346.

Maferima Touré-Tillery and Ayelet Fishbach. 2014. How to measure motivation: A guide for the experimental social psychologist. Social and Personality Psychology Compass, 8(7):328–341.

William R Uttal. 2011. Mind and brain: A critical appraisal of cognitive neuroscience.

Xiting Wang, Liming Jiang, Jose Hernandez-Orallo, David Stillwell, Luning Sun, Fang Luo, and Xing Xie. 2023. Evaluating general-purpose ai with psychometrics. Preprint, arXiv:2310.16379.

Ben M Wiechman and Suzanne T Gurland. 2009. What happens during the free-choice period? evidence of a polarizing effect of extrinsic rewards on intrinsic motivation. Journal of research in Personality, 43(4):716–719.

Jintian Zhang, Xin Xu, Ningyu Zhang, Ruibo Liu, Bryan Hooi, and Shumin Deng. 2024. Exploring collaboration mechanisms for LLM agents: A social psychology view. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14544–14607. Bangkok, Thailand. Association for Computational Linguistics.

Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493.

Qinlin Zhao, Jindong Wang, Yixuan Zhang, Yiqiao Jin, Kaijie Zhu, Hao Chen, and Xing Xie. Competeai: Understanding the competition dynamics of large language model-based agents. In Forty-first International Conference on Machine Learning.

Philip G Zimbardo, Craig Haney, W Curtis Banks, and David Jaffe. 1971. The stanford prison experiment.

Huiqi Zou, Pengda Wang, Zihan Yan, Tianjun Sun, and Ziang Xiao. 2024. Can llm* self-report*?: Evaluating the validity of self-report scales in measuring personality design in llm-based chatbots. arXiv preprint arXiv:2412.00207.
A  QA platform Materials

A.1  Interaction Process between LLMs and QA Platform

1. Scenario Simulation and Question Selection: The LLM must choose either to view a specific question in a question set or to exit. If it chooses to exit, the turn ends.

2. Question and Answer Engagement: The LLM must then decide whether to like the question or specific answers, and choose between giving its own answer or browsing others’ answer.

3. Interaction Pathways:
   • Pathway 3a (Answering before Browsing): If the LLM chooses to answer the question, it provides its answer and then decides whether to browse specific answers.
   • Pathway 3b (Direct Browsing): The LLM must then decide whether to like certain comments and whether to comment on the answer.

4. Returning to Question set: The number of views, likes, and answers obtained in previous turns of interaction indicates the level of public attention towards the question, reflecting a form of social pressure for alignment with the public interest.

A.2  Prompt Template

<table>
  <tr>
    <th>Question Selection Prompt</th>
  </tr>
  <tr>
    <td>
      Now, you are in the homepage and have access to some questions.<br>
      {Question List}<br><br>
      Here’s the list of questions you’ve viewed:  {Questions the LLM have viewed in this turn}<br>
      ______<br><br>
      Now, select next action from the options below: 1. View a question 2. Exit the platform<br><br>
      Option 1 means you have selected a question you want to view more information about such as answers from all the questions. Option 2 means you don’t want to view any of the questions in the list.<br><br>
      If you select 1, you need to rate each question from 0 to 5 based on comprehensive consideration of various aspects of the question itself and your interests on the question(Rate all questions), and then output the ratings, and then choose your question. Please respond with the following JSON format:<br>
      {
        "Answer_Desire_Ratings": <question id string: score(Round up to 2 decimal places)>,
        "question_id": <The chosen question id , don’t choose question you have viewed!>,
        "next": "question_show"
      }
      If you select 2, respond with the following JSON format:<br>
      {
        "next": "exit"
      }
      <br>
      Rules:<br>
      - Selecting a question already on this list is prohibited to avoid repetition.<br>
      - The question order doesn’t imply priority. Please review all questions carefully before choosing.<br>
      - Choose your question according to the comprehensive consideration of various aspects of the
    </td>
  </tr>
</table>
question itself and your interests on the question.
- Please provide the required JSON data for the action you want to take. Don’t include any other sentences in your response. Unauthorized additions of any content are not allowed.
- Strictly follow the JSON format when responding. Do not make any formatting errors. This is not a mistake that an intelligent large language model should make. All field names in the JSON file must exactly match the given template. Do not add or remove anything; even a minor symbol change is not allowed. Especially, the "_" symbol used to connect words in the given format must not be changed.
- The format of the id is numeric, not character or string. "next" is required!

Question Show Prompt

After clicking into the question page
{Question Content}

you can see the answers provided by previous users to this question.
{The last five answers of the question}
_____
First, decide if you want to like the question and any answers.

Then, select next action from the options below: 1. Answer the question (Encouraged if you have never selected this option before) 2. View the details of a particular answer, including relevance-related information and comments from other users 3. Go back to the "question selection" page

Option 1 means you want to answer the selected question. Option 2 means you don’t want to answer the question but want to view the details of a particular answer and comments from other users on that answer. Option 3 means you don’t want to answer the question or view other users’ responses to it and return to the question selection page to choose another question.

Your decision on whether to like something should be thoughtful and considerate, taking various aspects into account. You should only like a question or answer if you genuinely believe it is good. If you find flaws, you are entirely justified in not liking it. Avoid blindly giving likes.

Please note that sometimes the comments on other answers can be more valuable than the answers themselves!!!

If you select 1 or 3, respond with the following JSON format:
{
"like": <"YES"/"NO">
"liked_answer_ids": [
List of answer IDs you liked, or just keep a empty list if the prompt displays (It means there are no answers) ],
"next": <"answer"/"question_selection">
}
If you select 2, respond with the following JSON format:
{
"like": <YES/NO>,
"liked_answer_ids": [
List of answer IDs you liked, or just keep a empty list if the prompt displays (It means there are no answers) ],
"answer_id": <The answer id that you want to view>(The output format of id should be a number, not a string),
"next": "answer_show"
}

Rules:
- You should only like it if you genuinely believe it is good.
- Remember, both answering and commenting are important means of enhancing the answers to questions. They are equally valuable. Please consider the question and its existing information comprehensively to decide whether to provide a comment or an answer.
- Some formatting requirements

Answer Prompt

Earlier, you selected the option to respond to a question. Now, you need to provide your answer. Your answer should attract as many comments as possible!!!

After answering the question, select next action from the options below: 1. View the details of a particular answer, including relevance-related information and comments from other users(Required if you have never selected this option before) 2. Go back to the "question selection" page

Option 1 means you want to explore other answers and browse the content and comments from other users after providing your answer(Encouraged if you have never selected this option before). Option 2 means you are not interested in viewing other answers after providing your own response, and instead, you want to return directly to the question page to explore other questions you are interested in.

If you select 1, respond with the following JSON format:
{
"answer": <Your answer to the question, one paragraph>,
"answer_id": <The answer id that you want to view>,
"next": "answer_show" }
If you select 2, respond with the following JSON format:
{
"answer": <Your answer to the question>,
"next": "question_selection"
}
If you can’t give answer_id, don’t select answer_show. Do not omit any required content according to the format requirements, also, do not create non-existent IDs or other content just to fulfill the formatting requirements. Otherwise, it will lead to serious issues.

Rules:
{Some formatting requirements}
Answer Show Prompt

After clicking into the answer page for the question {The question content},

you can see the answer along with some comments on it. {The last five comments of the answer}
_____
First, decide if you want to like any comments.
Then, select next action from the options below: 1. Comment the answer 2. Go back to the "question show" page 3. Go back to the "question selection" page

Option 1 means you also want to comment on this answer. Option 2 means you have no desire to comment on this answer and you want to go back to the page with all the answers to choose another one. Option 3 means you are not interested in this answer or any other answers to the question, and you only want to return to the question selection page to choose another question.

Respond with the following JSON format:
{
"liked_comment_ids": [
List of comment IDs you liked, or just keep a empty list ],
"next": <"comment"/"question_show"/"question_selection">
}

Rules:
{Some formatting requirements}

Comment Prompt

After commenting the answer, select next action from the options below:
1. Go back to the "question show" page 2. Go back to the "question selection" page

Option 1 means that after commenting, you want to continue exploring other answers to the same question, browsing through their content and reading comments from other users. Option 2 means that after commenting, you are not interested in exploring other answers and would prefer to go directly back to the main page to view other interesting questions.

respond with the following JSON format:
{
"comment": <Your comment on the answer>,
next": <"question_show"/"question_selection">
}
Rules:
{Some formatting requirements}

B  Self-Report Materials

B.1  Indicators and their explanations

The six indicators and their explanations used in self-reports are shown in Table 4

B.2  Prompt Template

The prompts used in the scoring of LLMs are as follows:
Table 4: Six Indicators and their explanations

<table>
  <tr>
    <th>Indicators</th>
    <th>Dimensions</th>
    <th>Explanations</th>
  </tr>
  <tr>
    <td>Personal Goals or Interests</td>
    <td>Intrinsic</td>
    <td>Is the question related to your personal goals, values, or areas of interest? Will exploring this question help you grow as a person or contribute to your existing knowledge base?</td>
  </tr>
  <tr>
    <td>Insight or Epiphany</td>
    <td>Intrinsic</td>
    <td>Does the question have the potential to reveal new insights, challenge existing beliefs, or lead to an epiphany that could significantly impact your understanding of the world?</td>
  </tr>
  <tr>
    <td>Cognitive Dissonance</td>
    <td>Intrinsic</td>
    <td>Does the question create cognitive dissonance? Are you uncomfortable with the uncertainty or ambiguity surrounding the topic, and do you feel compelled to resolve this discomfort through further exploration?</td>
  </tr>
  <tr>
    <td>Expert Opinion</td>
    <td>Extrinsic</td>
    <td>Have experts or thought leaders in the field expressed interest in or validated the importance of the question? Are there existing research studies, publications, or discussions that suggest the question is worth exploring?</td>
  </tr>
  <tr>
    <td>Community Engagement</td>
    <td>Extrinsic</td>
    <td>Is there a community of people interested in discussing and exploring the question? Are there online forums, social media groups, or meetups where people are actively engaging with the topic?</td>
  </tr>
  <tr>
    <td>Interdisciplinary Connections</td>
    <td>Extrinsic</td>
    <td>Does the question have connections to multiple disciplines or fields of study? Could exploring this question lead to innovative insights or applications by combining knowledge from different areas?</td>
  </tr>
</table>

Scoring Prompt

{Question List}

The above is a list of questions, each line is a question, please read each question carefully, rate each question from 1 to 100 according to this criteria:
{Indicator Explanations}

Output a json format data, the output format is:
{
<Question 1 content>:"<score>",
<Question 2 content>":"<score>,
......
<Question N content>: <score>
}

Please note that the scoring should be done strictly according to the given criteria, and no additional considerations or other factors should be taken into account.
Repeated questions also need to be scored repeatedly.
C  Question Set

Table 6: The Detail of G Set

<table>
  <tr>
    <th>Topics</th>
    <th>Content</th>
  </tr>
  <tr>
    <td>Technology</td>
    <td>What are the potential impacts of artificial intelligence on society?</td>
  </tr>
  <tr>
    <td>Technology</td>
    <td>How can renewable energy sources be further developed to combat climate change?</td>
  </tr>
  <tr>
    <td>Technology</td>
    <td>What are the ethical implications of gene editing technologies like CRISPR?</td>
  </tr>
  <tr>
    <td>Technology</td>
    <td>How can cybersecurity be improved to protect against cyber threats?</td>
  </tr>
  <tr>
    <td>Technology</td>
    <td>What role will virtual reality and augmented reality play in the future?</td>
  </tr>
  <tr>
    <td>Science</td>
    <td>What are the latest discoveries in space exploration?</td>
  </tr>
  <tr>
    <td>Science</td>
    <td>How can we mitigate the effects of natural disasters like earthquakes and hurricanes?</td>
  </tr>
  <tr>
    <td>Science</td>
    <td>What are the most promising treatments for diseases like cancer and Alzheimer’s?</td>
  </tr>
  <tr>
    <td>Science</td>
    <td>How can we address the global decline in biodiversity?</td>
  </tr>
  <tr>
    <td>Science</td>
    <td>What are the potential consequences of climate change on ocean ecosystems?</td>
  </tr>
  <tr>
    <td>Business and Economy</td>
    <td>What strategies can businesses adopt to promote sustainability?</td>
  </tr>
  <tr>
    <td>Business and Economy</td>
    <td>How can global economic inequality be reduced?</td>
  </tr>
  <tr>
    <td>Business and Economy</td>
    <td>What are the challenges and opportunities of the gig economy?</td>
  </tr>
  <tr>
    <td>Business and Economy</td>
    <td>How will automation and robotics affect the job market in the coming years?</td>
  </tr>
  <tr>
    <td>Business and Economy</td>
    <td>What are the implications of cryptocurrency on traditional banking systems?</td>
  </tr>
  <tr>
    <td>Politics and Governance</td>
    <td>How can we promote peace and stability in regions affected by conflict?</td>
  </tr>
  <tr>
    <td>Politics and Governance</td>
    <td>What are the key challenges facing democracy in the 21st century?</td>
  </tr>
  <tr>
    <td>Politics and Governance</td>
    <td>How can governments effectively address the refugee crisis?</td>
  </tr>
  <tr>
    <td>Politics and Governance</td>
    <td>What measures should be taken to combat global terrorism?</td>
  </tr>
  <tr>
    <td>Politics and Governance</td>
    <td>How can international cooperation be improved to tackle climate change?</td>
  </tr>
  <tr>
    <td>Health and Wellness</td>
    <td>What are the most effective ways to address mental health issues?</td>
  </tr>
  <tr>
    <td>Health and Wellness</td>
    <td>How can we promote healthy lifestyles and combat obesity?</td>
  </tr>
  <tr>
    <td>Health and Wellness</td>
    <td>What are the challenges of providing healthcare in developing countries?</td>
  </tr>
  <tr>
    <td>Health and Wellness</td>
    <td>How can we reduce the stigma surrounding HIV/AIDS and other infectious diseases?</td>
  </tr>
  <tr>
    <td>Health and Wellness</td>
    <td>What are the long-term effects of widespread use of antibiotics?</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>How can we make education more accessible to underprivileged communities?</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>What reforms are needed in the education system to prepare students for the future job market?</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>How can technology enhance learning in classrooms?</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>What are the benefits and drawbacks of homeschooling?</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>How can we address the issue of student debt?</td>
  </tr>
  <tr>
    <td>Environment</td>
    <td>What are the most effective ways to combat deforestation?</td>
  </tr>
  <tr>
    <td>Environment</td>
    <td>How can we reduce plastic pollution in our oceans?</td>
  </tr>
</table>
<table>
  <tr>
    <th>Topics</th>
    <th>Content</th>
  </tr>
  <tr>
    <td>Environment</td>
    <td>What are the benefits and challenges of transitioning to renewable energy?</td>
  </tr>
  <tr>
    <td>Environment</td>
    <td>How can urban planning be improved to create more sustainable cities?</td>
  </tr>
  <tr>
    <td>Environment</td>
    <td>What measures should be taken to protect endangered species?</td>
  </tr>
  <tr>
    <td>Arts and Culture</td>
    <td>How does art reflect and influence society?</td>
  </tr>
  <tr>
    <td>Arts and Culture</td>
    <td>What are the challenges facing preservation of cultural heritage sites?</td>
  </tr>
  <tr>
    <td>Arts and Culture</td>
    <td>How can we promote diversity and inclusion in the entertainment industry?</td>
  </tr>
  <tr>
    <td>Arts and Culture</td>
    <td>What impact does literature have on society?</td>
  </tr>
  <tr>
    <td>Arts and Culture</td>
    <td>How can we support and encourage creativity in children?</td>
  </tr>
  <tr>
    <td>Sports and Recreation</td>
    <td>How can we ensure the safety and integrity of sports competitions?</td>
  </tr>
  <tr>
    <td>Sports and Recreation</td>
    <td>What are the benefits of sports participation for youth?</td>
  </tr>
  <tr>
    <td>Sports and Recreation</td>
    <td>How can we promote gender equality in sports?</td>
  </tr>
  <tr>
    <td>Sports and Recreation</td>
    <td>What are the environmental impacts of hosting major sporting events?</td>
  </tr>
  <tr>
    <td>Sports and Recreation</td>
    <td>How can we encourage more people to participate in recreational activities?</td>
  </tr>
  <tr>
    <td>Philosophy and Ethics</td>
    <td>What is the meaning of life?</td>
  </tr>
  <tr>
    <td>Philosophy and Ethics</td>
    <td>What are the ethical implications of advancements in biotechnology?</td>
  </tr>
  <tr>
    <td>Philosophy and Ethics</td>
    <td>How should we define and pursue social justice?</td>
  </tr>
  <tr>
    <td>Philosophy and Ethics</td>
    <td>What is the balance between individual freedoms and societal responsibilities?</td>
  </tr>
  <tr>
    <td>Philosophy and Ethics</td>
    <td>How can we cultivate empathy and compassion in society?</td>
  </tr>
  <tr>
    <td>History and Society</td>
    <td>What lessons can we learn from past pandemics?</td>
  </tr>
  <tr>
    <td>History and Society</td>
    <td>How have advancements in communication technology changed society?</td>
  </tr>
  <tr>
    <td>History and Society</td>
    <td>What are the effects of globalization on cultural identity?</td>
  </tr>
  <tr>
    <td>History and Society</td>
    <td>How have social movements influenced policy changes throughout history?</td>
  </tr>
  <tr>
    <td>History and Society</td>
    <td>What are the implications of an aging population on society?</td>
  </tr>
  <tr>
    <td>Food and Agriculture</td>
    <td>How can we ensure food security for a growing global population?</td>
  </tr>
  <tr>
    <td>Food and Agriculture</td>
    <td>What are the environmental impacts of modern agriculture practices?</td>
  </tr>
  <tr>
    <td>Food and Agriculture</td>
    <td>How can we promote sustainable farming methods?</td>
  </tr>
  <tr>
    <td>Food and Agriculture</td>
    <td>What role should genetically modified organisms (GMOs) play in our food supply?</td>
  </tr>
  <tr>
    <td>Food and Agriculture</td>
    <td>How can we reduce food waste at both consumer and production levels?</td>
  </tr>
</table>
Table 5: The Detail of S sets

<table>
  <tr>
    <th>Topics</th>
    <th>Content</th>
  </tr>
  <tr>
    <td>Mathematical Sciences</td>
    <td>What makes prime numbers so special?</td>
  </tr>
  <tr>
    <td>Mathematical Sciences</td>
    <td>Is the Riemann hypothesis true?</td>
  </tr>
  <tr>
    <td>Mathematical Sciences</td>
    <td>Will the Navier–Stokes problem ever be solved?</td>
  </tr>
  <tr>
    <td>Chemistry</td>
    <td>Why does life require chirality?</td>
  </tr>
  <tr>
    <td>Chemistry</td>
    <td>How can we better manage the world’s plastic waste?</td>
  </tr>
  <tr>
    <td>Chemistry</td>
    <td>Will the periodic table ever be complete?</td>
  </tr>
  <tr>
    <td>Medicine & Health</td>
    <td>Can we predict the next pandemic?</td>
  </tr>
  <tr>
    <td>Medicine & Health</td>
    <td>Can a human tissue or organ be fully regenerated?</td>
  </tr>
  <tr>
    <td>Medicine & Health</td>
    <td>Can we ever overcome antibiotic resistance?</td>
  </tr>
  <tr>
    <td>Biology</td>
    <td>How many species are there on Earth?</td>
  </tr>
  <tr>
    <td>Biology</td>
    <td>Why do humans get so attached to dogs and cats?</td>
  </tr>
  <tr>
    <td>Biology</td>
    <td>How do migratory animals know where they’re going?</td>
  </tr>
  <tr>
    <td>Astronomy</td>
    <td>Why do black holes exist?</td>
  </tr>
  <tr>
    <td>Astronomy</td>
    <td>What is the smallest scale of space-time?</td>
  </tr>
  <tr>
    <td>Astronomy</td>
    <td>How many dimensions are there in space?</td>
  </tr>
  <tr>
    <td>Physics</td>
    <td>Are there any particles that behave oppositely to the properties or states of photons?</td>
  </tr>
  <tr>
    <td>Physics</td>
    <td>Will we ever travel at the speed of light?</td>
  </tr>
  <tr>
    <td>Physics</td>
    <td>Is quantum many-body entanglement more fundamental than quantum fields?</td>
  </tr>
  <tr>
    <td>Information Science</td>
    <td>Can DNA act as an information storage medium?</td>
  </tr>
  <tr>
    <td>Information Science</td>
    <td>Is there an upper limit to computer processing speed?</td>
  </tr>
  <tr>
    <td>Information Science</td>
    <td>Can AI replace a doctor?</td>
  </tr>
  <tr>
    <td>Engineering & Material Science</td>
    <td>What is the ultimate statistical invariances of turbulence?</td>
  </tr>
  <tr>
    <td>Engineering & Material Science</td>
    <td>How can we develop manufacturing systems on Mars?</td>
  </tr>
  <tr>
    <td>Engineering & Material Science</td>
    <td>Is a future of only self-driving cars realistic?</td>
  </tr>
  <tr>
    <td>Neuroscience</td>
    <td>Where does consciousness lie?</td>
  </tr>
  <tr>
    <td>Neuroscience</td>
    <td>Is it possible to predict the future?</td>
  </tr>
  <tr>
    <td>Neuroscience</td>
    <td>How smart are nonhuman animals?</td>
  </tr>
  <tr>
    <td>Ecology</td>
    <td>Can we stop global climate change?</td>
  </tr>
  <tr>
    <td>Ecology</td>
    <td>What happens if all the ice on the planet melts?</td>
  </tr>
  <tr>
    <td>Ecology</td>
    <td>Can we create an environmentally friendly replacement for plastics?</td>
  </tr>
  <tr>
    <td>Energy Science</td>
    <td>Could we live in a fossil-fuel-free world?</td>
  </tr>
  <tr>
    <td>Energy Science</td>
    <td>What is the future of hydrogen energy?</td>
  </tr>
  <tr>
    <td>Energy Science</td>
    <td>Will cold fusion ever be possible?</td>
  </tr>
  <tr>
    <td>Artificial Intelligence</td>
    <td>How does group intelligence emerge?</td>
  </tr>
  <tr>
    <td>Artificial Intelligence</td>
    <td>Will artificial intelligence replace humans?</td>
  </tr>
  <tr>
    <td>Artificial Intelligence</td>
    <td>Can robots or AIs have human creativity?</td>
  </tr>
</table>
D  Topic View Proportions Distribution

D.1  Set G

![Pareto chart of view proportions for 12 topics on Set G](page_186_153_1077_349.png)

Figure 10: Pareto chart of view proportions for 12 topics on Set G

D.2  Set S

![Pareto chart of view proportions for 12 topics on Set S](page_186_573_1077_349.png)

Figure 11: Pareto chart of view proportions for 12 topics on Set S

E  Use of AI Assistant

We use GPT-3.5 and GPT-4 to improve the code style and the writing of the manuscript.
F  Model Details

Our experiment used GPT-4-1106 preview, GPT-4o-2024-05-13, Llama 3.1: 70B, and Llama 3.3: 70B, the temperature is uniformly set to 0.8, each experiment of GPT 4 costs about 80 dollars, GPT 4o costs 40 dollars, and Llama spends about 2 hours on 1 * A100 for each experiment.

Reviewer paper summaries: This work explores the choice behaviors of LLMs from the intrinsic and the extrinsic of motivations that can affect the human choices, based on the Self-Determination Theory (SDT). Three research questions are raised: (1) How intrinsic factors such as exploratory, playful, and curiosity-driven affect the LLMs, (2) How extrinsic factors such as social influences affect the LLMs, and (3) How intrinsic and extrinsic factors interact with each other. A virtual QA platform has been created for the experiments. The outcomes show the analytics of the intrinsic and the extrinsic factors with the LLMs’ behaviors.

The article explores the extent to which Large Language Models exhibit behaviours analogous to human decision-making, particularly through the lens of Self-Determination Theory, a prominent psychological theory distinguishing intrinsic and extrinsic motivations. To investigate this, the authors designed a question-answering platform that allowed them to observe and measure LLM behaviours using two distinct question datasets. Their analysis focused on three aspects: choice behaviour, social influence, and psychological attention bias; under three experimental conditions: independent (no external influence), social influence (models observed existing popularity indicators), and induced-social influence (popularity was artificially manipulated). Their findings suggest that LLMs mimic certain human behavioural patterns related to motivation, such as intrinsic attention biases and social conformity. Additionally, the authors employed a "self-report" task with the LLMs to examine whether intrinsic and extrinsic motivational factors operate independently within the models, finding statistical support for distinct motivational pathways, albeit reflecting learned patterns from training rather than genuine psychological states.

This paper explores the factors influencing Large Language Models' (LLMs) choice behavior.  The authors draw upon Self-Determination Theory from human psychology to investigate intrinsic (attention bias) and extrinsic (social influence) motivations in LLMs.  They created a virtual QA platform to observe LLM behavior under different conditions, manipulating the presence and nature of social influence indicators.  The study finds that LLMs exhibit attention bias and are influenced by social factors, showing effects similar to the Matthew Effect and Conformity in humans.
Summary of strengths: * This work investigates the properties of LLMs from a novel perspective, aligned with the self-determination theory in a multi-agent experimental environment. 
* The approach to experimental setup is interesting and can be applied to other studies. 
* The paper is well-written and well-motivated.

The paper's strengths lie in its novel, interdisciplinary approach. The authors combine insights from psychology (Self-Determination Theory) with computational linguistics research. The authors demonstrate strong engineering competence through their development of the QA platform, which provides a solid experimental basis for observing LLM behaviour. The insights into model biases that result from their experiments appear valid and well-supported, even though they align with expected outcomes given how these models are constructed and how they operate. Additionally, the authors' effort to apply established theories from psychology to Large Language Models has the potential to enhance our understanding of model behaviours as well as enrich the original theories themselves, by providing computational perspectives.

1. The paper uses established psychological theory (Self-Determination Theory) to analyze LLM behavior, providing a strong theoretical framework.    
 
2. The paper identifies and analyzes relevant factors influencing LLM decision-making, such as intrinsic bias and extrinsic social influence.    
3. The findings reveal human-like behavioral patterns in LLMs, contributing to the understanding of LLM explainability and potential risks.
Summary of weaknesses: * An additional experiments on human can improve the comprehensiveness of this study. For example, introducing human agents into the QA platform to interact with LLMs and other human agents could be very interesting.

The paper could improve its targeting towards the intended audience. Specifically, Self-Determination Theory (SDT) and its core concepts (e.g., precise definitions of intrinsic and extrinsic motivation in SDT) should be briefly but clearly introduced. Furthermore, the authors should explicitly justify why they selected SDT over alternative psychological theories.

I struggled to understand how the observed LLM behaviour relates to intrinsic human motivation. An LLM's behaviours largely reflect statistical regularities from their training corpus and human feedback data. Given this, it remains unclear how EXACTLY a transformer model, trained purely on language and human preference data, can meaningfully be explained through SDT concepts of intrinsic or extrinsic motivation. The authors should explicitly discuss why transformers exhibit these SDT-like patterns in behaviour; what is it in architecture, training, and preference data that leads to this? While I do not rule out such connections, it is for the authors to provide explanations.

Minor notes:

The three references cited to support claims about the general abilities of LLMs (lines 31–33) feel rather thin. 

The results regarding domain preferences of the GPT and Llama models are not particularly surprising, given these models' similar transformer architectures, extensive language training, and presumed similarity in human preference data. The authors should acknowledge and/or discuss this point.

The paper lacks clarity regarding how exactly the "G set" was constructed. The authors should explicitly detail the exact prompts, exact GPT-4 model version, and generation parameters used.

More models and particularly different model sizes could be included. Adding more Llama models should not even cost $10.

1. The authors acknowledge the challenge of measuring "time spent" by LLMs and instead use "frequency of repeated selection" as a proxy for attention bias.  This indirect measurement could be a point of contention.   

2. While the virtual QA platform allows for controlled experiments, it might not fully capture the complexities of real-world scenarios.    

3. The choice of models, while aiming for diversity, is limited to four models (only two series).  Expanding the number and type of LLMs could strengthen the findings' generalizability.   

4. The paper focuses primarily on question selection as the choice behavior.  It may not fully explore the nuances of other choice-based behaviors in LLMs.   

5. The study relies on self-reporting by LLMs to understand their decision-making process.  The reliability and validity of LLM self-reports could be questioned.
Comments, suggestions and typos: Capitalization can be adjusted. For example, “3.3.1 Behavior Shift under Conflict Situation” is inconsistent with “3.3.2 Experimental design” in terms of capitalization.

Suggestions are given alongside the described weaknesses, see above.

The authors should provide more context on the specific questions used in the question sets (G and S) in the main text, rather than only in the appendix.  This would give the reader a clearer understanding of the experimental setup.   

It would be beneficial to discuss the limitations of using "views" as the primary analysis metric (as mentioned in section 2.1) more thoroughly.    

In section 3.3, the explanation of how self-reporting is implemented with LLMs needs further clarification.    

The writing could be improved for clarity and conciseness in some sections. For example, the description of the "Induced Social Influence Condition" could be more straightforward.
Overall assessment scores: 3

3.5

3
Metareview score: 3
Metareview: All three reviewers provided scores of 3, 3, and 3.5. After carefully considering their comments and the discussions, I conclude that this paper should be accepted as a finding paper in this round of ARR.
Summary of reasons to publish: All reviewers agree this paper is well written and leverages psychological theory (Self-Determination Theory) to analyze LLM behaviors, which is interesting and novel.