InvestAlign: Align LLMs with Investor Decision-Making under Herd Behavior

Authors: Huisheng Wang, Zhuoshi Pan, Hangjing Zhang, Mingxiao Liu, H. Vicky Zhao
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: alignment, investment decision, large language model, supervised fine-tuning

Abstract:
Studying investor decision-making processes under herd behavior is of great significance in microeconomics and behavioral finance. Large Language Models (LLMs) can be leveraged to assist in solving specific classes of complex investment problems. However, the investment decisions generated by existing LLMs often deviate from real-user data. One method to align LLMs with investor decision-making processes is Supervised Fine-Tuning (SFT), which requires a substantial amount of real-user data that is costly to collect and raises concerns about privacy and security. To overcome data scarcity, in this work, we propose **InvestAlign**, a low-cost and high-quality method that constructs large-scale SFT training datasets based on the theoretical solution to a specific simpler optimal investment problem, rather than the original complex one. We theoretically demonstrate that fine-tuning LLMs with these datasets leads to faster parameter convergence compared to using real-user data. By fine-tuning LLMs, we obtain **InvestAgent**s, which align more closely with real-user data than pre-SFT LLMs in both the simple and original complex problems examined in our study. This highlights **InvestAlign** as a promising approach with the potential to address complex optimal investment problems and align LLMs with investor decision-making processes in economics and finance.

Decision: reject
Reviewer scores: 6, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730345558033

thread_type:
official_review

<thread_object>
creator: reviewer_7wmn

<content>
summary:
This paper proposes InvestAlign, a novel method to align Large Language Models (LLMs) with investor decision-making processes under herd behavior. The key innovation is using theoretical solutions from simpler investment problems to construct training datasets for fine-tuning LLMs, rather than relying on costly real-user data. The authors demonstrate both theoretically and empirically that this approach leads to faster parameter convergence compared to using real-user data, while achieving better alignment with actual investor behavior.

strengths:
The paper demonstrates significant originality in its approach to LLM alignment in the financial domain. Rather than following the conventional path of collecting extensive real-world data, it cleverly leverages theoretical solutions from simpler problems to generate training data. The empirical validation is comprehensive, testing multiple LLM architectures (GPT-3.5, GLM-4, Qwen-2, and Llama-3.1) and comparing performance against both real-user data and pre-SFT baselines.

weaknesses:
While the authors demonstrate effectiveness on two specific types of herd behavior (absolute and relative), it's unclear how well the method generalizes to other forms of behavioral biases in investment decisions.

questions:
How sensitive is the method to the choice of the "simpler problem" used to generate training data? What criteria should be used to select appropriate simplified problems?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730429973790

thread_type:
official_review

<thread_object>
creator: reviewer_mvgl

<content>
summary:
This paper presents InvestAlign, a methodology for constructing training datasets by leveraging theoretical solutions to similar and simpler problems, aimed at aligning large language models (LLMs) with the decision-making processes of investors exhibiting herd behavior. The paper's experiments demonstrate that LLMs fine-tuned using these theoretical solutions exhibit faster parameter convergence and superior alignment performance when addressing complex investment issues, in contrast to traditional methods reliant on authentic user data. While the results indicate the potential of InvestAlign, the authors also acknowledge that theoretical solutions may not fully capture the complexities of real investor behavior. Future research will explore hybrid approaches that integrate theoretical solutions with real user data to further enhance model performance.

strengths:
- The paper incorporates a multi-round experimental design to compare the performance of LLMs before fine-tuning with that of investment agents fine-tuned using InvestAlign. By employing various investment attributes and random seeds, the study ensures the robustness and credibility of the experimental results. This systematic approach to experimental design enhances the persuasive power of the research conclusions.
- The study employs a method for constructing SFT training datasets based on theoretical solutions, utilizing the deterministic characteristics of these solutions to enhance parameter convergence speed. This innovative approach demonstrates how effective model training can be achieved even in the absence of extensive authentic user data, thereby improving alignment with human decision-making processes.
- The paper clearly outlines the importance of understanding investor decision-making processes within the contexts of microeconomics and behavioral finance, as well as the cost and privacy issues associated with traditional data collection methods. The introduction of InvestAlign not only addresses this research gap but also provides new insights for the practical application of models in financial decision-making, possessing significant theoretical and practical implications.

weaknesses:
- Although the training datasets constructed from theoretical solutions offer an effective method for fine-tuning, this approach has only been demonstrated to be effective in simplified ideal scenarios, neglecting the diversity and nonlinear characteristics of complex investor behavior in real-world contexts. The numerous assumptions and simplifications inherent in theoretical models may lead to suboptimal performance of this method when confronted with authentic market data.
- The paper collects a limited amount of real user data, with only 119 samples for P2 and 90 samples for P1, which may be insufficient to support the validation of the theoretical solutions discussed in Section 3.3. Furthermore, in subsequent comparisons of experimental results, this limited dataset may not adequately represent a broader investor population and diverse market contexts, potentially introducing randomness that could impact the intuitiveness and accuracy of the results.
- While the paper emphasizes the importance of studying investor decision-making, it offers relatively limited discussion on existing related research and models in the current financial market. The lack of a thorough exploration of the differences between established methods and InvestAlign may result in an inadequate assessment of the innovative aspects of this approach.

questions:
- In constructing the SFT training dataset, the paper assumes that the probability distribution of the theoretically optimal decisions approximates a Pareto distribution, treating real user data as theoretical solutions plus white noise. Is this assumption consistently valid across different market environments? If market conditions change or investor behavior undergoes significant alterations, will this affect the relationship between theoretical solutions and real user data, thereby influencing the model's training effectiveness?
- The fine-tuning training dataset in the paper utilizes input-output pairs generated by a custom template. Which large model was employed to generate these data pairs? Were there any prompt settings for other linguistic contexts in this model? The experiments involve four fine-tuned LLM models; should there also be multiple tests conducted on the large model generating the dataset? Although the prompts provided in the paper are relatively formulaic, they may still exhibit certain hallucinatory tendencies and inaccuracies. Does the paper address strategies for mitigating and resolving these issues?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730561559011

thread_type:
official_review

<thread_object>
creator: reviewer_wahm

<content>
summary:
This paper proposes a method to instill human values into LLMs. To train LLMs on actual human values or decision-making criteria, a large amount of real human behavior data is typically required, and this work aims to address that challenge. When a complex task needs to be performed, the authors propose creating a simpler task instead, obtaining a theoretical solution, and using it for training, which they argue is more efficient than using real human behavior data.

strengths:
The study collected data through questionnaires from real participants and utilized this data.

weaknesses:
- Although the proposed P1 (the complex, real-world task) and P2 (a simpler task for which a theoretical solution can be obtained) differ significantly from an optimization perspective, they do not appear very distinct from the viewpoint of either LLMs or humans. Thus, the claim that complex tasks can be handled by replacing them with simpler tasks with theoretical solutions seems logically overstretched.
- Additionally, the proposed task seems to focus more on assessing how well the model follows theoretically calculated values rather than instilling genuine human values. Since real humans would naturally struggle to solve the problem accurately, it is unsurprising that training on a theoretical solution yields better results.
- Consequently, I feel that the logical leaps in the authors' claims are significant, and the experimental design may not be appropriate to support these claims.

questions:
Do the authors believe this approach could generally be used to create LLM agents that embody human values? To achieve this, it would be necessary to address far more complex and high-dimensional problems than the rigid ones presented in this paper. Is it feasible to transform such complex problems into simplified tasks with theoretical solutions, as proposed in this study?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730657763375

thread_type:
official_review

<thread_object>
creator: reviewer_ubav

<content>
summary:
This paper introduces a method to align large language models (LLMs) with real investor decision-making processes under herd behavior in financial markets. Recognizing the difficulty and expense of collecting real-user investment data, the authors propose InvestAlign, a supervised fine-tuning (SFT) approach that uses theoretically generated data from a simpler, theoretical investment problem that can be solved analytically. Then the data constructed from the theoretical solution provides a high-quality (synthetic) training set for LLMs, which enable them to simulate investor decisions in a cost-effective and privacy-preserving way.

Key contributions of the paper include (i) constructs a dataset based on the theoretical solution of a simpler investment model, and by fine-tuning LLMs on this theoretical data, they demonstrate closer alignment with real-user decision-making data than pre-trained models; (ii) the authors theoretically and empirically show that fine-tuning with theoretical data enables faster parameter convergence compared to real-user data; (iii) using real data collected from human user, they show that InvestAgents mimic investor decisions in both the simplified and original complex investment scenarios.

In sum, this paper proposes an efficient method framework for aligning LLMs with investor behaviors using theoretical solutions, where the authors demonstrated both theoretical and empirical success for their framework.

strengths:
The paper presents a novel approach that addresses the challenging problem of aligning LLMs with investor decision-making under herd behavior. The originality lies in creatively using *theoretical solutions* from simpler investment models that have established analytical results. Then the theoretical solution can be used to generate high-quality training data, which differs from conventional reliance on real-user data for alignment. By applying a simplified mathematical solution to a more complex decision-making process, I think this paper presents a very nice contribution, which proposes a unique, resource-efficient method for improving LLM performance in behavioral finance. This approach bridges theory-driven data generation and model alignment to LLM fine-tuning strategies in finance.

One thing that I like the most of the paper is that the authors validate the approach by demonstrating faster parameter convergence and more accurate alignment with investor behaviors from both the theoretical and empirical perspective. This theoretical basis is a very nice component. Moreover, the authors provide thorough experimental evidence, including a rigorous comparison between pre-trained LLMs, real-user data, and InvestAgents fine-tuned with theoretical data. These strong experimental results, combined with theoretical justification, highlight the quality and reliability of the proposed approach.

The paper is well-written and easy to follow. 

In sum, this work holds significant theoretical contribution and practical implications for AI applications in finance. The insights could also extend to other domains where user alignment is essential but data availability is limited, broadening the potential impact. Additionally, the method’s emphasis on optimizing parameter convergence presents a meaningful advancement for the efficiency of fine-tuning methods in LLMs.

weaknesses:
1. While the paper introduces an efficient method for data generation through simplified theoretical models, this approach may not fully capture the nuanced decision-making processes seen in real-world investors, especially under complex, multi-dimensional financial conditions. Relying solely on theoretical data might limit the model’s robustness. I suggest the authors consider supplementing theoretical data with smaller samples of real-user data to improve robustness.

2. The paper lacks a comparison with other alignment techniques for LLMs, particularly those that might use alternative strategies such as reinforcement learning from human feedback (RLHF) or existing behavioral finance datasets. The authors could improve the paper by adding more comparison. 

3. The paper’s focus on herd behavior is well-motivated but may limit the generalizability of its findings to other types of investor biases, such as overconfidence, or recency bias. The exclusive focus on herd behavior could make the model less generalizable. The authors could comment more on whether InvestAlign can be adapted to model other investor behaviors.

4. Still related to generalizability, it is not entirely clear how generalizable the approach is to different segments of the financial market or varying types of investors (e.g., retail vs. institutional). The simplified model may not capture the complexity of larger institutional investor decisions or differences in risk tolerance.

questions:
1) Do the authors believe the InvestAlign approach could be extended to other human behavior, such as recency bias or overconfidence, which also impact investor behavior? If so, what adaptations would be necessary? Also, are there specific limitations they foresee in applying this simplified model to unpredictable or multifactorial investment decisions in real world?

2) How about including comparisons with other alignment techniques, such as reinforcement learning from human feedback (RLHF) or using existing behavioral finance datasets?

3) Could the authors elaborate on how applicable InvestAlign is to different types of investors (e.g., retail vs. institutional) or market segments with varying characteristics? More importantly, given that the alignment was validated within a specific application (the investment context), how confident are the authors in the method’s generalizability to different financial decisions or even other high-stakes domains beyond finance?

4) What specific assumptions does the theoretical model used for data generation rely on? How might these assumptions impact the model’s alignment with real investor behavior?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732002556564

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers,

We sincerely appreciate the insightful critiques and valuable feedback from the reviewers. Addressing the key concerns raised, we would like to highlight the most critical issues and our corresponding revisions as follows.

1. **Generalizability and Robustness:** This has been identified as a paramount concern across all reviews (WAhM, MvGL, 7wmn, uBAV). To bolster the robustness and applicability of our theoretical model, we have taken significant steps. Firstly, we have supplemented our theoretical data with smaller samples of real-user data to enhance the model's robustness, as suggested in uBAV's review (AW1). This hybrid approach not only maintains the advantages of theoretical data in terms of parameter convergence but also grounds the model in real-world investor behavior. Additionally, we have expanded our comparison to include other alignment strategies, such as training on existing behavioral finance datasets, which were recommended by uBAV (AW2). This comparison underscores the orthogonal nature of our method to traditional Supervised Fine-Tuning (SFT), and how it can complement these methods.

2. **Clarification of Study Scope and Objective:** A key point we want to emphasize is the scope of our study. Our goal is not to propose a universal investment language model or a fully capable agent that can solve every complex financial decision-making problem. Instead, our primary objective is to address a specific challenge: data scarcity when training large language models (LLMs) in the context of investor decision-making. Real-world investor data is often limited due to privacy concerns and high collection costs, making it difficult to build extensive datasets for training. Thus, our work focuses on exploring a new potential method to overcome this data scarcity issue by constructing training datasets based on simplified problems with theoretical solutions. By using this approach, we aim to provide a feasible pathway for aligning LLMs with human-like decision-making behaviors in scenarios where real data is insufficient. This differentiation is crucial, as it highlights that our paper targets solving the data scarcity problem rather than developing a universal agent for all investment decision-making tasks.

3. **Assumptions of the Theoretical Model:** uBAV and WAhM questioned the assumptions underlying our theoretical model. We have provided a detailed explanation of these assumptions and their implications on model alignment with real investor behavior in uBAV's review (AQ4) and WAhM's review (AW1). We acknowledge the limitations and plan to relax these assumptions in future work to better reflect real market conditions.

4. **Applicability to Different Types of Investors and Behavior Biases:** uBAV inquired about the applicability of InvestAlign to various investor types and behavior biases. We have clarified that our current model focuses on a simplified investment scenario and plan to extend our analysis to other biases and more complex decision-making processes in future work, making our model more comprehensive (AW3 and AW4).

5. **Insufficient Real User Data:** MvGL pointed out the limitation of our real user data sample size. In response, we have increased our sample size to ensure that our findings are more representative and robust, aligning with statistical testing recommendations (AW2).
In summary, we have made substantial efforts to address the reviewers' concerns, particularly focusing on enhancing the generalizability and robustness of our theoretical model, which was the most critical feedback. We believe these revisions significantly strengthen our paper and its contributions to the field of aligning LLMs with investor decision-making processes.

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper aims to align large language models (LLMs) with real investor decision-making processes under herd behavior in financial markets. Reviewers agreed that this paper studies a very interesting and challenging problem and makes contributes to the community by collecting an extensive real-world data, presenting a multi-round experimental design, etc. However, reviewers also identified some limitations of this work, such as the lack of comparisons with other LLM alignment techniques, the design of tasks P1 and P2, generalization ability of the proposed approach, etc. Overall, this paper in its current version is not ready for publication at ICLR.