Motivation
Decision-making is complex, as it requires the evaluation of various determinants that can influence outcomes Eigner and Händler ( 2024 ) . This ability is crucial across multiple fields, ranging from healthcare, where decisions can determine patient health outcomes Lehman et al. ( 2022 ) , to finance, where investment choices can impact financial stability Keith and Stent ( 2019 ); Liu et al. ( 2023 ) . For LLMs to be effective, they must not only identify relevant facts but also weigh the favorable and unfavorable aspects to reach insightful conclusions. To date, it remains unclear whether LLMs can effectively balance multiple factors in complex scenarios to make rational decisions.

LLMs also produce lengthy, plain text explanations that can sometimes overwhelm users with too much information or ambiguity Vafa et al. ( 2021 ); Alkhamissi et al. ( 2023 ); Sharma et al. ( 2023 ); Ye et al. ( 2023 ); Wang et al. ( 2024 ) . As we increasingly rely on those LLMs for critical decision-making, it is important to prioritize transparency and accountability Ludan et al. ( 2023 ) . We propose structuring these explanations into a table format, where each fact is listed with a ‘strength level’ that measures its influence on the decision-making process. This approach not only facilitates review and modification of various facts by humans, but also enhances the transparency of the decisions made.

Further, a significant advantage of LLMs is their ability to reason through complex scenarios, which can enhance the decision-making processes (Shinn et al., 2023 ; Yao et al., 2023 ; Zeng et al., 2024 ; Band et al., 2024 ) . Notably, DeLLMa Liu et al. ( 2024 ) uses classical decision theory to help LLMs make decisions under uncertainty. It infers a utility function through prompting and optimizes the expected utility using Monte Carlo estimation. Feng et al. ( 2024 ) calculate decision probabilities using a Bayesian model and present results on datasets such as Common2Sense (Singh et al., 2021 ) and PlaSma (Brahman et al., 2023 ) . In contrast, our approach involves fine-tuning an LLM with domain-specific knowledge to ensure it prioritizes supporting facts accurately. Training instances are generated via a series of reflection steps, without relying on human annotations.

Our research explores the potential of using earnings call transcripts to forecast stock investment decisions Sawhney et al. ( 2020 ); Medya et al. ( 2022 ); Lopez-Lira and Tang ( 2023 ); Ni et al. ( 2024 ) . Publicly traded companies in the U.S. are mandated by the Securities and Exchange Commission (SEC) to regularly report their financial performance, often through earnings calls. These calls include presentations from senior executives , such as the CEO and CFO, followed by a Q&A session with financial analysts. The objective is to reassure investors about the company’s management and strategy. With the rise of LLMs in financial services Zhu et al. ( 2021 ); Sang and Bao ( 2022 ); Cao et al. ( 2024 ); Reddy et al. ( 2024 ) , analyzing earnings call transcripts to guide stock investment decisions presents a promising opportunity to test the effectiveness of LLM-assisted decision-making.

Our research contributions include: (a) we introduce STRUX, a novel framework designed to enhance the decision-making processes of LLMs. STRUX improves accuracy and transparency by meticulously constructing a fact table, analyzing these facts through a series of reflective steps, and fine-tuning the LLM to prioritize crucial information. (b) Our experiments demonstrate that STRUX surpasses strong baselines in forecasting stock investment decisions, proving its effectiveness. Its structured explanations further enhance decision transparency and represent a notable step towards practical decision-making with LLMs. 1 1 Our data are available at http://struxdata.github.io