Experimental Results
We evaluated our STRUX against strong baselines for forecasting stock investment decisions. This includes DeLLMa Liu et al. ( 2024 ) , which incorporates uncertainty into LLM decision-making using classical decision theory and has been tested on tasks such as agriculture planning and finance. Additionally, we tested gpt-4o-mini-2024-07-18 and Llama3-8b-Instruct by providing either full transcripts or concise fact tables to elicit investment decisions; see Appendix for the prompt.

System Comparisons. Table 1 shows the macro-averaged precision, recall, F-scores, and accuracy for the test set. STRUX outperforms strong baselines in accuracy and F-scores for stock investment decisions. Our findings indicate that adding reinforcement learning (STRUX+RL) leads to stronger performance compared to using the SFT method alone. We also find that direct prompting methods, e.g., GPT-4o-mini with Fact Table, tend to produce overly positive outcomes, often failing to suggest Strong Sell or Sell decisions. This bias can be traced back to the optimistic financial descriptions by company executives, and without fine-tuning, it leads LLMs to display a bias toward bullish predictions. It is also worth mentioning that our test set has an imbalanced label distribution. A random baseline achieves an accuracy of 19.11%, and our STRUX+RL model shows a notable improvement, reaching an accuracy of 25.55%. 2 2 2 We observe that OpenAI’s o1-mini-2024-09-12 , which generates a detailed internal thought process, only achieves a 16% accuracy on this task, possibly due to overthinking.

Supporting Facts. We analyzed the supporting facts identified by the model in cases of correct decisions after reflections. Statistics are presented in Table 3 . Each transcript is distilled into a table of about 40 facts, from which the model selects 9. The selection is predominantly positive, with 8 positive and 1 negative fact; about half of the negative fact has an impact strength of 2–3. This indicates that adding expert knowledge on potential negative factors such as financial risks could make the fact tables more comprehensive. Figure 2 illustrates our experiment in which the model selects supporting facts from three ranges during self-reflection: 3-6, 6-10, and 10-15. We found that selecting 6-10 facts consistently yielded the highest performance.

Decision Paths. STRUX performed 4 rounds of self-reflection, because there are 5 ground-truth decisions. Figure 3 presents the confusion matrices, with each round of reflection improving the model’s accuracy. The model initially favored ‘Hold’ as a conservative decision. After two rounds of reflection, it began to predict decisions more accurately. Ultimately, the errors arise from the model’s reluctance to recommend ‘Strong Sell’ likely due to the postive language in executive speeches.

Table 2 shows common decision paths during reflection. Interestingly, reflection can lead to abrupt decision changes, such as a direct jump from Buy to Strong Sell, instead of gradual shifts (e.g., Buy \rightarrow Hold \rightarrow Sell). Moreover, reflection does not always yield perfect outcomes; the model can repeat decisions from previous cycles despite being instructed not to. These observations suggest that guardrails for self-reflection may help stabilize the decision-making process and prevent radical changes.