RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
The STRUX System
STRUX is tasked with predicting a company’s post-earnings stock trend to inform the investment decision. It is set to select the most relevant facts from a provided fact table, ensuring a balanced representation of positive and negative facts affecting the stock price. Each selected fact must then be evaluated for its potential impact on the stock’s price movement. A “+” symbol indicates a positive impact, with the number of symbols varying from one (+) to three (+++) showing the increasing strength. Conversely, a “-” symbol denotes a negative impact, with one (-) to three (—) symbols reflecting the severity of the negative influence.

Our system then combines and analyzes all the selected facts to forecast the direction of the stock price movement. The outcomes include: Strongly Buy (SB), Buy (B), Hold (H), Sell (S), or Strongly Sell (SS). It also provides a justification elaborating on its rationale, focusing on the key facts that influence this decision. As illustrated in Figure 1 , our structured explanations consist of three components: {supporting facts, decision, and brief justification}. Supporting facts can be both favorable and adverse, along with their respective strengths.
Generating Structured Explanations Through Self-Reflection
We create a fact table from each company’s earnings call transcript to summarize key financial metrics, which are crucial for making informed investment decisions. Following Koa et al. ( 2024 ) , we input executive speeches from either the Prepared Remarks or Q&A sessions into the LLM. Summaries are proportional in input length. Each speech from the Prepared Remarks is summarized into 3-5 key facts, while those from the Q&A session are condensed into 1-3 key facts. The fact table was generated using OpenAI’s gpt-4o-mini-2024-07-18 ; refer to the Appendix for the prompt. It distills essential information from a lengthy transcript, highlighting key aspects of a company’s financials Cho et al. ( 2021 , 2022 ) .

Reflection. We use a series of reflective steps to create training instances without requiring human annotations. This reflection was performed by GPT-4o-mini, aiming to help the model learn from its mistakes. When the model makes a poor investment decision, we notify it of the error and prompt it to identify any significant flaws in its fact selection, strength assignment, or reasoning processes. We also provide a list of previous incorrect decisions, including the reasons behind those decisions. Importantly, we ask the model to come up with a different decision from its previous ones without revealing the correct answer . This approach allows us to observe the model’s independent decision-making that emerges from reflection. Our prompt used for reflection can be found in the Appendix.

Demonstrations and Comparisons. Our ‘demonstrations’ data contains training instances where output \mathbf{y} has a correct decision post-reflection. We utilize this data to fine-tune Llama3 , helping it prioritize relevant facts and make accurate decisions. The ‘comparisons’ data consists of paired outputs, \mathbf{y} and \mathbf{y}^{*} , where \mathbf{y}^{*} is the output with the correct decision, and \mathbf{y} is the prior model output in a series of reflections which has incorrect decision. These pairwise comparisons help train a reward model to favor outcomes that lead to correct decisions. Training instances that do not yield correct decisions after all reflections are excluded from demonstration or comparison data.
Fine-tuning LLMs for Decision-Making
STRUX+SFT. We start with the base LLM model, Llama3-8b-Instruct , and fine-tune it using our demonstrations data to develop the SFT model p_{\theta}(\mathbf{y}|\mathbf{x}) . Specifically, the input \mathbf{x} is a fact table created from an earnings call transcript, and the output \mathbf{y} includes structured explanations that contain {supporting facts, a decision, a brief justification}. As illustrated in Equation 1 , the fine-tuning process aims to minimize the negative log-likelihood of the data. Here, \mathbf{y}^{*}\sim\pi(\cdot|\mathbf{x}) represents the demonstrations provided by gpt-4o-mini-2024-07-18 , each of which contains the correct decision.

\mathcal{L}_{SFT}(\theta)=\mathop{-\mathbb{E}}_{\begin{subarray}{c}\mathbf{x}\sim\mathcal{D},\mathbf{y}^{*}\sim\pi(\cdot|\mathbf{x})\end{subarray}}\left[\,\log p_{\theta}(\mathbf{y}^{*}|\mathbf{x})\,\right] (1)

STRUX+RL. In reinforcement learning, we start with a policy p_{\theta^{\prime}}(\mathbf{y}|\mathbf{x})=p_{\theta}(\mathbf{y}|\mathbf{x}) and fine-tune the policy p_{\theta^{\prime}}(\mathbf{y}|\mathbf{x}) using a reward function r_{\phi}(\mathbf{x},\mathbf{y}) . We employ proximal policy optimization to optimize the expected reward. This process involves repeatedly choosing an instance from our training set, calculating the reward for the model’s response with the reward function, then updating model parameters towards maximizing the reward. Following Ziegler et al. ( 2020 ) , we include a penalty \beta\frac{p_{\theta^{\prime}}(\mathbf{y}|\mathbf{x})}{p_{\theta}(\mathbf{y}|\mathbf{x})} to the reward to prevent p_{\theta^{\prime}}(\mathbf{y}|\mathbf{x}) from diverging too far from p_{\theta}(\mathbf{y}|\mathbf{x}) where the learned reward r_{\phi}(\mathbf{x},\mathbf{y}) is valid; \beta is set to 0.2 in our study.

\mathcal{L}_{RL}(\theta^{\prime})=\mathop{-\mathbb{E}}_{\begin{subarray}{c}\mathbf{x}\sim\mathcal{D},\ \mathbf{y}\sim p_{\theta^{\prime}}(\cdot|\mathbf{x})\end{subarray}}\left[r_{\phi}(\mathbf{x},\mathbf{y})-\beta\frac{p_{\theta^{\prime}}(\mathbf{y}|\mathbf{x})}{p_{\theta}(\mathbf{y}|\mathbf{x})}\right]

The reward function r_{\phi}(\mathbf{x},\mathbf{y}) is trained using ‘comparisons’ data. For every input \mathbf{x} , a response with the correct decision \mathbf{y}^{*} is paired with \mathbf{y} , corresponding to the incorrect response prior to a successful reflection . Below, \sigma(r_{\phi}(\mathbf{x},\mathbf{y}^{*})-r_{\phi}(\mathbf{x},\mathbf{y})) represents the probability that \mathbf{y}^{*} is preferred over \mathbf{y} , denoted by p(\mathbf{y}^{*}\succ\mathbf{y}) . We implement the reward r_{\phi}(\mathbf{x},\mathbf{y}) as a linear function of the final embedding from the SFT model, and use this reward model to guide the policy learning during RL.

\mathcal{L}_{RM}(\phi)=\mathop{-\mathbb{E}}_{\begin{subarray}{c}\mathbf{x}\sim\mathcal{D},\ \mathbf{y},\mathbf{y^{*}}\sim\pi(\cdot|\mathbf{x})\end{subarray}}\left[\log\sigma(r_{\phi}(\mathbf{x},\mathbf{y}^{*})-r_{\phi}(\mathbf{x},\mathbf{y}))\right]
Earnings Call Transcripts
Our dataset includes 11,950 quarterly earnings call transcripts from the Motley Fool website, collected by Hu et al. ( 2024 ) , covering the period from 2017 to 2024. It contains transcripts from 869 companies listed on the NASDAQ 500 and S&P 500, with an average of 10,187 tokens per transcript. Due to resource limits, we construct a balanced training set with 100 transcripts from each of the 11 financial sectors. Our test set consists of 587 transcripts from 2024, carefully chosen to ensure they were not part of the LLM pretraining, which has a cutoff up to December 2023. Our study focuses on the textual information of these transcripts and excludes acoustic features. The ground-truth investment decisions are based on a stock’s performance 30 days post-earnings; they are categorized as Strongly Buy, Buy, Hold, Sell, or Strongly Sell.
Conclusion
STRUX marks a notable step in using LLMs for decision-making. It integrates structured explanations into the decision-making process through a series of reflective steps. STRUX not only leads to higher accuracy but also improves the transparency of LLM decisions, making it a valuable tool for complex decision-making scenarios.
Limitations
STRUX represents a significant advancement in using LLMs for decision-making, particularly in financial contexts. However, it’s crucial to refine its fact extraction capabilities, as inaccuracies in data selection can impact decision quality. Additionally, predicting stock movements is inherently complex and influenced by various external factors like data quality and market nuances. Users are advised to carefully consider these aspects to maximize STRUX’s effectiveness and accuracy in real-world applications. With ongoing enhancements, STRUX has the potential to revolutionize decision-making across diverse sectors.
Figure 1: STRUX’s explanations consist of three components: {supporting facts, a decision, and a brief justification}. Supporting facts can include both positive (green) and negative (red) aspects, along with their strengths.; STRUX’s explanations consist of three components: {supporting facts, a decision, and a brief justification}. Supporting facts can include both positive (green) and negative (red) aspects, along with their strengths.
Figure 2: Each iteration of self-reflection improves the accuracy of decision-making. We show the percentage of training instances that receive correct decisions after each iteration. Our STRUX model is instructed to select from three ranges of supporting facts: 3-6, 6-10, and 10-15. The selection of 6-10 supporting facts consistently yielded the highest accuracy.; Each iteration of self-reflection improves the accuracy of decision-making. We show the percentage of training instances that receive correct decisions after each iteration. Our STRUX model is instructed to select from three ranges of supporting facts: 3-6, 6-10, and 10-15. The selection of 6-10 supporting facts consistently yielded the highest accuracy.
Figure 3: Confusion matrix after each reflection.; Confusion matrix after each reflection.
Figure 4: We input executive speeches from the Prepared Remarks or Q&A sessions into the LLM. Summaries are proportional in input length. Each speech from the Prepared Remarks is summarized into 3-5 key facts, while those from the Q&A session are condensed into 1-3 key facts. Fact tables are generated using gpt-4o-mini-2024-07-18 .; We input executive speeches from the Prepared Remarks or Q&A sessions into the LLM. Summaries are proportional in input length. Each speech from the Prepared Remarks is summarized into 3-5 key facts, while those from the Q&A session are condensed into 1-3 key facts. Fact tables are generated using gpt-4o-mini-2024-07-18 .
Figure 5: STRUX is tasked with predicting a company’s post-earnings stock trend to inform the investment decision. It is set to select the most relevant facts from a provided fact table, ensuring a balanced representation of positive and negative facts affecting the stock price. Each selected fact is evaluated for its potential impact on the stock’s price movement. A “+” symbol indicates a positive impact, with the number of symbols varying from one (+) to three (+++) showing the increasing strength. Conversely, a “-” symbol denotes a negative impact, with one (-) to three (—) symbols reflecting the severity of the negative influence. The system then analyzes all the selected facts to forecast the direction of the stock price movement. The outcomes include: Strongly Buy (SB), Buy (B), Hold (H), Sell (S), or Strongly Sell (SS). It also provides a justification elaborating on its rationale, focusing on the key facts that influence this decision. Additionally, we tested gpt-4o-mini-2024-07-18 and Llama3-8b-Instruct using this prompt by providing either full transcripts or concise fact tables to elicit investment decisions.; STRUX is tasked with predicting a company’s post-earnings stock trend to inform the investment decision. It is set to select the most relevant facts from a provided fact table, ensuring a balanced representation of positive and negative facts affecting the stock price. Each selected fact is evaluated for its potential impact on the stock’s price movement. A “+” symbol indicates a positive impact, with the number of symbols varying from one (+) to three (+++) showing the increasing strength. Conversely, a “-” symbol denotes a negative impact, with one (-) to three (—) symbols reflecting the severity of the negative influence. The system then analyzes all the selected facts to forecast the direction of the stock price movement. The outcomes include: Strongly Buy (SB), Buy (B), Hold (H), Sell (S), or Strongly Sell (SS). It also provides a justification elaborating on its rationale, focusing on the key facts that influence this decision. Additionally, we tested gpt-4o-mini-2024-07-18 and Llama3-8b-Instruct using this prompt by providing either full transcripts or concise fact tables to elicit investment decisions.
Figure 6: We use a series of reflective steps to create training instances without requiring human annotations. This reflection was performed by gpt-4o-mini-2024-07-18 , aiming to help the model learn from its mistakes. When the model makes a poor investment decision, we notify it of the error and prompt it to identify any significant flaws in its fact selection, strength assignment, or reasoning processes. We also provide a list of previous incorrect decisions, including the reasons behind those decisions. Importantly, we ask the model to come up with a different decision from its previous ones without revealing the correct answer . This approach allows us to observe the model’s independent decision-making that emerges from reflection.; We use a series of reflective steps to create training instances without requiring human annotations. This reflection was performed by gpt-4o-mini-2024-07-18 , aiming to help the model learn from its mistakes. When the model makes a poor investment decision, we notify it of the error and prompt it to identify any significant flaws in its fact selection, strength assignment, or reasoning processes. We also provide a list of previous incorrect decisions, including the reasons behind those decisions. Importantly, we ask the model to come up with a different decision from its previous ones without revealing the correct answer . This approach allows us to observe the model’s independent decision-making that emerges from reflection.
Figure 7: An example of an earnings call transcript from Delta Air Lines (DAL) for Q3 2021.; An example of an earnings call transcript from Delta Air Lines (DAL) for Q3 2021.
Our STRUX system outperforms strong benchmarks in making stock investment decisions. We present macro-averaged precision, recall, F-scores, accuracy for the test set. LLMs evaluated are: Llama3-8b-Instruct and gpt-4o-mini-2024-07-18 .
The most common decision paths during reflection and their percentages in the training data.
Statistics of supporting facts.