EconNLI: Evaluating Large Language Models on Economics Reasoning (2024)

Abstract: Large Language Models (LLMs) are widely used for writing economic analysis reports or providing financial advice, but their ability to understand economic knowledge and reason about potential results of specific economic events lacks systematic evaluation. To address this gap, we propose a new dataset, natural language inference on economic events (EconNLI), to evaluate LLMs’ knowledge and reasoning abilities in the economic domain. We evaluate LLMs on (1) their ability to correctly classify whether a premise event will cause a hypothesis event and (2) their ability to generate reasonable events resulting from a given premise. Our experiments reveal that LLMs are not sophisticated in economic reasoning and may generate wrong or hallucinated answers. Our study raises awareness of the limitations of using LLMs for critical decisionmaking involving economic reasoning and analysis. The dataset and codes are available at https://github.com/Irenehere/EconNLI.

heading: 1 Introduction
text: Economics is a social science that studies the behavior and interactions of economic agents such as individuals, firms, and nations (Krugman and Wells, 2013). It has significant social impact as it covers nearly all vital aspects of human life, including social production, distribution, and consumption (Marshall, 2009; Parkes and Wellman, 2015; Mankiw and Scarth, 2010; Mankiw et al., 2020). Studying economics enables people to understand human society better and forecast future trends.
Recently, with the development of Large Language Models (LLMs), many financial institutions and investment firms are using these models to assist in writing economic analysis reports or providing financial advice (Yang et al., 2023b; Zhao et al., 2024; Lee et al., 2024). For example, an LLM may be used to analyze the impact of changes in interest rates on the stock market or to predict the
potential effects of a new government policy on a particular industry. These tasks require LLMs to understand economic knowledge and possess the ability to reason and infer the potential results that may arise from certain economic events or actions. However, there lacks a systematic evaluation of such tasks, and how well the LLMs can understand the economic knowledge and reason for the economic events is unknown.
To fill this research gap, we propose a new task, natural language inference on economic events (EconNLI), to evaluate the LLMs’ knowledge and reasoning ability on economic events. To facilitate this task, we construct an EconNLI dataset that contains sentence pairs of premise and hypothesis, both of which are economic events. Figure 1 provides an example ChatGPT cannot correctly answer. In this example, the premise is the event that the available amount of hard currency per person falls, and the hypothesis is the purchasing power of each unit of currency increases. Different from the traditional NLI datasets (Dagan et al., 2005; Bowman et al., 2015; Williams et al., 2018), the premise does not entail the hypothesis in semantics; instead,
982
it requires understanding the economic theory to conduct the inference. Inference on the example in Figure 1 is based on the quantity theory of money. In the case of a decrease in the available amount of hard currency per person, the money supply in the economy will decrease, leading to a decrease in the price level. As a result, the purchasing power of each unit of currency will increase.
We assess the performance of LLMs on EconNLI from two aspects: classification and generation. As shown in Figure 1, in terms of classification, we provide the premise and the hypothesis to the LLMs and evaluate whether the LLMs can correctly classify the causal relationship between the premise and the hypothesis. For generation evaluation, we ask the LLMs to generate the possible resulting events of the given premise and compare the generated events with the hypothesis of the positive example, which serves as a reference.
We conduct extensive experiments on various language models, from open-source to commercial LLMs, and from the general to the finance-specific domain models. For the classification task, we find that without supervised fine-tuning on the training set, the open-source LLMs perform closely to the random guess, and the advanced commercial LLMs such as ChatGPT and GPT-4 also have unsatisfactory performance. As for the generation task, we find that the LLMs can generate the wrong or hallucinated answers. Our findings suggest that LLMs are not sophisticated in economic reasoning, and there remains room for improving the reliability of LLMs in financial and economic analysis.
In conclusion, this paper presents a novel dataset designed to evaluate the reasoning ability of LLMs in the economic domain. The results of our experiments suggest that these models are far from perfect in their ability to economic reasoning. Given that LLMs are now widely applied in writing financial reports or providing financial advice without carefully evaluating their correctness, our study shows LLMs can sometimes lead to errors or unreliable results in economic analysis. Therefore, it is essential to be aware of the limitations of these models when using them for decision-making.

heading: 2 Related Works
text: NLP for Finance and Economics. The development of LLMs has revolutionized the solutions for many natural language processing (NLP) tasks (OpenAI, 2023; Touvron et al., 2023a,b),
with no exception in the financial domain (Guo et al., 2023a; Lee et al., 2024; Liu et al., 2024). Many domain-specific models have been developed to provide solutions to financial tasks. BloombergGPT (Wu et al., 2023) is the first LLM trained on vast financial data. Later, FinMA (Xie et al., 2023) adapts LLAMA (Touvron et al., 2023a) with instruction-tuning on the various financial NLP tasks. FinGPT (Yang et al., 2023a) is a series of models developed to solve some specific financial tasks, such as sentiment analysis or financial relation extraction. InvestLM (Yang et al., 2023b) is instruction-tuned to provide investment suggestions specifically.
Meanwhile, some works evaluate these LLMs’ capability on the financial tasks. Callanan et al. (2023) find the ChatGPT and GPT-4 struggle in answering the mock exam questions of the Chartered Financial Analyst (CFA) Program. Cheng et al. (2023) show GPT-4 performs comparably to human experts as a data analyst. Guo et al. (2023b) shows while some LLMs demonstrate notable performance on some financial NLP tasks, they generally lag behind the fine-tuned expert models. Besides, several datasets are developed to evaluate LLMs in various financial NLP tasks, such as sentiment analysis (Malo et al., 2014; FiQA), text classification (Sinha and Khandait, 2020; Shah et al., 2023) or question answering (Chen et al., 2021, 2022). However, none of them involve assessing the reasoning ability of LLMs in the financial and economic domain. To our knowledge, we are the first to assess LLMs from the aspect of reasoning on the relations between the economic events.
Natural Language Inference. Natural language inference (NLI), which aims to reason the entailment relationship between text fragments, has long been a key task in natural language processing (Yu et al., 2023). Various natural language inference tasks are proposed to address the inference ability from different aspects, such as RTE (Dagan et al., 2005), SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2018), SCiTail (Khot et al., 2018), Abductive NLI (Bhagavatula et al., 2019), defeasible NLI (Rudinger et al., 2020), SciNLI (Sadat and Caragea, 2022). However, none of them address the inference in economics and finance. Unlike traditional NLI tasks that rely on linguistic or common sense knowledge for reasoning, EconNLI demands an understanding of economic theories to perform event reasoning.

heading: 3 Natural Language Inference on Economic Events
text: We study the problem of causal relation inference on the economic events in the natural language form. In our study, we define an event as somebody or something having some actions or states. Economic events are the events that are related to the economic subject or phenomenon, for example, events describing production, distribution, and consumption (Krugman and Wells, 2013).
We construct a dataset, EconNLI, to evaluate LLMs’ causal relation inference ability on economic events. It consists of sentence pairs of (premise, hypothesis), both of which represent economic events. In positive pairs, the occurrence of the premise event leads to the occurrence of the hypothesis event based on economic theories. For negative pairs, no such causal relationship is supported by any theory. Table 1 provides five random positive examples from the EconNLI test set, illustrating when the premise can cause the hypothesis.
Note that in the real world, multiple premises may cause the hypothesis. For example, premise1 and premise2 may be the joint causes of the hypothesis. However, in our benchmarking, we only consider whether premise1 causes the hypothesis and premise2 causes the hypothesis
separately. If premise1 and premise2 are joint causes of the hypothesis, we exclude this example in our dataset.

heading: 4 EconNLI Dataset Construction
text: In this section, we describe the construction of EconNLI. The outline of the construction framework is shown in Figure 2. We first prepare a corpus about the economics from Wikipedia, and extract the events from the corpus using a supervised finetuned LLAMA2 model. Next, we generate candidates for premises and hypotheses based on a given reference sentence and assign a label. To ensure the dataset’s scale and quality, we use ChatGPT and GPT-4 to assign labels to the training set and annotate the testing set with manual inspection to align with human domain knowledge.

heading: 4.1 Corpus Preparation
text: To build a domain-specific corpus, we collected all pages related to economics and finance from Wikipedia, resulting in a corpus of 862 articles. To identify sentences containing causal relationships, we follow Sadat and Caragea (2022) and extract sentences with causal linking phrases. The linking phrases are provided in Appendix A. In total, we identify 5,077 sentences to construct the dataset.

heading: 4.2 Event Extraction
text: As our goal is to conduct inference among economic events, a crucial step involves extracting meaningful events from the source sentence. Directly splitting the sentence into two segments by the linking phrase can cause several issues: first, a sentence segment may not accurately express an event or contain multiple events; second, it can lead to potential grammar mistakes or unclear pronoun references. These problems can create confusion in the subsequent reasoning task.
To address this issue, we propose summarizing events with LLMs from the source sentence rather than generating events through syntax patterns. To ensure the LLM can summarize events as required, we manually write down all possible events for 125 sentences and perform supervised fine-tuning (SFT) on an LLAMA2 model for event generation. We use the following prompt for SFT: ### Question: An event is defined as somebody/something having some actions/states. List all events that can be inferred from the given sentence. If no event exists, simply respond "No event." The sentence:{source sentence} \n ### Answer: {human label}. After the SFT, we apply the fine-tuned LLAMA2 model to the corpus prepared in 4.1 and extract all possible events from every sentence in the corpus.

heading: 4.3 Positive Pairs Construction
text: After summarizing the events from the sentences, we construct positive pairs in EconNLI.

heading: 4.3.1 Candidates Generation
text: To create positive pairs in EconNLI, we first generate candidates for premises and hypotheses based on syntax patterns. This step makes creating prompts for asking ChatGPT and GPT-4 to generate training set labels easier in the next stage.
Given a sentence containing a causal linking phrase (e.g., s1 because s2), we split the sentence into two segments by the linking phrase (because) and denote the first sentence segment as s1, and the second segment as s2. We assume the premise event is contained in s2 and the hypothesis event is in s1. Under this assumption, for an event e summarized by LLAMA2, we compute the cosine similarity between the bag-of-words representation of event −→ve and the two segments −→vs1,−→vs2 respectively. If similarity(−→ve ,−→vs1) < similarity(−→ve ,−→vs2), we consider s2 to include the event e, and regard e as a premise candidate; otherwise, we regard e as a hypothesis candidate.
Likewise, for a sentence containing a linking phrase of effects (e.g. s1 resulting in s2), If similarity(−→ve ,−→vs1) < similarity(−→ve ,−→vp2), we assign the event e as a premise candidate; otherwise, we assign it as a hypothesis candidates.
Using this approach, after processing all events in a source sentence, we generate a (source sentence, premise candidates, hypothesis candidates) tuple for each source sentence, where the premise and hypothesis candidates are sets of events, respectively. Next, we assign the labels.

heading: 4.3.2 Assign Labels
text: To create the training set, for each possible (source sentence, premise, hypothesis) tuple derived from the (source sentence, premise candidates, hypothesis candidates), we ask ChatGPT and GPT-4 to assign a label. Using the prompt Given the sentence {source sentence}, can we infer that {premise} is a cause of {hypothesis}? Answer Yes or No, we ask ChatGPT and GPT-4 to classify whether the event in the premise can cause the event in the hypothesis based on the source sentence. By providing the reference sentence, we enhance the reliability of LLMs’ answers and reduce the chance of generating a hallucinated answer.
Creating the testing set requires human expertise in economics and finance. Therefore, instead of using crowd-sourced workers, the authors with relevant backgrounds manually annotate the testing set. We select sentences from the training set that ChatGPT identifies as positive and assess if the premise leads to the hypothesis based on relevant economic theories. We manually annotate 513 positive examples and then exclude them from the training set.

heading: 4.4 Negative Pairs Construction
text: We also include negative pairs in EconNLI, where the premise is not a valid cause of the hypothesis. We create negative hypotheses by sampling from all possible hypotheses. For each premise from the examples that ChatGPT identified as positive, we first retrieve 20 hypotheses with the highest cosine similarity to the target premise as candidates, where the similarity is based on the BERT embeddings. We then exclude candidates from the same Wikipedia page as the premise to avoid false negatives. Finally, we randomly sample one hypothesis from the candidates as the negative hypothesis.
To improve the dataset quality, we use ChatGPT and GPT-4 to classify the relation between the sampled premise and hypothesis using the prompt Can we infer that {premise} is a cause of {hypothesis}? Answer Yes or No on the training set. We only preserve the examples where either ChatGPT or GPT-4 answers no. Authors manually check all negative pairs for the testing set to ensure validity.

heading: 4.5 Dataset Statistics
text: Table 2 presents the descriptive statistics of the EconNLI dataset. The dataset is divided into two sets: the training set and the test set. The training
set includes 10,810 examples, covering 554 different economic topics (Wikipedia pages), with an average of 17.84 words per (premise, hypothesis) example. We provide two labels for each example in the training set, one from ChatGPT and the other from GPT-4. The Cohen’s kappa coefficient between ChatGPT and GPT-4 labels is 0.731.
The test set contains 1,026 examples covering 211 economic topics, with an average of 19.54 words per example. We also compare EconNLI with other popular financial NLP datasets in Appendix B.

heading: 5 EconNLI Evaluation
text: In this section, we use EconNLI to benchmark the LLMs’ performance on economic event reasoning. We examine the language models’ capabilities in inferring the correct relations between the event pairs and the correctness in generating the possible resulting events of a given premise.

heading: 5.1 Evaluated Language Models
text: Our experiment includes a wide range of language models, from the encoder-only to the decoder-only architecture. Specifically, we examine the following LLMs:
Encoder-only Language Models. For general domain LLMs, we consider (1) BERT (Devlin et al., 2019) and (2) RoBERTa (Liu et al., 2019): two popular language models for sequence classifications. Besides, we also evaluate the LLMs of the financial domain, including (1) FinBERT (Yang et al., 2020), a domain-specific model trained on financial text data. (4) FLANG-BERT and FLANG-ELECTRA (Shah et al., 2022) which use finance domainspecific pre-training with preferential masking to build more robust representations for the domain.
Decoder-only Language Models. We also evaluate the performance of various popular decoderonly language models: (1) LLAMA2 (Touvron et al., 2023b) is a popular open-source LLM pretrained on extensive online data, and we use the LLAMA2-Chat version, which is optimized for dialogue use cases. (2) Alpaca (Taori et al., 2023) is fine-tuned from a LLAMA-7B model (Touvron
et al., 2023a) on instruction-following data generated by the the technique of Self-Instruct (Wang et al., 2023). (3) FINMA (Xie et al., 2023) is a financial LLM based on fine-tuning LLAMA (Touvron et al., 2023a) with instruction data. (4) ChatGPT (Ouyang et al., 2022) and GPT-4 (OpenAI, 2023) are two advanced LLMs pre-trained on a wide array of textual data and reinforced by human feedback.

heading: 5.2 EconNLI for Classification Evaluation
text: In this subsection, we use EconNLI as a classification benchmark to evaluate the LLM’s capability to reason about the cause-effect relations between economic events. Specifically, given a premise p and a hypothesis h, the LLM performs as a classifier f : (p, h) → {0, 1}, where the label 1 represents the happening of p can result in the happening of h, and label 0 otherwise.

heading: 5.2.1 Experiment Setup
text: There are two main techniques for utilizing LLMs for sequence classification: one is supervised finetuning, typically for encoder-only models or some relatively small decoder-only language models, and the other is zero-shot or few-shot prompting, typically for large decoder-only models. We experiment with the classification of EconNLI with these two techniques in the following setup:
Supervised fine-tuning. For fine-tuning the encoder-only LLMs, we use the BERT (base,uncased), RoBERTa (base), FinBERT (pretrain), FLANG-BERT and FLANG-ELECTRA from Huggingface1, and the model fine-tuning is implemented via Trainer 2. Besides, we supervised fine-tuning (SFT) the LLAMA2-chat (7B, 13B) with the Huggingface SFT Trainer 3 using LoRa (Hu et al., 2022). The hyperparameters setting is
1https://huggingface.co/ 2https://huggingface.co/docs/transformers/
main_classes/trainer 3https://huggingface.co/docs/trl/sft_trainer
given in Appendix C. The prompt we used for SFT is provided in Appendix D.1. For all models, We randomly select 10% examples from the training set as the validation set and fine-tune the model for three epochs, and the model with the best validation performance is selected for evaluation. We repeat the fine-tuning experiments three times for each LLM, and report the average performance along with the standard deviation on the test set.
Prompting the decoder-only LLMs. We experiment with three types of prompting: zero-shot prompting, in-context learning (ICL) prompting, and chain-of-thought (CoT) prompting (Wei et al., 2022). The prompts for zero-shot setting, ICL, and CoT are provided in Appendix D.2, D.3 and D.4 respectively. The demonstrations in ICL, containing one positive example and one negative example, are randomly sampled from the training set. For ChatGPT and GPT-4, We use the "gpt-3.5-turbo" and "gpt-4" model API from OpenAI, respectively, retrieved in January 2024. Other open-source models are experimented on huggingface, and the answer generation is based on the greedy search.

heading: 5.2.2 Results
text: Table 3 presents the classification results of the language models after supervised fine-tuning. Despite being fine-tuned on 10,810 training data, encoderonly language models - from general to domainspecific - failed to perform well on EconNLI classification. The best encoder-only model, BERT, scored less than 0.8 F1 on this binary classification task. However, supervised fine-tuning on LLAMA2-chat models, which have a larger number of parameters, significantly improved the results from encoder-only models, as the 7B and 13B LLAMA2 models achieved similar results with around 0.87 F1 score. Nonetheless, the language models’ economic reasoning ability still has room for improvement.
Table 4 displays the classification results of the
language models with various prompting methods. The open-source models, with parameters ranging from 7B to 13B, all demonstrate performance similar to random guesses. The open-source model with the best performance is FINMA, indicating that tuning on financial instructions improves the model’s capability in economic event reasoning. ChatGPT and GPT-4 significantly surpass the opensource models, possibly due to their larger model size, richer pre-training corpora, and the knowledge gained from human feedback. Nevertheless, the results of both ChatGPT and GPT-4 still notably lag behind the LLAMA2 model after supervised finetuning, suggesting potential improvement from acquiring more related knowledge.
Upon comparing various prompting strategies, we have observed that different strategies have varying effects on open-source and GPT-series models. With ICL prompting, the open-source models consistently predict the positive label, which is undesired as it does not provide valid information. On the other hand, CoT prompting improves the results compared to zero-shot prompting, indicating that prompting the model to reason step by step can enhance reasoning accuracy. For the GPT-series model, ICL prompting hinders the performance of ChatGPT but improves the performance in GPT-4. In contrast, CoT prompting hindered the performance of both models, likely because these LLMs with reinforcement learning feedback can under-
stand the intention in zero-shot prompts, and providing extra reasoning steps is unnecessary.

heading: 5.3 EconNLI for Generation Evaluation
text: EconNLI is also an ideal dataset for evaluating the generation quality of the LLMs, as it provides sentence pairs describing the economic events and their consequential effects. Therefore, it can be used to assess whether the LLMs can generate the possible resulting event provided a premise event. As the LLMs are usually used to provide advice for economic analysis, correctly inferring the consequences of specific economic phenomena is crucial for providing reliable advice and solutions in the economic domain.

heading: 5.3.1 Experiment Setup
text: To evaluate the LLM’s generation quality, we provide an event from the EconNLI premises of the positive examples and ask LLM to generate the possible resulting event with the following prompt: If the event {premise} happens, what event will be caused? Answer the possible resulting event very briefly in one sentence. answer:.
As the real-world economic system is complicated, one economic event can have multiple consequences. As a result, the LLM’s answer may be an alternative event that is also correct but not included in our reference. Therefore, we avoid using the BLUE (Papineni et al., 2002) or ROUGE (Lin,
Premise Hypothesis(Reference) LLM’s Answer Explaination
The subordinate bonds are allocated any losses from the collateral before losses are allocated to the senior bonds The senior bonds are given a credit enhancement
ChatGPT: The resulting event would be a higher probability of default for the senior bonds. If losses occur, they are first allocated to the subordinate bonds, resulting in a lower probability of default for the senior bonds
2004) metrics that measure the similarity between the generated sequence and the reference sentence. Instead, we use GPT-4 to evaluate the relations between the LLM’s answer and our hypothesis (reference). We ask GPT-4 to classify the relations into entailment, contradiction, and irrelevant. The entailment class stands that the LLM’s answer and our reference describe similar economic phenomena, while the contradiction class means they describe the opposite economic phenomena. The irrelevant class means that the LLM provides an event that is different or unrelated to our reference, implying the correctness of the LLM answer is unknown. This setup considers the complications of the economic system, where the irrelevant answer may also be a
possible resulting event. The prompt we used on GPT-4 to verify the LLM’s answer is provided in Appendix D.5.

heading: 5.3.2 Results
text: Table 6 presents the quality of generating the consequent economic events given a premise. The results show that GPT-4 had the best generation quality, with the lowest contradiction/entailment rate of 4.71%. LLAMA2-chat (7B) and LLAMA2chat (13B) had similar rates of 8.72% and 8.02%, respectively, while Alpaca had a slightly higher rate of 9.21%. The model FINMA had the highest rate of contradiction/entailment at 70.73%, probably because it is instruction-tuned on the classification datasets, which impedes its ability to generate fluent and relevant results. The experiment results suggest that all generative language models have the probability of generating incorrect answers when analyzing the potential consequences of economic events. While some models performed better than others, none of them achieved perfect performance. Therefore, it is important for people to carefully check the correctness of the generated content when using LLMs to provide financial advice for economic analysis. This will help avoid misunderstandings and errors in decision-making and ensure the generated content is reliable and accurate.

heading: 5.3.3 Case Study
text: Table 5 presents a case study evaluating the performance of different LLMs in generating possible resulting events based on given premises. Table 5 identifies five cases where the LLMs generate wrong or hallucinated answers, providing an intuitive understanding of how LLMs fail the economics reasoning. The case study demonstrates the potential limitations of using LLMs for generating content related to economic events. While LLMs can help generate hypotheses and explore potential outcomes, it is important to recognize their limitations and potential for generating inaccurate or misleading results. As such, it is recommended that LLMs be used in conjunction with human expertise and judgment and that the generated content be carefully evaluated and verified before being used for decision-making.

heading: 6 Conclusions
text: In conclusion, we propose a new task to evaluate (financial) LLMs’ ability to reason about economics. To do so, we construct a new dataset, EconNLI, and apply it to benchmark LLM’s ability to perform classification and generation tasks. Through experiments, we find that LLMs have limitations in understanding economic knowledge and reasoning about the potential results of specific economic events. This work provides a new benchmark and insights for constructing financial LLMs. By emphasizing the need for continued research in this area, we hope to inspire future studies that will further enhance the reasoning capabilities of LLMs in the financial domain.

heading: 7 Limitations
text: There are two main limitations to this work that should be considered. Firstly, the focus of the study is on the ability of LLMs to reason about economics. The study does not address the ability of LLMs to reason about other domain-specific tasks, such as legal or clinical reasoning. Therefore, the findings of this study should only be generalized to other domains with further evaluation.
Secondly, the dataset was constructed based on Wikipedia, which may not fully represent the complexity and variability of real financial reports. The reasoning scenarios in writing a financial report may have unique and important problems that are not captured in the dataset. Future work should explore using more diverse and representative cor-
pora to provide a more comprehensive evaluation of LLMs’ performance in the financial domain.

heading: 8 Acknowledgement
text: We acknowledge Jiaxin Liu and Hanyu Duan for providing helpful feedback on labeling the EconNLI dataset.

heading: A Linking Phrase
text: Linking phrase of causes: because, because of, owing to, due to, caused by
Linking phrase of effects: therefore, hence, thus, as a result, as a consequence, consequently

heading: B Comparisons with Other Financial NLP Datasets
text: Table 7 compares EconNLI with other well-known datasets for financial NLP. As a domain-specific dataset, EconNLI stands out due to its relatively large scale compared to other financial NLP datasets. Additionally, EconNLI is unique in that it addresses the problem of reasoning in economics, which has not been previously explored in other datasets. This makes EconNLI a valuable resource for researchers and practitioners interested in developing and evaluating large language models for economic reasoning.

heading: C Hyper-parameters for SFT
text: SFT on BERT, RoBERTa, FinBERT, FLANGBERT: We fix the learning rate as 2×10−5, weight decay as 0.01, and the batch size as 48. Other hyperparameters remain the default in Trainer.
SFT on LLAMA2: We set the hyperparameters r = 16, α = 32 in LoRa, and train the model with 5× 10−5 learning rate and 24 batch size for three epochs.
All experiments are run on a machine with four Nvidia 3090 GPUs.

heading: D Prompts in Experiments
text: D.1 SFT prompting ### Question: Conduct inference on economic events. We provide a premise and a hypothesis, both of them are economical events. Infer whether the premise can cause the happening of the hypothesis. Only answer ’Yes’ or ’No’. premise: {premise}, hypothesis: {hypothesis}.
### Answer: {answer} + tokenizer.eos_token Note that the loss is only calculated on the com-
pletion after the term "### Answer:".
D.2 Zero-shot Prompting Conduct inference on economic events. We provide a premise and a hypothesis, both of them are economical events. Infer whether the premise can cause the happening of the hypothesis. Only answer ’Yes’ or ’No’. Premise: {provided premise}, hypothesis: {provided hypothesis}, answer:
D.3 ICL Prompting Conduct inference on economic events. We provide a premise and a hypothesis, both of them are economical events. Infer whether the premise can cause the happening of the hypothesis. Only answer ’Yes’ or ’No’. Here are some examples: premise:{premise of the positive example}, hypothesis: {hypothesis of the positive example}, answer:Yes
premise: {premise of the negative example}, hypothesis: {hypothesis of the negative example}, answer:No
Conduct inference on the following premise and hypothesis: premise: {premise for prediction}, hypothesis: {hypothesis for prediction}, answer:
D.4 COT Prompting ### Question: Conduct inference on economic events. We provide a premise and a hypothesis, both of them are economic events. Infer whether the premise can cause the hypothesis to happen. Write the reasoning chain on the first line, and summarize the answer as ’Yes’ or ’No’ in the second line. premise: demand increases, hypothesis: price increases.
### Answer: When demand for a product or service increases, more people want to buy it. This creates a situation where there are more buyers than available supply, which leads to an increase in competition among buyers. As a result, sellers can raise their prices because they know that buyers are willing to pay more to get the product or service they want. \n Yes. \n
### Question: Conduct inference on economic events. We provide a premise and a hypothesis, both of them are economic events. Infer whether the premise can cause the hypothesis to happen. Write the reasoning chain on the first line, and summarize the answer as ’Yes’ or ’No’ in the second line. premise: government borrowing cre-
ates higher demand for credit in the financial markets,hypothesis: interest rates decreases across the market.
### Answer: When the government borrows money, it creates higher demand for credit in the financial markets. This is because the government is competing with other borrowers for available funds, which can drive up interest rates. Therefore, it is unlikely that government borrowing would cause interest rates to decrease across the market. \n No. \n
### Question: Conduct inference on economic events. We provide a premise and a hypothesis, both of them are economic events. Infer whether the premise can cause the hypothesis to happen. Write the reasoning chain on the first line, and summarize the answer as ’Yes’ or ’No’ in the second line. premise:{premise}, hypothesis: {hypothesis}
### Answer:
D.5 Prompt for Generation Quality Evaluation
The prompt we used on GPT-4 to verify the LLM’s answer is: I will give you two economical events. Determine the connection between these two events, choosing from ’entailment’, ’contradiction’, and ’irrelevant’. ’Entailment’ means the two events describe similar economic phenomena, ’contradiction’ means the two events describe the opposite economic phenomena, ’irrelevant’ means the two events are unrelated or do not belong to one of the above two classes. Only provide the label.Event 1: {LLM answer}, event 2:{hypothesis}

heading: Future Work
text: Future work should explore using more diverse and representative cor-
pora to provide a more comprehensive evaluation of LLMs’ performance in the financial domain