Reasoning and Tools for Forecasting

Authors: elvis  hsieh
preston  fu
jonathan  chen
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-21 00:51:15

Abstract: Language models (LMs) trained on web-scale datasets are largely successful due to their ability to memorize large amounts of training data, even if only present in a few examples. These capabilities are often desirable in evaluation on tasks such as question answering but raise questions about whether these models can exhibit genuine reasoning or succeed only at mimicking patterns from the training data. This distinction is particularly salient in forecasting tasks, where the answer is not present in the training data, and the model must reason to make logical deductions. We present Reasoning and Tools for Forecasting (RTF), a framework of reasoning-and-acting (ReAct) agents that can dynamically retrieve updated information and run numerical simulation with equipped tools. We evaluate our model with questions from competitive forecasting platforms and demonstrate that our method is competitive with and can outperform human predictions. This suggests that LMs, with the right tools, can indeed think and adapt like humans, offering valuable insights for real-world decision-making.

Main paper: 
 introduction: 
 Forecasting is an essential tool today, playing a critical role in government, corporate, and personal decision-making. Weather forecasting provides essential information for agriculture, natural disaster preparedness for governments, and travel plans for individuals. During the COVID-19 pandemic, lockdown policies were largely determined by forecasts, which were required to be sufficiently accurate due to their global impact [9] . Forecasting methodologies fall into two main categories [23] : statistical and judgmental. Statistical forecasting leverages time-series modeling and excels with abundant data under stable conditions. Conversely, judgmental forecasting, which we refer to simply as "forecasting," typically relies on human expertise, integrating historical data, domain knowledge, and intuition to make predictions, and is particularly useful when data are sparse or conditions are volatile. By nature, forecasting requires not only accuracy but also the ability to continuously adapt to dynamic data streams. This is where traditional LMs often struggle: timely data updates may cause predictions to change considerably and past data to be irrelevant. 
related work: 
 Information retrieval Reliable and accurate predictions are largely dependent on the information available to the predictor. This is especially the case of LMs, which are trained on data preceding a knowledge cutoff and have been shown to perform better with information retrieval [20] .Language models model the likelihood p θ (y i |x, y <i ) for input sequences x and target sequences y. Retrieval-augmented generation (RAG) [12] proposes augmenting this approach with non-parametric memory, i.e. retrieving the top-k text documents z via p η (z|x) and conditioning the generator on Figure 1 : RTF: High-level agent oversees lowlevel agents, each equipped with distinct toolkits and data/document to accomplish various tasks, including API calling and Python simulation. the retrieved passages, p θ (y i |x, z, y <i ). In a forecasting context, RAG enables us to search for relevant documents z that may contain timely information about the forecasting task x not present in the training data. Therefore, we propose to equipping our forecasting agent with tools to retrieve up-to-date information beyond its knowledge cutoff.Prior approaches to LLM forecasting LMs have shown strong, interpretable reasoning capabilities [3, 2, 1], and have seen adoption across a wide variety of domains, even in cognitively demanding tasks. LMs have shown success across a large number of benchmarks, but previous work has shown that LMs predictly emit memorized training data [6, 8] . As a result, it seems plausible that success on these benchmarks merely corresponds to memorization. On the other hand, forecasting has proven to be a difficult task for LLMs. For example, Zou et al. [27] proposed a benchmark to use neural networks to automate prediction in prediction markets; the authors found that while language models can be trained to improve their performance on forecasting tasks, their accuracy remains significantly below those of human experts.Current methods aim to improve the accuracy of LLM forecasting by fine-tuning and scratchpad prompting [16, 10, 24] or ensembling [5, 18] in an attempt to approach human-level forecasting. However, fine-tuning requires extensive computational resources and dataset preparation, resulting in a costly process that limits scalability, especially for specialized applications such as forecasting.Concurrent work [17] benchmarks LLMs' forecasting capabilities using the GleanGen prediction market, an internal tool at Google. However, this approach did not accurately reflect real human crowd prediction distributions, and it relied on PaLM2 [4] , which performs worse than GPT models. We propose a zero-shot tool-usage LLM framework without costly fine-tuning and tedious scratchpadformat prompting.Ensembles Leveraging multiple LLM agents has demonstrated strong performance on a variety of tasks, and improve performance beyond that of a single agent [22, 13] . Schoenegger et al. [18] has shown that taking ensemble sizes up to 36 outperforms any individual forecasting agent. Recent work in tool learning has implemented task planning and execution with separate agents [21, 19] . We propose bridging this gap with a hierarchical structure to facilitate cooperation between highlevel reasoning and low-level execution agents, and demonstrate that a small ensemble suffices for human-level performance without costly fine-tuning and tedious scratchpad format prompting. 
reasoning and tools for forecasting: 
 Forecasting is a complex task solving environment, for which we would like to where we leverage a frozen LM p θ as reasoning. Successful forecasting agents rely on the most up-to-date information, and accordingly operate as agents that collect observations o t ∈ O and take actions a t ∈ A.The observation space O is natural language, as collected from the prompt itself or information on the internet. The agent's actions are distributed according to a t ∼ π(a t |c t ), where c t = (o 1 , a 1 , . . . , o t-1 , a t-1 ) is the context to the agents.Our proposed approach π satisfies the following criteria:• It is simple, scalable, and time-invariant. As we consider different datasets of forecasting questions or language models at least as capable as the current state-of-the-art, we would like our approach to work at least as well.• It can produce comprehensive responses through zero-shot prompting from factual information, which can be used to reliably support decision-making in downstream scenarios.• These responses should be consistent, i.e. they should correctly synthesize the up-to-date information the model collects.It's shown that CoT prompting, even with in-context examples, can iteratively hallucinate to produce incorrect responses on complex tasks [25] . CoT satisfies (i) but neither (ii) nor (iii). We find that CoT's lack of interaction with the environment (i.e. sole reliance on its training data) limits its reasoning abilities and over-emphasizes irrelevant information. Yao et al. [25] proposes ReAct for this setting: A = {search, lookup, finish}, and observations o t from search and lookup are collected from O ⊆ Wikipedia web API. The context is then augmented a thought ât ∼ p θ (â t |c t ) that composes information about the existing context. This method has shown to significantly enhance the model's ability to refine its responses continuously, reducing the likelihood of erroneous outputs due to lacking critical context information. Vanilla ReAct satisfies (i); as part of our framework, we show that it can additionally satisfy (ii) and (iii).We define π by an aggregate of a collection of hierarchical ReAct agents with tools for real-time data retrieval and simulation, expanding π's observations o t collected from O ⊆ Google Search API and Python interpreter.We propose hierarchical ReAct planning, where a LM agent acts as a high-level planner for handling abstract logic and forecasting principles based on the outputs collected from the low-level agents (Figure 1 ). When LLMs handle API directly with individual agents, it can consume a large portion of the context window. We delegate the reasoning and API calling to specialized agents to enhances efficiency, conserves tokens, and allows for more complex operations. The high-level agent interacts with the low-level agent by invoking it as a tool. We wrap API tools with another ReAct agent to form the low-level agent, which significantly increases API call success rates due to its self-correction mechanism [25] . Both classes of agents are implemented with GPT-4o backbones. 
setup: 
 Models and data Jin et al. [11] , Zou et al. [26] have proposed forecasting benchmarks to assess models' forecasting abilities, simulating forecasting by leveraging that models are only trained up to a cutoff date. However, these benchmarks, consisting of questions that resolved in 2022, are now outdated for evaluating the performance of models such as GPT-4o due to answer leakage in training data (knowledge cutoff October 2023; see Appendix A.1).We curated the dataset on April 15, 2024, when we scraped the platform for questions resolving within the next two weeks and corresponding human crowd predictions. We then filtered out vague questions, and ran every prediction method on these questions, enabling a fair comparison between each method and the human crowd. To prevent answer leakage from the Google API, we set the search range to prior to this date. Our final dataset consisted of 201 questions spanning across 9 diverse categories (see Apendiex B).None of our baselines have direct access to prediction market data, and empirically we found that this information was never scraped via Google search. That is, the prediction given by the ensemble of agents relies on only the agents themselves, with no human crowd influence. (By contrast, if deployed in the real world, this approach could benefit from incorporating the current human crowd performance as an input to the prediction due to the wisdom-of-crowds effect. Indeed, we observe in our experiments that human crowds are fairly well-calibrated.)Performance metrics Our n forecasting questions have true outcomes o i ∈ {0, 1} and probabilistic forecasts f i ∈ [0, 1]. We evaluate our forecasts using Brier scores [7] , i.e.1 n n i=1 (f i -o i ) 2, and accuracy, i.e. 1 n n i=1 1{1{f i > 0.5} = o i }. 2 3 In case LMs decline to give numerical answers, the question is dropped over all methods when evaluating scores.Baselines In Table 1 , we compare RTF ensemble to multiple baselines: (a) crowd scores given by the current traded values on Manifold Markets (see Appendix A.2), (b) scratchpad prompting, ensemble, and fine-tuning [10] , and (c) base models from different providers.Table 1 demonstrates that RTF significantly improves over CoT and scratchpad with fine-tuning. We also achieve comparable Brier score (0.169 vs. 0.172) and superior accuracy (73.9% vs. 73.8%) compared to human predictors using the median and mean of our ensemble, respectively.We also demonstrate that ensembles for RTF yield better performance than individual agents (Brier 0.169 vs. 0.180). However, this is not the case for base LMs (Brier 0.218 vs. 0.210 for GPT-4o). Base LMs tend to produce higher-variance outputs (standard deviation in ensemble size 4 of 0.150) compared to our better-calibrated ReAct agents (standard deviation in ensemble size 3 of 0.092), which satisfied (iii) as defined in Section 3.Ensembles only contribute to the final performance if each ensemble member is already sufficiently calibrated. Indeed, Brier scores given by randomly sampling our ReAct ensemble outputs, "React Sampled" in the table, achieved a score of 0.180, far better than was achieved by any of the base methods (which, aside from GPT-4o, perform worse than guessing 0.5 every time by Brier score).Ablation study To demonstrate the effectiveness of our introduced components, we conduct the ablation study. We showed each component is necessary for the fully functioning RTF framework.• ReAct: RTF itself without adequate guidance from ReAct struggles to properly use the tools provided by our low-level agents, which leads to misguided lines of reasoning that cascade downstream. This is consistent with the observation (B) in [25] , where groundedness and trustworthiness come at the cost of higher reasoning error rates. • Hierarchical Planning: Empirically, without the cooperation of high-and low-level agents, a single agent fails to call APIs and perform necessary reasoning, as it exhausted available tokens on API schemas. In our experiments, the single-agent approach frequently encountered time-out errors or exceeded rate limits when handling complex queries.Qualitative analysis While the baselines systematically evaluate multiple considerations, they do not consider interactions between these considerations. Empirically, we find in our samples that the prompting style we present is useful in generating a wide variety of arguments and providing reasonable estimates for how to weight each of those arguments. On the other hand, we see that this same prompt GPT-4o directly does this calibration in a sequential manner to update its final estimate, which may result in over-or under-estimate based on the recency of its considerations. In general, we find that RTF yield human-like reasoning trajectories, showing the robustness of interactive decision making, supporting goal (ii) from Section 3 (see Appendix D). predicts an event with a certain probability, the event should occur approximately that fraction of the time over many predictions.We calculate the calibration index asCI = 1 N K k=1 N k (f k -o k ) 2, where N is the total number of forecasts, N k is the number of forecasts in bin k, f k is the mean forecast probability in bin k, and o k is the observed probability with which events occur in bin k. We select bins as the K-quantiles of the forecasts.Comparing GPT-4o and React Mean, we see a significant decrease in calibration index (0.0194 vs. 0.0129), which shows that ensembling with ReAct not only increases forecasting accuracy, but also more accurately measures the specific magnitudes with which events occur. 
conclusion: 
 We present Reasoning and Tools for Forecasting, a framework to leverage LMs' reasoning capabilities by interacting with the latest information. It is competitive with the predictive capabilities of human forecasters on forecasting platforms. The RTF synthesizes information through a structured decisionmaking process, ensuring that the predictions are both current and relevant. Additionally, while previous work has shown that ensembling can improve prediction accuracy, a carefully calibrated smaller set of models is often more cost-effective than larger ensembles. By advancing LMs' abilities to reason and dynamically interact with new data, RTF offers a robust tool for real-world decisionmaking for tasks like forecasting.Limitations The evaluation dataset is based on prediction market data and popular questions rather than domain-specific questions. This facilitates a comparison with crowd prediction performance, but may not fully capture the nuances of more specialized domains. In addition, the Google Search API retrieves a list of URL links, titles, and text snippets, but lacks nuanced, context-specific understanding. As we can observe qualitatively (see Appendix D.2), interpreting incoherent observations from the API can be challenging. 
 
 Appendix: A.1 Models We list the details of models we evaluated in Table 3 , where the cutoffs have been retrieved from the model cards. For GPT models, we run them via the OpenAI API. We host Llama-3-7B on a single NVIDIA TITAN RTX 24GB via Ollama for roughly 0.5 GPU-hours. All other approaches are run through the OpenAI API, for roughly 1 hour per naive baseline, 6 hours for our reproduction of [10] , and 2.5 hours for our proposed method. For GPT models, we use temperature 0.5 for all the experiments.[10] finds that GPT-3.5 and GPT-4 do not have leakage due to post-training. We find that the same is true of GPT-4o and Llama-3-7B: prompting with "Answer this question without searching the web: Who was appointed to the Governor-General of Australia in 2024?" yielded a statement about its cutoff date, whereas the correct answer was given when prompted for the answer in 2019.On Manifold Markets, players make bets on the outcome of various events where the prices of bets are determined by a current aggregate of crowd predictions, which are prices in [0, 1]. As bets are made, the prices are adjusted by their automated market-makers [14] . As shown in [15] , the crowd prediction is a strong baseline and consistently outperform top forecaster in the prediction market.Our final dataset consisted of 201 questions from Manifold Markets. These question were all resolved after April 15, 2024, which was the knowledge cutoff date for our low-level agent supporting the Google Search API. We include a subset of the dataset for reference.From Manifold Markets, we initially filtered for questions that resolve between April 16, 2024 and May 15, 2024, inclusive. Then, to improve the quality of our questions, we filtered the question through prompting GPT-4 to return "Yes" if (i) it is possible to respond to the question with a yes or no answer and (ii) the question asks about an external event from the perspective of the majority of askers and respondents, and "No" otherwise. Finally, after the markets have resolved, we re-collect data using the API to extract the answers and compute Brier scores and accuracies. In the future, researchers can use our questions data for forecasting using LMs and information retrieval tools with cutoff dates before April 15, 2024 (see Section 4.1).We show the diversity of our dataset in Table 4 , with categories determined by GPT-3.5. Due to the popularity of Economics & Business and Politics & Governance questions on the forecasting platform, we have higher proportions of data in those categories. Our system prompt requested that the output satisfy elementary forecasting principles. Due to its length, the full prompt will be released along with the codebase and dataset. The forecasting principles are as follows:• You may find relevant time series data online. If the data is recent, you can use zeroth-order or first-order approximations (but for longer time horizons, recall that these may break down due to saturation effects). • Don't ignore base rates for low-probability events that haven't happened, but rather attempt to estimate those probabilities to inform your estimate. • Attempt to approximate the same quantity in multiple different ways. You are welcome to split your reasoning into multiple sections, where in each section you describe how you reached an estimate independently. If your estimates in these sections are very far off, reason through why the estimates are so different, potentially update those estimates, and theorize how confident you are in each estimate. Then, combine those forecasts using a weighted or trimmed mean or median. • Some data can be modeled well using a common distribution, e.g., normal, log-normal, power law, orPoisson distribution (but this is by no means a guarantee). If there's a good reason to believe this is the case, you should use information about those distributions.Due to length, we have only included one full sample below. In general, we observe that RTF yields more robust CoT-style outputs, whereas base LMs use more linear reasoning that may result in outputs unnecessarily biased towards specific considerations.Will ETH close above 3700? Background : None Resolution criteria : None Closure time : April 30 , 2024For reference, this prompt resolved as NO (0).[ Entering new AgentExecutor chain ...] To forecast the probability that ETH ( Ethereum ) will close above $3700 on April 30 , 2024 , I ' ll follow a structured approach :1. ** Reference Classes and Base Rates **:-Historical price trends of Ethereum .-Historical performance of similar cryptocurrencies .-General market conditions and trends in the cryptocurrency market . To forecast whether ETH ( Ethereum ) will close above $3700 by April 30 , 2024 , we need to consider several factors , including historical price data , market trends , and current events that could influence the price . Let 's break down the analysis step -by -step .### Step 1: Historical Data and Base Rates First , we need to look at the historical price data for ETH to understand its past performance and volatility . We can use this data to establish a base rate for how often ETH has closed above $3700 in the past .#### Historical Price Data -ETH has been highly volatile , with significant price swings over the past few years .-ETH reached an all -time high of around $4800 in November 2021.-Since then , the price has fluctuated , with periods of both significant gains and losses . Let 's assume we have the following historical data for ETH closing prices over the past few years : -2021: ETH closed above $3700 on 50 days . -Regulatory developments could either positively or negatively impact the market .-Macroeconomic factors , such as inflation and interest rates , could influence investor sentiment . #### Recent News -Ethereum 's transition to a proof -of -stake ( PoS ) consensus mechanism ( Ethereum 2.0) could have a significant impact on its price . -Any major security breaches or technological advancements could also affect the price . ### Step 3:Adjusting the Base Rate Based on the current market trends and recent news , we need to adjust our base rate . #### Positive Factors -Increased institutional interest and adoption could drive the price higher .-The successful implementation of Ethereum 2.0 could boost investor confidence .-Regulatory crackdowns could negatively impact the price .-Macroeconomic factors such as rising interest rates could lead to reduced investment in riskier assets like cryptocurrencies .We can combine our base rate with adjustments for current trends and news to arrive at a final probability estimate . The optimal strategy to minimize Brier scores is to forecast fi = P(oi = 1), so this scoring metric is unbiased. It is typical to compare Brier scores to 0.25, which can be achieved by fi = 0.5 for all i.Accuracy denotes whether fi and oi are on the same side of 0.5. 



Reviews: review: title: title: The paper introduces a framework called Reasoning and Tools for Forecasting (RTF), which enhances large language models' (LLMs) forecasting capabilities by integrating the ReAct (Reasoning and Acting) paradigm.
main_review: main_review: Quality:
The methodology seems to be sound and well thought. The paper details how a hierarchical structure of agents is used to interact with external tools like Google Search API and Python interpreters. The experiments are properly conducted, comparing RTF with multiple baselines, including human crowd predictions from Manifold Markets. The authors provide evaluations using Brier scores and accuracy metrics, showing that RTF outperforms or matches human forecasters. They also conduct ablation studies to validate the necessity of each component in their framework, such as the ReAct paradigm and hierarchical planning.

Clarity:
The paper is generally well-written and organized. The introduction provides a clear motivation for the work, highlighting the challenges of forecasting with traditional LLMs due to their reliance on static training data. The methodology section explains the RTF framework in detail with details on the hierarchical agent structure.

Originality:
The hierarchical structure of agents with tools for real-time data retrieval and simulation seems to be innovative.

Significance:
The significance of the work is in its potential to enhance LLMs' applicability in real-world decision-making tasks that require up-to-date information and dynamic reasoning. By demonstrating that LLMs can achieve or surpass human-level forecasting when equipped with appropriate tools, the paper opens up new avenues for research and practical applications. The RTF framework could be extended to other domains where real-time data access and numerical computations are crucial.

Pros:
- Demonstrates that RTF can match or outperform human forecasters on competitive platforms, achieving a Brier score of 0.169 and accuracy of 73.9%.
- Shows that a small ensemble of three agents is sufficient to achieve high performance, making the approach more computationally efficient than larger ensembles.
- Includes qualitative analysis and examples that illustrate the reasoning process of RTF, demonstrating its similarity to human thought processes.

Cons
- The evaluation is primarily based on data from Manifold Markets, focusing on popular forecasting questions. This may not generalize to specialized or less popular domains.
title: title: combining large language models (LLMs) with real-time data retrieval and computational tools to improve forecasting capabilities
main_review: main_review: The authors propose a hierarchical system of ReAct agents, where a high-level agent directs lower-level agents equipped with specialized toolkits, such as web search or simulation environments. Through experiments with competitive forecasting platforms, the paper demonstrates that RTF outperforms base LLMs and, in some cases, human forecasters.
Addressing the limitations of traditional language models (LMs) that rely heavily on memorized training data by combining hierarchical ReAct agents with tool usage to dynamically retrieve updated information and perform simulations is innovative.
The potential impact of this work is significant, especially in fields where accurate forecasting is crucial.

Pros:
- Strong empirical results.
- Innovative approach
- Well written

Cons:
- Lack of dataset diversity
- The hierarchical system, introduces a layer of complexity that might limit its adoption.


Overall, a good paper with strong empirical results. Future work could explore expanding the dataset to include more diverse and domain-specific questions.
title: title: This paper introduces Reasoning and Tools for Forecasting (RTF), a framework combining hierarchical ReAct agents with dynamic information retrieval and numerical simulation tools. RTF aims to improve language model performance on forecasting tasks, demonstrating human-level accuracy. The authors evaluate RTF on recent prediction market questions, showing it outperforms baselines and matches human crowd performance, while providing interpretable reasoning processes.
main_review: main_review: **Quality:**
##### Strengths:
- The paper presents a well-defined problem and a novel approach to address it.
- The experimental setup carefully addresses potential data leakage issues.
- The evaluation metrics (Brier score and accuracy) are appropriate and well-explained.

##### Weaknesses:
- The paper lacks a detailed discussion of the potential limitations of using Google Search API for information retrieval.
- There is no analysis of the computational resources required to run RTF compared to simpler methods.

**Clarity of writing:**
##### Strengths:
- The paper is generally well-structured, with clear sections for methodology and experiments.
- The problem formulation in Section 3 is precise and helps frame the rest of the paper.

##### Weaknesses:
- The related work section is relatively brief and could benefit from more detailed comparisons to existing methods.

**Originality of the paper:**
##### Strengths:
- The hierarchical planning approach combining high-level and low-level agents is novel in the context of forecasting.
- The use of a small ensemble (3 agents) with GPT-4o backbone is an interesting departure from larger ensembles in previous work.

##### Weaknesses:
- The core idea of using ReAct agents for forecasting builds upon existing work, though the specific implementation and application are novel.

**Significance of this work:**
##### Strengths:
- The paper addresses an important question about the genuine reasoning capabilities of language models beyond pattern matching.
- The potential applications of RTF in real-world decision-making scenarios highlight its significance.

##### Weaknesses:
- The limitations section acknowledges that the evaluation dataset may not fully capture the nuances of specialized domains, which could limit the immediate impact of the work.
- The paper does not extensively explore the ethical implications of using AI for forecasting in high-stakes scenarios.
title: title: Review of "Reasoning and Tools for Human-Level Forecasting
main_review: main_review: This article introduces the Inference and Prediction Tool (RTF) framework, which enhances the predictive ability of Language Models (LM) by integrating inference and dynamic data retrieval.
Quality: The RTF framework has a good structure and provides a powerful method, proving that LM can outperform human predictions in certain situations.
Clarity: Although most of the writing is clear, some complex concepts related to the ReAct framework can be enhanced by adding examples or charts to improve understanding.
Originality: This work innovatively combines reasoning with tools specifically designed for predictive tasks, marking a novel contribution to the field.
Meaning: For fields such as public health and the economy that require timely forecasting, its significance is significant and may affect the future application of artificial intelligence in decision-making.
advantage:
Innovative combination of reasoning and tools.
The empirical results show opposite effectiveness to human predictions.
Meet key requirements in practical applications.
Disadvantages:
The complexity of implementing the framework.
There is uncertainty regarding the universality of various prediction tasks.
Further exploration is needed to identify the limitations of LM in reasoning.
Overall, this article has made significant contributions to artificial intelligence and prediction, but requires minor modifications to improve clarity and address practical challenges.