MIRAI: Evaluating LLM Agents for International Event Forecasting

Authors: Chenchen Ye, Ziniu Hu, Yihe Deng, Zijie Huang, Mingyu Derek Ma, Yanqiao Zhu, Wei Wang
Primary area: datasets and benchmarks
Keywords: LLM Agents, Temporal Forecasting, Tool Use

Abstract:
We present MIRAI, a benchmark designed to systematically evaluate LLM agents as temporal forecasters to predict international events. Our benchmark features an agentic environment with tools for accessing an extensive database of historical, structured events and textual news articles. We refine the GDELT event database with careful cleaning and parsing to curate a series of relational prediction tasks with varying forecasting horizons, assessing LLM agents’ abilities from short-term to long-term forecasting. We further implement APIs to enable LLM agents to utilize different tools via a code-based interface. Notably, MIRAI features a dynamic data construction pipeline that supports periodically downloading recent news and events, and automatically generates the most recent test split. This allows us to evaluate any newly released model in a contamination-free manner as we can always construct a test split later than its knowledge cutoff date. MIRAI comprehensively evaluates the agents’ capabilities in three dimensions: 1) autonomously source and integrate critical information from large global databases; 2) write codes with both domain-specific APIs and libraries for tool-use; and 3) jointly reason over historical knowledge from diverse formats and timespan to accurately predict future events. Through comprehensive evaluation, we establish a reliable benchmark for assessing the capabilities of LLM agents in forecasting international events and contribute to the development of more accurate and trustworthy models for international relation analysis.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730482769982

thread_type:
official_review

<thread_object>
creator: reviewer_szj2

<content>
summary:
This paper introduces MIRAI, a practical benchmark designed to predict future interactions between two countries. It constructs a dataset based on news articles and existing information extraction and representation tools, and then exposes an API that allows an LLM agent to interact with the different data sources. The authors' extensive experiments provide valuable insights into how LLMs perform on this task.

strengths:
S1. The paper provides the code and data. Besides, it comes with an extensive description of the data and code, ensuring reproducibility and reuse.

S2. The authors ran a lot of experiments testing different scenarios and configurations. They obtained very insightful and broad results about the task.

S3. The paper is well illustrated, with many examples showcasing the performance of the systems. It is also well-written and easy to follow.

weaknesses:
W1. The paper heavily relies on the appendix (the paper is 73 pages long!). As a reviewer, I am not supposed to read it, but in many cases, I had to. For example, the previous work is mostly in the appendix (two small paragraphs are in the main paper) and the human evaluation is in the appendix. 

W2. The technical contribution of the paper could be clearer. The dataset construction pipeline is standard and relies on many existing tools with additional cleaning. The baselines are also coming from previous works.

W3. The scope of the task is very narrow: Predicting the future interaction between two countries.

Others/Typos

O1. Some figures are hard to read, especially in black and white (2, 4, 5, 7, 8)

O2. Human evaluation of the dataset is done in the appendix. Is it enough to only evaluate 51 events? Besides, 82% seems quite low to me.

O3. The authors include the confidence intervals but do not discuss them. In many cases, there are overlaps between the baseline, making the conclusions unclear.

O4. Adding the human evaluation in Table 2 is only one line, and it would be insightful.

O5. Having the RAG baseline would be very insightful in knowing if one needs the API interface.

T6. Line 420, problem citation (et al, 2024)

O7. Figure 8.b: In general, red is used to show incorrect things. Here, it is counter-intuitive.

questions:
Q1. Are the news articles freely available? Are there copyright issues?

Q2. Line 260: Is balancing this way fair? Some months are naturally more active than others, and some relationships are more frequent than others.

Q3. How does the number of parameters in the LLM impact the performance? In Table 2, GPT-3.5-turbo has more parameters than the other baselines but is competitive with GPT-4o-mini, which the oldness of GPT-3.5 can explain.

Q4. For the Direct IO baseline, isn't a main issue outputting the output following the standard of the answer? Do you provide the possible relationships in any way?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730590325213

thread_type:
official_review

<thread_object>
creator: reviewer_udbl

<content>
summary:
This paper introduces MIRAI, a benchmark to test LLMs for events forecasting. The article provides a detailed description of the process for generating predictions from LLMs which are based on recently extracted news and geopolitical events. This is based on a strategy called ReAct that includes the three steps of thinking, acting, and observing. An ontology called CAMEO is used for the geopolitical events analysis. Extensive experiments have been performed to compare MIRAI to other approaches and strategies. Different Large Language Models have been used to evaluate their performance.

strengths:
- The paper is well-written.
- Extensive experiments have been provided. These include comparing MIRAI process against other strategies and temporal reasoning 
  benchmarks and comparing base LLMs and smaller Language Models using MIRAI.
- The results of these experiments seem promising.

weaknesses:
- Only one concrete example is provided and discussed in the paper. More examples would facilitate a better understanding of the results.  For instance, examples that showcase different types of forecasting scenarios or that highlight the strengths of MIRAI could have been included.
- The work should have been also compared with RAG-based (Retrieval-Augmented generation) approaches. It is unclear how MIRAI's approach differs from RAG-based methods.
- Typos  line 088: ".. is able to better utilizes and benefits... " ->  ".. is able to better utilize and benefit... "

questions:
How LLMs would perform if the news were used as a direct source instead of following the ReAct strategy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730705219698

thread_type:
official_review

<thread_object>
creator: reviewer_azdo

<content>
summary:
The paper proposes a new LLM benchmark for short- and longterm forcasting international events, which is essentialy trying to predict relations between international parties. The general and fine-grained relations stem from an ontology, namely CAMEO. The authors define a think, act, oberserve loop, where the agent processes the query at hand and can give an answer right away, otherwise can act to get more information via coding and eventually observes the executed code. The benchmark equips agents to act by either single functions for data retrieval or code blocks for writing more complex code.

The results show that the task is challenging for LLMs by evaluating a set of baseline LLMs, where GPT-4o mini ends up performing the best. Next to the baseline LLMs, the authors also show the performance of fine-tuned methods from other benchmarks, underlining the benchmark is challenging. In addition, the authors run additional experiments and analyses, including observed generated code errors, boosting smaller architectures or investigating chosen action orders.

strengths:
- Novel relation prediction task formulation (wrt other benchmarks) 
- Sensibe LLM system design, enabling the to query for information or to write own code, and defining an agent-based behavioural loop
- Sufficiently broad evaluation, including diverse analyses and also including baselines from other competing benchmarks
- Empirical results are promising

weaknesses:
- Unclear if, beyond the integrated fine-tuned approaches from other works, the results for LLMs are superior if compared to other task formulations such as QA. Since LLMs have been used there as well, the question remains open if other types of system components or prompts significantly worked better in the past

questions:
- Did you also try a fine-tuned the baseline LLMs (given their are open-source)?
- Which competitor in the related work overview would be strongest compared to your approach?
- From the related work it becomes quite clear what the conceptual differences to prior LLM QA works are, but how much better does the new agent design actually work compared to more straightforward prior approaches?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730970299841

thread_type:
official_review

<thread_object>
creator: reviewer_zmpf

<content>
summary:
The paper presents MIRAI, a benchmark designed to assess the performance of LLM agents in predicting international event relations. It emphasizes APIs to enable LLM agents to utilize different tools, a refined GDELT event database, and a dynamic data construction pipeline to ensure contamination-free test sets, aiming to provide a reliable standard for assessing the capabilities of LLM agents in forecasting international events.

strengths:
•	Clarity and Structure: The paper is clearly written and easy to follow, which facilitates understanding and replicability.
•	Contamination-Free Test Sets: The focus on providing contamination-free test sets is a significant strength, ensuring unbiased performance evaluation. Regular updates to the test sets also add to its robustness and relevance over time, when newer models should be evaluated.
•	Insightful Visuals: Figure 2 offers a valuable overview of the distribution of relation types within the dataset.
•	Detailed Analysis: The authors provide an in-depth result analysis across diverse categories, such as action order and accuracy across different relation types

weaknesses:
1.	Absence of Baseline Comparisons: As a benchmark paper, a comparison with simple (heuristic) baselines is critical to contextualize the agents performance in the MIRAI benchmark. The lack of such comparisons with simple baselines is a notable drawback, reducing the ability to assess the evaluated agents true effectiveness.  See suggestion 3).
2.	Focus Solely on Relation Prediction: The exclusive focus on relation forecasting seems narrow. Incorporating link prediction tasks (i.e., predicting specific subjects and objects for given queries) would align it better with existing TKG forecasting approaches and provide a more comprehensive benchmark. See question 1)
3.	Doubts in Experimental Setting when comparing to "traditional" models: 
o	The "traditional" models that you compare to are a bit old, which may not reflect current advancements in temporal knowledge graph forecasting. Inclusion of more recent models, such as TiRGN (Time-Guided Recurrent Graph Network with Local-Global Historical Patterns for Temporal Knowledge Graph Reasoning), would enhance the relevance of comparisons. 
o	I have some doubts about the evaluation on these models, see question 5. For example, why finetune it only until 2023-06, even though some agents have more recent cutoff-dates. 

1.	Integrate Human Performance in Main Paper: Including the presented human forecasting performance in the main paper (currently in Appendix G.4) would strengthen the benchmark's validation, given the strong performance of human evaluators on most metrics.  I strongly suggest to include the human study in main paper.
2.	Condense the Appendix: I suggest to potentially not include the full API specification and implementation, and README of the github repo, but instead a link to the github repo containing these. This could streamline the paper, allowing readers to focus on MIRAI's core contributions.
3.	Include a simple Baseline for Comparison: To provide a more meaningful comparison, I strongly suggest to include a simple/ heuristic baseline to compare the agents to. For example, a baseline that predicts the same relations as previously occurred, e.g. a modification from [1] to relation prediction. 
4.	References: AutoGPT Documentation: this should at least contain a date and a link.
 [1] Gastinger, J., Meilicke, C., Errica, F., Sztyler, T., Schuelke, A., & Stuckenschmidt, H. (2024). History repeats itself: A Baseline for Temporal Knowledge Graph Forecasting. International Joint Conferences on Artificial Intelligence.

questions:
1.	Focus on Relation Forecasting: Why did the authors choose to focus solely on relation forecasting rather than including link prediction (predicting subjects and objects), which is more standard in TKG forecasting, and KG completion evaluation?
2.	Metric Selection: What motivated the choice of the current metrics, and why were more commonly reported metrics, like MRR and Hits@k, excluded?
3.	Dataset Update Commitment: The authors commit to updating the dataset split monthly. How long is this commitment expected to last, and will it be sustained indefinitely?
4.	Higher Thresholds for Test Data: What is the rationale behind applying higher thresholds to the test data (100 daily mentions, 5 news articles) than to the training data? 
5.	Clarification on the "traditional models" experiments: Section 3.2 3): 
o	What do you mean with "fine-tuned" in this case? Did you train the models on the provided GDELT event data? 
o	Assuming that with "fine-tune" you mean train: Why call it fine-tuned? 
o	Why do you fine-tune it only until 2023-6? considering that the query date is 2024-02, this is a very large gap. It seems also unfair, considering that e.g. for Llama-3.1-8B-Instruct the cutoff date is 2023-12. 
o	Do you apply RE-GCN in single-step or multi-step prediction mode, i.e. for predicting t, do you feed ground truth up until t-1?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732400429045

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Comment

comment:
We sincerely thank all the reviewers for their insightful and encouraging feedback on our benchmark manuscript. We appreciate the recognition of the clarity and reproducibility (noted by Reviewers ZMPf and SzJ2), featuring contamination-free test sets with regular updates (ZMPf) and comprehensive code documentation (SzJ2). We also thank the reviewers for noting the the extensive experimental evaluations (mentioned by all reviewers), including diverse analyses across different relation types (ZMPf), comparisons with competing benchmarks (AZdo, uDbL) and the well-crafted visualizations (ZMPf, SzJ2).

We have addressed the reviewers' comments with new experiments and substantial revisions. Our key additions include: 
- Adding experiments for heuristics-based forecasting baseline, Recurrency, with different data cutoffs (Reviewer ZMPf’s W1, W6, W7)
- Adding experiments for traditional temporal knowledge graph-based method, RE-CGN, with different training data cutoffs (Reviewer ZMPf’s W3, Q5)
- Adding ranking-based forecasting task formulation and corresponding experiments with LLM agents, ReAct (Rank) and ReAct (Rank w.Prob), studying the effect of rank list length and format (Reviewer ZMPf’s Q2)
- Adding retrieve-augmented generation (RAG) baselines, including structured event retrieval, textual news retrieval, and hybrid of the two (Reviewer AZdo’s W1, Q3; Reviewer uDbL’s W2, Q1; Reviewer SzJ2’s O5)
- Adding more agent forecasting examples and analysis (Reviewer uDbL’s W1)
- Adding experiments studying the effect of LLM parameter size for forecasting (Reviewer SzJ2’s Q3) 
- Adding a new baseline of augmenting CAMEO ontology to LLM Direct IO (Reviewer SzJ2’s Q4)


### Contributions of this paper

We further make the following clarification to highlight the contribution of our paper. 
- **Scope.** Most importantly, our manuscript clearly defined its scope in lines 10-14, 21-25, and 90-96: we introduce **a benchmark dataset** with an **agentic environment** to systematically evaluate LLM agents' forecasting capabilities, which is a newly emerging and promising direction in LLM development. 
- **Benchmark compatibility.** To facilitate meaningful comparisons, we have made further efforts so that our benchmark is compatible with traditional TKG methods, NLP based approaches and LLM based methods like retrieval-augmented generation (RAG), allowing us to better contextualize the LLM agents' performance. 
- **Focus.** Our contribution is not proposing state-of-the-art methods or advocating for current LLM agent baseline performances. Rather, we aim to **provide a foundational benchmark that can drive future development in this field**, particularly given LLMs' demonstrated strengths in tool utilization and environmental interaction.
- **Findings and analysis.** Our extensive experiments reveal challenges and opportunities for LLM agents in forecasting: diverse information gathering and tool-use are critical for temporal forecasting, advanced action types like Code Block benefit stronger LLM agents, scaling inference-time compute improves small LLMs, and there is substantial room for LLM agents to improve in code generation, tool-use planning, reasoning, and long-horizontal forecasting.
- **Fostering future research.** LLM agents have become a prominent focus in recent research [1-8], driven by interest in their ability to interact with dynamic environments through tool use, complex reasoning, and decision-making. This highlights the need for benchmarks tailored to real-world applications. MIRAI thus offers a uniquely designed platform centered around event forecasting, enabling researchers to explore and develop LLM agents in a focused yet flexible setting.
To further reflect this, we have also added a contribution statement in lines 91-98 of the revised manuscript.

[1] Large Language Model based Multi-Agents: A Survey of Progress and Challenges

[2] AgentGym: Evolving Large Language Model-based Agents across Diverse Environments.

[3] SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering

[4] OpenHands: An Open Platform for AI Software Developers as Generalist Agents

[5] WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?

[6] Voyager: An Open-Ended Embodied Agent with Large Language Models

[7] Approaching Human-Level Forecasting with Language Models

[8] From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents MIRAI, a practical benchmark for predicting future interactions between countries. Overall the reviews are mixed (two 5's and two 6's), and there was an in-depth discussion among the reviewers after the rebuttal. I will mainly quote the reviewers to respect their thoughts. On the positive side, this work is extremely serious, with all the components clearly explained and the code well-documented and reproducible. The authors' effort during the rebuttals demonstrated their desire to produce quality work. However, there are also remaining concerns: (1) The additional RAG-based experiments provide better results than the authors' approach; (2) While the authors claim to have provided additional forecasting results in the Appendix, they are still related to one specific example mentioned in the paper, the relations between China and Australia; (3) The authors claim to have provided an interactive agent demo in the following link, but this does not seem to work: https://drive.google.com/file/d/1kKvYdAYv5hed-sbF_QE1moP-dQbfYoXI/view; (4) Additionally, the repository linked in the paper is also not available: https://anonymous.4open.science/r/ForecastAgent-3419. (We note the anonymous git did work during the review period.) There was an agreement on the lack of relevance of the agent approach to this problem and uncertainty about the impact of the dataset. So while the work is quite thorough, and one therefore can build on the developed benchmark, there are doubts on the need for an agent-based approach and no clear picture on how relevant the benchmark would be for the audience. Overall, I think the paper can be improved and should go through another round of reviewing.