Title: Evaluating LLMs on Real-World Forecasting Against Expert Forecasters
Primary area: datasets and benchmarks
Keywords: automated forecasting, benchmarks

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities across diverse tasks, but their ability to forecast future events remains understudied. Large language models used to struggle to come close to the accuracy of a human crowd. I evaluate state-of-the-art LLMs on 464 forecasting questions from Metaculus, comparing their performance against top forecasters. Frontier models achieve Brier scores that ostensibly surpass the human crowd but still significantly underperform a group of experts.

TL;DR: LLM forecasting rivals crowd forecasting accuracy but still falls short of expert forecasters