Title: Look Before you Leap: Estimating LLM Benchmark Scores from Descriptions
Primary area: foundation or frontier models, including LLMs
Keywords: text-only performance forecasting, retrieval-augmented reasoning LLMs, anticipatory evaluation

Abstract: Progress in large language models is constrained by an evaluation bottleneck: build a benchmark, evaluate models and settings, then iterate. 
We therefore ask a simple question: can we forecast outcomes before running any experiments?
We study text-only performance forecasting—estimating a model’s score from a redacted task description and intended configuration, with no access to dataset instances. 
To support systematic study, we curate PRECOG, a corpus of redacted description–performance pairs spanning diverse tasks, domains, and metrics.
Experiments show the task is challenging but feasible: models equipped with a retrieval module that excludes source papers achieve moderate prediction performance with well-calibrated uncertainty, reaching mean absolute error as low as 8.7 on the Accuracy subset at high-confidence thresholds. 
Our analysis indicates that stronger reasoning models engage in diverse, iterative querying, whereas current open-source models lag and often skip retrieval or gather evidence with limited diversity.
We further test a zero-leakage setting—forecasting on newly released datasets or experiments before their papers are indexed—where GPT-5 with built-in web search still attains non-trivial prediction accuracy.
Overall, our corpus and analyses offer an initial step toward open-ended anticipatory evaluation—supporting difficulty estimation and smarter experiment prioritization.

TL;DR: We recast evaluation as forecasting: predict benchmark scores from redacted, schema-aligned descriptions, and curate PRECOG to study it. GPT-5+Search yields calibrated predictions, beats strong baselines, and generalizes to zero-leakage streaming.