yarrow-forecast

yarrow-forecast forecasts U.S. party primary and nomination outcomes. Given a question, its resolution criteria, and the evidence available as of a stated date, one forward pass returns a calibrated probability in [0, 1].

The model does not write an answer and does not abstain. It reads the probability from the next-token distribution. There is nothing to parse. The weights run locally: 27B parameters, 256K context, no hosted API in the loop.

Why use it

The marginal cost is about zero. The weights sit on your own GPU. A forward pass does not incur a per-call fee.

One forward pass, no generated text. Measured batch inference is 0.12–1.6 seconds per question. There is no sampling and no token-by-token decode. The probability is read from the next-token distribution.

No third-party service. Scoring does not depend on an outside API, a proxy, or a rate limit. The model answered every item in the evaluation.

The weights can be trained further. A LoRA update followed by a merge left yes/no calibration error at 0.001–0.003. On 15 other checked items, the answers were unchanged after the merge.

Model details

Model	yarrow-forecast
Parameters	27B
Context length	256K tokens
Task	P(the event resolves YES)
Validated domain	U.S. party primary and nomination elections
Output	One probability in [0, 1]
Inference	One forward pass, deterministic, 0% abstention
License	CC BY-NC-4.0

Input and output

Question	The event to forecast
Resolution criteria	How a YES or NO outcome is decided
Evidence	News, polls, and other material available at prediction time
As-of date	The date the forecast is conditioned on
Output	p ∈ [0, 1]

Usage

forecast.py loads the weights once. Each call returns only the calibrated probability.

from forecast import Forecaster

model = Forecaster("YarrowLab/yarrow-forecast")
p = model.forecast(
    title="Will candidate X win the nomination?",
    rules="Resolves YES if candidate X is declared the nominee.",
    evidence="[E1] ...",
    as_of="2026-09-05",
)
print(p)  # 0.71

ps = model.forecast_batch([
    {"title": "...", "rules": "...", "evidence": "...", "as_of": "2026-09-05"},
    {"title": "...", "rules": "...", "evidence": "...", "as_of": "2026-09-05"},
])

Evaluation

Validated domain: U.S. party primary and nomination questions with settled outcomes. Same questions, same evidence, same outcomes. Each model answered in its own setup. yarrow-forecast answered all 150.

Model	Brier ↓	AUC ↑	Directional accuracy	Answered
yarrow-forecast	0.175	0.81	76.0%	150
Claude Sonnet 4.6	0.171	0.84	70.1%	117
GPT-5.5	0.185	0.87	65.5%	148
Historical base rate	0.239	0.51	62.7%	150
GPT-5.4-mini	0.242	0.51	62.4%	149
OpenForecaster-8B	0.265	0.72	61.3%	93

Directional accuracy on the full set is 76.0%, above Claude Sonnet 4.6 (70.1%) and GPT-5.5 (65.5%). Brier is 0.175, below GPT-5.5 (0.185), GPT-5.4-mini (0.242), and OpenForecaster-8B (0.265), and about 24% lower than the historical base rate (0.239). Across five out-of-sample fold splits the Brier stays in 0.175–0.185 and directional accuracy in 72.7–76.0%. Abstention is 0%.

Claude Sonnet 4.6 scores 0.171 Brier on the 117 questions it returned. OpenForecaster-8B's 93 questions are the ones it returned. Unanswered items were left unanswered.

A later set of 166 settled questions from the same category scores 0.1653 Brier (95% interval 0.136–0.200, clustered by election night).

With the question and the resolution rule alone, AUC is about 0.51. With the evidence supplied, AUC is 0.808.

License

CC BY-NC-4.0. Attribution is required. Commercial use is not permitted.