CivTelescope

CivTelescope is the pairwise judge from Rome Wasn't Built in One Turn: Replay-Grounded Reward Signals for Long-Horizon Freeciv Agents. Given two Freeciv positions, it predicts which one reaches the higher terminal score when the game is played to the end. It is fine-tuned from Qwen2.5-7B-Instruct with LoRA (rank 16) on 10,800 position pairs whose labels come from CivHarness replays.

• Code: https://github.com/jamesnulliu/CivMarshmallow
• Policies trained with CivTelescope as a dense reward: CivMarshmallow/CivMarshmallow-Qwen3-8B
• Baseline policies: CivMarshmallow/Baselines-Qwen3-8B

Results

Evaluation	Accuracy
Held-out trap pairs	0.775
Held-out non-trap pairs	0.878
Trap pairs, unseen classic ruleset	0.592

A trap pair is a pair of positions where the position with the higher current score reaches the lower terminal score. On the same held-out trap pairs, five frontier models score 0.26–0.34.

Files

lora.pt holds the LoRA adapter for the q_proj and v_proj layers (rank 16, alpha 32) as a PyTorch state dict. It is not in PEFT format; merge it into the base model with the script in the code repository:

hf download CivMarshmallow/CivTelescope lora.pt --local-dir ckpts/civtelescope
python scripts/merge_lora.py --base-model Qwen/Qwen2.5-7B-Instruct \
    --adapters ckpts/civtelescope/lora.pt --out models/civtelescope

The merged model is what policy training serves as CIVTELESCOPE_HF; scripts/eval_civtelescope.py evaluates it on trap pairs with the paper's prompt.