ABForge-Qwen3-8B

The main ABForge model: a single unified checkpoint that performs both tasks of paper-grounded ablation design.

ABForge is a post-training pipeline for paper-grounded ablation design. This checkpoint is post-trained with the full ABForge pipeline on a 1:1 mixture of both tasks — supervised fine-tuning from Qwen/Qwen3-8B followed by rubric-guided GRPO with per-task reward routing (SFT → GRPO, RL update 200).

Renamed on 2026-08-13 from ABForge-Qwen3-8B-Combined; the weights are unchanged.

Tasks

Given the ablation-free context of a research paper, this one model handles both:

• Task 1: Ablation Objective Identification — propose candidate ablation objectives, each expressed as a Target Module (the component to ablate) paired with a Research Question it is meant to answer.
• Task 2: Ablation Experiment Synthesis — produce a concrete, executable ablation experiment plan (variants, controls, datasets, metrics, expected outcomes) for a given objective.

Training data

Mixed-task SFT on train/sft_task1_45961.jsonl + train/sft_task2_37019.jsonl, then mixed-task GRPO on train/RL_task1_30K.jsonl + train/RL_task2_30K.jsonl, from SlowGuess/abforge-data (derived from CC-licensed research papers). Both stages use a 1:1 task mixture, and during RL each rollout is routed to its task-specific reward by data_source.

Results

AblationBench, automated rubric-based LLM-as-a-Judge evaluation (eval/ablationbench_200.jsonl, 200 papers, judge claude-sonnet-4-6):

Model	Task 1	Task 2
Qwen/Qwen3-8B (base)	44.4	43.4
-SFT (unified, SFT only)	30.7	52.2
-RL (unified, RL only)	52.2	54.9
ABForge-Qwen3-8B (this model, SFT → GRPO)	55.9	62.4

The unified model surpasses the task-specific specialists on Task 2 while consolidating both capabilities into a single checkpoint; see the paper's ablation table for the task-specific comparison.

Per-paper generations and judge rationales for this model are released in the dataset repo under outputs/task{1,2}/{generations,judge_claude-sonnet-4-6}/abforge.jsonl, and the aggregate row is abforge in outputs/leaderboard.csv.

Related models

Post-training stages of this model (unified, both tasks):

• SlowGuess/ABForge-Qwen3-8B-SFT — SFT only (RL init)
• SlowGuess/ABForge-Qwen3-8B-RL — GRPO from base, no SFT

Task-specific specialists (ablation of task sharing):

• SlowGuess/ABForge-Qwen3-8B-Task1 (SFT · RL-only)
• SlowGuess/ABForge-Qwen3-8B-Task2 (SFT · RL-only)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SlowGuess/ABForge-Qwen3-8B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

Use the Task 1 / Task 2 prompt templates from the code release — the model is trained on those exact formats and the rubric-based evaluator expects the corresponding output structure. Low-temperature (greedy) decoding is recommended.

Evaluation

Reproduce AblationBench evaluation with the SlowGuess/Abforge_1 code:

git clone https://github.com/SlowGuess/Abforge_1 && cd Abforge_1
huggingface-cli download SlowGuess/abforge-data --repo-type dataset --local-dir data