Hugging Face's logo

SeanWang0027
/
ftb-sciworld-repro

Reinforcement Learning
English
on-policy-distillation
llm-agents
scienceworld
Model card Files
xet
 Community

ftb-sciworld-repro

A self-contained, runnable bundle for training five multi-turn on-policy distillation methods on ScienceWorld with a Qwen3-1.7B student and a Qwen3-32B teacher:

method what it does
OPD vanilla on-policy distillation: the student runs whole episodes alone, the teacher scores its tokens afterwards
Guided-OPD per-turn coin flip picks teacher or student; teacher turns get SFT cross-entropy, student turns get the OPD reverse-KL
TCOD-B2F replays a gold-action prefix, student takes over near the end; the takeover point walks backwards over training
TCOD-F2B student always starts at turn 1, but episodes are truncated short and grow over training
FTB FutureBridge: inserts a validated teacher "bridge" at the highest-disagreement student turn

Everything needed to run is in here except the model weights, the task data and the Python dependencies, which scripts/setup.sh fetches.

Requirements

  • 8 GPUs with ≥ 80 GB each, on one node. The teacher is dense Qwen3-32B at TP2 — roughly 32 GB of weights per device before any KV cache — so 40 GB cards cannot hold it at this parallelism. The split is 4 student rollout + 2 teacher
    • 2 FSDP trainer.
  • Linux with CUDA and NCCL
  • Python 3.10
  • Java 17+ on PATH — ScienceWorld starts a JVM per episode
  • ~80 GB free disk for models, plus ~4 GB per saved checkpoint (4 per run)

Quick start 

hf download SeanWang0027/ftb-sciworld-repro --local-dir ftb-sciworld-repro
cd ftb-sciworld-repro

# If the node already has a working Trinity/TCOD environment:
bash scripts/setup.sh

# If it does not, also install the pinned dependency set:
bash scripts/setup.sh --with-deps

# Reuse models already on the node instead of downloading ~68 GB:
bash scripts/setup.sh --models /path/to/models     # expects Qwen3-1.7B, Qwen3-32B

# Then, one method at a time (each wants all 8 GPUs):
bash scripts/run.sh ftb
bash scripts/run.sh opd
bash scripts/run.sh guided_opd
bash scripts/run.sh tcod_f2b
bash scripts/run.sh tcod_b2f

Behind a scheduler: sbatch -A <acct> -p <partition> --export=ALL,METHOD=ftb scripts/run_slurm.sbatch.

(The Hub does not preserve the executable bit, so invoke the scripts as bash scripts/... — as above — rather than ./scripts/....)

setup.sh is idempotent — re-run it after a partial failure. It pip-installs the vendored tcod/ (with --no-deps unless you pass --with-deps), copies the FutureBridge overlay over it, verifies all five workflow classes import, puts the models under ./models, downloads the task split to ./data/scienceworld, and creates ./outputs/.

A run is 200 training steps, batch 16 tasks × up to 30 environment turns, with a checkpoint every 50 steps under outputs/checkpoints/FutureBridge-OPD/<run_name>/global_step_*/. Expect hours to a day per method; B2F and FTB are slowest (they boot two ScienceWorld JVMs per training task for gold-path replay).

What is upstream and what is not

This matters if you are reproducing published numbers.

Upstream, redistributed unmodified (both Apache-2.0, see NOTICE):

Added here: configs/{opd,guided_opd,tcod_b2f,tcod_f2b}.yaml, configs/bench.yaml.tmpl, everything in scripts/, this README, NOTICE.

The FutureBridge release ships ScienceWorld configs for FutureBridge and its ablations only. For the four baselines it ships the workflow implementations but no configuration — instead its configs/README.md gives a "Baseline workflow mappings" table saying to swap default_workflow_type and keep the paper settings. The four baseline YAMLs here are a reconstruction from that table: every shared setting is identical to ftb.yaml, and the deltas are the workflow class, the method's own workflow_args, run/buffer names, the monitor backend (tensorboard rather than wandb), and — for Guided-OPD only — the mix policy-loss block.

Judgement calls the upstream table does not determine, in case you want to change them:

setting value here basis
B2F / F2B checkpoint_steps 5 / 6 upstream TCOD's own TCOD_examples/scienceworld/tcod_{b2f,f2b}.yaml
Guided-OPD beta_start/end, curriculum_ratio 1.0 / 0.0 / 0.8 defaults in guided_opd_workflow.py's own schedule function
Guided-OPD mu 0.5 weighs the forward-KL (teacher-turn SFT) and reverse-KL (student-turn OPD) terms equally. MIXPolicyLossFn's own default is 0.1. Nothing in the release fixes this value — treat it as a hyperparameter, not a reproduction.
mix batch-size args derived from the 8-GPU split not specified upstream

Reporting results

The upstream protocol does not evaluate. Every released config sets eval_interval: 9999 and eval_on_startup: false, so the declared eval task set never fires, and no evaluation entry point is shipped. Its configs/README.md says to "run the configured evaluation task set once after training" but provides no mechanism.

What the release's own analysis code reads is the training rollout metric — analysis/plot_training_dynamics.py maps rollout/env_done/mean to the panel labelled "Completion Rate". So the faithful reproduction is: run the five methods and plot rollout/env_done/mean against step.

One caveat if you compare those curves across methods: the training rollout is not produced by the same policy in each arm. OPD's trajectories are 100% student; B2F's begin with a gold-action prefix; Guided-OPD's contain teacher turns with probability β (≈1 early in training); FTB's contain both. Early-training completion is therefore partly the teacher's or the gold path's work, not the student's.

scripts/eval.sh is provided as an addition, not a reproduction: it scores a run's checkpoints on the held-out 1,308-task test split through a single shared workflow, so the arms are measured under identical conditions with no teacher intervention. See the header of that script for why it forces one workflow for all five methods.

Either way the metric is env_done. There is no score metric anywhere, here or upstream: ScienceWorld's 0–100 partial credit is computed and stored on Experience.reward, which never reaches the metrics dictionary that the monitor aggregates. And env_done counts any terminating episode — ScienceWorld also terminates on failure, by driving the score negative — so it is not a clean success rate.

Known properties worth knowing before you interpret runs

These are properties of the upstream implementation, not bugs introduced here. None of them are patched in this bundle.

1. The prompt context accumulates, and overflow stops the agent. Four of the five workflows (all but Guided-OPD, which keeps a 10-turn window) send the entire growing chat history every turn, on top of a 2-step summary embedded in each user message. Measured with the Qwen3-1.7B tokenizer on real ScienceWorld rooms, the prompt crosses max_prompt_tokens: 10240 somewhere around turn 11 (object-rich room) to turn 17 (sparse room) of a 30-turn episode. Past that point the vLLM engine is not called at all: the framework returns a placeholder experience with an all-zero action mask, so the turn contributes no gradient, the parsed action is empty, and the agent stops acting for the rest of the episode. Grep a run log for Prompt was truncated to to count how often it happens.

Upstream hit a more severe version of this and fixed half of it: TCOD commit 17a8af2 ("use compact action representation to avoid token overflow") replaced a ~1162-item full action×object list with a ~43-item templates+objects pair, because single-step prompts were already exceeding the limit. That shrank the per-turn prompt; it did not change the accumulation.

2. The <think> instruction is inert. Both the system prompt and every user message require reasoning "enclosed within <think> </think> tags", but the configs run Qwen3 with enable_thinking: false, and that chat template pre-emits and closes an empty <think>\n\n</think>\n\n before the model writes a token. Across 11,264 recorded generation turns from three models in earlier measurements, <think> opened in 0.0% of them. The instruction costs tokens every turn and parse_action only ever reads <action>.

3. The full action×object list is hidden from the model but used to judge it. FTB's _is_valid_action checks a proposed action against get_valid_action_object_combinations() — the same ~1000-item list that commit 17a8af2 removed from the prompt. The student must compose a string that lands exactly in that set from two separate short lists, and a miss discards the whole bridge candidate.

4. FTB is B2F plus bridging. FutureBridgeScienceWorldWorkflow subclasses TCOD_b2f_scienceworld_workflow and overrides only _finalize_turn_responses. If you want to know whether FTB's gain comes from the bridge or from the B2F curriculum underneath it, tcod_b2f is the controlled comparison — not opd.

Troubleshooting

symptom cause / fix
Failed to look up actor 'synchronizer' during weight sync an engine was set to TP1; keep tensor_parallel_size: 2 for both rollout and teacher
teacher engine OOM at startup cards smaller than 80 GB; there is no supported fallback at this parallelism
wandb hangs then kills the explorer only configs/ftb.yaml uses wandb (it is the upstream file). wandb login, or switch its monitor_type to tensorboard like the four baselines. WANDB_MODE=offline does not reach the Ray actor.
ModuleNotFoundError: scienceworld, or JVM errors mid-run pip install scienceworld==1.2.2; confirm java -version works
flash-attn build failure it compiles against the installed torch; install torch first, then pip install flash-attn==2.8.1 --no-build-isolation. Training can proceed without it.
stale Ray state after a crash ray stop, and delete the run's buffer db under outputs/buffers/ if restarting from scratch
two runs at once not supported: one Ray cluster, all 8 GPUs per run

Credits

Method and implementation are the work of the upstream authors:

This bundle only packages them for reproducible execution and adds the missing baseline configurations. Please cite the upstream work.

Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
Reinforcement Learning
loading