SEED-ALFWorld-3B

Overview

Built with Qwen2.5-3B-Instruct, SEED-ALFWorld-3B is trained for text-based embodied interaction in ALFWorld using SEED (Self-Evolving On-Policy Distillation). Training unfolds in two stages:

• Hindsight-skill SFT: the model learns to analyze a completed trajectory and summarize it as a concise, episode-level skill.
• Self-evolving on-policy distillation: in each reinforcement-learning iteration, the current policy both acts in the environment and analyzes the trajectories it has just collected. The generated skills are used to re-score the same sampled actions, providing dense token-level supervision alongside GRPO. Once updated, the improved policy takes on both roles in the next iteration, allowing its behavior and hindsight supervision to evolve together.

At inference time, the model acts from the standard interaction history alone, with no analyzer, skill bank, retrieval module, or additional skill prompt.

Key Features

• Self-evolving supervision: policy updates improve both action selection and the analyzer used to generate the next round of hindsight skills.
• On-policy skill generation: supervision follows the states, actions, and failure modes encountered by the current policy.
• Dense token-level credit: skill-induced log-probability shifts provide decision-specific guidance beyond terminal rewards.
• No inference overhead: privileged hindsight skills are used only during training and are internalized by the released policy.

Performance

The following results are success rates (%) on the ALFWorld seen test split. Avg. is the unweighted macro-average over the six task families. An asterisk indicates that the method also receives a skill prompt at evaluation; SEED does not use skills at evaluation.

Method	Pick	Look	Clean	Heat	Cool	Pick2	Avg.
Vanilla	44.4	11.1	6.2	15.4	28.6	12.5	21.9
Skill-Prompt*	51.7	66.7	48.4	0.0	4.3	10.0	28.9
OPSD	48.8	41.7	16.7	0.0	15.8	16.7	28.1
GRPO	91.2	62.5	96.2	61.9	65.0	47.4	75.0
Skill-GRPO	88.9	71.4	58.8	70.6	40.7	29.2	60.2
Skill-GRPO*	94.3	57.1	100.0	66.7	73.1	57.1	80.5
GRPO+OPSD	100.0	82.4	85.7	75.0	70.0	60.0	81.2
Skill-SD	88.2	50.0	96.2	52.4	65.0	57.9	73.4
RLSD	87.9	75.0	90.9	75.0	73.1	68.4	79.7
SDAR	97.1	62.5	100.0	61.9	75.0	84.2	84.4
SEED (ours)	100.0	100.0	100.0	100.0	70.6	80.0	91.8

On the ALFWorld unseen split, the same 3B SEED checkpoint reaches an 86.2 macro-average, compared with 70.9 for the matched GRPO baseline.

Method	Pick	Look	Clean	Heat	Cool	Pick2	Avg.
ReAct	17.4	6.7	8.8	7.4	9.1	0.0	8.2
GRPO	73.9	60.0	82.4	59.3	72.7	76.9	70.9
SEED (ours)	90.4	78.3	79.5	94.3	86.2	88.2	86.2

Quickstart

Install the runtime dependencies:

pip install "transformers>=4.37.0" accelerate torch

Load the checkpoint and generate one ALFWorld action:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Jinyang23/Seed-AlfWorld-3B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)


def build_alfworld_prompt(
    task_description,
    step_count,
    history_length,
    action_history,
    current_step,
    current_observation,
    admissible_actions,
):
    actions = ", ".join(admissible_actions)
    return f"""You are an expert agent operating in the ALFRED Embodied Environment. Your task is
to: {task_description}
Prior to this step, you have already taken {step_count} step(s). Below are the most recent
{history_length} observations and the corresponding actions you took: {action_history}
You are now at step {current_step} and your current observation is: {current_observation}
Your admissible actions of the current situation are: [{actions}].

Now it's your turn to take an action.

You should first reason step-by-step about the current situation. This reasoning process
MUST be enclosed within <think> </think> tags.
Once you've finished your reasoning, you should choose an admissible action for current
step and present it within <action> </action> tags."""


prompt = build_alfworld_prompt(
    task_description="put a candle in toilet.",
    step_count=0,
    history_length=0,
    action_history="None.",
    current_step=1,
    current_observation="You are in the middle of a room.",
    admissible_actions=["look", "go to shelf 1", "go to toilet 1"],
)

text = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=False,
)
new_tokens = outputs[:, inputs.input_ids.shape[1]:]
response = tokenizer.batch_decode(new_tokens, skip_special_tokens=True)[0]
print(response)

In a complete evaluation loop, replace the example values with the current ALFWorld task, recent observation-action history, observation, and admissible actions at every step. Only execute the action emitted inside <action> tags.

The model can also be served with vLLM:

vllm serve Jinyang23/Seed-AlfWorld-3B \
  --served-model-name Seed-AlfWorld-3B \
  --dtype bfloat16

Limitations

• The model is specialized for ALFWorld-style text interaction and may not transfer reliably to unrelated environments or prompting formats.

License

This checkpoint is a derivative of Qwen2.5-3B-Instruct and is distributed under the Qwen Research License Agreement.

Citation

Citation metadata will be added after the public paper release.

Acknowledgements

The SEED implementation builds on veRL, verl-agent, and SDAR.

Links

• Model: https://huggingface.co/Jinyang23/Seed-AlfWorld-3B
• Base model: https://huggingface.co/Qwen/Qwen2.5-3B-Instruct
• Paper:
• Code: