Instructions to use Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld")
model = AutoModelForCausalLM.from_pretrained("Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld

SGLang

How to use Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld with Docker Model Runner:
```
docker model run hf.co/Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld
```

RSIPO-Qwen2.5-3B-Instruct-ALFWorld

The ALFWorld policy trained with RSIPO from Qwen/Qwen2.5-3B-Instruct, reported in Table 3 of the paper: the checkpoint after 150 RL steps, in bf16. Built with Qwen.

RSIPO (Recursive Self-Improving Policy Optimization) is a self-evolving agentic RL framework in which agentic skills and the policy co-evolve during training. It builds on SEED's hybrid of RLVR and on-policy self-distillation and adds an execution check: the policy rewrites a skill from one of its rollouts, a skill-enhanced teacher replays the task from the same initial state, and the measured replay gain both rewards the skill (on-policy skill evolution) and sets the weight of the self-distillation loss (trust-calibrated self-distillation). No skill is used at test time.

Base model: Qwen/Qwen2.5-3B-Instruct
Training: skill SFT on the base model (500 tasks, 8 rollouts each), then 150 steps of RSIPO on ALFWorld
Result in the paper: 95.7 success rate (%) on ALFWorld
Code: https://github.com/zfkarl/RSIPO
Paper: coming soon

Usage

The policy acts with the agent prompts of the RSIPO code. To reproduce the evaluation, set up the repository as described in its README, then run:

hf download Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld --local-dir $MODELS_ROOT/RSIPO-Qwen2.5-3B-Instruct-ALFWorld
RSIPO_EVAL_HF=1 bash scripts/rsipo/evaluate.sh alfworld RSIPO-Qwen2.5-3B-Instruct-ALFWorld

This plays the seen (140 games) and unseen (134 games) splits of ALFWorld in three rounds with seeds 1, 2 and 3, with temperature 0.4, at most 50 turns and no skill in the prompt, and writes the success rates of every round to summary.json.

License

The Qwen RESEARCH LICENSE AGREEMENT of the base model Qwen/Qwen2.5-3B-Instruct (LICENSE, with the attribution notice in NOTICE): research and evaluation use only; commercial use needs a license from Alibaba Cloud. The weights are modified from the base model by fine-tuning.

Citation

The citation will be added when the paper is public.

Downloads last month: 41

Safetensors

Model size

3B params

Tensor type

BF16

Model tree for Karl28/RSIPO-Qwen2.5-3B-Instruct-ALFWorld

Base model

Qwen/Qwen2.5-3B

Finetuned

Qwen/Qwen2.5-3B-Instruct

Finetuned

(1664)

this model