Instructions to use YijiaFan/CaPtain-4B-SFT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use YijiaFan/CaPtain-4B-SFT with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="YijiaFan/CaPtain-4B-SFT")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("YijiaFan/CaPtain-4B-SFT")
model = AutoModelForMultimodalLM.from_pretrained("YijiaFan/CaPtain-4B-SFT", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use YijiaFan/CaPtain-4B-SFT with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "YijiaFan/CaPtain-4B-SFT"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YijiaFan/CaPtain-4B-SFT",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/YijiaFan/CaPtain-4B-SFT

SGLang

How to use YijiaFan/CaPtain-4B-SFT with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "YijiaFan/CaPtain-4B-SFT" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YijiaFan/CaPtain-4B-SFT",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "YijiaFan/CaPtain-4B-SFT" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "YijiaFan/CaPtain-4B-SFT",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use YijiaFan/CaPtain-4B-SFT with Docker Model Runner:
```
docker model run hf.co/YijiaFan/CaPtain-4B-SFT
```

CaPtain-4B-SFT

CaPtain-4B-SFT is the supervised fine-tuning (SFT) stage of CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents. It is Qwen3.5-4B fine-tuned on synthetic agent trajectories collected in CaPtain-Harness. It is the initialisation and the KL reference of the RL stage, which produces CaPtain-4B. For the best robot performance, use CaPtain-4B. Use this checkpoint to reproduce or modify the RL stage, or as an SFT baseline.


Paper	CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents
Project page	https://waltstephen.github.io/captain/
Pipeline code (SFT, RL)	https://github.com/waltstephen/CaPtain
Harness (required to run the model)	https://github.com/waltstephen/CaPtain-Harness
RL model	`YijiaFan/CaPtain-4B`
SFT data	`YijiaFan/CaPtain-SFT-Data`

What the model is

A code-writing agent, not a VLA. The model never outputs actions directly. At every request it writes Python for the harness's single python tool: a persistent, sandboxed Python 3.11 session. The robot API in that session has 22 low-level capabilities (perception, geometric measurement, IK-level guarded motion) plus finish(), and no grasp or place macros.
What it sees: camera images, its own code and output, and one-line receipts of its robot calls.
What it never sees: object poses and the success predicate.
Perception services. SAM 3 segmentation and Molmo2-8B pointing run behind the harness tools.
Format. bf16 safetensors with the Qwen3_5ForConditionalGeneration architecture and an untied lm_head. The tokenizer, processor and chat template are the same as Qwen3.5-4B. The vision tower is unchanged from the base model: it was frozen during SFT.

How to use

Use it exactly like CaPtain-4B, inside CaPtain-Harness.

vllm serve YijiaFan/CaPtain-4B-SFT --served-model-name captain-4b --dtype bfloat16 --max-model-len 81920 \
  --enable-prefix-caching --limit-mm-per-prompt '{"image":12}' --max-num-seqs 16 \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml --port 8600   # vLLM 0.29

# in a CaPtain-Harness checkout (https://github.com/waltstephen/CaPtain-Harness, installed as in INSTALL.md):
services/start_perception.sh                                    # SAM 3 :8114 + Molmo2-8B :8122
export POLICY_URL=http://127.0.0.1:8600/v1 STUDENT_PYTHON=~/envs/student-py311/bin/python EGL_DEVICE=0
eval/run_episode.sh libero_spatial 0 11 runs/demo

The paper's evaluation protocol:

sampling T = 1.0, top_p = 1.0, thinking on;
budgets per episode: 60 model requests, 1,080 s and 150 physical actions;
context: a 64k-token window with compaction.

To start RL from this checkpoint, see rl/README.md in the CaPtain repository.

Prompt format

The model was trained on the harness's exact request template:

the system prompt and environment context;
TASK.md, inlined in the system prompt;
the python tool schema (python-runtime/v1);
receipts, compaction notes and image placement.

The released SFT data show this template row by row. The model uses the standard Qwen3.5 chat template, with <think> reasoning and XML tool calls. Run it through CaPtain-Harness rather than with hand-written prompts, and do not change model-visible text when evaluating it.

Results

Success rates (%) in simulation, host / strict. Strict success also requires that the agent called finish() itself. The LIBERO row uses held-out official initial states 11–16 of the 40 tasks of the four standard suites.

Benchmark	Episodes	Qwen3.5-4B	CaPtain-4B-SFT	CaPtain-4B (after RL)
LIBERO, official held-out states 11–16	240	0.0 / 0.0	55.4 / 52.1	73.8 / 67.5
LIBERO-PRO Pos+Task, 6-cell mean	600	0.3 / 0.2	59.5 / 56.7	78.5 / 71.5
LIBERO-PRO Pos+Task, 8-cell mean	800	0.2 / 0.1	49.2 / 45.1	68.6 / 58.9
LIBERO-Plus, 7 categories	700	0.0 / 0.0	48.6 / 44.6	66.1 / 59.0
LIBERO-PRO, 4 categories	160	0.0 / 0.0	48.8 / 48.1	68.8 / 60.6
LIBERO-MAX Dynamic	80	0.0 / 0.0	40.0 / 40.0	61.2 / 50.0
LIBERO-MAX Base	80	0.0 / 0.0	51.2 / 45.0	70.0 / 63.8

The zero-shot transfer results of the paper (RoboTwin 2.0 22.6 on the official Hard protocol, SimplerEnv WidowX 83.3, CaP-Bench 75.0, RoboDojo Score 1.7) were measured with CaPtain-4B, not with this checkpoint.

Training

Base: Qwen3.5-4B, with the lm_head untied (sft/scripts/untie_base.py).
Data: CaPtain-SFT-Data:
- synthetic agent trajectories collected in CaPtain-Harness, generated with frontier models, plus successful episodes of the student itself;
- visual QA at loss weight 0.5.
Supervision: only assistant turns. A text-hint mask (p = 0.2) replaces earlier mark_point pixel coordinates with <masked>, so the model has to read the image.
Optimisation: full fine-tuning of the language model with the vision tower frozen. Peak LR 1e-5, 30 warmup steps, cosine schedule. One row per rank on 16 ranks, with a constant global per-token loss normaliser. DeepSpeed ZeRO-3, bf16, gradient clipping 1.0.
Length: 2,435 steps (1.5 epochs).

The trainer, launch script and exact configuration are in the CaPtain repository (sft/).

Limitations

Simulation-trained. The SFT trajectories are simulated episodes, almost all on LIBERO. The model was not trained on real-robot data.
Weaker than the RL model. It is clearly worse than CaPtain-4B on every benchmark above. Missed finishes and long-horizon or articulated tasks are its main failures.
Inputs. It depends on the harness's perception services (SAM 3, Molmo2-8B), depth, camera calibration, and the exact prompt and tool format.
One arm at a time. Two-arm coordination is not supported.
Research artifact. Not validated for safety-critical use or for unsupervised use on physical robots.

License

Apache-2.0, the license of the base model Qwen3.5-4B. The SFT data have their own terms; see the dataset card. Components you run with the model (LIBERO, SAM 3, Molmo2, simulators) keep their own licenses.

Citation

@article{fan2026captain,
  title   = {CaPtain: A Pipeline for Post-Training Code-as-Policy Robot Agents},
  author  = {Fan, Yijia and He, Xilin and Jiang, Yilei and Wu, Dongming and Wang, Jiaxu and Yue, Xiangyu},
  journal = {arXiv preprint},
  year    = {2026}
}

Contact: Xiangyu Yue (xyyue@ie.cuhk.edu.hk), MMLab, The Chinese University of Hong Kong.