EgoForesight-RoboTwin

RoboTwin 2.0 · full-data absolute-EEF policy

GitHub — placeholder: https://github.com/<GITHUB_ORG>/<EGOFORESIGHT_REPO>
Citation: BibTeX placeholder

EgoForesight policy for RoboTwin 2.0 (50 tasks), using full data and absolute EEF control.

This is a complete native EgoForesight inference checkpoint. It contains the fine-tuned Qwen3.5-4B backbone, frozen DINOv3 ViT-B/16 encoder, EMA MoT3 weights, and the original representation projector. Use the separate EgoForesight implementation with the retained starVLA namespace and Qwen35MoT3 framework.

Evaluation

Full-data RoboTwin 2.0.

Method	Clean	Randomized	Overall
EgoForesight	91.80%	90.16%	90.98%

Each setting contains 50 tasks × 50 episodes. Clean: 2,295 / 2,500 successes; randomized: 2,254 / 2,500; overall: 4,549 / 5,000. Overall pools the two equal-sized settings; no baseline results are included.

Evaluation uses 10 flow steps, CFG 1.0, visual-timestep gamma 1.0, seed 42, 48 executed actions per chunk, absolute EEF control, and no temporal ensemble. These are existing simulator results, independently of the 4-step export consistency check. The launch command below uses the evaluated 10-step setting; the saved model configuration retains its original 4-step default.

Evaluation details and all 100 task-setting results include the source run ID and SHA256 hashes of the original result logs. The results were checked against their final success counts; no new benchmark was launched.

Model

Property	Value
Architecture	Qwen3.5-4B + DINOv3 ViT-B/16 + MoT3 / Foresight Forcing
Benchmark	RoboTwin 2.0, 50 tasks
Training mixture	robotwin_full_aug_aligned
Weight selection	foresight_forcing EMA for all mot_model.* tensors
Action / state width	130 canonical coordinates
Predicted action chunk	48 timesteps
Default flow inference	4 steps, CFG scale 1.0
Normalization key	robotwin_aug_aligned
EEF representation	Absolute, head-camera OpenCV frame
Joint / gripper representation	Absolute
Export dtypes	Original mixed BF16 / FP32; EMA cast to matching model dtype

Inputs are the instruction, current composite RGB observation, and 14 joint / gripper coordinates (left six joints + gripper, right six joints + gripper). The same joint state is used for EEF and joint control. The native dataset layout is [2, 1, 384, 320, 3] for current/future paired images; inference uses the current frame. Canonical predictions are [batch, 48, 130] and predicted future visual features are [batch, 120, 768].

This checkpoint predicts absolute EEF poses, with enable_relative_action and enable_state_relative_eef both false. The camera_spatial configuration field is retained for compatibility; the actual absolute EEF coordinate frame is robotwin_head_opencv_v1. Use the matching RoboTwin client from EgoForesight, which converts camera-frame targets to simulator world-frame commands.

The native adapter accepts robotwin_control_mode="eef" and returns absolute_actions shaped [batch, 48, 14]: left xyz + rotvec + gripper, then right xyz + rotvec + gripper. In this absolute EEF mode it needs no reference EEF pose. robotwin_control_mode="joint" instead decodes absolute joint targets; the adapter defaults to joint mode, so select eef explicitly for EEF control. The canonical tensor must be decoded by the matching adapter before execution.

Use the native prompt format, for example:

Task: Pick up the bottle. Control Mode: ee absolute. Embodiment: aloha. Reference Frame: camera.

Files

EgoForesight-RoboTwin/
  checkpoints/pytorch_model.pt
  config.yaml
  dataset_statistics.json
  README.md
  CITATION.bib
  evaluation.json
  export_manifest.json
  validation.json
  SHA256SUMS
  requirements.txt
  LICENSE
  NOTICE.md
  licenses/
  .gitattributes
  .gitignore

The checkpoint has 1,627 tensors: 711 EMA replacements and 916 tensors preserved from the original online model. dataset_statistics.json is byte-identical to the training run's statistics. The package contains no optimizer state, EMA sidecar, dataset samples, training logs, or machine-specific asset paths.

Load and serve

Set up the EgoForesight code release following its docs/environment.md. requirements.txt records that implementation's pinned policy/training runtime; install PyTorch 2.8.0 / CUDA 12.8 before CUDA extension dependencies. The simulator uses its own environment. Run the following from the EgoForesight code root:

export PYTHONPATH="$PWD"
export MODEL_DIR=/absolute/path/to/EgoForesight-RoboTwin
export QWEN_MODEL=/absolute/path/to/Qwen3.5-4B
export DINO_REPO=/absolute/path/to/dinov3
export DINO_WEIGHTS=/absolute/path/to/dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth

python -m deployment.model_server.server_policy \
  --ckpt_path "$MODEL_DIR/checkpoints/pytorch_model.pt" \
  --ema_teacher online \
  --use_bf16 --seed 42 --num_inference_timesteps 10 \
  --cfg_scale 1.0 --inference_visual_timestep_alpha 1.0 \
  --host 127.0.0.1 --port 10093

Use --ema_teacher online: this file already contains the merged EMA. Selecting foresight_forcing would request a separate sidecar that is not part of this package.

Obtain the official Qwen3.5-4B files (including tokenizer and processor) and DINOv3 source/checkpoint separately under their terms. The current native constructor loads these assets before strictly restoring the full merged state dictionary, so they are still needed even though their model tensors are included here. QWEN_MODEL defaults to the official Hub ID if it is not set. DINO_REPO and DINO_WEIGHTS are required. Dataset files are not needed for inference.

For Python loading with the same environment variables:

import os
from pathlib import Path

import torch

from starVLA.model.framework.base_framework import baseframework

checkpoint = Path(os.environ["MODEL_DIR"]) / "checkpoints/pytorch_model.pt"
model = baseframework.from_pretrained(str(checkpoint))  # strict state-dict loading
model = model.to(dtype=torch.bfloat16, device="cuda").eval()
policy = model.build_inference_policy()
# Supply examples through the included RoboTwin client / adapter contract.

This native .pt format is loaded by EgoForesight rather than Transformers AutoModel. Retain the checkpoints/ subdirectory and both metadata files at the model root. To inspect tensors without constructing the model, use torch.load(checkpoint, map_location="cpu", weights_only=True, mmap=True).

Verification and provenance

export_manifest.json records source roles and SHA256 digests, EMA merge rules, tensor counts, and configuration changes. All saved tensors were checked against the corresponding source tensor, including shapes, dtypes and finite values. The release preserves the original model dtype for each parameter.

validation.json records strict native loading and a comparison of three real RoboTwin observations against the original online checkpoint plus the native EMA sidecar: normalized actions, predicted future visual features and decoded absolute EEF actions matched exactly (maximum absolute difference 0 on all three observations). It also records action-space compatibility and native adapter preprocessing. Existing audit observations are reused, with joint states recovered using the audit statistics and then normalized with this checkpoint's statistics; prompts are regenerated for absolute EEF control. This is an export/inference check, separate from the evaluation results above. No new simulator benchmark was run during export or packaging.

Verify the downloaded files from the model directory:

sha256sum --check SHA256SUMS

The export is an inference / warm-start artifact. The saved training schedule in config.yaml provides context, but optimizer and scheduler state are absent, and the original FP32 EMA history has been merged into model dtypes. Use the EgoForesight post-training recipes for a new run, with your own dataset location.

License and attribution

EgoForesight additions: Chubin Zhang, built on StarVLA. EgoForesight's original MoT3 / representation-projector weights, configuration and documentation are released under the MIT License.

The model also includes Qwen3.5 and DINOv3 components. Qwen retains Apache-2.0 terms, and the bundled DINOv3 weights retain the DINOv3 license. The MIT license does not replace those upstream terms. See NOTICE.md and the full component licenses in licenses/. Source datasets and simulator assets retain their upstream terms.

Citation

The final paper citation is a placeholder. Update CITATION.bib and the block below together.

% Placeholder only: replace with the final paper citation before publication.
@misc{egoforesight,
  title  = {EgoForesight: Effectively Scaling Policy Learning with Noisy Egocentric Actions},
  author = {TODO: Full author list},
  year   = {TODO},
  url    = {TODO: Paper URL}
}