EgoForesight-RobocasaGR1

RoboCasa GR1 tabletop · full-data policy

GitHub — placeholder: https://github.com/<GITHUB_ORG>/<EGOFORESIGHT_REPO>
Citation: BibTeX placeholder

EgoForesight policy for RoboCasa GR1 tabletop (24 tasks), trained with full data.

This is a complete native EgoForesight inference checkpoint. It contains the fine-tuned Qwen3.5-4B backbone, frozen DINOv3 ViT-B/16 encoder, EMA MoT3 weights, and the original representation projector. Use the separate EgoForesight implementation with the retained starVLA namespace and Qwen35MoT3 framework.

Evaluation

RoboCasa GR1 tabletop — full data (1,000 demonstrations per task).

Method	Pick-and-place	Articulated objects	Overall
EgoForesight	71.9%	80.0%	73.9%

The suite contains 18 pick-and-place + 6 articulated-object tasks. These are the reported full-data results supplied for this release, not a new benchmark of the repackaged checkpoint. Only EgoForesight results are shown. See evaluation.json for the source and scope; the loading example below is not a claim about the original table's inference protocol.

Model

Property	Value
Architecture	Qwen3.5-4B + DINOv3 ViT-B/16 + MoT3 / Foresight Forcing
Benchmark	RoboCasa GR1 tabletop, 24 tasks
Training mixture	robocasa_1000_aug_aligned
Weight selection	foresight_forcing EMA for all mot_model.* tensors
Action / state width	130 canonical coordinates
Predicted action chunk	16 timesteps
Default flow inference	4 steps, CFG scale 1.0
Normalization key	robocasa_gr1_aug_aligned
EEF representation	Relative, camera_spatial
Joint / fingertip representation	Absolute
Export dtypes	Original mixed BF16 / FP32; EMA cast to matching model dtype

Inputs are the instruction, current composite RGB observation, and GR1 state with the corresponding masks and embodiment metadata. The native dataset layout is [2, 1, 384, 320, 3] for current/future paired images; inference uses the current frame. Canonical predictions are [batch, 16, 130] and include predicted future visual features. The supplied action-space adapter handles normalization and decodes the 29-coordinate absolute joint output. Its aligned EEF/joint mode additionally requires current EEF reference poses.

Use the matching GR1 simulator client from EgoForesight. The canonical action tensor is not a direct simulator command. This checkpoint targets the GR1 tabletop benchmark, not RoboCasa365.

Files

EgoForesight-RobocasaGR1/
  checkpoints/pytorch_model.pt
  config.yaml
  dataset_statistics.json
  README.md
  CITATION.bib
  evaluation.json
  export_manifest.json
  validation.json
  SHA256SUMS
  requirements.txt
  LICENSE
  NOTICE.md
  licenses/
  .gitattributes
  .gitignore

The checkpoint has 1,627 tensors: 711 EMA replacements and 916 tensors preserved from the original online model. dataset_statistics.json is byte-identical to the training run's statistics. The package contains no optimizer state, EMA sidecar, dataset samples, training logs, or machine-specific asset paths.

Load and serve

Set up the EgoForesight code release following its docs/environment.md. requirements.txt records that implementation's pinned policy/training runtime; install PyTorch 2.8.0 / CUDA 12.8 before CUDA extension dependencies. The simulator uses its own environment. Run the following from the EgoForesight code root:

export PYTHONPATH="$PWD"
export MODEL_DIR=/absolute/path/to/EgoForesight-RobocasaGR1
export QWEN_MODEL=/absolute/path/to/Qwen3.5-4B
export DINO_REPO=/absolute/path/to/dinov3
export DINO_WEIGHTS=/absolute/path/to/dinov3_vitb16_pretrain_lvd1689m-73cec8be.pth

python -m deployment.model_server.server_policy \
  --ckpt_path "$MODEL_DIR/checkpoints/pytorch_model.pt" \
  --ema_teacher online \
  --use_bf16 --seed 42 --num_inference_timesteps 4 \
  --host 127.0.0.1 --port 10093

Use --ema_teacher online: this file already contains the merged EMA. Selecting foresight_forcing would request a separate sidecar that is not part of this package.

Obtain the official Qwen3.5-4B files (including tokenizer and processor) and DINOv3 source/checkpoint separately under their terms. The current native constructor loads these assets before strictly restoring the full merged state dictionary, so they are still needed even though their model tensors are included here. QWEN_MODEL defaults to the official Hub ID if it is not set. DINO_REPO and DINO_WEIGHTS are required. Dataset files are not needed for inference.

For Python loading with the same environment variables:

import os
from pathlib import Path

import torch

from starVLA.model.framework.base_framework import baseframework

checkpoint = Path(os.environ["MODEL_DIR"]) / "checkpoints/pytorch_model.pt"
model = baseframework.from_pretrained(str(checkpoint))  # strict state-dict loading
model = model.to(dtype=torch.bfloat16, device="cuda").eval()
policy = model.build_inference_policy()
# Supply examples through the included RoboCasa GR1 client / adapter contract.

This native .pt format is loaded by EgoForesight rather than Transformers AutoModel. Retain the checkpoints/ subdirectory and both metadata files at the model root. To inspect tensors without constructing the model, use torch.load(checkpoint, map_location="cpu", weights_only=True, mmap=True).

Verification and provenance

export_manifest.json records source roles and SHA256 digests, EMA merge rules, tensor counts, and configuration changes. All saved tensors were checked against the corresponding source tensor, including shapes, dtypes and finite values. The release preserves the original model dtype for each parameter.

validation.json records strict native loading and a comparison of three real RoboCasa samples against the original online checkpoint plus the native EMA sidecar: both normalized actions and predicted future visual features matched exactly (maximum absolute difference 0 on all three samples). It also records action-space compatibility and joint decoding. This is an export/inference check, separate from the evaluation results above. No new simulator benchmark was run during export or packaging.

Verify the downloaded files from the model directory:

sha256sum --check SHA256SUMS

The export is an inference / warm-start artifact. The saved training schedule in config.yaml provides context, but optimizer and scheduler state are absent, and the original FP32 EMA history has been merged into model dtypes. Use the EgoForesight post-training recipes for a new run, with your own dataset location.

License and attribution

EgoForesight additions: Chubin Zhang, built on StarVLA. EgoForesight's original MoT3 / representation-projector weights, configuration and documentation are released under the MIT License.

The model also includes Qwen3.5 and DINOv3 components. Qwen retains Apache-2.0 terms, and the bundled DINOv3 weights retain the DINOv3 license. The MIT license does not replace those upstream terms. See NOTICE.md and the full component licenses in licenses/. Source datasets and simulator assets retain their upstream terms.

Citation

The final paper citation is a placeholder. Update CITATION.bib and the block below together.

% Placeholder only: replace with the final paper citation before publication.
@misc{egoforesight,
  title  = {EgoForesight: Effectively Scaling Policy Learning with Noisy Egocentric Actions},
  author = {TODO: Full author list},
  year   = {TODO},
  url    = {TODO: Paper URL}
}