🏔️ ALPS: Visual State Encoders for World Models

ALPS (A Language-aligned, Pixel-renderable State encoder) is a pretrained ViT-B/16 that brings physical-state information, native language alignment, and pixel reconstruction into one set of patch tokens. Its frozen tokens serve as the visual state of world models and robot policies, its native text tower relates instructions and goals stated in words to the scene, and an evaluation decoder renders its tokens back into images.

Project page · Code · Paper

Files

File	Content	Size
alps_vitb16.safetensors	ALPS (Stage 1 + Stage 2): the image encoder and the language interface	786 MB
alps_stage1_vitb16.safetensors	ALPS (Stage 1), before cross-view refinement: the image encoder and the language interface	786 MB
libero_policies/	the eight ALPS policies of the LIBERO experiments (Table 2 of the paper), see below	8 x 35 MB
SHA256SUMS	checksums of every file above

Use

git clone https://github.com/UCSC-VLAA/ALPS.git && cd ALPS
pip install -e .                 # torch, numpy, pillow

import torch, alps
from PIL import Image

model = alps.load("alps-vitb16")                          # downloads alps_vitb16.safetensors from this repository
x = alps.center_crop_transform(Image.open("image.jpg"))   # [1, 3, 224, 224], normalized

with torch.no_grad():
    out = model.encode_image(x)
    z = out["tokens"]            # [1, 196, 768]  raw patch tokens: the visual state
    img = model.image_embedding(tokens=z)                      # L2-normalized
    txt = model.text_embedding(["a robot arm opens the top drawer", "a photo of a dog"])
    probs = (model.contrastive_scale() * img @ txt.T).softmax(-1)

alps.load("alps-stage1-vitb16") loads ALPS (Stage 1). Simulator frames: resize the whole frame to 224 (alps.resize_transform). Use float32 or bfloat16 (float16 overflows in the deeper blocks). More in the repository README.

Model

• Architecture: ViT-B/16 (width 768, 12 blocks, 12 heads, MLP 3,072), class token, fixed 2-D sine-cosine position embedding, no patch-embedding bias, LayerNorm epsilon 1e-6. 85.6M parameters.
• Output: 196 raw patch tokens of width 768 at 224 x 224 pixels (the visual state), without a final LayerNorm.
• Language interface (from Stage 1, unchanged by Stage 2): a 12-layer text encoder of width 768 (109.7M parameters, WordPiece vocabulary, 32 tokens), linear image projection of the mean token, linear text projection, learned contrastive scale.

Training

• Stage 1, from scratch: DataComp-1B images with their web captions and the synthetic captions of the LLaVA-LLaMA-3 captioner of Recap-DataComp-1B. Objectives: image-text contrastive alignment, caption prediction, pixel reconstruction, and cross-view dense correspondence, all on the same patch tokens. 12.8 billion samples at 128 pixels, then 1.0 billion samples at 224 pixels.
• Stage 2, cross-view self-distillation on single images: from an augmented crop, the student predicts the standardized tokens of the frozen Stage 1 encoder on a clean crop of the same image; only patches of the clean crop whose centers lie inside the augmented crop contribute to the loss. 5 million draws from a curated collection of public image-editing datasets, used as still images only (no edit pairs or instructions).
• Still images and text only: no video, no action supervision, no teacher model other than the frozen Stage 1 copy of ALPS itself.

Results (from the paper)

In a comparison with fifteen public encoders, all frozen and evaluated with the same protocols, ALPS has the lowest aggregate LIBERO state-probe error among the ViT-B encoders (.829, relative to ALPS (Stage 1)), matching the 3.5x larger V-JEPA 2-L; its Flickr30k image-to-text recall at 1 (93.2) is within 2 points of the best image-text encoders; and its reconstruction l1 error (.120) is lower than that of every compared encoder not trained to reconstruct pixels or latents. Relative to ALPS (Stage 1), cross-view refinement reduces state-probe error by 17% while changing image-text retrieval by at most 0.5 points.

LIBERO policies

libero_policies/ holds the eight ALPS policies of Table 2: the LIBERO transformer policy trained by behavior cloning on frozen ALPS tokens (50 demonstrations per task), with instructions embedded by LIBERO's BERT or by ALPS's own text tower. Each folder, named by its suite and instruction embedding, has the run's config.json (only experiment_dir and experiment_name follow the folder name) and multitask_model_ep<N>.safetensors, its 8.8M trainable parameters; the frozen encoder is alps_vitb16.safetensors of this repository.

Suite	Instructions	Folder	Success in Table 2 (%)
LIBERO-Long	BERT	libero_policies/LIBERO_10/bert	69.7
LIBERO-Goal	BERT	libero_policies/LIBERO_GOAL/bert	85.1
LIBERO-Spatial	BERT	libero_policies/LIBERO_SPATIAL/bert	86.5
LIBERO-Object	BERT	libero_policies/LIBERO_OBJECT/bert	92.2
LIBERO-Long	ALPS text tower	libero_policies/LIBERO_10/alps_text	78.5
LIBERO-Goal	ALPS text tower	libero_policies/LIBERO_GOAL/alps_text	87.4
LIBERO-Spatial	ALPS text tower	libero_policies/LIBERO_SPATIAL/alps_text	82.6
LIBERO-Object	ALPS text tower	libero_policies/LIBERO_OBJECT/alps_text	93.7

Success is the mean of three evaluations of each policy with 50 rollouts per task. The rollouts sample their actions and the simulator is not seeded, so a new evaluation differs from each of the paper's three about as much as they differ from one another. To evaluate a policy, set up the patched LIBERO as described in eval/libero_policy/README.md of the code repository, then:

huggingface-cli download UCSC-VLAA/ALPS alps_vitb16.safetensors --local-dir alps_hf
huggingface-cli download UCSC-VLAA/ALPS --include "libero_policies/*" --local-dir alps_hf
export LIBERO_ROOT=<patched LIBERO checkout> ALPS_WEIGHTS=$PWD/alps_hf/alps_vitb16.safetensors \
       TASK_EMBEDDINGS_DIR=<ALPS repository>/eval/libero_policy/paper_records/raw/task_embeddings
bash <ALPS repository>/eval/libero_policy/scripts/evaluate_policy.sh \
     alps_hf/libero_policies/LIBERO_OBJECT/bert 0

The files were written from the training checkpoints of the runs with scripts/export_policy_weights.py of the code repository (the frozen network dropped, every other tensor unchanged). For each policy, the frozen network stored in its checkpoint equals, bit for bit, the encoder of alps_vitb16.safetensors with the settings that evaluate_policy.py applies to these configurations.

Intended use and limitations

Intended use: a frozen visual state for world models, robot policies, and probes; image-text retrieval and zero-shot classification through the native text encoder; pixel reconstruction through an evaluation decoder trained on the frozen tokens.

• Evaluated with the ViT-B/16 backbone on simulation benchmarks (LIBERO, DINO-WM tasks, Reacher); real-robot use is untested.
• Pretrained on still images only; fine-grained temporal actions are not its strength.
• The language interface was trained on web and synthetic captions and inherits their biases.

License

Apache-2.0, for the weights in this repository and the code in UCSC-VLAA/ALPS.

Citation

@article{liu2026alps,
  title   = {ALPS: Visual State Encoders for World Models},
  author  = {Liu, Yanqing and Zhang, Letian and Liu, Chengzhi and Wang, Zijun and Mei, Jieru and Fei, Hongliang and Zhou, Yuyin and Xie, Cihang},
  journal = {arXiv preprint arXiv:2610.30120},
  year    = {2026}
}

Acknowledgements

We would like to thank the TPU Research Cloud (TRC) program, the Google Cloud Research Credits program, and the Google TPU Builder Award for supporting our computing needs.