XY-VLA

Code for paper: XY-VLA: Zero-Shot Cross-Embodiment UAV VLA via X-VLA.

By using the commands listed in the command.md file, can fast run XY-VLA.

Language-guided portal navigation in PyBullet using the X-VLA vision-language-action model. Fly a virtual end-effector through numbered rectangular portals, orbit frames, hover, or execute multi-leg missions from natural-language --cmd instructions.

Built on X-VLA 0.9B (WidowX Edition) with EE6D action semantics inside a WidowX arm workspace, extended with a three-phase navigation stack (coarse X-VLA planning → top-down A* → action-aware trajectory refinement).

Features

• Natural-language missions — billboard_id=N portal targeting, multi-clause chains (then, ;), and five basic actions: pass-through, fly-by, orbit, hover, collision (debug).
• Three-phase navigation — Phase 1 corridor + X-VLA coarse path, Phase 2 grid A* with portal-aware obstacles, Phase 3 per-object refinement and translucent feedback spheres.
• Closed-loop or coarse-plan modes — Per-step /act inference, or one-shot Phase 1+2+3 planning with optional simulation replay (no inference during replay).
• Rich simulation scene — 20 numbered rect_frame portals on upper/lower rings with random tilt; cubes, ramps, and local obstacle avoidance.
• Recording & visualization — Multi-view MP4/GIF under recordings/, top-down overlays, Phase 3 color feedback, optional GUI.
• Configurable schemes — config.json presets for GUI debug, fast batch runs, and demo playback (--scheme 1|2|3|widowx_ee6d).

Test Result

1-1_Fly through the yellow opening

1-2_Pass through the nearest red portal

2-1_Fly a figure-eight path in the air over the workspace

2-2_Fly a racetrack oval in the air over the workspace

Prerequisites

• Python 3.10 (conda recommended)
• CUDA GPU recommended for X-VLA inference (CPU works for smoke tests)
• Anaconda or Miniconda
• X-VLA model weights in xVLAModel/ (see Model setup)

Installation

1. Clone the repository

git clone <your-repo-url>
cd XY-VLA

2. Create the environment

One-click setup (creates conda env xy-vla, installs PyTorch and dependencies):

python set_enviroment_simple.py

Or with live pip output:

python set_enviroment.py

Recreate the environment from scratch:

python set_enviroment_simple.py --recreate

3. Model setup

Place the X-VLA WidowX checkpoint in xVLAModel/:

# Example: download from Hugging Face
# huggingface-cli download 2toINF/X-VLA-WidowX --local-dir xVLAModel

Required files include model.safetensors, config.json, modeling_xvla.py, processing_xvla.py, and tokenizer assets. Verify with:

python test_xVLA.py

Quick start

Single portal (language command)

python run_xyVLA.py \
  --cmd "Fly through only the rectangular portal marked billboard_id=3" \
  --sim-steps 200 \
  --infer-every 6 \
  --xvla-steps 4 \
  --speed 5

The demo auto-starts a local X-VLA server when auto_start_xvla_server is enabled in config.json.

With GUI

python run_xyVLA.py --gui --scheme 1 \
  --cmd "Fly through billboard_id=3 only"

Sequential 20-portal task (no --cmd)

Uses task_sequence from config.json (schemes.widowx_ee6d):

python run_xyVLA.py --scheme widowx_ee6d

When --cmd is provided, a non-empty task_sequence in JSON is ignored.

Multi-leg mission + Phase 3 refinement

python run_xyVLA.py \
  --cmd "Pass through portal billboard number 2 then pass through portal billboard number 3, then pass through portal billboard number 5" \
  --sim-steps 200 \
  --infer-every 6 \
  --xvla-steps 4

With default cmd_coarse_plan_once: true, this runs Phase 1+2+3 planning. Set navigation_use_phase3_refined_path_in_sim: true to replay the refined path in simulation without further /act calls.

Configuration

Main file: config.json

Key / scheme	Description
scheme	Top-level preset: 1 (GUI debug), 2 (fast headless), 3 (demo playback), or widowx_ee6d (full workspace).
--scheme	CLI override for top-level scheme.
schemes.widowx_ee6d	Workspace bounds, cubes, task_sequence, navigation flags, motion scales.
cmd_coarse_plan_once	Run Phase 1+2+3 once instead of closed-loop main sim.
navigation_use_phase3_refined_path_in_sim	Replay Phase 3 path in main simulation.
auto_start_xvla_server	Spawn local /act server on startup.

See inline _readme_* keys in config.json for detailed option documentation.

Project structure

XY-VLA/
├── run_xyVLA.py              # Main entry: PyBullet sim + X-VLA client
├── xvla_local_server.py      # Local X-VLA server helpers
├── config.json               # Schemes and runtime defaults
├── command.md                # Extended command examples and test matrix
├── test_xVLA.py              # Model + server smoke test
├── set_enviroment.py         # Conda environment setup (verbose)
├── set_enviroment_simple.py  # Conda environment setup (compact)
├── algorithms/
│   ├── phase1.py             # Phase 1 corridor + coarse planning
│   ├── phase2.py             # Phase 2 top-down A*
│   ├── phase3.py             # Phase 3 refinement + feedback zones
│   ├── phase3_actions.py     # Basic action geometry + keyword parsing
│   ├── phase3_xvla_actions.py# X-VLA action classification
│   ├── multi_leg.py          # Multi-leg mission stitching
│   ├── instruction_parse.py  # --cmd clause / billboard_id parsing
│   ├── portal_geometry.py    # Portal opening / collision geometry
│   └── phase_recording.py    # Recording and overlay rendering
├── xVLAModel/                # X-VLA WidowX weights + model code
├── recordings/               # Output videos and phase JSON artifacts
└── results.md                # Test-result gallery (four-view GIFs)

Portal billboard IDs

The default scene places 20 rect_frame portals (billboard_id 1–20) on upper and lower rings. Prefer explicit IDs in instructions:

Fly through only billboard_id=18
Pass through portal billboard number 12 only
Orbit billboard_id=3 twice for inspection, then pass through billboard_id=10

Color-only commands work for legacy compatibility but may be ambiguous when two portals share a color on different layers.

Common CLI flags

Flag	Purpose
--cmd	Natural-language mission (overrides task_sequence).
--config	Path to config.json.
--scheme	Config scheme key (1, 2, 3, widowx_ee6d).
--sim-steps	Simulation horizon.
--infer-every	Steps between X-VLA /act calls.
--xvla-steps	Action horizon per /act request.
--gui / --no-gui	PyBullet viewer.
--no-local-avoidance	Disable near-field obstacle repulsion.
--no-xvla-scene-catalog	Disable scene object catalog in instructions.
--cmd-motion-amplify	Scale language-only target displacement.
--infer-displacement-scale	Scale decoded target displacement per inference.

Full examples: see command.md.

Recording output

When record_visualization is enabled, outputs are written under recordings/<timestamp>/ (or a named mission folder):

• sim/ — main simulation multi-view video / GIF
• phase1/, phase2/, phase3/ — planning artifacts, overlays, and path JSON

Recorded missions are collected in results.md: 1-1 ~ 2-26 show four GIFs from the mission folder; 3-1 ~ 3-15 show four GIFs from each folder's sim/ directory.