LoveLogicAI/xros-shell

XROS Shell — a spatial OS for the Meta Quest 3

A working prototype of a spatial operating system shell combining three elements, inspired by the DiOS paper (An Extended Reality Operating System for the Metaverse, arXiv:2201.03256):

1. Spatial computing — panels anchored in your real room (passthrough), driven by a shared world model rather than per-app state.
2. A terminal — a command line you use inside VR, as the primary interface.
3. Generative UI — describe what you want in natural language and the system generates a panel (currently a rule-based stub; an LLM can be wired in later).

Architecture

Quest 3 (Unity)                          Your computer (Python)
┌─────────────────────────┐              ┌──────────────────────────┐
│ WorldPanelManager       │◄──spawn/move─┤  xros_server.py         │
│  (spatial panels)       │   /remove     │   ├─ WorldState         │
│ SpatialShellTerminal    │──command─────►│   └─ SpatialShell       │
│  (terminal panel)       │◄──output──────│      (spawn/move/ls/gen)│
│ XrosClient (WebSocket)  │              │                          │
└─────────────────────────┘              └──────────────────────────┘

The backend is the "OS brain": a shared world-state store plus a spatial shell (command interpreter). The Quest app is a thin client that renders the world and hosts the terminal.

Backend (Python)

File	Role
xros_world.py	World-state store: scene graph + object registry + event bus
xros_shell.py	Spatial shell: command interpreter (spawn, move, ls, gen, clear)
xros_server.py	WebSocket server exposing the world-state + shell API
xros_client.py	Python client library + demo
test_xros.py	End-to-end tests (unit + integration)

Run it:

pip install -r requirements.txt
python xros_server.py --host 0.0.0.0 --port 8765   # serve on your LAN
python test_xros.py                                 # verify it works

Shell commands: help, ls, spawn <type> [x y z] [--title T] [--color C], move <id> x y z, rm <id>, clear, gen <description>.

Frontend (Unity, Quest 3)

File	Role
XrosClient.cs	WebSocket client (Newtonsoft.Json, no other packages)
SpatialShellTerminal.cs	Terminal panel: command input + output log
WorldPanelManager.cs	Renders world events as spatial panels
UNITY_SETUP.md	Step-by-step Unity + Meta XR SDK setup and build guide

See UNITY_SETUP.md for the full assembly instructions (project setup, Meta XR SDK, passthrough, hand tracking, and building to the Quest 3).

Roadmap

• Phase 1 (done): shared world-state service + spatial shell + terminal.
• Phase 2: CV as a system service (shared scene understanding).
• Phase 3: agentic interface layer (AIOS-style scheduling/context).
• Phase 4: spatial developer experience (Code Cubes / structured editing).
• Phase 5: privacy/security at the OS layer (capability-based access).