VishalN17/ar-scene-grapher

AR Scene Graph Builder — OpenEnv Environment

AR spatial placement RL environment simulating the object-anchoring decisions made by Meta Ray-Ban / Orion AR glasses. Trains an agent to place virtual AR objects (notifications, screens, arrows, panels, widgets) into physical rooms with optimal gaze alignment, spatial validity, occlusion avoidance, and ergonomic comfort.

---

Overview

Property	Value
Domain	Augmented Reality / Spatial Computing
Action Space	{action_type, position, scale, anchor_surface_id, orientation, layer}
Observation	Room + users + placed objects + pending object
Reward Range	[0.0, 1.0] — shaped at every step
Tasks	3 tasks (Easy → Hard), 3 cases each
API	FastAPI on port 7860

---

RL Problem

The agent receives a description of a physical room (surfaces, furniture, users) and must decide where and how to place virtual AR objects. It is scored on five weighted criteria:

Criterion	Description
spatial_validity	Object inside room, not overlapping furniture
gaze_alignment	Angle between user gaze and object direction
occlusion	How much existing objects block the new placement
comfort_zone	Object within user's ergonomic comfort cone
priority_respect	High-priority objects placed in the sweet spot

---

Project Structure

ar-scene-graph-env/
│
├── Dockerfile
├── openenv.yaml
├── requirements.txt
├── README.md
├── inference.py
│
├── env/
│   ├── __init__.py
│   ├── models.py        ← Pydantic v2 data models
│   ├── tasks.py         ← Hardcoded task / case data (9 cases total)
│   ├── environment.py   ← ARSceneGraphEnv class
│   ├── scene_graph.py   ← Pure-numpy 3D geometry helpers
│   ├── reward.py        ← Shaped reward computation
│   └── graders.py       ← Episode-level graders (1 per task)
│
├── api/
│   ├── __init__.py
│   └── server.py        ← FastAPI app
│
└── tests/
    ├── test_env.py
    └── test_graders.py

---

Tasks

Task 1 — Easy (task1_easy)

• Room: 4 × 3 × 3 m, empty
• 1 user, 1 notification bubble to place
• Budget: 5 steps
• Reward threshold: 0.6

Task 2 — Medium (task2_medium)

• Room: 6 × 4 × 4 m, 3 furniture items
• 1 user, 4 mixed AR objects (virtual screen, notification, arrow, widget)
• Budget: 15 steps
• Reward threshold: 0.5

Task 3 — Hard (task3_hard)

• Room: 8 × 4 × 6 m, 8 furniture items
• 2 users with different positions and gaze vectors
• 6 objects (some shared between users, some user-specific)
• Budget: 25 steps
• Reward threshold: 0.4

---

API

Start the server

uvicorn api.server:app --host 0.0.0.0 --port 7860

POST /reset

{
  "task_id": "task1_easy",
  "case_index": 0,
  "seed": 42
}

All fields optional. Returns ResetResult with initial Observation.

POST /step

{
  "action_type": "place",
  "position": [2.0, 1.5, 2.8],
  "scale": 1.0,
  "anchor_surface_id": "wall_n",
  "orientation": [0.0, 0.0, 0.0],
  "layer": 0
}

Returns StepResult with reward (∈ [0, 1]), done, and per-component info.

GET /state

Returns full StateResult — works even before reset.

GET /health

{"status": "ok", "version": "1.0.0"}

---

Inference

Set environment variables and run:

export HF_TOKEN=<your-api-key>
export MODEL_NAME=gpt-4o-mini
export SERVER_URL=http://localhost:7860
python inference.py

Log format (exact):

[START] task=task1_easy_case0 env=ar-scene-graph-env model=gpt-4o-mini
[STEP] step=1 action=place reward=0.82 done=True error=None
[END] success=True steps=1 score=0.82 rewards=[0.82]

---

Docker

# Build
docker build -t ar-scene-graph-env .

# Run
docker run -p 7860:7860 ar-scene-graph-env

---

Testing

pip install -r requirements.txt
pytest tests/ -v

---

Tech Stack

• Python 3.10+
• FastAPI 0.110 + Uvicorn 0.29
• Pydantic v2 — full type annotations, model validators
• NumPy — all 3D geometry (no scipy / trimesh / etc.)
• OpenAI SDK — LLM inference client
• pytest + httpx — testing

---

Reward Details

reward = Σ(weight_i × score_i) − penalties
reward = clamp(round(reward, 4), 0.0, 1.0)

Penalties:
  skip action          −0.20
  scale out of [0.5,2] −0.10
  overlaps placed obj  −0.15
  placed behind user   −0.25

All geometry is pure NumPy. EPSILON = 0.001 for all float comparisons.

---

Design Notes

• state() never crashes — returns safe defaults before reset() is called.
• step() never raises — all exceptions caught, returns safe fallback.
• reset() is fully reproducible: same (task_id, case_index, seed) always produces identical observations.
• After done=True, all subsequent step() calls return reward=0.0.
• Scale is auto-corrected to [0.5, 2.0]; layer to [0, 10].