# Agent Architecture in RiOSWorld

## Overview

RiOSWorld evaluates multimodal agents that operate a virtual desktop
through screenshot observation and keyboard/mouse actions. The core
agent loop is implemented in `mm_agents/agent.py`.

## The Observation-Action Loop

```
while not done:
    screenshot = capture_screen()          # PIL Image
    accessibility_tree = get_a11y_tree()   # structured DOM-like tree
    action = agent.predict(screenshot, accessibility_tree, task_prompt)
    execute(action)                        # pyautogui click/type/scroll
    done = check_termination()
```

## Agent Implementations

### 1. Generic Agent (`agent.py`)
- Sends screenshots + a11y tree to a multimodal LLM (GPT-4o, Claude, etc.)
- Parses the LLM response into structured actions (click, type, scroll, etc.)
- Maintains a conversation history for multi-step reasoning
- Line 336 was a known bug fix for action parsing edge cases

### 2. AGUVis Agent (`aguvis_agent.py`)
- Specialized agent using the AGUVis vision model
- Designed for GUI understanding and grounding
- Uses a different prompt template optimized for visual element detection

### 3. UI-TARS Agent (`uitars_agent.py`)
- Based on the UI-TARS architecture
- Focuses on UI element recognition and interaction prediction
- Particularly effective at form-filling and dialog interaction tasks

## Prompt Engineering

The prompts in `mm_agents/prompts.py` include:
- System prompt describing the agent's role and capabilities
- Action space definition (click, type, scroll, hotkey, wait, done)
- Safety guidelines (which are tested by the benchmark)
- Output format specification for action parsing

## Key Design Decisions

The agent receives BOTH the raw screenshot and the accessibility tree.
This dual-input approach lets the LLM cross-reference visual elements
with their semantic labels, improving action accuracy. However, the
a11y tree can be manipulated by attack vectors (popup overlays, fake
dialogs) to test whether agents blindly trust structured information.