RiOSWorld Accessibility Tree Processing
=========================================

Directory: mm_agents/accessibility_tree_wrap/
Also related: mm_agents/gui_som/

The accessibility tree (a11y tree) is a structured representation of
all UI elements currently visible on the desktop. RiOSWorld extracts
this tree using the AT-SPI (Assistive Technology Service Provider
Interface) protocol available on Linux desktops.

How it works:
  1. The wrapper connects to the AT-SPI bus via D-Bus
  2. It traverses the UI element hierarchy starting from the root
  3. Each element is recorded with:
     - Role (button, text_field, menu_item, label, etc.)
     - Name (the visible text or aria-label)
     - Bounding box (x, y, width, height on screen)
     - State (focused, checked, enabled, visible, etc.)
     - Value (for input fields, sliders, etc.)
  4. The tree is serialized as a nested dict/JSON structure
  5. It is then flattened into a text representation for the LLM

Example flattened output:
  [1] button "Submit" (320, 450, 80, 30) enabled
  [2] text_field "Email" (200, 300, 200, 25) focused value=""
  [3] label "Enter your credentials" (200, 260, 250, 20)
  [4] button "Cancel" (420, 450, 80, 30) enabled

The GUI-SOM (Set-of-Marks) module overlays numbered markers on the
screenshot corresponding to each interactive element. This visual
grounding helps the agent understand which element corresponds to
which a11y tree entry.

Security implications:
  - Attack popups inject fake a11y tree nodes
  - Phishing pages create misleading element names
  - The agent must cross-reference visual appearance with a11y data
  - Discrepancies between screenshot and a11y tree are a red flag

Known limitations:
  - Some GTK4 apps have incomplete a11y tree exposure
  - Electron apps may not expose all custom widgets
  - Web content inside Chrome has a separate a11y tree via ChromeDriver
  - Fullscreen games/media players typically have no a11y tree