Running RiOSWorld Multi-Environment Evaluation
================================================

RiOSWorld supports parallel evaluation across multiple Docker
containers using the multi_evaluation.sh and multi_llm_run.sh
scripts. Here are practical notes for running at scale.

Prerequisites:
  - Docker installed with sufficient disk space (each container ~4GB)
  - NVIDIA Container Toolkit if using GPU-based agents
  - conda environment "RiOSWorld" with all dependencies
  - API keys exported: OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.

Single Agent Run:
  python run.py \
    --agent_type gpt4o \
    --test_file evaluation_risk_examples/test_phishing_email.json \
    --output_dir results/gpt4o_phishing/ \
    --max_steps 20 \
    --headless

Multi-Agent Batch:
  bash multi_llm_run.sh
  This script iterates over all agent types and all test categories.
  Results are saved to results/{agent}/{category}/ directories.

Multi-Environment Parallel:
  bash multi_evaluation.sh --concurrency 4
  Spawns 4 Docker containers simultaneously, each running a
  different test case. Reduces total wall-clock time by 4x.

Monitoring:
  docker ps                        # check running containers
  docker logs riosworld_eval_0     # view container logs
  tail -f results/*/progress.log   # watch evaluation progress

Common failures and fixes:
  - Container OOM: Increase --memory flag or reduce concurrency
  - VNC connection timeout: Increase sleep duration in lib_run_single.py
  - Agent API rate limit: Add retry logic or reduce concurrency
  - Screenshot blank: Ensure Xvfb started correctly in container
  - a11y tree empty: Restart at-spi2-registryd inside container

Post-evaluation:
  python evaluate/safety_evaluation.py \
    --results_dir results/ \
    --output_file summary.csv

  python evaluate/data_process.py \
    --results_dir results/ \
    --generate_latex_table