# ComponentBench v0.3.1 Patch Notes

## Overview

This directory contains patched results for ComponentBench v0.3.1. Two canonical
types (`markdown_editor` and `transfer_list`) were broken during the original
full benchmark runs and produced 0% success across webarena and som modes. These
tasks were rerun after fixing the site, and the results were spliced into the
original full-run data.

Pixel mode was rerun cleanly from scratch (all 2910 tasks) and is included
unpatched.

## Run IDs and Provenance

| Item | Value |
|------|-------|
| Base git commit | `b9168d3e3` |
| Patch git commit | `503620025` |
| Model | `Qwen/Qwen3-VL-235B-A22B-Thinking-FP8` |
| Patch run ID | `patch_20260216_180150` |
| Pixel clean run ID | `pixel_20260216_180155` |

### Base run sources (Qwen3-VL-235B, all 2910 tasks each)

- **webarena**: `/usr/project/xtmp/tg295/componentbench_zips/webarena/results.jsonl`
- **som**: `/usr/project/xtmp/tg295/componentbench_zips/som/results.jsonl`
- **pixel_grid**: `/usr/project/xtmp/tg295/componentbench_zips/pixel_grid/results.jsonl`

### Patch source

- `/usr/project/xtmp/tg295/componentbench_rerun_exports/patch_20260216_180150/results_merged.jsonl`
- 240 total rows: 60 per mode (webarena, som, pixel_grid, pixel) x 2 canonical types x ~30 tasks each

### Pixel clean source

- `/usr/project/xtmp/tg295/interfacegym-results/componentbench/pixel_20260216_180155/merged/results_merged.jsonl`

## Canonical Types Replaced

- `markdown_editor` (30 tasks: `markdown_editor-external-T01` through `T30`)
- `transfer_list` (30 tasks: `transfer_list-antd-T01` through `T30`)

## Per-Mode Patch Summary

| Mode | Replaced | Old Success (replaced) | New Success (replaced) | Old Overall | New Overall |
|------|----------|----------------------|----------------------|-------------|-------------|
| webarena | 60 | 0/60 (0.0%) | 27/60 (45.0%) | 2213/2910 (76.05%) | 2240/2910 (76.98%) |
| som | 60 | 0/60 (0.0%) | 20/60 (33.3%) | 1564/2910 (53.75%) | 1584/2910 (54.43%) |
| pixel_grid | 60 | 10/60 (16.7%) | 15/60 (25.0%) | 1418/2910 (48.73%) | 1423/2910 (48.90%) |
| pixel | -- | -- | -- | -- | 1470/2910 (50.52%) |

Pixel mode was not patched; it was rerun cleanly from scratch.

## Files in This Directory

### Patched results (3 modes)

- `{webarena,som,pixel_grid}_results_patched.jsonl` -- 2910 lines each, base rows with 60 replaced
- `{webarena,som,pixel_grid}_summary_patched.json` -- recomputed summary from patched JSONL
- `{webarena,som,pixel_grid}_patch_report.json` -- detailed patch metadata (task IDs, before/after counts)

### Clean pixel results

- `pixel_results_clean.jsonl` -- 2910 lines, clean full rerun
- `pixel_summary_clean.json` -- recomputed summary

### Log archives (excluding screenshots/videos)

- `logs_base_webarena.tar.zst` -- experiment logs from original webarena run
- `logs_base_som.tar.zst` -- experiment logs from original som run
- `logs_base_pixel_grid.tar.zst` -- experiment logs from original pixel_grid run
- `logs_patch_20260216_180150.tar.zst` -- experiment logs from patch rerun (all 4 modes)
- `logs_pixel_20260216_180155.tar.zst` -- experiment logs from clean pixel rerun

## Methodology

Patching was performed by `scripts/patch_componentbench_results.py`:

1. Base JSONL loaded in original row order
2. Patch JSONL filtered by mode and canonical type
3. Each base row with `canonical_type in {markdown_editor, transfer_list}` replaced
   by the matching patch row (exact `task_id` match)
4. Assertions verified: exactly 60 replacements, 2910 output lines, 2910 unique task_ids
5. Summary recomputed from patched data