LCA-on-the-Line Table 2 Reproduction
Domain: computing_math — AI Engineering, Safety & CS Research

Summary: Reproduce the 40-cell correlation table (Table 2) from the ICML 2024 paper "LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies" (arXiv:2407.16067). The provided codebase ships precomputed per-model metric dictionaries sufficient to regenerate all 40 R² / Pearson correlations without running GPU inference, making the task CPU-feasible.

Task prompt:
You are reproducing Table 2 of the ICML 2024 paper "LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies" (arXiv:2407.16067).

## Inputs (under `input/`)

- `paper.pdf` - local copy of the paper; do not re-fetch from arXiv.
- `codebase.zip` - snapshot of the official repo `github.com/ElvishElvis/LCA-on-the-line` at commit 8c0e74d. The archive ships precomputed per-model statistics (`ICML_dict_result_metric_dict`, `ICML_dict_agreement_dict_*.npy`) that encode Top-1 / Top-5 / LCA-distance numbers for 75 pretrained models across the 6 evaluation sets used by the paper. You do NOT need to re-run the full inference pipeline on raw images to reproduce Table 2 - the shipped `main.py` / `draw_plot.py` already consume those cached artifacts.
- `README.md` - short orientation note.

## Software

- `software/python` is a stable Python 3 entry point on the VM.
- The VM is CPU-only. Running 75 pretrained models over five full OOD validation splits is infeasible here; the precomputed-artifact path in the codebase is the intended CPU route.
- You are responsible for any additional Python packages (`torch`, `timm`, `open_clip_torch`, `numpy`, `pandas`, `scipy`, `scikit-learn`, `statsmodels`) that the codebase imports at runtime.

## Your Task

1. Read `paper.pdf` and inspect `codebase.zip` to confirm that Table 2 is the correlation table you are asked to reproduce.
2. Unpack the codebase and run (or adapt) its correlation pipeline. The shipped `ICML_dict_result_metric_dict` plus `ICML_dict_agreement_dict_*.npy` are sufficient to regenerate all 40 Table 2 correlation cells without downloading the OOD validation images themselves.
3. If you do choose to download any of the five ImageNet-OOD datasets listed in the paper (ImageNet-V2, ImageNet-Sketch, ImageNet-R, ImageNet-A, ObjectNet), put them on this VM and list what you actually downloaded in `datasets_downloaded`. Claiming a dataset you did not download will not earn credit - the evaluator will look for it on disk.
4. Write `output/results.json` with the schema below.

## Output Schema

```json
{
  "identified_key_table": "Table 2",
  "datasets_downloaded": ["imagenet-v2", "imagenet-s", "imagenet-r", "imagenet-a", "objectnet"],
  "table2_values": {
    "<ID>_<OOD>_<dataset>_<metric>": <float>,
    ...
  }
}
```

- `identified_key_table` must be exactly `"Table 2"`.
- `datasets_downloaded` must be a JSON array using the tokens `imagenet-v2`, `imagenet-s`, `imagenet-r`, `imagenet-a`, `objectnet` (lower case, hyphenated). List only the OOD splits you actually downloaded.
- `table2_values` holds the 4 row-pairs x 5 datasets x 2 metrics = 40 cells of Table 2. Each key follows `<ID>_<OOD>_<dataset>_<metric>` where:
    - ID in {`Top1`, `LCA`} - the in-distribution (ImageNet) indicator.
    - OOD in {`Top1`, `Top5`} - the out-of-distribution accuracy target.
    - dataset in {`ImgNv2`, `ImgNS`, `ImgNR`, `ImgNA`, `ObjN`}.
    - metric in {`R2`, `PEA`} - coefficient of determination and Pearson correlation. Per the paper's Table 2 caption, report the absolute value of Pearson.
  Example: `Top1_Top1_ImgNv2_R2 = 0.962`, `LCA_Top1_ObjN_PEA = 0.956`.

## Rules

- Write only under `output/`.
- Do not modify anything under `input/`.
- `results.json` must be valid UTF-8 JSON.

The agent must:
Read `paper.pdf` and identify Table 2 as the correlation table (R² and Pearson between ID LCA/Top1 and OOD Top1/Top5 across five ImageNet-OOD datasets and 75 pretrained models).
Unpack `input/codebase.zip` and use the precomputed `ICML_dict_result_metric_dict` + `ICML_dict_agreement_dict_*.npy` artifacts to regenerate the 40 Table 2 correlation cells without re-running 75-model inference (the VM is CPU-only).
Optionally download any of the five ImageNet-OOD validation splits you want to claim credit for; the evaluator verifies claimed datasets against on-disk directories before counting them.
Write `output/results.json` with `identified_key_table`, `datasets_downloaded`, and 40 `table2_values` cells using the exact key naming convention `<ID>_<OOD>_<dataset>_<metric>` documented in the prompt.
Use absolute-value Pearson correlations for PEA cells (per the paper's Table 2 caption) and raw R² values for R2 cells.

Software: Python
PyTorch
NumPy
pandas
SciPy
scikit-learn
statsmodels

Input files:
name: Paper PDF
format: pdf
description: ICML 2024 paper "LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies" (arXiv:2407.16067v1). Local copy.
name: Codebase snapshot
format: zip
description: Snapshot of `github.com/ElvishElvis/LCA-on-the-line` @ 8c0e74d. Ships `main.py`, `draw_plot.py`, `ICML_dict_result_metric_dict` (per-model Top-1/Top-5/LCA stats), and `ICML_dict_agreement_dict_*.npy` - enough to regenerate Table 2 without GPU inference.
name: Task README
format: md
description: Agent-facing orientation note.