agents-last-exam/agents-last-exam huggingface dataset:
Title: Sanding Performance Image Scoring
Domain: engineering — Manufacturing & Industrial Systems

Summary: Compute pixel-level paint-removal scores from reference and sanded panel images, then write a structured CSV with quantity, uniformity, and composite metrics.

Task prompt:
You are completing a sanding-performance image scoring task on a Windows VM.

Input files are staged in:
`base\input`

Read `algorithm_spec.md` in that directory. It defines how to use the reference images `panel_painted.png` and `panel_unpainted.png` to score:
- `panel_sanded_01.png`
- `panel_sanded_02.png`
- `panel_sanded_03.png`

Use RGB image values and follow the formulas in `algorithm_spec.md`.

Write exactly one output file:
`base\output\sanding_scores.csv`

The CSV must have this header:
`panel_id,quantity_pct,uniformity_pct,composite_pct`

It must contain exactly one row for each sanded panel:
- `panel_sanded_01`
- `panel_sanded_02`
- `panel_sanded_03`

Round the three numeric metric columns to one decimal place. Do not modify the files under `input/`.


The agent must:
Read the visible algorithm specification and five panel images from the staged input directory.
Compute quantity, uniformity, and composite sanding scores for each of the three sanded panels.
Write `sanding_scores.csv` under the runtime output directory with the required schema and one row per sanded panel.

Software: Python

Input files:
name: algorithm_spec.md
format: Markdown
path: input/algorithm_spec.md
description: Visible task algorithm and output schema.
name: panel_painted.png
format: PNG image
path: input/panel_painted.png
description: Painted reference panel.
name: panel_unpainted.png
format: PNG image
path: input/panel_unpainted.png
description: Unpainted reference panel.
name: panel_sanded_01.png, panel_sanded_02.png, panel_sanded_03.png
format: PNG images
path: input/
description: Sanded panel images to score.
Title: Clinical Variant Annotation
Domain: health_medicine — Genomics & Sequence Analysis

Summary: Analyze a staged clinical VCF, identify the pathogenic BRCA1-like candidate, and document the supporting evidence in the required output tables.

Task prompt:
You are given a small clinical VCF containing 200 candidate variants from a breast-cancer case.

Your task is to identify the single pathogenic BRCA1-like candidate and document the evidence in the required output files.

Task directory:
- `base`

Input:
- `base/input/patient_variants.vcf`

Available tools:
- a Linux terminal is open at `base`
- `python3`
- `curl`
- `jq`

External resources:
1. gnomAD GraphQL API
2. Ensembl VEP REST API
3. NCBI ClinVar / E-utilities

Required outputs under `base/output`:
- `variant_count.txt` with a single integer
- `vep_results.csv`
- `gnomad_results.csv`
- `clinvar_results.csv`
- `final_candidates.csv`

Rules:
- Append rows to the pre-created CSV files without changing headers.
- It is acceptable to use `NA` for unavailable fields.
- The final candidate file must include the pathogenic evidence in the justification column.
- Do not ask for confirmation. Execute directly.

The agent must:
Read a small patient VCF, count variants, annotate candidates against public knowledge bases, and write a final pathogenic BRCA1-like candidate.
Write outputs only under `output/` while keeping committed fixtures untouched.
Work directly on Ubuntu; Windows + WSL is no longer the canonical runtime.


Input files:
name: patient_variants.vcf
format: vcf
path: input/patient_variants.vcf
Title: IHDP ITE Estimation
Domain: health_medicine — Economics & Quantitative Social Research

Summary: Build a reusable hidden-test inference script for the IHDP semi-synthetic benchmark that estimates individualized treatment effects and emits the required diagnostics artifacts.

Task prompt:
You are working on a Linux causal-inference benchmark.

Read these staged solve-time inputs first:
- `base/input/task_brief.md`
- `base/input/task_readme.md`
- `base/input/feature_schema.json`
- `base/input/train.csv`
- `base/input/dummy_test/test.csv`

If you want the pinned scientific Python stack, use:
- `base/software/python_with_task_deps.sh`

Your job is to author `base/output/predict.sh` so that the evaluator can later run:

`bash output/predict.sh <hidden_test_dir> output/output.csv`

from the task root.

When your script runs, it must create these required files:
- `base/output/output.csv`
- `base/output/artifacts/model_selection.csv`
- `base/output/artifacts/overlap_diagnostics.csv`
- `base/output/artifacts/subgroup_ite.csv`
- `base/output/artifacts/run_notes.txt`

`output/output.csv` must have exactly these columns:
`replication,unit_id,mu0_hat,mu1_hat,ite_hat`

Rules:
- `ite_hat` must equal `mu1_hat - mu0_hat` row by row.
- You may keep helper code under `output/` and may emit extra artifact files if you want.
- Do not modify files under `input/`.
- Write solver-created files only under `base/output`.


The agent must:
Read the staged IHDP training data, schema, benchmark brief, and task readme from `input/`.
Build a reusable individualized treatment-effect estimator over the 100 IHDP replications.
Author the required `output/predict.sh` prediction entrypoint.
Write `output/output.csv` plus the required diagnostics artifacts under `output/artifacts/`.

Software: Python
uv
bash
numpy
pandas
scipy
scikit-learn

Input files:
name: task_brief.md
format: Markdown
path: input/task_brief.md
description: Canonical agent-facing task brief with the exact `predict.sh` delivery contract.
name: task_readme.md
format: Markdown
path: input/task_readme.md
description: Normalized solve-time instructions derived from the original submission materials.
name: feature_schema.json
format: JSON
path: input/feature_schema.json
description: Covariate schema and feature metadata.
name: train.csv
format: CSV
path: input/train.csv
description: IHDP observational training data with factual outcomes across 100 replications.
name: dummy_test/test.csv
format: CSV
path: input/dummy_test/test.csv
description: Schema-and-order stand-in for the hidden official test set.
name: runtime_env/
format: Python runtime bundle
path: input/runtime_env/
description: Pinned offline scientific-runtime manifest and wheelhouse exposed through `software/python_with_task_deps.sh`.
Title: CRF To SDTM Mapping Specification 4
Domain: health_medicine — Clinical Research & Trial Operations

Summary: Create field-level mapping CSVs from clinical CRF and define.xml source documents to SDTM AE/SUPPAE or DM/SUPPDM variables.

Task prompt:
You are preparing a field-level CRF-to-SDTM mapping specification on Linux.

## Your Task
Create the Adverse Events mapping for study C4591001.

## Visible Inputs
- Task brief: `base/input/task_brief.md`
- Output contract: `base/input/output_contract.json`
- Source documents directory: `base/input/source_documents`
- Optional Python runtime manifest: `base/input/runtime_env`

The source documents include the sample CRF PDF, annotated CRF PDF, SDTM define.xml,
and supplemental define.xml material. Use those files to identify form fields and
their target SDTM variables for `AE` and, where applicable,
`SUPPAE` supplemental qualifiers.

## Required Output
Save exactly one CSV file:

```text
base/output/ae_mapping.csv
```

The CSV must follow the column order and constraints in `base/input/output_contract.json`.
`goes_to_suppqual` must mark AE rows as NO and SUPPAE rows as YES.

## Constraints
- Produce a mapping specification only, not subject-level records or XPT datasets.
- Keep all generated files inside `base/output`.
- Do not modify files under `base/input`.
- Use the visible task files only; do not use external web sources.


The agent must:
Read the variant's task brief and output contract.
Inspect the sample CRF, annotated CRF, SDTM define.xml, and supplemental define.xml source documents.
Identify CRF field-level mappings for the variant's primary SDTM dataset and supplemental qualifier dataset.
Write exactly one mapping CSV with the required columns in the selected output directory.

Software: Python
uv
pandas
pypdf
lxml

Input files:
name: task_brief.md
format: md
path: input/task_brief.md
description: Variant-specific instructions and output filename.
name: output_contract.json
format: json
path: input/output_contract.json
description: Required CSV columns, allowed datasets, and format constraints.
name: source_documents
format: pdf/xml
path: input/source_documents/
description: Sample CRF, annotated CRF, SDTM define.xml, and supplemental define.xml files.
name: runtime_env
format: pyproject/uv.lock
path: input/runtime_env/
description: Optional Python dependency manifest for agents that want pandas, pypdf, or lxml.
Title: NSCLC Group-Sequential SAP
Domain: health_medicine — Clinical Research & Trial Operations

Summary: Produce a statistical analysis plan package for a Phase III NSCLC trial with log-rank sample size, O'Brien-Fleming interim boundaries, futility, Hochberg secondary testing, and plots.

Task prompt:
You are preparing a clinical-trial Statistical Analysis Plan on a Linux VM.

## Task Folder
`base`

## Inputs
- Protocol JSON: `base/input/protocol.json`
- Detailed instructions: `base/input/task_instructions.md`

## Goal
Create the full SAP deliverable set in:
`base/output`

Required output filenames:
- `SAP.md`
- `analysis.R`
- `sample_size.json`
- `boundaries.csv`
- `multiple_testing.json`
- `power_curve.csv`
- `power_curve.png`
- `boundary_plot.png`

Read the protocol and instructions, compute the group-sequential design quantities, document the SAP, and include a reproducible R script. Do not use internet access.

The agent must:
Read `input/protocol.json` and `input/task_instructions.md`.
Compute the sample size, interim monitoring boundaries, futility rule, multiplicity allocation, and power curve.
Write the required JSON, CSV, PNG, Markdown, and R script outputs under `output/`.
Use only the staged protocol and instructions; do not use internet access.

Software: R
gsDesign
rpact
ggplot2

Input files:
name: Protocol
format: json
path: input/protocol.json
description: Sanitized self-contained NSCLC trial protocol parameters.
name: Task Instructions
format: md
path: input/task_instructions.md
description: Output contract and statistical requirements.
name: Runtime Note
format: txt
path: input/runtime_env.txt
description: Recommended R package stack.
Title: TCGA LUAD KRAS Survival Analysis
Domain: health_medicine — Public Health & Epidemiology

Summary: Download public TCGA-LUAD data from GDC, build a KRAS-stratified survival cohort, and export Kaplan-Meier and Cox proportional hazards results.

Task prompt:
You are a bioinformatics analyst performing a TCGA-LUAD KRAS survival analysis on Linux.

Task directory:
- `base`

Visible files and directories:
- Task specification: `base/input/task_spec.json`
- Detailed instructions: `base/input/task_description.txt`
- Output contract: `base/input/output_requirements.txt`
- Runtime notes: `base/input/runtime_notes.txt`
- Software notes: `base/software/README.txt`

What you need to do:
1. Use public TCGA-LUAD data from the GDC Data Portal.
2. Download the clinical data and STAR-count expression data for the LUAD cohort.
3. Extract KRAS (`ENSG00000133703`) using `tpm_unstranded`.
4. For multiple primary-tumor samples per patient, use the lexicographically smallest 15-character sample submitter ID.
5. Merge expression with survival and clinical covariates, then assign values greater than or equal to the cohort median to the KRAS-high group.
6. Fit Kaplan-Meier, log-rank, and Cox proportional-hazards analyses with KRAS group, age, and stage grouping.
7. Save every required output under `base/output`.

Required output files:
- `cohort.csv`
- `km_plot.png`
- `cox_results.json`
- `analysis.R`

Do not modify staged inputs.
Do not ask for confirmation. Execute directly.


The agent must:
Read the staged task specification and output contract.
Download public TCGA-LUAD clinical and STAR-count expression data from GDC.
Extract KRAS expression from `tpm_unstranded` and merge it with survival and clinical covariates.
Use the lexicographically smallest primary-tumor sample submitter ID per patient.
Stratify patients using the cohort median, assigning ties to the high group.
Fit Kaplan-Meier, log-rank, and Cox proportional hazards analyses.
Write `cohort.csv`, `km_plot.png`, `cox_results.json`, and `analysis.R` under `output/`.

Software: R
TCGAbiolinks
survival
survminer
ggplot2
jsonlite

Input files:
name: task_spec.json
format: JSON
path: input/task_spec.json
description: Original submission specification describing the TCGA-LUAD KRAS survival task.
name: task_description.txt
format: TXT
path: input/task_description.txt
description: Benchmark-owned execution brief derived from the submission.
name: output_requirements.txt
format: TXT
path: input/output_requirements.txt
description: Lists the required output filenames and minimum content expectations.
name: runtime_notes.txt
format: TXT
path: input/runtime_notes.txt
description: Notes about the public-data runtime workflow.
name: software/README.txt
format: TXT
path: software/README.txt
description: Operational notes about the planned benchmark-owned R runtime surface.
Title: Build A Pinned Clinical Variant Annotation Table
Domain: health_medicine — Genomics & Sequence Analysis

Summary: Parse a staged 50-variant NA12878-derived VEP snapshot bundle, recover gene / consequence / frequency / ClinVar annotations, and emit the exact reportable subset.

Task prompt:
You are working on a Linux clinical-variant annotation task.

Task directory:
- `base`

Visible inputs:
- Variant list: `base/input/variants_to_annotate.tsv`
- Raw VEP snapshots: `base/input/annotation_snapshots/vep_batch_responses.jsonl`
- Rule / provenance manifest: `base/input/source_manifest.json`
- Task-local runtime manifest: `base/input/runtime_env/pyproject.toml`
- Task-local runtime lockfile: `base/input/runtime_env/uv.lock`

Your job:
1. Work from a writable directory under `base/output`.
2. Read the 50 staged variants in order.
3. Parse the raw VEP snapshot JSON lines and derive, for each variant:
   - `gene`
   - `consequence`
   - `max_population_af`
   - `clinvar_significance`
   - `is_reportable`
4. Filter the reportable subset exactly according to `base/input/source_manifest.json`.
5. Write these files under `base/output`:
   - `annotated_variants.tsv`
   - `reportable_variants.tsv`
   - `pipeline.py`
   - `run_log.json`

Output rules:
- `annotated_variants.tsv` and `reportable_variants.tsv` must use tab-separated columns with this exact header:
  `variant_id chrom pos ref alt gene consequence max_population_af clinvar_significance is_reportable`
- `reportable_variants.tsv` must contain only rows where `is_reportable=yes`
- `pipeline.py` should be your solver-authored pipeline implementation
- `run_log.json` must be valid UTF-8 JSON and include the emitted row counts
- Treat `base/input` as read-only source data

Recommended setup:
- You may work with the system `python`, or create a task-local environment from `base/input/runtime_env` with:
  `uv sync --frozen --project input/runtime_env`

Do not ask for confirmation. Execute directly.


The agent must:
Read the staged 50-variant TSV in order.
Parse the raw VEP JSONL snapshot bundle and derive gene / consequence / AF / ClinVar fields.
Apply the reportable rule from `input/source_manifest.json` exactly.
Write `annotated_variants.tsv`, `reportable_variants.tsv`, `pipeline.py`, and `run_log.json` under `output/`.

Software: Python
uv
pandas
Ensembl VEP snapshot

Input files:
name: Variant list
format: tsv
path: input/variants_to_annotate.tsv
description: 50 pinned NA12878-derived coding variants in canonical output order.
name: VEP snapshot bundle
format: jsonl
path: input/annotation_snapshots/vep_batch_responses.jsonl
description: Raw VEP responses the agent must parse.
name: Rule manifest
format: json
path: input/source_manifest.json
description: Visible reportable-variant rule and provenance notes.
name: Runtime manifest
format: toml
path: input/runtime_env/pyproject.toml
description: Optional task-local dependency manifest.
name: Runtime lockfile
format: lock
path: input/runtime_env/uv.lock
description: Locked dependency set for deterministic `uv sync --frozen`.
Title: NHANES Confounder Sensitivity Analysis
Domain: health_medicine — Public Health & Epidemiology

Summary: Rebuild a clinical sensitivity-analysis cohort from staged NHANES tables and produce the required participant subset plus logistic-regression summary outputs.

Task prompt:
You are working on a Linux precision-health data-analysis task.

Read the staged benchmark brief and output contract first:
- `base/input/task_spec.txt`
- `base/input/output_contract.json`

Visible task inputs:
- `base/input/3.demographics_clean_cycle1.csv`
- `base/input/7.questionnaire_clean_cycle1.csv`
- `base/input/8.response_clean_cycle1.csv`
- `base/input/5.medications_clean_cycle1.csv`

If you want the pinned Python environment, use:
- `base/software/python_with_task_deps.sh`

Your job is to rebuild the analytic cohort and export exactly these files under `base/output`:
- `base/output/hpyl_nsaid_subset.csv`
- `base/output/sensitivity_summary.csv`

Requirements:
1. Follow the exact cohort construction, variable derivations, and model definitions in `task_spec.txt`.
2. Use `output_contract.json` for the exact output filenames, column order, formulas, and metadata fields.
4. Write only the required task outputs under `base/output`.


The agent must:
Read `input/task_spec.txt` and `input/output_contract.json` before doing any analysis.
Rebuild the analytic cohort and the non-missing `LBXHP1` subset from the four staged NHANES CSVs.
Write `output/hpyl_nsaid_subset.csv` with the exact required participant-level schema.
Write `output/sensitivity_summary.csv` with the exact required three-row model summary schema.

Software: Python
uv
pandas
NumPy
SciPy
statsmodels

Input files:
name: task_spec.txt
format: text
path: input/task_spec.txt
description: Agent-visible benchmark brief describing the cohort construction and model requirements.
name: output_contract.json
format: json
path: input/output_contract.json
description: Exact output schema, formulas, and metadata contract for the two required CSV files.
name: 3.demographics_clean_cycle1.csv
format: csv
path: input/3.demographics_clean_cycle1.csv
description: Demographics table keyed by `SEQN_new` and `SDDSRVYR`.
name: 7.questionnaire_clean_cycle1.csv
format: csv
path: input/7.questionnaire_clean_cycle1.csv
description: Questionnaire table containing ulcer and chronic-condition fields.
name: 8.response_clean_cycle1.csv
format: csv
path: input/8.response_clean_cycle1.csv
description: Response table containing BMI and `LBXHP1` serology values.
name: 5.medications_clean_cycle1.csv
format: csv
path: input/5.medications_clean_cycle1.csv
description: Medication exposure table with free-text `RXDDRUG` entries.
name: runtime_env
format: uv Python environment
path: input/runtime_env
description: Pinned optional Python runtime manifest for pandas/numpy/scipy/statsmodels.
Title: NSCLC Radiomics Cox Signature
Domain: health_medicine — Clinical Diagnostics & Imaging

Summary: Build a CPU-only radiomics survival-risk model for non-small-cell lung cancer from CT volumes, tumor masks, training clinical outcomes, and fixed held-out prediction IDs.

Task prompt:
You are building an Aerts-2014-style radiomic prognostic risk model for non-small-cell lung cancer.

## Input

Use the data under `base/input`:

- `images/` contains 422 lung CT NIfTI volumes (`.nii.gz`).
- `masks/` contains matched GTV-1 tumor segmentation masks for the usable cohort.
- `clinical/train_clinical.csv` contains survival labels for the training patients only.
- `prediction_ids.csv` lists held-out patients that need risk scores.
- `split.json` documents the fixed train/test split and the one excluded missing-mask case.
- `runtime_env/pyproject.toml` pins a Python 3.10-compatible scientific stack for this task.

## Task

Build any CPU-only radiomics survival pipeline you choose. You may use PyRadiomics, SimpleITK, lifelines, scikit-learn, pandas, and NumPy. Extract features from the CT/mask pairs, fit a Cox-style or other survival-risk model using the labeled training patients, and generate risk scores for the held-out patients in `prediction_ids.csv`.

## Required Output

Write exactly these files under `base/output`:

1. `risk_scores.csv` with one row for each held-out patient and columns:
   - `PatientID`
   - `risk_score`

Higher `risk_score` must mean higher predicted hazard / shorter survival. Use only the supplied training labels, and do not modify input files. CPU time budget is 2 hours.


The agent must:
Read CT images and tumor masks from `input/images/` and `input/masks/`.
Use only `input/clinical/train_clinical.csv` as labeled survival training data.
Generate one held-out prediction for each patient listed in `input/prediction_ids.csv`.
Write `risk_scores.csv` with columns `PatientID` and `risk_score` to the configured output directory.

Software: Python
uv
PyRadiomics
SimpleITK
lifelines
pandas
NumPy

Input files:
name: CT volumes
format: NIfTI gzip
path: input/images/*.nii.gz
description: Lung CT volumes for the submitted NSCLC cohort.
name: Tumor masks
format: NIfTI gzip
path: input/masks/*.nii.gz
description: GTV-1 tumor segmentation masks for the usable cohort.
name: Training clinical outcomes
format: CSV
path: input/clinical/train_clinical.csv
description: Survival labels for the training patients only.
name: Held-out prediction IDs
format: CSV
path: input/prediction_ids.csv
description: Patient IDs that require output risk scores.
name: Split metadata
format: JSON
path: input/split.json
description: Fixed train/test split and the excluded missing-mask case.
name: Runtime environment manifest
format: TOML
path: input/runtime_env/pyproject.toml
description: Agent-visible Python dependency manifest for the radiomics stack.
Domain: health_medicine — Public Health & Epidemiology

Summary: Audit an Obermeyer-style healthcare risk-score bias using `risk_score_t` as the baseline, build a counterfactual revised ranking from medical need, and emit `full_predictions.csv` plus baseline and revised analysis reports.

Task prompt:
You are working on a Linux VM to reproduce an Obermeyer-style healthcare algorithmic bias audit.

Read the task instructions and cohort data staged under:
- `base/input/task_instructions.md`
- `base/input/analysis_data.csv`
- `base/input/dummy_test.csv`

The helper `base/software/python_with_task_deps.sh` runs Python from the staged UV runtime environment if you want pandas and numpy available; it writes dependency state under the writable output directory.

Your job is to audit the observed risk score using `risk_score_t` directly as the baseline, build a reproducible counterfactual revised ranking based on medical need, explain the disparity mechanism, and save `full_predictions.csv`, `baseline_analysis_report.md`, and `revised_analysis_report.md` under `base/output`.

`full_predictions.csv` must have exactly these columns: `patient_id,baseline_score,revised_score`.


The agent must:


Input files: