andyc03/handoff_repo

DTAP os-filesystem — all-in-one eval image (Slurm handoff)

A single self-contained image that runs the whole DTAP eval inside one container, with no docker daemon and no root (DTAP_NO_DOCKER): the os-filesystem world-state service runs as a local process, the judge runs locally, and the eval talks to both over 127.0.0.1. Meant for clusters (Enroot/Pyxis or Apptainer) that have no docker daemon — you submit one Slurm job, it runs the container, and it writes results to a shared folder.

Validated end-to-end (benign and malicious tasks): one container runs a whole split sequentially and, per task, writes the full trace set, not just the final verdict: judge_result.json, trajectory_*.json, traces/*.jsonl, task.log.

Files

• dtap-osfs-smoke.image.tar.gz — the prebuilt image (~474 MB, secrets scrubbed).
• run_on_slurm.sbatch — the Slurm job you submit (Enroot or Apptainer).
• env.template — copy to my.env and fill in your victim/judge credentials.
• incontainer_smoke.sh — the in-container wrapper (how the env/API is set up + eval launched); already baked into the image, included here for reference.
• tests/smoke_osfs_benign2.jsonl — 2 benign tasks (quick "does it run").
• tests/smoke_osfs_malicious2.jsonl — 2 malicious tasks (exercises the attack/defense path; judge reports attack_success).

End-to-end, step by step

Everything below lives on shared scratch the compute nodes can see, e.g. /scratch/$USER/handoff/.

1) Get the image onto the cluster (one time).

gunzip -c dtap-osfs-smoke.image.tar.gz | docker load          # if you have docker somewhere; then push to your registry
# Apptainer (no docker needed): build a .sif from the archive
gunzip dtap-osfs-smoke.image.tar.gz
apptainer build /scratch/$USER/handoff/dtap-osfs-smoke.sif docker-archive://dtap-osfs-smoke.image.tar
# Enroot: enroot import docker://<your-registry>/dtap-osfs-smoke:latest ; enroot create --name dtap-osfs-smoke <img>.sqsh

2) Fill in credentials.

cp env.template /scratch/$USER/handoff/my.env
# edit my.env: pick a victim (OpenRouter / your own OpenAI-compatible endpoint / Bedrock) + judge.

3) Submit the job. Edit the paths at the top of run_on_slurm.sbatch first (image, my.env, OUTDIR).

# benign test (baked split, no mount):
sbatch run_on_slurm.sbatch smoke_osfs_benign2 dtap_def_v0

# malicious test (mount the split file):
SPLITFILE=/scratch/$USER/handoff/tests/smoke_osfs_malicious2.jsonl \
  sbatch run_on_slurm.sbatch smoke_osfs_malicious2 dtap_def_v0

4) Collect results. Everything lands under OUTDIR/_test_<split>/<defense>/, one folder per task with judge_result.json + trajectory_*.json + traces/*.jsonl + task.log. Send us that OUTDIR (or its .tar.gz). The job log ends with SMOKE PASS if every task produced a judge result.

Change things at run time — NO rebuild

run_on_slurm.sbatch already mounts these; the knobs:

Change	How
Defense	sbatch run_on_slurm.sbatch <split> <DEFENSE_NAME> + (for a custom one) mount it into .../dt_defenses/<name>
Split / task set	pass a SPLITFILE=/path/<name>.jsonl and use <name> as the split arg
Victim / judge API	edit my.env (DTAP_MODEL, OPENAI_API_BASE, keys, JUDGE_MODEL)
Harness code	mount your dtap-src over /harness/dtap_ossystem/benchmarks/dtap-src
Results dir	OUTDIR=/path

Notes

• The image ships with REPLACE_ME for real keys — it will only run once you provide my.env.
• Give a split N tasks → the container runs them sequentially and emits N full trace sets.
• In-container parallelism is off (one shared world-state service): keep MAXP=1 and get throughput by submitting more jobs (each an independent container), not by raising MAXP.
• DTAP_NO_DOCKER is an off-by-default harness switch baked into the image; normal docker runs elsewhere are unaffected.