Dhruv1000/Liars_Are_Information

AIP — Adversarial Informativeness Pooling

"Liars Are Information": Byzantine-robust decentralized LLM swarms.

N agents answer the same question, broadcast an answer plus a confidence, and each agent locally aggregates what it receives. A fraction f of agents are Byzantine. Prior art (Krum, geometric median, trimmed mean, confidence weighting, Dawid–Skene, filter-and-refine) discards adversarial input. AIP inverts coherent adversaries and pools them as information. This repository tests, with real LLMs, whether accuracy can be made monotone-increasing in f.

Status: end of Phase 1 (skeleton, registry, task loaders, tests). No inference has been run and no experimental results exist yet. Every number this project eventually reports comes from executed code — nothing is estimated or filled in by hand.

Design principles

1. Cache-once inference. All LLM generation happens in Phase A (and Phase D for adversaries whose outputs depend on injections) and is written to data/cache/. Aggregation, baselines, and analysis read that cache offline. No aggregation experiment may trigger inference.
2. Normalization parity. Trust/weight-based schemes inflate their weights with self-vote mass. Every baseline comparison runs under matched self-vote share, and every table states its parity condition.
3. Statistical rigor. Every reported number carries a bootstrap 95% CI (≥1000 resamples), logged fixed seeds, and a per-cell sample size.

Setup

curl -LsSf https://astral.sh/uv/install.sh | sh     # if uv is not installed
uv python install 3.11
uv venv --python 3.11
source .venv/bin/activate
uv pip install -e ".[dev]"        # offline phases + tests (no GPU needed)
uv pip install -e ".[gpu]"        # adds vLLM; required only for Phases A and D

Gated models (Llama, Mistral) additionally need export HF_TOKEN=... and an accepted licence on the Hub. The models enabled by default are ungated.

Hardware

configs/models.yaml is tuned to the machine it runs on. This node has a single NVIDIA L4 (22.49 GiB).

Residency is sequential and there is no other mode. One model is resident at a time (load, generate, del + torch.cuda.empty_cache(), next). Co-resident mode was removed from the code, not merely defaulted off, so that no config can accidentally assume models share the device. Because of the cache-once principle this is scientifically equivalent to co-residency and costs only wall-clock time. gpu_memory_utilization therefore means "fraction of the device this model may use while it is the resident model", and the ≤ 0.90 budget is per model.

bf16 only. No quantised checkpoints anywhere in the study: quantisation shifts the token logprob distribution, which is both the confidence surface the attacks target and the error-correlation structure Phase B measures.

Run the preflight before downloading anything:

python scripts/preflight.py --all

It checks three things and exits non-zero if any enabled model fails:

1. Fit — bf16 weights plus KV cache at the model's full max_model_len against the physical card, requiring min_slack_gib (default 4 GiB) of headroom for activations, CUDA graphs, and fragmentation.
2. Architecture agreement — the arch block in the registry is re-fetched from the Hub's config.json and compared, so the KV arithmetic can never drift from the real checkpoint. params_b is checked the same way.
3. Access — gated status plus an authenticated read, so a 401/403 surfaces in a second rather than mid-download.

GPU driver constraint

The driver here is 550.127.08, which caps at CUDA 12.4. vLLM ≥ 0.20 resolves to a CUDA 13 build of torch and will not run on this driver. The [gpu] extra is therefore pinned to vllm==0.17.0, which ships CUDA 12.8 wheels (compatible with a 12.4 driver under CUDA minor-version compatibility) and still supports every architecture in the registry, Qwen3.5 included.

Phase order

Phase	Script	Produces	GPU
1	—	skeleton, registry, task loaders, tests	no
A	scripts/phase_a_cache.py	data/cache/{benchmark}/{model}.parquet	yes
A′	scripts/phase_a_cache.py --temperature 0.7 --sample-index 1	data/cache_t07/gsm8k/{model}.parquet	yes
B	scripts/phase_b_correlation.py	pairwise error correlations, correlation_heatmap.pdf	no
C	scripts/phase_c_aggregate.py	method × f × p_obs results table	no
D	scripts/phase_d_adversarial.py	real-LLM adversary cache, re-run sweep	yes
E	scripts/phase_e_figures.py	paper figures and tables	no

Each phase stops for review before the next begins.

Reproducing

Every phase is resumable: re-running skips completed work and writes a manifest (config hash, git commit, seeds, timestamps, GPU) beside its outputs.

python scripts/preflight.py --all          # GPU + Hub check; never skip this
python scripts/make_task_lists.py          # freeze the task selection (once; committed)
python scripts/phase_a_cache.py --dry-run  # 5 tasks per cell, prints the five gates
python scripts/phase_a_cache.py            # full run, greedy, sample_index=0

# Phase B needs a second sample per task; GSM8K only, to bound the cost.
python scripts/phase_a_cache.py --benchmarks gsm8k --temperature 0.7 \
    --sample-index 1 --cache-root data/cache_t07 --no-self-report

Phase A is complete: 1600 cached predictions (12 cells × 100 greedy, plus 4 × 100 at temperature 0.7), 31 minutes of wall clock on one L4.

Frozen tasks

configs/task_lists.json is committed and is the authority on which questions the study runs on — 100 per benchmark under seed 20250825, MMLU stratified across all 57 subjects. Phase D's adversarial runs read the same file, so honest and Byzantine broadcasts provably refer to the same questions. A recorded id that has vanished from the dataset raises rather than silently shrinking the study.

Two reproducibility caveats

Both are recorded in every run manifest rather than papered over:

• VLLM_BATCH_INVARIANT is on for all cache-generating runs. Without it, identical batches return logprobs differing at ~1e-6, enough to flip a near-tie argmax. With it, output and logprobs are bit-identical across batch sizes.
• Seven task ids still did not reproduce byte-identically across re-runs; they are listed under reproducibility.known_nonreproducing. Downstream phases read the cache, so results stay internally consistent — what is not guaranteed is regenerating a bit-identical cache from scratch.

Figure-by-figure reproduction commands land in Phase E, one command per figure.

Layout

configs/          models.yaml (registry), experiments/*.yaml (one per experiment)
src/aip/
  models/         registry.py, inference.py (vLLM), confidence.py (logprob spans)
  tasks/          gsm8k.py, math500.py, mmlu.py — loaders, prompts, extractors, scorers
  swarm/          broadcast.py (p_obs), topology.py, assignment.py
  attacks/        6 attack classes + adaptive bandit adversary
  aggregation/    aip.py, correlation.py, baselines.py
  analysis/       bootstrap.py, tables.py, figures.py, regret.py
  harness/        runner.py, cache.py, logging.py, manifest.py
scripts/          phase_a … phase_e entry points
tests/            scorers, extractors, registry, cache, confidence, manifests
data/             (gitignored) benchmark + inference caches
results/          parquet outputs, run manifests, JSONL logs
figures/          final paper figures (PDF)

Two confidence surfaces

Each cached prediction stores both, and they are not interchangeable:

• logprob_confidence — mean token logprob over the answer span only, mapped through exp to (0, 1]. A prompt-injected agent cannot inflate this by asserting confidence in prose.
• self_reported_confidence — elicited by a separate 0–100 prompt. This is the surface the falsified_confidence attack corrupts, which is the point: baselines that read it should break while AIP's logprob channel does not.

Tests

pytest              # fast unit tests
pytest -m slow      # + dataset downloads and real-gold round-trip checks

The slow suite formats every real gold answer the way the prompt asks for it, then requires the extractor and normalizer to recover and score it — a normalizer regression fails loudly instead of quietly mis-scoring Phase A.

Licence

MIT.