InfraBench — Pipeline Architecture

InfraBench
Pipeline Architecture

Pipeline
Concentric
Bands
Paper
Docs ↗

Stack (L1–L6)

L6 ServiceContainer

L5 PlatformContainer/K8s

L4 StorageDistributed

L3 KernelVM/Bare

L2 VirtVM

L1 HardwareBare metal

Higher layers depend on every layer below. Faults can be injected at any level.

1 · Setup

impl

Provision the machine: CPU/mem/network, cache OS/app images.

CloudLab (bare metal)

infra/baremetal/cloudlab.py

libvirt + KVM (VM)

infra/vm/standalone.py · cluster.py

Docker (container)

infra/container/docker.py

TrialQueue · asyncio.Semaphore(n) trial/queue.py:39

2 · Bootstrap

impl

Start the application infra (Redis / Ceph / HDFS / etcd / Cassandra) and drive it to the desired test state.

environment/setup.sh

Once per image: apt install, image cache

environment/bootstrap.sh

Per trial: launch services → broken/desired state

Per-trial isolation: unique VM, disk, libvirt network.

3 · Management

impl

Scenario engine orchestrates agent + faults + verification.

ScenarioEngine

fault/scenario/executor.py · 3 modes

Fault injectors (6)

kill :71 · partition :95 · disk-delay :159

disk-fill :223 · cpu :268 · netem network.py

Dynamic & cascading fault timelines

Adaptive injection driven by live state; chained L1→L4→L6 propagation.

Periodic Verifier → EventLog

fault/scenario/periodic_verifier.py:20 · events.py

Operations monitoring

Dangerous-op detection (rm -rf, dd /dev/…) + cascading-fault trigger watch → EventLog.

4 · Decommission

impl (core)

Clean teardown after each trial.

stop(delete=…)

infra/base.py:291

heal_all()

fault/injection.py:320

network_cleanup

infra/vm/network_cleanup.py

Restart · container level

docker / pod restart of a single service; re-verify state survives.

Restart · device level

Host reboot, kernel reload, device re-attach; config & data persist.

Planned

Data migration / backup hooks before teardown.

Planned — not yet implemented

Long-term monitoring — background scoring after agent exits.

Lifecycle / longevity — sustained workload regression check.

Multi-tenant impact — noisy-neighbour on shared infra.

Formal task taxonomy — layer × fault class × phase coverage.