Skill: quochungto/bookforge-distributed-failure-analyzer (version 1.0.0)

--- name: distributed-failure-analyzer description: | Diagnose distributed system failures caused by network faults, unreliable clocks, or process pauses — and map each to its correct mitigation. Use when: a node is intermittently timing out with no clear network outage; a lock-holder or leader keeps acting after being declared dead (zombie leader / split brain via distributed locking, not replication topology — use replication-failure-analyzer for replica split brain); stale reads persist beyond expected replication lag; wall-clock-based lease checks or last-write-wins conflict resolution is producing data loss under clock skew; or cascading node-death declarations are occurring under load. Also use proactively to audit timing assumptions in new system designs (absence of fencing tokens, NTP drift exposure, GC pause risk). Distinct from replication-failure-analyzer (replication lag anomalies, failover pitfalls, quorum edge cases). Produces a structured failure report: symptom → fault category → mechanism → mitigation. Covers: asynchronous network behavior, timeout tuning and cascade risk, NTP drift and clock jump mechanics, process pause causes (GC, VM migration, paging, SIGSTOP), fencing tokens with ZooKeeper zxid/cversion, Byzantine fault scoping, and system model selection (crash-stop vs. crash-recovery vs. Byzantine; synchronous vs. partially synchronous vs. asynchronous). version: 1.0.0 homepage: https://github.com/bookforge-ai/bookforge-skills/tree/main/books/designing-data-intensive-applications/skills/distributed-failure-analyzer metadata: {"openclaw":{"emoji":"📚","homepage":"https://github.com/bookforge-ai/bookforge-skills"}} status: draft depends-on: - replication-strategy-selector - consistency-model-selector source-books: - id: designing-data-intensive-applications title: "Designing Data-Intensive Applications" authors: ["Martin Kleppmann"] chapters: [8] tags: - distributed-systems - failure-analysis - network-faults - unreliable-clocks - process-pauses - fencing-tokens - zombie-leader - split-brain - distributed-locking - last-write-wins - clock-skew - ntp-drift - garbage-collection - partial-failure - timeout-tuning - byzantine-faults - system-model - crash-recovery - quorum - zookeeper - lease-expiry - partial-synchrony execution: tier: 2 mode: hybrid inputs: - type: codebase description: "Application source code, infrastructure configs (docker-compose, k8s manifests), or architecture description revealing timing assumptions, locking patterns, and clock usage" - type: document description: "Incident report, runbook, or architecture description if no codebase is available" - type: description description: "User-described symptoms: what failed, when, topology, observed behavior vs. expected behavior" tools-required: [Read, Write, Grep] tools-optional: [Bash] mcps-required: [] environment: "Run inside a project directory with codebase or configuration files, or accept a verbal description of the failure. Produces a written failure analysis report." discovery: goal: "Produce a structured failure analysis report: classify each symptom into its fault category, identify the specific mechanism, and recommend concrete mitigations" tasks: - "Gather failure symptoms, system topology, and observed vs. expected behavior" - "Classify each symptom into network fault, clock unreliability, or process pause" - "Identify the specific failure mechanism within the category" - "Scan codebase for anti-patterns (clock-based ordering, unguarded lease checks, missing fencing tokens)" - "Recommend mitigations matched to the root cause" - "Assess Byzantine fault risk and whether it is relevant to this context" - "Select appropriate system model assumptions for the architecture" audience: roles: ["backend-engineer", "software-architect", "site-reliability-engineer", "tech-lead", "data-engineer"] experience: "intermediate-to-advanced — assumes familiarity with distributed systems concepts" triggers: - "Intermittent timeouts with no clear network cause" - "Data disappearing or being silently overwritten after concurrent writes" - "A node or leader that continues acting after being declared dead" - "Stale reads that persist longer than replication lag explains" - "Cascading node-death declarations under load" - "Distributed lock / lease behavior that appears correct but causes corruption under failure" - "Designing a system and wanting to audit timing and locking assumptions before shipping" not_for: - "Choosing a replication topology — use replication-strategy-selector" - "Selecting consistency and isolation levels — use consistency-model-selector" - "Diagnosing replication lag anomalies specifically — use replication-failure-analyzer" --- ## When to Use You are diagnosing or preventing a failure in a distributed system and the root cause is not immediately obvious. The symptom could be: a timeout that might mean the remote node is dead, or might mean the network is congested, or might mean the node is alive but paused. A write that appeared to succeed but whose data is now missing. A leader that is still writing after another leader was elected. A lock that was held correctly but still allowed two writers simultaneously. This skill imposes a diagnostic framework: every unexplained distributed system failure traces to one of three root fault categories — **network faults**, **clock unreliability**, or **process pauses** — and each category has a bounded set of mechanisms and well-understood mitigations. The skill maps symptoms to categories, categories to mechanisms, and mechanisms to concrete fixes. Use it reactively (incident post-mortem, production debugging) or proactively (design review, codebase audit for timing anti-patterns). Cross-references: - `replication-failure-analyzer` — for failures specific to replication lag, failover, and quorum behavior - `consistency-model-selector` — for selecting isolation and consistency guarantees that prevent a class of failures at the application layer --- ## Context and Input Gathering Before analysis, collect the following. Ask the user for any that are missing. **Required:** 1. **Symptom description** — what was observed vs. what was expected. Be concrete: "node A sent a write that was confirmed, but the data is now absent on node B" is better than "data loss." 2. **System topology** — number of nodes, roles (leader/follower, lock-holder/client, partitioned shards), deployment environment (bare metal, VMs, cloud, containers). 3. **Failure timeline** — when did it start, is it reproducible, does it correlate with load spikes, deploys, or maintenance events? **Useful:** 4. **Infrastructure configuration** — timeout values, NTP setup, GC settings (heap size, collector type), VM migration policies. 5. **Relevant code** — any section that checks wall-clock time, holds a lease or lock, does conflict resolution (especially last-write-wins), or makes assumptions about execution timing. 6. **Logs or metrics** — round-trip time distributions, GC pause logs, NTP offset metrics, CPU steal time. **If no codebase is available:** accept a verbal architecture description and produce an analysis based on stated behavior patterns. The output will note which findings are speculative vs. confirmed. --- ## Process ### Step 1 — Map symptoms to the fault taxonomy WHY: Distributed failures are non-deterministic and the same symptom can arise from multiple root causes. The three-category taxonomy forces explicit elimination: you cannot address "network issues" without first determining which of the six network failure modes is actually occurring, or whether it is a process pause mimicking a network failure. For each reported symptom, classify it against this taxonomy: **Category A: Network faults** The network is an asynchronous packet network with unbounded delays. When a request is sent and no response arrives, it is impossible to distinguish which of these occurred: 1. Request lost in transit (cable, queue drop) 2. Request queued, not yet delivered (network congestion, switch buffer overflow) 3. Remote node crashed or was powered down 4. Remote node temporarily unresponsive (GC pause — overlaps Category C) 5. Response lost in transit (misconfigured switch, asymmetric fault) 6. Response delayed (network or receiver overloaded) Key diagnostic signal: **timeouts tell you nothing about whether the remote node executed your request.** A timeout means you gave up waiting; it does not mean the request failed. **Category B: Clock unreliability** Each node has its own hardware clock (quartz oscillator). Clocks drift and can jump: - **Time-of-day clock** (wall-clock time): synchronized via NTP, but NTP accuracy is limited by network round-trip time. On a congested network, NTP error can be 35ms to over 100ms. Clocks may jump backward if they are ahead of the NTP server. - **Monotonic clock**: suitable for measuring elapsed time (durations, timeouts) on a single node. Cannot be compared across nodes — the absolute value is meaningless. - **Sources of clock error**: quartz drift (up to 200 ppm = 17 seconds/day without sync), NTP misconfiguration (firewall blocking NTP traffic), NTP server errors (some servers are wrong or misconfigured), leap seconds (a minute can be 59 or 61 seconds), VM virtualization (CPU time-sharing pauses the VM; from the application's view, the clock jumps forward). **Category C: Process pauses** A thread can be preempted at any point in its execution and paused for an arbitrary duration without being notified. Pause causes: - **Stop-the-world GC**: JVM, .NET, Ruby, and others pause all threads during major GC. Pauses of several seconds are possible even with "concurrent" collectors like CMS (which still has stop-the-world phases). - **VM suspension**: a hypervisor may suspend a VM (save memory to disk) and resume it seconds or minutes later — live migration between hosts does this. - **OS context switches / CPU steal**: in multi-tenant environments, another VM consuming a shared CPU core creates "steal time" — the paused VM's threads wait with no awareness that real time is passing. - **Disk I/O**: synchronous disk access pauses the thread. In Java, class loading can trigger disk I/O unexpectedly. On network-attached storage (EBS, NFS), I/O latency inherits network variability. - **Memory paging (thrashing)**: if the OS swaps pages to disk under memory pressure, a thread can be paused waiting for page-in. In extreme cases the system spends most of its time paging. - **SIGSTOP**: a Unix signal that immediately halts a process until SIGCONT. Sent accidentally by operators (Ctrl-Z in a shell), or by tooling. After categorizing, state the most probable category per symptom and note any ambiguity. --- ### Step 2 — Identify the specific failure mechanism WHY: The category narrows the space; the mechanism identifies the specific causal chain and determines which mitigation is effective. A "clock unreliability" failure caused by NTP being blocked at the firewall needs a different fix than one caused by VM time virtualization. **For network faults:** - Distinguish between node death (crash-stop) and node pause (crash-recovery or process pause). If the node later recovers, it is likely a pause, not a crash. - Check for asymmetric faults: a node can receive messages but its outgoing messages are dropped. This node will appear dead to the rest of the cluster despite being healthy. Symptom: the "dead" node does not know it has been declared dead. - Check for timeout calibration: is the timeout shorter than the 99th percentile round-trip time under load? Premature timeouts under load cause cascading failures — a node declared dead transfers its load to other nodes, which increases their load, which makes them slower, which causes more timeouts, which declares more nodes dead. **For clock unreliability:** - Identify whether the code uses wall-clock time for **ordering events across nodes** (anti-pattern) or for **measuring elapsed time on a single node** (acceptable with monotonic clock). - Last-write-wins (LWW) conflict resolution using wall-clock timestamps is the canonical clock anti-pattern: a node with a lagging clock will silently discard writes from a node with a fast clock, because the lagging node's writes appear "more recent." Data is lost with no error reported to the application. - Lease checks using wall-clock time are unsafe if the clock can jump or if a process pause occurs between the check and the protected operation (see Step 4). **For process pauses:** - GC pause: correlate with GC logs. Look for stop-the-world events. Check heap size and object allocation rate. Long-lived objects accumulate and force full GC. - VM migration: check hypervisor or cloud provider logs for live migration events. - Thrashing: check system swap metrics and page fault rates. - The defining characteristic: **the paused node does not know it was paused.** When it resumes, it checks its clock and finds that very little time has passed (from its perspective). It has no awareness that the rest of the cluster has moved on, elected a new leader, or acquired a new lock. --- ### Step 3 — Scan the codebase for anti-patterns (if codebase is available) WHY: Many distributed system bugs are latent — they exist in the code but only manifest under specific timing conditions (high load, GC pressure, VM migration). Proactive scanning finds them before they trigger in production. Search for: **Anti-pattern 1: Wall-clock time used for cross-node event ordering** ``` # Look for: System.currentTimeMillis(), time.time(), Date.now(), new Date() # used to timestamp events that are replicated or compared across nodes ``` Risk: If last-write-wins is the conflict resolution strategy and timestamps come from node clocks, any clock skew causes silent data loss. Fix: Use logical clocks (version vectors, Lamport timestamps) for causal ordering. Use physical clocks only for duration measurement (monotonic clock). **Anti-pattern 2: Client-side lease/lock validity check before a protected operation** ``` # Look for: if (lease.isValid()) { ... do protected operation ... } # or: if (System.currentTimeMillis() < leaseExpiryTime) { ... } ``` Risk: A process pause between the check and the operation can cause the lease to expire. The node proceeds, believing it still holds the lock, while another node has already acquired it. Fix: The resource itself must enforce the fencing token (see Step 4). Client-side checks are defense-in-depth only, not a correctness guarantee. **Anti-pattern 3: Timeout values hard-coded without accounting for load variability** ``` # Look for: timeout = 5000 (fixed constant), no jitter, no adaptive adjustment ``` Risk: Timeouts calibrated for p50 latency will fire on p99 latency spikes, causing false node-death declarations. Fix: Measure round-trip time distribution empirically. Set timeout at p99 + safety margin. Consider adaptive failure detectors (Phi Accrual, used in Akka and Cassandra) that adjust based on observed jitter. **Anti-pattern 4: Distributed locking without fencing tokens** ``` # Look for: acquire lock → use resource → release lock # without any monotonically-increasing token passed to the resource ``` Risk: A zombie lock-holder (paused during lock hold, revived after expiry) will corrupt shared state by writing concurrently with the legitimate new lock-holder. Fix: See Step 4. --- ### Step 4 — Apply the fencing token pattern (for zombie leader / zombie lock-holder failures) WHY: Any lease or lock system that relies on the lock-holder checking its own lease status is vulnerable to process pauses. The lock-holder cannot detect that it was paused; the check passes because, from the holder's perspective, no time has elapsed. The only correct solution puts enforcement in the resource, not the client. **The fencing token pattern:** Every time the lock service grants a lock or lease, it returns a **fencing token** — a monotonically increasing integer (each new grant increments the counter). The lock-holder includes its token in every request to the protected resource. The resource tracks the highest token it has seen and rejects any request with a lower token. Concrete mechanics: 1. Client 1 acquires lock → receives token 33. 2. Client 1 goes into a stop-the-world GC pause for 15 seconds. 3. Client 1's lock expires. Client 2 acquires lock → receives token 34. 4. Client 2 writes to the resource with token 34. Resource records "highest seen: 34." 5. Client 1 resumes. It believes it still holds the lock (lease check passes — its clock says almost no time has elapsed). It sends a write with token 33. 6. Resource rejects Client 1's write: token 33 < 34. Corruption prevented. **ZooKeeper integration:** If ZooKeeper is used as the lock service, the transaction ID `zxid` or the node version `cversion` serve as fencing tokens. They are guaranteed to be monotonically increasing. **When the resource does not support fencing tokens natively:** For a file storage service, encode the fencing token in the filename or as a conditional write (compare-and-swap on a version field). Some kind of server-side enforcement is required — client-side enforcement alone is not sufficient for correctness. **Important:** Fencing tokens protect against *inadvertent* zombie behavior (a node that does not know it has been declared dead). They do not protect against *deliberate* misbehavior — that requires Byzantine fault tolerance (see Step 5). --- ### Step 5 — Assess Byzantine fault risk WHY: Byzantine fault tolerance is expensive and complex. For most datacenter systems it is unnecessary. The analysis must be explicit about whether Byzantine faults are in scope, so engineering effort is not misallocated. **Byzantine faults are relevant when:** - Nodes may send arbitrary, incorrect, or malicious messages (not just slow or silent). - The system spans multiple organizations where participants may have conflicting incentives (blockchain, inter-bank settlement). - Hardware operates in high-radiation environments (aerospace, embedded safety systems) where memory or CPU registers may be silently corrupted. **Byzantine faults are NOT relevant for most server-side data systems when:** - All nodes are controlled by your organization. - The threat model is accidental failure, not adversarial behavior. - Radiation levels are low enough that memory corruption is not a realistic concern. For typical datacenter systems: assume crash-recovery faults (nodes may fail and restart, stable storage survives crashes, in-memory state is lost on crash), not Byzantine faults. Standard authentication, checksums in the application protocol, and NTP with multiple servers cover the "weak lying" cases (corrupted packets, misconfigured servers) without the overhead of full Byzantine fault-tolerant protocols. --- ### Step 6 — Select the appropriate system model WHY: Distributed algorithms are designed for specific system model assumptions. Running an algorithm in a system whose actual behavior violates its model assumptions causes correctness failures. This step makes the model explicit so algorithm selection is grounded. **Timing models:** | Model | Assumption | Reality fit | |---|---|---| | Synchronous | Bounded network delay, bounded process pauses, bounded clock error | Not realistic for most packet networks and commodity hardware | | Partially synchronous | Usually synchronous, occasionally exceeds bounds | Realistic for most production systems | | Asynchronous | No timing assumptions, no timeouts | Very restrictive; few practical algorithms can operate without timeouts | **Node failure models:** | Model | Assumption | When to use | |---|---|---| | Crash-stop | Node fails by stopping; never recovers | Simplest; safe assumption for algorithm design even if nodes do recover | | Crash-recovery | Nodes may crash and restart; stable storage survives; in-memory state is lost | Realistic for server-side systems with durable storage | | Byzantine | Nodes may do anything, including sending false messages | Only for multi-party untrusted environments or high-radiation hardware | **Recommended default for most server-side data systems:** partially synchronous model + crash-recovery faults. State the chosen model explicitly in the analysis report. Any algorithm the team adopts (leader election, distributed locking, consensus) must be evaluated against this model. --- ### Step 7 — Produce the failure analysis report Output a structured report with: 1. **Executive summary** — one paragraph: what failed, root fault category, severity. 2. **Symptom-to-mechanism mapping** — for each symptom: category, mechanism, confidence level. 3. **Anti-patterns found** — code locations (if codebase available), description, risk. 4. **Mitigations** — prioritized list, matched to root cause, with implementation notes. 5. **System model recommendation** — which timing model and node failure model the system should be designed against. 6. **Open questions** — what additional information (logs, metrics, code) would increase diagnostic confidence. --- ## Common Misdiagnoses These are the mistakes most frequently made when reasoning about distributed failures. Treat each as an active risk to check. **"The network is reliable inside our datacenter — this must be a software bug."** Network faults occur inside datacenters. One study found ~12 network faults per month in a medium-sized datacenter, half disconnecting an entire rack. Redundant networking gear does not reduce faults proportionally because human error (misconfiguration) is a primary cause. **"The timeout fired, so the node must be dead."** A timeout means you stopped waiting. The remote node may have received your request and processed it, with only the response being lost or delayed. Acting on a false timeout (e.g., promoting a new leader) can create two leaders simultaneously. **"We use NTP so our clocks are synchronized."** NTP accuracy is limited by network round-trip time. On a congested network, NTP error can exceed 100ms. VMs are paused by the hypervisor and see their clock jump forward when resumed. Clocks behind a firewall that blocks NTP silently drift. "Synchronized" does not mean "accurate to the millisecond." **"The node checked its lease and it was valid, so it was safe to proceed."** A check-then-act sequence is not atomic in a distributed system. A GC pause, VM migration, or OS context switch between the check and the act can make the lease expire. Only server-side enforcement (fencing tokens) provides correctness. **"Last-write-wins is fine because our timestamps are accurate enough."** LWW silently discards writes when a node with a lagging clock overwrites values from a node with a fast clock. Clock skew between nodes under 3ms can cause this. The application receives no error. The data is simply gone. **"The node is dead — it stopped responding."** The node may be in a stop-the-world GC pause. It will resume, discover that it was declared dead, and attempt to continue its previous role. Without fencing tokens, this zombie behavior can corrupt state. **"We need Byzantine fault tolerance because we can't trust all nodes."** In a datacenter where your organization controls all nodes, Byzantine fault tolerance is almost certainly not needed and its cost (algorithmic complexity, performance overhead) is not justified. Standard authentication and checksums handle the realistic "lying" cases. --- ## Examples ### Example 1: Silent data loss in a multi-leader database **Scenario:** A team runs Cassandra with multi-datacenter replication. After a network partition heals, some writes from one datacenter appear to be silently overwritten or lost. No errors were logged. **Trigger:** Post-incident investigation after user complaints about missing data. Developer wonders if there is a replication bug. **Process:** 1. Symptom classification: data loss with no error, after concurrent writes during a partition → Category B (clock unreliability) combined with conflict resolution policy. 2. Mechanism: Cassandra uses last-write-wins by default. During the partition, both datacenters accepted writes to the same key. On reconciliation, the write with the higher timestamp wins. If the losing datacenter's clock was ahead, its writes had higher timestamps and survived — but those writes occurred causally *before* the winning datacenter's writes. The causally later write is silently discarded. 3. Anti-pattern confirmed: timestamp-based conflict resolution with wall-clock time. 4. Mitigation: Replace LWW with application-level conflict resolution using version vectors (CRDTs for counters/sets, explicit merge logic for other data types). If LWW is retained, monitor clock offsets between datacenters and alert when skew exceeds acceptable threshold. Declare nodes with excessive clock drift dead and remove them from the cluster. **Output:** Failure analysis report identifying LWW + clock skew as root cause. Migration plan to version-vector-based conflict resolution. Alert thresholds for inter-datacenter clock offset monitoring. --- ### Example 2: Zombie leader corrupts shared file storage **Scenario:** A service uses a ZooKeeper-based distributed lock before writing to shared object storage. Occasionally, two clients write to the same object simultaneously, corrupting it. The team has confirmed the locking code "looks correct." **Trigger:** Data corruption incident. Developer audits the locking implementation. **Process:** 1. Symptom classification: two concurrent writers despite a mutual exclusion lock → Category C (process pause) causing zombie lock-holder behavior. 2. Mechanism: Client 1 acquires the lock and begins writing. A stop-the-world GC pause of 18 seconds occurs. The lock expires. Client 2 acquires the lock and begins writing. Client 1 resumes, checks `lease.isValid()`, finds the clock says only milliseconds have elapsed, concludes it still holds the lock, and continues writing. Two writers are now active simultaneously. 3. Anti-pattern confirmed: client-side lease validity check without server-side fencing token enforcement. 4. Mitigation: Implement fencing tokens. ZooKeeper's `zxid` is a suitable fencing token — it is monotonically increasing and returned with each lock grant. Pass the `zxid` with every write to the storage service. The storage service rejects writes with a `zxid` lower than the highest it has seen. Alternatively: use conditional writes (compare-and-swap on an object version field) at the storage layer to detect and reject stale writes. **Output:** Failure analysis report. Code diff showing where to extract the `zxid` from ZooKeeper and pass it to storage writes. Storage service modification to track and enforce the fencing token. --- ### Example 3: Cascading node-death declarations under load spike **Scenario:** During a traffic spike, a distributed database cluster declares multiple nodes dead in rapid succession. The cluster degrades severely. When traffic drops, all nodes recover and show healthy, but the cluster has partially lost quorum and requires manual intervention. **Trigger:** Post-incident review. SRE team wants to understand why nodes were declared dead when they were actually alive. **Process:** 1. Symptom classification: multiple simultaneous timeout-triggered node-death declarations under load → Category A (network fault) + timeout calibration error. 2. Mechanism: Timeouts were configured at 2 seconds — calibrated for p50 latency. During the load spike, p99 latency jumped to 4–6 seconds due to network congestion and CPU queueing. Health-check RPCs timed out. Nodes were declared dead. Their load was redistributed to remaining nodes, increasing their latency further, causing more timeouts, triggering more declarations — a cascading failure. 3. Anti-pattern: static timeout value calibrated for median, not tail, latency. 4. Mitigation: (a) Recalibrate timeout to p99.9 latency under expected peak load. (b) Add jitter to timeout values to prevent synchronized timeout storms. (c) Implement backpressure: before declaring a node dead, check whether the local node is itself under load (a proxy for network congestion). (d) Consider a Phi Accrual failure detector (used in Akka, Cassandra) that adapts to observed jitter. (e) Tune load shedding to shed excess traffic rather than redistributing it to already-loaded nodes. **Output:** Failure analysis report. Timeout recalibration recommendation with p99.9 measurement methodology. Configuration changes. Recommendation to evaluate Phi Accrual failure detector. --- ## References - `references/failure-taxonomy.md` — complete taxonomy: network fault modes, clock error sources, process pause causes, with detection signals and mitigation options - `references/fencing-token-pattern.md` — fencing token mechanics, ZooKeeper integration, implementation for storage services without native fencing support - `references/system-models.md` — timing models (synchronous, partially synchronous, asynchronous), node failure models (crash-stop, crash-recovery, Byzantine), safety vs. liveness properties - `references/clock-pitfalls.md` — NTP accuracy limits, LWW data loss mechanics, clock confidence intervals, Google Spanner TrueTime approach ## License This skill is licensed under [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). Source: [BookForge](https://github.com/bookforge-ai/bookforge-skills) — Designing Data-Intensive Applications by Martin Kleppmann. ## Related BookForge Skills Install related skills from ClawhHub: - `clawhub install bookforge-replication-strategy-selector` - `clawhub install bookforge-consistency-model-selector` Or install the full book set from GitHub: [bookforge-skills](https://github.com/bookforge-ai/bookforge-skills)

path: references/clock-pitfalls.md
content: # Clock Pitfalls in Distributed Systems

Full reference for Step 2 (clock unreliability) of the distributed-failure-analyzer skill.

---

## The Core Problem

Each node in a distributed system has its own hardware clock. Clocks are not perfectly accurate and cannot be perfectly synchronized. Software that assumes clock accuracy across nodes will behave incorrectly in ways that are subtle, silent, and difficult to reproduce.

"Synchronized" does not mean "accurate." Even with NTP running correctly, clocks across nodes may differ by tens of milliseconds. Under network congestion, errors can exceed 100 ms. In virtualized environments, clocks can jump forward by arbitrary amounts.

---

## Time-of-Day vs. Monotonic Clocks

### Time-of-day clock (wall-clock time)
- **API examples:** `clock_gettime(CLOCK_REALTIME)` (Linux), `System.currentTimeMillis()` (Java), `time.time()` (Python), `Date.now()` (JavaScript)
- **Returns:** seconds (or milliseconds) since the Unix epoch (midnight UTC, January 1, 1970)
- **Synchronized via:** NTP
- **Jump risk:** YES — if the local clock is ahead of the NTP server, it may be forcibly reset backward. Applications observing time before and after this reset see time go backward or jump forward.
- **Use for:** timestamps of events for human display, expiry dates, calendar scheduling.
- **Do NOT use for:** measuring elapsed time (can jump), ordering events across nodes (skew causes incorrect orderings).

### Monotonic clock
- **API examples:** `clock_gettime(CLOCK_MONOTONIC)` (Linux), `System.nanoTime()` (Java), `time.monotonic()` (Python)
- **Returns:** nanoseconds since an arbitrary point (usually system boot). The absolute value is meaningless.
- **Synchronized via:** not synchronized — does not need to be, because it is only used to measure duration on a single node.
- **Jump risk:** NO — guaranteed to always move forward. NTP may *slew* (speed up or slow down) the monotonic clock by up to 0.05%, but it cannot jump.
- **Use for:** timeouts, measuring response latency, measuring elapsed time on a single node.
- **Do NOT use for:** comparing times across different nodes (absolute values are incomparable).

**Rule:** Use monotonic clocks for timeouts and durations. Use time-of-day clocks only where you need a point in calendar time, and never for cross-node ordering.

---

## Sources of Clock Error

### 1. Quartz drift
- Quartz crystal oscillators run faster or slower depending on temperature.
- Google's assumption for server hardware: 200 ppm drift.
- 200 ppm = 6 ms drift for a clock resynchronized every 30 seconds; 17 ms/day for a clock synced once daily.
- Even with NTP, clock drift between syncs accumulates.

### 2. NTP accuracy limits
- NTP accuracy is bounded by network round-trip time to the NTP server.
- Minimum achievable error (local network, well-behaved): ~35 ms.
- Under network congestion: errors spike to over 100 ms.
- On public internet NTP servers: accuracy is limited further by variable internet latency.

### 3. NTP misconfiguration and blocking
- If a firewall blocks NTP traffic (UDP port 123), the NTP client cannot sync.
- The misconfiguration is silent — the clock keeps running and drifting.
- Anecdotal evidence: this happens in practice and goes unnoticed for extended periods.
- If the NTP client detects it is too far from the server's time, it may refuse to sync (to avoid destabilizing the system with a large jump).

### 4. Wrong or misconfigured NTP servers
- Some public NTP servers report incorrect times (hours off).
- NTP clients query multiple servers and reject outliers, but this protection is not perfect.

### 5. Leap seconds
- A leap second inserts an extra second (23:59:60) or deletes a second, making a minute 61 or 59 seconds.
- Systems not designed with leap seconds in mind have crashed.
- Best practice: configure NTP servers to "smear" the leap second — spread the adjustment gradually over the course of a day, so clocks see a slightly slower/faster rate of advance rather than a discontinuity.
- NTP server behavior on leap seconds varies in practice.

### 6. VM clock virtualization
- When a VM is paused (CPU time given to another VM), the VM's hardware clock is frozen.
- When the VM resumes, the hypervisor updates the VM's clock to the current time.
- From the application's perspective, the clock suddenly jumps forward by the pause duration.
- Pause durations for live migration: seconds to minutes, depending on memory write rate.
- This manifests as a time-of-day clock jump on resume.

---

## Clock-Based Anti-Patterns

### Anti-pattern 1: Last-write-wins with wall-clock timestamps

**How it fails:**
- In a multi-leader or leaderless database, concurrent writes to the same key from different nodes are resolved by keeping the write with the higher timestamp.
- If Node A's clock is 5 ms ahead of Node B's, a causally later write from Node B (timestamp = Node B's time) may have a lower timestamp than an earlier write from Node A (timestamp = Node A's time).
- Node B's write is discarded. No error is logged. The application has no idea.
- Example: Cassandra and Riak use LWW as a conflict resolution option.

**Concrete illustration (from the book, Figure 8-3):**
- Client A writes `x=1` on Node 1 at timestamp `42.004`.
- Client B increments `x` on Node 3, producing `x=2` at timestamp `42.003` (Node 3 is 1 ms behind).
- After replication, Node 2 receives both. LWW picks `x=1` (higher timestamp). Client B's increment is lost.
- Clock skew of 1 ms is sufficient to cause data loss. Typical inter-datacenter skew is much larger.

**Fix:**
- Replace LWW timestamp-based resolution with logical clocks (version vectors) for causal ordering.
- If LWW is required by the storage system, monitor clock offsets between nodes and alert when skew exceeds an acceptable threshold. Nodes with excessive drift should be removed.

### Anti-pattern 2: Using clock timestamps as transaction IDs for ordering

**How it fails:**
- Generating monotonically increasing transaction IDs from wall-clock time in a distributed system requires that clocks are synchronized well enough that a later transaction on any node always gets a higher timestamp than earlier transactions.
- This is not achievable with NTP alone. Two transactions on different nodes within the same millisecond may receive the same timestamp.
- Transaction ID ordering breaks; snapshot isolation semantics are violated.

**Partial solution (Google Spanner's TrueTime):**
- Spanner's TrueTime API returns `[earliest, latest]` for the current time.
- Spanner waits out the uncertainty interval before committing a transaction — ensuring that any future transaction's earliest possible timestamp is after the current transaction's latest possible timestamp.
- This provides correct ordering at the cost of commit latency proportional to the uncertainty interval (~7 ms with GPS/atomic clocks in each datacenter).
- Not practical without GPS/atomic clock infrastructure.

**Alternative:** Use logical clocks (Lamport timestamps, vector clocks) or a centralized sequence number generator for transaction IDs. Twitter's Snowflake generates approximately monotonically increasing IDs in a distributed way (but cannot guarantee consistency with causality).

### Anti-pattern 3: Clock-based lease validity check before a protected operation

Documented separately in `fencing-token-pattern.md`. In brief: checking `System.currentTimeMillis() < leaseExpiryTime` before acting on the lease is not atomic with the action. A process pause between the check and the action can cause the lease to have expired by the time the action executes.

---

## Clock Monitoring

Incorrect clocks are insidious because most things continue to work. The system does not crash or throw errors — it silently produces wrong results or drops data. This is why explicit monitoring is essential.

**Recommended monitoring:**
1. **NTP offset metrics**: track the offset between each node's clock and the NTP server. Alert when offset exceeds your acceptable threshold (typically 50–100 ms for most applications; much tighter for financial or high-precision systems).
2. **Inter-node clock skew**: for systems using timestamp-based conflict resolution, directly measure clock differences between nodes. Alert when skew approaches the granularity of your timestamps.
3. **Node removal on excessive drift**: any node whose clock drifts too far from the others should be declared dead and removed from the cluster. The node's incorrect timestamps can corrupt data or cause incorrect ordering.

**Tools:** `ntpq -p` (NTP status), `chronyc tracking` (chrony), Prometheus `node_timex_offset_seconds` metric, AWS CloudWatch `ClockErrorBound` for Spanner-equivalent services.

---

## When Clock Accuracy Really Matters

For most server-side data systems, the mitigations above are sufficient. But some domains require sub-millisecond clock accuracy:

- **High-frequency trading:** MiFID II (European regulation) requires financial institutions to synchronize clocks to within 100 microseconds of UTC.
- **Distributed tracing and log correlation:** millisecond-accurate clocks are usually sufficient; sub-millisecond is rarely needed.
- **Distributed snapshot isolation (Spanner-style):** requires GPS receivers or atomic clocks per datacenter.

For these use cases, the Precision Time Protocol (PTP, IEEE 1588) and GPS/atomic clock hardware are the relevant tools, not NTP.


path: references/failure-taxonomy.md
content: # Failure Taxonomy: Network, Clock, Process

Full reference for Step 1 and Step 2 of the distributed-failure-analyzer skill.

---

## Category A: Network Faults

### Background
The distributed systems discussed in this context are **shared-nothing systems**: nodes have their own memory and disk; communication happens only via messages over a network. Most internal datacenter networks (Ethernet) and the internet are **asynchronous packet networks**: there is no upper bound on delivery time, and no delivery guarantee.

### The Six Network Failure Modes (for a single request/response)

When a client sends a request and receives no response, any of these may have occurred:

| # | What failed | Distinguishable? | Notes |
|---|---|---|---|
| 1 | Request lost in transit | No | Cable unplugged, queue drop, routing failure |
| 2 | Request queued, not yet delivered | No | Network congestion, switch buffer full |
| 3 | Remote node crashed | No | Indistinguishable from 1, 5, 6 via timeout alone |
| 4 | Remote node temporarily unresponsive | No | GC pause, high load — node will recover |
| 5 | Response lost in transit | No | Misconfigured switch, asymmetric fault |
| 6 | Response delayed | No | Overloaded network or receiver node |

**Critical implication:** A timeout tells you that you stopped waiting. It does not tell you whether the remote node executed your request. If you retry after a timeout, the operation may execute twice.

### Network Fault Detection Signals

- **TCP RST or FIN**: the OS on the remote machine closed the port (process crashed, port not listening). Does not tell you how much work the process completed before crashing.
- **ICMP Destination Unreachable**: the router believes the IP is unreachable. Subject to the same limitations as other participants.
- **Management interface query**: for switches you control, can confirm link-level failure at hardware level. Not available in shared/cloud environments.
- **Application-level acknowledgment**: the only reliable confirmation that a request was processed correctly. TCP acknowledgment confirms delivery to the OS, not to the application.

### Network Fault Statistics (real-world)
- One study of a medium-sized datacenter: ~12 network faults per month. Half disconnected a single machine; half disconnected an entire rack.
- Adding redundant networking gear does not proportionally reduce faults — human error (misconfiguration) is a primary cause.
- Public clouds (e.g., EC2): frequent transient network glitches are documented.
- Asymmetric faults observed in practice: a network interface that drops all inbound packets but sends outbound packets successfully.

### Timeout Calibration

**Problem:** A long timeout delays failure detection. A short timeout causes false positives — declaring a live-but-slow node dead.

**Cascade failure mechanism:** Node declared dead → load redistributed to remaining nodes → remaining nodes become slower → their health checks time out → more nodes declared dead → cluster collapses.

**Calibration approach:**
1. Measure round-trip time distribution over an extended period, across many machines.
2. Set timeout at p99 or p99.9 latency under expected peak load, not median.
3. Add jitter to prevent synchronized timeout storms.
4. Consider adaptive failure detectors: **Phi Accrual** (used in Akka, Cassandra, and in TCP retransmission) dynamically adjusts the failure threshold based on observed jitter.

**Formula for a theoretical synchronous network:** If max packet delay is `d` and max processing time is `r`, timeout = `2d + r`. In asynchronous networks there is no `d`, so there is no correct value — only empirical calibration.

---

## Category B: Clock Unreliability

### Background
Each node has its own hardware clock — typically a quartz crystal oscillator. Clocks drift (run faster or slower than real time). Distributed systems cannot share a clock.

### Clock Types

| Type | Purpose | Cross-node comparison | Jump risk |
|---|---|---|---|
| Time-of-day (wall-clock) | Points in time (timestamps, event ordering) | Not safe | Yes — can jump backward on NTP reset |
| Monotonic | Elapsed time (durations, timeouts) | Never safe | No — guaranteed always forward |

**Rule of thumb:** Use monotonic clocks for timeouts and measuring durations on a single node. Never use time-of-day clocks to order events across nodes.

### Sources of Clock Error

| Source | Typical magnitude | Notes |
|---|---|---|
| Quartz drift | Up to 200 ppm (Google's server assumption) | 200 ppm = 6 ms drift if synced every 30s; 17 ms/day if synced once a day |
| NTP sync error | ~35 ms minimum on congested networks; spikes to >100 ms | Limited by network round-trip time |
| NTP firewall block | Unbounded — grows without bound | Goes unnoticed until clock is severely wrong |
| Wrong/misconfigured NTP servers | Hours off | Some public NTP servers are misconfigured |
| Leap seconds | ±1 second | A minute may be 59 or 61 seconds; has crashed production systems |
| VM time virtualization | Tens of milliseconds per pause | When a VM is paused (CPU given to another VM), the clock jumps forward on resume |
| User-controlled clocks | Arbitrary | On user devices (mobile, embedded); users may set clocks to incorrect values intentionally |

### Clock Confidence Interval
A clock reading is not a point in time — it is a range. `clock_gettime()` returns a single value but does not expose its uncertainty. The actual time may be anywhere within a confidence interval.

Google's **TrueTime API** (used in Spanner) explicitly returns `[earliest, latest]` — the earliest and latest possible current time. Spanner waits out the confidence interval before committing read-write transactions to ensure causal ordering. This requires GPS receivers or atomic clocks in each datacenter (uncertainty: ~7 ms).

For most systems without TrueTime: assume clock uncertainty of tens of milliseconds on a LAN, and design accordingly.

### Last-Write-Wins (LWW) Data Loss Mechanism

1. Client A writes `x=1` on Node 1 at wall-clock time `t=42.004`.
2. Client B increments `x` on Node 3, resulting in `x=2` at wall-clock time `t=42.003` (Node 3's clock is 1 ms behind Node 1's).
3. Both writes replicate to Node 2.
4. Node 2 applies LWW: `t=42.004 > t=42.003`, so `x=1` wins.
5. Client B's increment (`x=2`) is silently discarded. No error is reported.

**Detection:** Monitor inter-node clock offset. Alert when skew exceeds the acceptable threshold for your LWW use case. Nodes with excessive drift should be removed from the cluster before they cause data loss.

**Fix:** Replace LWW timestamp-based conflict resolution with **logical clocks** (version vectors, Lamport timestamps) for causal ordering. Logical clocks do not measure wall-clock time — they track only causal happens-before relationships, which are correct regardless of clock skew.

---

## Category C: Process Pauses

### Background
A thread can be preempted at any arbitrary point in its execution and paused for an arbitrary duration. The thread has no awareness that it was paused — when it resumes, it continues from where it left off, with no knowledge that real time has passed.

### Pause Causes

| Cause | Typical duration | Notes |
|---|---|---|
| Stop-the-world GC | Milliseconds to several seconds | JVM, .NET, Ruby. "Concurrent" collectors (CMS) still have stop-the-world phases. |
| VM suspension / live migration | Seconds to minutes | Hypervisor saves VM memory to disk; restores on another host. Cloud providers do this without notice. |
| OS context switch | Microseconds to milliseconds | Normal, but under CPU steal (multi-tenant) the paused thread may wait much longer. |
| CPU steal time | Variable | Another VM is using the shared CPU core. The paused VM's threads wait with no awareness. |
| Synchronous disk I/O | Milliseconds to seconds | Worse on network-attached storage (EBS, NFS). Java class loading can trigger this unexpectedly. |
| Memory paging (thrashing) | Seconds | OS swaps pages to disk under memory pressure. Extreme: system spends most time swapping. |
| SIGSTOP signal | Arbitrary — until SIGCONT | Sent by Ctrl-Z in a shell, or accidentally by operations tooling. |

### The Zombie Problem
When a paused node resumes:
1. It does not know it was paused.
2. Its wall-clock time jumps forward (in virtualized environments).
3. It checks its state (e.g., lease validity) and finds it appears valid.
4. It continues its previous role — even though the rest of the cluster has declared it dead, elected a new leader, or granted a new lock.
5. Result: two nodes acting as leader simultaneously, or two lock-holders writing to the same resource.

### GC Pause Mitigation (without real-time systems)
- **Treat GC pauses as planned node outages**: drain requests from the node before GC, let other nodes handle traffic during GC, resume the node after collection.
- **Use short-lived object patterns**: restart processes periodically before long-lived objects accumulate enough to trigger a full GC. This works like a rolling restart.
- **Tune GC settings**: increase heap size (delays GC), tune allocation patterns. Reduces frequency but does not eliminate pauses.
- **Monitor**: track GC pause durations. Alert on pauses that exceed your timeout budget.

---

## Summary: Symptom-to-Category Quick Reference

| Symptom | Most likely category | Secondary category |
|---|---|---|
| Intermittent timeout, node recovers | A (network) or C (process pause) | — |
| Node declared dead, then recovers with no awareness of downtime | C (process pause) | A (asymmetric network fault) |
| Two leaders active simultaneously | C (pause caused zombie) or A (split brain) | — |
| Silent data loss after concurrent writes | B (clock / LWW) | — |
| Lock corruption despite mutual exclusion code | C (pause caused zombie) | — |
| Stale reads longer than replication lag | A (delayed response) or B (clock) | — |
| Cascading node-death declarations under load | A (timeout miscalibration) | — |
| Clock-ordered events appear out of causal order | B (clock skew) | — |


path: references/fencing-token-pattern.md
content: # Fencing Token Pattern

Full reference for Step 4 of the distributed-failure-analyzer skill.

---

## Problem

A distributed lock or lease is meant to ensure that only one node acts as the chosen one (leader, lock-holder) at any time. But any locking scheme based on time-limited leases is vulnerable to process pauses:

1. Client 1 acquires a lease with a 30-second expiry.
2. Client 1 enters a stop-the-world GC pause for 40 seconds.
3. The lease expires. Client 2 acquires the lease.
4. Client 2 begins writing to the shared resource.
5. Client 1 resumes. Its clock has not advanced by 40 seconds (virtualized clock, or monotonic clock from its perspective). It checks `lease.isValid()` — the check passes. It begins writing.
6. Two writers are now active simultaneously. The shared resource is corrupted.

**This bug is documented in the HBase codebase** and is not theoretical.

The client-side lease check is the anti-pattern. The check is not atomic with the protected operation — any pause between them invalidates the check's conclusion.

---

## Solution: The Fencing Token

The lock service issues a **fencing token** with every lock or lease grant. A fencing token is a monotonically increasing integer — each new grant increments the counter. The lock-holder must include its token in every request to the protected resource. The resource tracks the highest token it has processed and rejects any request with a lower token.

### Sequence Diagram

```
Lock service                Client 1                Client 2                Storage
     |                          |                       |                       |
     |<-- acquire lock ---------|                       |                       |
     |-- ok, token=33 --------->|                       |                       |
     |                          |                       |                       |
     |           [Client 1: stop-the-world GC pause — 40 seconds]              |
     |                          |                       |                       |
     |                    lease expired                 |                       |
     |<-- acquire lock ---------|--------------------->|                       |
     |-- ok, token=34 --------------------------------->|                       |
     |                          |                       |-- write, token=34 -->|
     |                          |                       |<-- ok ----------------|
     |                          |                       |                       |
     |           [Client 1 resumes — does not know it was paused]              |
     |                          |                       |                       |
     |                          |-- write, token=33 ----------------------->|
     |                          |<-- REJECTED: token 33 < 34 seen ----------|
```

Client 1's write is rejected. No corruption.

---

## Implementation

### Requirements on the lock service
- Every lock grant returns a token that is strictly greater than all previously issued tokens.
- Tokens need not be sequential (gaps are allowed) — they must only be monotonically increasing.

### Requirements on the resource
- Maintain state: `highest_token_seen`.
- On receiving a write request with token `t`:
  - If `t > highest_token_seen`: accept the write, update `highest_token_seen = t`.
  - If `t <= highest_token_seen`: reject the write with an error indicating stale token.
- This state must be persisted durably (survives crashes).

### Requirements on the client
- Extract the fencing token from the lock service response.
- Pass the token with every request to the protected resource.
- Handle rejections: a rejection means the lock has been superseded. The client should not retry — it should abandon the operation and re-acquire the lock if it still needs access.

---

## ZooKeeper Integration

If ZooKeeper is used as the lock service:

- **`zxid` (ZooKeeper transaction ID)**: a globally monotonically increasing ID assigned to every transaction in ZooKeeper. When a client acquires a lock (creates an ephemeral node), the `zxid` of that transaction serves as the fencing token.
- **`cversion` (child version)**: the version number of a node's children, incremented on each child modification. Also monotonically increasing and usable as a fencing token.

Both are guaranteed to be monotonically increasing and are available from the ZooKeeper API response.

**Example (Java, using Curator):**
```java
InterProcessMutex lock = new InterProcessMutex(client, "/locks/my-resource");
lock.acquire();
long fencingToken = client.getZookeeperClient()
    .getZooKeeper()
    .exists("/locks/my-resource", false)
    .getCzxid();  // creation zxid — use as fencing token

storageService.write(data, fencingToken);  // pass token to resource
lock.release();
```

---

## When the Resource Does Not Natively Support Fencing Tokens

Not all storage services expose a fencing token check API. Options:

### Option 1: Conditional writes (compare-and-swap)
Most databases support conditional updates. Use an object version field:
```sql
-- Write only if version matches what we last read
UPDATE resource SET data = ?, version = version + 1
WHERE id = ? AND version = ?
```
This is not exactly fencing tokens, but achieves a similar effect: stale writers whose version does not match the current version will have their writes rejected.

### Option 2: Encode the token in the resource name
For file storage: include the fencing token in the filename or object key.
- Client acquires lock with token 33: writes to `data-33.json`.
- Client acquires lock with token 34: writes to `data-34.json`.
- Readers always use the file with the highest token number.
This is a workaround, not a full solution — it requires readers to be aware of the convention.

### Option 3: External serialization layer
Wrap the resource access in a proxy or middleware that enforces fencing token ordering before forwarding requests to the underlying resource.

---

## What Fencing Tokens Do NOT Protect Against

Fencing tokens protect against **inadvertent** zombie behavior — a node that does not know it has been declared dead, acting in good faith.

They do NOT protect against:
- A node that **deliberately** sends a faked high fencing token to bypass the check. This is a Byzantine fault.
- A resource that does not enforce the token check (client-side checking alone is insufficient).
- Race conditions at the storage layer if the fencing token check and the write are not atomic.

---

## Relationship to Leases and Heartbeats

A lease (lock with timeout) combined with fencing tokens is a robust pattern:
- The lease ensures the lock is eventually released if the holder crashes (liveness).
- The fencing token ensures that a zombie holder cannot corrupt the resource after expiry (safety).

Heartbeats alone (leader sends keep-alive to remain leader) do not solve the zombie problem — a paused leader stops sending heartbeats, gets declared dead, then resumes and continues acting as leader.

---

## Further Reading
- Kleppmann, "Designing Data-Intensive Applications," Chapter 8, pp. 302–303 (fencing tokens) and p. 295–297 (process pauses and the lease anti-pattern).
- Martin Kleppmann, "How to do distributed locking" (blog post) — expands on why Redlock (Redis-based distributed locking without fencing tokens) is unsafe.


path: references/system-models.md
content: # System Models for Distributed Algorithms

Full reference for Step 6 of the distributed-failure-analyzer skill.

---

## Why System Models Matter

A distributed algorithm is correct only within a specific system model. If the actual system violates the model's assumptions, the algorithm's correctness guarantees no longer hold. Making the model explicit allows:
1. Selecting algorithms that match the system's real behavior.
2. Identifying cases where the implementation may fail even if the algorithm is "correct."
3. Reasoning about safety and liveness properties in the presence of faults.

---

## Timing Models

### Synchronous Model
**Assumptions:** Bounded network delay, bounded process pauses, bounded clock error.

Does not mean perfectly synchronized clocks or zero delay — only that known fixed upper bounds exist.

**Reality fit:** Not realistic for most packet-switched datacenter networks or commodity hardware running general-purpose operating systems. Packet delays are unbounded; GC pauses are unbounded; clock drift has no fixed bound without dedicated hardware (GPS, atomic clocks).

**Use case:** Algorithm analysis baseline. Some embedded and real-time systems (RTOS + dedicated hardware) can approximate this model.

### Partially Synchronous Model
**Assumptions:** The system behaves like a synchronous system *most of the time*, but occasionally exceeds the bounds for network delay, process pauses, and clock drift.

**Reality fit:** Realistic for most production server-side systems. Networks are usually well-behaved; processes usually respond quickly. But any timing assumption may be violated occasionally (load spikes, GC events, VM migrations, network congestion).

**Implication:** Algorithms designed for this model must be correct even when timing bounds are temporarily exceeded. Safety properties must hold always; liveness properties are allowed to wait until the system returns to synchronous behavior.

**Recommended default for most datacenter systems.**

### Asynchronous Model
**Assumptions:** No timing assumptions whatsoever. No clock, no timeouts.

**Reality fit:** Very conservative. Some algorithms (e.g., certain consensus protocols) are designed for this model, making them maximally portable. However, the model is extremely restrictive — without timeouts, failure detection is impossible, and many practical algorithms cannot be expressed.

---

## Node Failure Models

### Crash-Stop Faults
**Assumptions:** A node fails by stopping responding. It never comes back. Once a node is declared dead, it stays dead.

**Notes:** Simplest model. Safe to use as a design assumption even if nodes do recover in practice — algorithms designed for crash-stop remain correct if nodes happen to recover (they just may not take advantage of the recovery).

### Crash-Recovery Faults
**Assumptions:** Nodes may crash at any time. They may start responding again after an unknown amount of time. On crash-recovery: stable storage (non-volatile disk) is preserved across crashes; in-memory state is lost.

**Notes:** Realistic for server-side systems with durable storage (databases, write-ahead logs). Most production distributed databases assume this model.

**Implication:** Algorithms must handle the case where a recovered node has forgotten its in-memory state. This is why distributed databases write decisions to disk before responding — the disk is the source of truth after recovery.

### Byzantine (Arbitrary) Faults
**Assumptions:** Nodes may do anything — crash, respond slowly, send incorrect or conflicting messages, lie about their state.

**Notes:** Covers adversarial behavior and hardware memory corruption. Requires supermajority quorums (>2/3 of nodes functioning correctly). Protocols are significantly more complex and expensive.

**When relevant:**
- Multi-party untrusted environments (blockchain, inter-bank settlement, peer-to-peer networks).
- High-radiation environments (aerospace, space systems) where memory corruption is a realistic concern.
- NOT relevant for typical datacenter systems where your organization controls all nodes.

---

## Safety and Liveness Properties

Every correctness property of a distributed algorithm is classified as either safety or liveness.

### Safety Properties
**Definition:** "Nothing bad happens."

Formally: if a safety property is violated, there exists a specific point in time at which the violation occurred. The violation cannot be undone — the damage is done.

**Examples:**
- Uniqueness (fencing tokens): no two requests for a fencing token return the same value.
- Monotonic sequence: tokens are always increasing.
- Linearizability: reads always see the most recent write.

**Requirement:** Safety properties must hold in *all* situations in the system model — even if all nodes crash or all network delays become infinite. The algorithm must never return a wrong result.

### Liveness Properties
**Definition:** "Something good eventually happens."

Liveness properties often include the word "eventually" in their definition.

**Examples:**
- Availability: a node that requests a fencing token and does not crash eventually receives a response.
- Eventual consistency: if no new writes occur, all replicas eventually converge to the same value.

**Requirement:** Liveness properties may include caveats. For example: "a request receives a response *if* a majority of nodes are functional *and* the network eventually recovers." The partially synchronous model requires that any period of timing violation is finite.

**Note:** Eventual consistency is a liveness property. Linearizability is a safety property. These have fundamentally different guarantees and operational implications.

---

## Recommended System Model for Most Server-Side Data Systems

**Timing model:** Partially synchronous
**Node failure model:** Crash-recovery

This combination is used by most well-known distributed algorithms (Raft, Paxos, ZooKeeper's ZAB, PBFT with crash-recovery modifications).

**Implications for your system:**
1. Timeouts will sometimes fire incorrectly — design for this.
2. A recovered node must re-read its state from stable storage before acting.
3. Safety properties (uniqueness, linearizability) must hold even during timing violations.
4. Liveness properties (eventually getting a response, eventual leader election) are allowed to stall during partitions or excessive delays.

---

## Mapping System Models to Reality

System models are abstractions. Real systems frequently encounter scenarios the model does not cover:

- **Stable storage assumed to survive crashes** — but what if the disk is corrupted by a firmware bug? What if the server fails to recognize its drives on reboot?
- **Crash-recovery assumed** — but what if a node has amnesia and forgets previously stored data (hardware failure wiping the disk)?
- **Non-Byzantine assumed** — but what if a software bug causes a node to send incorrect responses (a bug is effectively a Byzantine fault that a Byzantine-tolerant algorithm cannot save you from, if all nodes run the same buggy code)?

These edge cases do not make system models useless. They mean that:
1. Theoretical algorithm analysis and empirical testing are both required.
2. Real implementations must handle cases the model declares impossible — even if that handling is just an alert + manual intervention.
3. Safety properties in a real system are best-effort, with monitoring as the backstop.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)