Skill: quochungto/bookforge-replication-failure-analyzer (version 1.0.0)

--- name: replication-failure-analyzer description: | Diagnose active replication failures by mapping symptoms to leader failover pitfalls, replication lag anomalies, or quorum edge cases — and produce a structured remediation plan. Use when: data just written disappears or shows stale on re-read (read-after-write violation); records appear and vanish on refresh (monotonic reads violation); causally related events appear in impossible order (consistent prefix reads violation); a failover produces duplicate primary keys, write rejections, or incorrect routing; two replica nodes are both accepting writes (split brain in replication topology — for split brain via distributed locking, use distributed-failure-analyzer); quorum reads return stale values despite w + r > n; or a sloppy quorum with incomplete hinted handoff is serving old data. Applies to PostgreSQL, MySQL (single-leader), Cassandra, Riak, Voldemort, DynamoDB (leaderless). Use replication-strategy-selector first if the topology has not yet been chosen. Produces: symptom → failure class → mechanism → mitigation report, leader failover checklist, replication lag anomaly guide, and quorum edge case catalog (six ways w + r > n still fails). version: 1.0.0 homepage: https://github.com/bookforge-ai/bookforge-skills/tree/main/books/designing-data-intensive-applications/skills/replication-failure-analyzer metadata: {"openclaw":{"emoji":"📚","homepage":"https://github.com/bookforge-ai/bookforge-skills"}} status: draft depends-on: - replication-strategy-selector source-books: - id: designing-data-intensive-applications title: "Designing Data-Intensive Applications" authors: ["Martin Kleppmann"] chapters: [5] tags: - replication - failover - split-brain - replication-lag - read-after-write - monotonic-reads - consistent-prefix-reads - quorum - sloppy-quorum - hinted-handoff - leaderless - single-leader - cassandra - riak - postgresql - mysql - primary-key-conflict - data-loss - stale-reads - anti-entropy - version-vectors - last-write-wins - concurrent-writes - failure-analysis execution: tier: 2 mode: full inputs: - type: description description: "Current replication topology (single-leader, multi-leader, or leaderless), observed symptoms, and whether a failover recently occurred" - type: codebase description: "Application source code, docker-compose, or database configuration files that reveal quorum settings, read routing, conflict resolution strategy, and primary key generation" - type: document description: "Incident report, runbook, or architecture description if no codebase is available" tools-required: [Read, Write, Grep] tools-optional: [Bash] mcps-required: [] environment: "Run inside a project directory with codebase or configuration files, or accept a verbal description of the failure. Produces a written replication failure analysis report." discovery: goal: "Produce a structured replication failure analysis report: classify each symptom into its failure class, identify the specific mechanism, and recommend concrete mitigations" tasks: - "Gather symptom description, replication topology, and recent operational events (failover, partition, node restart)" - "Classify each symptom into leader failover failure, replication lag anomaly, or quorum edge case" - "Identify the specific mechanism within the class" - "Scan configuration and codebase for anti-patterns (timeout values, read routing, quorum parameters, primary key generation)" - "Recommend mitigations matched to root cause" - "Produce the failure analysis report" audience: roles: ["backend-engineer", "software-architect", "site-reliability-engineer", "data-engineer", "tech-lead"] experience: "intermediate-to-advanced — assumes basic replication familiarity (what a leader is, what a follower is, what a quorum is)" triggers: - "User's submitted data disappeared or shows stale immediately after a write" - "Data appears then vanishes on refresh — time appears to move backward" - "Causally related records appear out of causal order (answer before question)" - "Failover completed but writes are rejected, misrouted, or creating duplicate key conflicts" - "Two nodes are both accepting writes simultaneously" - "Quorum reads returning stale values despite w + r > n" - "Hinted handoff in progress and fresh reads returning old data" - "Infrequently-read keys diverging silently across replicas" - "Proactive review of replication config before shipping to production" not_for: - "Choosing a replication topology — use replication-strategy-selector" - "Diagnosing network faults, zombie leaders, or clock unreliability — use distributed-failure-analyzer" - "Selecting consistency and isolation guarantees at the transaction level — use consistency-model-selector or transaction-isolation-selector" --- ## When to Use You have a replication failure in progress or an anomaly you cannot explain. The symptom is one of: data that was written cannot be read back; data that was visible has disappeared; records arrive in causally impossible order; two nodes are writing simultaneously; a quorum read is stale despite the math being correct. This skill imposes a diagnostic framework: every replication failure traces to one of three classes — **leader failover pitfalls**, **replication lag anomalies**, or **quorum edge cases**. Each class has a bounded set of mechanisms with known mitigations. The skill maps your symptoms to a class, narrows to the mechanism, and produces a remediation plan. This is the companion to `replication-strategy-selector`. That skill helps you choose. This skill helps you diagnose. Use this one when something is already wrong, or when you want to audit a configuration for latent failure before it manifests. Cross-references: - `replication-strategy-selector` — for choosing topology, sync mode, quorum values, and conflict resolution strategy from scratch - `distributed-failure-analyzer` — for failures whose root cause is a network fault, clock unreliability, or process pause (zombie leaders, LWW data loss via clock skew, cascading timeouts) - `consistency-model-selector` — for selecting the right consistency and isolation guarantees to prevent a class of anomalies at the application layer --- ## Context and Input Gathering Before analysis, collect the following. Ask the user for any that are missing. **Required:** 1. **Symptom description** — what was observed vs. what was expected. Be specific: "a user submitted a comment and immediately reloaded the page but did not see their comment" is better than "stale reads." 2. **Replication topology** — single-leader, multi-leader, or leaderless (Dynamo-style). If leaderless: number of replicas (n), write quorum (w), read quorum (r). If single-leader: synchronous, semi-synchronous, or asynchronous? 3. **Recent operational events** — did a failover happen recently? Was a node restarted? Was there a network partition? Was a follower promoted? Any configuration changes in the past 48 hours? **Useful:** 4. **Read routing strategy** — do reads go to the leader only, to any replica, or to a specific replica? Is there a load balancer routing reads randomly? 5. **Primary key / ID generation strategy** — autoincrement, UUID, application-generated? Is any external system (cache, secondary store) keyed on these IDs? 6. **Quorum configuration** — for leaderless systems: exact w, r, n values; whether sloppy quorums are enabled; whether an anti-entropy background process is configured. 7. **Replication log metrics** — replication lag in seconds or write offset delta, if available. **If no codebase or configuration is available:** accept a verbal description and produce an analysis. The report will note which findings are confirmed vs. inferred. --- ## Process ### Step 1 — Classify the failure WHY: The three failure classes have different root causes and completely different mitigations. Treating a replication lag anomaly as a failover problem (or vice versa) leads to wasted effort and leaves the actual failure in place. Classification first prevents this. **Class A: Leader failover pitfalls** Applies to single-leader replication. A failover is the process of promoting a follower to be the new leader when the current leader fails. Automatic failover typically follows three steps: (1) detect the leader has failed via timeout, (2) elect a new leader (usually the most up-to-date replica, chosen by election or by a previously elected controller node), (3) reconfigure clients to route writes to the new leader and ensure the old leader becomes a follower if it recovers. Failover is "fraught with things that can go wrong." The four documented failure modes are: | Failure mode | Mechanism | Signal | |---|---|---| | **Async data loss** | New leader was an async follower — had not received all writes from old leader before failure. Old leader's unreplicated writes are discarded. | Writes confirmed to client are missing after failover | | **Primary key conflict** | New leader's autoincrement counter lagged behind old leader's. New leader reissues keys already assigned by the old leader. Any system keyed on these IDs (Redis cache, secondary DB, audit log) develops cross-system inconsistency. | Duplicate key errors or wrong data returned for existing IDs in external systems | | **Split brain** | Old leader recovers and does not recognize the new leader. Both nodes accept writes simultaneously. Without a process to resolve conflicts, data is lost or corrupted. Some systems "shut down one node if two leaders are detected" — but if this mechanism is misconfigured, both nodes may shut down. | Two nodes both reporting as leader; writes going to both; diverging replica state | | **Timeout miscalibration** | Timeout too short: unnecessary failovers under load spike or network glitch, making the situation worse. Timeout too long: prolonged unavailability during genuine failures. A temporary load spike can cause response time to exceed the timeout, triggering a failover that increases load further. | Repeated failovers during traffic spikes; or prolonged unavailability before failover triggers | **Class B: Replication lag anomalies** Applies to single-leader asynchronous replication with read-scaling (reads routed to followers). The replication lag — the delay between a write being applied on the leader and being reflected on a follower — may be milliseconds under normal conditions, but can grow to seconds or minutes under load or network issues. Three named anomaly patterns arise: | Anomaly | Description | Mechanism | Named guarantee required | |---|---|---|---| | **Read-after-write violation** | User submits data; immediately reads it back; does not see it. From the user's perspective, the submission was lost. | Read was routed to a follower that had not yet received the write. | Read-after-write consistency (also called read-your-writes) | | **Monotonic reads violation** | User reads data (e.g., a comment); reloads the page; the data is gone. Time appears to move backward. | Sequential reads were routed to different replicas with different lag. The second read went to a more-lagged replica that had not yet received the write the first read saw. | Monotonic reads | | **Consistent prefix reads violation** | User sees causally related records in an impossible order — an answer appearing before the question it answers. | In a partitioned database where partitions operate independently, partition A (carrying the reply) had low lag and partition B (carrying the question) had high lag. The observer read partition A first. | Consistent prefix reads | **Class C: Quorum edge cases** Applies to leaderless (Dynamo-style) replication: Cassandra, Riak, Voldemort, DynamoDB. The quorum condition w + r > n is designed to ensure that at least one node in every read set has seen every acknowledged write. However, six scenarios break this guarantee in practice even when the condition is mathematically satisfied: | Edge case | Mechanism | |---|---| | **Sloppy quorum active** | A network interruption isolated the client from the n "home" nodes for a value. Writes were accepted by w nodes outside the home set (sloppy quorum). Even though w + r > n, the r read nodes are the home nodes — they have not seen the writes yet. Hinted handoff has not completed. | | **Concurrent writes, no clear ordering** | Two writes to the same key occurred simultaneously. The quorum condition does not determine which write happened first. If last-write-wins is the conflict resolution strategy, the write with the lower timestamp (possibly the causally later write, if clocks are skewed) is silently discarded. | | **Write concurrent with read** | A write was in-flight when a read was issued. The write was reflected on some of the r replicas but not others. The read may return the old value, the new value, or — in the worst case — the read returns the old value and the write is subsequently applied, but a future read may still return the old value from a different replica subset. | | **Partial write success, no rollback** | A write succeeded on some replicas but failed on others (e.g., disk full) and was reported as failed overall (fewer than w acknowledgements). The replicas that did succeed are not rolled back. Subsequent reads may or may not see the partially-written value. | | **Node restored from stale replica** | A node carrying a new value fails and its data is restored from a replica carrying an old value. The number of replicas storing the new value falls below w, breaking the quorum condition retrospectively. | | **Timing edge cases at linearizability boundary** | Even with w + r > n fully satisfied, quorum reads are not linearizable — there are race conditions where unlucky timing can produce stale reads. Quorums provide eventual consistency, not linearizability. | --- ### Step 2 — Identify the specific mechanism WHY: The class narrows the diagnostic space; the specific mechanism determines which mitigation is effective. "Replication lag anomaly" does not tell you whether you need sticky routing, a timestamp-based threshold, or causal consistency at the partition level — only identifying the exact anomaly pattern does. **For Class A (leader failover):** Confirm which of the four failure modes is active by asking: - Are writes that were confirmed missing after failover? → Async data loss. Check whether the promoted follower was in synchronous or asynchronous mode with the old leader. - Are duplicate key errors or cross-system inconsistencies appearing after failover? → Primary key conflict. Check whether the new leader's autoincrement counter was reset or is behind the old leader's. Check for any external systems keyed on the same IDs. - Are two nodes both accepting writes? → Split brain. Check each node's leader status. Check whether fencing / STONITH (Shoot The Other Node In The Head) is configured and whether it fired correctly. - Are failovers happening repeatedly or under load? → Timeout miscalibration. Check the configured timeout value against observed p99 latency under load. **For Class B (replication lag anomalies):** Confirm which anomaly pattern is active by asking: - Does the stale read affect only the user who just wrote, immediately after a write? → Read-after-write violation. - Does a previously visible record disappear on refresh? → Monotonic reads violation. - Does a reply appear before its question, or an effect appear before its cause? → Consistent prefix reads violation. For each anomaly, identify whether the read routing layer can be changed (application-level) or whether the database must be configured to provide the guarantee (database-level). **For Class C (quorum edge cases):** Confirm which edge case applies: - Was there a network partition recently? Is hinted handoff in progress? → Sloppy quorum edge case. - Were concurrent writes made to the same key? Is LWW the conflict resolution strategy? → LWW + concurrent write data loss. Check clock skew between nodes. - Is the write partially applied (error reported on write, but no rollback)? → Partial write success. - Was a node recently restored from backup or from a replica? → Stale node restoration. - Is the failure intermittent under high concurrency? → Timing edge case at linearizability boundary. --- ### Step 3 — Apply the mitigation WHY: Each mechanism has a specific mitigation. Applying the wrong mitigation (e.g., increasing quorum for a read-after-write problem in a single-leader system) wastes effort and may introduce new problems. This step matches mechanism to fix precisely. #### Class A mitigations: leader failover **Async data loss:** - Short-term: accept that unreplicated writes from the old leader are lost. Inform users; roll back any downstream effects (cache entries, external system records). - Long-term: configure at least one synchronous follower (semi-synchronous replication). PostgreSQL's `synchronous_standby_names`, MySQL's semi-sync replication. Accept the latency cost on writes. Alternatively: use a consensus-based replication protocol (Group Replication, Galera) where a write is not confirmed until a majority has applied it. **Primary key conflict:** - Immediate: reset the new leader's autoincrement counter to a value above the old leader's highest known key. Query the old leader's `AUTO_INCREMENT` status before decommissioning it; if the old leader is unavailable, query all external systems for the highest key they have seen. - Invalidate or reconcile any external system (Redis, cache, secondary DB) keyed on the affected ID range. - Long-term: switch to UUIDs or application-generated IDs that are globally unique and do not depend on a per-node counter. This eliminates the class of failure entirely. **Split brain:** - Immediate: manually fence the old leader — force it offline, revoke its write permissions at the network or storage layer, or trigger STONITH if configured. - Reconcile diverged writes: identify writes accepted by both nodes during the split brain window. Resolve conflicts using the conflict resolution strategy appropriate to the data (last-write-wins if losing some writes is acceptable; application-level merge otherwise). - Long-term: configure a fencing mechanism. Ensure the fencing path is tested periodically. Consider whether a consensus protocol (Raft, Paxos) for leader election would prevent the situation — these protocols guarantee that only one leader is active at a time. **Timeout miscalibration:** - Measure round-trip time distribution empirically under peak load. Set the failure-detection timeout at p99 or p99.9 latency plus a safety margin. - If repeated unnecessary failovers are occurring: increase the timeout. If the timeout is too long and genuine failures are causing excessive unavailability: decrease it — but only after measuring the latency distribution. - Consider a Phi Accrual failure detector (used in Cassandra, Akka) that adapts its failure threshold based on observed heartbeat jitter, rather than a fixed timeout. - Some operations teams prefer manual failover even when automatic failover is available, specifically to avoid miscalibrated-timeout failures during load spikes. #### Class B mitigations: replication lag anomalies **Read-after-write violation:** Multiple techniques can implement read-after-write consistency: 1. **Route the user's own data reads to the leader.** For data that only the user themselves can modify (e.g., a user profile), always read from the leader. Read other users' data from followers. This requires knowing, at read time, which data belongs to the current user. 2. **Time-window leader reads.** For one minute (or some configurable interval) after the user's last write, route all that user's reads to the leader. This covers the case where most things in the application are potentially editable by the user. 3. **Client-side write timestamp, replica lag threshold.** The client records the timestamp of its last write (or, better, the replication log position / log sequence number returned by the write). When reading, route to any replica that has advanced past that position. If no such replica is available, either wait or route to the leader. 4. **Cross-device consistency (additional complexity).** If the user accesses your service from multiple devices, the write timestamp must be stored centrally (not just in local storage on the device that made the write). All reads from all of that user's devices must be routed according to that centralized timestamp. **Monotonic reads violation:** Ensure that a given user's sequential reads always go to the same replica. The replica can be chosen based on a hash of the user ID rather than randomly. This ensures the user's observed state only moves forward in time. Caveat: if the assigned replica fails, the user's reads must be rerouted to another replica. At that moment, monotonic reads may be violated for the duration until the new replica catches up. This is generally acceptable — the guarantee is "best effort" in the face of replica failure. **Consistent prefix reads violation:** Ensure that writes with causal dependencies are written to the same partition. This prevents the ordering inversion — if the question and answer go to the same partition, the follower will always apply them in the correct order. If causally related writes cannot always be co-located on the same partition (because the data model makes this impractical): use a database or middleware layer that tracks causal dependencies explicitly (causal consistency via version vectors) and ensures that a read does not return a causally later write without also returning its causal prerequisites. #### Class C mitigations: quorum edge cases **Sloppy quorum / hinted handoff in progress:** - Wait for hinted handoff to complete before serving reads that require up-to-date data. Monitor the hinted handoff queue to determine when it is empty. - If waiting is not acceptable: disable sloppy quorums (`durable_writes = true` in Cassandra, `allow_offline_hnodes = false` in Riak) to get strict quorum behavior at the cost of lower availability during network partitions. - Do not treat sloppy quorums as equivalent to strict quorums. They are a durability guarantee only — "w nodes somewhere accepted the write" — not a freshness guarantee. **LWW + concurrent writes:** - If data loss is not acceptable: do not use last-write-wins as the conflict resolution strategy. Switch to version vectors (Riak supports these natively as "dotted version vectors") which allow conflicting writes to be detected and merged rather than silently discarded. - If LWW must be retained: use a UUID as the key so each write operation gets a unique key, making concurrent writes to the same key impossible. This is the recommended Cassandra pattern for LWW safety. - Monitor clock skew between nodes. Alert when skew exceeds the acceptable data-loss threshold. **Partial write success:** - Accept that a reported-failed write may have partially applied. Applications that require strict atomicity across all replicas must use a database with transaction support, not an eventually consistent leaderless store. - Design operations to be idempotent where possible so they can be safely retried or re-applied during read repair. **Stale node restoration:** - Before restoring a node from backup or from another replica, assess how many replicas currently hold the latest value for affected keys. If restoring would drop the count below w, delay the restoration until a full-cluster repair can be run first. - After restoring a node, run a full repair (Cassandra: `nodetool repair`; Riak: `riak-admin repair`) before routing reads to it. **Linearizability requirement:** - If the application genuinely needs linearizable reads (not just eventual consistency), quorums alone are insufficient. Options: (1) route all reads and writes through the single-leader (leaderless systems can designate a read-repair coordinator), (2) use a consensus protocol (Raft, Paxos) which provides linearizability, or (3) upgrade to a database with linearizable transaction support (refer to `transaction-isolation-selector`). --- ### Step 4 — Scan configuration and codebase for anti-patterns WHY: Latent replication failures exist in configuration and code before they manifest in production. Proactive scanning finds them at low cost. These are the specific patterns to search for. **Anti-pattern 1: Reads always routed to any random follower** ``` # Look for: round-robin read balancing, random replica selection # or: load balancer distributing reads across all replicas without session affinity ``` Risk: Read-after-write and monotonic reads violations under any replication lag. Any write may be invisible to a read that lands on a different, more-lagged follower. Fix: Implement user-session sticky reads or timestamp-gated replica selection. **Anti-pattern 2: Autoincrement primary keys with asynchronous replication and external systems** ``` # Look for: AUTO_INCREMENT columns, SERIAL columns, sequences # combined with: external system (Redis, Elasticsearch, audit log) using the same IDs # combined with: asynchronous replication with manual or automatic failover ``` Risk: After failover, the new leader reissues IDs that were already assigned by the old leader but not yet replicated. The external system retains entries for the old IDs; the new leader's records point to different data. Fix: Use UUIDs or application-generated globally unique IDs. Or, ensure the autoincrement sequence is advanced past the old leader's maximum before the new leader begins accepting writes. **Anti-pattern 3: No fencing / STONITH configured for leader failover** ``` # Look for: automatic failover configuration without a fencing mechanism # e.g., Patroni without fencing, MHA without power fencing, manual failover runbooks ``` Risk: The old leader recovers and does not know it has been demoted. Both nodes accept writes. Split brain. Fix: Configure a fencing mechanism. Test it periodically. See `distributed-failure-analyzer` for fencing token implementation details. **Anti-pattern 4: Sloppy quorums enabled in a system requiring read freshness** ``` # Cassandra: read_repair_chance and dclocal_read_repair_chance < 1.0 # with no anti-entropy (nodetool repair) schedule # Riak: allow_mult = false (no sibling handling) with sloppy quorums # Voldemort: default config enables sloppy quorums ``` Risk: During and after network partitions, reads return stale data despite w + r > n being satisfied, because the w writes went to non-home nodes. Fix: Either disable sloppy quorums (strict quorum mode) or implement application-layer awareness of hinted handoff status. **Anti-pattern 5: No anti-entropy process, relying solely on read repair** ``` # Cassandra: nodetool repair not scheduled # Voldemort: no anti-entropy configured # Custom leaderless system: no background reconciliation ``` Risk: Values that are rarely read will diverge permanently across replicas. Read repair only runs when a value is actually read. Infrequently-read keys can remain stale indefinitely, violating durability guarantees. Fix: Schedule regular anti-entropy runs. For Cassandra: `nodetool repair` on a weekly schedule (or more frequently for high-write workloads). Ensure the interval is shorter than the gc_grace_seconds (tombstone expiry period) to prevent deleted data from "coming back." --- ### Step 5 — Produce the failure analysis report Output a structured report with: 1. **Executive summary** — what failed, which failure class, operational impact. 2. **Symptom-to-mechanism mapping** — for each symptom: class, specific mechanism, confidence level (confirmed / inferred / possible). 3. **Immediate remediation** — ordered list of steps to stop the active failure. 4. **Anti-patterns found** — code or configuration locations (if available), description, risk. 5. **Long-term fixes** — architectural changes to prevent recurrence. 6. **Open questions** — what additional information (metrics, config values, code) would increase diagnostic confidence. --- ## Common Misdiagnoses **"The read is stale — this must be a replication bug."** Stale reads in asynchronous replication are expected behavior, not a bug. The replication lag is working as designed. The issue is that the application assumed synchronous replication behavior but is running in asynchronous mode. The fix is application-level read routing, not replication reconfiguration. **"We set w + r > n so our reads must be consistent."** The quorum condition ensures overlap between write and read node sets under normal conditions. It does not guarantee freshness when: a sloppy quorum was used (writes went to non-home nodes), concurrent writes occurred with LWW resolution and clock skew, or a write partially succeeded. Quorums provide eventual consistency by default, not linearizability. **"The failover succeeded — why are there duplicate key errors?"** The promoted follower's autoincrement counter reflects the writes it received before failover. If the old leader had advanced its counter further (on writes not yet replicated), the new leader's counter is behind. When the new leader issues new IDs, it reuses IDs the old leader already assigned. This is especially dangerous when an external system (Redis, Elasticsearch) is keyed on these IDs — the external system retains entries for the old IDs, and the new leader's records now point to different data in the external system. **"We have two leaders — one of them must be wrong."** Both leaders may believe they are legitimate. The old leader did not receive the demotion message (it may have been partitioned when the new leader was elected, or the fencing mechanism failed to fire). The solution is not to query which one is "right" but to forcibly fence the old leader and then reconcile the writes it accepted during the split brain window. **"Monotonic reads just means we need stronger consistency."** Monotonic reads is a weaker guarantee than strong consistency. It only requires that a single user's reads do not observe an older state after having observed a newer state. It does not require that all users see the same state at the same time. Implementing it with sticky replica routing is significantly cheaper than requiring strong consistency across the cluster. **"The quorum write failed, so the data wasn't written."** A failed quorum write means fewer than w nodes acknowledged. But the nodes that did acknowledge are not rolled back. The write may be partially applied across some replicas. Subsequent reads may or may not return the partially-written value, depending on which replicas the read contacts. Applications that retry a failed write without making it idempotent can create inconsistencies. --- ## Examples ### Example 1: GitHub-style primary key conflict after MySQL failover **Scenario:** A team runs MySQL with a single-leader replication topology. During maintenance, a follower is promoted to leader. Shortly after, users start seeing other users' private data — profile photos and messages belonging to a different account. **Trigger:** Security incident report. Immediate investigation required. **Process:** 1. Classify: failover occurred recently + cross-user data leakage → Class A, primary key conflict mechanism. 2. Mechanism: The old leader had issued autoincrement IDs for rows not yet replicated to the promoted follower. The new leader's `AUTO_INCREMENT` counter started below the old leader's maximum. The new leader reissued IDs already assigned by the old leader. A Redis cache was storing user profile data keyed on MySQL row IDs. Redis entries for the old IDs now returned different users' profile data because the new leader's rows with those IDs belong to different users. 3. Immediate remediation: (a) Take the service offline. (b) Query Redis for the highest user ID present. (c) Advance the new leader's `AUTO_INCREMENT` past that value. (d) Invalidate all Redis entries in the affected ID range. (e) Audit which users were served incorrect data and notify them. 4. Long-term fix: (a) Switch to UUID primary keys. (b) Decouple external system keys from database autoincrement IDs. (c) Add a post-failover checklist that explicitly includes advancing the autoincrement counter before opening traffic. **Output:** Failure analysis report identifying primary key conflict as root cause. Immediate remediation steps. UUID migration plan. Updated failover runbook with autoincrement counter validation step. --- ### Example 2: Read-after-write and monotonic reads violations in a social feed **Scenario:** A social application uses a single-leader MySQL setup with five followers. Reads are distributed across all followers via a round-robin load balancer. Users regularly report that comments they just posted do not appear when they reload the page. Occasionally, a comment that was visible disappears and then reappears. **Trigger:** Support ticket volume on "my posts disappear" exceeds threshold. Product team requests investigation. **Process:** 1. Classify: user's own writes not visible on reload → Class B, read-after-write violation. Comments appearing and disappearing → Class B, monotonic reads violation. 2. Mechanism for read-after-write: Round-robin routing sends the reload request to a different follower than the one that received the previous read (or to any follower, including heavily lagged ones). The follower has not yet received the write. 3. Mechanism for monotonic reads: The first read lands on a follower with low lag (comment visible). The reload lands on a follower with higher lag (comment not yet present). The user observes time moving backward. 4. Mitigation: (a) For the user's own content: route reads to the leader for one minute after any write from that user. Implement by storing `last_write_at` in the user session. After one minute, route to followers. (b) For monotonic reads: hash user ID to always route to the same follower. If that follower fails, reroute — accepting a brief monotonic reads violation during failover. (c) Scan the load balancer configuration to confirm round-robin routing is what is actually configured (vs. session-sticky or latency-aware routing). **Output:** Failure analysis report. Read routing change specification. Session-layer implementation plan for `last_write_at` tracking. Load balancer reconfiguration recommendation. --- ### Example 3: Stale quorum reads after network partition in Cassandra **Scenario:** A team runs Cassandra with n=3, w=2, r=2 (satisfying w + r > n). After a 20-minute network partition between two datacenters, quorum reads are returning values that are several minutes old. The partition healed 10 minutes ago. **Trigger:** Monitoring alert showing read staleness exceeding acceptable threshold after a network event. **Process:** 1. Classify: quorum reads stale after partition healing, despite w + r > n → Class C, sloppy quorum edge case. 2. Mechanism: During the partition, the datacenter-1 nodes were isolated from the three "home" nodes for affected keys. Cassandra's sloppy quorums (enabled by default in many configurations) allowed writes to be accepted by datacenter-1 nodes that are not in the home node set. After partition healing, hinted handoff is replaying those writes to the home nodes — but the process is not yet complete. Reads contacting the home nodes return the pre-partition values. w + r > n is satisfied in terms of node counts, but the r read nodes are not the same nodes that received the w sloppy-quorum writes. 3. Check: confirm that `nodetool tpstats` shows a non-empty hints queue. Monitor the queue draining rate. 4. Mitigation: (a) Wait for hinted handoff to complete (monitor `HintsService` metrics). (b) For values that must be current: force a read repair by issuing a `CONSISTENCY QUORUM` read or running `nodetool repair` on the affected keyspace. (c) Long-term: evaluate whether sloppy quorums provide acceptable trade-offs for this keyspace. For data requiring freshness guarantees, configure `LOCAL_QUORUM` with `durable_writes = true` and disable sloppy quorums. For data tolerating eventual consistency, sloppy quorums increase availability and are appropriate. **Output:** Failure analysis report. Hinted handoff monitoring procedure. Decision framework for which keyspaces should use strict vs. sloppy quorums. Updated runbook for post-partition recovery validation. --- ## References - `references/failover-checklist.md` — step-by-step leader failover checklist: pre-failover verification, the four failure modes and their per-mode checks, post-failover validation steps, and rollback procedure - `references/lag-anomaly-patterns.md` — complete replication lag anomaly reference: read-after-write, monotonic reads, and consistent prefix reads — each with formal definition, concrete example, implementation techniques, and cross-device complexity considerations - `references/quorum-edge-cases.md` — the six quorum edge cases in detail: conditions that trigger each, detection signals, mitigation options, and the distinction between sloppy quorums (durability guarantee) and strict quorums (freshness guarantee) ## License This skill is licensed under [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). Source: [BookForge](https://github.com/bookforge-ai/bookforge-skills) — Designing Data-Intensive Applications by Martin Kleppmann. ## Related BookForge Skills Install related skills from ClawhHub: - `clawhub install bookforge-replication-strategy-selector` Or install the full book set from GitHub: [bookforge-skills](https://github.com/bookforge-ai/bookforge-skills)

path: references/failover-checklist.md
content: # Leader Failover Checklist

Use this checklist when performing or recovering from a leader failover in a single-leader replication setup (PostgreSQL, MySQL, or any leader-follower topology).

---

## Pre-Failover Verification (Planned Failover)

Before initiating a planned failover (e.g., maintenance, upgrade):

1. **Identify the most up-to-date follower.**
   - Check replication lag for each follower. Choose the one with the smallest lag.
   - Ideal: zero lag (follower is fully caught up). If no follower is at zero lag, accept the one with the lowest lag and document the expected data loss window.
   - PostgreSQL: `SELECT * FROM pg_stat_replication;` (on leader). Check `write_lag`, `flush_lag`, `replay_lag`.
   - MySQL: `SHOW SLAVE STATUS\G` on each follower. Check `Seconds_Behind_Master`.

2. **Check the autoincrement / sequence counter on the old leader.**
   - MySQL: `SHOW TABLE STATUS LIKE 'tablename'` → `Auto_increment` column.
   - PostgreSQL: `SELECT last_value FROM tablename_id_seq;`
   - Record this value. It will be needed to advance the new leader's counter after promotion.

3. **Check for any external systems keyed on primary IDs.**
   - Identify Redis caches, Elasticsearch indices, audit logs, or secondary databases that store records keyed on the same IDs.
   - List them. You will need to reconcile or invalidate these after failover.

4. **Verify fencing mechanism is operational.**
   - Confirm that STONITH (Shoot The Other Node In The Head), power fencing, or network-level fencing is configured and can fire.
   - Test: if this is a staging environment, trigger the fencing mechanism manually to confirm it works before you need it in production.

---

## During Failover

5. **Stop writes to the old leader** (planned failover only).
   - Gracefully drain connections.
   - Wait for replication lag to reach zero on the chosen follower.

6. **Promote the follower.**
   - PostgreSQL: `pg_promote()` or `touch /tmp/postgresql.trigger`.
   - MySQL: `STOP SLAVE; RESET SLAVE ALL;` on the new leader.
   - Patroni, MHA, Orchestrator: use their promotion commands rather than manual promotion.

7. **Reconfigure clients.**
   - Update the write endpoint (connection string, load balancer, service discovery record) to point to the new leader.
   - Existing connections to the old leader will fail. Applications must handle reconnection.

8. **Ensure the old leader becomes a follower** (if it recovers).
   - Configure the old leader to replicate from the new leader when it comes back online.
   - Do not allow the old leader to resume accepting writes.

---

## Post-Failover Validation

9. **Advance the new leader's autoincrement counter.**
   - MySQL: `ALTER TABLE tablename AUTO_INCREMENT = <value above old leader's max>;`
   - PostgreSQL: `SELECT setval('tablename_id_seq', <value above old leader's max>);`
   - WHY: If the old leader had issued IDs that were not replicated, the new leader's counter may be behind the old leader's maximum. The new leader will reissue those IDs, causing conflicts in any external system that has already indexed them.

10. **Reconcile or invalidate external systems.**
    - For each external system identified in step 3: invalidate any cached entries in the ID range between (new leader's starting counter) and (old leader's last known max).
    - If the old leader is available: query it for the exact set of IDs it issued that were not replicated. Compare with the new leader.

11. **Verify write traffic is routing to the new leader only.**
    - Check application logs. Look for write attempts to the old leader's address.
    - Check the old leader's write count metric — it should be zero.

12. **Confirm no split brain.**
    - Verify that the old leader (if it recovered) is in follower mode.
    - Check its replication status — it should be replicating from the new leader, not accepting independent writes.
    - If the old leader is accepting writes: immediately fence it (trigger STONITH or revoke its network write access). Then reconcile diverged writes.

---

## The Four Failure Modes: Quick Reference

| Mode | Signal | Immediate action |
|---|---|---|
| Async data loss | Writes confirmed before failover are missing on new leader | Accept loss; roll back downstream effects; advance autoincrement; notify affected users |
| Primary key conflict | Duplicate key errors in application; cross-user data leakage in external systems | Take service offline; advance autoincrement counter past old leader's max; invalidate external system ID range |
| Split brain | Two nodes both reporting as leader; write divergence | Fence old leader immediately; reconcile diverged writes; audit data integrity |
| Timeout miscalibration | Repeated unnecessary failovers under load; or excessive unavailability before failover fires | Measure p99 latency under load; recalibrate timeout; add load-sensitive trigger guard |

---

## Rollback Procedure

If the failover cannot be completed cleanly (e.g., the promoted follower has corruption or the new leader cannot accept connections):

1. Fence the new leader (prevent it from accepting writes).
2. Restore the old leader to primary status.
3. Re-establish replication from the old leader to all followers.
4. Investigate the root cause before attempting another failover.

Do not attempt a second failover while the first failover's state is unresolved. You can accumulate two concurrent leaders if the fencing from the first failover was not confirmed.


path: references/lag-anomaly-patterns.md
content: # Replication Lag Anomaly Patterns

Three named anomaly patterns arise when single-leader asynchronous replication is combined with read-scaling (reads routed to followers). Each has a formal definition, a concrete example, implementation techniques for the mitigation, and additional complexity notes for multi-device scenarios.

---

## Pattern 1: Read-After-Write Violation

**Also known as:** read-your-writes consistency violation.

**Formal definition:** A user makes a write, then issues a read. The read does not reflect the write. The user observes their own submission as "lost."

**Why it happens:** The write goes to the leader. The read is routed to a follower. If the follower has not yet received and applied the write (replication lag is nonzero), the follower returns the pre-write value. The user sees no data.

**Concrete example:** A user edits their profile bio and clicks Save. They reload their profile page. Their new bio is not shown — the old bio is still displayed. From the user's perspective, the edit failed silently.

**Distinguishing characteristic:** The anomaly affects only the user who made the write, and only their own data. Other users' reads are unaffected (they never had the new value, so they are not surprised by its absence).

**Mitigation techniques:**

*Technique 1: Selective leader reads for own data.*
Route reads for data that the user themselves can modify directly to the leader. Route reads for other users' data to followers. This requires knowing at read time which data belongs to the requesting user.
- Example: user profile pages always read the current user's profile from the leader; other users' profiles are read from followers.
- Drawback: if the application is highly personalized and most data is potentially modifiable by the user, most reads must go to the leader, negating the read-scaling benefit.

*Technique 2: Time-window leader reads.*
For a fixed interval after the user's last write (e.g., one minute), route all reads from that user to the leader. After the interval, resume reading from followers.
- Requires storing `last_write_at` per user in session state.
- The interval should be longer than the expected maximum replication lag. If lag occasionally exceeds the interval, violations can still occur — this is a heuristic, not a guarantee.

*Technique 3: Replication position tracking.*
The write returns the replication log position (PostgreSQL: LSN; MySQL: binlog coordinates) at which the write was applied on the leader. The client stores this position. On subsequent reads, route to any follower whose replication position is at or beyond the stored position. If no such follower is available: either wait for one to catch up, or route to the leader.
- This provides a hard guarantee rather than a heuristic.
- Requires the load balancer or client driver to query each follower's current replication position before routing.
- Supported by some drivers: Vitess supports this via `@primary` and position-aware routing. PgBouncer can be configured for this with custom routing logic.

**Cross-device complexity:** If the user submits data on one device (desktop) and reads it on another (mobile), technique 1 and 2 may fail because the `last_write_at` is stored in the desktop session, which the mobile device does not have access to. Centralizing the write timestamp in the user record (database or server-side session store) is required for correct cross-device read-after-write consistency.

---

## Pattern 2: Monotonic Reads Violation

**Formal definition:** A user makes multiple sequential reads. The second read returns a state that is older than the state returned by the first read. Time appears to move backward.

**Why it happens:** Sequential reads from the same user are routed to different replicas (e.g., random or round-robin routing). Replica A has low lag and returns the new value. Replica B has higher lag and has not yet received the write. The second read goes to Replica B and returns the old value.

**Concrete example:** A user refreshes a social media feed. The first refresh shows a comment posted by another user. The second refresh does not show the comment — the second request was routed to a more-lagged replica. The comment appears to have been deleted (but it was never deleted).

**Distinguishing characteristic:** The anomaly manifests as disappearing data on successive reads by the same user. It does not require the user to have made any writes. It is caused entirely by read routing hitting replicas with different lag.

**Mitigation technique:**

*Sticky replica routing:* Route all reads from a given user to the same replica. Choose the replica based on a hash of the user ID (or session ID) rather than randomly. This ensures that as long as the user's reads land on the same replica, the replica's state only moves forward — even if that replica is behind the leader, its local state is monotonically advancing.

```
replica_index = hash(user_id) % number_of_replicas
route_to = replicas[replica_index]
```

**Important caveat:** If the assigned replica fails, the user's reads must be rerouted. At the moment of rerouting, the new replica may be in a different lag state (possibly behind the failed replica's last-seen state). A brief monotonic reads violation is acceptable at this point — the failure case is typically handled by logging in the user out and back in (resetting their read state), or simply accepting the anomaly as a known rare occurrence.

**Interaction with read-after-write:** If sticky routing assigns the user to a follower that has not yet received their own write, a read-after-write violation can occur even with sticky routing in place. Sticky routing solves monotonic reads but does not solve read-after-write. Both mitigations may be needed simultaneously. The read-after-write mitigation (leader reads after a write) overrides sticky routing for the relevant time window.

---

## Pattern 3: Consistent Prefix Reads Violation

**Formal definition:** A user reads a sequence of writes. The writes were causally ordered (write B depends on write A). The user sees write B but not write A — they see the effect before the cause.

**Why it happens:** In a partitioned (sharded) database, different partitions operate independently with no global write ordering. Write A goes to partition 1; write B (which depends on A) goes to partition 2. Partition 2 has low replication lag; partition 1 has high replication lag. The user's read contacts both partitions and sees partition 2's current state (write B present) but partition 1's stale state (write A not yet applied).

**Concrete example (from Kleppmann):** Mr. Poons asks: "How far into the future can you see?" Mrs. Cake replies: "About ten seconds usually." These are written to different partitions. An observer reads Mrs. Cake's reply (partition 2, low lag) before Mr. Poons' question (partition 1, high lag). The observer sees: "About ten seconds usually" followed by "How far into the future can you see?" — the answer precedes the question.

**Distinguishing characteristic:** The anomaly involves causally related writes that are written to different partitions. It is a problem of ordering, not of lag per se — even small lag differences can cause it if partitions diverge for even a moment.

**Mitigation techniques:**

*Technique 1: Causal co-location.*
Ensure that writes with causal dependencies go to the same partition. If the question and answer are always written to the same user's partition, they will always be applied in order by that partition's follower.
- Practical implementation: use a consistent routing key for causally related writes (e.g., a conversation ID, a user ID, an entity ID). All writes for the same conversation go to the same partition.
- Limitation: does not work when causally related writes must span different users' data or different entity types.

*Technique 2: Causal dependency tracking.*
Track causal dependencies explicitly using version vectors or logical clocks. A read does not return a causally later value without also verifying that all its causal prerequisites are present.
- More complex to implement and requires database or middleware support.
- Some databases provide this as a built-in feature (causal consistency in MongoDB, some Cassandra consistency levels).
- References: version vectors (Riak), Lamport timestamps, hybrid logical clocks (CockroachDB).

*Technique 3: Single-partition writes for related data.*
Redesign the data model so that causally related data lives in the same partition by default. For the conversation example: store both the question and answer as records within the same conversation partition, rather than in different user partitions.

---

## Replication Lag Monitoring

Regardless of which anomaly mitigation is implemented, monitoring replication lag is essential:

- **Leader-based replication:** The database exposes per-replica lag metrics. PostgreSQL: `pg_stat_replication.replay_lag`. MySQL: `Seconds_Behind_Master` in `SHOW SLAVE STATUS`. These can be fed into a monitoring system (Prometheus, Datadog, CloudWatch).

- **Leaderless replication:** There is no single leader to compare against. Staleness must be inferred from version numbers or timestamps returned by reads. Some databases expose node-level metrics (Cassandra: `nodetool netstats`, `nodetool tpstats`), but a unified lag figure is not available the same way it is in leader-based systems.

Alert thresholds:
- Warning: lag > 5 seconds (investigate whether the follower is falling behind)
- Critical: lag > 60 seconds (all read-after-write time-window mitigations assume lag is short; long lag makes them unsafe)
- Emergency: lag growing continuously (follower cannot keep up; consider removing it from the read pool)


path: references/quorum-edge-cases.md
content: # Quorum Edge Cases

The quorum condition w + r > n is designed to guarantee that the read node set and the write node set overlap, ensuring at least one node in every read has the latest value. In practice, six scenarios break this guarantee even when the condition is mathematically satisfied.

Applies to leaderless (Dynamo-style) replication: Cassandra, Riak, Voldemort, DynamoDB, and custom Dynamo-inspired implementations.

---

## The Quorum Baseline

For n replicas:
- Every write must be confirmed by w nodes to be considered successful.
- Every read must query r nodes and return the most recent value seen.
- As long as w + r > n, at least one node in the r-node read set must have seen the w-node write set.

Example: n=3, w=2, r=2. Any two nodes overlap in at least one member. If the write was confirmed by nodes {A, B}, any read contacting {A, C} or {B, C} or {A, B} will include at least one node that has the latest value.

This guarantee holds under the assumptions: strict quorums, no concurrent writes, no partial failures, no node recovery from stale state, and no clock skew in conflict resolution. The six edge cases each violate one of these assumptions.

---

## Edge Case 1: Sloppy Quorum / Hinted Handoff in Progress

**Condition:** A network interruption isolated the client from the n "home" nodes for a key. The database accepted the write on w "non-home" nodes (sloppy quorum) to maintain write availability. After the partition heals, hinted handoff is replaying those writes to the home nodes — but the process is not yet complete.

**Why the guarantee breaks:** The w writes are on non-home nodes. The r reads go to the home nodes. Even though w + r > n is satisfied in terms of counts, there is no overlap between the write set (non-home nodes) and the read set (home nodes) until hinted handoff completes.

**Detection signal:** Recent network partition or node outage. Hinted handoff queue non-empty. Reads returning pre-partition values.

**Mitigation:**
- Wait for hinted handoff to complete before serving reads. Monitor `HintsService` metrics (Cassandra: `nodetool tpstats`; Riak: hints queue length).
- Force read repair: issue a `CONSISTENCY QUORUM` read or run `nodetool repair`.
- Long-term: for keyspaces requiring freshness guarantees, disable sloppy quorums. For Cassandra: set `durable_writes = true`. For Riak: configure per-bucket quorum settings with `pw` (primary write quorum) rather than relying on sloppy handoff.

**Key distinction:** A sloppy quorum is a durability guarantee — "w nodes somewhere in the cluster hold this write." It is not a freshness guarantee. Treating sloppy quorum confirmations as equivalent to strict quorum confirmations is the error.

---

## Edge Case 2: Concurrent Writes with Last-Write-Wins Resolution

**Condition:** Two clients write to the same key at approximately the same time. Last-write-wins (LWW) is the conflict resolution strategy. Clocks between nodes have skew.

**Why the guarantee breaks:** The quorum condition says nothing about which write wins when two writes are concurrent. The only safe solution is to merge concurrent writes (using version vectors or application-level conflict resolution). LWW picks a winner based on timestamp — but timestamps come from node clocks, which have skew. The write with the higher timestamp wins, even if it was causally earlier. The causally later write — which the application intended to be the final value — is silently discarded. No error is reported.

**Detection signal:** Concurrent writes to the same key. LWW conflict resolution enabled (Cassandra default). Clock skew present between nodes. Missing writes with no error in the application.

**Mitigation:**
- Replace LWW with version vectors (Riak: dotted version vectors; custom systems: per-replica version numbers). Version vectors detect concurrent writes and preserve both versions as siblings, allowing application-level or CRDT-based merge.
- If LWW must be retained: use UUIDs as keys so each write has a unique key and concurrent writes to the same key cannot occur. This is the recommended Cassandra pattern for LWW safety.
- Monitor inter-node clock skew. Alert when skew exceeds the acceptable loss threshold (even 3ms skew can cause LWW data loss under high write concurrency).

**Key distinction:** LWW achieves eventual convergence at the cost of durability. All w writes were durably stored; LWW then deliberately discards some of them. This is a design choice, not a quorum failure — but it violates the application's intuition that "confirmed writes survive."

---

## Edge Case 3: Write Concurrent with Read

**Condition:** A write is in-flight when a read is issued. The write has been applied to some replicas but not others. The read contacts r replicas.

**Why the guarantee breaks:** If some of the r replicas have the new value and some have the old value, the read may return either value. If the read takes the old value and the write subsequently completes on all nodes, a later read might return the new value — but another later read (contacting a different replica subset) might still return the old value if that subset did not all receive the write in time. The system is in a transient inconsistent state during the in-flight write window.

**Detection signal:** Very high write concurrency. Intermittent stale reads on keys that are actively being written. Reads sometimes returning new values and sometimes old values non-deterministically.

**Mitigation:**
- For strong consistency requirements: use a consensus protocol (Raft, Paxos) rather than quorums. Consensus protocols serialize reads and writes and provide linearizability.
- For eventual consistency workloads: this edge case is inherent and expected. Ensure the application can tolerate reading either the old or new value during the write window.
- Read repair helps: if a read returns conflicting versions from different replicas, it writes the latest version back to the stale replicas, accelerating convergence.

---

## Edge Case 4: Partial Write Success, No Rollback

**Condition:** A write succeeded on some replicas but failed on others (e.g., a replica's disk is full, or it was temporarily unreachable). The overall write returned an error to the client (fewer than w acknowledgements). The replicas that did succeed are not rolled back.

**Why the guarantee breaks:** The write was reported as failed, so the client may retry it. But the replicas that succeeded already have the new value. On the next read, those replicas may return the partially-written value. The state is now: some replicas at new value, some at old value, and the client does not know which is "correct."

**Detection signal:** Write errors under disk pressure or replica unavailability. Subsequent reads intermittently returning values the client believes were never written.

**Mitigation:**
- Design operations to be idempotent. A safe retry of a partially-applied write must produce the same result as a full write.
- If atomicity across all replicas is required: use a database that provides transactional writes with rollback, not an eventually consistent leaderless store.
- Use conditional writes (compare-and-swap) where available to make partial writes detectable: the write includes a precondition (expected version), and only succeeds if the precondition is met. If the write partially applied, the replicas that got it are at a new version; replicas that did not are at the old version. The next attempt can detect the split by checking versions.

---

## Edge Case 5: Node Restored from Stale Replica

**Condition:** A node that held the latest value for a key fails. Its data is restored from a backup or from another replica that held an older value. After restoration, the number of replicas storing the latest value for that key falls below w.

**Why the guarantee breaks:** The quorum condition was satisfied when the write was made (w nodes confirmed). But the quorum is a snapshot in time — it is not a permanent guarantee. If nodes fail and are restored with stale data, the count of replicas holding the latest value can shrink below w, retroactively breaking the quorum guarantee for that write.

**Detection signal:** Node failure followed by restoration from backup. Reads intermittently returning values that should have been superseded. `nodetool repair` showing large amounts of data being resynchronized.

**Mitigation:**
- Before restoring a node from backup: run `nodetool repair` on the affected keyspace across all remaining healthy replicas first. This ensures the latest values are fully propagated before the restored node's stale data can contaminate reads.
- After restoring a node: do not route reads to it until a repair completes. Mark it as not eligible for read quorum until it has been resynchronized.
- Schedule regular anti-entropy runs (see below) to prevent silent value drift that makes restoration events worse.

---

## Edge Case 6: Timing Edge Cases at the Linearizability Boundary

**Condition:** No specific failure event has occurred. The system is operating normally with w + r > n fully satisfied. Yet under certain timing conditions, a read returns a stale value.

**Why the guarantee breaks:** Quorum reads provide eventual consistency by default — they are not linearizable. Linearizability requires that every read reflects the most recently completed write as of the moment the read is issued. Quorum reads do not guarantee this: the r replicas contacted may all have received the write, but the read request may have been issued before all r replies were collected. There are race conditions where unlucky timing causes stale reads even when the quorum condition is met.

**Detection signal:** Intermittent stale reads under high concurrency with no concurrent writes or node failures. Reads appearing stale by only milliseconds. The anomaly is not reproducible consistently.

**Mitigation:**
- Accept that quorum reads provide eventual consistency, not linearizability. Design the application to tolerate occasional stale reads.
- If linearizability is genuinely required: route reads through a single leader (leaderless systems can designate one replica as the read coordinator for linearizable reads), or use a consensus protocol (Raft, Paxos). Stronger guarantees require transactions or consensus — see `transaction-isolation-selector` and `consistency-model-selector`.

---

## Anti-Entropy: The Missing Piece

All six edge cases are worsened by the absence of an anti-entropy background process.

**Read repair** (the default in most Dynamo-style databases): when a read detects that some replicas returned a stale value, the reading client writes the latest value back to the stale replicas. This only runs when a value is actually read. Infrequently-read keys can diverge permanently.

**Anti-entropy process**: a background process that continuously compares replicas and propagates missing writes. Unlike read repair, it does not depend on reads being issued. Cassandra: `nodetool repair`. Riak: background anti-entropy (enabled by default). Voldemort: does not implement anti-entropy — relies solely on read repair.

**Without anti-entropy:** values that are rarely read may be missing from some replicas indefinitely. This reduces the effective durability below the theoretical guarantee (if only 1 of n replicas has a value for an infrequently-read key, that single replica's failure causes permanent data loss).

**Anti-entropy schedule recommendation:**
- Run `nodetool repair` on a schedule shorter than the gc_grace_seconds period (Cassandra default: 10 days).
- If gc_grace_seconds expires before repair runs, tombstones (deletion markers) are garbage-collected. If a replica that was offline during the original deletion comes back online after tombstone GC, it will "resurrect" the deleted data — deleted rows will reappear.
- Recommended: weekly repair at minimum. Daily repair for high-write workloads or large clusters.

---

## Quick Reference: Quorum Edge Cases

| Edge case | w + r > n satisfied? | Root cause | Mitigation |
|---|---|---|---|
| Sloppy quorum / hinted handoff | Yes (count-wise) | Write and read node sets don't overlap | Wait for hinted handoff; force read repair; consider strict quorums |
| LWW + concurrent writes | Yes | Conflict resolution discards valid writes | Use version vectors; use UUID keys; monitor clock skew |
| Write concurrent with read | Yes | Transient in-flight state | Design for eventual consistency; use consensus for linearizability |
| Partial write, no rollback | No (write failed) | Partial state left on some replicas | Idempotent operations; conditional writes; transactional DB if atomicity needed |
| Stale node restoration | Was yes, now no | Node count with latest value fell below w | Repair before routing reads to restored node; schedule regular anti-entropy |
| Timing at linearizability boundary | Yes | Quorums are not linearizable | Accept eventual consistency; use consensus for linearizable reads |


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)