Skill: quochungto/bookforge-replication-strategy-selector (version 1.0.0)

--- name: replication-strategy-selector description: | Choose a replication topology (single-leader, multi-leader, or leaderless) and configure it correctly — including sync vs. async mode, quorum parameters (w + r > n), and consistency guarantees. Use when designing replication for a new system, configuring quorum values for Cassandra/Riak/DynamoDB, deciding how to handle multi-leader write conflicts, or comparing PostgreSQL/MySQL streaming replication vs. CouchDB multi-leader vs. Cassandra leaderless for your architecture. Also use for: selecting a conflict resolution strategy (last-write-wins vs. version vectors); designing multi-datacenter replication; choosing between WAL shipping, logical replication, and statement-based replication log formats. For diagnosing an existing replication failure (failover gone wrong, lag spike, quorum misconfiguration, split brain), use replication-failure-analyzer instead. For consistency model selection (eventual vs. causal vs. linearizable), use consistency-model-selector instead. For partitioning strategy, use partitioning-strategy-advisor instead. version: 1.0.0 homepage: https://github.com/bookforge-ai/bookforge-skills/tree/main/books/designing-data-intensive-applications/skills/replication-strategy-selector metadata: {"openclaw":{"emoji":"📚","homepage":"https://github.com/bookforge-ai/bookforge-skills"}} status: draft depends-on: - consistency-model-selector - replication-failure-analyzer source-books: - id: designing-data-intensive-applications title: "Designing Data-Intensive Applications" authors: ["Martin Kleppmann"] chapters: [5] tags: - replication - single-leader - multi-leader - leaderless - quorum - consistency - failover - replication-lag - read-after-write - monotonic-reads - consistent-prefix-reads - cassandra - riak - postgresql - kafka - dynamo - conflict-resolution - last-write-wins - version-vectors - split-brain - sloppy-quorum - hinted-handoff - wal-shipping - logical-replication - multi-datacenter - availability - durability execution: tier: 2 mode: hybrid inputs: - type: codebase description: "Application codebase, docker-compose, database config files, or architecture description that reveals data access patterns and consistency requirements" - type: document description: "System requirements document or architecture description if no codebase is available" tools-required: [Read, Write, Bash] tools-optional: [Grep] mcps-required: [] environment: "Run inside a project directory with codebase or configuration files. Falls back to document/description input." discovery: goal: "Produce a concrete replication strategy recommendation: topology + sync mode + quorum config + consistency guarantees + conflict resolution" tasks: - "Classify the system's availability vs. consistency priority" - "Select the replication topology (single-leader, multi-leader, leaderless)" - "Select synchronous, asynchronous, or semi-synchronous replication mode" - "Configure quorum parameters (n, w, r) if leaderless" - "Select consistency guarantees required by the application" - "Select a conflict resolution strategy if multi-leader or leaderless" - "Document failover risks and replication lag anomalies" audience: roles: ["backend-engineer", "software-architect", "data-engineer", "site-reliability-engineer", "tech-lead"] experience: "intermediate-to-advanced — assumes experience with distributed systems and databases" triggers: - "User is choosing between replication strategies for a new database deployment" - "User is configuring Cassandra, Riak, or DynamoDB quorum parameters" - "User wants to add multi-datacenter replication to an existing system" - "User is debugging stale reads, inconsistencies, or read-after-write failures" - "User is assessing failover risk in a single-leader setup" - "User is evaluating whether to use multi-leader replication for offline or multi-datacenter operation" - "User is choosing a conflict resolution strategy for concurrent writes" not_for: - "Diagnosing replication failures in production — use replication-failure-analyzer" - "Selecting consistency and isolation levels for transactions — use consistency-model-selector" - "Choosing a partitioning/sharding scheme — use partitioning-strategy-advisor" --- # Replication Strategy Selector ## When to Use You are designing or evaluating a data system that replicates data across multiple nodes and need to select a replication topology, configure how data is propagated, and define what consistency guarantees the application requires. This skill applies when the replication architecture is open (new system), contested (existing system with lag or consistency problems), or needs documented justification (team alignment, architecture decision record). It produces a concrete recommendation covering topology, sync mode, quorum configuration, consistency guarantees, and conflict resolution strategy. **Prerequisite check:** - If you need to understand what consistency model (linearizability, causal, eventual) the application requires before selecting a topology, run `consistency-model-selector` first. - If you are diagnosing an active replication failure or anomaly in production, run `replication-failure-analyzer` instead. - This skill does not cover partitioning/sharding — if the dataset is too large for a single machine, run `partitioning-strategy-advisor` after this skill. --- ## Context & Input Gathering ### Required Context (must have — ask if missing) - **Availability vs. consistency priority:** Why: This is the primary axis. Single-leader replication is the most consistent topology (all writes go through one node, so ordering is defined). Leaderless replication is the most available (writes are accepted even when nodes are down). Multi-leader sits between them but introduces conflict risk. You cannot choose a topology without knowing which trade-off matters more. - Check prompt for: "high availability", "must not go down", "consistency is critical", "ACID", "always online", "offline-capable", "multi-region" - Check environment for: docker-compose (replication factor configs, consistency-level settings), database config files (sync_commit, consistency_level), requirements.md - If still missing, ask: "If a network partition separates some of your database nodes, which is more important: (a) all nodes continue accepting writes even if they may diverge temporarily, or (b) writes are rejected until you can confirm consistency across nodes?" - **Geographic distribution:** Why: Multi-datacenter operation strongly favors multi-leader or leaderless replication. In a single-leader setup with multiple datacenters, all writes must route to the datacenter containing the leader, adding cross-datacenter latency to every write. If users are geographically distributed and write latency matters, multi-leader or leaderless is necessary. - Check prompt for: "multi-region", "multiple datacenters", "users in Europe and US", "global", "CDN", "geographic proximity" - Check environment for: deployment configs (Kubernetes regions, terraform provider regions), docker-compose (multi-DC labels), architecture.md - If still missing, ask: "Are your database nodes in a single datacenter, or are they spread across multiple geographic regions?" - **Write conflict probability:** Why: Multi-leader and leaderless replication can produce write conflicts — two nodes concurrently accepting writes to the same key. If writes to the same record from different clients or regions are likely, conflict resolution must be planned. If conflicts are unlikely (e.g., each user only writes to their own data), multi-leader is more tractable. - Check prompt for: "concurrent writes", "multiple clients updating the same record", "collaborative editing", "last-write-wins is fine", "offline sync", "shopping cart" - Check environment for: application code (multi-writer patterns), schema (shared mutable records, counters, aggregates) - If still missing, ask: "Is it possible for two different clients or datacenters to write to the same record at approximately the same time? For example, two users editing the same document, or two datacenters updating the same account balance?" - **Read/write ratio and latency requirements:** Why: Affects whether asynchronous replication and read scaling are appropriate. Single-leader with many asynchronous followers enables horizontal read scaling. Leaderless quorums let you tune r and w to bias toward read or write performance. If write latency is critical, synchronous replication to all nodes is disqualifying. - Check prompt for: throughput numbers, "read-heavy", "write-heavy", latency SLA (ms), "real-time", "user-facing" - Check environment for: requirements.md, architecture.md (SLA definitions), application code (read/write ratios) - If still missing, ask: "What is the approximate read-to-write ratio and your latency requirement? For example: 90% reads with p99 < 50ms, or mostly writes with best-effort latency?" ### Observable Context (gather from environment) - **Existing database and replication config:** Look for `docker-compose.yml`, `postgresql.conf` (synchronous_standby_names, wal_level), `cassandra.yaml` (replication_factor, consistency_level), `my.cnf` (binlog_format, rpl_semi_sync_master_enabled). Reveals current topology and whether this is a greenfield or migration decision. - **Number of replicas/nodes:** Look for replica count configurations. Affects quorum math (n must be odd for majority quorums to work cleanly; n=3 tolerates 1 failure, n=5 tolerates 2). - **Durability configuration:** Look for fsync settings, WAL configuration, backup policies. Signals how much data loss risk is currently accepted. - **Application consistency patterns:** Grep codebase for session tokens, user-ID-based routing, "read from primary" comments, timestamp-based staleness checks — signals that the team is already working around replication lag. ### Default Assumptions - Replication factor unknown → assume n=3 (standard minimum for fault tolerance; tolerates 1 node failure) - Geographic distribution unknown → assume single-datacenter - Write conflict probability unknown → assume low (most applications have per-user or per-entity write ownership) - Latency SLA unknown → assume best-effort; synchronous replication to all replicas is not required --- ## Process ### Step 1: Classify the System Profile **ACTION:** Determine the system profile across three axes: (a) availability vs. consistency priority, (b) geographic distribution, (c) write conflict probability. **WHY:** The replication topology selection is determined almost entirely by these three axes. Geographic distribution eliminates single-leader as a low-latency option. High conflict probability makes leaderless difficult to use safely. Strict consistency requirements eliminate fully asynchronous leaderless replication. Getting the profile right prevents the most common mistake: choosing Cassandra (leaderless, eventual) for a system that actually needs read-after-write guarantees, or using single-leader in a multi-datacenter deployment where the cross-datacenter write path adds unacceptable latency. Produce a one-line profile: ``` Profile: [consistency-priority | availability-priority | balanced] [single-datacenter | multi-datacenter] [low-conflict | high-conflict | conflict-acceptable] ``` **IF** the user describes offline-first mobile clients → note that multi-leader (or leaderless) is structurally required; each device is effectively a leader. **IF** the user describes collaborative editing (multiple users editing the same document) → flag that conflict resolution complexity is high; consider whether single-leader with leader-per-document is feasible. --- ### Step 2: Select the Replication Topology **ACTION:** Choose between single-leader, multi-leader, and leaderless replication based on the Step 1 profile. Apply disqualifying filters first. **WHY:** Each topology has a different fundamental design: who can accept writes, in what order writes are applied, and what happens when nodes disagree. These are architectural commitments — changing from single-leader to leaderless after data is in production requires schema redesign, operational migration, and application changes. The topology decision must be made with full awareness of what each option cannot do. **Disqualifying filters:** | Condition | Disqualifies | |-----------|-------------| | Users distributed across 2+ geographic regions AND write latency < 100ms required | Single-leader (all writes must cross datacenter to reach the leader) | | Strict ACID transactions required across multiple records | Leaderless (no defined write ordering; transactions require coordination layers) | | Writes must never be lost (zero data loss on failure) | Fully asynchronous single-leader (async follower may lag behind when leader fails) | | Application cannot tolerate write conflicts (financial transactions, inventory reservation) | Multi-leader and leaderless (concurrent writes to same record may produce conflicts) | | Clients must work offline and sync later | Single-leader (requires connectivity to the leader for writes) | **Three-way topology comparison:** | Dimension | Single-leader | Multi-leader | Leaderless | |-----------|--------------|-------------|-----------| | **Write ordering** | Total order (leader serializes all writes) | Partial order (writes within a leader are ordered; cross-leader conflicts possible) | No global order (concurrent writes to same key are possible) | | **Write availability during leader failure** | Unavailable until failover completes (~30s typical) | Available (other datacenters continue accepting writes) | Available (quorum of remaining nodes accepts writes) | | **Write latency in multi-datacenter** | High (all writes route to leader's datacenter) | Low (writes go to local datacenter's leader) | Low (writes go to any available replica; quorum locally) | | **Conflict risk** | None (only one writer) | High (two datacenters can write to same record concurrently) | Medium (concurrent writes possible; quorum overlap reduces risk) | | **Operational complexity** | Low (well-understood; all major DBs support it) | High (conflict resolution logic required; retrofitted in many DBs) | Medium (quorum config requires care; read repair needed) | | **Consistency guarantees available** | Read-after-write, monotonic reads, consistent prefix (achievable with techniques) | Eventual at minimum; stronger guarantees require cross-leader coordination | Tunable via quorum config; does not naturally provide the session guarantees above | | **Representative systems** | PostgreSQL, MySQL, MongoDB (default), Kafka | Tungsten (MySQL), BDR (PostgreSQL), CouchDB, multi-DC Cassandra with multi-leader config | Cassandra, Riak, Voldemort, DynamoDB | **Selection rules:** ``` IF single-datacenter AND consistency-priority → Single-leader IF single-datacenter AND availability-priority AND low-conflict → Leaderless IF multi-datacenter AND write-latency-critical → Multi-leader OR Leaderless IF offline-clients OR multi-datacenter-availability → Multi-leader IF high-conflict AND no-conflict-resolution-capacity → Single-leader (simplest) IF consistency-priority AND ACID-required → Single-leader (only safe choice) ``` **WHY the selection rules are structured this way:** Single-leader is the simplest and most consistent option — there is only one writer, so write ordering is total and conflicts are impossible. The cost is write availability during failover and cross-datacenter write latency. Multi-leader trades conflict risk for write availability across datacenters. Leaderless trades global ordering for the ability to tune availability and consistency independently via quorum parameters. Most systems should default to single-leader unless there is a concrete reason (multi-datacenter write latency, offline operation) that requires one of the more complex options. **Record the selected topology and the primary disqualifiers applied.** --- ### Step 3: Select Synchronous vs. Asynchronous Replication Mode **ACTION:** For the selected topology, determine whether replication to followers/replicas should be synchronous, asynchronous, or semi-synchronous. **WHY:** Synchronous and asynchronous replication represent a direct trade-off between durability and write availability. Synchronous replication guarantees that a write is on at least two nodes before confirming to the client — if the leader fails immediately after acknowledging the write, the data is not lost. Asynchronous replication confirms the write to the client as soon as the leader applies it locally — if the leader fails before replicating, the write is lost even though the client was told it succeeded. This is not a subtle theoretical concern; it is the primary cause of data loss in leader-based database failures. **For single-leader and multi-leader:** | Mode | Behavior | When to use | |------|----------|-------------| | **Fully synchronous** | All followers must confirm before leader reports success. Write is durable on all nodes. | Never recommended for all followers — any single follower failure blocks all writes. | | **Semi-synchronous (recommended default)** | One follower is synchronous; all others are asynchronous. If the synchronous follower fails, an async follower is promoted to synchronous. | When you need at least 2 durable copies of every write, while keeping write availability when some followers lag. This is PostgreSQL's `synchronous_commit` behavior. | | **Fully asynchronous** | Leader confirms immediately; followers catch up in background. Replication lag may be milliseconds to minutes depending on load. | When write throughput is the primary constraint and some data loss on leader failure is acceptable (e.g., log aggregation, analytics events, non-critical telemetry). | **Configuration implications:** - PostgreSQL: `synchronous_commit = on` (synchronous to named standby) or `remote_write` (semi-sync). Controlled by `synchronous_standby_names`. - MySQL: `rpl_semi_sync_master_enabled = 1` enables semi-synchronous. At least one slave must acknowledge before leader reports success. - Kafka: `acks=all` is synchronous to all in-sync replicas (ISR). `acks=1` is semi-sync (leader only). `acks=0` is fire-and-forget. **WHY semi-synchronous is the recommended default:** Fully synchronous (all followers) means any follower slowdown or failure stalls all writes — one network blip kills write availability for the entire cluster. Fully asynchronous means leader failure always risks losing writes that were confirmed to the client, which violates most durability expectations. Semi-synchronous gives you at least 2 durable copies (leader + 1 synchronous follower) while keeping write availability when some nodes are slow or down. **For leaderless:** Synchronous vs. asynchronous is expressed as quorum configuration (Step 4). Skip to Step 4. --- ### Step 4: Configure Quorum Parameters (Leaderless Only) **ACTION:** If leaderless replication is selected, determine the values of n (replication factor), w (write quorum), and r (read quorum) using the formula w + r > n. **WHY:** The quorum formula w + r > n guarantees that at least one node in every read set has seen the most recent write. This is because the w write nodes and r read nodes must overlap by at least one node (by the pigeonhole principle). Without this overlap, a read could hit only stale nodes and return an outdated value without knowing it. Quorum configuration is the primary lever for trading off availability vs. consistency in a leaderless system. **Common configurations:** | n | w | r | w+r>n? | Fault tolerance | Bias | |---|---|---|--------|----------------|------| | 3 | 2 | 2 | 4>3 ✓ | 1 node failure | Balanced (standard) | | 3 | 3 | 1 | 4>3 ✓ | 0 for writes | Write-consistent, read-fast | | 3 | 1 | 3 | 4>3 ✓ | 0 for reads | Write-fast, read-consistent | | 5 | 3 | 3 | 6>5 ✓ | 2 node failures | Balanced | | 5 | 5 | 1 | 6>5 ✓ | 0 for writes | Write durable, read-fast | | 3 | 1 | 1 | 2>3 ✗ | N/A | High availability, stale reads possible | **Quorum selection rules:** ``` High availability priority → lower w and r (set w+r ≤ n for sloppy quorum behavior) Read-heavy workload → lower r (r=1 if stale reads acceptable); higher w to keep writes durable Write-heavy workload → lower w; higher r to compensate for write distribution Strong consistency needed → w = n or w+r > n strictly (strict quorum) Multi-datacenter → configure per-datacenter quorum in Cassandra LOCAL_QUORUM ``` **WHY the common default is n=3, w=2, r=2:** This is the minimum configuration that provides both fault tolerance (1 node can fail) and quorum overlap (the 2 write nodes and 2 read nodes must share at least 1 node). Lowering to w=1 or r=1 increases availability but risks returning stale data — the read may miss the node that has the latest write. **Sloppy quorums and hinted handoff:** During a network partition that cuts a client off from enough nodes to reach quorum, the database faces a choice: reject all writes (strict quorum) or accept writes on any available nodes even if they are not the designated "home" nodes for a key (sloppy quorum). Sloppy quorums improve write availability during partitions but break the quorum consistency guarantee — a read of r nodes is no longer guaranteed to overlap with the w write nodes. After the partition heals, hinted handoff sends the temporarily-routed writes to their home nodes. Sloppy quorums are appropriate for use cases that can tolerate occasional stale reads (Riak enables by default; Cassandra and Voldemort disable by default). **WHY sloppy quorums break the consistency guarantee:** With a strict quorum, w + r > n ensures the read and write node sets overlap. With a sloppy quorum, writes may land on "neighbor" nodes not in the normal n-node set. When you read from r nodes in the normal set, those nodes may not include the neighbor nodes that received the write — so w + r > n no longer guarantees overlap. Sloppy quorum is a write-availability mechanism, not a consistency mechanism. --- ### Step 5: Select Consistency Guarantees **ACTION:** Determine which consistency guarantees the application requires and verify that the selected topology and sync mode can provide them. **WHY:** Replication lag is not theoretical — in asynchronous systems, followers can lag by milliseconds to minutes. Applications that read from followers without accounting for this lag expose users to anomalies: seeing a write they just made disappear (violating read-after-write), watching data appear to move backward in time (violating monotonic reads), or seeing an answer before the question that caused it (violating consistent prefix reads). Each anomaly requires a specific mitigation technique. Choosing the right mitigation depends on knowing which anomaly is unacceptable for your application's users. **The three replication lag anomalies and their mitigations:** **1. Read-after-write consistency (read-your-writes)** - Anomaly: User submits data, immediately reads it back from a stale follower, sees the old value. Appears to the user as if their write was lost. - When it matters: Any application where users write and immediately view their own data (profile updates, comment submission, form submission confirmation). - Mitigations (apply one): - Read the user's own data from the leader; read other users' data from followers. (Requires knowing which data the user can have modified.) - For one minute after any write, route all that user's reads to the leader (or prevent reads from followers lagging > 1 minute). - Track the timestamp of each user's most recent write as a logical timestamp (log sequence number). On read, only serve from replicas that have applied up to that timestamp. - For multi-device access (user writes on phone, reads on laptop): centralize the timestamp metadata and route all devices to the same datacenter. **2. Monotonic reads** - Anomaly: User makes two successive reads; the second read returns data older than the first (because it hit a more-lagged replica). Data appears to move backward in time. - When it matters: Any application where users make multiple reads of the same data in a session (social feeds, dashboards, chat messages). - Mitigation: Route each user's reads to the same replica (e.g., hash the user ID to a replica). If that replica fails, reroute — but accept that the user may briefly see data go backward. **3. Consistent prefix reads** - Anomaly: Observer reads writes in a different order than they were written, violating causality. An answer is visible before the question that prompted it. - When it matters: Any application where the order of writes is semantically meaningful (conversation threads, audit logs, event sourcing, financial transaction history). - Mitigation: Ensure causally related writes go to the same partition (so they are applied in the same order at all replicas). Alternatively, track causal dependencies with version vectors. **Consistency guarantee availability by topology:** | Guarantee | Single-leader | Multi-leader | Leaderless | |-----------|-------------|-------------|-----------| | Read-after-write | Achievable (read from leader; timestamp-based routing) | Complex (requires routing to the same leader that processed the write) | Not guaranteed by quorum alone; application must implement timestamp tracking | | Monotonic reads | Achievable (sticky-session replica routing) | Complex (requires routing to same datacenter that processed the write) | Not guaranteed by quorum; implement user-to-replica affinity | | Consistent prefix reads | Natural for single-partition data; requires causal tracking for partitioned data | Very hard (cross-datacenter write ordering is not defined) | Not guaranteed; requires version vectors and causal tracking | **WHY leaderless quorums do not automatically provide session guarantees:** Even with w + r > n, the overlap node is not guaranteed to be the node you actually read from — quorum reads are sent to r nodes, and the freshest value is returned, but if two concurrent writes happened, the "freshest" determination uses version numbers or timestamps that can be unreliable (especially with clock skew). The session guarantees (read-after-write, monotonic reads, consistent prefix) require the application or database to track causal dependencies explicitly — a capability that is present in single-leader systems by construction (because the leader serializes all writes) but must be built explicitly in leaderless systems. --- ### Step 6: Select a Conflict Resolution Strategy (Multi-leader or Leaderless) **ACTION:** If the topology is multi-leader or leaderless, select a conflict resolution strategy for concurrent writes to the same key. **WHY:** In multi-leader and leaderless systems, two nodes can concurrently accept writes to the same key. When replication converges, the system must decide what the final value of that key should be. There is no "correct" answer — the right resolution depends on the application semantics. Making this choice explicit now prevents silent data loss from the default strategy (usually last-write-wins, which discards writes) surprising the team later. **Conflict detection:** Two writes are concurrent if neither operation knew about the other when it was sent. Version vectors track which version each write was based on. If client A's write is based on version v1 and client B's write is also based on v1 (not knowing about A's write), both writes are concurrent — they must be resolved. If B's write is based on v2 (which incorporated A's write), B's write happens-after A's, and the later write wins without conflict. **Resolution strategies:** | Strategy | How it works | Data loss? | When to use | |----------|-------------|-----------|------------| | **Last write wins (LWW)** | Each write is timestamped; the write with the highest timestamp wins. Other writes are silently discarded. | Yes — concurrent writes are lost | Only when data loss is acceptable (caches, analytics events, idempotent operations). Cassandra's default. | | **Conflict avoidance** | Route all writes for a given record through the same leader. Conflicts cannot occur if one leader "owns" each record. | No | When write locality is predictable (user X always writes to datacenter A). Breaks down if user location changes. | | **Merge / union** | Merge all concurrent versions into a combined value. For sets/lists: take the union. | No (but requires tombstones for deletes) | Collaborative data structures (shopping carts, sets, counters). Riak siblings; CRDTs. | | **Application-level resolution on read** | Store all conflicting versions; return them all to the application on the next read. Application code resolves and writes back the merged value. | No | When only the application has the semantic context to resolve (CouchDB model). Requires application code changes. | | **Application-level resolution on write** | Conflict handler executes immediately when conflict is detected in the replication log. Must execute quickly (no user prompts). | Depends on handler | When automated resolution is possible (Bucardo for PostgreSQL). | **WHY LWW is dangerous and widely misused:** LWW requires a reliable total ordering of writes. In distributed systems, clocks cannot be trusted to provide this — two nodes' system clocks can differ by milliseconds or more (see clock skew), meaning a write with an earlier local timestamp may have actually occurred later in causal time. LWW silently discards writes that are reported as successful to the client. The only safe use of LWW is when each key is written exactly once and then treated as immutable — for example, using a UUID as the key so concurrent updates cannot happen. **WHY conflict avoidance is often the best strategy:** Most multi-leader deployments can be designed so that a given record has a "home" leader — the user's nearest datacenter, for example. If routing ensures that user X always writes to leader A, concurrent writes to the same record from two leaders cannot happen. Conflict avoidance is effectively making multi-leader behave like single-leader on a per-record basis. The limitation is that if the home leader changes (datacenter failure, user relocation), conflicts become possible again. --- ### Step 7: Produce the Recommendation **ACTION:** Write a structured recommendation document covering all decisions made in Steps 1–6. **WHY:** A concrete documented recommendation enables team alignment, prevents relitigating the decision, and creates an artifact for the architecture decision record. The "What Can Go Wrong" section is essential — it prevents future surprise when production behavior diverges from the expected replication model. **Output format:** Write the following to a file named `replication-strategy-recommendation.md` in the project root (or in `docs/architecture/` if that directory exists): ```markdown ## Replication Strategy Recommendation ### System Profile [One-line profile from Step 1] ### Recommendation **Topology:** [Single-leader | Multi-leader | Leaderless] **Sync Mode:** [Synchronous | Semi-synchronous | Asynchronous | N/A (leaderless)] **Quorum (if leaderless):** n=[x], w=[x], r=[x] (w+r=[x] > n=[x] ✓) **Sloppy Quorum:** [Enabled | Disabled | Not applicable] ### Primary Database Products | Role | Product | Configuration | |------|---------|--------------| | [Leader / All replicas] | [PostgreSQL / Cassandra / etc.] | [key config settings] | | [Follower / Replica] | [as above] | [sync mode setting] | ### Consistency Guarantees Provided | Guarantee | Provided? | Implementation | |-----------|----------|----------------| | Read-after-write | [Yes / No / Application-layer] | [Technique from Step 5] | | Monotonic reads | [Yes / No / Application-layer] | [Technique from Step 5] | | Consistent prefix reads | [Yes / No / Application-layer] | [Technique from Step 5] | ### Conflict Resolution (if multi-leader or leaderless) **Strategy:** [LWW | Conflict avoidance | Merge | Application-level] **Rationale:** [Why this strategy fits the application semantics] ### Why [Topology] [2-3 sentences connecting the system profile to the topology choice] ### What We're Giving Up [1-2 sentences on the primary trade-off and how to mitigate it] ### Disqualifiers Applied [If any topology was disqualified, state the condition and why] ### What Can Go Wrong [See "What Can Go Wrong" section below — include the relevant items for this topology] ``` --- ## What Can Go Wrong **Failover data loss (single-leader, async replication).** When an asynchronous leader fails, the new leader is the replica with the most up-to-date data — but "most up-to-date" does not mean "fully up-to-date." Any writes the old leader processed but had not yet replicated are lost when the new leader takes over, even though the client received an acknowledgment. This is the fundamental cost of asynchronous replication. If the old leader later comes back online, it may still have those writes — the system must ensure it discards them and follows the new leader, or data inconsistency results permanently. **Split brain (single-leader and multi-leader).** In certain network partition scenarios, two nodes can each believe they are the leader. If both accept writes, and there is no mechanism to resolve conflicts, data diverges permanently. The standard defense is fencing: the system shuts down one of the two nodes when it detects two leaders. If the fencing mechanism itself has a bug (or is not implemented), the system can end up with both nodes shut down — a worse outcome than split brain. This is described as "shoot the other node in the head" (STONITH) in the operations literature. **Replication lag spikes during leader failure or high load.** Asynchronous followers can fall behind by seconds or minutes when the system is under load or recovering from a failure. If the application reads from followers assuming the lag is negligible, users see stale data. The lag can increase to several minutes during follower recovery, maintenance, or network problems. Monitor the replication lag metric on every follower (PostgreSQL: `pg_replication_slots`, `pg_stat_replication`; MySQL: `Seconds_Behind_Master`). Alert when lag exceeds your acceptable staleness threshold. **Timeout misconfiguration in failover.** Automatic leader election uses a timeout to detect leader failure. If the timeout is too short, a temporarily slow network causes an unnecessary failover — the old leader is still alive but is now competing with the new leader (split brain risk). If the timeout is too long, the cluster is unavailable for writes for longer than necessary after a real failure. There is no universally correct timeout. Start with 30 seconds and calibrate based on observed network behavior. High-load systems should use longer timeouts because response times are naturally higher. **Old leader rejoining after failover.** When the old leader comes back online after a failover, it may still believe it is the leader — particularly if the old leader was partitioned (slow, not dead). It will try to accept writes. The system must force the old leader to recognize the new leader and demote itself to follower. Without this mechanism, writes go to two nodes simultaneously. This is a known problem in many open-source databases and is a primary reason some operations teams prefer manual failovers even when software supports automatic failover. **LWW silently discards acknowledged writes.** Last-write-wins conflict resolution is the default in Cassandra and other Dynamo-style databases. If two clients write to the same key concurrently, LWW picks one value and discards the other — even though both clients received a success acknowledgment. The client whose write was discarded has no way to know this happened. For any data where losing a write is unacceptable (financial records, reservations, inventory counts), LWW is a correctness bug, not just a performance trade-off. Use conflict avoidance, application-level resolution, or CRDTs instead. **Quorum does not guarantee strong consistency.** Even with w + r > n, stale reads can occur in edge cases: (a) if two writes happen concurrently, the ordering between them is not defined and LWW may pick the wrong winner; (b) if a sloppy quorum is enabled, the write nodes and read nodes may not overlap; (c) if a write fails on some replicas but succeeds on others (partial write), and the write is reported as failed to the client but applied to w nodes, subsequent reads may return that "failed" write's value. Quorum is a probabilistic durability guarantee, not a linearizability guarantee. For linearizability, use a consensus protocol (Raft, Paxos — as in etcd, ZooKeeper, CockroachDB). **Multi-leader conflict resolution logic is hard to get right.** Amazon's early DynamoDB implementation had a conflict resolution handler for shopping carts that merged adds but not removes — items removed from the cart would reappear after a conflict merge. The bug went undetected because conflicts are infrequent and the test suite did not cover the concurrent write case. Conflict resolution code must be explicitly tested with concurrent write scenarios; it cannot be reasoned about by inspecting the code alone. Build a test harness that simulates concurrent writes to the same key before deploying multi-leader. --- ## Key Principles **Default to single-leader unless there is a concrete requirement for multi-leader or leaderless.** Single-leader replication is the simplest topology — it provides total write ordering, makes consistency guarantees achievable, and is well-supported by every major database. Multi-leader and leaderless replication solve real problems (multi-datacenter write latency, offline operation, high write availability), but they introduce conflict risk and operational complexity that is routinely underestimated. The right default is single-leader with semi-synchronous replication. **Replication mode is a durability commitment, not a performance setting.** Choosing asynchronous replication to improve write latency means accepting that some acknowledged writes will be lost if the leader fails before replication completes. This is a business and correctness decision, not purely a performance optimization. Make this trade-off explicitly with the team, document it, and build monitoring that alerts when replication lag reaches a level where data loss would be significant. **The replication lag anomalies (read-after-write, monotonic reads, consistent prefix) are application bugs, not database bugs.** When an application reads from an asynchronous follower and returns stale data to the user, the database is behaving correctly — it told you it uses eventual consistency. The application is the layer responsible for implementing the session guarantees that mask this behavior. Either implement the mitigation techniques in Step 5 or use a database that provides these guarantees at the database layer (PostgreSQL with synchronous_commit, or a database with sessions backed by the leader). **Quorum configuration is not set-and-forget — it must be re-evaluated as load and node count change.** A system that starts with n=3, w=2, r=2 and later grows to n=5 may have stale quorum configuration. Adding replicas without updating quorum parameters means the quorum overlap guarantee may be weaker or stronger than intended. Revisit quorum configuration whenever the replication factor changes or when the availability vs. consistency balance of the system changes. **Conflict avoidance is almost always better than conflict resolution.** Every conflict resolution strategy (LWW, merge, application-level) is either lossy, complex, or application-specific. The cleanest approach is to avoid conflicts entirely by routing all writes for a given record through a single leader. If the system allows it, design the data model and routing rules so that conflicts cannot happen — then multi-leader or leaderless replication behaves like single-leader on a per-record basis, with the added benefit of datacenter-local write acceptance for other records. --- ## Examples **Scenario: Global e-commerce platform with regional datacenters** Trigger: "We have users in the US, EU, and Asia. Right now all writes go to our US leader. EU and Asia users complain about high write latency. We're considering multi-leader replication." Process: - Step 1: Multi-datacenter, write-latency-critical, low-conflict (each user writes to their own orders/cart) - Step 2: Single-leader disqualified (cross-datacenter write latency). Multi-leader selected. Low conflict probability → conflict avoidance feasible. - Step 3: Semi-synchronous within each datacenter; asynchronous between datacenters (inter-DC link is less reliable). - Step 5: Read-after-write: route reads to the user's home datacenter's leader (same datacenter that accepted the write). Monotonic reads: user session pinned to home datacenter. - Step 6: Conflict avoidance — each user's records are owned by their home datacenter. If home datacenter changes, implement a brief lock during migration. Output: Multi-leader with one leader per datacenter, semi-sync within datacenter, async cross-datacenter, conflict avoidance via user-to-datacenter affinity. Read-after-write provided by datacenter-local routing. **Scenario: Real-time leaderboard for a mobile game** Trigger: "We need a leaderboard that survives node failures without going down. Players' scores update constantly. Occasional stale reads are acceptable — seeing a score a few seconds old is fine." Process: - Step 1: Availability-priority, single-datacenter, high-write, stale-reads-acceptable - Step 2: Single-leader would work but creates write bottleneck for high-frequency score updates. Leaderless selected — high write throughput with tunable availability. - Step 4: n=3, w=2, r=1. w+r=3, not > n=3. This is intentionally a sloppy configuration for high read availability at the cost of occasional staleness. For scores where exact order matters, increase r=2. - Step 5: Consistent prefix not required (leaderboard order is eventually consistent by design). Read-after-write not required (seeing a slightly old score is acceptable). - Step 6: LWW acceptable — score updates are monotonically increasing; overwriting with an older score is unlikely because score writes are always increments, not overwrites. Use version vectors to detect if older value would overwrite newer. Output: Leaderless (Cassandra), n=3, w=2, r=1. Sloppy quorum enabled for maximum availability. LWW with version tracking. Monitoring alert if replication lag exceeds 5 seconds. **Scenario: Financial transaction ledger** Trigger: "We're building an account balance system. Every debit and credit must be consistent. We cannot lose writes. We're using PostgreSQL and considering adding read replicas for reporting." Process: - Step 1: Consistency-priority, single-datacenter, write-conflict-unacceptable (double-debit catastrophic) - Step 2: Multi-leader disqualified (write conflicts are unacceptable; two DCs could both debit the same account). Leaderless disqualified (no strong ordering; quorum does not prevent double-debit). Single-leader selected. - Step 3: Semi-synchronous. At least one standby must acknowledge before the leader confirms. Fully async is disqualified — data loss on leader failure is not acceptable for financial records. - Step 5: Read-after-write: all balance reads for the user's own account come from the leader. Reporting queries (dashboards, audit logs) can use async followers with acceptable staleness. - Step 6: N/A — single-leader, no conflicts possible. Output: PostgreSQL single-leader, `synchronous_commit = remote_write` (semi-synchronous), one sync standby, multiple async replicas for reporting. All balance reads routed to leader. Replication lag monitoring alert at 10 seconds. --- ## References For topology-specific deep dives, failure mode details, and quorum calculators, see the references directory: - For quorum parameter calculations and availability modeling, see [quorum-calculator.md](references/quorum-calculator.md) - For replication log format comparison (WAL shipping vs. logical/row-based vs. statement-based), see [replication-log-formats.md](references/replication-log-formats.md) - For conflict resolution strategy comparison and implementation patterns, see [conflict-resolution-strategies.md](references/conflict-resolution-strategies.md) - For failover procedure and the 3-step leader election process, see [failover-playbook.md](references/failover-playbook.md) - For replication lag monitoring metrics by database (PostgreSQL, MySQL, Cassandra), see [replication-lag-monitoring.md](references/replication-lag-monitoring.md) - Cross-reference: `replication-failure-analyzer` — for diagnosing active replication anomalies in production - Cross-reference: `consistency-model-selector` — for selecting the right consistency model before this skill - Cross-reference: `partitioning-strategy-advisor` — for sharding strategy after replication topology is decided ## License This skill is licensed under [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). Source: [BookForge](https://github.com/bookforge-ai/bookforge-skills) — Designing Data-Intensive Applications by Martin Kleppmann. ## Related BookForge Skills Install related skills from ClawhHub: - `clawhub install bookforge-consistency-model-selector` - `clawhub install bookforge-replication-failure-analyzer` Or install the full book set from GitHub: [bookforge-skills](https://github.com/bookforge-ai/bookforge-skills)

path: references/conflict-resolution-strategies.md
content: # Conflict Resolution Strategies

Use this reference when selecting a conflict resolution strategy for multi-leader or leaderless replication. A write conflict occurs when two nodes independently accept writes to the same key without either knowing about the other's write. All replicas must eventually converge to the same value — conflict resolution defines how that convergence happens.

## What Makes a Conflict

Two writes are **concurrent** if neither operation was aware of the other when it was submitted. Version vectors (or vector clocks) track which version each write was based on:
- If client B's write is based on version v2 (which included client A's write), B's write happens-after A's — no conflict.
- If both A and B wrote based on version v1 (neither knew about the other), both writes are concurrent — conflict.

Two writes are **not concurrent** (and therefore not a conflict) if one causally depends on the other.

## Strategy 1: Last Write Wins (LWW)

**How it works:** Attach a timestamp to every write. When two concurrent writes are detected, the write with the higher timestamp "wins" — the other is silently discarded.

**Data loss:** Yes. The losing write is dropped, even though the client received a success acknowledgment.

**When to use:**
- Data loss is acceptable (caches, idempotent operations, analytics events that can be replayed)
- Each key is written at most once and then immutable (use UUID as key to prevent reuse)
- You need the simplest possible convergence with no application code

**When NOT to use:**
- Financial records, inventory counts, reservation systems — any write that cannot be lost
- Counters or aggregates (two increments become one)
- Any write where the client expects their acknowledgment means the write is permanent

**The clock skew problem:** LWW relies on timestamps to pick the "most recent" write. In distributed systems, node clocks cannot be perfectly synchronized. A write with a "later" local timestamp may have actually been submitted before a write with an "earlier" timestamp on another node (due to clock drift, NTP adjustments, or clock skew). LWW will silently pick the wrong winner in these cases.

**Safe use of LWW:** The only safe pattern is to use a globally unique key (UUID) for every write, so two writes to the same key cannot happen. Cassandra documentation recommends this pattern: use a UUID as the primary key, making each write operation a unique key, so concurrent updates cannot occur by construction.

**Implementation:**
- Cassandra: LWW is the default conflict resolution. Timestamp is the write time on the client.
- Riak: LWW is optional; version vectors are the default.
- DynamoDB: Conditional writes (compare-and-set) prevent LWW data loss for critical updates.

---

## Strategy 2: Conflict Avoidance

**How it works:** Design the system so that all writes for a given record always go through the same leader. If one leader "owns" each record, concurrent writes to the same record from two leaders cannot happen — there is no conflict to resolve.

**Data loss:** No.

**When to use:**
- Each record has a natural "home" (user's home datacenter, record owner, geographic affinity)
- Write locality is predictable (a given user always writes from the same region)
- The system can tolerate routing all writes for a record to a single datacenter

**Limitation:** Conflict avoidance breaks down if the home leader changes. If a user moves to a different region, or if the home datacenter fails and traffic is rerouted, writes may go to a different leader than expected — creating conflicts. Implementing a clean leader-change handoff (lock the record, drain writes, switch leader, unlock) is non-trivial.

**Implementation:** Application-level routing. Configure your load balancer or service layer to send writes for a given key (e.g., user ID, account ID) to the datacenter that "owns" that key. Can be implemented with consistent hashing or explicit key-to-datacenter mappings.

---

## Strategy 3: Merge / Union

**How it works:** When concurrent writes produce multiple versions of the same key, keep all versions and merge them into a single value. For sets and lists, merge = union.

**Data loss:** No — but merge may produce a result that is not exactly what either client intended.

**When to use:**
- The data is a collection (set, list, map) where union is semantically correct
- Collaborative data structures (shopping carts, task lists, tagged content)
- CRDTs (Conflict-free Replicated Data Types) — data structures designed for automatic merge

**Limitation:** Merge only works for additive operations. Deletions are problematic: if client A deletes an item and client B adds an item, and the two writes are concurrent, the union may bring the deleted item back. The solution is tombstones — a special marker indicating "this item was deleted." The merge algorithm must treat tombstones correctly.

**CRDTs (Conflict-free Replicated Data Types):**
CRDTs are data structures designed so that concurrent modifications can always be merged automatically without conflict. Examples:
- **G-counter (grow-only counter):** Each node has its own counter; the global count is the sum of all per-node counts. Concurrent increments never conflict.
- **OR-Set (observed-remove set):** Adds and removes are tracked with unique identifiers; concurrent add+remove is resolved by keeping the add (with a tombstone for the remove).
- **LWW-register:** A single-value register using LWW — safe only when writes to the same key are not truly concurrent (because LWW is used).

Implementations: Riak 2.0 (G-counters, sets, maps, registers, flags), Redis (basic CRDTs), Akka Distributed Data (various CRDTs).

---

## Strategy 4: Application-Level Resolution on Read

**How it works:** When concurrent versions of a key exist, all versions are stored ("siblings" in Riak terminology). On the next read, the database returns all conflicting versions to the application. The application code resolves the conflict and writes back the merged value.

**Data loss:** No — but requires application code to handle multi-version responses correctly.

**When to use:**
- Only the application has the semantic context to merge two concurrent values correctly
- Conflict frequency is low (application complexity of handling multi-version responses is justified)
- The application already handles eventual consistency gracefully

**Limitation:** Every read code path must handle the case of multiple returned values. This is easy to get wrong — a naive implementation that picks the first value, or crashes on multi-value responses, will silently corrupt data.

**Implementation:** CouchDB uses this model. A read returns all conflicting revisions; the application picks or merges them and writes a new revision that resolves the conflict.

---

## Strategy 5: Application-Level Resolution on Write

**How it works:** A conflict handler is registered with the database. When the replication log detects a conflict during synchronization, the handler is called immediately. The handler must run quickly (it executes in the background replication process) and cannot prompt users.

**Data loss:** Depends on handler implementation.

**When to use:**
- Automated conflict resolution is possible without user input
- The conflict resolution logic is deterministic and fast
- The team can implement and test the handler thoroughly

**Limitation:** The handler must not block the replication process. It has no access to the user and cannot make external API calls. Bugs in the handler are difficult to detect because conflicts are infrequent — the handler may go untested in production for months.

**Implementation:** Bucardo (PostgreSQL), Oracle GoldenGate (custom PL/SQL handlers).

---

## Strategy Comparison Matrix

| Strategy | Data loss | Application code changes required | Conflict frequency sensitivity | Best use case |
|----------|----------|----------------------------------|-------------------------------|--------------|
| Last write wins | Yes | No | Low (conflicts infrequent) | Caches, idempotent ops |
| Conflict avoidance | No | Yes (routing) | N/A (no conflicts) | User-owned data with stable home |
| Merge / CRDTs | No | Sometimes (tombstones) | High (designed for concurrent writes) | Collections, collaborative data |
| App-level on read | No | Yes (multi-version handling) | Low-medium | Semantic conflicts only app can resolve |
| App-level on write | Depends | Yes (handler code) | Low-medium | Automated resolution possible |

## Recommendation Decision Tree

```
Can all writes for a given record always route through one leader?
  YES → Conflict avoidance (simplest; no data loss; no application code)
  NO ↓

Is the data a collection (set, list, counter)?
  YES → CRDTs / merge (Riak, or implement a CRDT data structure)
  NO ↓

Can the conflict be resolved automatically without user input?
  YES AND resolution is fast → App-level on write (conflict handler)
  YES AND resolution needs read context → App-level on read
  NO ↓

Is data loss acceptable (cache, analytics, idempotent)?
  YES → Last write wins (with UUID keys if possible)
  NO → Do not use multi-leader or leaderless; use single-leader instead
```


path: references/failover-playbook.md
content: # Failover Playbook

Use this reference when planning or executing a leader failover in a single-leader or multi-leader replication setup. Failover is the process of promoting a follower to become the new leader after the current leader fails. It is the most operationally risky event in a leader-based replication system.

## The Three-Step Failover Process

### Step 1: Determine that the leader has failed

There is no foolproof way to detect leader failure. The standard approach is a timeout:
- If the leader does not respond within a configured period (commonly 30 seconds), assume it is dead.
- If it has not responded within that period, initiate failover.

**The timeout calibration problem:**
- Too short (< 10 seconds): A temporary load spike or network hiccup causes false positives. The system performs an unnecessary failover while the leader is still alive — creating split-brain risk.
- Too long (> 60 seconds): The cluster is unavailable for writes for longer than necessary after a real failure.
- Starting point: 30 seconds. Adjust upward if your system operates under sustained high load (response times naturally higher). Adjust downward only after observing false-positive rate in production.

### Step 2: Choose a new leader

**By election:** The remaining replicas vote; the replica with the most up-to-date data is elected. This requires consensus (a notoriously hard distributed systems problem — see Raft, Paxos, Zab in Chapter 9 of DDIA).

**By appointment:** A previously elected controller node (e.g., ZooKeeper) appoints the new leader. The controller is itself a replicated consensus system.

**Heuristic for the best candidate:** The replica that is most up-to-date with the old leader (fewest unreplicated writes) should become the new leader, to minimize data loss.

### Step 3: Reconfigure the system to use the new leader

- All clients must send write requests to the new leader.
- All followers must consume the replication log from the new leader, not the old one.
- If the old leader comes back online, it must recognize the new leader and demote itself to follower. Without this, it may still accept writes as leader — creating split brain.

## Automatic vs. Manual Failover

| | Automatic failover | Manual failover |
|---|---|---|
| Recovery time | ~30s (timeout + election) | Minutes to hours (human response time) |
| Risk | Split brain, premature election, stale leader confusion | Human error during high-stress incident |
| Recommended for | Systems with high availability SLA, well-tested failover path | Systems where the failover code is untested or where human judgment is critical |

**Recommendation:** Use automatic failover with a conservative timeout (30-60 seconds). Test the failover path regularly (chaos engineering — kill the leader in staging, observe behavior). Do not rely on automatic failover as a substitute for testing.

## Failover Pitfalls

### Pitfall 1: Data loss from async replication

When the old leader fails, the new leader is chosen as the most up-to-date replica. But "most up-to-date" does not mean "identical to the old leader." Any writes the old leader applied but had not yet replicated are lost.

**How much data can be lost?** Determined by the replication lag at the moment of failure. With semi-synchronous replication (one synchronous follower), the loss is bounded: the synchronous follower was always up-to-date, so it becomes the new leader with no data loss. With fully asynchronous replication, the lag can be seconds to minutes.

**Mitigation:** Use semi-synchronous replication so at least one follower is always current.

### Pitfall 2: The old leader's unreplicated writes cause conflict when it rejoins

When the old leader comes back online, it has writes that the new leader does not have. The most common solution is to discard those writes — the old leader becomes a follower and its unreplicated writes are thrown away.

**Why discarding is dangerous:** The client received a success acknowledgment for those writes. The database silently breaks the durability guarantee. If those writes were coordinated with other systems (e.g., a Redis counter was incremented based on the DB write), the external system is now inconsistent with the database.

**Example:** At GitHub in 2012, a MySQL follower that was behind the leader was promoted to leader during a failover. The lagging new leader had missed some writes, including auto-incrementing primary keys. Those primary key values had already been used by the old leader's rows (which were stored in a Redis cache). The reuse of primary keys caused data from different users to be returned to the wrong users.

### Pitfall 3: Split brain — two nodes both believe they are the leader

In certain network partition scenarios, the old leader is partitioned from the cluster but not actually dead. The cluster elects a new leader. Now two nodes believe they are the leader and both accept writes.

**Defense — fencing (STONITH):** The system shuts down one node when it detects two leaders. "Shoot The Other Node In The Head" — the node that loses the election is forcibly terminated. 

**The fencing catch:** If the fencing mechanism is improperly designed, it can shut down both nodes. A split-brain situation where both nodes are shut down is worse than a split-brain where both nodes are alive (at least the system is running in the latter case, even if inconsistently). Design fencing carefully and test it explicitly.

### Pitfall 4: Premature failover during load spikes

High system load causes response times to increase. If the leader is under sustained high load, it may not respond to heartbeats within the timeout — even though it is alive and processing writes. This triggers an unnecessary failover.

**Symptoms:** Failovers that happen during high-traffic periods (not hardware failures). Post-failover analysis shows the old leader was still running and had no hardware failure.

**Mitigation:** Increase the timeout during known high-load windows. Monitor the leader's response time trend — if p99 response times are approaching the failover timeout, alert before a failover is triggered. Separate heartbeat traffic from write traffic so write backpressure does not cause heartbeat failures.

## Failover Configuration by Database

### PostgreSQL

- **pg_auto_failover:** Automated failover with a monitor node. Monitor detects primary failure, promotes standby.
- **Patroni:** ZooKeeper or etcd-backed leader election. The current leader must hold a lock; if it cannot renew the lock, it demotes itself.
- **Repmgr:** Simpler failover automation; relies on monitoring script, not consensus.
- Key config: `synchronous_standby_names` (for semi-sync); `wal_level = replica` (for streaming replication).

### MySQL

- **Orchestrator (GitHub):** Detects leader failure, elects most up-to-date replica, reconfigures replication topology automatically.
- **MHA (MySQL High Availability):** Similar to Orchestrator; widely used.
- Key config: `rpl_semi_sync_master_enabled` (semi-synchronous); `GTID_MODE = ON` (for easier failover topology reconfiguration).

### Kafka

- Kafka uses an ISR (In-Sync Replicas) list managed by ZooKeeper (older) or KRaft (newer, built-in consensus). Leader election happens automatically among ISR members.
- Key config: `acks=all` (write must be acknowledged by all ISR replicas before success); `min.insync.replicas` (minimum ISR size for writes to be accepted).

## Post-Failover Checklist

After every failover (automatic or manual), verify:

- [ ] Old leader has fully demoted itself to follower (not accepting writes)
- [ ] All clients are writing to the new leader
- [ ] New leader's replication lag to its followers is decreasing (not growing)
- [ ] Any data loss is documented (how many writes were discarded? Were they coordinated with external systems?)
- [ ] External systems that relied on the old leader's data (caches, search indexes) are invalidated or refreshed
- [ ] The root cause of the failover is identified (hardware failure, network partition, timeout misconfiguration, load spike)
- [ ] The failover timeout setting is re-evaluated based on the root cause


path: references/quorum-calculator.md
content: # Quorum Calculator

Use this reference when configuring leaderless replication (Cassandra, Riak, Voldemort, DynamoDB) to determine the correct values of n, w, and r for your availability and consistency requirements.

## The Core Formula

```
w + r > n
```

- **n** = replication factor (total number of nodes that store each key)
- **w** = write quorum (number of nodes that must acknowledge a write for it to succeed)
- **r** = read quorum (number of nodes queried on each read; freshest value returned)

When w + r > n, the write set and read set must overlap by at least one node. That overlapping node has the most recent write. This guarantees the read returns the latest value — provided no concurrent writes are happening and no sloppy quorum is in use.

## Worked Examples

### Standard balanced configuration (n=3, w=2, r=2)

```
w + r = 4 > n = 3  ✓
Fault tolerance: 1 node can be unavailable (writes need 2/3; reads need 2/3)
Staleness risk: Low — any read will hit at least 1 node with the latest write
Space cost: 3x data stored
```

Best for: General-purpose workloads where both availability and consistency matter. This is the default for most Cassandra deployments.

### Write-heavy, reads can be stale (n=3, w=1, r=2)

```
w + r = 3 = n  ✗ (quorum condition not strictly satisfied)
Fault tolerance: 2 nodes can be unavailable for writes; 1 for reads
Staleness risk: High — write is on 1 node; read may hit 2 nodes that don't have it
Space cost: 3x data stored
```

Best for: High-throughput event ingestion where eventual consistency is explicitly acceptable. Do not use if read-after-write is required.

### Read-fast, write-durable (n=5, w=3, r=1)

```
w + r = 4 < n = 5  ✗ (quorum condition not satisfied)
```

Wait — this means reads can return stale values. To fix this:

```
n=5, w=3, r=3: w + r = 6 > 5 ✓
Fault tolerance: 2 nodes can fail
Read cost: 3 parallel reads required per operation
Write cost: 3 acknowledgments required
```

### High availability bias — sloppy quorum (n=3, w=1, r=1)

```
w + r = 2 < n = 3  ✗ (intentionally not a strict quorum)
```

Used when write availability is the top priority. Writes are accepted on any available node. Reads may return stale values. Enable hinted handoff to propagate writes to home nodes when partition heals.

**Do not use if:** data loss is unacceptable, read-after-write is required, or application logic requires seeing the latest value.

## Availability vs. Consistency Trade-off Table

| n | w | r | w+r>n | Node failures tolerated | Stale read risk |
|---|---|---|-------|------------------------|----------------|
| 3 | 3 | 1 | 4>3 ✓ | 0 writes, 2 reads | Very low |
| 3 | 2 | 2 | 4>3 ✓ | 1 | Low |
| 3 | 1 | 3 | 4>3 ✓ | 2 writes, 0 reads | Low |
| 3 | 2 | 1 | 3=3 ✗ | 1 writes, 2 reads | High |
| 3 | 1 | 2 | 3=3 ✗ | 2 writes, 1 reads | High |
| 5 | 3 | 3 | 6>5 ✓ | 2 | Low |
| 5 | 4 | 2 | 6>5 ✓ | 1 writes, 3 reads | Low |
| 5 | 2 | 4 | 6>5 ✓ | 3 writes, 1 reads | Low |

## Cassandra Consistency Level Mapping

In Cassandra, you configure consistency levels per-query rather than global n/w/r. Cassandra translates consistency levels to the quorum formula automatically based on the replication factor.

| Cassandra CL | Meaning | Equivalent |
|-------------|---------|-----------|
| ONE | w=1 or r=1 | Any single replica |
| QUORUM | w=(n/2)+1 or r=(n/2)+1 | Majority of replicas |
| LOCAL_QUORUM | Majority within local datacenter only | Multi-DC: quorum per-DC |
| ALL | w=n or r=n | All replicas |
| ANY | w=1, accepts hinted handoff node | Maximum write availability |
| SERIAL | Lightweight transactions (Paxos) | Linearizable per-key |

**For multi-datacenter Cassandra:** Use `LOCAL_QUORUM` for both reads and writes. This satisfies quorum within each datacenter independently and avoids cross-datacenter latency on every operation.

## Sloppy Quorum Configuration by Database

| Database | Sloppy quorum setting | Default |
|----------|----------------------|---------|
| Riak | `allow_mult = true`, sloppy quorum enabled | Enabled by default |
| Cassandra | Implicit via `hinted_handoff_enabled = true` | Enabled, but strict quorum by default |
| Voldemort | Configurable `prefer-reads`, `prefer-writes` | Disabled by default |
| DynamoDB | Not directly configurable; managed by AWS | AWS-managed |

## Replication Factor Selection Rules

- **n=1:** No fault tolerance. Only acceptable for development or non-critical caches.
- **n=3:** Standard minimum. Tolerates 1 node failure with majority quorum (w=2, r=2).
- **n=5:** Higher fault tolerance (2 node failures). Use for mission-critical data or when rolling restarts (which take nodes offline one at a time) should not reduce quorum.
- **Odd n is preferred:** Simplifies majority quorum math. n=4 does not give more fault tolerance than n=3 for majority quorums (both tolerate 1 failure before majority is lost).


path: references/replication-lag-monitoring.md
content: # Replication Lag Monitoring

Use this reference when setting up monitoring for replication lag, or when diagnosing replication lag anomalies. Replication lag is the delay between when a write is applied on the leader and when it is visible on a follower/replica. Unmonitored lag is the primary cause of read-after-write, monotonic reads, and consistent prefix reads violations in production.

## Why Monitoring Lag Is Critical

Replication lag is not constant. In normal operation it may be milliseconds — imperceptible. But during:
- High write load
- Follower recovery after a crash or restart
- Network congestion between leader and follower
- Compaction or heavy disk I/O on the follower

...lag can increase to seconds or minutes. An application that reads from followers without accounting for lag will silently return stale data to users. This is not a theoretical concern — it is a common production incident.

Monitoring must alert before lag reaches the threshold where user-visible anomalies occur.

## PostgreSQL Lag Monitoring

### View current replication state

```sql
-- On the primary: see each standby's lag
SELECT
  client_addr,
  state,
  sent_lsn,
  write_lsn,
  flush_lsn,
  replay_lsn,
  write_lag,
  flush_lag,
  replay_lag,
  sync_state
FROM pg_stat_replication;
```

- `replay_lag`: Time between the primary writing a WAL record and the standby applying it. This is the most meaningful lag metric for data visibility.
- `sync_state`: `sync` (synchronous standby), `async` (asynchronous), `potential` (can become synchronous).

### Check lag from the standby

```sql
-- On the standby: how far behind is this replica?
SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;
```

Returns a time interval. Alert if this exceeds your staleness threshold (e.g., 30 seconds for user-facing reads, 5 minutes for analytics).

### Check bytes of lag

```sql
-- On the primary: bytes of WAL not yet replicated
SELECT
  client_addr,
  pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS lag_bytes
FROM pg_stat_replication;
```

Large lag in bytes (> 1GB) indicates the replica is significantly behind and may take a long time to catch up even after load normalizes.

### Replication slot lag (prevents WAL recycling)

```sql
-- Check replication slot lag — can prevent WAL cleanup if lag is large
SELECT
  slot_name,
  active,
  restart_lsn,
  pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS lag_bytes
FROM pg_replication_slots;
```

Alert if a replication slot has lag > 10GB — WAL is accumulating and can fill the disk.

### Recommended alerts

| Metric | Warning threshold | Critical threshold |
|--------|------------------|-------------------|
| `replay_lag` (time) | 30 seconds | 5 minutes |
| Lag bytes | 500 MB | 5 GB |
| Replication slot lag bytes | 5 GB | 20 GB |
| `pg_stat_replication` row missing | Immediately | N/A (standby disconnected) |

---

## MySQL Lag Monitoring

### Check lag from the replica

```sql
SHOW REPLICA STATUS\G
```

Key fields:
- `Seconds_Behind_Master`: Approximate lag in seconds. Warning: this metric can be misleading — it shows 0 when the replica SQL thread has caught up but the IO thread is lagging.
- `Replica_IO_Running`: Must be `Yes`. If `No`, the replica cannot receive new writes from the primary.
- `Replica_SQL_Running`: Must be `Yes`. If `No`, the replica is not applying received writes.
- `Last_SQL_Error`: Error message if replica SQL thread stopped. Common cause: a write that succeeded on the primary fails on the replica (schema divergence, constraint violation, duplicate key).

### GTID-based lag (more accurate)

```sql
-- On the primary
SELECT @@global.gtid_executed;

-- On the replica
SELECT @@global.gtid_executed;
-- Compare the two GTID sets to find unapplied transactions
```

GTID (Global Transaction ID) tracking gives a precise count of transactions not yet applied, rather than a time estimate.

### Recommended alerts

| Metric | Warning threshold | Critical threshold |
|--------|------------------|-------------------|
| `Seconds_Behind_Master` | 30 seconds | 5 minutes |
| `Replica_IO_Running` = No | Immediately | N/A |
| `Replica_SQL_Running` = No | Immediately | N/A |
| `Last_SQL_Error` non-empty | Immediately | N/A |

---

## Cassandra Lag Monitoring

Cassandra uses leaderless replication, so there is no single "leader lag" metric. Instead, monitor:

### Repair status

Cassandra's anti-entropy repair process ensures all replicas have all data. If repair is not running regularly, stale data can persist indefinitely on replicas that missed writes.

```bash
nodetool repair --full <keyspace>       # Force a full repair
nodetool compactionstats                # Check compaction backlog
nodetool tpstats                        # Thread pool stats including repair threads
```

Key metrics:
- `AntiEntropyStage` pending tasks: if this grows, repair is not keeping up
- `RepairStage` pending tasks: repair jobs queued but not yet executed

### Hinted handoff

When a node is down and writes are directed to it via hinted handoff, the hints are stored on other nodes. Check hint delivery:

```bash
nodetool tpstats | grep HintedHandoff
```

If hints are accumulating and not being delivered, the target node may not have come back online, or the hinted handoff timeout has expired (hints are discarded after `max_hint_window_in_ms`, default 3 hours).

### Read repair monitoring

Track `ReadRepair` metrics in Cassandra's JMX interface. Low read repair rates in a read-heavy system mean replicas are diverging without correction.

### Recommended alerts

| Metric | Warning threshold | Critical threshold |
|--------|------------------|-------------------|
| AntiEntropyStage pending tasks | > 100 | > 500 |
| HintedHandoffManager active hints | > 10,000 | Growing unbounded |
| Node status (nodetool status) | Any node DN (down) | Multiple nodes DN |
| Dropped messages (nodetool tpstats) | Any | Growing (indicates overload) |

---

## General Monitoring Principles

**Measure actual staleness, not just lag time.** Time-based lag metrics (`Seconds_Behind_Master`) measure when the replica last applied a write, not how stale a specific read is. A replica that hasn't received any writes in the last 10 seconds may show a 10-second lag but is actually fully up-to-date — no writes have happened. For user-facing staleness, track the timestamp of the most recently applied write per key, not just replication lag.

**Set a staleness budget based on your application's guarantees.** If your application provides read-after-write consistency by routing reads to the leader for 1 minute after a write, your lag alert threshold should be much lower than 1 minute — alerting at 30 seconds gives you time to route reads back to the leader before the user notices.

**Monitor the trend, not just the current value.** Lag that is at 10 seconds and growing is more dangerous than lag at 30 seconds and stable. Configure lag rate-of-change alerts in addition to threshold alerts.

**Lag monitoring for leaderless systems requires different approaches.** Without a single replication log, there is no single lag metric. Instead, monitor: (1) anti-entropy repair completion rate, (2) hinted handoff delivery, (3) read repair frequency, and (4) node availability (a node that is down will accumulate lag invisibly until it rejoins).


path: references/replication-log-formats.md
content: # Replication Log Formats

Leader-based replication (single-leader and multi-leader) requires the leader to send a stream of data changes to followers. How those changes are encoded determines upgrade flexibility, cross-database compatibility, and the risk of divergence between replicas.

## The Three Formats

### 1. Statement-Based Replication

**How it works:** The leader logs every write SQL statement (INSERT, UPDATE, DELETE) and forwards the statement text to followers. Each follower re-executes the statement.

**Problems:**
- Nondeterministic functions (`NOW()`, `RAND()`, `UUID()`) produce different values on each replica. The leader must replace these with fixed values before logging — a subtle requirement that is easy to miss.
- Statements that depend on existing data (e.g., `UPDATE table WHERE condition`) must execute in the same order on all replicas. Concurrent transactions can cause divergence if statements are re-ordered.
- Statements with side effects (triggers, stored procedures, user-defined functions) may produce different effects on each replica unless side effects are deterministic.

**Used by:** MySQL before version 5.1 (now row-based by default). VoltDB uses statement-based replication but requires all transactions to be deterministic.

**When appropriate:** Only for systems where all statements are guaranteed to be deterministic. Not recommended for general-purpose use.

---

### 2. Write-Ahead Log (WAL) Shipping

**How it works:** The storage engine appends every write to a WAL (write-ahead log) — an append-only sequence of bytes describing every change to every disk block. The leader ships this raw log to followers. Followers apply the log bytes directly to their own storage, building an identical disk structure to the leader.

**Advantages:**
- Byte-for-byte identical disk structure on all replicas. Divergence is structurally impossible.
- Used by PostgreSQL and Oracle (among others).

**Problems:**
- The WAL contains physical details: which bytes changed in which disk blocks. This is tightly coupled to the specific storage engine version and format.
- If the leader and follower run different database software versions, the WAL format may differ — making zero-downtime upgrades impossible. Upgrading requires taking followers offline or using logical replication instead.
- Cannot be used to replicate between different database products (PostgreSQL WAL cannot be consumed by MySQL).

**Used by:** PostgreSQL (`wal_level = replica` or `logical`), Oracle Data Guard.

**When appropriate:** Single-database deployments where version homogeneity is maintained and operational simplicity is valued over upgrade flexibility.

---

### 3. Logical (Row-Based) Log Replication

**How it works:** A separate log format — decoupled from the storage engine's physical format — describes writes at the granularity of database rows:
- **Inserted row:** new values of all columns
- **Deleted row:** enough information to identify the row (primary key, or all column values if no primary key)
- **Updated row:** row identifier + new values of changed columns

**Advantages:**
- Decoupled from the storage engine internals → leader and follower can run different database versions during rolling upgrades.
- Easier to parse by external systems (data warehouses, caches, search indexes via change data capture).
- Can replicate between different database products if both understand the logical format.

**Problems:**
- More data volume than statement-based replication (full row contents vs. one SQL statement).
- Conflict detection in multi-leader setups must operate at the row level (which column changed? was the whole row replaced?).

**Used by:** MySQL binlog (`binlog_format = ROW`), PostgreSQL logical replication (`wal_level = logical`), Debezium (CDC).

**When appropriate:** The recommended default for most production deployments. Required if zero-downtime upgrades are needed. Required if change data capture (CDC) feeds a data warehouse or search index.

---

### 4. Trigger-Based Replication

**How it works:** Application-level replication using database triggers or stored procedures. A trigger fires on every write, logs the change to a separate table, and an external process reads that table and replicates the change.

**Advantages:**
- Maximum flexibility: can replicate a subset of data, apply transformation logic, or replicate between different database systems.
- Does not require changes to the database software itself.

**Problems:**
- Higher overhead than built-in replication (trigger execution + log table writes on every write).
- More bugs and edge cases than built-in replication methods.
- Requires careful design to avoid trigger recursion and partial-write anomalies.

**Used by:** Databus (Oracle to Oracle/MySQL), Bucardo (PostgreSQL), Oracle GoldenGate.

**When appropriate:** When built-in replication does not support the required topology (e.g., cross-database replication, selective replication, custom transformation during replication).

---

## Comparison Table

| Format | Decoupled from storage engine? | Zero-downtime upgrades? | External consumption (CDC)? | Divergence risk |
|--------|-------------------------------|------------------------|----------------------------|----------------|
| Statement-based | Yes | Yes | Difficult (parse SQL) | High (nondeterminism) |
| WAL shipping | No | No | Difficult (binary format) | Very low |
| Logical/row-based | Yes | Yes | Easy (structured rows) | Low |
| Trigger-based | Yes | Yes | Easy (custom table) | Medium (trigger bugs) |

## Recommendation

**Default choice:** Logical (row-based) replication. It provides upgrade flexibility, external CDC compatibility, and low divergence risk without the nondeterminism problems of statement-based replication.

**Use WAL shipping only if:** You want the simplest possible setup and guarantee that the leader and all followers will always run the same database version. Operational teams that can enforce version homogeneity may prefer WAL shipping for its byte-for-byte replica guarantee.

**Avoid statement-based replication** unless you can guarantee all statements are deterministic (VoltDB model) or you are constrained to a legacy MySQL version.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (MEDIUM)