Skill: quochungto/bookforge-consistency-model-selector (version 1.0.0)

--- name: consistency-model-selector description: | Choose the correct consistency model (linearizability, causal consistency, or eventual consistency) for each operation in a distributed system, and select the matching implementation mechanism. Use when designing a new distributed data system, deciding whether ZooKeeper or etcd is needed for coordination, evaluating whether two-phase commit is appropriate for cross-node transactions, debugging correctness violations (stale reads, split-brain, uniqueness constraint failures), or distinguishing linearizability from serializability. Also use when applying the CAP theorem correctly (beyond the "pick 2 of 3" oversimplification), selecting total order broadcast as a consensus primitive, evaluating 2PC failure modes and lock-holding cost, or assessing whether causal consistency is sufficient in place of linearizability. Produces a per-operation consistency recommendation with replication mechanism, ordering guarantee, and — when consensus is needed — protocol selection (Raft, Zab, Paxos) with documented failure modes. Does not cover replication topology or failure recovery strategy (see replication-strategy-selector, distributed-failure-analyzer). version: 1.0.0 homepage: https://github.com/bookforge-ai/bookforge-skills/tree/main/books/designing-data-intensive-applications/skills/consistency-model-selector metadata: {"openclaw":{"emoji":"📚","homepage":"https://github.com/bookforge-ai/bookforge-skills"}} status: draft depends-on: - replication-strategy-selector - distributed-failure-analyzer source-books: - id: designing-data-intensive-applications title: "Designing Data-Intensive Applications" authors: ["Martin Kleppmann"] chapters: [9] tags: - consistency - linearizability - causal-consistency - eventual-consistency - consensus - total-order-broadcast - two-phase-commit - cap-theorem - zookeeper - etcd - raft - paxos - lamport-timestamps - distributed-transactions - atomic-commit - serializability - ordering-guarantees - leader-election - uniqueness-constraints - distributed-locks execution: tier: 1 mode: hybrid inputs: - type: codebase description: "Application codebase, docker-compose, database config, or architecture description revealing operations and their correctness requirements" - type: document description: "System requirements document, architecture diagram, or written description of operations if no codebase is available" tools-required: [Read, Write] tools-optional: [Grep, Bash] mcps-required: [] environment: "Run inside a project directory with codebase or architecture artifacts. Falls back to interactive document/description input." discovery: goal: "Produce a per-operation consistency recommendation: model selection + replication mechanism + ordering guarantee + consensus protocol if needed" tasks: - "Classify each operation by its consistency requirement" - "Select the minimum sufficient consistency model per operation (eventual / causal / linearizable)" - "Identify which operations require consensus and match them to the 6-problem checklist" - "Select a replication mechanism compatible with the chosen model" - "Evaluate 2PC applicability and document failure modes if selected" - "Recommend ZooKeeper/etcd for consensus-dependent coordination tasks" - "Document the CAP trade-off in force for each linearizable operation" audience: roles: ["backend-engineer", "software-architect", "data-engineer", "site-reliability-engineer", "tech-lead"] experience: "intermediate-to-advanced — assumes familiarity with distributed systems, replication, and transactions" triggers: - "Choosing consistency guarantees for a new distributed data system" - "Deciding whether to use ZooKeeper, etcd, or a custom leader election mechanism" - "Evaluating whether two-phase commit is appropriate for cross-service transactions" - "Debugging stale reads, causality violations, or uniqueness constraint failures" - "Assessing whether leaderless replication is strong enough for an operation" - "Distinguishing when serializability is sufficient vs. when linearizability is required" - "Designing distributed lock or leader election infrastructure" not_for: - "Selecting isolation levels for single-node transactions — use transaction-isolation-selector" - "Diagnosing replication lag and failover failures — use replication-failure-analyzer" - "Choosing a replication topology — use replication-strategy-selector" --- ## When to Use Use this skill when you need to decide **how strongly consistent** a distributed operation must be, and what mechanism enforces that guarantee. Invoke it for: - Any operation where two nodes could disagree on the current value (leader election, uniqueness check, account balance enforcement, distributed lock) - Cross-service or cross-database transactions where atomic commit is required - Systems where stale reads cause incorrect application behavior (not just stale UI) - Designing coordination infrastructure (service registry, job scheduler, partition assignment) - Evaluating whether a Dynamo-style leaderless database is strong enough for a given workload Do not invoke it for single-node transaction isolation tuning — that is the domain of `transaction-isolation-selector`. --- ## Context and Input Gathering Before selecting a model, collect the following per operation or component: 1. **Operation type**: read, write, read-modify-write, uniqueness check, lock acquisition, leader election, atomic commit across nodes 2. **Correctness requirement**: Can stale data cause incorrect behavior? Can two nodes diverge temporarily? Is the worst-case outcome data loss, a user-visible error, or a constraint violation? 3. **Availability requirement**: Must this operation succeed during a network partition, or is it acceptable to return an error? 4. **Cross-channel timing dependencies**: Does any other system or user observe an out-of-band signal about the write (e.g., a webhook, message queue, user notification) before reading back? 5. **Replication topology in use**: single-leader, multi-leader, or leaderless (see `replication-strategy-selector`) 6. **Throughput and latency constraints**: Are response times acceptable with a synchronous round-trip to a leader or quorum? If a codebase is available, search for: - Database client configuration (isolation level, quorum parameters `w`, `r`, `n`) - Distributed lock acquisition code - Uniqueness constraint enforcement logic - Leader election or service discovery configuration - Cross-service transaction boundaries --- ## Process ### Step 1 — Identify the Required Guarantee for Each Operation **WHY**: Different operations have fundamentally different correctness requirements. Over-provisioning consistency wastes latency and availability; under-provisioning introduces bugs that only appear under concurrency or network faults — the hardest class of bugs to detect in testing. Apply this decision table per operation: | Scenario | Minimum Required Model | |---|---| | Display a user's own recent writes to that same user | Read-your-writes (causal) | | Show a feed where replies never appear before questions | Causal consistency | | Enforce a hard uniqueness constraint (username, seat, stock limit) | Linearizability | | Acquire a distributed lock that prevents split-brain | Linearizability | | Elect a single leader across nodes | Linearizability (via consensus) | | Atomically commit a transaction across multiple nodes | Atomic commit (consensus-equivalent) | | Show analytics dashboard (staleness of seconds acceptable) | Eventual consistency | | Show a social feed where ordering is approximate | Eventual consistency | | Replicate writes across datacenters for disaster recovery | Eventual consistency (async replication) | **Key distinction — linearizability vs. serializability** (commonly confused): - **Serializability** is an isolation property of *transactions*: multi-object operations behave as if they executed in some serial order. It says nothing about recency. - **Linearizability** is a recency guarantee on *individual objects*: once a write completes, all subsequent reads see that value — no stale reads from any replica. - A system can have one without the other. Serializable snapshot isolation (SSI) is explicitly *not* linearizable by design — reads come from a consistent snapshot, which may not include the latest write. - When you need both: use strict serializability (strong one-copy serializability), implemented by systems that combine 2-phase locking with single-leader replication, or by actual serial execution. ### Step 2 — Select the Minimum Sufficient Consistency Model **WHY**: Causal consistency is the strongest model that does not slow down under network delays and remains available during network failures. Linearizability is strictly stronger but imposes real latency costs proportional to network uncertainty. Always use the weakest model that is correct. **Eventual consistency** — use when: - Staleness is acceptable or expected (analytics, social feeds, DNS lookups) - High availability during partitions is the primary requirement - Conflict resolution is application-managed or last-write-wins is acceptable - Systems: Cassandra (default), Riak, DynamoDB (eventual mode), CouchDB **Causal consistency** — use when: - Operations have cause-and-effect ordering that users would notice if violated (reply before question, update before its acknowledgment) - Multiple communication channels exist (e.g., message queue + file storage), and race conditions between them would cause incorrect behavior - You need read-your-writes, monotonic reads, and consistent prefix reads across replicas - Linearizability overhead is too high but eventual is too weak - Implementation: single-leader replication with reads from leader or synchronously updated follower; causal dependency tracking via version vectors or Lamport timestamps - Note: causal consistency is not a standard off-the-shelf setting in most databases — it requires careful design of how reads are routed and how dependency information is propagated **Linearizability** — use when: - A uniqueness constraint must be enforced as data is written (username, email, seat booking, stock level) - A distributed lock must be held by exactly one node at a time (split-brain prevention) - A leader election must produce exactly one agreed-upon leader - A cross-channel timing dependency exists and cannot be controlled (an external system reads after an out-of-band signal) - The operation is a compare-and-set that must be atomic across replicas - Systems: single-leader replication with reads from leader (potentially), consensus algorithms (ZooKeeper, etcd — linearizable writes, quorum reads for linearizable reads), 2-phase locking + single-leader, actual serial execution - **Not provided by**: multi-leader replication (concurrent writes to different leaders), leaderless/Dynamo-style replication (last-write-wins with clock skew is not linearizable; even strict quorums with variable network delays are not linearizable — see reference) ### Step 3 — Check If the Operation Requires Consensus **WHY**: Consensus is harder than it looks. Many operations that appear simple are actually reducible to consensus, meaning they require a consensus algorithm to implement correctly in a fault-tolerant way. Identifying this early prevents building brittle custom solutions. The following 6 problems are all equivalent to consensus — if you need to solve any one of them in a fault-tolerant distributed system, you need a consensus algorithm: 1. **Linearizable compare-and-set registers** — atomically decide whether to set a value based on its current state 2. **Atomic transaction commit** — decide whether to commit or abort a distributed transaction (all nodes must agree) 3. **Total order broadcast** — decide the order in which messages are delivered to all nodes 4. **Distributed locks and leases** — decide which client successfully acquired the lock 5. **Membership/coordination service** — decide which nodes are alive and should be considered current members 6. **Uniqueness constraints** — decide which of concurrent conflicting writes wins If your operation matches any of the above, evaluate whether to: - Use an existing consensus service (ZooKeeper, etcd — preferred, see Step 5) - Use a database that internally implements consensus (VoltDB, CockroachDB, Spanner) - Implement two-phase commit for atomic cross-node transactions (see Step 4 for failure modes) ### Step 4 — Evaluate Two-Phase Commit (2PC) If Cross-Node Atomic Commit Is Required **WHY**: 2PC is the standard algorithm for atomic commit across multiple nodes. It solves a real problem but introduces a single point of failure (the coordinator) and can block indefinitely — understanding its failure modes is essential before choosing it. **How 2PC works**: 1. Coordinator sends `prepare` to all participant nodes 2. Each participant votes `yes` (promises it can commit) or `no` (aborts) 3. If all vote `yes`, coordinator writes commit decision to its log (the commit point), then sends `commit` to all participants 4. If any vote `no`, coordinator sends `abort` to all participants **The critical failure mode — coordinator crash after prepare**: - Once a participant has voted `yes`, it cannot unilaterally abort or commit — it must wait for the coordinator's decision - If the coordinator crashes after receiving `yes` votes but before sending the commit/abort: participants are **in-doubt** and blocked indefinitely - In-doubt participants hold row-level locks on all modified rows — no other transaction can proceed on those rows - The only resolution is coordinator recovery (reads its log) or manual administrator intervention - Orphaned in-doubt transactions (coordinator log lost or corrupted) may hold locks permanently until manual rollback **2PC failure mode catalog**: | Failure | Outcome | |---|---| | Coordinator crashes before prepare | Safe: participants can abort | | Participant crashes before voting | Coordinator aborts on timeout | | Network partitions participant from coordinator | Coordinator aborts on timeout (before commit point) | | Coordinator crashes after commit point, before all `commit` sent | Remaining participants are in-doubt; blocked until coordinator recovers | | Coordinator log lost after crash | Orphaned transactions; manual intervention required | | Long coordinator restart (e.g., 20 minutes) | All participant locks held for that duration; application may be largely unavailable | **2PC performance cost**: Disk fsyncs at each phase, additional network round-trips, lock-holding during coordination. MySQL distributed transactions reported at 10x slower than single-node. **When 2PC is appropriate**: - Database-internal distributed transactions (all nodes run the same software — e.g., VoltDB, MySQL Cluster NDB) — failure modes are more manageable - Heterogeneous systems that all support XA transactions (PostgreSQL, MySQL, Oracle + JTA-compatible message brokers) **When to avoid 2PC**: - High-availability requirements: a coordinator crash can make the application unavailable - Heterogeneous systems where not all participants support XA or 2PC - When the coordinator is not replicated (single point of failure for the entire transaction system) - Stateless application server deployments where coordinator logs can be lost on restart **Alternatives to 2PC for cross-service correctness**: - Sagas (compensating transactions) — eventual consistency with explicit rollback logic - Outbox pattern — atomic local write + reliable event relay (see `distributed-failure-analyzer`) - Total order broadcast + idempotent consumers — exactly-once processing without 2PC ### Step 5 — Select the Consensus Implementation **WHY**: Implementing consensus from scratch has a very poor success record. Well-tested consensus systems exist and should be used as building blocks. The key insight is that total order broadcast (the core of Raft, Zab, Paxos) is the practical primitive that enables all 6 consensus-equivalent problems to be solved safely. **Total order broadcast** (also called atomic broadcast) provides: - **Reliable delivery**: if a message is delivered to one node, it is delivered to all - **Totally ordered delivery**: all nodes receive messages in exactly the same order - This is equivalent to repeated rounds of consensus — and it is what ZooKeeper, etcd, and Kafka (with ZooKeeper or KRaft) implement **Consensus algorithm selection**: | Algorithm | Implemented by | Notes | |---|---|---| | Raft | etcd, CockroachDB, TiKV, Consul | Well-specified, good documentation, widely adopted | | Zab | ZooKeeper | Total order broadcast directly; basis of Hadoop/HBase/Kafka coordination | | Multi-Paxos | Google Chubby, Spanner | Highly proven, complex to implement correctly | | Viewstamped Replication | Basis for VR-based systems | Theoretically important, less common in production | **Do not implement your own consensus algorithm.** Use one of the above systems. **ZooKeeper / etcd as outsourced consensus** — prefer this model when: - You need distributed locks, leader election, service discovery, or membership tracking - Your application should not embed consensus logic - You need the combination: linearizable atomic operations + total ordering + failure detection + change notifications - Note: ZooKeeper writes are linearizable by default; reads may be stale unless you request a linearizable read (quorum read in etcd, `sync()` call in ZooKeeper) **Fault-tolerant consensus limitations**: - Requires a strict majority: tolerate 1 failure → need 3 nodes; tolerate 2 failures → need 5 nodes - Safety properties (agreement, integrity, validity) are always maintained even if a majority fails - Liveness (termination) requires fewer than half the nodes to be failed or unreachable - Performance degrades with high network variance (frequent false leader timeouts trigger leader elections, reducing throughput) - Fixed membership by default — dynamic membership extensions exist but are less well-understood ### Step 6 — Document the CAP Trade-Off Per Linearizable Operation **WHY**: CAP is widely misunderstood. The correct framing is: when a network partition occurs, a linearizable system must choose between staying consistent (refusing requests) or becoming available (serving potentially stale data). This is not a design choice you make once — it is a per-operation consequence of requiring linearizability. **What CAP actually says**: - If an application *requires* linearizability: when a replica is disconnected from others, it must wait or return an error — it becomes *unavailable* - If an application *does not require* linearizability: replicas can process requests independently during a partition — it remains *available* but behavior is not linearizable - The "pick 2 of 3" framing is misleading: network partitions are not optional. A better framing: *either Consistent or Available when Partitioned* - CAP is narrow in scope — it only addresses linearizability and network partition faults, says nothing about latency, node crashes, or other faults. It has been superseded by more precise results for system design purposes **Practical consequence**: For each operation you mark as requiring linearizability, document the expected behavior during a partition: error returned, request queued, or application component unavailable. Stakeholders should understand this before the system is deployed. --- ## Examples ### Example 1 — Seat Booking Service **Scenario**: An event ticketing platform lets users book the last seat in a venue. Two users submit requests concurrently. **Trigger**: "We're seeing double-bookings in our seat reservation system." **Process**: 1. Gather: the seat availability check + reservation write is a uniqueness constraint — exactly one booking must win 2. Model selection: linearizability required. Both concurrent reads see availability = 1; without a linearizable compare-and-set, both can commit. 3. Consensus check: uniqueness constraint → problem 6 in the consensus-equivalent list → requires consensus 4. Implementation: use a single-leader database with a serializable transaction (or at minimum a linearizable compare-and-set on the seat record). A Dynamo-style leaderless database with last-write-wins is not safe here. 5. 2PC: only needed if the seat record and the payment record live in different databases. If so, use a saga with compensation (refund) instead of 2PC to avoid coordinator-failure blocking. **Output**: Linearizability required for the reservation write. Use a single-leader database with serializable isolation for the seat-payment transaction. If cross-database: saga pattern with idempotent payment reversal. ### Example 2 — Multi-Region Comment Feed **Scenario**: A social platform shows threaded comments. Replies should never appear before the question being replied to. Comments can be up to 2 seconds stale. The system uses multi-leader replication across 3 regions. **Trigger**: "Users in Asia see replies to questions that haven't appeared yet." **Process**: 1. Gather: the anomaly is a causal ordering violation — replies appearing before their parent 2. Model selection: causal consistency is sufficient. Linearizability is not required (no uniqueness constraint, no lock, stale display is acceptable within 2 seconds). 3. Multi-leader replication is not causally consistent by default — writes on different leaders can be applied in any order on followers. 4. Options: (a) route all reads and writes for a given thread to a single-leader per partition; (b) propagate causal dependency metadata (version vectors) with writes and delay delivery of writes whose dependencies haven't arrived yet; (c) use a single-leader database and accept the write-latency increase for cross-region authors. 5. Total order broadcast is overkill — causal ordering per thread is sufficient. **Output**: Causal consistency required for comment ordering. Recommended: single-leader-per-partition with reads from leader. Multi-leader without dependency tracking is not safe for this workload. ### Example 3 — Leader Election for a Job Scheduler **Scenario**: A distributed job scheduler must have exactly one active scheduler node at a time. If two nodes believe they are the leader, jobs execute twice. **Trigger**: "We have a split-brain problem — two scheduler instances both claim leadership and jobs are running twice." **Process**: 1. Gather: leader election is problem 5 in the consensus-equivalent list (membership/coordination) and requires a linearizable lock 2. Model selection: linearizability required. The lock must be held by exactly one node at a time — all nodes must agree who holds it. 3. Consensus check: leader election → consensus required 4. Implementation: use ZooKeeper or etcd for leader election. Acquire an ephemeral node (ZooKeeper) or a lease (etcd). Use fencing tokens (the monotonically increasing `zxid` in ZooKeeper) to prevent a slow previous leader from acting on a stale lock after a new leader is elected. 5. 2PC is not relevant — this is a lock acquisition, not a cross-node transaction. 6. Do not implement with a custom distributed lock using a regular database row — it will not handle coordinator failure correctly. **Output**: Linearizability required. Use etcd or ZooKeeper for leader election with ephemeral leases and fencing tokens. Never use a custom distributed lock without a consensus-backed service. --- ## References - [consistency-model-spectrum.md](references/consistency-model-spectrum.md) — Formal definitions of all consistency models with ordering properties - [linearizability-vs-serializability.md](references/linearizability-vs-serializability.md) — Side-by-side distinction with examples of systems that provide each - [cap-theorem-analysis.md](references/cap-theorem-analysis.md) — What CAP actually says, its limitations, and how to apply it correctly - [consensus-equivalence-checklist.md](references/consensus-equivalence-checklist.md) — The 6 consensus-equivalent problems with implementation guidance - [2pc-failure-modes.md](references/2pc-failure-modes.md) — Complete 2PC failure catalog, in-doubt recovery, XA limitations, alternatives - [total-order-broadcast-primitives.md](references/total-order-broadcast-primitives.md) — How to build linearizable storage and uniqueness constraints from total order broadcast Cross-references: - `replication-strategy-selector` — replication topology must be compatible with the selected consistency model - `distributed-failure-analyzer` — for failure mode diagnosis when consistency violations appear in production - `transaction-isolation-selector` — for isolation level selection within a single-node or single-leader transaction context ## License This skill is licensed under [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). Source: [BookForge](https://github.com/bookforge-ai/bookforge-skills) — Designing Data-Intensive Applications by Martin Kleppmann. ## Related BookForge Skills Install related skills from ClawhHub: - `clawhub install bookforge-replication-strategy-selector` - `clawhub install bookforge-distributed-failure-analyzer` Or install the full book set from GitHub: [bookforge-skills](https://github.com/bookforge-ai/bookforge-skills)

path: references/2pc-failure-modes.md
content: # Two-Phase Commit: Failure Modes and Alternatives

## How 2PC Works (Normal Path)

Two-phase commit (2PC) ensures atomic transaction commit across multiple nodes. It requires a *coordinator* (transaction manager) and one or more *participants* (database nodes).

**Phase 1 — Prepare**:
1. Application requests a globally unique transaction ID from the coordinator
2. Application executes the transaction on each participant under that transaction ID
3. When ready to commit, coordinator sends `prepare` to all participants
4. Each participant either:
   - Replies `yes`: writes all transaction data to disk, promises it can commit under any future circumstance (crash, power failure, etc. are no longer excuses for refusing to commit later)
   - Replies `no`: aborts the transaction

**Phase 2 — Commit or Abort**:
5. If all participants replied `yes`: coordinator writes commit decision to its own durable log (the *commit point*), then sends `commit` to all participants
6. If any participant replied `no`: coordinator sends `abort` to all participants
7. Coordinator retries `commit` or `abort` indefinitely until all participants acknowledge

**The commit point** is the critical moment: writing the commit decision to the coordinator's log is the irrevocable decision. Everything before this point can still be aborted; everything after must be committed.

## Failure Modes

### Participant Failure Before Voting
**What happens**: Coordinator detects timeout; aborts the transaction and sends `abort` to all other participants.  
**Safe**: The participant hasn't voted yet, so it can roll back on recovery.

### Participant Failure After Voting "Yes"
**What happens**: Participant is in-doubt. When it recovers, it checks in with the coordinator for the commit/abort decision and applies it.  
**Safe**: The coordinator still holds the decision.

### Coordinator Failure Before Sending Prepare
**What happens**: Coordinator crashes before phase 1. Participants have no in-doubt transactions; they can safely abort.  
**Safe**: No participant has committed anything yet.

### Coordinator Failure After Prepare, Before Commit Point (Most Dangerous)
**What happens**:
- Participants have voted `yes` — they are in-doubt
- Participants CANNOT unilaterally abort (another participant may have committed)
- Participants CANNOT unilaterally commit (another participant may have aborted)
- Participants hold row-level locks on all modified rows — no other transaction can proceed on those rows
- Participants wait indefinitely until the coordinator recovers

**Recovery**: When the coordinator recovers, it reads its transaction log:
- If the commit record is present: send `commit` to in-doubt participants
- If no commit record: the commit point was not reached; send `abort` to all

**Lock-holding duration**: If the coordinator takes 20 minutes to restart, participant locks are held for 20 minutes. During this time, other transactions accessing those rows are blocked.

### Coordinator Log Lost or Corrupted After Crash
**What happens**: Orphaned in-doubt transactions — the coordinator cannot determine the correct outcome.  
**Resolution**: Manual administrator intervention required. Administrator must:
1. Examine the state of each participant (has any participant committed or aborted?)
2. Apply the same outcome to all remaining participants

**Operational risk**: This typically happens during a serious production outage, under time pressure, requiring expert judgment about distributed transaction state.

### XA Heuristic Decisions (Emergency Escape Hatch)
Some XA transaction implementations allow a *heuristic decision* — a participant unilaterally decides to commit or abort an in-doubt transaction without a definitive decision from the coordinator.

This is defined as "probably breaking atomicity." It is intended only for getting out of catastrophic situations (coordinator permanently down, locks held for an unacceptable duration). Regular use violates 2PC's correctness guarantees.

## Complete Failure Catalog

| Failure Timing | Blocked? | Data Safe? | Resolution |
|---|---|---|---|
| Before prepare sent | No | Yes | Coordinator aborts on timeout |
| Participant fails before voting | No | Yes | Coordinator aborts on timeout |
| Participant fails after voting yes | Participant blocked | Yes | Coordinator sends decision on participant recovery |
| Coordinator fails before prepare | No | Yes | Participants can abort |
| Coordinator fails after prepare, before commit | **Yes — in-doubt** | Yes (no commit yet) | Wait for coordinator recovery |
| Coordinator fails after commit point, partial sends | **Partial commit — in-doubt** | Partially committed | Wait for coordinator recovery |
| Coordinator log lost after crash | **Indefinitely blocked** | Unknown | Manual administrator intervention |

## Performance Cost

- **Disk fsyncs**: Coordinator must fsync its log before sending commit (at commit point). Each participant must fsync transaction data before replying yes to prepare. Total: 2+ fsyncs per transaction.
- **Network round-trips**: At minimum 2 round-trips (prepare + commit) vs. 1 for single-node commit.
- **Lock-holding**: Participants hold locks throughout both phases — much longer than single-node transactions.
- **Benchmark**: MySQL distributed transactions reported at 10x slower than single-node transactions.

## 2PC vs. Fault-Tolerant Consensus

| Property | 2PC | Fault-Tolerant Consensus (Raft/Zab) |
|---|---|---|
| Uniform agreement | Yes | Yes |
| Integrity | Yes | Yes |
| Validity | Yes | Yes |
| Termination | **No** (blocks if coordinator crashes) | Yes (majority quorum sufficient) |
| Coordinator | Single point of failure | Leader elected by consensus |
| Requires all nodes? | **Yes** — any participant failure can block | No — majority quorum sufficient |
| Amplifies failures | Yes — one broken participant blocks all | No — minority failures are tolerated |

2PC is a kind of consensus algorithm, but not a very good one — it does not satisfy the termination property.

## Distributed Transaction Types

**Database-internal distributed transactions**: All participants run the same database software (VoltDB, MySQL Cluster NDB, CockroachDB). The coordinator and protocol are optimized for that specific system. Failure modes are more manageable; performance is better.

**Heterogeneous distributed transactions (XA)**: Participants are different technologies (PostgreSQL + Oracle + ActiveMQ). XA is the standard C API for cross-technology 2PC (Java Transaction API/JTA wraps XA for Java). Limitations:
- Cannot detect deadlocks across systems (no shared lock information)
- Does not work with SSI (requires conflict detection across different systems)
- Coordinator log on application server makes the server stateful — breaks horizontal scaling model
- Coordinator is a single point of failure unless explicitly replicated (most XA coordinators are not highly available by default)

## Alternatives to 2PC

### Saga Pattern (Compensating Transactions)
Break a multi-step transaction into a sequence of local transactions, each with a compensating action for rollback.
- Provides eventual atomicity (not strict atomicity)
- Requires idempotent compensating actions
- Application must handle partial state during saga execution
- Best for: long-running business transactions where strict atomicity is not required

### Outbox Pattern
Write the event/message to the same database as the main write, in the same local transaction. A relay process reads the outbox and publishes to the message broker.
- Guarantees at-least-once delivery without cross-system 2PC
- Requires idempotent consumers
- Best for: reliably publishing events to a message broker after a database write

### Total Order Broadcast + Idempotent Consumers
Use a log (Kafka, Kinesis) as the source of truth. All consumers process messages in the same order. Idempotency handles duplicate delivery.
- Best for: systems where all state is derived from a log (event sourcing, CQRS)
- Does not require 2PC across the producer and consumer systems


path: references/cap-theorem-analysis.md
content: # CAP Theorem Analysis

## What CAP Actually Says

CAP (Consistency, Availability, Partition tolerance) is often summarized as "pick 2 of 3." This framing is misleading. The correct interpretation:

**When a network partition occurs, a linearizable (consistent) system must choose between**:
- **Remaining consistent**: refuse requests from nodes that cannot contact the majority — become *unavailable*
- **Remaining available**: process requests from disconnected nodes — violate *linearizability*

A better statement: **either Consistent or Available when Partitioned (CAAP)**.

## Why "Pick 2 of 3" Is Wrong

Network partitions are not a design choice. They happen whether you want them to or not — packets are delayed, lost, or reordered; hardware fails; datacenter links go down. You cannot opt out of partition tolerance. The real choice is what to do *when* a partition occurs.

Additionally:
- CAP's definition of "availability" is idiosyncratic — it means every request receives a response, not that the system is "highly available" in the operational sense. Many "highly available" systems (that use consensus) do not meet CAP's availability definition.
- CAP only considers one consistency model (linearizability) and one fault type (network partitions). It says nothing about node crashes, network delays, or other faults.
- CAP has been superseded by more precise results (PACELC, etc.) for system design purposes.

## Correct Application

For each operation that requires linearizability, document the expected behavior during a network partition:

| Situation | Linearizable System Behavior | Non-Linearizable System Behavior |
|---|---|---|
| Node disconnected from leader | Must wait or return error | Can serve requests (potentially stale) |
| Cross-datacenter link down (single-leader) | Follower datacenter unavailable for writes and linearizable reads | Multi-leader: both datacenters continue, may conflict |
| Minority partition | Minority nodes unavailable (cannot form quorum) | May continue serving (leaderless, eventual) |

## Partition Behavior by Replication Strategy

**Single-leader (linearizable reads from leader)**:
- Clients connected to leader datacenter: unaffected
- Clients connected to follower datacenter: unavailable for writes and linearizable reads during partition
- This is the correct linearizability/availability trade-off

**Multi-leader**:
- Both datacenters continue operating during partition
- Writes accepted independently; conflict resolution required when partition heals
- Behavior is NOT linearizable

**Leaderless (Dynamo-style)**:
- Sloppy quorums or degraded quorums may serve requests during partition
- Data may diverge; read repair resolves after partition heals
- Behavior is NOT linearizable

**Consensus algorithm (Raft/Zab)**:
- Majority partition: continues normally
- Minority partition: unavailable (cannot make progress without quorum)
- Behavior IS linearizable for the majority partition

## The Latency-Consistency Trade-off (Beyond CAP)

Even on a network that is working correctly (no partition), linearizability is slow. The Attiya-Welch theorem proves that in a network with variable delays, the response time of linearizable reads and writes is at least proportional to the uncertainty of those delays.

This means: in a geographically distributed system with high inter-region latency, linearizable operations will be slow even when nothing is failing. Many distributed databases choose weaker consistency models primarily for latency reasons, not partition tolerance.

**Practical implication**: Even within a single datacenter, RAM on a modern multi-core CPU is not linearizable (CPU caches create multiple copies of data, updated asynchronously). Linearizability was sacrificed for performance — memory barriers (fences) re-introduce ordering when needed.

## When to Apply CAP Analysis

Apply CAP analysis to each operation you have marked as requiring linearizability:

1. What happens during a network partition? Which clients lose access?
2. Is partial availability acceptable (clients on majority partition continue; minority partition returns errors)?
3. What is the expected duration of partitions in your network? (Data center link failure: minutes to hours. Within-DC: milliseconds.)
4. Is the cost of unavailability during a partition acceptable, given how rare partitions are?

If unavailability during a partition is not acceptable, reconsider whether linearizability is truly required for that operation, or whether causal consistency with application-level conflict handling is sufficient.

## CAP Does Not Apply to Causal Consistency

Causal consistency is the strongest model that remains available during network failures and does not slow down due to network delays. The CAP theorem does not apply to causally consistent systems because they do not make the recency guarantee that linearizability requires.

This is why, for many operations that appear to need linearizability (but actually only need causal ordering), causal consistency is the correct — and much cheaper — choice.


path: references/consensus-equivalence-checklist.md
content: # Consensus Equivalence Checklist

## What Consensus Means

Consensus means getting several nodes to agree on a single value, such that:
- **Uniform agreement**: No two nodes decide differently
- **Integrity**: No node decides twice
- **Validity**: The decided value was proposed by some node (not invented)
- **Termination**: Every non-crashed node eventually decides

The key challenge is termination under faults. 2PC satisfies the first three properties but fails termination (in-doubt participants block indefinitely when the coordinator crashes). True consensus algorithms (Raft, Zab, Paxos) satisfy all four.

## The 6 Consensus-Equivalent Problems

These problems are all reducible to each other. If you can solve one in a fault-tolerant way, you can transform that solution to solve any of the others. All require a consensus algorithm for a correct fault-tolerant implementation.

### 1. Linearizable Compare-and-Set Registers
**Problem**: Atomically read a value, compare it to an expected value, and set it to a new value only if the comparison succeeds — and have all nodes agree on whether the operation succeeded.  
**Why it needs consensus**: The "compare" and "set" must be atomic across replicas. Without consensus, two nodes can each see the old value and both succeed their CAS, producing split state.  
**Implementation**: ZooKeeper's `setData` with version check, etcd's transactional `compare-and-swap`.  
**Practical use**: Username uniqueness, lock acquisition, leader election tokens, optimistic concurrency control.

### 2. Atomic Transaction Commit
**Problem**: A transaction spans multiple nodes. Either all nodes commit, or all abort — no partial commits.  
**Why it needs consensus**: Nodes must agree on the commit/abort decision despite failures. 2PC approximates this but fails the termination property when the coordinator crashes.  
**Implementation**: Two-phase commit (approximate, not full consensus), or database-internal consensus (VoltDB, CockroachDB).  
**Practical use**: Cross-shard transactions, multi-table atomic updates in partitioned databases.

### 3. Total Order Broadcast
**Problem**: All nodes must receive messages in exactly the same order, with no messages lost.  
**Why it needs consensus**: Deciding the position of each message in the total order requires agreement among nodes. A single leader can serialize messages, but choosing that leader — and handling leader failures — requires consensus.  
**Implementation**: ZooKeeper (Zab), etcd (Raft), Kafka (KRaft or ZooKeeper-based).  
**Practical use**: Database replication log, serializable transactions via stored procedures, event sourcing with global ordering.

### 4. Distributed Locks and Leases
**Problem**: Multiple clients race to acquire a lock. Exactly one must succeed. The lock must be released on client failure (leases with expiry).  
**Why it needs consensus**: The decision of "which client holds the lock" must be agreed upon by all nodes. Without consensus, a network partition can cause two clients to each believe they hold the lock (split-brain).  
**Implementation**: ZooKeeper ephemeral nodes, etcd leases. Always use fencing tokens to handle slow lock holders.  
**Practical use**: Single-active-instance patterns (job schedulers, cron, leader-only tasks).

**Fencing token requirement**: A distributed lock is not sufficient on its own. If the lock holder is paused (garbage collection, slow network) after acquiring the lock, it may act on the lock after a new holder has been elected. Fencing tokens — a monotonically increasing number attached to every request that uses the lock — allow the resource being protected to reject stale requests from the previous lock holder. ZooKeeper provides this via `zxid`.

### 5. Membership and Coordination Service
**Problem**: Nodes must agree on which other nodes are currently alive and part of the cluster. A failed node should be removed from membership; a recovered node should be re-admitted.  
**Why it needs consensus**: Without agreement on membership, different nodes have divergent views of the cluster. Operations that depend on "the current member set" (partition assignment, quorum calculation) become inconsistent.  
**Implementation**: ZooKeeper (failure detection via session timeouts + ephemeral nodes), etcd (similar model).  
**Practical use**: Service discovery, partition assignment, quorum determination, failover coordination.

### 6. Uniqueness Constraints
**Problem**: Multiple concurrent operations attempt to create records with the same unique key (same username, same email, same seat). Exactly one must succeed; the rest must fail with a constraint violation.  
**Why it needs consensus**: The decision of "which request won" must be agreed upon by all nodes before any of them responds to the client. Lamport timestamps can define a total order after the fact, but a node receiving a request cannot know in real time whether another node is concurrently processing a conflicting request with a lower timestamp.  
**Implementation**: Single-leader database with a unique index, or a linearizable compare-and-set register (which itself requires consensus).  
**Practical use**: Username registration, email uniqueness, seat booking, inventory reservation.

## Quick Checklist: Do I Need Consensus?

Answer yes to any of the following → consensus is required:

- [ ] Two concurrent operations could both believe they succeeded but only one should
- [ ] I need to enforce a uniqueness constraint as data is written (not just detect violations after the fact)
- [ ] I need a distributed lock that prevents any form of split-brain
- [ ] I need to elect a single leader and all nodes must agree who it is
- [ ] A transaction spans multiple nodes or databases and must be atomic
- [ ] Messages must be delivered to all nodes in the same order for correctness
- [ ] I need to know which nodes are live members of a cluster with agreement

## How to Use the Equivalence

If you already have ZooKeeper or etcd (which implement total order broadcast = consensus):
- You can build all 6 primitives on top of it — linearizable CAS, atomic commit coordination, distributed locks, membership, uniqueness constraints, and message ordering
- You do not need to implement a separate consensus mechanism for each problem

If you do not have a consensus service:
- Do not build your own consensus algorithm (very high failure rate)
- Deploy ZooKeeper or etcd as infrastructure and use it for all consensus-equivalent operations
- Alternatively, use a database that internally implements consensus (CockroachDB, TiKV, Spanner, VoltDB)


path: references/consistency-model-spectrum.md
content: # Consistency Model Spectrum

Consistency models form a hierarchy from weakest to strongest. Stronger models are easier to reason about but impose higher latency and availability costs.

## The Spectrum (Weakest to Strongest)

```
Eventual Consistency
      |
  Causal Consistency          ← strongest model with no network-delay slowdown
      |
  Sequential Consistency
      |
  Linearizability             ← strongest common model; recency guarantee
      |
  Strict Serializability      ← linearizability + serializability combined
```

## Definitions

### Eventual Consistency (Convergence)
**Guarantee**: If you stop writing, all replicas will eventually converge to the same value.  
**What it does NOT guarantee**: When convergence happens. Until convergence, reads may return anything — including stale values, missing values, or different values from different replicas.  
**Ordering**: None. Operations are not ordered relative to each other.  
**Examples**: Cassandra (default), Riak, DynamoDB (eventual mode), CouchDB, DNS.  
**Best for**: High-availability workloads where staleness is acceptable.

### Causal Consistency
**Guarantee**: Operations that are causally related (cause → effect) are seen in that order by all nodes. Operations with no causal relationship may be seen in any order (they are concurrent).  
**Ordering**: Partial order. Causal dependencies define a directed acyclic graph; concurrent operations are incomparable.  
**Key property**: Causal consistency is the *strongest* model that does not slow down due to network delays and remains available during network failures. CAP theorem does not apply to causal consistency.  
**Subsumes**: Read-your-writes, monotonic reads, consistent prefix reads.  
**Implementation**: Version vectors (per-key causal tracking), Lamport timestamps (total order consistent with causality), or single-leader replication with reads from the leader.  
**Examples**: Snapshot isolation provides causal consistency. COPS, Eiger, Occult (research systems).

### Sequential Consistency
**Guarantee**: All nodes see operations in the same total order, and that order is consistent with the program order on each individual node.  
**What it does NOT guarantee**: That the total order reflects real time. A write may be "seen" after a long delay, but once seen, it appears to have happened at a consistent point relative to other operations.  
**Note**: Less common in distributed databases; more common in CPU memory models.

### Linearizability (Atomic Consistency, Strong Consistency, Immediate Consistency, External Consistency)
**Guarantee**: The system behaves as if there is a single copy of the data, and all operations are atomic. Once a write completes, all subsequent reads — from any replica — see that write. This is a *recency guarantee*.  
**Ordering**: Total order. There is one global timeline; no operation is concurrent with another in the observable history.  
**Key property**: After any one read returns a new value, all following reads (on the same or any other client) must also return the new value.  
**Cost**: Response time is proportional to the uncertainty of network delays — linearizability is inherently slow in high-latency networks.  
**Examples**: ZooKeeper (writes), etcd (writes + linearizable reads), single-leader replication with reads from leader (potentially), Spanner, CockroachDB.

### Strict Serializability (Strong One-Copy Serializability)
**Guarantee**: Transactions behave as if they executed serially on a single copy of the data, AND that serial order is consistent with real time (linearizable).  
**Implementation**: Two-phase locking (2PL) + single-leader replication, or actual serial execution.  
**Examples**: FaunaDB, older VoltDB configurations.  
**Note**: Serializable snapshot isolation (SSI) is serializable but NOT linearizable — reads come from a snapshot that may not include the most recent writes.

## Ordering Properties Compared

| Model | Total Order? | Recency Guarantee? | Available During Partition? | Network-delay-free? |
|---|---|---|---|---|
| Eventual | No | No | Yes | Yes |
| Causal | Partial | No | Yes | Yes |
| Sequential | Total (program-order-consistent) | No | No | No |
| Linearizable | Total (real-time-consistent) | Yes | No | No |
| Strict Serializable | Total (real-time-consistent, multi-object) | Yes | No | No |

## Replication Method Compatibility

| Replication Method | Linearizable? | Notes |
|---|---|---|
| Single-leader, reads from leader | Potentially | Not every single-leader DB is linearizable; snapshot isolation breaks it |
| Single-leader, reads from follower | No | Followers may lag |
| Consensus algorithm (Raft/Zab) | Yes | This is how ZooKeeper/etcd achieve it |
| Multi-leader | No | Concurrent writes to different leaders; conflicts must be resolved |
| Leaderless (Dynamo-style) | Probably not | Quorums with variable network delays are not linearizable; LWW with clock skew is definitely not |
| Leaderless with synchronous read repair | Possible for reads/writes, not CAS | Cassandra does not do this by default; CAS requires consensus |


path: references/linearizability-vs-serializability.md
content: # Linearizability vs. Serializability

These two terms are among the most commonly confused in distributed systems. They sound similar and both involve ordering, but they are orthogonal guarantees about different things.

## The Core Distinction

| Property | Serializability | Linearizability |
|---|---|---|
| What it applies to | *Transactions* (multi-object operations) | *Individual objects* (single registers/keys) |
| What it guarantees | Transactions behave as if they ran in *some* serial order | Reads always return the most recently written value |
| Recency guarantee? | No — the serial order can differ from real time | Yes — once a write completes, all reads see it |
| Concurrent operations? | Concurrent transactions allowed; conflicts prevented or serialized | No concurrent operations: one global timeline |
| Prevents write skew? | Yes (serializable transactions prevent write skew) | No — write skew requires multi-object transactions |
| Prevents stale reads? | No — serializable reads may come from a snapshot | Yes — no stale reads from any replica |

## Serializability

Serializability is a property of a *transaction execution schedule*. It guarantees that the result of executing a set of concurrent transactions is the same as if they had been executed one at a time, in *some* serial order — but that serial order need not match the wall-clock order in which the transactions actually ran.

**What serializability does NOT prevent**:
- Stale reads from a replica that hasn't received the latest write
- A read seeing data that is "in the past" relative to real time

**What serializability DOES prevent**:
- Write skew (two transactions reading overlapping data and each making a write that is only safe based on what the other hasn't written yet)
- Lost updates, dirty reads (with sufficient isolation level)

**Implementations**:
- Two-phase locking (2PL): serializable, and typically also linearizable for individual objects
- Actual serial execution: serializable, linearizable for individual objects
- Serializable snapshot isolation (SSI): serializable, but **NOT linearizable**

**SSI and non-linearizability**: SSI deliberately reads from a consistent snapshot taken at transaction start. Writes that happen after that snapshot is taken are not visible to the transaction. This means a read within an SSI transaction may not see the latest committed write — this is the definition of non-linearizability.

## Linearizability

Linearizability is a property of a *single-object execution history*. It guarantees that the system behaves as if there is only one copy of the data, and every operation takes effect atomically at a single point in time between its start and completion.

**What linearizability does NOT prevent**:
- Write skew across multiple objects (because it applies per-object, not per-transaction)
- Dirty reads in multi-object transactions (no transaction concept)

**What linearizability DOES prevent**:
- Stale reads: once a write completes, no subsequent read from any client on any replica may return the old value
- Time-travel reads: reads cannot go "backward" in the value sequence

**Implementations**:
- Single-leader with reads from leader: potentially linearizable (depends on implementation)
- Consensus algorithms (Raft, Zab): linearizable writes; linearizable reads require quorum reads or sync()
- Multi-leader, leaderless: NOT linearizable

## Combining Both: Strict Serializability

When a system provides both serializability and linearizability, it is called *strict serializability* or *strong one-copy serializability (strong-1SR)*. This is the strongest practical guarantee:
- Multi-object transactions behave as if executed serially (serializability)
- That serial order is consistent with real time (linearizability)

**Implementations**: 2PL + single-leader replication, actual serial execution on a single leader.

## Decision Guide

Use this to select which guarantee you need:

**Do you need multi-object atomicity (atomic read-modify-write across multiple keys or tables)?**
- Yes → You need serializability (or at minimum, a weaker isolation level that prevents the specific anomaly you care about — see `transaction-isolation-selector`)
- No → Serializability is not needed

**Do you need recency: once a write completes, every subsequent read must see it, from any replica?**
- Yes → You need linearizability
- No → A weaker model (causal or eventual) may suffice

**Combining both needs → strict serializability**. Be aware this has the highest latency and availability cost.

## Practical Examples

| System / Config | Serializable? | Linearizable? |
|---|---|---|
| PostgreSQL with SERIALIZABLE isolation | Yes (SSI) | No (snapshot reads) |
| PostgreSQL with SERIALIZABLE + synchronous replication, reads from leader | Yes | Yes (for single objects) |
| MySQL with 2PL (SERIALIZABLE isolation) | Yes (2PL) | Yes (single objects) |
| Cassandra (default) | No | No |
| ZooKeeper writes | N/A (single-object) | Yes |
| ZooKeeper reads (without sync()) | N/A | No (may be stale) |
| etcd writes | N/A | Yes |
| etcd linearizable reads (default) | N/A | Yes |
| CockroachDB | Yes (SSI variant) | Yes |
| Spanner | Yes | Yes |


path: references/total-order-broadcast-primitives.md
content: # Total Order Broadcast as a Practical Consensus Primitive

## What Total Order Broadcast Provides

Total order broadcast (also called atomic broadcast) is a messaging protocol with two safety properties that must hold even during faults:

1. **Reliable delivery**: No messages are lost — if a message is delivered to one node, it is delivered to all nodes
2. **Totally ordered delivery**: Messages are delivered to every node in exactly the same order

These two properties, together, are equivalent to repeated rounds of consensus — each delivery decision is one consensus round. This is why fault-tolerant consensus algorithms (Raft, Zab, Multi-Paxos) implement total order broadcast directly.

## Relationship to Consensus

Total order broadcast and consensus are equivalent in expressive power:
- **Total order broadcast → consensus**: To reach consensus on a value, broadcast it; the first message in the total order is the consensus decision
- **Consensus → total order broadcast**: Use repeated consensus rounds to decide the next message in the sequence

This equivalence means: if you have ZooKeeper or etcd (which implement total order broadcast), you already have consensus, and all 6 consensus-equivalent problems can be solved using it.

## Relationship to Linearizability

Total order broadcast and linearizability are related but not identical:
- Total order broadcast is **asynchronous**: messages are delivered in a fixed order, but there is no guarantee about *when* a message will be delivered (one node may lag behind)
- Linearizability is a **recency guarantee**: a read is guaranteed to return the most recently written value

**From total order broadcast → linearizable storage**:
You can build linearizable read-write storage on top of total order broadcast. Example — linearizable username uniqueness:
1. Append a message to the log: "I want to claim username X"
2. Wait for the message to be delivered back to you (it's now in the global order)
3. Scan all preceding log entries for any claim on username X
4. If your entry is first: claim succeeds. If another claim appears before yours: abort.

This ensures that because all nodes receive log entries in the same order, they will all agree on who claimed the username first. Writes are linearizable by this procedure.

**Making reads linearizable** (choose one):
- Sequence reads through the log: append a read message, perform the read when the message is delivered back (used by etcd quorum reads)
- Fetch the latest log position in a linearizable way, wait for all entries up to that position, then read (ZooKeeper `sync()`)
- Read from a replica that is synchronously updated on every write (chain replication)

**From linearizable storage → total order broadcast**:
If you have a linearizable register with an atomic increment-and-get operation, you can assign monotonically increasing sequence numbers to messages and implement total order broadcast. This is why linearizable sequence number generators inevitably lead to consensus algorithms when you think about fault tolerance.

## Total Order Broadcast as a Replication Log

Total order broadcast is exactly what is needed for database replication:
- Each message represents a write to the database
- Every replica processes the same writes in the same order
- Replicas remain consistent with each other (aside from temporary replication lag)

This principle is called *state machine replication* — if you start from the same initial state and apply the same sequence of deterministic operations, you arrive at the same final state.

**Kafka** uses this model: producers append messages to a partition log; consumers process messages in partition order. Kafka's ordering guarantee is per-partition total order (not global total order across partitions).

**ZooKeeper** uses Zab to implement total order broadcast for its own replication, making ZooKeeper itself a strongly consistent, fault-tolerant data store.

## Building Linearizable Operations from Total Order Broadcast

### Linearizable Compare-and-Set (CAS)
1. Append a CAS proposal to the log: `CAS(key, expected_value, new_value)`
2. Wait for delivery back to yourself
3. Apply all preceding log entries
4. Check if the key's current value matches `expected_value`
5. If yes: apply the CAS (commit it with another log entry or inline in step 4)
6. If no: reject the CAS

All nodes apply the same log in the same order, so all agree on the CAS outcome.

### Atomic Transaction Coordination
Total order broadcast can coordinate the commit/abort decision that 2PC's coordinator holds:
- Broadcast the "prepare" outcome (yes/no votes from all participants) via total order broadcast
- All nodes see the same votes in the same order; all can independently determine the commit/abort outcome
- Eliminates the coordinator as a single point of failure

### Serializable Multi-Object Transactions
If every message represents a deterministic stored procedure, and every node processes those procedures in the same order, the result is serializable multi-object transactions — this is the model used by VoltDB and H-Store (actual serial execution).

## Fault-Tolerant Consensus Algorithm Summary

| Algorithm | Key Systems | Notes |
|---|---|---|
| Raft | etcd, CockroachDB, TiKV, Consul | Well-documented; leader-based; strong safety |
| Zab | ZooKeeper | Total order broadcast natively; basis of Hadoop/Kafka/HBase coordination |
| Multi-Paxos | Google Chubby, Spanner, some Cassandra configs | Highly proven; complex to implement correctly; "nobody really understands Paxos" |
| Viewstamped Replication (VSR) | Research, some production | Theoretically important; similar structure to Raft |

**Epoch numbers and quorums**: All these algorithms use some form of epoch (term, ballot, view) numbering. Each epoch has a unique leader. If a conflict arises between leaders from different epochs, the higher epoch wins. Leaders must collect a quorum of votes before making a decision, and the quorum for leader election must overlap with the quorum for proposals — ensuring that any new leader knows about all committed decisions from previous epochs.

**Key difference from 2PC**: Fault-tolerant consensus requires votes from a *majority* of nodes. 2PC requires a `yes` from *every* participant. This means:
- A single slow or failed participant can block 2PC indefinitely
- A minority of failures does not block Raft/Zab/Paxos

## When to Use ZooKeeper / etcd vs. Database-Internal Consensus

**Use ZooKeeper or etcd** for:
- Distributed locks and leases (with fencing tokens)
- Leader election for application components
- Service discovery and membership tracking
- Configuration that must be consistent across all nodes
- Partition assignment coordination (Kafka, HBase, YARN)

**Use database-internal consensus** (CockroachDB, Spanner, TiKV) for:
- Linearizable data storage at scale
- Cross-shard transactions with linearizability
- When you want the database to handle consensus transparently

**Do not use** ZooKeeper/etcd for:
- High-throughput application data storage — it is designed for small, slow-changing coordination data
- Per-request state — the overhead of consensus is not worth it for data that can be eventually consistent


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)