Skill: quochungto/bookforge-data-integration-architect (version 1.0.0)

--- name: data-integration-architect description: | Design the integration architecture for systems with multiple specialized data stores (Postgres, Elasticsearch, Redis, data warehouses) that must stay in sync. Use when deciding how data flows between components, avoiding dual writes, reasoning about correctness across system boundaries (idempotency, end-to-end operation identifiers), choosing between Lambda and Kappa architecture, or applying the "unbundling databases" pattern to compose specialized tools instead of relying on a single monolith. Trigger phrases: "how do I keep Postgres and Elasticsearch in sync?", "should I use CDC or event sourcing to propagate data?", "how do I avoid dual writes across microservices?", "my downstream systems are going out of sync — how do I fix the architecture?", "how do I design derived data pipelines?", "what is the system of record pattern?", "how do I integrate OLTP with a search index and an analytics warehouse?", "how do I design for end-to-end idempotency?". This is the capstone skill for data systems design — it synthesizes batch pipelines, stream integration, consistency, and replication into a single architecture recommendation. Produces a component map (systems of record vs derived views), data flow diagram, and correctness analysis. Does not replace batch-pipeline-designer or stream-processing-designer — delegates to them for pipeline internals. model: sonnet context: 1M execution: tier: 1 mode: hybrid inputs: - type: document description: "Current architecture description, data flow diagram, or system design document (architecture.md, data-flow.md)" - type: codebase description: "docker-compose.yml, schema.sql, config files showing current infrastructure and database choices" - type: none description: "Skill can work from a verbal description of the system and its data stores" tools-required: [Read, Write, TodoWrite] tools-optional: [Grep, Bash] environment: "Any agent environment. Codebase access enables concrete analysis of existing infrastructure. Works equally well from a written system description." depends-on: - batch-pipeline-designer - stream-processing-designer - consistency-model-selector - replication-strategy-selector source-books: - id: designing-data-intensive-applications title: "Designing Data-Intensive Applications" authors: ["Martin Kleppmann"] chapters: [12] tags: - data-integration - derived-data - system-of-record - unbundling-databases - lambda-architecture - kappa-architecture - change-data-capture - event-sourcing - total-order-broadcast - idempotency - end-to-end-correctness - dataflow - federated-database - write-path - read-path - dual-writes - multi-system-consistency - operation-identifiers - coordination-avoiding - timeliness - integrity version: 1.0.0 homepage: https://github.com/bookforge-ai/bookforge-skills/tree/main/books/designing-data-intensive-applications/skills/data-integration-architect metadata: {"openclaw":{"emoji":"📚","homepage":"https://github.com/bookforge-ai/bookforge-skills"}} status: draft --- # Data Integration Architect ## When to Use Use this skill when you are designing or evaluating an architecture where **multiple specialized data stores need to work together** and you must reason about how data flows between them, stays consistent, and remains correct in the face of faults. Concrete preconditions: - You have (or are planning) more than one data store: for example, a primary relational database plus a search index, an analytics warehouse, a cache, or a machine learning feature store. - You need to decide the **integration strategy**: should writes go to all stores directly, or should one store be the source of truth with others derived from it? - You need to choose the **propagation mechanism**: synchronous distributed transactions, change data capture, event sourcing, or batch ETL. - You need to evaluate a **processing architecture**: Lambda (separate batch and stream layers), Kappa (unified stream layer), or a single integrated system. - You have a correctness problem: downstream systems are drifting, duplicate events are being applied, or a system boundary is leaking bugs. **Do not use this skill** if you have a single-database workload with no integration requirements — use `oltp-olap-workload-classifier` and `storage-engine-selector` instead. --- ## Context and Input Gathering Before proceeding, gather the following. If any are unknown, ask the user. **Required:** 1. **Current data stores** — list every database, index, cache, or data warehouse in the system (e.g., PostgreSQL, Elasticsearch, Redis, Snowflake, Kafka). 2. **Data sources** — what writes new data into the system, and at what volume/rate? 3. **Data consumers** — who reads from each store, and what access patterns do they require (full-text search, OLAP aggregation, low-latency lookup, ML training)? 4. **Consistency requirements** — which parts of the system require linearizability (user-facing correctness), and which can tolerate eventual consistency (derived views)? 5. **Growth trajectory** — current data volume, expected growth rate, and peak write throughput. **Optional but valuable:** - Existing architecture diagram or `architecture.md` / `docker-compose.yml` - Current integration mechanism (if any: ETL scripts, dual writes, triggers, CDC) - Known pain points: lag, drift, data loss, duplicate processing, schema migration friction **If dependencies are installed**, invoke them to fill gaps: - `consistency-model-selector` → determine per-operation consistency requirements - `replication-strategy-selector` → determine replication topology for the primary store - `batch-pipeline-designer` → design the batch processing layer if historical reprocessing is needed - `stream-processing-designer` → design the stream processing layer if low-latency propagation is needed --- ## Process ### Step 1: Map the Current Dataflow **ACTION:** Identify every system that stores data and every transformation that moves data between systems. Draw (or describe) the current state: ``` [Write source] → [Store A] → ??? → [Store B] → ??? → [Store C] ``` For each data path, classify the current integration mechanism: - **Dual writes** (application writes to A and B directly) — flag as anti-pattern - **Distributed transactions** (2PC across stores) — note heterogeneity constraints - **Change data capture** (CDC from A's replication log to B) - **Event sourcing** (events written to a log; A and B derived independently) - **Batch ETL** (periodic full or incremental extract from A to B) **WHY:** You cannot improve what you cannot see. Dual writes are the most common source of drift between systems — if the write to Store B fails after the write to Store A succeeds, the stores diverge permanently. Making the dataflow explicit immediately surfaces where this risk exists. --- ### Step 2: Identify the System of Record **ACTION:** For each logical data entity (user, order, product, event), determine which single store is the **system of record** — the authoritative source of truth that other representations are derived from. Apply this test: if Store A and Store B disagree about the value of an entity, which one is correct by definition? - If the answer is "A" → A is the system of record; B is a derived view. - If the answer is "it depends" or "neither" → you have a multi-master problem that must be resolved by architecture (see Step 4). Document the result as a table: | Entity | System of Record | Derived Views | |--------|-----------------|---------------| | User profile | PostgreSQL users table | Elasticsearch user index, Redis session cache | | Order | PostgreSQL orders table | Kafka order-events log, Snowflake orders_fact | | Search index | (none — fully derived) | Elasticsearch ← PostgreSQL via CDC | **WHY:** The system of record designation is not just documentation — it determines write authority. Only the system of record should accept writes for its entities. All other stores are read-optimized derived views that are populated by processing the source of truth. This is what prevents split-brain inconsistency between stores. --- ### Step 3: Choose the Integration Mechanism **ACTION:** For each system of record → derived view pair, select the integration mechanism from this decision framework: **Decision tree:** ``` Is the derived view latency requirement < 1 minute? ├─ YES → Use stream-based propagation (CDC or event sourcing) │ → invoke stream-processing-designer for details └─ NO → Can you afford to reprocess all historical data? ├─ YES → Consider batch ETL (simpler, replayable) │ → invoke batch-pipeline-designer for details └─ NO → Use incremental batch or hybrid (CDC + periodic batch) ``` **For the propagation mechanism, choose:** | Mechanism | Use when | Trade-off | |-----------|----------|-----------| | **Change data capture (CDC)** | Existing database cannot be changed; need low-latency propagation | Requires access to replication log; schema changes need care | | **Event sourcing** | New system or greenfield; want full audit log; need replayability | Application must be redesigned around immutable events | | **Batch ETL** | High latency acceptable; historical reprocessing needed regularly | Simple but creates lag windows; schema evolution is manual | | **Log-based messaging (Kafka)** | High throughput; multiple consumers; need replay; decoupled teams | Operational complexity; ordering only within partition | **Key principle — prefer the event log over distributed transactions:** Distributed transactions (2PC) across heterogeneous stores are fragile: they have poor fault tolerance (coordinator failure leaves participants in-doubt), poor performance (blocking protocol), and require all participating systems to speak the same transaction protocol. An ordered log of events with idempotent consumers achieves the same correctness properties with better fault isolation and looser coupling. A fault in one consumer is contained locally; it does not abort the writes to all other systems. **WHY:** The integration mechanism determines whether failures in one part of the system cascade to other parts. Synchronous coupling (dual writes, distributed transactions) amplifies failures across system boundaries. Asynchronous event logs contain failures: a slow or failed consumer falls behind, but the producer and other consumers continue unaffected. --- ### Step 4: Decide — Single System vs. Composed Pipeline **ACTION:** Evaluate whether the requirements can be satisfied by a single integrated system or require a composed pipeline of specialized tools. Apply this test: **Use a single system if:** - One technology satisfies all access patterns with adequate performance (e.g., PostgreSQL with full-text search is sufficient for the scale) - The team lacks operational capacity to run multiple systems - The workload fits within the throughput and latency envelope of a single product **Use a composed pipeline if:** - No single technology satisfies all access patterns (OLTP + full-text + analytics + ML feature serving) - Scale requirements exceed what a single system can handle - Different teams own different parts of the data, requiring independent deployment **The unbundling principle:** A database internally implements secondary indexes, materialized views, replication logs, and caching. When you compose specialized systems — a primary database plus a search index plus an analytics warehouse plus a feature store — you are "unbundling" those features into separate, independently deployable components. The event log plays the role of the replication log that connects them. The goal of unbundling is **breadth**: achieving good performance across a wider range of workloads than any single system supports. It is not to maximize the number of moving parts. **WHY:** Many teams reach for multi-system architectures prematurely and add operational complexity without proportionate benefit. The decision must be grounded in actual access pattern requirements that cannot be satisfied by a simpler alternative. --- ### Step 5: Choose the Processing Architecture **ACTION:** If a composed pipeline is warranted, select the processing architecture: **Lambda Architecture:** - Two parallel layers: a batch layer (reprocesses all historical data; produces accurate views) and a speed layer (processes recent events; produces approximate views) - Reads merge results from both layers - **Use when:** Batch processing is significantly simpler and less bug-prone than the stream processor you can operate, AND the latency of batch processing alone is unacceptable - **Problems:** Maintaining the same logic in two frameworks doubles complexity; merging batch and stream outputs is non-trivial for operations beyond simple aggregations; batch layer often ends up doing incremental processing anyway, negating its simplicity advantage **Kappa Architecture:** - Single stream processing layer that handles both historical reprocessing (by replaying the log) and recent events - **Use when:** A stream processor with replay capability (e.g., Apache Flink reading from Kafka with historical retention, or Apache Beam on Google Cloud Dataflow) can match the correctness guarantees of batch processing - **Required capabilities:** Log-based broker with configurable retention (replay); exactly-once semantics in the stream processor; event-time windowing (not processing-time windowing, which is meaningless during replay) - **Preferred** when these capabilities are available — eliminates the dual-system burden of Lambda **Single unified system:** - Modern stream processors (Flink, Beam) unify batch and stream by treating batch as a bounded stream - **Use when** the team can operate one such system well; eliminates the architecture choice entirely **WHY:** Lambda architecture was an important idea that made event reprocessing central to data system design. But the practical problems — dual codebases, merged outputs, incremental batch complexity — are significant. The Kappa architecture achieves the same benefits (replayability, correctness, low latency) without the dual-system burden, provided the stream processor supports replay and exactly-once semantics. --- ### Step 6: Design for Total Ordering **ACTION:** Identify which data flows require a total order and evaluate whether total order is achievable given your deployment constraints. **Total ordering is required when:** - Multiple systems derive state from the same events and must remain consistent with each other (they must process events in the same order) - A uniqueness constraint must be enforced (who claimed the username first?) - Causal dependencies exist between events in different partitions or services (the "unfriend before message" example: a message notification system must not deliver a message to a user who was unfriended before the message was sent) **Constraints on total ordering:** | Constraint | Effect | |------------|--------| | Single leader | Total order is feasible; the leader serializes all writes | | Multiple partitions | Order within a partition is guaranteed; across partitions is not | | Multiple datacenters | Total order requires synchronous cross-datacenter coordination (high cost) | | Independent microservices | Events originating in different services have no defined order | **For causal ordering without total order, consider:** - **Logical timestamps** (Lamport clocks, vector clocks): provide causal ordering without a single leader; see `consistency-model-selector` - **Causal event identifiers**: log the state the user saw before making a decision (the event identifier); downstream consumers can reference this to reconstruct the causal dependency - **Partition routing by entity**: route all events for a given entity (user ID, order ID) to the same partition, ensuring per-entity ordering **WHY:** Total order broadcast is equivalent to consensus — it requires a single node (or Raft/Paxos cluster) to serialize all events. This scales well on a single machine but becomes a bottleneck at very high throughput or across geographically distributed systems. Understanding exactly which parts of your data flow require total ordering — versus which can tolerate partial ordering — lets you apply the strong guarantee only where it is necessary. --- ### Step 7: Enforce End-to-End Correctness **ACTION:** Apply the end-to-end argument to the integration architecture. Verify that correctness guarantees are not assumed from any single layer. **The end-to-end argument:** TCP suppresses duplicate packets within a connection. Databases provide transactional atomicity. Stream processors provide exactly-once semantics within the processing framework. But none of these individually prevent a user from submitting a duplicate request after a network timeout. Solving the problem requires passing a unique operation identifier all the way from the end-user client to the final data store. **Correctness checklist — apply to every data flow:** 1. **Idempotency at the consumer:** Can the consumer safely process the same event twice? If not, implement idempotency using a deduplication table keyed by operation ID. ```sql -- Pattern: unique constraint on request_id suppresses duplicates ALTER TABLE requests ADD UNIQUE (request_id); INSERT INTO requests (request_id, ...) VALUES ('uuid-here', ...); -- Second identical insert fails at the constraint level — safe to retry ``` 2. **Operation identifier propagation:** Is a unique operation ID generated by the client (UUID or hash of request fields) and passed through every hop — HTTP request → message broker → stream processor → database write? 3. **Single-message atomicity for multi-partition operations:** When an operation must affect multiple partitions (e.g., debit account A and credit account B), do not use distributed atomic commit. Instead: - Write the entire request as a single message to a log partition (by request ID) - A stream processor reads the request and emits individual instructions to each partition's stream (with the request ID included) - Downstream processors for each partition deduplicate by request ID - This achieves equivalent correctness to atomic commit without cross-partition coordination 4. **Timeliness vs. integrity separation:** Distinguish what must be linearizable (users must see their own writes immediately) from what requires only integrity (data must not be lost or corrupted). Violations of timeliness are eventually consistent; violations of integrity are permanent corruption. - **Timeliness failures:** the user sees stale data temporarily → recover by waiting - **Integrity failures:** data is lost, double-charged, or corrupted → cannot self-heal Design the architecture so integrity is maintained in all cases, even if timeliness guarantees are weak. 5. **Loose constraint enforcement:** Not every uniqueness constraint requires synchronous linearizable enforcement. Evaluate whether the business cost of a constraint violation is recoverable: - If two users claim the same username concurrently and one succeeds, the other gets a rejection — recoverable - If a financial transaction is applied twice to an account — not recoverable For recoverable constraints, asynchronous detection and compensation (apology workflow) may be sufficient, enabling coordination-avoiding architectures with better availability and performance. **WHY:** This is the most critical step and the most commonly skipped. Engineers assume that because their database provides transactions and their message broker provides exactly-once delivery, their system is correct. The end-to-end argument shows this is false: each layer handles its own scope, but correctness across the entire request path — from client to database — requires an explicit end-to-end mechanism. The operation identifier is the minimal such mechanism. --- ### Step 8: Produce the Integration Architecture Document **ACTION:** Write an `integration-architecture.md` with three sections: **Section 1: Component Map** List every data store with its role (system of record vs. derived view), the entity types it owns or serves, and the access patterns it satisfies. ``` [System of Record] PostgreSQL — users, orders, products Access: OLTP reads/writes, foreign key enforcement [Derived Views — propagated via Kafka CDC] Elasticsearch — full-text search on products and orders Snowflake — orders_fact, products_fact for OLAP analytics Redis — user session cache, hot product cache [Processing Layer] Kafka — event log (ordered message delivery, 7-day retention) Debezium — CDC from PostgreSQL to Kafka Apache Flink — stream processor (CDC → Elasticsearch, Snowflake) ``` **Section 2: Data Flow Diagram** For each entity type, show the write path (how new data enters) and the read path (how consumers access the derived view). ``` Write Path: User action → Application server → PostgreSQL (system of record) → Debezium reads WAL → Kafka topic (ordered) → Flink consumer → Elasticsearch (search index) → Flink consumer → Snowflake (analytics) Read Path: Full-text search → Elasticsearch OLAP query → Snowflake OLTP query → PostgreSQL Session lookup → Redis (populated from PostgreSQL on login) ``` **Section 3: Correctness Analysis** Document the ordering guarantees, idempotency mechanisms, and constraint enforcement strategy for each critical data flow. --- ## Examples ### Example 1: E-commerce Platform — OLTP + Search + Analytics **Scenario:** Online retailer with PostgreSQL for orders/products, Elasticsearch for product search, Snowflake for business analytics. Current architecture uses dual writes from application code. Products sometimes appear in search before inventory is updated; analytics dashboards occasionally show orders that do not exist in PostgreSQL. **Trigger:** "Our Elasticsearch and PostgreSQL are drifting. Products show in search that are out of stock. How do we fix the architecture?" **Process:** - Step 1 identifies dual writes as the anti-pattern causing drift - Step 2 designates PostgreSQL as system of record for products, orders, inventory - Step 3 selects CDC (Debezium) from PostgreSQL WAL → Kafka → Elasticsearch/Snowflake consumers - Step 6 confirms that per-product ordering (route by product_id partition) is sufficient; total ordering across all products is not required - Step 7 verifies Flink consumers are idempotent (upsert by product_id handles redelivery) **Output:** Replace dual writes with Debezium CDC pipeline. Elasticsearch and Snowflake become read-only derived views, populated exclusively from the Kafka event log. Drift is eliminated because both stores process the same ordered event sequence from the same source. --- ### Example 2: Financial Services — Multi-Partition Transfer with End-to-End Correctness **Scenario:** Payment processing system where transferring money requires debiting one account (partition A) and crediting another (partition B). Current implementation uses two-phase commit across partitions; this causes availability problems when the coordinator fails. **Trigger:** "We're using 2PC for cross-account transfers and it's killing our availability. How do we redesign this?" **Process:** - Step 3 selects event sourcing: log the transfer request as a single message - Step 6 identifies that per-request ordering (route by request_id) is needed to prevent duplicate application - Step 7 applies the multi-partition correctness pattern: 1. Client generates UUID request_id; application appends transfer request to Kafka (keyed by request_id) 2. Stream processor reads request; emits debit instruction to partition A's stream (with request_id) and credit instruction to partition B's stream (with request_id) 3. Account processors for A and B each deduplicate by request_id using a unique constraint on a requests table - If the stream processor crashes and reprocesses the request, it produces identical debit/credit instructions; the unique constraint suppresses the duplicates **Output:** Remove 2PC. Achieve equivalent correctness (every transfer applied exactly once to both accounts) without cross-partition coordination. Availability improves because no coordinator failure mode exists. --- ### Example 3: Social Platform — Causal Ordering Across Services **Scenario:** Social network with friendship status stored in service A, notification delivery in service B. Users report receiving notifications from people they have unfriended — the unfriend event and the message-send event are processed in the wrong order. **Trigger:** "Users are getting messages from people they unfriended. The unfriend event seems to arrive after the message sometimes." **Process:** - Step 1 identifies that friendship and messaging are independent event streams; no total order exists between them - Step 6 identifies a causal dependency: the message-send event causally depends on the friendship status the sender observed - Fix: When the sender sends a message, log the event identifier of the most recent friendship-status read (the state the sender saw). The notification service checks this event ID; if it has not yet processed past that event in the friendship log, it defers the notification. - Alternative: Route all friendship and messaging events through a single Kafka topic partitioned by the pair of user IDs, imposing a per-pair total order **Output:** Causal dependency captured via event identifiers. Notification service becomes causally consistent without requiring total ordering across all users. --- ## References - [Architecture patterns: federated databases vs. unbundled databases](references/unbundling-patterns.md) - [Lambda vs. Kappa architecture decision guide](references/lambda-kappa-decision.md) - [End-to-end correctness and operation identifier implementation](references/end-to-end-correctness.md) - [Timeliness vs. integrity: distinguishing the two dimensions of consistency](references/timeliness-integrity.md) **Cross-skill references:** - `batch-pipeline-designer` — design the batch reprocessing layer (historical data, schema migrations) - `stream-processing-designer` — design the stream propagation layer (CDC, event sourcing, window types, join types) - `consistency-model-selector` — determine per-operation consistency requirements; distinguish linearizability from serializability; select consensus mechanisms where needed - `replication-strategy-selector` — determine the replication topology of the primary system of record - `transaction-isolation-selector` — evaluate isolation requirements for the system of record's OLTP workload - `distributed-failure-analyzer` — diagnose correctness failures in the existing integration **Source:** Designing Data-Intensive Applications, Martin Kleppmann (O'Reilly, 2017), Chapter 12: The Future of Data Systems, pp. 489–544. ## License This skill is licensed under [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). Source: [BookForge](https://github.com/bookforge-ai/bookforge-skills) — Designing Data-Intensive Applications by Martin Kleppmann. ## Related BookForge Skills Install related skills from ClawhHub: - `clawhub install bookforge-batch-pipeline-designer` - `clawhub install bookforge-stream-processing-designer` - `clawhub install bookforge-consistency-model-selector` - `clawhub install bookforge-replication-strategy-selector` Or install the full book set from GitHub: [bookforge-skills](https://github.com/bookforge-ai/bookforge-skills)

path: references/end-to-end-correctness.md
content: # End-to-End Correctness and Operation Identifier Implementation

## The End-to-End Argument

The end-to-end argument (Saltzer, Reed, and Clark, 1984) states:

> The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the endpoints of the communication system. Therefore, providing that questioned function as a feature of the communication system itself is not possible.

Applied to data systems: **correctness guarantees provided by individual layers — TCP, database transactions, stream processor exactly-once — cannot, by themselves, guarantee correctness across the entire request path from client to database.**

### The Concrete Problem

A user submits a bank transfer via a web form:

1. Browser sends HTTP POST to application server
2. Application server opens a database transaction
3. Transaction updates two account balances
4. Transaction commits (ACID guarantee: atomically)
5. Database sends success response to application server
6. Application server sends 200 OK to browser

**What if the network fails at step 6?** The transaction committed in step 4, but the browser never received the success response. The browser shows an error. The user clicks "Submit" again. Now:

- TCP duplicate suppression: does not help — this is a new TCP connection
- Database transaction: does not help — this is a new transaction; the database does not know it is a retry
- 2PC: does not help — it handles coordinator failure during the commit protocol, not client-level retries after commit

The transfer executes twice. The user is charged $22 instead of $11.

**Real banks do not work this way.** Real banks use end-to-end idempotency: each transfer request carries a unique identifier that is stored in the database. A retry of the same request is detected and rejected at the application level.

## The Operation Identifier Pattern

### Structure

Every mutating operation that must be idempotent end-to-end needs a unique operation identifier:

1. **Generated at the client** — not at the application server. The client generates the ID before the first attempt and reuses it on every retry.
2. **Derived deterministically** — either a UUID (random, globally unique) or a hash of the request's meaningful fields (from_account, to_account, amount, initiating_session). Deterministic derivation allows reconstruction if the client forgets the ID.
3. **Passed through every hop** — HTTP header, message broker key, stream processor message field, database column. The ID is never dropped or regenerated at any intermediate layer.
4. **Enforced at the final storage layer** — a unique constraint on the operation ID in the database prevents duplicate execution, even under concurrent retries.

### Implementation

```sql
-- Schema: operation ID as a unique constraint
CREATE TABLE money_transfers (
    request_id    UUID        PRIMARY KEY,
    from_account  BIGINT      NOT NULL,
    to_account    BIGINT      NOT NULL,
    amount        DECIMAL     NOT NULL,
    initiated_at  TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

-- The unique constraint on request_id makes concurrent retries safe
-- even at weak isolation levels (read committed, snapshot isolation)
-- because uniqueness constraints are enforced correctly at all isolation levels
-- in most relational databases
```

```sql
-- Execution: idempotent transfer using operation identifier
BEGIN TRANSACTION;

-- Insert the transfer request — fails with a unique violation if already processed
INSERT INTO money_transfers (request_id, from_account, to_account, amount)
VALUES ('0286FDB8-D7E1-423F-B40B-792B3608036C', 4321, 1234, 11.00)
ON CONFLICT (request_id) DO NOTHING;  -- Idempotent: second attempt is a no-op

-- Conditionally apply the debit/credit only if the insert succeeded
-- (i.e., this is the first time this request_id has been processed)
UPDATE accounts SET balance = balance + 11.00 WHERE account_id = 1234
  AND EXISTS (SELECT 1 FROM money_transfers WHERE request_id = '0286FDB8-D7E1-423F-B40B-792B3608036C');

UPDATE accounts SET balance = balance - 11.00 WHERE account_id = 4321
  AND EXISTS (SELECT 1 FROM money_transfers WHERE request_id = '0286FDB8-D7E1-423F-B40B-792B3608036C');

COMMIT;
```

**Note on isolation level:** The unique constraint on `request_id` works correctly even at read committed isolation, because uniqueness constraints are enforced at a lower level than MVCC snapshot visibility. A concurrent insert of the same `request_id` will either serialize (one succeeds, one fails) or abort. The `ON CONFLICT DO NOTHING` clause makes the retry a safe no-op.

### The requests Table as an Event Log

The `money_transfers` table in the pattern above functions as an event sourcing log for transfer requests. The account balance updates do not need to happen in the same transaction as the transfer request insertion — they can be applied asynchronously by a downstream consumer that reads the `money_transfers` table as an event log, because:

- The request is durably recorded (the INSERT committed)
- The downstream consumer processes each request exactly once (it deduplicates by `request_id`)
- The downstream consumer is idempotent (applying the same transfer twice is prevented by the deduplication check)

This is the event sourcing pattern: record the intent durably first; derive the effects asynchronously.

## Multi-Partition Correctness Without Distributed Transactions

When an operation spans multiple partitions — for example, debiting account A (in partition 1) and crediting account B (in partition 2) — distributed atomic commit (2PC) is the traditional solution. 2PC achieves atomicity but has high cost: blocking during coordinator failure, heterogeneous protocol requirements, poor fault tolerance.

The alternative achieves equivalent correctness without cross-partition coordination:

### The Pattern

**Step 1:** Client generates a unique `request_id`. Application server appends a single transfer-request message to a Kafka topic partitioned by `request_id`. The message contains `{request_id, from_account, to_account, amount}`. This is a single-object write — atomic in almost all message brokers.

**Step 2:** A stream processor reads the transfer-request log. For each message, it emits two derived messages:
- A debit instruction to the `account-debits` topic, partitioned by `from_account`, containing `{request_id, from_account, amount}`
- A credit instruction to the `account-credits` topic, partitioned by `to_account`, containing `{request_id, to_account, amount}`

If the stream processor crashes and replays from its last checkpoint, it re-emits the same debit and credit instructions. Because the derivation is deterministic and the `request_id` is preserved, the downstream processors can safely deduplicate.

**Step 3:** Separate account processors for debits and credits each consume their respective topics. They maintain a deduplication table keyed by `request_id`. For each message:
- If `request_id` already in the deduplication table → skip (already applied)
- Otherwise → apply the debit/credit and record the `request_id`

**Why this achieves the same correctness as 2PC:**
- Every valid transfer request is applied exactly once to both the payer and payee accounts
- This holds even in the presence of stream processor crashes and message redelivery
- No cross-partition coordination is required at any step
- A failure of the debit processor does not affect the credit processor (they are independent consumers)

**The key insight:** Distributing the operation into two separately partitioned stages, connected by a durably logged request, avoids the need for atomicity across partitions. The `request_id` propagated through all stages is the mechanism that ties the distributed operation together and ensures exactly-once application.

## Timeliness vs. Integrity

These are two distinct dimensions of correctness that are conflated by the term "consistency":

**Timeliness:** Users observe the system in an up-to-date state. A timeliness violation means a user reads stale data — they see a version of the state that has since been updated. Timeliness violations are **temporary**: waiting and retrying will eventually show the correct state (assuming eventual consistency).

**Integrity:** No data is lost and no contradictory or false data exists. An integrity violation means the data is permanently corrupted — a record is missing, an account balance is wrong, a derived view contains a record that does not exist in the source. Integrity violations are **permanent**: waiting and retrying does not fix database corruption.

| Property | Violation | Recovery | Example |
|----------|-----------|----------|---------|
| Timeliness | Stale read | Wait and retry | User's profile update not yet visible in search index |
| Integrity | Data corruption | Manual repair | Account charged twice; search index contains records deleted from the source |

**Design principle:** In most applications, integrity is far more important than timeliness. Design the integration architecture to preserve integrity unconditionally — even at the cost of weak timeliness guarantees. Stale data is annoying; corrupted data is catastrophic.

The event log + idempotent consumer pattern preserves integrity while accepting weak timeliness: derived views will eventually be consistent with the source of record (when the consumer catches up), and the derivation will not corrupt data even if it runs multiple times.

## Auditing and Self-Verification

The end-to-end argument implies that end-to-end integrity checks are valuable. Because each individual layer assumes the layers above and below it are working correctly, bugs in any layer can produce corruption that only manifests at the application level.

**Useful audit patterns:**

1. **Periodic reconciliation:** Count or sum records in the source of record and each derived view. Discrepancies indicate lost events or failed derivations.

2. **Event log as audit trail:** If the event log is immutable and append-only, it provides a complete, replayable record of all writes. Any derived view can be verified by replaying the log and comparing the output to the current state.

3. **Deterministic reprocessing as verification:** Run the derivation function twice on the same input. If the outputs differ, the function is non-deterministic (a bug). If the output differs from the stored derived view, events were lost or applied out of order.

4. **Cryptographic integrity:** Hash the event log periodically and store the hashes. A mismatch indicates tampering or silent corruption. This is the principle behind Merkle trees used in certificate transparency systems.


path: references/lambda-kappa-decision.md
content: # Lambda vs. Kappa Architecture Decision Guide

## The Problem Both Architectures Solve

When batch processing is used to reprocess historical data (producing accurate, complete views) and stream processing is used to handle recent events (producing low-latency, approximate views), you need to decide how to combine them. Lambda and Kappa are the two main architectural patterns for this decision.

The underlying principle they share: **incoming data should be recorded as an immutable append-only log of events. Read-optimized views are derived from this log.** The difference is in how many systems derive those views.

## Lambda Architecture

### Structure

```
Incoming events
     │
     ├──► [Batch Layer]  ──────────────────────────────────────────────►  Batch views (accurate)
     │     Hadoop MapReduce / Spark                                              │
     │     Reprocesses all historical data                                       │
     │     Produces corrected, complete views                                    ▼
     │                                                               [Serving Layer]  ──► Query response
     └──► [Speed Layer]  ──────────────────────────────────────────►  Real-time views
          Apache Storm / Kafka Streams                                 (approximate)
          Processes recent events quickly
          Produces approximate, low-latency views
```

### Core Idea

- The batch layer is authoritative and simple: it reprocesses everything, so there is no state to manage, no fault tolerance to implement beyond restarting the job, and no incremental complexity.
- The speed layer is fast but imprecise: it handles the lag between the last batch run and now, accepting that its output may have small errors.
- Queries merge results from both layers at read time.

### When Lambda Makes Sense

- The batch processing logic is significantly simpler and less bug-prone than the equivalent stream processing logic.
- The team's stream processing infrastructure is less mature or reliable than their batch infrastructure.
- The workload uses approximate algorithms in the speed layer (e.g., HyperLogLog cardinality estimates, sketches) that are fast but slightly inaccurate, while the batch layer computes exact results.

### Practical Problems with Lambda

1. **Dual codebase burden.** The same business logic must be implemented in two different frameworks — the batch framework (Spark, MapReduce) and the streaming framework (Storm, Flink, Kafka Streams). These frameworks have very different APIs, semantics, and operational models. Any change to the logic requires updating both implementations.

   Libraries like Apache Beam or Summingbird attempt to abstract this by providing a single API that compiles to both batch and streaming backends, but debugging, tuning, and operational behavior still differ between the two runtimes.

2. **Merging batch and stream outputs is non-trivial.** Merging a simple aggregation (sum, count) over a tumbling window is straightforward. Merging session-based aggregations, joins, or stateful computations is much harder. The serving layer must implement query-time merge logic that mirrors the derivation logic — a third implementation of the same business logic.

3. **Batch layer ends up doing incremental processing.** Reprocessing all historical data from scratch on every batch run is expensive at scale. The batch layer is usually configured to process only recent data (e.g., the last hour), which means it faces the same windowing and late-data problems as the stream layer — negating its simplicity advantage.

4. **Two systems to operate.** Maintaining two independent distributed systems (a batch cluster and a streaming cluster) doubles infrastructure and operational complexity.

## Kappa Architecture

### Structure

```
Incoming events (immutable log, long retention)
     │
     └──► [Stream Processing Layer]  ──────────────────────────────────► Views
          Apache Flink / Apache Beam / Kafka Streams
          Processes both historical (replay) and recent events
          Produces accurate views via exactly-once semantics
```

### Core Idea

Unify batch and stream processing in a single system by treating batch jobs as a special case of stream processing: a job over a bounded (finite) stream. The stream processor reads from the beginning of the log for historical reprocessing, and from the current offset for ongoing processing. No separate batch layer is needed.

### Required Capabilities for Kappa

For Kappa to work correctly, three capabilities are needed:

1. **Log-based broker with configurable retention and replay.** The message broker must retain events long enough to replay all history needed for reprocessing. Kafka's configurable log retention (time-based or size-based) satisfies this. The stream processor reads from offset 0 to reprocess history, then continues from the current offset.

2. **Exactly-once semantics in the stream processor.** The output of a reprocessing run must be the same as if no faults had occurred. This requires: (a) exactly-once message delivery within the processor, (b) idempotent output writes, or (c) transactional output commits. Apache Flink's checkpointing with transactional sinks achieves this.

3. **Event-time windowing.** When reprocessing historical data, processing-time timestamps are meaningless (the job runs now, but the events are from the past). The stream processor must window on the event's original timestamp, not on when the processor handles it. Apache Beam and Flink both support event-time windowing with watermarks for handling late-arriving events.

### When Kappa Wins

- A mature stream processor with the three required capabilities (replay, exactly-once, event-time) is available and the team can operate it.
- Business logic is complex enough that maintaining two implementations is a significant burden.
- Schema evolution or business logic changes are frequent (each change requires reprocessing the log; with Kappa, this is one reprocessing job, not a coordinated update to two systems).

### When Lambda May Still Be Preferred

- The stream processor available to the team lacks replay capability or exactly-once semantics.
- The batch computation is a genuinely simpler algorithm than the equivalent streaming computation (e.g., sorting a bounded dataset vs. approximate sorting in a stream).
- The team has significantly more operational expertise in batch systems than streaming systems.

## The Convergence Trend

Modern processing engines are converging toward the Kappa model:

- **Apache Spark:** Originally batch-only (MapReduce replacement). Added Structured Streaming as a micro-batch stream processor. Spark can read Kafka topics as a stream or as a bounded dataset for batch — same API.
- **Apache Flink:** Originally a stream processor. Added DataSet API for batch processing. Flink 1.12+ unified both under a single streaming model: batch is a bounded stream.
- **Apache Beam:** Provides a single portable API for both batch and streaming. Backends include Flink, Dataflow, and Spark.

The practical consequence: if the team uses Flink or Beam, the Lambda vs. Kappa choice largely dissolves — the same code runs in both modes.

## Decision Matrix

| Criterion | Lambda | Kappa |
|-----------|--------|-------|
| Separate batch and stream codebases | Required | Not needed |
| Exactly-once stream semantics required | No (batch corrects errors) | Yes |
| Log replay capability required | No | Yes (long retention) |
| Query-time merge of batch + stream outputs | Required | Not needed |
| Appropriate when stream processor is immature | Yes | No |
| Appropriate when team has high streaming maturity | Less relevant | Yes |
| Schema evolution / logic change cost | High (update 2 codebases + reprocess) | Lower (one reprocessing job) |
| Operational complexity | High (2 clusters) | Medium (1 cluster) |

## Recommendation

**Default to Kappa** if the team can operate a stream processor with replay, exactly-once, and event-time capabilities (Flink, Beam on Dataflow, or Kafka Streams with Kafka log retention).

**Use Lambda** only if the stream processing capability is genuinely immature relative to the batch capability, or if the batch algorithms being used are fundamentally simpler than their streaming equivalents (which is increasingly rare).

**Use a unified batch/stream system** (Flink, Spark with Structured Streaming, Beam) to make the choice irrelevant.


path: references/timeliness-integrity.md
content: # Timeliness vs. Integrity: Two Dimensions of Consistency

## The Core Distinction

The term "consistency" in distributed systems conflates two properties that are worth separating:

**Timeliness** — ensuring that users observe the system in an up-to-date state. If data has been written, any subsequent read should return the new value, not a stale version.

**Integrity** — ensuring the absence of corruption; no data loss, no contradictory or false data. If a derived dataset is maintained as a view onto underlying data, the derivation must be correct and complete.

### The Slogan

- Violations of timeliness are **"eventual consistency"** — the system will catch up.
- Violations of integrity are **"perpetual inconsistency"** — the corruption is permanent.

## Why the Distinction Matters for Data Integration

### Traditional Transactions Conflate Both

ACID transactions provide both timeliness (linearizability — a committed write is immediately visible to all subsequent reads) and integrity (atomicity and durability — a write either commits completely or rolls back completely, with no partial effects).

Because transactions provide both together, architects using transactional systems rarely need to distinguish them. This creates a false assumption: that any system providing "consistency" must provide both simultaneously.

### Asynchronous Log-Based Systems Decouple Them

Event-log-based derived data systems (CDC, event sourcing, stream processing) are designed to preserve integrity while accepting weak timeliness.

- **Integrity is central:** Exactly-once semantics, idempotent consumers, and operation identifiers ensure that every event is applied exactly once to every derived view. No data is lost; no event is applied twice.
- **Timeliness is weak:** Derived views lag behind the source of record by the propagation latency (milliseconds to minutes, depending on the system). A read from the derived view may return stale data.

This is not a bug — it is a deliberate design trade-off that enables better performance, availability, and fault isolation than synchronous distributed transactions.

## Practical Implications

### What Integrity Requires in a Composed Pipeline

1. **Exactly-once event delivery to each consumer.** An event that is lost causes the derived view to permanently miss that update. An event that is applied twice may corrupt the derived view (unless the consumer is idempotent).

2. **Correct ordering where ordering matters.** If two events must be applied in a specific order (a delete must not precede the record's creation), the consumer must receive them in that order or detect and handle out-of-order delivery.

3. **Idempotent consumers or deduplication.** Because message delivery may provide at-least-once semantics (duplicates possible), consumers must be idempotent (applying the same event twice produces the same result) or must deduplicate by event ID.

4. **Durable event log.** If the event log is lost, derived views cannot be reconstructed. The log must be replicated and retained for at least as long as any consumer might need to replay it.

### What Timeliness Requires

1. **Low propagation latency.** The gap between a write to the source of record and its visibility in derived views must be small enough for the use case (sub-second for real-time dashboards; minutes may be acceptable for analytics).

2. **Read-your-own-writes consistency** (if required). A user who updates their profile and immediately views the result should see their new profile, not the pre-update version. This requires either routing the user's reads to the system of record (not the derived view) for a period after the write, or using causal consistency mechanisms.

3. **Causal consistency** (if required). If Event B causally depends on Event A (B happened because of A), any user who observes B should also observe A. See the causal ordering section of the main skill.

### The Trade-Off Is Asymmetric

In most applications, integrity violations are far more costly than timeliness violations:

| Scenario | Timeliness violation | Integrity violation |
|----------|---------------------|---------------------|
| E-commerce order | Product shows as available for 30s after selling out | Customer is charged twice for the same order |
| Bank account | Balance shows yesterday's value for 500ms | $11 is debited but only $5.50 is credited |
| Social profile | Updated bio visible to some users 2s before others | Profile update is permanently lost |
| Search index | New document not searchable for 10s | Deleted document remains in search results forever |

Timeliness violations are annoying and recoverable. Integrity violations are catastrophic and may be unrecoverable without manual intervention.

## Coordination-Avoiding Architectures

An important corollary: if integrity is more important than timeliness, and integrity can be maintained without synchronous cross-system coordination, then coordination-avoiding architectures are strictly better for most workloads.

**Coordination-avoiding systems** maintain integrity guarantees on derived data without requiring:
- Atomic commit across multiple stores
- Linearizability (synchronous coordination for recency guarantee)
- Synchronous cross-partition or cross-region coordination

They achieve this through:
- Deterministic derivation functions (same input → same output, always)
- Exactly-once processing (idempotent consumers, operation identifiers)
- Immutable event logs (replay reconstructs any derived view)

A coordination-avoiding system can operate across multiple datacenters in a multi-leader configuration, with weak timeliness guarantees, while still maintaining strong integrity guarantees. This is the sweet spot for most large-scale data integration scenarios.

## Loose Constraint Enforcement

Not every constraint must be enforced synchronously. Many business constraints can be violated temporarily and repaired after the fact ("apologize later").

### Hard Constraints (must be synchronous)
- Double-charging a payment card
- Permanently deleting data that a user wants back
- Any operation that cannot be reversed or compensated

### Soft Constraints (can be eventual or compensating)
- Two users claiming the same username simultaneously — apologize to the loser; ask them to choose another
- Overselling a product — apologize to the excess purchasers; offer a refund or backorder
- Overbooking an airline seat — apologize; offer compensation; bump a passenger

The cost of the apology (money, reputation, customer service effort) determines whether the constraint must be enforced synchronously. If the apology cost is low and the apology workflow must exist anyway (for other reasons like fraud, cancellations, etc.), strict synchronous enforcement may be unnecessary.

**The key question:** "What is the cost of temporarily violating this constraint, and can we detect and repair the violation before it causes irreversible harm?"

If the answer is "low cost, yes we can detect and repair" → soft constraint, coordination-avoiding enforcement is sufficient.
If the answer is "high cost or irreversible" → hard constraint, synchronous enforcement required.

## Applying This in Practice

When designing a data integration architecture, apply this framework to each constraint:

1. Identify the constraint (uniqueness, referential integrity, balance non-negativity, etc.)
2. Classify: hard (synchronous enforcement required) vs. soft (can be eventually enforced with compensation)
3. For hard constraints: design the partition routing to ensure conflicts go to a single partition, enabling a stream processor to enforce the constraint without distributed coordination
4. For soft constraints: design the async detection and compensation workflow; keep the write path fast and coordination-free
5. Ensure integrity (no data loss, no double-application) in all cases, regardless of timeliness classification


path: references/unbundling-patterns.md
content: # Unbundling Patterns: Federated Databases vs. Composed Pipelines

## The Core Analogy

A relational database internally maintains several subsystems:
- **Secondary indexes** — allow efficient lookup by non-primary-key fields
- **Materialized views** — precomputed query results stored for fast read access
- **Replication logs** — propagate changes to follower replicas
- **Full-text indexes** — built into some databases (PostgreSQL tsvector, MySQL FULLTEXT)
- **Query optimizer** — selects efficient execution plans across indexes and joins

When you compose specialized systems — a primary OLTP database, a search index, an analytics warehouse, a feature store — you are implementing these same subsystems outside the database, as independent services connected by an event log. The event log plays the role of the replication log.

This is the "unbundling databases" or "database inside-out" approach: taking features that are built into databases as tightly coupled subsystems and implementing them as loosely coupled independent services.

## Two Approaches to Composition

### Federated Databases (Unified Reads)

A federated database (or polystore) provides a unified query interface over multiple underlying storage engines. Users query a single endpoint; the federated layer routes subqueries to the appropriate backend and merges results.

**Examples:** PostgreSQL Foreign Data Wrappers (FDW), BigQuery federated queries, Presto/Trino.

**Strengths:**
- Single query language for users
- No data movement required for read-only analytical queries
- Applications that need a specialized data model still have direct access to their native storage

**Weaknesses:**
- Query optimization across heterogeneous engines is complex
- No good answer to synchronizing writes across backends — reads are unified, writes are not
- Performance depends on the least-capable backend for cross-store queries

**When to use:** When you need unified read access to existing independent stores and can tolerate the write-synchronization problem being handled separately.

### Composed Pipelines (Unified Writes)

The composed pipeline approach focuses on write synchronization: one system is the source of truth; all others are derived. The event log synchronizes writes deterministically.

**The analogy to Unix pipes:**
```
# Unix: small tools, uniform interface (byte streams), composable
cat access.log | grep "ERROR" | awk '{print $1}' | sort | uniq -c

# Data systems: small tools, uniform interface (event log), composable
PostgreSQL → Debezium CDC → Kafka → Flink → Elasticsearch
                                          → Snowflake
                                          → Redis
```

Each tool does one thing well. The event log is the pipe. Application code is the shell script that wires them together.

**Strengths:**
- Loose coupling: a fault in Elasticsearch does not affect PostgreSQL or Snowflake
- Each store is optimized for its specific access pattern
- Historical reprocessing: replay the event log to rebuild any derived view
- Independent team ownership: each team operates their downstream consumer independently

**Weaknesses:**
- Operational complexity: more systems to deploy, monitor, and upgrade
- Asynchronous propagation: derived views lag behind the source of truth
- No standardized "pipe" yet: unlike Unix pipes, there is no universal protocol for composing storage systems

## Creating an Index as the Template

The `CREATE INDEX` operation in a relational database illustrates the pattern:

1. The database takes a consistent snapshot of the table
2. It scans all rows, extracts the indexed field values, sorts them, and writes the index structure
3. It applies any writes that arrived during the scan from the replication backlog
4. It continues maintaining the index on every subsequent write

This is exactly the bootstrap pattern for setting up a new derived view in a composed pipeline:
1. Take a consistent snapshot of the source of record (initial export)
2. Apply the snapshot to the derived store
3. Switch to consuming the ongoing event log from the point of the snapshot
4. Continue consuming the log indefinitely

The difference is that `CREATE INDEX` is built into the database and runs automatically. In a composed pipeline, you implement this bootstrap manually — but the logic is identical.

## The Missing Piece

Unix has the shell as a high-level language for composing tools via pipes. Data systems do not yet have an equivalent. The vision:

```
# Hypothetical declarative composition
mysql | elasticsearch  # = CREATE INDEX in MySQL, continuously replicated to Elasticsearch
```

This would mean: take all documents in MySQL, index them in Elasticsearch, and continuously apply all future changes. Tools like Debezium + Kafka + a stream processor approximate this, but with significantly more configuration and operational overhead than a pipe operator.

## When a Single System Wins

The goal of unbundling is breadth — serving more access patterns than any single system can. It is not a goal in itself.

**Use a single system when:**
- One technology satisfies all access patterns at the required scale
- The marginal benefit of specialized stores does not justify the operational cost
- The team does not have capacity to operate multiple systems reliably

"If there is a single technology that does everything you need, you're most likely best off simply using that product rather than trying to reimplement it yourself from lower-level components." — Kleppmann, Ch. 12

Premature unbundling — composing systems before the single-system ceiling is reached — adds complexity without benefit and may lock the team into an inflexible design.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)