Skill: quochungto/bookforge-partitioning-strategy-advisor (version 1.0.0)

--- name: partitioning-strategy-advisor description: | Select the right partitioning (sharding) strategy — range, hash, or compound key — and configure secondary indexes, rebalancing, and request routing for a distributed database. Use when: designing a partition key for a new system; diagnosing write hotspots on monotonically increasing keys (timestamps, auto-increment IDs); evaluating whether an existing sharding scheme supports required query patterns; choosing between document-partitioned (local) vs. term-partitioned (global) secondary indexes and weighing scatter/gather read costs against global index write amplification; or selecting a rebalancing approach (fixed partitions, dynamic partitions, proportional-to-nodes) and routing topology (gossip, ZooKeeper coordination, partition-aware client). Covers Cassandra compound primary key patterns for range queries within hash-distributed partitions, HBase/SSTables range partitioning, Riak consistent hashing, and MongoDB/Elasticsearch index partitioning. Distinct from replication-strategy-selector (topology and consistency) and data-model-selector (schema design). Produces a concrete recommendation: partition key, partitioning method, secondary index approach, rebalancing configuration, and routing topology. Depends on data-model-selector for schema and access pattern context. version: 1.0.0 homepage: https://github.com/bookforge-ai/bookforge-skills/tree/main/books/designing-data-intensive-applications/skills/partitioning-strategy-advisor metadata: {"openclaw":{"emoji":"📚","homepage":"https://github.com/bookforge-ai/bookforge-skills"}} status: draft depends-on: - data-model-selector source-books: - id: designing-data-intensive-applications title: "Designing Data-Intensive Applications" authors: ["Martin Kleppmann"] chapters: [6] tags: - partitioning - sharding - range-partitioning - hash-partitioning - compound-key - hotspot - skew - secondary-index - local-index - global-index - document-partitioned-index - term-partitioned-index - scatter-gather - rebalancing - dynamic-partitioning - fixed-partitions - proportional-partitioning - consistent-hashing - request-routing - zookeeper - gossip-protocol - cassandra - hbase - mongodb - riak - elasticsearch execution: tier: 2 mode: hybrid inputs: - type: codebase description: "Application codebase, docker-compose, database schema files, or architecture description that reveals data access patterns, query patterns, and write volume characteristics" - type: document description: "System requirements document or architecture description if no codebase is available" tools-required: [Read, Write, Bash] tools-optional: [Grep] mcps-required: [] environment: "Run inside a project directory with codebase or configuration files. Falls back to document/description input." discovery: goal: "Produce a concrete partitioning recommendation: partition key, partitioning method, secondary index approach, rebalancing configuration, and request routing topology" tasks: - "Identify the primary data entity and its natural access key" - "Classify the data distribution risk (monotonic keys, celebrity keys, uniform keys)" - "Select partitioning method: range, hash, or compound" - "Assess secondary index requirements and select local vs. global indexing" - "Select rebalancing approach: fixed, dynamic, or proportional-to-nodes" - "Select request routing approach: gossip, coordination service, or partition-aware client" - "Document hotspot risks and mitigation strategies" audience: roles: ["backend-engineer", "software-architect", "data-engineer", "site-reliability-engineer", "tech-lead"] experience: "intermediate-to-advanced — assumes experience with distributed databases and data modeling" triggers: - "User is designing a partitioning scheme for a new distributed database deployment" - "User is experiencing write hotspots or uneven partition load" - "User needs efficient range queries on a hash-partitioned database" - "User is choosing between local and global secondary indexes" - "User is configuring rebalancing for a growing cluster" - "User is evaluating whether their sharding key causes skew" - "User is deciding how to route requests to the correct partition" not_for: - "Choosing a replication topology — use replication-strategy-selector" - "Selecting a storage engine or data model — use storage-engine-selector or data-model-selector" - "Diagnosing distributed system failures — use distributed-failure-analyzer" --- # Partitioning Strategy Advisor ## When to Use You are designing or evaluating a distributed database where data must be spread across multiple nodes (sharding) and need to select a partitioning key, a partitioning method, secondary index approach, rebalancing configuration, and request routing topology. This skill applies when the partitioning scheme is open (new system), problematic (existing system with hotspots or unsupported query patterns), or needs documented justification (architecture decision record, team alignment). It produces a concrete recommendation covering partition key choice, range vs. hash vs. compound method, secondary index strategy, rebalancing approach, and routing topology. **Prerequisite check:** - If you haven't selected a data model (relational, document, graph) for your system, run `data-model-selector` first — partitioning key design depends on the schema and access patterns that the data model determines. - If you are also designing replication across nodes, run `replication-strategy-selector` after this skill — partitioning and replication are largely independent choices, but the replication factor affects partition sizing. - If you need to diagnose hotspots or routing failures in a live production system, use `distributed-failure-analyzer` for root-cause analysis after using this skill to select the corrected strategy. --- ## Context & Input Gathering ### Required Context (must have — ask if missing) **1. Primary access pattern — how records are most frequently looked up** Why: The access pattern determines what the partition key must be. If records are always looked up by a single primary key (user ID, order ID), hashing that key achieves even distribution. If records are frequently fetched in sorted ranges (sensor readings by time, events by date), range partitioning on that key enables efficient range scans. Choosing the wrong key forces scatter/gather reads or hot single partitions on every query. - Check prompt for: "look up by", "query by", "fetch all records for", "time-series", "range query", "primary key is" - Check environment for: schema.sql (PRIMARY KEY, index definitions), application code (WHERE clauses, query methods), architecture.md (data access descriptions) - If still missing, ask: "How does your application most frequently look up records? By a single ID, by a range of values (e.g., date range), or by filtering on secondary attributes (e.g., color, status)?" **2. Write distribution characteristics — is the natural key monotonically increasing?** Why: Monotonically increasing keys (timestamps, auto-increment IDs, sequential order numbers) create sequential write patterns that concentrate all writes on the latest partition in range partitioning. This is the most common hotspot source. Even under hash partitioning, extremely popular individual keys (celebrity users, viral content IDs) can overwhelm a single partition. Understanding write distribution is required before selecting a partition key. - Check prompt for: "timestamp", "created_at", "auto-increment", "sequential ID", "time-series writes", "celebrity", "viral", "high-traffic key" - Check environment for: schema.sql (SERIAL, AUTO_INCREMENT, TIMESTAMP columns as primary keys), application code (INSERT patterns, event ingestion pipelines) - If still missing, ask: "Does your most common write operation use a key that increases sequentially (like a timestamp or auto-increment ID)? And are any individual records likely to receive dramatically more writes than others (e.g., a popular user or trending item)?" **3. Secondary query requirements — does the application filter on non-primary-key attributes?** Why: Secondary indexes on partitioned data require a choice between document-partitioned (local) and term-partitioned (global) indexes. This choice forces a fundamental trade-off: local indexes are cheap to write (only the local partition is updated) but expensive to read (scatter/gather across all partitions). Global indexes are cheap to read (single partition lookup) but expensive to write (updates span multiple partition index entries, often asynchronously). You must know which queries are required before designing the index strategy. - Check prompt for: "filter by", "search by color", "find all users where", "full-text search", "secondary index", "non-primary attribute lookup" - Check environment for: schema.sql (CREATE INDEX, secondary index definitions), application code (multi-attribute WHERE clauses), architecture.md (search or filter requirements) - If still missing, ask: "Does your application need to query records by attributes other than the primary key — for example, filtering orders by status, searching products by category, or looking up users by email?" **4. Data volume and growth trajectory** Why: Affects rebalancing strategy selection. If total data volume is small and known, a fixed number of partitions sized correctly upfront is simplest. If data will grow significantly over time, dynamic partitioning (split/merge) adapts partition count automatically. If the cluster will scale out by adding nodes, proportional-to-nodes partitioning keeps partition size stable. Choosing a fixed partition count that's too small locks out future growth; too large creates excessive overhead. - Check prompt for: "expected data size", "TB", "GB", "millions of records", "growing fast", "scaling out", "adding nodes" - Check environment for: docker-compose (volume mounts, sizing comments), requirements.md (capacity planning), architecture.md (growth estimates) - If still missing, ask: "Roughly how much data will this system store now, and how is it expected to grow over the next 1–2 years? Will you be adding database nodes as it grows?" **5. Target database** Why: Different databases implement partitioning in fundamentally different ways — HBase and RethinkDB use dynamic key-range partitioning; Cassandra and Riak use hash partitioning with proportional-to-nodes rebalancing; Elasticsearch uses fixed-partition hash sharding; MongoDB supports both. The recommendation must fit what the target database actually supports. Additionally, Cassandra's compound primary key pattern is only available in Cassandra. - Check prompt for: database names (Cassandra, HBase, MongoDB, DynamoDB, Riak, Elasticsearch, PostgreSQL with Citus, etc.) - Check environment for: docker-compose (image names), requirements.md (technology constraints), package files (database drivers) - If still missing, ask: "Which database are you partitioning? (e.g., Cassandra, HBase, MongoDB, DynamoDB, Elasticsearch, or a custom partitioning layer on top of another store)" ### Optional Context (enriches recommendation) - **Consistency requirements under rebalancing:** If the system requires zero-downtime rebalancing with continued read/write availability, manual-approval rebalancing (Couchbase, Riak, Voldemort model) is safer than fully automatic. - **Request routing constraints:** If the system cannot depend on an external coordination service like ZooKeeper, gossip-based routing (Cassandra, Riak) or partition-aware client libraries are required. - **Read/write ratio:** Heavily read-dominated workloads benefit more from global (term-partitioned) secondary indexes; heavily write-dominated workloads benefit from local (document-partitioned) secondary indexes. --- ## Process ### Step 1 — Classify Data Distribution Risk Categorize the workload's skew profile before selecting any partitioning method. **Why:** Partitioning is only useful if load is evenly distributed. A poorly chosen partition key can negate all the benefits of horizontal scaling by concentrating all reads or writes on one node (a hotspot). Identifying skew risk upfront determines whether extra mitigation (key prefixing, salting, compound keys) is needed on top of the base partitioning method. Classify into one of three profiles: | Profile | Signal | Risk | |---|---|---| | **Monotonic key** | Timestamps, sequential IDs, auto-increment PKs | All writes go to the single "latest" partition under range partitioning | | **Celebrity key** | A small number of keys receive orders of magnitude more traffic (viral content, celebrity users) | Single partition overloaded even under hash partitioning | | **Uniform key** | Random UUIDs, user IDs with no natural ordering, hashed values | Low hotspot risk; both range and hash partitioning are viable | **Mitigation for monotonic keys (range partitioning):** Prefix the timestamp with a high-cardinality dimension (e.g., sensor name + timestamp, or user ID + timestamp). This distributes writes across the key space while preserving range-scan ability within each prefix bucket. **Mitigation for celebrity keys (hash partitioning):** Append a two-digit random suffix (00–99) to the hot key at write time. This splits writes across 100 partitions. Reads must then query all 100 suffix variants and merge — track which keys are "hot" in a separate lookup table to apply this only where needed. ### Step 2 — Select Partitioning Method Choose range partitioning, hash partitioning, or compound key (hybrid) based on access patterns and skew profile. **Why:** This is the central decision. Range partitioning preserves sort order, enabling efficient range scans but requiring careful key design to avoid hotspots. Hash partitioning destroys sort order (making range queries require scatter/gather) but distributes load evenly. Compound keys (Cassandra model) combine both — hashing on the first part distributes across partitions, sorting within the partition on the remaining parts supports range scans within a single partition. **Decision framework:** ``` PRIMARY ACCESS PATTERN ├── Range queries are critical (fetch records within a time window, │ scan sorted ranges, support range-based pagination) │ ├── Key is monotonically increasing → RANGE with prefix mitigation │ │ Example: (sensor_name, timestamp) compound range key │ └── Key has natural, non-monotonic distribution → RANGE partitioning │ Example: alphabetically distributed last names (encyclopedia volumes) │ ├── Point lookups dominate (fetch a single record by ID) AND │ range queries are not needed │ → HASH partitioning │ Example: user profiles by user_id, order records by order_id │ └── Point lookups on partition key AND range queries within that key → COMPOUND KEY (hash partition key + range sort key) Example: (user_id [hash], update_timestamp [range]) in Cassandra → Efficiently fetch all updates for a user, sorted by time, in a single partition lookup ``` **Range partitioning:** Used by Bigtable, HBase, RethinkDB, MongoDB (pre-2.4). Partition boundaries must adapt to data distribution — automatic boundary management (dynamic partitioning) is strongly preferred over manually configured boundaries to avoid under/over-filled partitions. **Hash partitioning:** Used by Cassandra, Riak, Voldemort, MongoDB (hash mode). The hash function must be stable across processes — language-native hash functions (Java's `Object.hashCode()`, Ruby's `Object#hash`) are not safe for this purpose because they can return different values in different processes. Use a purpose-built stable function: Cassandra and MongoDB use MD5, Voldemort uses Fowler-Noll-Vo. Assign each partition a range of hash values (not `hash mod N` — see rebalancing step). **Compound key (Cassandra):** The partition key (first component) is hashed to determine the partition. The clustering columns (remaining components) determine sort order within the partition. A query that fixes the partition key can perform an efficient range scan over the clustering columns without touching other partitions. This is the primary pattern for one-to-many relationships in Cassandra: `(user_id, post_timestamp)` stores all posts for a user in one partition, sorted by time. ### Step 3 — Design the Secondary Index Strategy If the application requires queries on non-primary-key attributes, choose between document-partitioned (local) and term-partitioned (global) secondary indexes. **Why:** Secondary indexes do not map cleanly to partitions. The choice of local vs. global index makes a fundamental trade-off between write cost and read cost that cannot be undone without a full reindex. Making this decision explicitly — and documenting it — prevents later surprises when a secondary index query turns out to require scatter/gather across every partition in the cluster. **Document-partitioned (local) index:** - Each partition maintains its own secondary index covering only the documents in that partition. - Write: only the partition being written must be updated. Fast, local, no cross-partition coordination. - Read: a secondary index query must be sent to all partitions and results merged (scatter/gather). Tail latency amplification — the query takes as long as the slowest partition. - Used by: MongoDB, Riak, Cassandra, Elasticsearch, SolrCloud, VoltDB. - Choose when: writes are frequent and latency-sensitive; read queries on secondary indexes are acceptable to be slower; secondary index queries usually target a single partition (structured such that the secondary query attribute correlates with the partition key). **Term-partitioned (global) index:** - A single global index covers all partitions, but is itself partitioned (by the indexed term or a hash of the term). - Read: a secondary index query hits a single index partition — fast, no scatter/gather. - Write: a single document write may need to update index entries spread across multiple index partitions. Updates are typically asynchronous (DynamoDB global secondary indexes update within a fraction of a second normally, but with potential delays under faults). - Used by: Amazon DynamoDB (global secondary indexes), Riak's search feature, Oracle data warehouse. - Choose when: secondary index read frequency and latency are critical; writes are less frequent or can tolerate asynchronous index propagation; consistency requirements can accept eventually-consistent secondary indexes. **Index partition boundary choice (for global indexes):** - Range-partitioned term index: supports range scans on the indexed attribute (e.g., price ranges). - Hash-partitioned term index: more uniform load distribution, no range scan support on the index. ### Step 4 — Select Rebalancing Approach Choose fixed-partition, dynamic, or proportional-to-nodes rebalancing. **Why:** As data grows and nodes are added or removed, partitions must be redistributed. The rebalancing strategy determines how disruptive this is, how much data moves, and how much operational overhead it requires. Using `hash mod N` — the naive approach — is explicitly wrong: it causes the vast majority of keys to move when N changes, making rebalancing catastrophically expensive. **Fixed number of partitions:** - Create many more partitions than nodes at setup (e.g., 1,000 partitions for a 10-node cluster = ~100 partitions/node). - When a node is added, it steals a few partitions from every existing node. Only entire partitions move; no keys are remapped. - Partition count is fixed at creation time. Choose it high enough to accommodate maximum expected cluster size. - Operationally simple. Used by Riak, Elasticsearch, Couchbase, Voldemort. - Risk: if data size is highly variable, partition size grows proportionally to total data, making very large partitions expensive to rebalance and recover from failure. **Dynamic partitioning:** - Partitions split when they exceed a configured size threshold (HBase default: 10 GB); partitions merge when they shrink below a lower threshold. - Partition count adapts to data volume: small datasets use few partitions (low overhead), large datasets use many. - Requires pre-splitting on an empty database (pre-splitting) — without it, all writes initially hit a single partition until the first split. - Used by HBase, RethinkDB. MongoDB supports both dynamic and fixed partitioning. - Choose when: data volume is highly variable or expected to grow significantly; range-partitioned databases where fixed boundaries would be very wrong. **Proportional to nodes:** - Fixed number of partitions per node (Cassandra default: 256 per node). - When a node is added, it randomly splits existing partitions and takes half of each split. - Partition size remains stable as the cluster grows because adding nodes also increases partition count. - Used by Cassandra and Ketama. Requires hash-based partitioning (partition boundaries are drawn from the hash space). - Choose when: using Cassandra or a gossip-based hash-partitioned system; cluster will scale out by adding nodes. **Automatic vs. manual rebalancing:** - Fully automatic: the system decides when and how to move partitions, without operator intervention. More convenient but unpredictable — a node that is temporarily slow may be misidentified as dead, triggering rebalancing that adds load to the already-overloaded node, potentially causing cascading failure. - Semi-automatic (recommended for production): the system generates a suggested rebalancing plan, but an operator must approve it before it executes. Used by Couchbase, Riak, Voldemort. - Fully manual: an administrator configures partition assignment explicitly. - **Recommendation:** Use semi-automatic rebalancing in production. The human-in-the-loop prevents rebalancing storms triggered by false-positive failure detection. ### Step 5 — Select Request Routing Approach Choose how clients discover which node owns a given partition. **Why:** As partitions rebalance, the mapping from partition to node changes. Clients or proxies must track these changes to route requests to the correct node. Using stale routing metadata results in forwarded requests (extra latency), re-tried requests, or errors. The routing approach must match the database's coordination model. Three options: **Option 1 — Any-node with forwarding (gossip-based):** - Clients send requests to any node. If that node owns the partition, it handles the request; otherwise it forwards to the correct node. - Nodes disseminate routing metadata via gossip protocol — no external coordination service required. - Used by Cassandra and Riak. - Trade-off: adds complexity to database nodes; routing metadata may be slightly stale (eventual convergence). Good for systems that must avoid a single-point-of-failure coordination service. **Option 2 — Routing tier (coordination service):** - A dedicated routing layer (partition-aware load balancer) receives all requests, consults a coordination service (ZooKeeper, etcd), and forwards to the correct node. - ZooKeeper maintains the authoritative partition-to-node mapping. Nodes register themselves; the routing tier subscribes to changes. - Used by HBase, SolrCloud, Kafka (ZooKeeper), LinkedIn Espresso (Helix+ZooKeeper). - Trade-off: introduces ZooKeeper as an operational dependency; provides strong consistency for routing metadata. Best for systems with complex multi-partition query routing. **Option 3 — Partition-aware client:** - The client library maintains a local copy of the partition-to-node mapping and connects directly to the correct node. - Requires a mechanism to learn about partition changes — usually subscribing to ZooKeeper, or refreshing from a config server. - Used by MongoDB (mongos daemon), some Cassandra drivers in "token-aware" mode. - Trade-off: pushes routing logic into each client; reduces hop count for simple key lookups. --- ## What Can Go Wrong **Hotspot from monotonically increasing write key** The most common partitioning failure. A table keyed on `created_at`, `id SERIAL`, or any auto-increment column with range partitioning sends 100% of writes to the partition covering the current moment. The newest partition is overloaded while all others sit idle. Fix: Switch to compound key with a high-cardinality prefix, or switch to hash partitioning if range queries are not needed. For time-series with range-query requirements, use `(source_id, timestamp)` compound range key. **Scatter/gather latency on secondary index queries** Document-partitioned (local) secondary indexes require querying all partitions. With 100 partitions, the query waits for the slowest of 100 responses (tail latency amplification). Secondary index queries that were fast in a single-node database become 10–100x slower after partitioning. Fix: If secondary index reads are frequent and latency-sensitive, switch to a global (term-partitioned) index and accept asynchronous write propagation. Alternatively, structure the primary partition key to co-locate records that will be queried together (e.g., partition by tenant_id if secondary queries are always within a tenant). **Rebalancing storm from automatic rebalancing + false-positive failure detection** A temporarily overloaded node responds slowly. Other nodes declare it dead. Automatic rebalancing begins moving its partitions to other nodes. The extra rebalancing traffic overloads the already-struggling node further. The added load on other nodes triggers further detection false-positives. Fix: Use semi-automatic (operator-approved) rebalancing in production. Tune failure detection timeouts conservatively. Set rebalancing rate limits to bound the bandwidth consumed during rebalancing. **Wrong partition count with fixed-partition scheme** Fixed partitions chosen at setup cannot be changed later (in databases that do not support dynamic partitioning). Too few partitions caps how many nodes can be added. Too many creates excessive per-partition overhead. Choosing 10 partitions for a system that grows to 100 nodes means 10 nodes must carry the entire load. Fix: For fixed-partition systems, choose initial partition count 10–20x the expected maximum node count. For variable-growth scenarios, prefer dynamic partitioning. **Language-native hash functions used for partitioning** Java's `Object.hashCode()` and Ruby's `Object#hash` return different values for the same string in different JVM processes (randomization is intentional for security). Using these for partitioning produces incorrect routing — the partition computed at write time differs from the partition computed at read time. Fix: Use a purpose-built stable hash function: MD5 (Cassandra, MongoDB), Fowler-Noll-Vo (Voldemort), MurmurHash, or similar. Verify hash stability across process restarts before deploying. **Stale routing metadata causes misrouted requests** If routing metadata is cached client-side and not refreshed after rebalancing, requests are sent to nodes that no longer own the target partition. The node either returns an error or silently returns stale data. Fix: Use a routing approach that subscribes to partition assignment changes (ZooKeeper watches, gossip convergence). Test routing after every rebalancing event. Implement retry-with-redirect at the client layer. --- ## Examples ### Example 1 — IoT sensor time-series platform **Scenario:** Storing sensor readings where each reading has `(sensor_id, timestamp, value)`. Queries: fetch all readings for a sensor within a date range; write throughput is high and continuous. **Trigger:** "We're building a sensor data platform on HBase. Our current key is just the timestamp, and we're seeing one region server getting all the writes." **Process:** 1. Skew classification: Monotonic key — `timestamp` as the sole partition key causes all writes to go to the "now" partition. 2. Method selection: Range partitioning is required (range scans by date are the primary read pattern). Apply compound range key with sensor name as prefix: key = `(sensor_name, timestamp)`. 3. Secondary index: None required — all queries fix `sensor_name` (partition prefix) and scan a `timestamp` range within the partition. 4. Rebalancing: Dynamic partitioning (HBase default). Pre-split with known sensor names to avoid cold-start single-partition writes. 5. Routing: ZooKeeper-based routing tier (HBase default via HMaster). **Output:** Compound range key `(sensor_name, timestamp)`. All writes for a sensor go to the same partition, distributed across sensors. Range queries within a sensor are efficient single-partition scans. Dynamic rebalancing handles data growth. --- ### Example 2 — Social media post feed (Cassandra) **Scenario:** Storing user posts where queries are: "fetch all posts by user X sorted by time" (dominant) and "fetch a single post by post_id" (secondary). Write pattern: each user writes to their own posts; no cross-user write conflicts. **Trigger:** "We're modeling user posts in Cassandra. We need to efficiently retrieve all posts by a user, sorted from newest to oldest, but we also need to look up individual posts by ID." **Process:** 1. Skew classification: User ID as partition key is approximately uniform (assuming large user base). No celebrity-user mitigation needed unless specific users are known to be viral. 2. Method selection: Compound key pattern — `user_id` as hash partition key (evenly distributes users across nodes), `post_timestamp DESC` as clustering column (sorts posts within the partition, newest first). 3. Secondary index: Post-by-ID lookup can be handled by a separate table keyed on `post_id` (denormalization, Cassandra idiom) rather than a secondary index, avoiding scatter/gather entirely. 4. Rebalancing: Proportional-to-nodes (Cassandra default, 256 vnodes/node). Automatic with gossip-based partition assignment. 5. Routing: Gossip protocol with token-aware driver (Cassandra default). No ZooKeeper dependency. **Output:** Compound primary key `(user_id, post_timestamp DESC)`. Fetching all posts by user is a single-partition range scan. Individual post lookup uses a separate `posts_by_id` table. Scales horizontally by adding Cassandra nodes. --- ### Example 3 — E-commerce product catalog with multi-attribute search **Scenario:** Product records with attributes `(product_id, category, color, price)`. Queries: look up by `product_id` (dominant write target), filter by `category + color` (secondary index query), filter by `price` range (secondary index query). **Trigger:** "We have 50 million products partitioned by product_id hash across 20 Elasticsearch shards. Our 'filter by category and color' queries are slow and getting slower as we add shards." **Process:** 1. Skew classification: `product_id` hash partitioning is uniform — no hotspot risk on writes. 2. Method selection: Hash partitioning on `product_id` is correct for primary key lookups. Do not change this. 3. Secondary index analysis: Elasticsearch uses document-partitioned (local) indexes by default. The `category + color` filter query is scatter/gathered across all 20 shards. Query latency scales with shard count. 4. Secondary index strategy: Two options: - If `category` queries are always within a tenant/store, consider routing documents by `(store_id, product_id)` compound key to co-locate products from the same store, reducing scatter/gather to intra-store shards. - If global catalog search is required, accept scatter/gather and optimize by reducing shard count (fewer, larger shards perform better for scatter/gather than many small shards) or pre-aggregate category+color facets. 5. Rebalancing: Fixed partitions (Elasticsearch uses fixed shard count). Choose initial shard count = projected node count × 2–3. Do not over-shard. 6. Routing: Elasticsearch coordinator node (partition-aware proxy, equivalent to routing tier option). **Output:** Keep hash partitioning on `product_id`. Accept scatter/gather for secondary index queries — this is the Elasticsearch design. Reduce total shard count to the minimum needed; over-sharding amplifies scatter/gather cost. Add replica shards for read scaling on hot index queries. --- ## Cross-References - **replication-strategy-selector** — Select replication topology (single-leader, multi-leader, leaderless) after partitioning is configured. Replication factor affects partition sizing. - **data-model-selector** — Must be run before this skill. The schema and access pattern decisions made there determine viable partition keys and secondary index requirements. - **storage-engine-selector** — The storage engine (LSM-tree vs. B-tree) affects how range scans perform within a partition. Relevant when range partitioning is selected. --- ## References See `references/` for: - `partitioning-decision-matrix.md` — Scoring rubric comparing range, hash, and compound strategies across 6 workload dimensions - `rebalancing-strategies.md` — Detailed comparison of fixed, dynamic, and proportional rebalancing with configuration guidance - `secondary-index-trade-offs.md` — Local vs. global index trade-off analysis with cost model - `hotspot-mitigation-patterns.md` — Patterns for monotonic key hotspots, celebrity key hotspots, and write skew mitigation - `request-routing-comparison.md` — Gossip vs. ZooKeeper vs. partition-aware client with operational trade-offs per database ## License This skill is licensed under [CC-BY-SA-4.0](https://creativecommons.org/licenses/by-sa/4.0/). Source: [BookForge](https://github.com/bookforge-ai/bookforge-skills) — Designing Data-Intensive Applications by Martin Kleppmann. ## Related BookForge Skills Install related skills from ClawhHub: - `clawhub install bookforge-data-model-selector` Or install the full book set from GitHub: [bookforge-skills](https://github.com/bookforge-ai/bookforge-skills)

path: references/hotspot-mitigation-patterns.md
content: # Hotspot Mitigation Patterns

Concrete patterns for detecting and relieving the most common partitioning hotspot types.

---

## Pattern 1: Monotonic Key Hotspot (Range Partitioning)

### Symptom
All writes are routed to a single partition — typically the "latest" one. Other partitions are idle. The affected partition's CPU and I/O are saturated while cluster utilization is very low overall.

### Root cause
The partition key is monotonically increasing: `created_at TIMESTAMP`, `id BIGSERIAL`, `event_sequence BIGINT`, `order_id INT AUTO_INCREMENT`. Range partitioning assigns contiguous key ranges to partitions. All new data falls in the highest range, and all current writes go to the partition holding that range.

### Mitigation: Compound prefix key

Prepend a high-cardinality, non-monotonic value to the key before the monotonic component.

**Before (bad):**
```
partition key: timestamp
range: 2024-01-01T00:00:00 → 2024-01-01T23:59:59  →  Partition 0
range: 2024-01-02T00:00:00 → 2024-01-02T23:59:59  →  Partition 1  ← ALL current writes
range: 2024-01-03T00:00:00 → ...                  →  Partition 2  ← empty
```

**After (good):**
```
partition key: (sensor_name, timestamp)
range: sensor_A, 2024-01-01 → sensor_A, 2024-01-03  →  Partition 0  ← sensor A writes
range: sensor_B, 2024-01-01 → sensor_B, 2024-01-03  →  Partition 1  ← sensor B writes
range: sensor_C, 2024-01-01 → sensor_C, 2024-01-03  →  Partition 2  ← sensor C writes
```

**Query impact:** To fetch all sensor readings within a time range (regardless of sensor), you must issue one range query per sensor prefix. This is an accepted trade-off. If you need a global time range query, consider a secondary index on `timestamp` or accept scatter/gather.

**Alternative prefix choices:** User ID, device ID, region code, category ID — any dimension with sufficient cardinality to spread load, which is also available at write time.

---

## Pattern 2: Monotonic Key Hotspot (Hash Partitioning on Sequential IDs)

### Symptom
Hash partitioning is in use, but a new auto-increment ID or UUID-v1 (time-based) is the key. With UUID-v1, the time component is in the high-order bits and produces similar hash values for keys created at the same time, clustering them in the same partition.

### Root cause
UUID-v1 and ULID encode timestamp in a way that may produce sequential values over short intervals, partially defeating hash partitioning.

### Mitigation
- Use UUID-v4 (random) as the partition key — fully random hash distribution.
- Use ULID but reverse the timestamp component to distribute writes: `timestamp_reversed + random_component`.
- Use application-generated random IDs rather than database-generated sequential IDs.
- If you must retain sequential IDs for business reasons, use a hash of the sequential ID as the partition key (not the ID itself), and store the original ID as a separate non-partitioning column.

---

## Pattern 3: Celebrity Key Hotspot (Hash Partitioning)

### Symptom
One or a few specific keys receive orders of magnitude more traffic than all others. Hash partitioning distributes keys evenly in aggregate, but the hot keys are each still concentrated in a single partition. The partition containing the celebrity key is overloaded; others are not.

### Root cause
The partitioning is correct, but the data distribution is inherently skewed: a celebrity user, a viral post, a globally-shared resource (e.g., a global counter, a shared cart, a trending topic ID). Hashing cannot help — `hash(celebrity_key)` is still a single value.

### Mitigation: Write-time key salting

Append a random suffix (00–99) to hot keys at write time, spreading one logical key across 100 physical keys.

**Write:**
```python
import random

def write_celebrity_key(db, key, value):
    # Only apply to known-hot keys
    if is_hot_key(key):
        suffix = random.randint(0, 99)
        physical_key = f"{key}_{suffix:02d}"
    else:
        physical_key = key
    db.write(physical_key, value)
```

**Read:**
```python
def read_celebrity_key(db, key):
    if is_hot_key(key):
        # Must read all 100 suffix variants and merge
        results = []
        for i in range(100):
            physical_key = f"{key}_{i:02d}"
            results.append(db.read(physical_key))
        return merge(results)
    else:
        return db.read(key)
```

**Bookkeeping:** Maintain a lookup table (or application config) of which keys are "hot" so that the salting and merge logic is applied selectively. Applying it to all keys is unnecessary overhead for the vast majority that are not hot.

**Trade-off:** Reads on hot keys now require fetching 100 records and merging. This is acceptable when writes vastly outnumber reads for the hot key, or when the merge is simple (e.g., summing a counter across suffix partitions).

### Alternative: Application-level sharding for celebrity resources

For resources like a global counter or a shared leaderboard that receive extreme write volume, use application-level partitioned counters:

- Maintain N counter shards in the application layer.
- Each write randomly increments one shard.
- To read the total, sum all N shards.
- N can be tuned based on write throughput requirements.

---

## Pattern 4: Write Skew from Unbalanced Partition Boundaries

### Symptom
Some partitions are very large; others are nearly empty. This occurs most often with range partitioning where partition boundaries were chosen based on assumed key distribution that did not match actual data.

### Root cause
In range partitioning, partition boundaries must match the actual data distribution, not an assumed uniform distribution. For example, partitioning a name field alphabetically with equal letter ranges (A–B, C–D, E–F, ...) will produce very unequal partitions because letters do not appear with equal frequency.

### Mitigation
- Use dynamic partitioning (HBase, RethinkDB, MongoDB): boundaries adapt automatically based on actual data volume.
- If using fixed boundaries, analyze actual key distribution before setting boundaries. HBase supports pre-splitting with custom split points derived from a key distribution analysis.
- Use hash partitioning if range queries are not required — hash partitioning produces near-uniform distribution regardless of key distribution.

---

## Detection: How to Identify Hotspots Before They Cause Outages

**Metrics to monitor per partition:**
- Write requests/second per partition (look for orders-of-magnitude differences)
- Storage size per partition (look for one partition growing much faster)
- CPU and I/O utilization per node (hotspot node will be saturated)

**Query to identify hot partition in Cassandra:**
```bash
nodetool tpstats           # Thread pool stats — saturated pools on one node
nodetool cfhistograms      # Per-table latency and request histograms
nodetool tablestats        # Per-table metrics including partition size
```

**Query to identify hot shards in Elasticsearch:**
```bash
GET _cat/shards?v&s=store   # Shards sorted by store size
GET _nodes/stats/indices    # Per-node indexing/search rate
```

**Signal:** If one partition/shard/node shows 10x or more the request rate or storage of others, a hotspot is present.


path: references/partitioning-decision-matrix.md
content: # Partitioning Decision Matrix

Scoring rubric for selecting among range, hash, and compound key partitioning strategies. Score each dimension 1–5, then total. The highest-scoring method is the primary recommendation; use the runner-up to inform mitigation strategies.

---

## Dimensions

### 1. Range Query Support
Does the application require efficient range scans on the partition key (e.g., date ranges, alphabetical ranges, numeric ranges)?

| Score | Meaning |
|---|---|
| 5 | Range queries are the primary access pattern — efficient range scan is critical |
| 3 | Range queries are needed but not dominant |
| 1 | All queries are point lookups — range queries are never needed |

### 2. Write Distribution (inverted — higher = more uniform)
How uniformly are writes distributed across the key space?

| Score | Meaning |
|---|---|
| 5 | Keys are randomly/uniformly distributed (UUIDs, random hashes) |
| 3 | Moderate skew — some keys receive more writes but not extreme concentration |
| 1 | Strongly monotonic or celebrity-key workload — severe skew without mitigation |

### 3. Cross-Partition Query Avoidance
Does the application's primary access pattern allow most queries to be served from a single partition?

| Score | Meaning |
|---|---|
| 5 | Primary queries always fix the partition key (single-partition lookups) |
| 3 | Mixed — some queries span partitions, most do not |
| 1 | Most queries fan out across all partitions (analytics, global aggregations) |

### 4. Secondary Index Requirements
How demanding are secondary index query patterns (non-primary-key attribute filters)?

| Score | Meaning |
|---|---|
| 5 | No secondary index queries — all queries use the primary key |
| 3 | Secondary index queries exist but are infrequent or can tolerate scatter/gather |
| 1 | Secondary index queries are frequent, latency-sensitive, and cannot scatter/gather |

### 5. One-to-Many Relationship Access
Does the application frequently fetch all child records for a given parent entity, sorted?

| Score | Meaning |
|---|---|
| 5 | Yes — dominant pattern is "fetch all X for user Y, sorted by timestamp" |
| 3 | Occasionally needed |
| 1 | Never — all lookups are for individual records |

### 6. Operational Simplicity
How much operational complexity can the team absorb for partition boundary management?

| Score | Meaning |
|---|---|
| 5 | Prefer fully managed/automatic boundary assignment |
| 3 | Can manage semi-automatic with operator approval |
| 1 | Team has capacity for manual partition boundary administration |

---

## Scoring by Strategy

For each dimension, score the strategy being evaluated:

| Dimension | Range | Hash | Compound |
|---|---|---|---|
| Range query support | High score when range queries are critical | Low score — hash destroys sort order | High score — range scans within partition |
| Write distribution | Low score for monotonic keys | High score — hash distributes evenly | Medium — distributes on hash part, sequential within |
| Cross-partition query avoidance | High score if queries always fix prefix | High score for point lookups | High score — compound key collocates related records |
| Secondary index requirements | Medium | Medium | Medium (same trade-offs apply to secondary indexes) |
| One-to-many relationship access | Low score — all parents share the key space | Low score — cannot sort within partition | High score — clustering columns sort within partition |
| Operational simplicity | Medium — dynamic partitioning helps | High — hash ranges are simple to manage | Medium — compound key design requires upfront modeling |

---

## Quick-Decision Table

| Access pattern | Write distribution | Recommendation |
|---|---|---|
| Range queries dominant | Uniform keys | Range partitioning |
| Range queries dominant | Monotonic keys | Range + compound prefix (prefix + timestamp) |
| Point lookups only | Any | Hash partitioning |
| Point lookup on entity + range scan within entity | Any | Compound key (hash entity, range sort key) |
| Analytics / multi-attribute filter | Any | Hash + accept scatter/gather, or global secondary index |

---

## Example Scores

### IoT sensor platform (sensor_name + timestamp key)
| Dimension | Score |
|---|---|
| Range queries (date range scans) | 5 |
| Write distribution (timestamp = monotonic → needs prefix) | 3 (with prefix mitigation) |
| Cross-partition avoidance (prefix fixes partition) | 5 |
| Secondary index requirements (none needed) | 5 |
| One-to-many (all readings per sensor) | 4 |
| Operational simplicity (HBase dynamic) | 4 |
| **Total** | **26 / 30** |

**Winner: Range with compound prefix**

### User post feed (Cassandra: user_id + post_timestamp)
| Dimension | Score |
|---|---|
| Range queries (fetch posts by time) | 4 |
| Write distribution (user IDs uniform) | 5 |
| Cross-partition avoidance (user_id fixes partition) | 5 |
| Secondary index requirements (none — denormalized) | 5 |
| One-to-many (all posts per user) | 5 |
| Operational simplicity (Cassandra default) | 5 |
| **Total** | **29 / 30** |

**Winner: Compound key (hash + range)**


path: references/rebalancing-strategies.md
content: # Rebalancing Strategies

Detailed comparison of the three main rebalancing approaches, with configuration guidance and known failure modes.

---

## Why Not hash mod N

The naive approach to rebalancing is `partition = hash(key) mod N` where N is the number of nodes.

**The problem:** When N changes (a node is added or removed), the modulo result changes for nearly every key. If you had 10 nodes and add one (N = 11), `hash(key) mod 10` ≠ `hash(key) mod 11` for most keys — they all need to move. This makes rebalancing catastrophically expensive: adding a single node triggers a full data shuffle.

**The fix:** Assign each partition a fixed range of hash values (not a modulo). When a node is added, it takes over some partitions' hash ranges from existing nodes. Keys within a given hash range always map to the same partition; only the partition-to-node assignment changes, not the key-to-partition mapping.

---

## Strategy 1: Fixed Number of Partitions

**How it works:**
- Create many more partitions than nodes at initial setup. Common ratio: 10:1 to 100:1 (e.g., 1,000 partitions for a 10-node cluster).
- When a node is added: it steals a few complete partitions from every existing node until load is balanced.
- When a node is removed: its partitions are distributed to remaining nodes.
- Key-to-partition mapping never changes. Only partition-to-node assignment changes.
- Used by: Riak, Elasticsearch, Couchbase, Voldemort.

**Configuration:**
- Initial partition count: set at cluster creation, cannot be changed later in most fixed-partition databases.
- Rule of thumb: `initial_partitions = max_expected_nodes × 10`
- Elasticsearch: `number_of_shards` per index (set at index creation, immutable without reindex)
- Riak: `ring_size` (power of 2, set at cluster creation)

**When to choose:**
- Total data size is known and bounded — partition count can be sized correctly upfront.
- Operational simplicity is preferred — no dynamic splitting/merging to manage.
- Database supports it natively (Riak, Elasticsearch, Couchbase, Voldemort).

**Failure modes:**
- Too few partitions: limits maximum cluster size. A 10-partition cluster can never have more than 10 nodes effectively.
- Too many partitions: each partition has per-partition overhead (metadata, connections, background processes). Very high partition counts degrade performance.
- Data grows much larger than expected: partition size grows proportionally, making each partition very large and recovery from node failure very slow.

---

## Strategy 2: Dynamic Partitioning

**How it works:**
- Partitions split when they exceed a size threshold; they merge when they fall below a lower threshold.
- Partition count adapts to total data volume automatically.
- After a large partition splits, one of the two halves can be transferred to another node to balance load.
- Used by: HBase (default 10 GB threshold), RethinkDB, MongoDB (both hash and range modes).

**Configuration (HBase example):**
```
hbase.regionserver.region.split.policy = IncreasingToUpperBoundRegionSplitPolicy
hbase.hregion.max.filesize = 10737418240   # 10 GB — split threshold
```

**Pre-splitting (required for empty databases):**
An empty database starts with a single partition. All writes hit one node until the first split. For key-range partitioning, pre-splits require knowing the key distribution in advance.

```
# HBase: create a table with 10 pre-splits for known key prefixes
create 'sensor_data', 'readings', SPLITS => ['sensor01', 'sensor02', ..., 'sensor09']
```

For hash partitioning, MongoDB supports pre-splitting by hash value ranges.

**When to choose:**
- Data volume is highly variable or will grow significantly and unpredictably.
- Key-range partitioning is used (dynamic partitioning is the natural complement).
- The database natively supports it (HBase, RethinkDB, MongoDB).

**Failure modes:**
- Cold-start bottleneck: without pre-splitting, all early writes go to one partition/node.
- Split storms: a sudden data ingestion spike can trigger many rapid splits simultaneously, creating high I/O load on the splitting nodes.
- Pre-split key guessing: key-range pre-splits require knowing key distribution upfront, which may not be available for new systems.

---

## Strategy 3: Proportional to Nodes

**How it works:**
- Fixed number of partitions per node (Cassandra default: 256 vnodes per node).
- Total partition count = nodes × partitions_per_node.
- When a new node joins: it randomly selects existing partitions, splits them, and takes ownership of half of each split partition.
- Partition size stays roughly constant as nodes are added (more nodes = more partitions = same data per partition).
- Used by: Cassandra (virtual nodes / vnodes), Ketama.

**Configuration (Cassandra):**
```yaml
# cassandra.yaml
num_tokens: 256          # vnodes per node (partitions per node)
                         # Cassandra 3.0+: use vnodes for automatic load balancing
                         # Lower values (16, 32) reduce overhead but decrease balance quality
```

**Virtual nodes (vnodes):** Rather than assigning each node a single contiguous range of the hash space, each node owns many small non-contiguous ranges. This improves load distribution when nodes have heterogeneous capacity (a stronger node gets more vnodes). It also reduces the data movement needed when a node fails (failure load is spread across all other nodes, not just neighbors in the ring).

**When to choose:**
- Using Cassandra, Ketama, or a gossip-based ring system.
- Cluster will grow by adding nodes — partition size stays stable automatically.
- Hash partitioning is in use (proportional rebalancing requires hash-based boundary selection).

**Failure modes:**
- Unfair splits from randomization: with few partitions per node, random split selection can create unbalanced partition sizes. Cassandra 3.0 introduced a deterministic allocation algorithm to address this.
- New node join overhead: a new node must stream data from all nodes it takes partitions from, generating simultaneous I/O on multiple existing nodes.

---

## Automatic vs. Manual Rebalancing Decision

| Approach | Operational burden | Predictability | Risk |
|---|---|---|---|
| Fully automatic | Low — system handles all moves | Low — unpredictable timing | Rebalancing storm from false-positive failure detection |
| Semi-automatic (recommended) | Medium — operator approves proposed plan | High — operator sees full plan before execution | Delayed response to cluster changes |
| Fully manual | High — DBA configures partition assignment | High | Human error in assignment |

**Recommendation for production:** Use semi-automatic. The operator-approval step costs minutes but prevents the failure mode where a temporarily slow node triggers a rebalancing cascade that makes the situation worse.

Systems that support semi-automatic rebalancing: Couchbase, Riak, Voldemort (generate plan → operator commits → execution begins).


path: references/request-routing-comparison.md
content: # Request Routing Comparison

Detailed comparison of the three request routing approaches for partitioned databases, with operational trade-offs and database-specific guidance.

---

## The Problem

When a client sends a read or write request, the database must route it to the node that owns the target partition. As partitions rebalance (nodes added, removed, or failed over), the mapping from partition to node changes. Every component that makes routing decisions must have an up-to-date copy of this mapping.

Using stale routing metadata results in:
- Request forwarded to wrong node → error or extra hop latency
- Silent stale reads (wrong node returns stale data)
- Write to wrong node → data on the correct node is not updated

---

## Option 1: Any-Node with Forwarding (Gossip Protocol)

### How it works
1. Client sends request to any node (e.g., via round-robin load balancer or DNS).
2. If the receiving node owns the target partition, it handles the request directly.
3. If not, the receiving node looks up the correct owner in its local routing table and forwards the request to that node.
4. The correct node handles the request and returns the result (either directly to client or via the forwarding node).

Nodes maintain an up-to-date routing table by gossiping — periodically exchanging cluster state information with random peers. Changes propagate through the cluster over time (eventual consistency in metadata, not data).

### Used by
- Apache Cassandra (gossip protocol, token-aware drivers)
- Riak (riak_core gossip)

### Cassandra-specific
Cassandra drivers support "token-aware routing": the driver computes the partition key hash and connects directly to the owning node, bypassing the forwarding step. This reduces latency by eliminating the unnecessary hop.

```python
# Python cassandra-driver: token-aware load balancing policy
from cassandra.policies import TokenAwarePolicy, DCAwareRoundRobinPolicy
from cassandra.cluster import Cluster

cluster = Cluster(
    contact_points=['10.0.0.1', '10.0.0.2'],
    load_balancing_policy=TokenAwarePolicy(DCAwareRoundRobinPolicy())
)
```

### Operational trade-offs
| Dimension | Assessment |
|---|---|
| External dependency | None — no ZooKeeper or etcd required |
| Routing consistency | Eventual — gossip convergence may take seconds; brief stale routing is possible |
| Node complexity | Higher — routing logic lives in each database node |
| Failure tolerance | Good — no single coordinator to fail; gossip continues without central coordination |
| Configuration | Minimal — gossip is enabled by default in Cassandra/Riak |

### Best for
Systems that must avoid external coordination service dependencies. Multi-datacenter deployments where cross-DC ZooKeeper consensus would add latency. Teams with operational experience with Cassandra or Riak.

---

## Option 2: Routing Tier with Coordination Service

### How it works
1. A dedicated routing tier (partition-aware proxy/load balancer) sits between clients and database nodes.
2. Routing tier consults a coordination service (ZooKeeper, etcd) to look up the current partition-to-node mapping.
3. Routing tier forwards the request to the correct node and returns the response to the client.
4. Nodes register their partition assignments in ZooKeeper on join/leave. ZooKeeper notifies the routing tier of changes via watches.

### Used by
- Apache HBase (HMaster + ZooKeeper)
- Apache SolrCloud (ZooKeeper)
- Apache Kafka (ZooKeeper, pre-KRaft)
- LinkedIn Espresso (Helix + ZooKeeper)
- MongoDB (config server + mongos routing daemon)

### ZooKeeper-based routing (HBase/SolrCloud pattern)
```
Client → Routing Tier → ZooKeeper (watches partition assignment)
                      ↓
                  Node N (owns target partition)
```

ZooKeeper stores:
- Partition-to-node assignment map
- Node health/liveness registration
- Cluster epoch (version of the assignment)

Routing tier subscribes to ZooKeeper watches on the assignment node. When a partition moves, ZooKeeper notifies all subscribed routing tier instances, which update their local routing tables.

### MongoDB routing (mongos)
MongoDB uses a dedicated `config server` (replicated MongoDB deployment) to store shard metadata. `mongos` daemons act as the routing tier, caching shard metadata locally and refreshing from config server on stale-route errors.

```
Client → mongos → config server (shard map)
               ↓
           shard N (mongod)
```

### Operational trade-offs
| Dimension | Assessment |
|---|---|
| External dependency | Required — ZooKeeper/etcd cluster must be maintained and highly available |
| Routing consistency | Strong — ZooKeeper watch notifications are reliable; routing tier can have up-to-date metadata at all times |
| Node complexity | Lower — routing logic is isolated to the routing tier and coordination service |
| Failure tolerance | ZooKeeper itself becomes a critical path; ZooKeeper outage impairs routing tier operation |
| Operational overhead | Higher — ZooKeeper cluster requires its own deployment, monitoring, and maintenance |

### Best for
Systems with complex multi-partition query routing (e.g., analytics queries spanning multiple shards). Systems where routing correctness is critical and brief stale routing is unacceptable. Teams already operating ZooKeeper for other services (e.g., Kafka, HBase).

---

## Option 3: Partition-Aware Client

### How it works
1. The client library maintains a local cache of the partition-to-node mapping.
2. Client computes the target partition for the request and connects directly to the owning node.
3. Client refreshes its local routing map from a known source (ZooKeeper, config server, a seed node) on startup and periodically, or upon receiving a routing error from a node.

### Used by
- Cassandra drivers in token-aware mode (driver holds the token ring map)
- MongoDB application drivers (can bypass mongos and connect directly with partition info)
- DynamoDB (fully managed — AWS handles routing; client only calls the service endpoint)

### Cassandra token-aware driver (detailed)
The Cassandra driver downloads the full token ring topology from the cluster (via the `system.peers` table). For each write/read, the driver computes `hash(partition_key)` to determine the token, then looks up the token ring to find the owning node's IP address, and connects directly.

When the token ring changes (node added/removed), the driver receives a topology change event and updates its local copy. This update is asynchronous — there is a brief window where the driver has stale routing information.

### Operational trade-offs
| Dimension | Assessment |
|---|---|
| External dependency | None to minimal — may need initial seed node or config server endpoint |
| Routing consistency | Eventually consistent — client cache is refreshed asynchronously |
| Node complexity | Low — database nodes have no routing responsibility |
| Client complexity | Higher — routing logic is in each client library; library versions must be kept current |
| Latency | Lowest — zero-hop direct connection to owning node |

### Best for
Performance-critical applications where eliminating the forwarding hop matters. Systems using Cassandra with token-aware drivers. Managed database services (DynamoDB, Azure Cosmos DB) where routing is fully abstracted.

---

## Summary Comparison

| | Any-node + Forward | Routing Tier + Coord Service | Partition-Aware Client |
|---|---|---|---|
| Routing consistency | Eventual (gossip) | Strong (ZooKeeper watch) | Eventual (client cache refresh) |
| Hop count | 1–2 | 2 | 1 (direct) |
| External dependency | None | ZooKeeper/etcd | None (or seed node) |
| Operational complexity | Low | High | Medium |
| Best databases | Cassandra, Riak | HBase, SolrCloud, Kafka | Cassandra (token-aware), managed DBs |
| Failure of routing component | Gossip-based self-healing | ZooKeeper outage impairs routing | Client cache staleness |

---

## Handling Stale Routing in All Approaches

Regardless of which approach is used, clients should implement a stale-routing recovery path:

1. Send request using current routing information.
2. If the target node returns an error indicating it no longer owns the partition (e.g., Cassandra's `UnavailableException` with a "wrong coordinator" hint), refresh routing information from the authoritative source.
3. Retry the request with updated routing information.
4. If retry fails, surface the error to the application layer.

This retry-on-stale-route pattern ensures correctness even during rebalancing events when routing information is temporarily inconsistent.


path: references/secondary-index-trade-offs.md
content: # Secondary Index Trade-offs: Local vs. Global

This reference covers the detailed trade-off analysis between document-partitioned (local) and term-partitioned (global) secondary indexes in a partitioned database.

---

## The Core Problem

Secondary indexes identify records by attributes that are not the partition key. Because the partition key determines which node holds a record, a secondary index on a different attribute spans all partitions — records with a given attribute value may exist in every partition.

There is no way to partition both the primary data and the secondary index by the same key simultaneously. Every secondary index strategy involves a trade-off between write complexity and read complexity.

---

## Document-Partitioned (Local) Index

### Architecture

Each partition maintains its own secondary index, containing entries only for documents stored in that partition.

```
Partition 0                          Partition 1
PRIMARY DATA                         PRIMARY DATA
 product_id=191  color=red            product_id=515  color=silver
 product_id=214  color=black          product_id=768  color=red
 product_id=306  color=red            product_id=893  color=silver

LOCAL SECONDARY INDEX                LOCAL SECONDARY INDEX
 color:red   → [191, 306]             color:red   → [768]
 color:black → [214]                  color:silver → [515, 893]
```

### Write path
A write to `product_id=191` goes to Partition 0. Only Partition 0 updates its local index. No other partition is involved.

### Read path
A query for `color=red` must be sent to Partition 0 AND Partition 1 (scatter), wait for both to respond, then merge the results (gather). With N partitions, the query fans out to all N partitions.

### Tail latency amplification
Scatter/gather means the query's latency is determined by the slowest partition's response. With 20 partitions, the query is as slow as the slowest of 20 responses. Adding partitions worsens this unless all partitions are identical in performance.

### When to choose local indexes
- Writes are frequent and write latency is critical.
- Secondary index queries are infrequent or can tolerate higher latency.
- Secondary index queries are structured so they can be scoped to a single partition (e.g., always filter by tenant_id first, and data is partitioned by tenant_id — so the secondary index query hits only one partition).
- The database of choice uses local indexes by default and you cannot change it (MongoDB, Cassandra, Elasticsearch, SolrCloud, Riak, VoltDB).

### Mitigation for scatter/gather cost
- Reduce partition count (fewer partitions = less scatter). Over-sharding (e.g., 500 shards for a dataset that fits in 20) amplifies scatter/gather with no benefit.
- Ensure secondary index queries always include the partition key as a filter, so the query can be routed to a single partition.
- Precompute frequently used aggregate queries (materialized views, denormalized rollup tables) instead of relying on secondary index scatter/gather for high-frequency queries.

---

## Term-Partitioned (Global) Index

### Architecture

A single global index covers all partitions, but the index itself is partitioned by the indexed term (or a hash of the term).

```
Global Index Partition 0              Global Index Partition 1
(terms a–r)                           (terms s–z)
 color:black → [191, 515]             color:silver → [515, 893]
 color:red   → [191, 306, 768]        make:Volvo   → [768]
 make:Audi   → [893]
 make:Dodge  → [214]
 make:Ford   → [306, 515]
 make:Honda  → [191]
```

The primary data is still partitioned by `product_id`. The global index is separately partitioned by the indexed attribute value.

### Write path
A write to `product_id=191` (color=red, make=Honda):
- Updates primary data in the `product_id` partition (e.g., Partition 0)
- Must also update `color:red` in Global Index Partition 0
- Must also update `make:Honda` in Global Index Partition 0

A single write now touches multiple partitions (the data partition + all relevant index partitions). In a term-partitioned global index, writes require distributed transactions across partitions — not supported by all databases, so global secondary indexes are often updated **asynchronously**.

### Read path
A query for `color=red` goes to exactly one global index partition (the one covering terms starting with "r"). That partition returns the full list of matching document IDs from all primary partitions. The query can then fetch the actual documents, potentially with targeted lookups to specific primary partitions.

### Asynchronous index updates
Because requiring synchronous updates across multiple index partitions would need distributed transactions, most global secondary index implementations update asynchronously. Amazon DynamoDB's global secondary indexes are normally updated within a fraction of a second, but may lag during infrastructure faults.

**Implication:** If you read the global index immediately after a write, the change may not yet be reflected. Design applications that use global secondary indexes to tolerate brief eventual consistency in index reads.

### Global index partition strategy
- **Range-partitioned term index:** Terms are assigned to partitions by sorted ranges. Supports range queries on the indexed attribute (e.g., price between 100 and 500). Used when the indexed attribute has range-query requirements.
- **Hash-partitioned term index:** Terms are assigned to partitions by hash of the term. More uniform load distribution. No range scan support on the index itself.

### When to choose global indexes
- Secondary index reads are frequent and latency-sensitive.
- Writes are less frequent or can tolerate asynchronous propagation.
- The application can tolerate eventual consistency in secondary index reads.
- The database supports global secondary indexes (DynamoDB, Oracle, Riak search feature).

---

## Comparison Summary

| Dimension | Local (Document-Partitioned) | Global (Term-Partitioned) |
|---|---|---|
| Write cost | Low — single partition update | High — multiple index partitions updated |
| Write consistency | Synchronous, always up-to-date | Often asynchronous — brief eventual consistency |
| Read cost (secondary index query) | High — scatter/gather all partitions | Low — single index partition |
| Read tail latency | Amplified by partition count | Stable — single partition |
| Operational complexity | Low — no separate index management | Higher — global index is a separate partitioned structure |
| Databases | MongoDB, Cassandra, Elasticsearch, Riak, VoltDB | DynamoDB global secondary indexes, Oracle, Riak search |

---

## Hybrid Pattern: Co-location to Avoid Secondary Index

In some cases, secondary index overhead can be eliminated entirely by redesigning the partition key to co-locate records that will be queried together.

**Pattern:** If secondary index queries always filter by a high-cardinality attribute (e.g., `tenant_id`, `store_id`, `user_id`), partition the primary data by that attribute. Secondary index queries within a tenant are now single-partition lookups — no scatter/gather, no global index needed.

**Trade-off:** This works only when secondary index queries are always scoped to the partition key value. If global cross-partition secondary index queries are also needed, co-location does not eliminate the scatter/gather requirement for those queries.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)