Skill: ivangdavila/mongodb (version 1.0.1)

--- name: MongoDB slug: mongodb version: 1.0.1 description: Design MongoDB schemas with proper embedding, indexing, aggregation, and production-ready patterns. metadata: {"clawdbot":{"emoji":"🍃","requires":{"anyBins":["mongosh","mongo"]},"os":["linux","darwin","win32"]}} --- ## When to Use User needs MongoDB expertise — from schema design to production optimization. Agent handles document modeling, indexing strategies, aggregation pipelines, consistency patterns, and scaling. ## Quick Reference | Topic | File | |-------|------| | Schema design patterns | `schema.md` | | Index strategies | `indexes.md` | | Aggregation pipeline | `aggregation.md` | | Production configuration | `production.md` | ## Schema Design Philosophy - Embed when data is queried together and doesn't grow unboundedly - Reference when data is large, accessed independently, or many-to-many - Denormalize for read performance, accept update complexity—no JOINs means duplicate data - Design for your queries, not for normalized elegance ## Document Size Traps - 16MB max per document—plan for this from day one; use GridFS for large files - Arrays that grow infinitely = disaster—use bucketing pattern instead - BSON overhead: field names repeated per document—short names save space at scale - Nested depth limit 100 levels—rarely hit but exists ## Array Traps - Arrays > 1000 elements hurt performance—pagination inside documents is hard - `$push` without `$slice` = unbounded growth; use `$push: {$each: [...], $slice: -100}` - Multikey indexes on arrays: index entry per element—can explode index size - Can't have multikey index on more than one array field in compound index ## $lookup Traps - `$lookup` performance degrades with collection size—no index on foreign collection (until 5.0) - One `$lookup` per pipeline stage—nested lookups get complex and slow - `$lookup` with pipeline (5.0+) can filter before joining—massive improvement - Consider: if you $lookup frequently, maybe embed instead ## Index Strategy - ESR rule: Equality fields first, Sort fields next, Range fields last - MongoDB doesn't do efficient index intersection—single compound index often better - Only one text index per collection—plan carefully; use Atlas Search for complex text - TTL index for auto-expiration: `{createdAt: 1}, {expireAfterSeconds: 86400}` ## Consistency Traps - Default read/write concern not fully consistent—`{w: "majority", readConcern: "majority"}` for strong - Multi-document transactions since 4.0—but add latency and lock overhead; design to minimize - Single-document operations are atomic—exploit this by embedding related data - `retryWrites: true` in connection string—handles transient failures automatically ## Read Preference Traps - Stale reads on secondaries—replication lag can be seconds - `nearest` for lowest latency—but may read stale data - Write always goes to primary—read preference doesn't affect writes - Read your own writes: use `primary` or session-based causal consistency ## ObjectId Traps - Contains timestamp: `ObjectId.getTimestamp()`—extract creation time without extra field - Roughly time-ordered—can sort by `_id` for creation order without createdAt - Not random—predictable if you know creation time; don't rely on for security tokens ## Performance Mindset - `explain("executionStats")` shows actual execution—not just theoretical plan - `totalDocsExamined` vs `nReturned` ratio should be ~1—otherwise index missing - `COLLSCAN` in explain = full collection scan—add appropriate index - Covered queries: `IXSCAN` + `totalDocsExamined: 0`—all data from index ## Aggregation Philosophy - Pipeline stages are transformations—think of data flowing through - Filter early (`$match`), project early (`$project`)—reduce data volume ASAP - `$match` at start can use indexes; `$match` after `$unwind` cannot - Test complex pipelines stage by stage—build incrementally ## Common Mistakes - Treating MongoDB as "schemaless"—still need schema design; just enforced in app not DB - Not adding indexes—scans entire collection; every query pattern needs index - Giant documents via array pushes—hit 16MB limit or slow BSON parsing - Ignoring write concern—data may appear written but not persisted/replicated

path: aggregation.md
content: # Aggregation Pipeline

## Pipeline Philosophy

- Think of data as flowing through transformation stages
- Each stage receives documents from previous stage
- Order matters enormously—for both correctness and performance
- Build incrementally—add one stage, verify, add next

## Stage Order for Performance

- `$match` FIRST—reduce document count before anything else
- `$project` early—drop unneeded fields to reduce memory
- Only `$match` at START can use indexes—after any transformation, no index
- `$sort` with `$limit` together—can use index efficiently

## Memory Limits

- 100MB memory limit per stage—hits faster than you expect
- `allowDiskUse: true` for large aggregations—slower but won't fail
- `$group` on high cardinality field = memory explosion
- `$unwind` on large arrays multiplies documents—memory spikes

## $unwind Traps

- Creates one document per array element—100 element array = 100 documents
- Empty/missing array = document DROPPED—use `preserveNullAndEmptyArrays: true`
- Order in array is lost—use `includeArrayIndex` if position matters
- Think before unwinding—maybe `$filter` or `$reduce` is enough

## $group Patterns

- `_id: null` aggregates entire collection—single output document
- `$first`/`$last` depend on input order—need `$sort` before for meaningful results
- `$push` creates array in output—watch for 16MB limit on result
- `$sum` of non-numeric field = 0—silent wrong result, no error

## $lookup (Join)

- No index on foreign collection until 5.0—full collection scan each time
- Results in array even for 1:1—often need `$unwind` after
- Pipeline syntax (5.0+) filters foreign BEFORE joining—much faster
- Recursive lookup via `$graphLookup`—for tree/graph structures

## $facet

- Multiple parallel pipelines on same input documents
- All facets process ALL input documents—can be expensive
- Results in single document—watch 16MB limit
- Great for dashboards with multiple aggregations

## $merge and $out

- `$out` replaces target collection—atomic but destructive
- `$merge` can upsert/update—more flexible but complex
- Use for materialized views—pre-aggregate expensive reports
- Schedule with cron, read from materialized collection

## Debugging Aggregations

- Add `$limit: 1` at end while developing—see one result quickly
- Use `$project` to see intermediate state—remove it after
- Comment out stages to isolate problems
- `explain()` works on aggregations too—check index usage


path: indexes.md
content: # Index Strategies

## Index Design Philosophy

- Design indexes for your queries, not your schema
- Every query pattern needs an index—no index = collection scan
- Start with slowest queries and work backwards
- Measure with explain(), don't guess

## Compound Index Order (ESR Rule)

- **E**quality fields first—exact match conditions
- **S**ort fields next—for ORDER BY without extra sort step
- **R**ange fields last—$gt, $lt, $in (multiple values)
- Order matters: `{status: 1, createdAt: -1}` different from `{createdAt: -1, status: 1}`

## Covered Queries

- Query uses only indexed fields = no document fetch—maximum speed
- Include fields from query AND projection in index
- Check explain for `IXSCAN` + `totalDocsExamined: 0`
- Worth it for frequent, simple queries

## Multikey Index Behavior

- Array field = multikey index—one entry per array element
- Index size explodes with large arrays—1000 element array = 1000 index entries
- Only ONE array field per compound index allowed
- Can still be useful—just understand the cost

## Partial Indexes

- Index only documents matching a filter: `{partialFilterExpression: {status: "active"}}`
- Smaller index = faster queries AND less storage
- Query MUST include filter condition to use partial index
- Great for "hot" subset of data—index only active users, recent orders

## TTL Indexes

- Auto-delete documents after time: `{expireAfterSeconds: 86400}`
- Only on date field—expires based on that date
- Background thread runs ~every 60 seconds—not instant deletion
- Doesn't work on capped collections

## Text Indexes

- Only ONE text index per collection—plan carefully
- Requires `$text` in query—regular find can't use it
- Stemming and stop words are language-specific
- For complex text search, use Atlas Search instead

## Index Intersection

- MongoDB CAN use multiple indexes on one query—but rarely efficient
- Don't rely on it—single compound index almost always better
- If explain shows `AND_SORTED` or `AND_HASH`, consider compound index
- Exception: covered queries with different projections

## Monitoring and Maintenance

- `db.collection.getIndexes()`—see all indexes
- `db.collection.aggregate([{$indexStats: {}}])`—usage statistics
- Drop unused indexes—they still cost writes and storage
- `background: true` when creating on production—but still impacts performance


path: production.md
content: # Production Configuration

## Before Going to Production

- Test with production-like data volume—aggregations behave differently at scale
- Enable profiling for slow queries—catch problems early
- Set up monitoring: connections, replication lag, disk space, index usage
- Plan sharding strategy before you need it—resharding is painful

## Write Concern

- Default `w: 1` acknowledges primary only—data could be lost if primary fails
- `w: "majority"` waits for majority of replicas—durable but slower
- `j: true` waits for journal flush—most durable, slowest
- Match write concern to data importance—not everything needs majority

## Read Concern

- Default may read uncommitted data—"dirty reads" possible
- `readConcern: "majority"` only reads committed data—safe
- `readConcern: "linearizable"` strongest—but slow and availability issues
- Combine with read preference for full consistency strategy

## Read Preference

- `primary` = always read from primary—consistent but all load on one node
- `primaryPreferred` = primary unless unavailable—slight risk of stale
- `secondary` = read from secondaries—scales reads but stale data
- `nearest` = lowest latency—may be stale, may vary between requests

## Connection Management

- Use connection pool—creating connections is expensive
- Default pool size often too small—tune `maxPoolSize` based on load
- Set appropriate timeouts—`serverSelectionTimeoutMS`, `socketTimeoutMS`
- `retryWrites: true`—handles transient failures automatically

## Replica Set Operations

- Always connect to replica set URI, not individual nodes—automatic failover
- Monitor replication lag: `rs.printReplicationInfo()`
- Test failover in staging—know what happens before production emergency
- Secondary reads: understand that data may be seconds/minutes stale

## Sharding Considerations

- Choose shard key carefully—can't change it easily
- High cardinality key—many possible values
- Even distribution—avoid "hot" shards
- Write distribution matters more than read—reads can be scattered

## Backup Strategy

- `mongodump` for logical backup—portable but slow on large data
- Filesystem snapshots for large databases—faster but requires consistency
- Ops Manager/Cloud Manager for managed backups
- Point-in-time recovery needs oplog—enable with `--oplogReplay`

## Monitoring Essentials

- Connections: current vs available—hitting limit = operations queue
- Replication lag: `rs.printReplicationInfo()`—if growing, investigate
- Disk space: MongoDB doesn't shrink files—plan for growth
- Query performance: enable profiler, review slow queries
- Lock percentage: high lock wait = contention issues


path: schema.md
content: # Schema Design Patterns

## Embedding vs Referencing Decision

- Embed for 1:1 and 1:few relationships—data always accessed together
- Reference for 1:many and many:many—data accessed independently
- Embed when child doesn't exist without parent—natural containment
- Reference when entity is shared across multiple parents—avoid duplication

## Embedding Advantages

- Single read gets all data—no additional queries
- Atomic updates on single document—no transactions needed
- Data locality—related data physically together on disk
- Simpler application code—no manual joining

## Embedding Dangers

- Document growth causes relocation—performance hit on updates
- 16MB limit approached gradually then suddenly—plan ahead
- Duplicated embedded data becomes stale—update complexity
- Can't query embedded documents independently

## Referencing Patterns

- Store `_id` of related document—manual lookup required
- Use `$lookup` for server-side join—but has performance cost
- Denormalize frequently accessed fields—store name WITH reference
- Accept eventual consistency of denormalized copies

## Bucketing Pattern

- Instead of array per document, create fixed-size buckets
- Example: `{sensor_id, date, readings: [...], count: 50}`
- When count reaches limit, create new bucket document
- Great for time-series, IoT, logs—predictable document size

## Attribute Pattern

- Many similar optional fields → hard to index all separately
- Convert to array of key-value pairs: `attrs: [{k: "color", v: "red"}]`
- Single index covers all attributes: `{attrs.k: 1, attrs.v: 1}`
- Query: `{attrs: {$elemMatch: {k: "color", v: "red"}}}`

## Polymorphic Pattern

- Different document structures in same collection—flexible schema
- Use `type` or `kind` field to discriminate
- Partial indexes per type: `{age: 1}, {partialFilterExpression: {type: "person"}}`
- Application code must handle all shapes

## Computed Pattern

- Store pre-computed values to avoid repeated aggregation
- Update computed fields on write—single source of truth
- `$inc` for counters: `{$inc: {viewCount: 1}}`—atomic, no read needed
- Trade write complexity for read performance

## Anti-Patterns to Avoid

- Massive arrays that grow unbounded—time bomb
- Deep nesting > 3 levels—hard to query and index
- Storing large blobs inline—use GridFS
- One collection per user/tenant—operational nightmare


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: clean (LOW)