joliver/EventStore — pull request #81 by CaptainCodeman (2011-10-29 18:19:19+00:00)
Title: Mongodb optimizations

MongoDB Optimizations to reduce both storage and index sizes and make Commits collection more shard-friendly.

These savings reduce storage space & disk IO, network traffic and, most importantly, RAM usage so a larger working-set can stay resident in memory.

See: http://blog.boxedice.com/2010/10/23/on-shortened-field-names-in-mongodb/

While it does make the data a little less human-readable which would normally be a problem when using the schema directly, for the EventStore this is not such a consideration as it is more of a black-box.

One things to note: the Dispatched flag has been changed to a 'queued for dispatch' datetime field which is removed when dispatch is complete. A sparse index is used to only return the commits that need to be dispatched - this means this index is very small and will only contain a few entries (compared to the boolean flag which had very low cardinality). This does mean that MongoDB 1.8 is now required.

# Mapping
The field mappings are shown below:
## Commits Collection:
```
_id.StreamId => i
_id.CommitSequence => n
CommitId => _id
Headers => h
Payload => p
Dispatched => q
CommitStamp => s
Events => e
StreamRevision => r
Payload => p
Body => b
Headers => h
```
## Streams Collection:
```
HeadRevision => h
SnapshotRevision => s
Unsnapshotted => u
```
## Snapshots Collection:
```
_id.StreamId => _id.i
_id.StreamRevision => _id.r
Payload => p
```
# Savings
The actual savings work out to roughly:

38 bytes per stream + 60 bytes x number of commits + 28 x number of events in the commit (and some minor savings for the snapshots).

On my project I see savings of about 17% for the storage and 21% for the indexes.
This is with some quite large events - if the event payload is smaller then the savings could easily be around 30% - 50% (especially if the field names in the event mappings themselves are also trimmed).

# Example
Here's an example of the before and after versions:

## Old Format
``` json
{
"_id" : {
"StreamId" : new BinData(3, "+9IwFskUQE6jDBQWdEnjRw=="),
"CommitSequence" : 1
},
"CommitId" : new BinData(3, "y8ROiHM0/kW6Et8KHUo9Mw=="),
"CommitStamp" : ISODate("2012-10-29T16:27:44.215Z"),
"Headers" : {
"A header" : "A string value",
"Another header" : 2
},
"Events" : [{
"StreamRevision" : 1,
"Payload" : {
"Headers" : { },
"Body" : {
"_t" : "SomeDomainEvent",
"SomeProperty" : "Test"
}
}
}, {
"StreamRevision" : 2,
"Payload" : {
"Headers" : { },
"Body" : {
"_t" : "SomeDomainEvent",
"SomeProperty" : "Test2"
}
}
}],
"Dispatched" : true
}
```
## New Format
``` json
{
"_id" : new BinData(3, "y8ROiHM0/kW6Et8KHUo9Mw=="),
"i" : new BinData(3, "+9IwFskUQE6jDBQWdEnjRw=="),
"n" : 1,
"h" : {
"A header" : "A string value",
"Another header" : 2
},
"s" : ISODate("2012-10-29T16:27:44.215Z"),
"e" : [{
"r" : 1,
"p" : {
"h" : { },
"b" : {
"_t" : "SomeDomainEvent",
"SomeProperty" : "Test"
}
}
}, {
"r" : 2,
"p" : {
"h" : { },
"b" : {
"_t" : "SomeDomainEvent",
"SomeProperty" : "Test2"
}
}
}]
}
```
# Migration Script
The 2011-10-28-migration.js script can be used to convert an existing EventStore to the new format (see contents of file for execution instructions).