customerio/esdb — pull request #3 by jrallison (2014-10-02 19:24:42+00:00)
Title: Stream

ESDB is a write once, static file format. ESDB stream is a variation on the pattern which writes a queryable, append-only, binary log file format.

This allows for writing a stream of events, while still being able to efficiently query the log for indexed events.

For each log file, index information is kept in memory until the log file is closed, when they're written to the end of the file. Using a series of timestamped log files, closing and creating a new one when a certain threshold is reached, will allow efficient querying of events, while maintaining a fairly low memory overhead.