Switched to using postgres instead of SQLite for the DB as I wanted to try something else as well as all the small bits of text we've got a full stored proc supporting environment could prove to be useful for the deduplication stuff.
Ripped out the deduplication stuff that was disabled earlier as we aren't going to do it here anymore.
Ripped out the code around file splitting. This was commented out earlier and is no longer needed so lets get rid of it.
Added some flags for processing that we will filter on after the log files are loaded. Might consider adding a property to identify some auditing for what lines are done as part of which log files.
restricting the default stats to only pull back the past month to avoid causing stack overflows for the poor ruby interpreter. :p