maxogden/dat — pull request #105 by oncemoreification (2014-05-04 21:07:54+00:00)
Title: Ws updates ii

This is interesting. I found that what was actually causing the performance hit was not the package size, but the fact that 2 key shortcuts were no longer taken.

1) writeStream utilizes the `override` option to skip db checks

2) writeStream manages the schema on its own right now, and for csv data in particular it only checks the schema once (knowing that the csv is tabular)

Additionally, I changed how the row count is updated so that count increments are batched before being written to the db.

This pull addresses https://github.com/maxogden/dat/issues/86 and also https://github.com/maxogden/dat/issues/99 (kinda)

In terms of performance, I'm seeing about a 5% performance hit on my machine, which seems tolerable?