flush to disk + tests
generating 500k records down from 28 min to 10 sec by writing every 5k records to disk and not growing memory size beyond that