added regular flushing of indexes at batch size (10M)