starting to rewrite data collection to work in incremental batches, storing in the DB after every batch