need smaller batch sizes / less data reduplication