we need to reduce the dataset size because the hash table is going to
take a lot of space on disk