Shuffle before write, remove duplicates, performance improvements.