- split into chunks of 500000 lines each to avoid heapspace error