process test files in chunks of 1000