It seems to me that the second argument to _batch_iter should be a function of the length of raw_document and the number of jobs. Indeed, having too many iterations is counter productive, as you are going to fill in your memory with pending jobs.