To be honest, I'm not actually very sure that all the added complexity is warranted for this use case, running NLTK lemmatizers. I submitted a patch to their tokenization code some time ago to make it four times faster just by fixing a few regexes; similarly I sped up the Porter stemmer in Gensim (same code that's in NLTK also) by 20% by inlining some function calls and using string operations in a smarter way.

So if the NLTK folks fix their performance issues, we're stuck with unnecessary complexity.

What I would find more interesting is a simple threading approach where one thread fetches documents off the disk, and another does the processing, so the CPU and the disk are both at work at the same time. For slow disks, that can help a lot (Vowpal Wabbit does it).