amueller/joblib — comment on commit 4df1fabdda2b7e00a16b02848b760aa4076a96e5
File: joblib/parallel.py line 513
By GaelVaroquaux:

As I mentioned earlier, this is a big no-no: this forces evaluation of the whole iterable. It will break the 'pre_dispath' mechanism, that is important to avoid explosion of the memory usage.

I think that it would be better to slot in using the job-level dispatch mechanism, rather than the iterable-level dispatcher. In other words, you want to replace multiprocessing's 'apply_async', rather than 'map_sync'.

The good news is that if everything works out, this amounts to providing an equivalent of the multiprocessing 'pool', stored in self._pool, and the rest of the code should be unchanged, as long as it implements 'apply_async', 'join', 'close' and 'terminate'.