Each chunk will call 10,000 times the underlying function. If at the end of the function everything is properly garbage collected the memory should not increase. If the underlying function is updating a cache, however, we may indeed run out of memory. This is something to check, especially in risk calculations. The user can still tune the number of generated tasks by changing the parameter concurrent_tasks. By increasing it, more tasks will be generated and the chunksize will get smaller. So we are not losing control, we are trading the parameter block_size with the parameter concurrent_tasks.

Notice that I have not changed the risk calculators: they inherit from base.Calculator which is using the traditional distribution (i.e. no parallelize).