nipy/dipy — pull request #244 by gabknight (2013-09-11 20:26:35+00:00)
Title: Parallel peaks from model 

Hi, this PR is for multiprocessing the peaks_from_model function. This is the first effort to parallelize dipy functions and we hope that will give us ideas for the future on this topic.

I added the function peaks_from_model_parallel in dipy/reconst/odf.py
- The function divises the input data into chunks and calls peaks_from_model for each chunk
- It uses the multiprocessing.pool.map function to parallelize the model fitting
- examples: doc/examples/reconst_csa_parallel.py, doc/examples/reconst_csd_parallel.py
- tests: test_peaksFromModelParallel() in dipy/reconst/test/test_odf.py

This shows a speedup of about 4.5 using 8 cores for csd model (10 mins for the full brain!!!) and about a factor 3 using 4 cores for csa model. Performance decreases using 8 cores for csa and for small dataset due to the multiprocessing overhead. Examples are done using 4 cores.

In order to make this parallelization happen we had to fix an important bug in dipy that concerns most models.
the multiprocessing.pool.map function requires that all input are serializable (can be pickled). The models using the @multi_voxel_model class decorator cannot be pickled and therefore cannot be parallelized.
As a solution to this problem, the class decorator have been changed to a method decorator, turning a single voxel model fit definition into a multi voxel model fit definition. With this change both the API stays simple and the object is parallelizable. This should also fix a problem with the automatic documentation of the docstrings of models using that decorator.