sightmachine/SimpleCV — pull request #576 by victsou (2013-09-17 05:06:31+00:00)
Title: Implements ImageSet.multimap

multimap aims to speed up applying one function to all the images in an imageset,
as long as it is cpu-intensive, so that the improvements are greater than the
overhead from the creation of different processes.

Usage:

```python

i = ImageSet("/path/to/directory/with/images")
palettized = iset.multimap(Image.palettize)

#or

i = ImageSet("/path/to/directory/with/images")
skintoneMasks = iset.multimap(Image.getSkintoneMask, **{"dilate_iter":2})
```

I did a brief analysis of the results. At first, we'll just create an ImageSet with 5 images from the sampleimages folder. If you want to try it, remember to change the path!

```python
from SimpleCV import *

#Directory with 5 images from the sampleimages folder.
iset = ImageSet("/Users/victsou/dev/SimpleCV/SimpleCV/multiprocessing_images/palettize/")
```

Common ImageSet operation: applying i.getSkintoneMask to every image in the
image set, grabbing the results and making that a new imageset.

```python
#res = ImageSet([i.getSkintoneMask(dilate_iter=2) for i in iset])
```

victsou-mba@~/dev/SimpleCV/SimpleCV $ time python multimap.py

real 0m0.752s
user 0m0.574s
sys 0m0.179s

#-------------------------------------------------------------------------------

Same operation, but now using the multiprocessing pool with the default number
of processes (which is defined by a call to multiprocessing.cpu_count())

```python
res = iset.multimap(Image.getSkintoneMask, **{"dilate_iter":2})
```

victsou-mba@~/dev/SimpleCV/SimpleCV $ time python multimap.py

real 0m0.817s
user 0m0.592s
sys 0m0.215s

We can see that using the multimap is no better than the common way of doing it.
That's most likely due to the overhead of the multiprocessing being greater than
any improvements we'd get from it.

#-------------------------------------------------------------------------------

Now we're going to try a more cpu-intensive function. This applies palletize to
every image in the imageset.

```python
res = ImageSet(map(Image.palettize, iset))
```

victsou-mba@~/dev/SimpleCV/SimpleCV $ time python multimap.py

real 0m18.991s
user 0m17.954s
sys 0m1.030s

#-------------------------------------------------------------------------------

Finally, we'll do the same cpu-intensive thing, but now with multimap. This
image doesn't take any argument, so you obviously don't need to pass any
arguments.

```python
res = iset.multimap(Image.palettize)
```

victsou-mba@~/dev/SimpleCV/SimpleCV $ time python multimap.py

real 0m12.234s
user 0m25.562s
sys 0m1.006s

Yay! That's a 35.5 percent decrease in run time (18.991s to 12.234s). You can see that the time on
the user row increased. In rough terms, it means that we used 25.562seconds of
CPU in 12.234 seconds! That's the beauty of multiprocessing!

This comparison can be easily verified with the code on this gist: https://gist.github.com/victsou/6590134.

I also created a test, but it unfortunately doesn't behave well inside tests.py, but works fine separately. https://gist.github.com/victsou/6590126