fabric/fabric — pull request #877 by sorccu (2013-04-02 05:12:06+00:00)
Title: Substantial speedup of parallel tasks

(Also #868, but my `hub` client refused to let me join the existing issue. I'm probably using it wrong)

I came across an issue where issuing a simple command in parallel would often take twice as long as serial execution. This would get even worse with nested tasks (run with `execute()`. Mostly it just seemed to be waiting for something. Reading the source I realized the same thing #868 describes.

As it turns out, the timeout in `_fill_results ` doesn't seem to be necessary at all. `_fill_results` (which contains the blocking `self._comms_queue.get(timeout=1)`) is accessed twice, once inside the run loop and once when it's done. Inside the loop, we don't care if there are any results, we are only trying to get them out of the way. So, absolutely no need to block here. Also, after the loop, all the workers will already have finished, which means that there's no need to wait for more results here either. So, I simply changed it to `.get_nowait()`, and got the following results on a 5-node deploy:

fab -P -- echo 1 3.93s user 0.12s system 283% cpu 1.432 total
fab -- echo 1 2.51s user 0.07s system 73% cpu 3.502 total

Before this patch:

fab -P -- echo 1 3.89s user 0.11s system 62% cpu 6.448 total
fab -- echo 1 2.50s user 0.07s system 76% cpu 3.343 total

The increase in CPU usage seems a bit scary, but in reality the peak usage doesn't really change. It looks lower in the "before" version because for the last 5 seconds it was simply waiting for the blocking timeouts, bringing the number down.

For my original deploy (with all service restarts etc replaced with simple echos), and `pool_size=2`, the results are (see the total):

# w/o patch
fab deploy 20.67s user 0.45s system 37% cpu 56.827 total
# w/ patch
fab deploy 21.20s user 0.48s system 177% cpu 12.224 total

With tweaked deploy settings (calling methods directly rather than via `execute()`, i.e. making Fabric see fewer tasks):

# w/o patch
fab deploy 15.39s user 0.33s system 61% cpu 25.643 total
# w/ patch
fab deploy 15.78s user 0.35s system 162% cpu 9.915 total

For fun, a fully serial run would be

fab deploy 7.45s user 0.20s system 48% cpu 15.742 total

A parallel run with a `pool_size=5`

# w/ path
fab deploy 18.96s user 0.35s system 311% cpu 6.193 total

Of course, if the tasks actually did something that took some time, the parallel version would be even more favorable.