synthicity/synthpop — pull request #14 by jiffyclub (2014-09-05 05:34:20+00:00)
Title: IPU Performance Upgrades

After profiling the IPU code it became clear that Pandas was slowing things down. Indexing Series and doing arithmetic on Series are both quite slow, you apparently don't want that happening in looping code. So I changed things around so at the beginning of `household_weights` it converts everything to NumPy arrays, runs the algorithm, and then passes back a Series. After all the tuning my test case when from running in 4.1 seconds to 0.15 seconds. So the lesson is here is to not use Pandas in iterative numerical algorithms, instead drop back to NumPy.

As an illustration, here's a profile of `household_weights` before I started tuning:

![screen shot 2014-09-04 at 5 26 48 pm](https://cloud.githubusercontent.com/assets/920492/4160550/c75a6db6-34bd-11e4-9956-0c7897968eb8.png)

And here's the after picture:

![screen shot 2014-09-04 at 5 26 53 pm](https://cloud.githubusercontent.com/assets/920492/4160556/d8853c4c-34bd-11e4-96a1-89465300e2e9.png)

Pretty much all the stuff that disappeared is Pandas indexing, replaced with NumPy C fastness.