Revert "TravisCI should now provide NumPy under Python 3.2"

This reverts commit fdf97c8cd49e1039f6ceffe2185cfb7167dd2411.
Still not working, at least for me. Go back to manually installing
NumPy under Python 3.2
https://github.com/travis-ci/travis-cookbooks/issues/48