Don't build 'scipy' in the tests

That should save considerable time on Travis.