Get rid of scipy dependency, since it takes like 10-15 minutes just
to build scipy.