You still need to test for scipy (and then raise SkipTest if not available).  This is why you are getting the test failures.