Move tests around to properly skip tests that need NumPy and SciPy