You have to add tests to make sure the numpy and scipy.sparse tests are run only when those packages are installed.  See above in `test_matrixutils.py` for examples.