SimTk/msmbuilder — pull request #48 by rmcgibbo (2012-09-16 11:54:59+00:00)
Title: Testing

I think we've all realized in the last week that our testing is not that good. In no particular order

- the testing code itself is really crufty.
- some of this is garbage that I wrote, like `test_msmlib.py`
- most of it is in `TestWrappers.py` though
- the `TestWrappers.py` code is mostly loading files from disk -- the amount of logic in there is a lot less than the number of lines of code.
- the tests are distributed in two places, and the reference data are in 3 places
- the `TestWrappers.py` tests annoyingly all depend on one another, meaning that if you break one thing you get a million failures. this makes it hard to figure out what you broke / easy to just ignore the tests
- if you kill the tests halfway through their execution, you have to clean everything up yourself because the "working directory" is not reliably cleaned.
- the tests reference the tutorial directory, so they can only be run from the project directory
- they take a pretty long time to run
- more?

Proposal:

the test wrappers should only have to be this long:

```
from msmbuilder.testing import eq, get

def test_BuildMSM():
pops, mapping, tprob, tcounts, assignments = BuildMSM.run(LagTime=1, get('Assignments.h5'), Symmetrize='MLE')

eq(pops, get('Populations.dat')
eq(mapping, get('Mapping.dat')
eq(mapping, get('tProb.mtx')
eq(mapping, get('tCounts.mtx')
eq(mapping, get('Assignments.Fixed.h5')
```

- the key is `eq()` and `get()`. I think both of these can be done well enough that this pattern will work for 90% of stuff.

- `eq` needs to ensure the two arguments are the same type, and then dispatch off to a few specialzed methods based on the type. for numpy arrays, use `assert_array_equal` if they're ints(`dtype.kind = 'i'`) and `assert_array_almost_equal if they're floats. For trajectories or sparse matrices or whatever other types we often have, we can have special handlers for those.

- `get()` will find the file in the ReferenceData part of the package, that will get installed with the egg, and then try to apply some heuristics to load up the data. I think 90% can be done just by inspecting the extension, and then maybe get can provide an optional kwarg `just_filename` if you need to do the loading yourself.

- finally, I think we should make every wrapper test be independent -- it only makes it complicated when they depend on each other, and then you need to care about their ordering, etc. And then you could also just test a single script, without needing other scripts to build its input files.