[MRG] Test common refactor

This actually makes the tests faster and hopefully it will make the actual testing file way better to maintain.
+1 to make something like in `sklearn/metrics/tests/test_metrics.py`