Make sure to run tests both in a normal and a broken environment

From conversation in #179