Added some `@no_dry_run` tests.

We weren't testing the case where dry_run is *not* enabled.

We ought to refactor more of our test suite, so that it's easier to understand *what* its testing; right now there's too much redundant code to easily get an idea of what all of the cases we're testing are.

A note about the "actual test cases" at the bottom - it would be nice if we could turn this sort of thing into nested for loops, so that in larger cases, it's obvious at a glance that we're getting all possible combinations. It's really clumbsy to do this though, since pytest finds tests by looking for functions starting with `test_` - generating them dynamically is a pain.