Do not run tests twice when DEBUG is set to true (related: issue 33)

Hey!

Apparently it is possible to simply disable reloader, so tests don't run twice when DEBUG is true. I think this makes issue 37 less severe.