Fix break + autoreload tests

Autoreload setting was being left disabled after the tests causing
random failures in other places.