Guarantee consistent state when changing config.

This refactoring guarantees the application is not left in an
inconsistent state in case an error is raised during the test run.