Guarantee consistent state when changing config.

This refactoring guarantees the application is not left in an
inconsistent state in case an error is raised during the test run.
Merge pull request #528 from caike/master

Guarantee consistent state when changing config.