Add integration test for configuration loading.

After we had merged #356 I realized that while the new way of configuration loading was tested with unit tests, there certainly was an integration spec missing. This PR introduces a proper integration which I believe covers most if not all use cases.