Actually cause the config to be reloaded and update tests to support changes

Follow-up to #29