Better fix for the config issue

Need to keep hold on which config path we loaded from so that if we need
to update it, we update the one we loaded.