Make sure we throw the no-config-error before trying to require a non-existing config file