Don't try to load configs if they don't exist

This prevents a 'grep *' from happening in the current working directory.