Make sure config directory is prepared