If contactmanager isn't loaded, don't load contacts