Graceful fail when removing non-existent groups