Gracefully handle removing user from priv cache

If a user is not in the dict, a KeyError could occur throwing a nasty error into the room