Gracefully handle removing user from priv cache

If a user is not in the dict, a KeyError could occur throwing a nasty
error into the room.wq