Removing user association with votes.