A downvote isn't the removal of a vote it's a vote in the negative direction.