We agree that society should be blind to certain criteria - race, color, ethnicity, whatever. Ah, but we're not there yet, goes one familiar argument. Agreed, and the way to get there is to prosecute rigorously every instance of discrimination, not to institutionalize it. Doing away with race-based remedies is the only way to make us look at those who have been discriminated against, not as members of a victim class, but as individuals.
Affirmative action continues the victimization, but merely designates a different group of victims.
And it perpetuates the division, the hostility, between the favored and the unfavored. One of the insidious aspects of reverse discrimination, accounting for the eagerness to do away with the policy, is the difficulty of guessing what will satisfy enforcers. That is, in a workforce of twenty people, it might be that population statistics call for 3.2 to be of a particular race. You may be prosecuted if you only have three, but if you have four then one of the other groups is underrepresented. Suppose an applicant is gay, black, and Jamaican.