The same words may mean different things to different people. If you're in a discussion, you may need to be specific about what you and your conversation partners are each talking about. Don't assume that a conversation on these topics can be quick and casual for everyone. For some people, these issues are subjects for dispassionate, intellectual discussion. For others, affirmative action is a deeply felt, very personal issue in which attacking certain ideas equals attacking them personally.
Many people are in between.
Some are not interested in these topics. How other people hear your words is not in your control. What someone hears can be influenced by past experiences, work stress, even the morning's commute. What you can control is what you say and how you say it. Pay attention to your words.