America's legacy of slavery is well-documented. But at the same time, this is the same America that, in the '60s, looked into the mirror and blinked. The consciousness-raising civil rights reforms followed. Things have changed. Things do get better.
People make unthinking remarks.
Often this is ignorance. Yes, it is frequently race-biased ignorance. But more often it is simple, garden-variety ignorance. But ignorant, unthinking remarks do not necessarily mean the speaker wishes harm or, more to the point, has the power to inflict it. The same rules apply to you as to all Americans.