They are afraid they will become the target of hostile attitudes. The same holds true for the Confederate flag. Threats of boycotts have already been aimed at merchants and others who don't follow the accepted line. Boycotts, of course, would likely hurt the innocent. Free speech isn't so popular then.
Free speech is for all, though.
Even for those we don't like or with whom we disagree. In some cases. Sometimes, it's whites who use stereotypical thinking about blacks to prejudice frank and free discussions. Automatic charges of racism over anything about blacks makes them no different from biased people of any color or ethnicity. There are blacks who break laws and create all kinds of problems in schools.