" As you know , if you have too many white people at your rally , then your cause is racist , but if you have too many people of color at your rally , then you must be asking for something : special rights -- like eating in restaurants or piggy back rides -- something that we as a society are not ready to give , " he added .