As I've noted before, by that logic, Thomas Jefferson, Strom Thurmond and every other white man in power who fathered children with black women weren't racists, either.
I'm not so sure that those black women, their families or any of the other black people they were demeaning and discriminating against at the same time would agree.