No. However, the group that holds the power, Whites gets to exercise their prejudices more readily. Racism is usually defined as prejudice plus power and so most often applied to White people as the main power holders in our society.  Racial prejudice is often the more widely applied term to describe the sin all humanity is subject to it.