Some people might not follow you back because they don't agree with your opinions or they don't want to be associated with you.