Shrugs have always been poor manners, or at least that is what we were taught. When people are asked a question, they should give a feasible verbal answer. The rudeness is not the fact of the shrug being nonverbal but rather that it shows disrespect.