Of course, there is first the brutal idea of privilege, that of the strong over the weak. There is the privilege of wealth. If a man has more money than another, he wants a little privilege over those who have less. And last of all, and the worst, because the most tyrannical, is the privilege of spirituality. If some persons think they know more of spirituality, of God, they claim a superior privilege over everyone else.
None can be Vedantists, and at the same time admit of privilege for anyone.
The question of caste and its relation to privilege was one with which Swami Vivekananda struggled long and hard. We can see from some of his early letters to Pramadadas Mitra, a learned scholar for whom Swamiji had great respect, how troubled Swamiji was with certain aspects of caste. One of the letters which Swamiji wrote to him from the Baranagore Math raised several questions with regard to caste, specifically concerning hereditary caste and the rights of Sudas to study the scriptures. Swamiji's opinion on caste in general is not always entirely clear. In some of his writings and lectures, especially when responding to criticisms of the caste system from the West, he defends the concept of caste as representing a sensible and necessary division of labor.