According to Narada-bhakti-sutra. 7-8, you can get rid of worldly desires by avoiding worldly activities and also by developing desire for Bhagavan.