Worry makes you less effective in everything. If you don't want to do something, then don't do it . Do everything your own way. If you know that you know best, then don't allow yourself to buckle down mentally. There is always more to any person you meet than appears at first sight.
Many people carry an useen and mostly-unused bag of both good and bad tricks.
It is often possible to strike up a way of interaction, useful to you, with the most stupid and ignorant people. Your friends form your primary mental environment. For each person you know and meet, a level of closeness of interaction, between him and you, can be found that is best and most useful to you. Because you and others constantly change, this level is dynamic, and needs to be constantly gradually changed too. Decrease the level of closeness with people who are of no use to you, who irritate you and who suck away your energy.