Most of these people would say that the company has a positive impact on them. Our economy is based on voluntary exchange. No company can force customers to buy from it, or demand that workers work for it or insist that suppliers sell it supplies. Any particular company has both positive and negative effects on society. The very fact that a company makes a profit is a sign that it is making a contribution to society.
Profit is a reliable measure of a company's overall efficiency.
A company only receives money from the sale of its stock when the stock is first issued. Unless your mutual fund is buying the stock at the initial offering, your money is going to another investor, not the company itself. Avoiding purchase might make you feel good, but beyond that it would be difficult to detect any result. You might say that even though being socially conscious has no discernible effect on companies' behavior, you simply do not want to be associated with a company that makes, for example, cigarettes. But the complexity of today's corporate ownership structures makes it hard to avoid dealing with the cigarette makers.