Democrats have Lyndon LaRoach. Republicans have David Duke. Evangelism has the Jims, Baaker and Swagart. Most of us know enough about these institutions to recognize a misfit when we see one. The person who knows only one or two people called libertarians is likely to judge all libertarians by the one or two.
Libertarians have another smear on their image.
Liberty in the Land of the Free has slipped to the point that a return to liberty would require major changes. It is human nature to resist change. Many would find the changes proposed by libertarians frightening, even if they understood the proposed changes. Even changes that eventually gain near universal acclaim were unwelcome when they burst upon the scene. The Luddites destroyed machines and factories because they felt threatened by industrialization.