When you first start performing an effect, you naturally have to concentrate a bit more on every move than when you become more familiar with it. This can transmit itself to your spectator without your even being aware of it. In addition, no matter how much you've practiced, you will always be susceptible at first to the distractions posed by performing a trick for a real person. As you get more and more experience in performing a trick, you will get more and more skillful at it and you will usually find its deceptiveness increasing steadily. You are performing for the wrong people.
Some people see it as a blow to their ego to be fooled by a magician.
They may grab at the props or try to view what the magician is doing from an inconvenient angle. But no matter how you handle it, some people will still see your magic as a challenge to their ego. Fortunately, as an amateur magician, you have the luxury of choosing your audiences. If a spectator proves confrontational, simply move on to somebody else. There are plenty of people in the world, and no shortage of those who will make appreciative spectators.