No way! Unless you have a sensory sensitivity to the peel, it is actually great to leave the potatoes unpeeled. Particularly with yellow potatoes, you will hardly even be aware that the peel is there once everything is mashed together.