Just tell visitors that your pediatrician is very strict about everyone washing hands before touching the baby. Your concern over the health of your newborn is a sound one. Be up front and ask people to wash their hands before picking the baby up. When going for a walk, place a blanket over the hood of the stroller so people cannot just walk up and reach in to touch your baby. At large family gatherings, my husband monitors his relatives and speaks up when necessary, while I do the same thing with my relatives.
Small children are the most difficult to keep away.
If you see several small ones approaching, pick the baby up. Excuses can be made such as you are late for a feeding, or the baby has a dirty diaper. My babies were full-term, healthy and breastfed, and I never asked anyone to wash his or her hands before touching them. I didn't consider any extra precautions necessary. If your baby's immune system is compromised, explain that to people.