If your baby has a history of allergies, especially to orange veggies, you should consult your pediatrician before starting sweet potatoes. If your baby does not have a history of allergies, watch for signs of an allergic reaction such as hives, swelling, difficulty breathing, or vomiting after eating sweet potatoes.