People who are pregnant, breastfeeding, or have autoimmune conditions or thyroid disorders should avoid ashwagandha unless advised by a healthcare provider.