Bitter gourd should be avoided by pregnant women, people with low blood sugar, and those who are allergic to it. It is also advised to consume bitter gourd in moderation as excessive intake may cause gastrointestinal problems.