Young children and pregnant women should definitely avoid caffeine. Also, people who suffer from huge bouts of anxiety and diarrhea should also avoid consuming caffeine.