Depending on the pathogen, if only one or two such microbes are ingested, hopefully the body's immune system will identify and defeat the invader. If a few more microbes are ingested, that might make some people sick. In other words, if everyone in a group swallows the same dose, some people may get sick while others do not. Animals, like humans, tend to seek clean water sources. As you know, clean water is less likely to cary pathogens.
Different microorganisms make different species sick.
A virus that makes human beings sick might not have an effect on a deer or a raccoon.