I stand guard right at the door. If my son doesn't come out in short order, I give a yell in the door. No answer, I go right in. Older boys learn not to dawdle, so mom isn't waltzing into the men's room. This is relatively easy for a mother to get away with.
Dads are better off to ask a woman going into a facility to check up on a young daughter.
When in doubt, always check it out. It's better to err on the side of caution. Teach your children to keep themselves safe. Besides watching small children around swimming pools, beware of lakes, creeks and ponds in parks. Don't depend on older children to keep an eye on the young children.