The presence of many bees, wasps or hornets flying around your home or the observance of a nest is the main indicator of a problem.
Bees are beneficial to humans because they pollinate plants, whereas wasps and hornets help out by feeding their larvae on various insects, mites or spiders. Therefore, unless the nest is close to a door or activity area in your yard, it is better to avoid the nest than to remove it.