There is always a reason for concern, but depending on location, size, and spread of disease, it is ultimately a case by case basis. If you notice a hole/cavity, contact an arborist for a free estimate.