Birds, rabbits and chipmonks are the spawn of the devil. They'll pick your fruit before you do. Chipmonks chew the branches off of the trees to get the fruit to the ground. Rabbits will strip the bark off of various trees in the winter. Aphids, worms and other small critters will make life difficult at times.
Black knot is a problem on plums as is peach leaf curl on the peaches and nectarines.
The only relief is a lime sulphur spray done before the leaves open. You will need to tie the branches to your home so that they can't rub. Spraying of herbicidal oils should be done before the leaves open. You don't need to use special oils. Soap is a contact poison and will kill only on contact.