You might think that the lichen is damaging your tree, but that's not true. Lichens never do any actual harm. If you wanted to, you could remove the lichen with a stiff brush. But it would come back and that wouldn't help your problem. You see, when you find lichens on a tree, it indicates that something is wrong.
All trees have their own natural defenses.
A healthy tree is perfectly capable of keeping lichens off its branches. However, when a tree is weak, a lichen is able to establish itself. For example, dogwoods do best when planted in the shade but sometimes people plant them in full sun. The bright sunlight on this plant tends to stress it out. Therefore, a stressed dogwood often has lichens on it.