Termites has underground colonies, can dig tunnels and move towards location where they can find food which is cellulose (found in wood). Termites can eat into a tree and weaken the trunk creating a risk of it falling onto your house. They can also move towards your house seeking wood too. So, the answer is YES, termite infestation in your garden is a matter of concern.