As long as the damage is from insects and not from a fungal or bacterial issue, you should leave the leaves on the plant. These leaves are still able to create food for the plant so there’s no harm in leaving them. If holes in the leaves are caused by anthracnose, angular leaf spot, or another infection, the leaves should be removed to help prevent the spread. The leaves will eventually fall to the ground where spores can seep into the ground, or be spread to other plants.