Normally they damage only a few fruits and there are plenty left over for the family and friends. The small bugs in the fruit are scavengers and help to dispose of it. They are nature's garbagemen. I tried soap spray to no avail. A. Sevin is about the only spray safe for insects chewing on citrus leaves.
Chances are it is nocturnal caterpillars, leaf cutter bees or beetles chewing on the foliage.
Damage is very minor from chewing insects, so I am not inclined to worry excessively about them. I personally would not spray. If you do spray, follow label directions exactly. Do Not Spray if the plants are blooming . Sevin is lethal to bees.