Killing a few caterpillars in the garden will not harm the species, but caterpillars on celery plants may not indicate a serious problem. Because eastern swallowtails do not congregate in large numbers like some butterflies, you may only find a few larval worms on celery.