Oak worm caterpillars are not poisonous and harmful to humans. But some of the moths can cause breathing problems, skin irritation, and problems in the throat due to the allergy.