Typically, yellow jackets do not cause structural damage to homes. They might, however, build nests in attics or walls and defend them. On occasion, the pests chew through drywall to enter living spaces.