Not at all. I personally like the bit of crunch they add to the streusel, but if you dislike nuts in your baked goods or have a food allergy, then simply eliminate them.