The poppy seeds do not contribute much when it comes to flavor, but add in a ton of texture. If you are not a fan of the texture, feel free to leave them out. It won't change the flavor of the cake at all.