If you don’t like poppy seeds, you can certainly leave them out, however, I highly recommend them.  They add a lovely subtle crunch and sweetness.