Absolutely! The poppy seeds don't really affect the recipe that much at all. You will just have a lemon bread then, and it will still be delicious.