This can happen if your poppy seeds aren't ground finely enough or if you haven't cooked the mixture for long enough. Try adding a bit of honey or spoonful of jam for a quick fix.