No. You don't HAVE to. But you can if you want to. They are totally edible. I would not bother scraping them out in this particular recipe.