No you don't have to but all those crumbs are what make this crumb cake new york style. They are meant to all be used.