You can, but they add a necessary texture difference in the peanut butter layer. Without the graham cracker crumbs, the peanut butter layer can stick to the roof of your mouth.