You absolutely can! We love the richness and the crunch of the nuts in the topping, but if someone's allergic or you just don't have any nuts on hand, you can totally leave them out.