You totally can leave off the nuts. In fact, my sister always does since one of her kids is allergic. It works great for her!