You're welcome to leave it out, or replace it with another aromatic, sweet spice. Nutmeg, Allspice, or even Pumpkin Pie Spice would work in its place.