If you don't have whole nutmeg, then you can totally skip it. But, store bought ground nutmeg does not come anywhere close to freshly grated nutmeg.