Yes, you can certainly leave out the ginger. The ginger really adds a large burst of flavor that sets them apart from other pumpkin desserts, we highly recommend you try it.