You can...but I hope you don't! The pumpkin adds a nice creaminess and a very subtle flavor. If you really don't want to add it, you can leave it out.