You can, whole milk will just provide a richer flavor. 2% milk also runs a higher risk of crystallization when freezing.