You can use either! For a creamier cocoa, use whole milk. For a lighter cup, use plain water.