You can give your baby water but it is usually unnecessary and may not help your baby sleep all night. Instead, we recommend you get your baby used to sleeping straight through without expecting milk or water.