Yes, absolutely. It's OK to give room temperature or even cold formula to your baby. The liquid should feel lukewarm, not hot.