Room temperature is best for this. Take a sip and swallow slowly. Close your eyes and relax. Spend about five minutes drinking the rest of the water in the same way. It really helps to calm your body and relax the muscles in your face and neck where tension can add to your headache.
Dehydration is the common cause of a lot of headaches.
I have found this to be very useful in getting rid of headaches. My daughter had a headache one day, but has trouble swallowing aspirin. I suggested she try this technique. She thought it sounded a little silly. But given the choice of swallowing a pill or drinking water slowly she opted for the water.