It's probably due to a number of different reasons. So in the end, there's less water left to cool off. The temperature in the water isn't the same everywhere. As the hot water cools, the hotter parts of the water and the cooler parts of the water actually end up swirling around each other, helping it to cool down. This happens less in the cool water since there's less temperature difference.
Hot water has less gases disolved in it than cold water does.
It seems that the gases in the cold water actually prevent it from cooling down as quickly. For example, if you have two metal buckets of water, it's more likely that the cooler one will freeze first whereas if you have two thermoses with only the lid open, the warmer one may freeze faster. An interesting example of how this effect is used is in making ice cream. For a long time, many ice cream makers have made their ice cream using warmed milk since it can be made to freeze faster than cold milk. This article has all sorts of references to scientific work on this as well as a wonderful history of how the effect was discovered and why it's named after a Tanzanian high school student.