Comets are little bits of ice left over from the beginning of our solar system.  They formed in the same disk of gas and dust that formed the eight planets, but because they formed in the very cold regions far away from the sun, they include a lot more ice.
Astronomers are still measuring comet composition and running computer simulations to learn exactly where and how comets formed, but they likely formed in the same region of our solar system where Kuiper Belt Objects like Pluto orbit today, and got scattered out onto huge, elliptical orbits when the giant planets changed positions very early in the history of the solar system.