Born in Kemeloi in Kenya's Marakwet District, Kipyego was drawn to distance running by his older brother, Christopher Kipyego, who was a marathon runner. (Their younger sibling, Sally Kipyego, would also become a professional runner.) He began competing in the 3000 metres steeplechase as a teenager, choosing the same specialist event as locally born world champion Moses Kiptanui. His first international selection for Kenya was the 1999 World Youth Championships in Athletics, where he came eighth in the 3000 metres flat. He joined Colm O'Connell's St. Patrick's school in Iten in 2000 and success followed shortly after: he was the silver medallist in the steeplechase at the 2001 African Junior Athletics Championships behind Ezekiel Kemboi.