Born in Vugizo, in southern Burundi, Niyongabo won a silver medal in the 1500 m at the 1992 World Junior Championships and also came fourth over 800 metres. He competed in his first senior tournament the following year, but was eliminated in the 1500 m semi-finals of the World Championships in Stuttgart. Niyongabo soon became one of the top 1500 m runners in the world, winning several major races in 1994 and 1995. He also won a bronze medal at the World Championships in Gothenburg, finishing behind Noureddine Morceli and Hicham El Guerrouj.