He served as prime minister from 1999 to 2004 and briefly in 1996 and 1998 .