Razaf was born in Washington, D.C. His birth name was Andriamanantena Paul Razafinkarefo. He was the son of Henri Razafinkarefo, nephew of Queen Ranavalona III of Imerina kingdom in Madagascar, and Jennie Razafinkarefo (née Waller), the daughter of John L. Waller, the first African American consul to Imerina. The French invasion of Madagascar left his father dead, and forced his pregnant 15-year-old mother to escape to the United States, where he was born in 1895.