After graduating in English literature from Makerere University in 1972, Alibhai-Brown left Uganda for Britain, along with her niece, Farah Damji, shortly before the expulsion of Ugandan Asians by Idi Amin, and completed a Master of Philosophy degree in literature at Linacre College, University of Oxford, in 1975. After working as a teacher, particularly with immigrants and refugees, she moved into journalism in her mid-thirties. She is married to Colin Brown, former chairman of the Consumer Services Panel of the Financial Services Authority. The couple have a daughter, and Alibhai-Brown has a son from a previous marriage.