All of his sons joined the British Army and all but one died while serving. Their eldest son, William Livingstone, was born in 1791, and served alongside his father in the Peninsular War before dying at the Battle of Waterloo in 1815. Their second son, Alexander Watt (1793–1849), was a lieutenant-colonel of the Royal Engineers, and their third son, Thomas Congreve (1799–1853) a lieutenant-colonel of the Royal Artillery. Their fourth son, Frederick Holt (1801–1871), was major-general and colonel of the 95th Regiment of Foot, and the Governor of South Australia. The fifth son, George Mountain Sewell (1802–1825), was a lieutenant in the 26th Bengal Native Infantry, served as adjutant in the First Burmese War, and died on passage to Chittagong. Robe's daughters were unmarried. The youngest, Vimiera, died in December 1893 at Woolwich. She presented to the Royal Artillery Institution at Woolwich all the medals, orders, and decorations of her father and eldest brother, together with miniature portraits of each of them.