Waller was born in Galway, Ireland. His family were descendants of Richard Waller, a lieutenant in Cromwell's New Model Army during the conquest of Ireland, who was awarded Castle Cully and its lands near Newport, County Tipperary in 1666. Richard Waller's great-grandson William Thomas Waller married Eliza Augusta Guinness, the granddaughter of Arthur Guinness. His eldest son George Arthur Waller became Guinness' Chief Brewer and engineer, while his fourth son Francis Albert Waller was the managing director of Banagher Maltings (F.A. Waller &amp; Co. Ltd.) in Shannon Grove, County Galway, near Banagher, County Offaly. Francis Albert Waller married Frances Otway on 7 September 1872, and they had nine children, of whom Albert Gregory Waller was the youngest.