Panter-Downes was born to Major Edward Martin Panter-Downes (died 1914 at Mons) and Marie Kathleen Cowley who was of Irish origin.