Ernle married firstly under a settlement made on 1 March 1646, Susan Howe, daughter of Sir John Howe, 1st Baronet of Little Compton, Withington, Gloucestershire and had two sons and seven daughters. He married secondly on 19 September 1672, Elizabeth Seymour widow of Charles Seymour, 2nd Baron Seymour of Trowbridge and daughter of William Alington, 1st Baron Alington of Killard. He was the father of Sir John Ernle, a notable naval officer of the Third Anglo-Dutch War.