He was born on June 20, 1937, the son of Capt. John Davidson Hill Hoyt (1898–1943), who served with the 307th Bomb Group, 372d Bombardment Squadron and was killed in a crash off the coast of Kauai, Hawaiian Islands. His paternal grandparents were William B. Hoyt, an executive with the Pierce-Arrow Motor Car Company, and Esther Lapham (née Hill) Hoyt. His aunt, Esther Hoyt, was the wife of Ansley Wilcox Sawyer, a nephew of Ansley Wilcox (a friend of Theodore Roosevelt whose home on Delaware Avenue was the site of Roosevelt's inauguration following the assassination of President William McKinley).