His father, who married twice more and died in 1808, was a clergyman for the Church of England and a fierce Loyalist at the time of the American Revolution.
John came from an extended family of Loyalists, with his uncles Edmund Fanning and Parker Wickham being two of the most noted Loyalists from the Long Island area.