His paternal grandparents were Maria (née Salisbury) Van Bergen and Col. Anthony Van Bergen, a descendant of Mayor Dirck Wesselse Ten Broeck. His grandfather led the 11th Regiment of the Albany County militia (known as Van Bergen's Regiment) in the Revolutionary War. His maternal grandparents were Elizabeth (née Whitbeck) Houghtaling and Capt. Thomas Houghtaling, who fought in the second Battle of Saratoga under his paternal grandfather. His maternal uncle, Assemblyman Coenradt T. Houghtaling, was the wife of his paternal aunt, Catharina Van Bergen. and were the parents of one son: