Van Bergen was married to Hester Houghtaling (1768–1824), the only daughter of Elizabeth (née Whitbeck) Houghtaling and Capt. Thomas Houghtaling, who fought in the second Battle of Saratoga under Van Bergen's father and Lt.-Col. DuBois. Hester's brother, Coenradt T. Houghtaling, was the husband of Peter's younger sister Catharina. They lived in Catskill and were the parents of one son: