Van Bergen was born in Paris on April 15, 1871. He was the second of three children born to wealthy Americans Julia Augusta (née Peirson) Van Bergen (1843–1897) and Anthony T. Van Bergen, the Paris representative of Arnold Constable &amp; Co., the Equitable Life Assurance Society, and an American Commissioner to the Paris exhibitions of 1878 and 1889. He had two siblings, an elder brother, Dr. Charles Peirson Van Bergen, a University of Paris trained doctor, and a younger sister, Alice Van Bergen, who married Count Otto of Schloss Varchentin in 1900. His niece, Countess Antoinette Julia Grote, married Prince Dietrich of Wied, a son of William Frederick, 6th Prince of Wied and Princess Pauline of Württemberg (the elder daughter of King William II of Württemberg), in 1928.