Antonio Duvergé Duval was born in Mayagüez, Puerto Rico to a French man, Joseph Duverger, and a French-Dominican, Maria Duval. His grandfather Alexander, was born Nantes. A member of De la Rochejacquelein family, he left France during the Drownings at Nantes for being a royalist. Alexander fought for the French against Haitian General Toussaint Louverture. After the invasion of Santo Domingo by Jean-Jacques Dessalines, he was exiled to Puerto Rico with his son Joseph Duverger and his wife Maria Duval. In 1808, when Antonio Duvergé was one year old, the family returned to Santo Domingo, settling in El Seibo and later in San Cristobal. At that time the Dominican Republic was occupied by Haiti, Jean-Pierre Boyer unified the Spanish island under his rule. The family changed their surname from Duverger to Duvergé in order to sound more Spanish and not to be confused with Haitians who were the enemy at the time.