Weld was born in Smithtown, New York. Weld's father, David (1911–1972), was an investment banker; his mother, Mary Nichols Weld (1913–1986), was a descendant of William Floyd, a signatory of the U.S. Declaration of Independence. His ancestor Edmund Weld was among the earliest students (Class of 1650) at Harvard College; eighteen other Welds have attended Harvard, and two Harvard buildings are named for the family. A distant cousin, General Stephen Minot Weld Jr., fought with distinction in the Civil War.