Betsy was born in Baltimore, Maryland on February 6, 1785. She was the daughter of Dorcas (née Spear) Patterson (1761–1814) and William Patterson (1752–1835). Her mother was the daughter of a Baltimore flour merchant and her father, an Irish-born Presbyterian who came to North America from Donegal prior to the Revolutionary War, was the second wealthiest man in Maryland after Charles Carroll of Carrollton, the last surviving signatory of the Declaration of Independence of the United States.