Downey was born on June 24, 1827 in the townland of Castlesampson, Taughmaconnell parish, County Roscommon, in central Ireland, to Denis Downey and Bridget Gately. Castlesampson is 12 kilometres west of the town of Athlone. He emigrated with his family at the age of 14 to the United States in 1842, before the famine years. Settling in Charles County, Maryland, the Downeys joined two stepsisters who had already settled in the U.S. Dwindling family finances forced Downey to halt his education at age 16 and start working to become independent. He apprenticed at an apothecary in Washington, D.C. until 1846.