Bannon was born in Maryland on May 1, 1874. His great-grandfather was an officer in the rebellious Irish army during the Rebellion of 1798. His father, Michael Bannon, was born in County Tyrone, Ireland in 1827 and emigrated to the United States, where he arrived in Baltimore in 1847 with only ten cents. He worked as a teacher and eventually received an education in law and established a real estate business. Michael Bannon entered politics in Anne Arundel County and served as a Maryland State Senator. His brother, James P. Bannon, was a prominent Anne Arundel County politician and lawyer.