Bryantown stands on land known as Boarman's Manor, a manor granted to Major William Boarman in 1674. Bryantown is associated with the flight in 1865 of President Abraham Lincoln's assassin John Wilkes Booth, and its St. Mary's Catholic Church is the burial place of Dr. Samuel Mudd, who lived to the north, and who in November 1864 had first met Booth at the church.