Thornbury Castle was initially in possession of Aylward during the time of King Athelstan (925-940). The Stafford family owned the Castle in the 15th and 16th centuries, till King Henry VIII confiscated it after Edward Stafford, then Duke of Buckingham was beheaded for treason. The Howard family finally owned the Castle from 1640 till the 1960s.