Middleham Castle was built near the site of an earlier motte and bailey castle, called William's Hill, the site of which can still be seen nearby, although there is no evidence of stonework or defensive structures to the former castle site. Historians believe that the defensive walls of the original castle were constructed from timber. In 1270 the new Middleham Castle came into the hands of the Neville family, the most notable member of which was Richard Neville, 16th Earl of Warwick, known to history as the "Kingmaker", a leading figure in the Wars of the Roses. Following the death of Richard, Duke of York, at Wakefield in December 1460, his younger son, Richard, Duke of Gloucester, came into Warwick's care, and lived at Middleham with Warwick's own family. His brother King Edward IV was imprisoned at Middleham for a short time, having been captured by Warwick in 1469. Following Warwick's death at Barnet in 1471 and Edward's restoration to the throne, his brother Richard married Anne Neville, Warwick's younger daughter, and made Middleham his main home. Their son Edward (known as Edward of Middleham), was also born at the castle around 1476 and later also died there in 1484.