 John of Gaunt's ownership marked the start of Peveril Castle's decline. He was the richest nobleman in England and held several castles. As Peveril Castle was relatively unimportant, John decided not to maintain it and in 1374 gave orders to strip the lead from the buildings for re-use at Pontefract Castle. It was inherited by his son Henry Bolingbroke, later King Henry IV, and remained under royal control, administered by the Duchy of Lancaster. During the 15th century, Peveril became less important as administrative functions were moved elsewhere. Although other castles administered by the Duchy of Lancaster were repaired in 1480, there is no indication that this happened at Peveril. A survey conducted for the Duchy in 1561 revealed that Peveril was in a state of decay, and as a result, along with Donnington, was one of two castles that were subsequently abandoned. The castle however hosted local courts until 1600. A survey in 1609 found that Peveril was "very ruinous and serveth for no use". At some point in the post-medieval period the keep's facing stone was removed from three sides. The steep slope prevented the removal of the stone from the fourth side. At one point, the castle was used to house animals.
