 Roger Clifford died at Skipton Castle in 1389, and the Clifford family began to lose interest in Westmorland. The Cliffords preferred their properties in Yorkshire to their dilapidated castles in Westmorland, which had been ravaged by wars with Scotland. Brougham Castle is not known to have been in use as a residence again until 1421, when a man was accused of forging coins in the castle. Although little is known about Brougham during this period, historians believe it likely that repairs were undertaken, and a rivalry emerged between the Clifford family and the House of Neville that would later have consequences for Brougham. The familial enmity meant that the Earl of Salisbury, a Neville, used his position as lord of Penrith to antagonise the Cliffords; it is likely that Brougham Castle was kept garrisoned due to its proximity to Penrith. In the Wars of the Roses (1455 – 1485), the two families were on opposing sides, the Cliffords supporting the House of Lancaster and the Nevilles supporting the House of York. When the Yorkist Edward IV usurped the throne in 1461 the lands of John Clifford, 9th Baron de Clifford were confiscated. In 1471, Edward IV granted Sir William Parr Brougham Castle and other properties which had belonged to the Cliffords. A year later Henry Clifford, John's son and heir, was pardoned and when the Lancastrian Henry Tudor overthrew the Yorkists and took the throne as Henry VII, Henry Clifford appealed for the return of the Clifford estates. This was granted in November 1485.
