In the Despenser War of 1321, Edward II undertook a campaign against the Mortimers, by then a potentially rebellious Marcher Lord family. After the short war, the crown again occupied Bridgwater until 1326 to prevent Roger Mortimer, 1st Earl of March using it as a base for operations if he should escape from custody or return from exile in France. Roger Mortimer did return from France with Edward's wife, Isabella; once they had seized the throne, the castle was returned to the Mortimer family but its upkeep was neglected and part of the moat was filled in. Only St Mark's Chapel and a barn were repaired, until the 1380s and 1390s when the towers, gatehouse and barbican were reinforced. By 1450 private houses had been built within the walls, Fore Street had been extended over the moat and its military value was reducing. It the 1540s John Leland described the castle as "all going to mere ruin" and part collapsed in 1548, with a house being built on the site.