A subsequent rebuilding saw the timber palisades of the inner bailey replaced by stone walls and towers. Exactly when this happened is unclear, but it may have been under Peter of Savoy, the Earl of Richmond, who was granted the castle by Henry III in 1246. There is no record of the rebuilding but in 1254 Peter ended the feudal requirement to maintain the palisades and replaced it with cash payments. This probably reflected the replacement of the palisades with the stone walls and towers visible today. The castle faced a lengthy siege only a decade later during the Second Barons' War from the rebel baron Simon de Montfort, following Henry's defeat in the Battle of Lewes. Defeated members of the royalist army fled to Pevensey, pursued by de Montford's forces, but the garrison refused an invitation to surrender and endured over a year of besiegement. Their adversaries were unable to stop supplies reaching the castle despite digging a ditch to cut it off from the mainland; its garrison raided the surrounding countryside and sought to obtain fresh supplies of men and weapons by sea in December 1264. The costly and ineffective siege was eventually lifted in July 1265. It caused significant damage to the castle, with the Roman wall toppled on the south side. The parish churches at Pevensey and Westham also suffered damage, which the attackers may have caused in using them as siege castles (temporary fortresses and artillery platforms).