 By the 1330s, Southampton's prosperity was in a slow decline. In 1338 there was a successful French attack on the town, during which various buildings were burned and castle was damaged. The house may have been one of those damaged in the raid, as the south-western corner of the building collapsed around that time and had to be quickly rebuilt; other alterations, including the addition of a fireplace, may have been carried out at the same time. Southampton's economy collapsed in the aftermath of the attacks and never fully recovered. The character of French Street began to change, as many houses were sub-divided or redeveloped to fit in more buildings. The Medieval Merchant's House ceased to be used by major merchants and by 1392 appears to have been rented out to tenants by Thomas Fryke and John Barflet, the latter a descendent of John Fortin, for whom the house was originally built.
