From around 1093, the Marmion family became lords of the manor, and eight generations of Marmions inhabited Tamworth Castle until 1294. It was the Marmions who were largely responsible for building the present sandstone fort at Tamworth Castle, replacing the original wooden Norman structure. During the period of The Anarchy in the 12th century, Robert Marmion supported King Steven in his fight with Empress Matilda. In the ensuing struggle, Tamworth Castle was taken and occupied by the forces of Matilda, but was returned to the Marmions when Steven finally prevailed in the war. In 1215 King John threatened to have Tamworth Castle destroyed, in revenge for the 3rd Baron Marmion's support for the baronial revolt against the King. However this threat was not carried out.