The castle is believed to stand on the site of an earlier wooden fort built in 924 following the capture of the town ramparts in 901 by King Æthelwold of Wessex, and subsequently fortified with a motte by King Edward the Elder. After the Norman Conquest of 1066 the castle's defences were strengthened by the addition of a ditch and bailey surrounded by a wooden palisade. The wooden fort was eventually replaced with a stone keep. The Norman castle was a strategic defensive structure controlled by the King's constable until King Henry I () granted it as the caput of a feudal barony to his cousin Richard de Redvers (died 1107), feudal baron of Plympton, Devon. While the Saxon defences had been against outside threats such as Viking raiders, the Norman fortress was more concerned with subduing the local populace. It was also a useful base for enforcing the New Forest Laws. The castle controlled the harbour and inland access via the Rivers Avon and Stour. The earliest masonry has been dated to , and there is documentary evidence of the castle existing in . Richard de Redvers is often named as the castle's first builder, although there is little documentary evidence of this, and this castle may well have been on the site of the earlier Saxon castle. The castle was rebuilt by Baldwin de Redvers to resist King Stephen during the civil war with the Empress Matilda in 1147. A great tower was built probably around 1300. The castle again saw action during the Civil War of 1642–1651. The Parliamentarians were allowed to take control of the castle in 1644. The Royalists laid siege to it for 3 days, demolishing houses on the corner of Church Street and Castle Street in order to site their cannons. The Royalists were unable to take the castle and Oliver Cromwell, fearing such a powerful stronghold, ordered it to be slighted in 1652.