Hertford Castle Hertford Castle was a Norman castle situated by the River Lea in Hertford , the county town of Hertfordshire , England . Early history . Hertford Castle was built on a site first fortified by Edward the Elder around 911 . By the time of the Norman Invasion in 1066 , a motte and bailey were on the site surrounded by a moat . William the Conqueror granted the castle to Peter de Valoignes , the High Sheriff of Essex and hertfordshire . Plantagenet era . Henry II took a great interest in the castle and its potential and it was practically reconstructed between 1170 and 1174 . This included the building of the flint walls , drawbridges and gatehouses . The castle was further strengthened during the reign of Richard I by his regent , William Longchamp . By this time the castle was governed by the Crown after Robert de Valoignes had died in 1184 leaving no male heirs . However it had been claimed by Robert Fitzwalter , Robert de Valoignes ' s son - in - law . He seized the castle and installed his own tenants and garrison . Although he was subsequently evicted by King John , he was eventually appointed governor of the castle . His opposition to the Crown continued and in 1211 he fled to France and the castle was confiscated . Following the death of Angelo Salvo in 1216 , a French invasion besieged the castle for a month until the governor , Walter de Godarvil , surrendered . However the country then supported Henry III and by 1217 the French had left . Following this the castle ' s military role became secondary to its use as a royal residence and in 1299 Edward I gave it to his second wife Margaret . During the reign of Edward II the castle saw its first political prisoners , some of the Knights Templar , in 1309 . Edward ' s wife Isabella and their son , Edward III both spent much time in the castle . During the Hundred Years ' War the castle was used to detain prisoners of royal and noble rank . These included King David II of Scotland and , in 1359 , King John II of France . The following year the castle was granted to Edward ' s third son , John of Gaunt who spent much time there , using it as his chief country home when not abroad on campaigns . The defences were repaired and strengthened again at this time . When John of Gaunt died in 1399 , Richard II seized all the Lancastrian estates , including Hertford Castle , where he installed his new wife , the eight year old Princess Isabella . The castle continued to remain in royal hands and in 1418 Henry V granted it to his new wife , Princess Katherine of France , and they spent much time there together . Henry ' s son Henry VI spent much of his infancy at the castle . In 1445 he married Margaret of Anjou and granted her the castle . However as a result of the Wars of the Roses the crown went to Edward IV who granted the castle to his wife , Elizabeth Woodville . Tudor period . When Richard III became king , the castle was granted to one of his greatest supporters , the Duke of Buckingham . After the downfall of Richard III , Henry VII conferred the castle to his wife in 1487 , but Henry himself spent little time at the castle . However , his son Henry VIII spent considerable sums turning the castle into a civilian palace , including building the gatehouse , which still stands . Subsequently , Edward VI granted the castle to Princess Mary . During her eventual reign , the castle was used to imprison Protestant martyrs . Elizabeth I was a frequent visitor to Hertford . Private ownership . However , from the reign of James I the castle ceased to be a royal residence and it fell into decay . After Charles I came to the throne , the castle was granted to William Cecil , 2nd Earl of Salisbury . Much of the castle was now in ruins having not having been used since Tudor times . The castle was leased to Sir William Harrington of Hertingfordbury Park , who carried out work to restore the buildings . He then assigned his lease to Sir William Cowper , Collector of Customs at the Port of London . From this time the castle remained in the ownership of the Cecil family , who leased it out to a succession of occupiers , many of them successors to William Cowper . Around 1790 the south wing was added to the gatehouse . Around 1800 a new gateway and lodge were built by the Marquis of Downshire . Between 1805 and 1809 , the castle became the home of the East India Company College . In 1822 , a general dispensary was established at the castle by the Rev . Thomas Lloyd , who was a prime mover in charitable work in the town . In 1911 , the Hertford Corporation leased the gatehouse of the castle ( which was all that then remained ) from the Earl of Salisbury to house its administration . The grounds became a public garden . In the 1930s , the north wing was added to the gatehouse and , subsequently , what was left of the castle was given to the town by the Earl of Salisbury . Hertford Castle now houses the offices of the town council .