The building was originally a small castle on the crest of a ridge on the road inland from the Lincolnshire fen edge towards the Great North Road. It is said to have been begun by Gilbert de Gant, Earl of Lincoln in the early 13th century. However, he was the first and last in this creation of the Earldom of Lincoln and he died in 1156. Gilbert's heyday was the peak time of castle building in England, during the Anarchy. It is quite possible that the castle was built around 1140. However, the tower at the south-east corner of the present building is usually said to have been part of the original castle and it is known as King John's Tower. The naming of King John's tower seems to have led to a misattribution of the castle's origin to his time.