Thanks to its favourable position on the border, the castle was repeatedly attacked and hard fought for before being conquered in 772 A.D. during the Saxon Wars by the Frankish king, Charlemagne. Charlemagne had the Irminsul, a pagan religious site here or in the vicinity, destroyed. In 779, he charged the Fulda abbot, Sturmius, with mission service in this area and directed him to this location. As early as 784/785 Charlemagne overwintered at Eresburg and had a church built, probably on the site of the former Saxon temple. Pope Leo III is supposed to have stayed at the Eresburg in 799 on his way to Paderborn. In 915 there was a bloody feud at the Eresburg between the Saxon Duke Henry the Fowler and the East Francian king, Conrad. The Bishop of Paderborn, Theoderic is also supposed to have participated personally in the battle. Also in 915 at Eresburg the invading Hungarians defeated a German army.