Possibility as early as the 11th century the fortified church and refuge castle became a nobleman's castle. In the 11th or 12th century a square tower was built on the southern side of the hill top. Nothing is known about the first noblemen at Solavers. In the 13th century, the castle was acquired by the Lords of Aspermont. They controlled the lower Prättigau valley from Solavers and the upper valley from Castels Castle in Luzein. By the early 14th century the Aspermont family was in financial difficulties and in 1338 sold the Prättigau valley to the von Matsch family and the Counts of Toggenburg. In 1344 the Matsch and Toggenburg families divided their estates with Toggenburg receiving Solavers. The last count, Frederick VII may have been born at Solavers. His death in 1436, led to the creation of the League of the Ten Jurisdictions which was a confederation of ten communities in the region which sought freedom from foreign overlords. Frederick VII's death also started the Toggenburg War and a long running dispute over the inheritance of the castle.