In 1365–66 and 1384–85 the town served as a refuge for the Bishop of Constance when he was expelled from the Constance. Before 1324 and between 1371–86 the city was pawned to outside owners as collateral for a loan. From 1359 to 1403 the castle was owned by the locals. In 1374 they were granted an exemption of the war tax. The episcopal rule was weakened by the transition of sovereignty from the Habsburg to the County of Baden in 1415. Due to its location on the religious boundary Kaiserstuhl presented an important bridgehead for the Catholic regions. In both the First War of Villmergen, in 1655/56, and the Toggenburg War (or Second War of Villmergen), in 1712, the village was occupied by Zurich. During the religious conflicts, traffic to Kaiserstuhl was forced to instead cross the Rhine in Zurich controlled areas. As the town did not have the right to tax ship traffic on the Rhine, this loss of land traffic cost the town most of its custom revenue. However, the town remained prosperous due to viticulture and animal husbandry, through the credit and banking, and through foreign service. The prosperity is reflected in the stately houses built during this era. (Gasthaus zur Krone in 1550, administration hall of the monastery of St. Blaise in 1563 and 1612, and the "Marschallhaus" in 1764). The buildings that allowed Kaiserstuhl to function as a regional center were on the right bank of the Rhine, and were lost in 1798 with the creation of the Canton of Baden. After the destruction of the bridge by the French in 1799 border traffic through Kaiserstuhl ended for almost two decades.