In 1552, during the Second Margrave War the castle fell unopposed into the hands of the Margrave of Brandenburg-Kulmbach. In August that year, troops from Ulm retook the castle after several days of shelling with heavy stone cannonballs. Having lost its significance and value as a fortification, the city of Ulm started to demolish the castle starting in 1552, sending all usable materials to Ulm. The ruins were left and the site became overgrown over the following centuries.