The oldest parts of the castle were built in the first quarter of the 12th century as a residence in the Barony of Waischenfeld. At the same time the "ministerialis" family of Rabenstein, who were the builders of the castle and bore the raven on their coat of arms, were recorded as being in the service of the barons of Waischenfeld. In the early 13th century, the outer ward was expanded. During the following centuries ownership and occupancy of the castle changed hands several times. Its occupants included the House of Schlüsselberg and its lords, the burgraves of Nuremberg. In 1450 the castle was destroyed in the First Margrave War and in 1489 was rebuilt by Conz of Wirsberg.