In 1530 Martin I Pfinzing acquired the castle. It thus passed into the possession of an old Nuremberg patrician family, after which it is occasionally called "Pfinzing Castle" to the present day. At the beginning of their landlordship, Henfenfeld was also caught up in the destruction of the Second Margraviate War and, like many other fortifications in Middle Franconia, was burned down in 1553 when Margravial troops invaded, whose partisan Hans von Egloffstein saw himself deprived of his property by the Pfinzings. The most famous family representative and lord of the castle was Paul Pfinzing (1554-1599). The Pfinzing von Henfenfeld contributed the castle to a family foundation (called "Vorschickung" in the Nuremberg patriciate), which was administered by the family elder. They remained the lords of the castle until the extinction of the main branch of the family in 1764, after which the administration fell to the brothers of the widow, a native of Haller von Hallerstein.