His sister Jelena was the mother of the future Ban of Croatia and hero of the Siege of Szigetvár (1566), Nikola IV Zrinski. Karlović nephews Nikola and Ivan Zrinski in 1541 took care to carve the inscription on the tombstone, saying "Sepultus genere Spectabilis militiaque praeditus magnificus dominus Torquatus, comes Corbaviae regnorumque Croatiae et Sclavoniae banus mole sub hac tegitur", however the inscription with the coat of arms got lost over the centuries. In a 16th century Glagolithic document his seal and coats of arms were described to have a goose on a shield, above them letters I. C., meaning Joannes Caroli. In 1736, Hungarian polymath Samuel Timon described the alleged coat of arms on the tombstone, and according to it in 1802 Károly Wagner described the color, but they were inspired by 17th-century armorials like "Opus Insignium Armorumque" (1687–1688) by Johann Weikhard von Valvasor.