The manor originates from around the late 1300s, where it was owned by Anders Jacobsen Ulfeldt. The manor was kept in the Ulfeldt family, and in 1534-1536 during the Count's Feud, the manor was owned by Knud Ebbesen Ulfeldt, who defended the castle of Gjorslev on Sjælland against the peasants. He supported Christian III after the war. Knud Ebbesen Ulfeldt died in 1540, and the manor was given to his son, Corfitz Ulfeldt. He, however, was secretary at the Danish chancery, and the day-to-day management of the manor was instead performed by his mother, Anne Eriksdatter Hardenberg. Jacob Ulfeldt gained the manor after Anne Eriksdatter Hardenberg's death in 1564. Jacob Ulfeldt fought under both Herluf Trolle and Daniel Rantzau in the Seven Years' War. After performing well in the war, he was sent to Russia as a delegation, but failed to improve Denmark's relationship with Russia. This upset the king, and Jacob Ulfedt spent the rest of his days taking care of Holcekhavn Castle. He made large improvements, and expanded the manor significantly during this time. In 1616, the manor was sold to Ellen Marsvin, who made big changes to the interior of the castle. During the Dano-Swedish War, the manor was owned by Leonora Christine and Corfitz Ulfeldt. Holckenhavn Castle was largely ignored by the Swedes, due to Corfitz Ulfedt's relationship with the Swedish king, and he later joined the Swedish army. After the war, Corfitz Ulfeldt escaped to Germany to avoid the charges for treason, while Leonora Christine was imprisoned. In 1672, Holckenhavn was awarded to Eiler Holck, after his success in the Dano-Swedish War. The war had taken its economical toll on the manor though, and it wasn't until 1760 that the owner Erik Rosenkrantz Holck managed to turn this around. The manor has been owned by the Holck family since then.