Holsøe was born at Øster Egesborg near Vordingborg on the island of Zealand, Denmark. He was the son of pastor Lauritz Christian Holsøe (1789-1862) and Vilhelmine Euphrosyne Margrethe, née Feddersen (1797-1871). He was initially trained as a mason, and later was a private student of Gustav Friedrich Hetsch (1788–1864). Afterwards he attended the Royal Danish Academy of Fine Arts building school and ornament school from 1842 to 1849 and again from 1851 to 1852, interrupted by his participation in the First Schleswig War.