The gardens were established in 1853 and was the second botanical garden to be founded in the Cape Colony. A prospectus was compiled for the establishment of the gardens in 1846, but the establishment was delayed by the Frontier Wars. Sir Harry Smith, 1st Baronet and Sir George Cathcart were both present at the formal opening. The grounds are allegedly haunted by Smith's wife, Juana. The gardens were declared a national monument on July 4, 1994.