Woolloomooloo Gates, Botanical Gardens