Over the years it has been used as home to the city's gunner, the Town Gaol and even as storage for the Southampton Harbour Board