It is said that an old whale oil refinery was once situated at Point Baleine. The caves are below ground. A long staircase provides entry to the bottom of the caves. The main cave contains a deep pool of seawater that seeps in through numerous cracks and crevices. The caves are of limestone formation and have given rise to the formation of stalactites and stalagmites.
Stalagmites are the columns, which rise from the floor of the caves.
Stalactites are those which hang from the roof of the cave. There are many different shapes of stalactites and stalagmites found in the Gasparee caves. Names like the Altar and the Virgin are given to some stalagmites and stalactites fund in the cave because of their peculiar shapes noted to resemble the shape of an altar, and someone standing up like the virgin. Dripping water from the limestone roof of the cave over the years is what gives shape to Stalagmites and Stalactites. Bats are a common sight in the Gasparee caves today.