There are only five caves on the map; two are located in the forest, one in the burnt forest, one in the desert, and one in the snow biome.