There are 4 different kinds of caves. A few people in Africa use sandstone caves as homes.
Sandstone erodes away easily.
It can be carved out by wind, water or hand tools. This scientific article about Limestone Caves tells the orgin of limestone caves and how they are formed. The caves that are not caverns have a different tale to tell. Take sea caves, for instance. Thousands of years ago the earths crust had a sudden movement that caused a crack in the cliffs that lined the shore.