It should have two shoulder straps. Both straps may not always be needed but it is better to have two straps than to struggle with only one strap in a walking passage. Seams need to be protected and reinforced against wear, especially around the base. Handles on the side and bottom make it more versatile and easier to maneuver in different passages. Keep it as small as possible.
Large packs are more difficult to get through tight spots.
Fill you pack with extra supplies, bulbs, batteries, trash bags for warmth, hard candy for trip leaders, drinking water, high energy snack and extra clothing as needed. Padding for knees and elbows will make caving more comfortable and reduce damage to joints. There are a large variety of kneepads available. You can get pads at most sporting goods, and department stores. Of course there are kneepads designed just for caving.