Never disturb animal life in the cave. This includes bats, fish, crickets, etc. Never touch formations with your bare hands. Never remove anything from the cave that you didn't carry in with you, unless it is another careless caver's trash. Have fun but be careful.
Caves are a dangerous place.
They can put you in touch with local caving groups in your area. You could also try a wild tour at Wyandotte Cave . All three are good tours at a good price and will help you get started in caving. Marengo Cave also offers a wild tour, but I've never been on it.