When your basement or crawl space has ambient moisture and all of the problems that come with it - namely mold - the affected air circulates through your house. In addition to spreading allergens, the air just plain smells. But your home generally has enough activity that you might not notice it.
Closets, however, are a little more stagnant and spend a lot of time with the doors closed. This allows air to stagnate further and makes the smell even more noticeable when you open the doors.
Unless you’re trying to recreate Lady Gaga’s meat dress with your entire wardrobe, a smelly closet is a sign of water problems under your home.