Me, I'd have to say the taste they impart. Hot peppers vary considerably in type, the location where they're grown, the temperature of the growing season, when they're harvested and how they're processed. McIlheny, for example, puts their picked peppers in a barrel with salt and lets them age naturally three years before processing. The result is a wonderful, smoky, rich and deep flavour. Some companies, on the other hand, use chemical enhancers to artificially age their products, sometimes giving them an unpleasant aftertaste and a thin, watery tang.
Chile peppers are the fruit of a perennial plant related to the nightshade.
Perhaps it's that association with danger that attracts us. The highest concentrations of capsaicins are in the seeds and the placenta - those white 'ribs' you find inside. And size has nothing to do with heat. The Habanero is considerably smaller than a banana pepper. Capsaicin is only minimally water-soluble.