The top of the stamp can be damaged as easily as the back. The ink surface can rub off on the back of the preceding album page or stock sheet. Rough handling can abrade the surface leaving rubbed-out places. The result is a damaged stamp. Many of these same photogravure stamps also run in water when soaked.
Sunlight can bleach out or change the colors of many stamps.
Keep your collection out of the sun. Be careful, especially when you mount your stamps. Hinges leave marks and in extreme cases overmoistening leads to thins. Personally, I collect hinged stamps. In fact, I hinge every NH stamp I can get my hands on, but I try to do it carefully.