Age and genetics are two of the key causes of sagging skin. External aggressors like UV rays, diet and pollution can also damage skin and cause it to lose firmness prematurely.