The loss of skin elasticity due to aging is the most common cause of saggy breasts. Another factor is smoking, which accelerates aging and thus contributes to sagging breasts, sometimes even earlier in life.