His father also had them in his later life. Just as you've described, most folks who develop baggy eyelids inherit the tendency to develop them with age. But several things can bring on the condition a lot sooner. Chronic sun damage to the skin on the face and around the eyes leads to premature wrinkles. If you've got itchy and watery eyes from allergies of one form or another, the constant rubbing can cause swelling of your lower eyelid area.
Smoking can also cause premature aging of the skin.
Thanks to the effects of gravity, fluid will naturally have a tendency to collect in the lower eyelids, and wrinkled, loose skin will inevitably occur. Everyone, to some degree, will develop eyelid bags. The key is to protect your skin when you're young. Using moisturizers and sunblock, avoiding smoking, and treating allergies to lessen the urge to rub your eyes are the main preventive actions to take. It's not something I endorse.