When is the Best Time to Retire 59 1/2 - This is when you can access your retirement accounts with no penalty. 62 - This is the average age because you can start collecting Social Security benefits. 65 - This is the age that Medicare benefits begin. 70 - This is when your Social Security bonus stops adding to itself.