There's no set age for what's considered early retirement. Given that the average retirement ages are 65 for men and 63 for women, anything before that could be considered early retirement. In terms of Social Security, early retirement is drawing Social security benefits before your full retirement age. For those born in 1960 or later, full retirement age is 67.