Yes. Taking early or late retirement will directly impact the amount of your Social Security benefits for the rest of your life. Likewise, the longer you wait to dip into your retirement savings, the more it can grow before you start using it.