However, there are several significant drawbacks to 401(k) loans. Even though you'll pay yourself back, a significant disadvantage is that you're still taking money out of your retirement account while it's still growing tax-free. Additionally, less money in your plan means less money that will increase in value over time.