The biggest downside of paying off your house early is missing out on investment opportunities that could give you a higher return rate. Instead of using your extra cash in paying off your loan, you could've invested it elsewhere for the remaining life of the loan.