Paying off your mortgage early could limit your ability to save, reducing your retirement savings, especially if you have contribution room remaining in your tax-free and registered accounts.
If you have higher-interest debts like credit card debt, focusing on your mortgage could cost you more, as these debts likely have much higher interest rates, plus their minimum payment only covers the interest, unlike a mortgage payment.