The most significant downside to an FHA loan is the higher total mortgage insurance costs, including a monthly mortgage insurance premium and an upfront mortgage insurance premium, regardless of down payment. Often, FHA homebuyers will refinance their loan so they can remove their mortgage insurance.