Most lenders require a credit score of 620 to refinance to a conventional loan. FHA loans have a 500 minimum median qualifying credit score. However, most FHA-approved lenders set their own credit limits. Rocket Mortgage requires a minimum 580 credit score to qualify.