Some lenders have higher average mortgage rates than others, regardless of your credit or financial situation.