However , many holders of non-conforming loans , might not have been able to refinance and those borrowers with adjustable-rate mortgages might now be sweating it out because their impending re-set dates are approaching just as rates are trending higher , he said .