Unfortunately, one of the immovable standards for conforming loans is the loan limit — you can only borrow so much and no more. Loan limits are set by the Federal Housing Finance Agency (FHFA) and are generally adjusted each year, with higher limits for properties with two, three and four units (as long as you live in one of the units). One workaround is a piggyback loan, in which you get a smaller mortgage atop a larger one: Together they add up to enough to finance the home.