If you have to pay back gift money for a mortgage, you're actually taking out a loan rather than receiving a gift. In that scenario, you may be committing mortgage fraud. Ultimately, this is why mortgage lenders typically require gift letters explaining that the gift money is a true gift that never needs to be repaid.