Homeowners live in their second homes for at least a portion of the calendar year. Although each mortgage lender will have their own eligibility requirements, the IRS says a second home is a residence that you visit for at least 14 days each year, or 10% of the total days that you rent it out.