You can also expect to pay interest rates that are 0.25% to 0.5% higher than you would for a primary residence. For illustration purposes, if you've managed to save $50,000 towards a down payment, then you could afford a second home with a $250,000 purchase price ($50,000 is 20% of $250,000).