No, it's generally not possible to buy a vacation home with only 3% down. Most lenders require higher down payments for second homes or investment properties. A down payment of 10% is commonly needed for these types of properties.