Yes, you can be held liable for breaking the contract, because that is exactly what you would be doing breaking the contract. Your second thoughts, even if based on legimate concerns, are not a legal basis for escaping the contract. At a minimum, you could lose your deposit or earnest money--the buyer would not have to return it. He could likely also sue you for other amounts, such as his taxes, utilities, insurance, and mortgage for however many months he has to hold the property before selling it again, and for any lawyer, etc. fees he's incurred in trying to sell it to you.