If there was some early termination or cancellaton clause in the agreement, you'd have to let him terminate/cancel in accordance with the terms of the agreement. If there is not, then he is bound by the contract: there is no inherent right to terminate or cancel a contract, and once it is signed (and a deposit made), it is binding. He cannot get out of the contract entirely; if he refuses to go ahead, you are entitled to an amount equal to the sum of expenses you  have already *committed to or paid* (i.e. ones you will have to pay no matter what) and your reasonable or expected profit; all he could get is an amount equal to any expenses you will not have to foot or bear, since you will not have to lay out those costs.