The deposit goes to the person or company acting as the middleman. In most cases, this is an estate agent or an attorney.