The lawyer is entitled to be paid for all work done on your behalf while retained.  Discuss with the lawyer how much work has been performed and whether he  would consider a return of any unused portion of the retainer.