If the agreement is with the firm, then yes--you would have to pay the agreed-upon contingency fee (assuming, of course, that you win) to the firm. When you contract with any company or business, unless the contract guarantees you the work/services of a specific employee, the fact that an employee or even partner (e.g. a lawyer) leaves does not invalidate or terminate the agreement, or give you any right to go with the departing employee. So unless you want to pay this firm and also separately pay the attorney for his work if you go with him, stay with the firm.