Always remove our reference to the client.

If for some reason the registrar fails to unsubscribe the client, we
still don't want to keep the client alive by retaining a reference to
them.