Defer freeing of auth structs as we can't free the fde_t structures from

inside the i/o loop as release_auth_client does