Remove listeners on TCP error and close.