fix potential infinite loops in both LocalAddr and RemoteAddr