Fix bad IP being remembered on successful peer connection