Remove the port from the RemoteAddr