remove ip/port from sdp list once destroy flag is set