Close our end properly when the remote peer closes theirs