Fix race condition with peer.closed