Fix assert invariant in Peers::close_all