Self-Consistent Learning: Cooperation between Generators and Discriminators
Authors: Tong Wu
Hao Wang
Zhongshen Zeng
Wei Wang
Hai-Tao Zheng
Jiaxing Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Cooperative Closed-loop Training
Language Modeling
Sentence Semantic Matching

Abstract: Using generated data to improve the performance of downstream discriminative models has recently gained popularity due to the great development of pre-trained language models. In most previous studies, generative models and discriminative models are trained separately and thus could not adapt to any changes in each other. As a result, the generated samples can easily deviate from the real data distribution, while the improvement of the discriminative model quickly reaches saturation. Generative adversarial networks (GANs) train generative models via an adversarial process with discriminative models to achieve joint training. However, the training of standard GANs is notoriously unstable and often falls short of convergence. In this paper, to address these issues, we propose a $\textit{self-consistent learning}$ framework, in which a discriminator and a generator are cooperatively trained in a closed-loop form. The discriminator and the generator enhance each other during multiple rounds of alternating training until a scoring consensus is reached. This framework proves to be easy to train and free from instabilities such as mode collapse and non-convergence. Extensive experiments on sentence semantic matching demonstrate the effectiveness of the proposed framework: the discriminator achieves 10+ AP of improvement on the zero-shot setting and new state-of-the-art performance on the full-data setting. 

Under review as a conference paper at ICLR 2023
SELF-CONSISTEN LEARNING: COOPERATION BE- TWEEN GENERATORS AND DISCRIMINATORS
Anonymous authors Paper under double-blind review
ABSTRACT
Using generated data to improve the performance of downstream discriminative models has recently gained popularity due to the great development of pre-trained language models. In most previous studies, generative models and discriminative models are trained separately and thus could not adapt to any changes each other. As result, the generated samples distribution, while the improvement of the discriminative model quickly reaches saturation. Generative adversarial networks (GANs) train generative models via an adversarial process with discriminative models to achieve joint training. How- ever, the training of standard GANs is notoriously unstable and often falls short of convergence. In this paper, to address these issues, we propose a self-consistemt learning framework, in which a discriminator and generator trained in a closed-loop form. The discriminator and the generator enhance each other during multiple rounds of alternating training until scoring consensus reached. This framework proves to be easy to train and free from instabilities such as mode collapse and non-convergence. Extensive experiments on sentence semantic matching demonstrate the effectiveness of the proposed framework: the discriminator achieves 10+ AP improvement on the zero-shot setting and new state-of-the-an aht performance on the full-data setting.
easily deviate from the real data
cooperatively
INTRODUCTION
The advance of Pre-trained Language Models (PLMs) (Brown et al., 2020; Chowdhery et al. 2022) has substantially improved the performance of deep neural networks across a variety of Natural Language Processing (NLP) tasks. Various language models, based on the Transformer (Vaswani jet al. 2017) architecture, have been proposed, leading to state-of-the-art (SOTA) performance on the fundamental discrimination tasks. These models are first trained with self-supervised training objectives (e.g., predicting masked tokens according to surrounding tokens) on massive unlabeled text data, then fine-tuned annotated data to adapt to downstream tasks of interest. However, annotated data is usually limited to a wide range of downstream tasks, which results in overfitting and lack of generalization unseen data.
One straightforward way to deal with this data scarcity problem is data augmentation (Xie et al,, 2020), and incorporating generative models to perform data augmentation has been widely adopted recently (Carlini et al. 2021 Gangal et al. (022). Despite its popularity, the generated text can easily deviate from the real data distribution without exploiting any of the signals passed back from the discrimination task. In previous studies, generative data augmentation and discrimination have been well studied as separate problems, but is less clear how these two can be leveraged one framework and how their performances can be improved simultaneously.
Generative Adversarial Networks (GANs) (Goodfellow al., 2014 Gulrajani et al. 2017 are good attempts to couple generative and discriminative models in an adversarial manner, where two- player minimax game between learners carefully crafted, GANs have achieved tremendous suc- cess in domains such image generation (Denton et al., 2015). and related studies have also shown their effectiveness in semi cemi-supervised learning (Salimans et al., 2016 Kumar et al., 2017). However, GANs notoriously difficult train, most training objectives work well for only the discriminator or the generator, so rarely both learners can be optimal at the same time (Arjovsky)
model, either
& Bottou 2017 Wiatrak et al. 2019). This essentially arises from the adversarial nature of GANs,

---

Under review as a conference paper at ICLR 2023
that during the process. optimizing one learner can easily destroy the learning ability of the other, making GANs fail to converge.
Another limitation of simultancously optimizing the generator and the discriminator comes from the discrete nature of text in NLP, as no gradient propagation can be done from discriminators to gener- ators. One theoretically sound attempt is to reinforcement learning (RL), but the sparsity and the high variance of the rewards in NLP make the training particularly unstable (Caccia et al.) (020).
To address these shortcomings, we novelly introduce self-consistent learning framework based on one generator and one discrininator: the generator and the discriminator are alternately trained by way of cooperation instead of competition, and the samples are used as the medium to pass the feedback signal from the discriminator. Specifically, in each round of training, the samples generated by the generator are synthetically labeled by the discriminator, and then only part of them would be selected based on dynamic thresholds and used for the training of the discriminator and the generator in the next round. Several benefits can be discovered from this cooperative training process. First, a closed-loop form of cooperation can be established so that we can get the optimal generator and discriminator at the same time. Second, this framework helps improve the generation quality while ensuring the domain specificity of generator, which in turn contributes to training. Third, steady stream of diverse synthetic samples can be added to the training in each round and lead to continuous improvement of the performance of all learners. Finally, we can start the training with only domain- related corpus and obtain strong results, while these data can be easily sampled with little cost or supervision. Also, the performance on labeled datasets can be further boosted based on the SOTA level. As an example to demonstrate the effectiveness of our framework, we examine it on the task of sentence semantic matching. The experiments show that our method significantly improves over standalone state-of-the-ar discriminative models on zero-shot and full-data settings.
Our contributions are summarized as follows,
We propose self-consistent learning framework that incorporates the generator and the discrimi- nator, in which both achieve remarkable performance gains simultaneously.
We propose a dynamic selection mechanism such that cooperation between the generator and the discriminator drives the convergence to reach their scoring consensus.
Experimental results show that our proposed framework significantly outperforms the state-of-the- art methods for the task of sentence semantic matching.
2 RELATED WORKS
To alleviate the lack of annotated data in supervised learning in NLP, semi-supervised learning (SSL) has been a popular line of research (Van Engelen & Hoos 2020). The sources of the unlabeled data required by SSL are either collected from the domains or generated by generative language models. Then NLU models can learn from the unlabeled data by pseudo-labeling (Arazo et al. 2020 Bankaalehi-eekkrrii & Zhang 2021) and consistent regularization (Jeong et al. 2019] [Sohn et al, 2020). However, collecting unlabeled data comes at cthouhh smaller than labeling data), and the total amount is limited. Even with generative models, there is no guarantee of the quality of the generated samples, because the model cannot tune the generating results based on the performance of the downstream tasks. In contrast, our method usually includes continuously updated generative model, which dynamically adjusts its generation according to the performance of downstream tasks.
In GANs, the generator is adversarially trained with the discriminator, Unlike conventional GANs in continuous domains, language GANs usually employ Gumbel-Softma differentiation (Jang et al.)
2017 Yin et al. (020)) Reinforcement Learning (Ri. (Yu et al. 2017 Wu et al. [202)). or modi- fied training objectives (Montahaei et al., (021)) to update the generator, to use the non-differential signals from the discriminator. However, language GANs are often criticized for underperforming Maximum likelihood estimation (MLE) and are very difficult to train, even the single optimality of either the generator or the discriminator cannot be guaranteed (Alvarez- et al.) (022)). In comparison, our proposed framework allows us to cooperatively couple the generator and the dis- criminator, leading to continuous improvement for both learners.
2

---

Under review as a conference paper at ICLR 2023
3 METHODOLOGY
3.1 COOPERATIVE OR ADVERSARIAL
Following the principle of self-consistency outlined in|Ma al, (2022). a closed-loop training needs to be built between the generator and the discriminator, either cooperatively or adversarially. GANs are typical examples adversarial learning, but training GANs remains quite unstable. Let us consider extreme case show the possible instability: the discriminator can perfectly distinguish real data and fake data generated by the generator, and the generator can fully reproduce the real data distribution. Then the discriminator has only 50% probability of selecting all samples that are generated by the generator. Therefore, any further updates to the generator parameters based on the feedback from the discriminator deviate the generator from the optimum. Neither the generator nor the discriminator can likely be optimal (Arjovsky & Bottou, 2017 Lamprier et al.) (022).. In practice, very delicate balance needs be maintained between the discriminator and the generator to keep the training stable. In terms of cooperatively closed-loop learning, as discussed below, it does not suffer from instability: the generator and the diseriminator usually enhance each other.
3.2 SELF-CONSISTENT LEARNING FRAMEWORK
In this section, we introduce self-consistent learning (SCL) framework.
Supplement with filter data
Expand demain
Domain-daa data
fitter ED
Pretraining-dake
Discriminator
Gemerated-data
Fiteree-generaatedddatt
by
Figure 1: Overview of the flow chart for the SCL framework.
As shown in Figure our framework, similar to the GANs, consists of a generator and a discrimina- model. However, contrasting to the GANs, these two parts in our framework work cooperatively to enhance each other. Specifically, for any given class k, the generator G now become conditional generator that takes in an input sentence st and generate an output sentence st. The discriminator D is then responsible for discriminating the sentence using a dynamic threshold ep. The discriminated sentence used as positive or negative data for that specific class to continue the training process. Once the new discriminator is trained, the sentence is discriminated again by the new discriminator with a different dynamic threshold eg. This time only the positive data passed to the generator as the training data for the new round. In this way, closed loop of cooperation is formed.
the above closed-loop training, we propose a selection mechanism that uses dynamic thresholds to filter samples. This mechanism is empirically shown to play critical role in closing the gap between the generator and the discriminator, and thus makes this cooperation loop a virtuous circle. Specifically, shown in Equation the output probability (D kt|s)) that the sentence { belongs class A is calculated from the embedding representation H of {sh).
PD k(s{) softmax
We follow Reimers & Gurevych 2019) and use the embedding representation of CT.S-token as the sen- representation H.
3

---

Under review as a conference paper at ICLR 2023
where y represents the class label. Then, through the filtering function filter (-) in round for the k-th class in Equation 2 er, while other generated samples whose confidence lower than threshold Et, are discarded.
we keep samples whose output probability is not less than threshold
Ellter? - (D(k)))) Et,
where et, represents the dynamic threshold for accepting {st] negative or positive samples in the f-th round. The generalized threshold function for Er,b is defined as,
6t, = f(,, 1, 1t-1
(3)
where 11- and 1,.1, represent the discriminator loss and threshold for round t 1, respectively. La.k is set as O and Co = X, where À represents hyperparameter.
Theorem At round given the previous round discriminator D''', the aim of optimizing the generator Sg boils down to:
min O (-))
where DEL is the standard KL divergence, pgg() (-) refers to the degree of confidence that the sentences generated by the generator belongs to a given class (we can either train the generator to express its confidence in the generated sentences et al. 2022) or use a fixed third-party model (-) to score them Gao al. 20211), and 1)) the probability of being classified into class k given by the discriminator.
Theorem T shows that the generator at round is encouraged approximate the probability distri- bution given by the previous round discriminator, The proof is given in Appendix A In particular, on the basis of well-pretrained discriminator, the generated distribution of the generator can be guaranteed to be faithful to the real data distribution.
Why Cooperative, Not Adversarial? (1) the generator is no longer a challenger to the discriminator that only provides negative data points to fool it, but now serves as a data augmenter to provide both positive and negative data points to enhance the discriminator; (2) the generator no longer updates its parameters through the policy gradients guided by the signals from the discriminator, but rather by utilizing the filtered data points to further improve its conditional generation quality. Note that by deliberately choosing the conditional generation paradigm along with the selection mechanism, not only make the training more stable due to the different training goals, but also bypass the mode collapse problem of GANs (see Section[4.6ffo further discussion). Besides, by iterating through the loops, our framework achieves self-consistency by honing the domain specificity of the generator and increasing the domain data exposure of the discriminator.
3.3 SENTENCE SEMANTIC MATCHING
We leverage the sentence semantic matching task (i.e. k = 2) example to demonstrate the effectiveness of our method. At this time, corresponding Equation k = 1/00 represents the posi-
tive/negative class, and fi l ter represents the filter function in round / for the positive/negative class respectively. First, let introduce the formal definition of this task. Given s" = {wf, wy w2, and tokens in the sentences, and t, and ls indicate the length of s and s². The goal of this task is to learn a discriminator D to precisely predict the label = D(s", s²), where y E y = (0, 1} indicates whether the two sentences similar.
sentences (w², w..... w², }. where w and wh represent the i-th and j-th
In our task, G is trained to generate a similar sentence sh from any given sentence sa and D is trained predict label y from any given sentence pair s"). As demonstrated in Figure there are mainly two training processes in the entire framework: fix G to train D and fix D train G. We introduce the training procedures in detail with the t-th round training.
4

---

Under review as a conference paper at ICLR 2023
Training D: We first randomly sample if from domain- corpus C, and then input st to G to generate si- Next, we feed sentence pair (5),))) into D to predict the label Ut-1, and filter {sf; st, Mt-1 using threshold < Finally, we train D on the selected data and pre-training data P to get improved discriminator D Note that the filtered data have both positive and negative samples. The update process of 'D seeks to minimize the cross-entropy loss over all instances:
|a
1 8 1y log PD 1|s s + y.) -log(1 PD(y: 11s s(i) (4)
(D(s.y)
Training G: We feed the generated sentence pairs (5),))) into De to predict new labels yt. and then filter st yr using threshold eg and additional rules p Note that the filtered data has only positive samples. For the filtered data, supplement it with the pre-training data P to update G to G F We also take out of from the filtered data and add them to the domain-related corpus. The expanded domain corpus used sample conditional sentences in the next round of generation. The update procedure of G employs the negative log- eigliiklihood function over all instances:
La(8" 888
s D log pa P (
For the selection mechanism, we adopt the form c² m X after comparing the effects of different functions through experiments according to Equation where is the increment of the threshold for each round, À is the initial threshold, and is the threshold for rounds t.
In the process of training since the sentences generated in each round added the domain- related corpus, the source of domain-specific data is thus monotonically expanding by iterating the self-consistent learning loop. The formalized process is shown in Algorithm
Algorithm 1 Self-consistent Learning (SCL)
Require: Generator G; Discriminator D; Domain-Related Corpus C:; Pre-training Data P. 1: Initialize go and D with pre-trained language models;
2: Warm-up G and D with pre-training data P to get G' and D)
3: for each round E [1, ni] do
4: if Two consecutive rounds of discriminator still improve then
5: Generate similar sentences s Pg: (1))) from sampled sentences 8 from C;
6: Predict pseudo-labels y ~ PD (-|s" sh)
7: Use threshold e select data {s", sh y²) } to train Di+1:
8: Predict pscudo-labels yt Pri 1 (-1s", s²);
9: Use threshold eg and additional rules to select data on {s", s", t to train gi+1;
10: end if
11:
end for
EXPERIMENTS
4.1 TASKS DESIGN
In experiments, the pre-training datasets used to warm up the discriminator and generator, and the domain-related corpus is a set of independent sentences. To avoid label leakage, none of the
The additional rules used exclude sentences which too long, short, similar according the longest common substring algorithm.
Note that the pre-training data P is used to warm up & and 22. Although pre- pre-training data is not mandatory in subsequent training, empirically found that including it when training G can prevent language degenera- tion and improve downstream performances.
5

---

Under review as a conference paper at ICLR 2023
training datasets participate in the pre-training of the generator and discriminator. In other words, the datasets in pre-trale tre-training and self self-consistent training are two non-overlapped datasets.
Zero-Shot Baseline: We utilize the best- pesttperforming Chinese model RoBERTa-wwm-cek jct al. 2020 202T) and English model ALBERT ALBERT-xxlarge-22 (Lan et al. 2020 as the base discrimi- nators in self-consistent learning framework.
CCui
Fine-Tume Baseline: We compare our model with several state-of-the thee-o--thherrttt semantic matching mod- els, including the following. Chinese models MacBERT (Cui et al. 2020). StructBERT (Wang
et al.
2020), RoFormer (Su et al.
2021 Su et al. 2022), XLNet (Xu et al. (020). ELECTRA (Cui
al.
2020). ALBERT (Lan al. 2020), RoBERTa (Cui al. 2020 2021 and English models
BERT
Devlin ct al.
2019b). XLM-RoBERTH (XLM-R) (Conneau et al.
2020 XLNet (Yang et al.
2019b),
ELECTRA (Clark al.,
2020). ALBERT (Lan et al. (020), RoBERTa (Liu et al. 2019
For a fair comparison, we use models that are as close in size as possible.
4.2
EXPERIMENTS SETUP
4.2.1 DATASETS
We conduct experiments on three Chinese semantic matching datasets AFQMC (Xu et al.) 2020).
CHIP-STS (Zhang al., 2022a). Chinese-QQP (Wang al. 2019) and English semantic match-
ing dataset MRPC (Wang et al. 2019). More details about the datasets are given in Appendix
4.2.2 MODEL PRE-TRAINING
We adopt the well-established Transformer-XL (Dai et /0P (Zhang et 2022b archi- tectures as the Chinese/English generator. To enable the generator to generate similar sentences with better linguistic quality, we pre-train Transformer-Xe model with 5.0 billion parameters and incrementally pre-train an OPT model with 2.7 billion parameters on the corpus consisting of plain texts and similar sentence pairs. Cleaned large-scale Chinese corpus WuDaoCorpora (Yuan et al.) 2021) and English corpus WikiText (Merity et al. (017] are used as plain texts. Similar sentence pairs that do not overlap with downstream datasets are used in the pre-training, and the designed prompts are employed to guide the generation of similar sentences. More details regarding model pre-training can be found in Appendix D
4.3 ZERO-SHOT RESULTS
Table how the Fl score of the discriminator varies with the number of self-consistent learn- ing rounds on different datasets the zero-shot task. According to Algorithm [T the training is stopped when the discriminator no longer improves for two consecutive rounds. In addition, these four datasets are collected from different domains to further reflect the generality of our method in different domains. Specific training settings recorded in Appendix F
Table 1: Shoo-Shot Performance of the Discriminator, (F1 score (%)
AFQMC (Financia)
CHIP-STS (Medical)
Chinese-QOO QQP
MRPC (News)
Round Num
(Common)
baseline
38.25
58,82
$7.88
68.54
39.61
62.89
60.08
75.47
2
44,98
67.24
58.57
76.63
3
45,99
71.38
60.30
83.00
45.71
71.45
61.31
83,90
48.01
74.06
64.47
84.24
6
50.41
74.08
66.44
84.50
7
50.68
76.66
63.88
84.32
8
51.36
76.30
65.46
84.61
76.67
68.08
10
77.42
70.51
+13.11
+18.60
+12.63
+16.07

---

Under review as a conference paper at ICLR 2023
The scores in the last line of Table | give the improvement of discriminator in the last round relative to the first round. We can see that the score gradually increases after each training round, eventually reaching 10+ absolute percentage (AP) improvement. We believe what drives the improvement of the discriminator the self-consistency, which it acquires with the generator step by step during the loop.
To verify that the generator also improves after self-consistent training, we adopt Perplexity and Bertscore (Zhang al. 2020) to measure the language fluency and the semantic similarity (i.e. domain specificity) respectively, For different generators in different rounds, we first select 8 in similar sentence pairs from the same test set as the original sentences input, and generate similar sentences s" with greedy search. The reason for not using other sampling methods is to ensure re- producibility. Given the generated sentences, we introduce an additional GPT2 09922moode model to calculate the perplexity of generated similar sentences, and use third-party tibraryite calculate the bertscore between the original and generated similar sentences. The results are shown in Table
Table 2: Zero-Shot Performance of the Generator.
AFQMC
CHIP-STS
Chinese-QQP
MRPC
Perplexity first round Perplexity
10.13
6.86
12.94
28.71
laast round
8.43
5.97
12.27
17.56
Bertscore first round Bertscore
0.79
0.84
0.87
0.94
0.80
0.85
0.89
0.97
last round
We can see that the perplexity/bertscore of the last round in Table 2 has decreesse//mproved com- pared to the first round. Note that lower perplexity indicates a more fluent sentence, while a higher bertscore indicates more similar sentence. It suggests that after self-consistent training, the gener- ator is gradually improved in language fluency and semantic similarity (i.e. domain specificity). The reason why the improvement of the generator is the size of the generator is several times that of the discriminator, and the total number of training samples is limited. In Appendix G the generated samples of the generator different rounds are given to show the changes in the generation.
obvious that of the discriminator that
4.4 FINE-TUNE RESULTS
Our method not only works well in the zero-shot case, but also achieves good results in the full-data case. For the sake of fair comparison, reproduce several state-of-the-art semantic matching models on the four training sets, and their performances on the test sets are shown in Table
Our approach uses the berf-pero model on single test set the base discriminator for self- consistent learning. The bold scores in the last line of Table 3 show that our method outperforms the SOTA results (shaded in gray) by AP on all four test datasets, indicating the potential of self-consistent learning to further improve the model performance and establish new SOTA.
4.5 EVALUATING SELF-CONSISTENCY
In this section, we evaluate the consistency between the generator and the discriminator as the learn- ing loop unfolds. We follow the same method used in Section 4.3 and use greedy search generate similar sentences on the same test set. Then we take the confidence of the discriminator Rp as the score of the discriminator, which is calculated for the original sentences s and the generated similar sentences s according to Equation 5
1Wenzhong-GPT2-1100 Wang et al. 2022 for Chinese data, GPT2-base Radffrd et al. 2019 for
English data.
7

---

Under review as a conference paper at ICLR 2023
Table 3: Fl Score(%) of Different Discriminators on the Test Datasets.
#Param(zh/en)
AFQMC
CHIP-STS
Chinese-QQ0
MRPC
AVG
BERT lurge
-/335M
82.51
XLM-RAi M-Raase
-/278M
84.27
MacBERTuar large
326M/-
61.11
85.94
72.94
73.33
StructBERT
326M/-
60.56
85.17
76.33
74.02
RoFormertarg
316M/-
64.19
84.16
76.56
74.97
XLNeloore
360M/360M
50.31
82.97
64.96
79.51
69.44
ELECTRA Aturge
324M/334M
54.59
84.97
71.81
89.64
75.25
ALBERT large
221M/223M
56.87
86.32
70.52
91.21
76.23
RoBERTalary
326M/355M
57.29
86.93
74.58
90.24
77.26
Our Method
66.59
88.39
78.43
92.78
81.55
RD PDly 888
(5)
where yt represents positive label.
However, for the generator, to the best of our knowledge, there is no reliable way to measure how similar 8 and & are by using the generator itself. Therefore, to quantify this similarity, we intro- duce third-party static model SimCSE D to get the embedding representation A, B of sentences ,",s". The cosine similarity Rg between and B then calculated according to Equation 6 to approximate the score of the generator.
A, B Encoder(s") E.coder(ss)
A
Rg = A |B
(6)
where and B both represent the embedding representation at the (CL.S position. Note that the original sentence s remains unchanged in each round, while the generated sentence sh changes.
Finally, for the trained discriminator and generator in each round f, we can obtain two score distributions R$ and Rt correspondingly. According Theorem I draw the curves of KL divergence between Rt, and Rt, in each round for the four datasets: AFQMC, CHIP-STS, Chinese-QQR QQP, and MRPC. As illustrated in Figure 2 all the curves show clear downward trend, indicating that the distance between the two score distributions decreases with the increase in the number of training rounds until a score consensus is reached. Table 4 shows the values of KL divergence in the first and last rounds. Numerically, it is more evident that the distances are significantly reduced on the four datasets.
4.6 EFFECT OF PRE-TRAINING DATA AND SELECTION MECHANISM
We perform ablation experiments on the pre-training data and the selection mechanism in the zero- shot case. As described in Section 4.1 the pre-training data is used pre-train the generator and discriminator, completely independent of the experimental datasets in self-consistent training.
To explore the influence of pre-training data self-consistent training, round when training the discriminator, and only the generated data is used. But when the generator trained, pre-training data still retained prevent language degeneration and lack of expressive diversity of the generation. The result of removing pre- pre-training data is shown as the green curves in Figure 3 With all other training parameters being the same, after the same number of training rounds, the discriminator is slightly worse compared to the original method (red curves in Figure
longer add in each
We use SimCSF-BERT-har calculate scores Chinese datasets and Hap-SimCSF-BFEIE wawee---ab--ucccceee English datasets. (Gao et al. 2021
8

---

Under review as a conference paper at ICLR 2023
APUME CHIP-STS
0.20
Chinsso COP
9.15
MAPC
KI. Divergence -first round
KI. Divergence -last round
AFQMC
0.29
0.26
CHIP-STS
0.16
0.08
Chinese-QQi QQP
0.18
0.06
Round Numss
MRPC 0.22
0.02
Figure 2:
The KL Divergence be- tween the score distributions of the Discriminator and the Generator.
Table 4: The KL divergence in the first and last rounds.
However, the green maintain upward trend and very close to the red curves in all datasets except CHIP-STS. This shows that the generated data plays a key role in continuously improving the discriminator, while the pre-training data has limited role.
AFOMC
CHIP-STS
Chinese-QOP
MRPC
6.5
0.725
0.752
0.325
0.82
0.775
0.875
6.4
0.625
0.31
Procosed Tool
Pricesed Metool
Prepesed Metioo
PrrensediMettv Pertos
0.93
0.75
Figure 3: Results of ablation experiments on pre-training data and selection mechanism. Results of the proposed method, results without pre-training data, and results without the selection mechanism are given in red, green, and blue, respectively.
In order explore the effect of the selection mechanism training the discriminator, we remove the selection mechanism when training the discriminator, while the training of the generator remains unchanged. The blue curves in Figure depict the performance of the discriminator in each round after removing the selection mechanism. Compared to the original method (red curves), the discrim- inator only improves in the first round after removing the selection mechanism, which demonstrates the importance of the selection mechanism on the discriminator for the convergence of the self- consistent learning framework.
CONCLUSION
In this paper, we propose self-consistent learning framework enable cooperative training of the generator and the discriminator. During the training process, the generator and the discrimina- tor continuously enhance each other until reaching score consensus. This framework can utilize both limited labeled data and large-scale unlabeled domain-related corpus. Experimental results four Chinese/English sentence semantic matching datasets demonstrate that as a form closed- loop training, proposed framework achieve improved generators and discriminators.
state-of-the-an results with continuously
For future work, will explore the effectiveness of our self-consistent learning framework more NLP tasks, since the framework is straightforward and has no additional requirements on generators and discriminators.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We now discuss the efforts that have been made to ensure the reproducibility of our work. We have packaged the executable code and data into supplementary materials, which can be downloaded and run directly. In addition, we also provide detailed experimental parameters in the appendix to reproduce the experimental results. All datasets and code platforms (Pytorch) we use are public.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
David Alvarez-Melis, Vikas Garg, and Adam Tauman Kalai. Why gans are overkill for nlp. arXiv preprint arXiv:2205. 2008:0088880 2022. URL hRtpsss:s//axx org/ /as///b////////// 20s//2055 09838
Eric Arazo, Diego Ortego. Paul Albert, Noel E Connnr, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8, IEEE, 2020.
Martín Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In 5th International Conference Leurning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017. URL https: //openzeviww.oo formm id-ii4dddkkkk
qw5xe
Amin Banitalebi-Dehkone and Yong Zhang. Repaint: Improving the generalization of down- stream visual tasks by generating multiple instances of training examples. In 32nd British Ma- chine Vision Conference 2021, BMVC 2021, Online, November 22-25, 2021, pp. 122. BMVA wwm.cco//sssss. /ssetts
Press, 2021. URL https:/ /ttp:///wwwwwo www. www.bbbwe222111 wwwp:::2211viitttte -virtua
papers/ 0068. pdf
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Kerbert-Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL A.t. 4155 html L457c0a6bfcb4936 (11774411188881144414441111
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
net/ www.foorumiiiiie /tttp:://openrevi/ htr.ea/wwweeneettooorrrro.eeeea -oruuEr-------3
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp- 2633-2650, 2021.
Jing Chen, Qingcai Chen, Xin Liu, Haijun Yang, Daohe Lu, and Buzhou Tang. The BQ corpus: A large-scale domain-specific Chinese corpus for sentence semantic equivalence identification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4946-4951, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/ w1/D18-1536. URL http::::://alaaahhoooo...o
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard. Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omemicck, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Er- ica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellsterr Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language model- ing with pathways. CORR, abs/2204.02311. 2022. doi: 1445550/rrxiw.22440022 02.111. URL https: /tops:///oii. www./oo...o.o. 4tcg/1..4555/arrae 2204.22242000 .23111

---

Under review as a conference paper at ICLR 2023
Kevin Clark, Minh-Thame Luong, Quoe V. Le, and Christopher D. Manning. ELECTRA: pre- training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenRe- net. iitoormm?iddiie XMH Btv
view.net, 2020.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, 2020. As- URL https:
sociation for Computational Linguisties. doi: 1/22000a// 7/aclanthologee ortpyy..rr////0/ mma-mmiinnn....... 747
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre- trained models for Chinese natural language processing. In Findings of the Association for Com- putational Linguistics: EMNLP 2020. pp. 657-668, Online, 2020. Association for Computational Linguisties. doi: 10.18653/v 1t133311122000 1/20200 findings-emnl.55.. URL https: https::/aclaatholloo
org/20200 www.:/iiiiinnsiieeemoe emn 58
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. Pre-training with whole word masking for chinese bert. IEEE/ACM Transuctions on Audio, Speech, and Language Processing. 29:3504-3514 3514, 2021.
doi: 10. 1109/TAS....2022.333222. 33124365.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. - Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988 Flo- rence, Italy, 2019. Association for Computational Linguistics. doi: 10.1865//1//11--2225 URL
1285
Emily L. Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 28: Annual Conference on Neural Infor- mation Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. neurips............ wtt.iips.cccpppeee wttpc/pppp///////aa//..pp/ Abstrbatt.
1486-1494, 2015. URL aal
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Min- nesota, 2019a. Association for Computational Linguistics. doi: 10. 10.11863//1/N19 www.aa/wwwh.oooo.oooorooa-----
URL.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4171 4171-4186, Minneapolis, Min- nesota, 2019b. Association for Computational Linguistics. doi: 10. 11111186///11111444 URL http:////accaaatccoooooroorr////
Varun Gangal, Steven Y Feng, Malihe Alikhani, Teruko Mitamura, and Eduard Hovy. Nareor: The narrative reordering problem. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 11645-10653, 2022.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, 2021.
Ian J. Goodfellow, Jean Abadee-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks, 2014. URL https:
2661
12

---

Under review as a conference paper at ICLR 2023
Ishaan Gulrajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. In Isabelle Guyon, Ulrike von Luxburg, Samy Ben- gio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 5767-5777, 2017. URL 692c3blc5dccd5299 (tw.a//222336670000006688800 Abstract.ht.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL http:://ppenrevieew
net peet/formm20 HygGQyrFvH
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with gumble-softmar In International Conference on Learning Representations (ICLR 2017). OpenReview. net, 2017.
Jisoo Jeong, Seungeui Lee, Jeesoo Kim, and Nojun Kwak. Consistency-baset semi-supervised learning for object detection. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' D'Alché-Buc, Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 10758-10767, 2019. URL httpts::/ppopseeeeeettt ngs neurips. wttpp.cccpaaeee wctppppr//0119 - .html
Abhishek Kumar, Prasanna Sattigeri, and Tom Fletcher. Semi-supervised learning with gans: Manifold invariance with improved inference. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5534 5544, 2017. URL http:tpsss//prrceeennggg//ee neur ips. CC/ http:/papr/2211//mm/ html
Sylvain Lamprier, Thomas Scialom, Antoine Chaffin, Vincent Claveau, Ewa Kijak, Jacopo Staiano, and Benjamin Piwowarski. Generative cooperative networks for natural language generation. arXiv preprint arXiv:2201. 2022. /ttp:://s//
org abs 2201. 12320
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, net/ forum?id
April 26-30, 2020. OpenReview.net, 2020. URL /etttps://ppezeeeeee
#leA7ABEVS
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205. 14334, 2022.
Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. LCQMC: large-scale Chinese question matching corpus. In Proceedings of the 27th Interna- tional Conference on Computational Linguistics, pp. 1952-1962, Santa Fe, New Mexico, USA, 2018. Association for Computational Linguistics. URL https://aclanthollee .rrg C18-1166
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv e-prints, pp. arXiv-1907, 2019.
Yi Ma, Doris Tsao, and Heung- Yeung Shum. On the principles of parsimony and self-consistency for the emergence of intelligence. Frontiers of Information Technology & Electronic Engineering, pp. 1-26, 2022.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL //openrevies ew. net. wtw.e//o.r///eee d 7uudxe
13

---

Under review as a conference paper at ICLR 2023
Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. Dgsan: discrete genera- tive adlf-aaveraliaal network. Neurocomputting. 448:364-33 -399,,4-72227 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Nils Reimers and Iryna Gurevych. Sentence-BERY Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing Hong Kong, China, November 2019. Association for Com- (EMNLP-IJCNLP), pp. 1...1853//1/DD-11 119111410.111.... URL putational Linguistics. doi: 10.
/
D19- 1410
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference
Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 2226 2234-2234, 2234, 2016. URL httt:::/pprcceedingg nwwrips.rcccoo paper/2015 hash 8a33833333330 792 cb2d8 11166611666650a00 - Abstract .html
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-660 608,66668222000 2020.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: En- hanced Transformer with Rotary Position Embedding. arXiv e-prints, art. arXiv:2104.09864, 2021. TtLLhttps:/// /ttps::///rvvvii
abs / 21022110400 09864
Jianlin Su, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformerv2: A faster and better roformer
zhuiyiai. Technical report, 2022. URL. htttps:///ithhbp cwm/Zhut Technol o
roformer-v2
Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learn ing, 109(2):373-440, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL httpss: neur.s.............. 3FFa2445577dee1110000eeccceaaaa3 abte::icccc..eaaaaaaaattt.......
http/paaer2201//aaaeee
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding, In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, net/ fttt/ffrummidd
May 6-9, 2019. OpenReview.net, 2019. URL https::/
R54km2
Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, Jianheng Zhuo Qi Yang. Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang, Xiaojun Wu, Zhong- shen Zeng, Chongpei Chen, Ruyi Gan, and Jiaxing Zhang. Fengshenbang 1.0: Being the foundation of chinese cognitive intelligence. arXiv preprint arXiv:2209.02970 2022. URL https: /ttp://aaxvvvvv.v abtpg/ab//2209 02970
Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng. and Luo Si. Struct- bert: Incorporating language structures into pre-training for deep language understanding. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. BtLhhttss://// /ttp:////pp///eee/e///// net/ forum?id BJqQ41 SSPPH
URLAttpp.///ppeeeeveee
14

---

Under review as a conference paper at ICLR 2023
Maciej Wiatrak, Stefano V Albrecht, and Andrew Nystrom. Stabilizing generative adversarial net- works: A survey, arXiv preprint arXiv:1910.00992, 2019. wwtp / 1910.00922
Qingyang Wu, Lei Li, and Zhou Yu. Textgail: Generative adversarial imitation learning for text generation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp- 14067-14075, 2021.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33:6256-6268, 2020.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao. Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, Hc Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. CLUE: A Chinese language understanding evaluation bench- mark. In Proceedings of the 28th International Conference on Computational Linguistics, pp- 1762-4772, Barcelona, Spain (Online), 2020. International Committee on Computational Lin- guisties. doi: 10.18653/11/2220.colll 1tw.:00-oiiigmmii.411111 main.419. URL https: http://aalanthhloge , org/ 2020. col.colingg main. 419
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) pp. 3687-3692, Hong Kong, China, 2019a. Association for Computational Linguistics. doi: 10.18653/v1/D19- 1382. URL https: https::/aclaathologgy
org Skg/Di9-10 1001-11822 1382
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. Xlnet: Generalized autoregressive pretraining for language understanding, In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, 2019b. URL https: http:://cccceeeeeeeeee nwwriesrrpp.... cttssccc/ppe cc/paper/ Canada, pp. 22019//aah/cccaee651/e1833333333
Abst aatteettttettttttttttaaaa.aaae htm
Haiyan Yin, Dingcheng Li, Xu Li, and Ping Li. Meta-cotgan: A meta cooperative training paradigm for improving adversarial text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 9466-9473, 2020.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. In Satinder P. Singh and Shaul Markovitch (eds.), Proceedings of the Thirty-Firs, Firrty-Fiistaaeee AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, Cal- isornia, USA, pp. 2852-2858. AAAI Press, 2017. URL hnttp://aaet org//ca nndx.. php/ AA//AAAT///A / /ttp///AAA////aa///eieeeeie 14344
Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, and Jie Tang. Wudancorpora: super large-scale chinese corpora for pre-training language models. Al Open, 2:65-68, 2021. ISSN 2666-6510. doi: https//do.oorg/1..00///aaannnpp
0021.06.001. URL https: /www///wwww///////o/////ww scwencediiiicccoccoooooooo wwwttttomeeeeeeeee.ooo.oo...ttot wwtm///cieeeee/ttticee.p.. 2266665102100000
Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang. Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, Luo Si, Yuan Ni, Guotong Xie, Zhifang Sui, Baobao Chang, Hui Zong, Zheng Yuan, Linfeng Li, Jun Yan, Hongying Zan, Kunli Zhang, Buzhou Tang, and Qingcai Chen. CBLUE: A Chinese biomedical language understanding evaluation benchmark. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguiisiiii ( (Volume 1:
Long Papers), pp. 7888-799 7918,7915, Dublin, Ireland, 2022a. Association for Computational Linguis- tics. doi: 18653/v3/v l/2/22.acl -ong..0....44. URL ht / /at/acaaantthtooo wrtp:yyy.rrg2000....
aac oog.on..44.
15

---

Under review as a conference paper at ICLR 2023
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transforme language models. arXiv preprint arXtv:2205.01068, 2022b. URL
abs/2205.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Eval- uating text generation with BERT. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30. 2020, OpenReview.net, 2020. URL
hitpss///ppereet ew. net for
16

---

Under review as a conference paper at ICLR 2023
A PROOFS
A.1 PROOF OF THEOREM 1
Theorem At round Lgiiee the previous round discriminator D the aim of the optimization of the generator Gg boils down to,
minDKilPpg (.)Pagg())
0
where DEL is the standard KL divergence, p (-) refers to the degree of confidence that the sen- tences generated by the generator belong to a given class k (we can either train the generator to express its confidence in the generated sentencestLin al. 2002) or use a fixed third-party model to score them Gao et al. 2)21)),, and (-)) by the discriminator.
(-) the probability of being classified into class k given
Proof. We use the previous round generator gg generate samples, and filter them using the previous round discriminator & D' with a threshold εt-1, then these samples are used for training of the current round generator Gg. Therefore, the optimization of 9 will tend to maximize the probability that the generated samples pass the discrimination for the fixed D For given class k, we have
max E
Pog (x) s.t. Filter (x)=1 =1
where the definition of function filter (t- (.) has been given in Equation
The above objective is equivalent to sampling from the generator being optimized round t and making these samples pass the discrimination in round
much as possible, which gives
max B Bu p (x)
9
where (x) is fixed.
A further transformation of the formula shows that
max Egapt p (z)
max
doVB B
k (x)
og
Vo log log (z)
max
N
diVo Σ
(log Olggr (1 (1:))
( (1:))
max
mjn Min (), (-)) d
where (i) uses the integral property that integrating the derivative of a function gives the original function along with constant, (fi) takes advantage of the derivative property of the logarithmic & function, (iii) approximates the expectation of the probability distribution Pg; (.) by using averag-
ing on samples sampling from P (-). and adding a constant term with respect to 0 under the summation would not change its derivative, and (iv) cancels out the integral and the derivative and uses the definition of KL divergence. The above concludes our proof.
B EXPERIMENTS ON DIFFERENT FILTER FUNCTIONS
To compare the effect of different filter functions on the final result, we use four type of functions, including oscillatory function (cosine), constant function and monotonically increasing functions
17

---

Under review as a conference paper at ICLR 2023
(quadratic and linear). For the fairness of comparison, we keep the maxima and minima the same for all functions(excep) for the constant threshold), and the values are given in Appendix F In addition, the number of training rounds for different functions the same dataset remains the same.
In the results below, the best results and the second-ber best results are bold and underlined, respectively.
Table 5: Performance of Different Filter Functions in Zero-Shot Setting. (F1 Score( (%))
AFQMC
CHIP-STS
Chinese-QQP
MRPC
AVG
Baseline
38.25
58.82
57.88
68.54
55.87
Cosine
47.38
74.26
64.39
83.48
67.38
Constant
47.06
74.15
68.67
84.11
68.50
Quadratic
51.75
73.09
70.85
83.48
69.79
Linear
51.36
77.42
70.51
84.61
70.98
As can he seen from the Table 5 in the zero-shot setting, the chosen linear function outperforms the other functions, and all the filter functions show an averaging 10+ AP improvement relative to the baseline. Therefore, the self-consistent learning framework makes it casy choose a certain threshold function and perform well, and the results are not so sensitive to the choice of the functions.
AFOMC
CHIP-STS
Chinese DOP
MRAC
313
34.4
345
6.00
Uree
1.25
Unee
3.00
Danial
Figure 4: Results of contrast experiments on Cosine(green), Constant(orange)) Quadratic(blue) and Linear(red) function.
Figure dipicts the comparison results in each round. The linear function (red line) is significantly better than the other functions on both CHIP-STS and MRPC datasets. In the AFQMC and Chinese- QQP datasets, the quadratic function (blue line) is slightly more effective than the linear function. In general, we can intuitively see that all functions show significant increase relative to the starting point.
Table 6: Performance of Different Filter Functions in Fine-Tune Setting. (FI Score %))
AFQMC
CHIP-STS
Chinese-QOP
MRPC
AVG
Baseline
64.19
86.93
76.56
91.21
79.72
Cosine
66.43
88.01
77.33
92.63
81.10
Constant
66.57
88.15
78.45
92.51
81.42
Quadratic
66.37
87.76
79.26
92.75
81.54
Linear
66.59
88.39
78.43
92.78
81.55
Table 6 shows the effects of different filter functions in the fine-tune experiment. It can be that all functions have a 2 AP increase relative to the baseline, and the chosen linear function achieves the best performance all datasets except Chinese-QQP.
18

---

Under review as a conference paper at ICLR 2023
C CONTRASTIVE EXPERIMENTS WITH ADVERSARIAL TRAINING
In this section, We further demonstrate the superiority of the cooperative approach by comparing the results with adversarial experiments. All experimental settings independent of the training method remain the same in the adversarial training.
During the experiments, the generator is longer trained using the samples filtered by the discrim- inator, but the rewards passed by the discriminator assist the training. All generated samples are treated as negative samples when training the discriminator.
Specifically, G takes the prompt "s"" is similar to and the first M tokens of s as input to get M sentence pairs < s", sth 2, where is from M. Note that we repeat the process of generating sentences N times to reduce the negative impact caused by the large variance of the rewards The sentence pair Imm of the m-th token the n-th time is formalized as
In f< 5" Sm >= Go(s Is mm s"; ;N)
Once the M * N sentence pairs 9m are generated, they are passed as input to the D to obtain the probability score Qm for each of them. We take the average of Qm over N as the reward Q. corresponding to the m-th token. If the sentence length of s" is greater than M, the rewards of the remaining tokens are all the same those of the M-th token. Taking the m-th token as an example, the rewards Qm can be formalized as
Q (m) =
EN n= (o
m S M m > M
Q(M)
Therefore, the objective function for training the generator G is,
L 88)
1 8
loging (st (8112,8)))) 80) * Qc)
The loss function of training the discriminator remains the same Equation 4 but differing from cooperative training, the generated samples are regarded as negative samples to the discriminator, and the training target for the discriminator can be given by
min Ex-pas [log D -X Ex- [log (1 Dx))))]
The results of zero-shot and fine-tune on the four datasets are shown in Tables and
As can be seen from Table 7 in the zero-shot setting, training in an adversarial manner does not give any improvement over the baseline. Because the initial diseriminator in the zero-shot setting is very weak in distinguishing positive and negative samples, it is reasonable to believe that if all generated samples are considered negative samples from the very beginning, is difficult for the discriminator to know how to distinguish positive samples. As result, the F1 scores on both AFQMC and CHIP-STS datasets end up being O, while the scores on the Chinese-QQP and MRPC datasets fluctuate intensively with the number of rounds, which further validates the instability of the adversarial training in the zero setting.
For the fine-tune experiments, Table shows that training in adversarial manner slightly improve the performance on the Chinese-QQP and MRPC datasets, but is still worse than the coop- erative training. On the AFQMC and CHIP-STS dataset, adversarial training makes it even worse relative to the baseline. It worth noting that the whole process of adversarial training is so unstable and it is easy to collapse after a few training rounds.
In practice, we take M 5, N for of calculation.
19

---

Under review as a conference paper at ICLR 2023
Table 7: Zero-Shot Performance of Adversarial Trained Discriminator. (Fl score (%))
AFQMC (Financial)
CHIP-STS (Medical)
Chinese-QQP (Common)
MRPC (News)
Round Num
baseline
38.25
58.82
57.88
68.54
0.0
8.73
21.71
4.19
0.02
7.13
49.30
7.06
3
0.0
0.29
42.94
5.32
4
0.0
1.09
41.13
0.0
0.0
0.10
43.10
1.72
0.39
34.30
67.38
0.0
0.20
42.62
48.31
8
0.0
0.20
34.95
37.97
0.20
41.81
10
0.20
40.00
Table 8: Fine-Tune Performance of the Discriminator, (Fl score (%))
AFQMC
CHIP-STS Chinese-QQP
MRPC
AVG
Bascline
64.19
86.93 80.46
76.56
91.21
79.72
Adversarial
58.37
77.93
92.18
77.24
Cooperative 66.59
88.39
78.43
92.78
81.55
(Cur Method)
D MODEL DETAILS
The 5.0B Transformer-Xi is pre-trained on 32 A100s with 40G memory for 45 days, the batch size is set 32*8=256. After running 445k steps, the final validation loss reduces to about 2.4. The 2.7B OPT incrementally trained on the basis of the open-source model.
During the pre-training of the generator model, Transformer-X XL and design a special attention mask to concatenate the multiple input sentences into sample, reduce the number of the padding token in batch and therefore increase the number of effective tokens. To make the generation more robust, we add noise to the original sen- tences by randomly replacing or discarding tokens with 5% probability. In addition, the prompts that we use for Chinese generation and English generation are as follows,
utilize the memory-cacht mechanism of
Chinese prompt: " (en: A similar sentence to "ga" "ghn.)
English prompt: "s"" is similar to "sh"
When training the discriminator, following the usage of special tokens in BERT (Devlin et al. 2019a), we use [SEP] to concatenate two sentences and take the embedding at the [CLS] posi- tion to represent the whole sentence to predict the label. Moreover, we utilize the mask method in BERT randomly mask 15% of the input tokens.
DATASET DETAILS
The statistics of the experimental datasets reported in Table 9
Other Chinese datasets (LCQMC (Liu et al. 2018). OPPO, PAWS-X-zh (Yang et al. 2019a). BQ CCen al. 2018). CCKS, Chinese-STSs STS-B /Wang et al. 2019 and English datasets (QQP (Wang et al. 2019). STS-B (Wang et al., 2019). PAWS-X (Yang et al. 2019a)) are collected and used as the corpus of similar sentence pairs for pre- pre-training the generator.
The Chinese- QQP dataset contains 9000 pieces of data randomly selected and translated from the English QQP dataset, which is then divided into training and test set in ratio of 3:2.
20

---

Under review as a conference paper at ICLR 2023
Table 9: Statistics of Experimental Datasets
AFQMC
CHIP-STS
Chinese-QQP
MRPC(en)
domain-related
68668
32000
10800
8,152
Zero-Shot
corpus
test dataset
4316
4000
3600
1725
training dataset
34334
16000
5400
4076
Fine-Tume Tune
test dataset
4316
4000
3600
1725
F PARAMETER SETTINGS
The training parameters zero-shot are shown Table 10 The three thresholds are used to select positive and negative examples for training the discriminator and positive examples for training the generator, respectively. We adopt cosine annealing learning rate decay strategy during training.
Table 10: Parameter Settings of Zero-Shot.
AFQMC
CHIP-STS
Chinese -Qinese-QQP
MRPC
Maximum Threshold
0.8
0.9
0.95
0.95
-negative
Minimum Threshold
0.6
0.7
0.8
0.8
-negative
Maximum Threshold
0.8
0.9
0.95
0.95
ppssitiv
Minimum Threshold
0.6
0.7
0.8
0.8
-pssitive
Maximum Threshold
0.6
0.9
0.95
0.95
-enerator
Minimum Threshold
0.6
0.7
0.8
0.8
-generator
Threshold Increase
0.07
0.1
0.05
0.05
Sentence Num
6000
6000
3000
4000
Learning Rate
2e-5
Warm Up Steps
40
Early Stopping Patience
Generator Batch Size
2(concatenate 30 samples)
24
-training
Generator Batch Size
512
100
-predicting
Discriminator Batch Size
64
32
-training
Discriminator Batch Size -predicting
384
96
The training parameters of fine-tuning are shown in Table 11
G
GENERATE SAMPLES
The generators use nucleus sampling (Holzzman et al., 2020) to generate similar sentences. Gener- ated examples in English are shown Table 12 and in Chinese Table [3
21

---

Under review as a conference paper at ICLR 2023
Table 11: Parameter Settings of Fine-Tune.
AFQMC
CHIP-STS
Chinese-QQUl
MRPC
Maximum Threshold
0.98
0.98
0.84
0.8
-negative
Minimum Threshold
0.9
0.7
0.6
0.6
-negative
Maximum
Threshold
0.98
0.98
0.98
0.8
-positive
Minimum
Threshold
0.9
0.7
0.9
0.6
positive
Maximum Threshold
0.98
0.98
0.98
0.8
generator
Minimum Threshold
0.9
0.7
0.9
0.6
-gnerattr
Threshold Increase
0.07
0.07
0.07
0.2
Sentence Num
6000
6000
3000
3000
Learning Rate
5e-6
Warm Up Steps
40
Early Stopping Patience
Generator Batch Size
2(concatenate 30 samples)
24
-training
Generator Batch Size
512
100
-predicting
Diseriminator Batch Size
32
64
32
32
-training
Discriminator Batch Size
256
384
256
96
-predicting
Table 12: Generated Samples in English
MRPC(News)
origin: Hong Kong was flat, Australia, Singapore and South Korea lost 0.2-0.4 percent. first round: the market is flat, the US is flat, Europe is flat, Japan is flat, Malaysia is flat, Philippines is flat, Vietnam is flat, Singapore is last round: Hong Kong unchanged, Australia, Singapore and South Korea were down 0.2-0.4 percent.
origin: Ballmer has been vocal in the past warning that Linux a threat to Microsoft. first round: Microsoft is a threat to Linux.
last round: Ballmer has been vocal the past about the threat Linux poses to Microsoft.
origin: Last year, Congress passed similar, though less expensive, buyout legislation for peanut farmers, ending that Depression-era program.
first round: The govermment is going buy your farm and give you check for $1,00 11,000,000.000
You can't 't refuse.
last round: Congress passed a similar bill last year end Depression-era program for peanut farmers, ending that program.
origin: Licensing revenue slid percent, however, to $107.6 million.
first round: The company's 's net income fell to $1.1 million, or 2 cents per share,
in the quarter ended Sept. 30, from $1.3 million, or 3 cents per share, a year earlier.
last round: Licensing revenue fell 21 percent to $107.6 million in the third quarter.
origin: SCO says the pricing terms for license will not be announced for weeks. first round: we're not going tell you how much it costs buy a car, but we're going to tell you how much it costs to rent one.
last round: The pricing terms for a license for the SCO software will not be announced for several weeks.
22

---

Under review as a conference paper at ICLR 2023
Table 13: Generated Samples in Chinese.
origin: pj T
X TE
(Can I use my Alipay to repay someone else's 's Huabei) first round: AE मं IT T Afting (Can someone else pay for the money from Huabei)
AFQMC (Financial)
last round: TE AE
(Can my Huabei help others repay?)
origin: TEEEEE 2 KE के.
(How does Huabei pay off the arrears)
first round: TEEL A A thtt
(Why does Huabei still have electricity bills)
last round: T 7T
(How to pay off Huabei)
origin:
(QR code for my Huabei payment)
first round: (ि 7E
(Merchant Huabei's QR code)
last round: 7E #: 6%
64 7
(Where is the QR code for Huabei payment)
origin: X Mf 2 5e f 2
(What is the cause of AIDS)
first round: Yd क Ee ft A??
CHIP-STS (Medical)
(What is HIV?)
last round: X E
1 & DH ا
(What causes AIDS?)
origin: T ALE E 1
(High blood pressure is always drooling what is going on)
first round: 75 dtt E E 2 [ $
(What about high blood pressure)
last round: e di > ft 2, & M l
(Why does high blood pressure cause drooling?)
origin: 1 J 6E bKi M LETEEE RH J TE
(Why am getting thinner and thinner now that I have diabetes)
first round: Ks AA & [ $
(What's ' going on with diabetes now?)
last round: A At 7 2/ / ?
(Why do people with diabetes lose weight?)
origin: 1 AE
(How do delete my account from this site?)
first round: E L IIl MR (How to delete a website)
Chinese-QQU (Common)
last round: fxI / tl ER DRO ?
(How can I delete my account?)
origin: * T- K Bbt 13
(What are some good books on electronics?) first round: f 2 F Fr #ft (Any good electronics recommendations) last round: # T- di #3? (What are some good books about electronics?)
origin: 2 TT A
[a] #
(Why does no one see the difference between endless and infinite?)
first round: A/ft/ 2 3 4 A AE
(Why is there no limit in the universe) last round: 1 / H A PEA सहe 2 [ BJ?
(Why does no one know the difference between infinite and finite?)
23


Reviews:
[{"review_id": "vUmOlxWsF7", "reviewer": "Reviewer_r4Wp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work introduced a self-consistent learning framework, where a generator and a discriminator are trained cooperatively and alternatively. A selection mechanism is proposed to filter samples in the training process. The sentence semantic matching task is demonstrated as an example to support the effectiveness of the proposed framework. The results show the framework outperforms standalone discriminative models on four datasets in both zero-shot and full-data settings. ", "summary_of_the_review": "The paper includes interesting ideas and shows the proposed approach can improve performance for sentence sematic matching task. It may be more influential if fundamental evidences or analysis are available to compare cooperative and adversarial setting. ", "strength_and_weaknesses": "Strength: The paper is well-organized, and the main ideas are interesting, and easy to understand. The proposed method is evaluated on multiple datasets with good performance in sentence sematic matching. \n\nWeakness: 1. The work included both positive and negative samples in the training of discriminator, and claimed cooperative setting can lead to better training stability than adversarial setting. The idea makes sense but it is not clear to me how this point is supported either in theoretical or experimental aspects. 2. The definition of the output probability for generator $p_G$ cannot be found. 3. It is mentioned in Eq (1) that $p_D$ depends on the embedding representation H of generated samples. Since it is shown in the paper that the selective mechanism is important to the performance, I think it would be helpful to provide more details on this embedding process, and some evidence or explanation to justify the resulting output probability is somehow “good enough” for selective mechanism. 4. It is mentioned in page 4 that both positive and negative samples would be included in the training of discriminator, (which is a key difference between this method and GAN.) Due to the lack of information on the embedding representation, I fail to see why Eq (2) can make sure of this. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OBpfGApPbz", "reviewer": "Reviewer_YrRh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a co-operative technique for Generator and Discriminator to cooperate and aid each other instead of the standard adversarial game of of the GAN. The proposed technique uses an augmentation scheme to augment the learning data of the generator and data to aid each other's learning progress. ", "summary_of_the_review": "The paper significantly lacks novelty. The proposed technique should ideally also be beneficial to image generation. However, the lack of results raise the suspicion on the generalizability of the technique. The metrics reported on the text based datasets are moderate at best. ", "strength_and_weaknesses": "Strengths:\n\n* Simple technique for assisting the learning of generators and discriminators using a co-operative technique rather than adversarial technique \n\nWeakness:\n\n* Several cooperative GAN based augmentation systems have been proposed and the novelty is extremely low.\n* The experiments only show some NLP based experiments and no Vision based experiments. \n* Ideally the model should also work with image generation but the conspicous absence indicates the non-generality of the technique\n* The proofs are hand wavy and not concrete. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "MN0z08ywamr", "reviewer": "Reviewer_8YkB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a a self-consistent learning framework to train a discriminator and a generator cooperatively in a closed-loop form. The discriminator and the generator enhance each other during multiple rounds of alternating training. Experimnetal results are given to validate that the proposed framework is easy to train and free from instabilities such as mode collapse and non-convergence. ", "summary_of_the_review": "The paper proposes a self-consistent learning framework to enable cooperative training of the generator and the discriminator. The framework can utilize both limited labeled data and large-scale unlabeled domain-related corpus. But the proof of the main theorem still needs to be validated.", "strength_and_weaknesses": "Strength:\nThe paper proposes a self-consistent learning framework to enable cooperative training of the generator and the discriminator. The generator and the discriminator are trained alternatively until reaching a score consensus.  This framework makes plenty use of the limited labeled data and large-scale unlabeled domain-related corpus. \n\nWeaknesses:\nThe paper formulates the main theoretical result into theorem 1, but the proof of theorem in the Appendix is not obvious. In particular, how do you derive from the first line optimization problem to the second optimization problem. Could you provide further detailed proof for the theorem?\nThe choice of the dynamic threshold looks quite arbitrary, could you provide some criterion for the parameter selection?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Kaq4fAMa7h4", "reviewer": "Reviewer_6Yjm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this study, authors proposed a self-consistent learning algorithm to generate data samples to improve downstream performance. The self-consistent learning was designed on generator-discriminator framework which is similar as GAN, but different from that of GAN, the generator and discriminator were cooperatively learned to enhance each other. In the self-consistent learning, authors designed a data selection mechanism to select the generated data with a dynamic threshold, and the selected data were further applied to enhance the generator and discriminator (similar as data augmentation). Finally, authors examined their idea on several sentence semantic matching experiments, and confirmed the effectiveness of their method.", "summary_of_the_review": "The paper proposed a novel cooperative training of both generator and discriminator to generate data for enhance model training. In the model, authors designed a dynamic threshold for data generated data selection in a self-consistent learning framework. Authors experiments showed effectiveness of their idea.  ", "strength_and_weaknesses": "Strength: different from GAN where generator and discriminator are in a competitive learning strategy which is difficult to train, and difficult to use the generator data in further training (it is difficult to judge the quality of the generated data samples ), authors' idea with self-consistency learning could learn both the generator and discriminator in a cooperative way to enhance each other. Therefore, the training is stable and guarantee to converge.\n\nWeakness: The dynamic threshold design as given in Eq. 3 seems important in deciding the selection of the data, however, authors gave a simple incremental formulation in later part for designing the dynamic threshold. And there is no further explanation and ablation study on the design of the dynamic threshold. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject