Pre-training Language Model as a Multi-perspective Course Learner
Year: 2023

Abstract:
ELECTRA (Clark et al., 2020), the generatordiscriminator pre-training framework, has achieved impressive semantic construction capability among various downstream tasks. Despite the convincing performance, ELECTRA still faces the challenges of monotonous training and deficient interaction. Generator with only masked language modeling (MLM) leads to biased learning and label imbalance for discriminator, decreasing learning efficiency; no explicit feedback loop from discriminator to generator results in the chasm between these two components, underutilizing the course learning. In this study, a multi-perspective course learning (MCL) method is proposed to fetch a many degrees and visual angles for sample-efficient pre-training, and to fully leverage the relationship between generator and discriminator. Concretely, three self-supervision courses are designed to alleviate inherent flaws of MLM and balance the label in a multiperspective way. Besides, two self-correction courses are proposed to bridge the chasm between the two encoders by creating a “correction notebook” for secondary-supervision. Moreover, a course soups trial is conducted to solve the “tug-of-war” dynamics problem of MCL, evolving a stronger pre-trained model. Experimental results show that our method significantly improves ELECTRA’s average performance by 2.8% and 3.2% absolute points respectively on GLUE and SQuAD 2.0 benchmarks, and overshadows recent advanced ELECTRA-style models under the same settings. The pre-trained MCL model is available at https://huggingface.co/McmanusChen/MCLbase.

heading: 1 Introduction
text: Language models pre-training (Radford et al., 2018, 2019; Devlin et al., 2019; Liu et al., 2019) has shown great success in endowing machines with
∗Contribution during internship at Microsoft. † Corresponding author.
the ability to understand and process various downstream NLP tasks. A wide range of pre-training strategies have been proposed, among which the most prevailing object is the masked language modeling (MLM) (Devlin et al., 2019). Such autoencoding language modeling objects (Vincent et al., 2008) typically first randomly corrupt a certain percentage of training corpus with masked tokens, and then encourage encoders to restore the original corpus. To reduce the randomness of pre-training and produce a sample-efficient method, ELECTRAstyle frameworks (Clark et al., 2020) leverage an Transformer-based (Vaswani et al., 2017) generator training with MLM to build challenging ennoising sentences for a discriminator in the similar structure to carry out the denoising procedure.
Typically in the ELECTRA-style training, the generator first constructs its semantic representations through MLM training and cloze these masked sentences with pseudo words; in the meanwhile, the discriminator inherits the information from the former and distinguish the originality of every token, which is like a step-by-step course learning. However, only MLM-based generator training may lead to monotonous learning of data, which conduces to the incomprehensive generation and imbalanced label of corrupted sentences for the discriminator (Hao et al., 2021). Besides, interactivity between the two encoders stops abruptly except the sharing of embedding layers (Xu et al., 2020; Meng et al., 2021), since there is no direct feedback loop from discriminator to generator.
To enhance the efficiency of training data and to adequately utilize the relationship of generator and discriminator, in this work, we propose a sampleefficient method named multi-perspective course learning (MCL). In the first phase of MCL, to fetch a many degrees and visual angles to impel initial semantic construction, three self-supervision courses are designed, including cloze test, word rearrangement and slot detection. These courses instruct
114
language models to deconstruct and dissect the exact same corpus from multi perspectives under the ELECTRA-style framework. In the second phase, two self-correction courses are tasked to refine both generator and discriminator. A confusion matrix regarding to discriminator’s recognition of each sentence is analyzed and applied to the construction of revision corpora. Secondary learning is carried out for the two components in response to the deficiencies in the previous course learning. At last, the model mines the same batch of data from multiple perspectives, and implement progressive semantic learning through the self-correction courses.
Experiments on the most widely accepted benchmarks GLUE (Wang et al., 2019) and SQuAD 2.0 (Rajpurkar et al., 2018) demonstrate the effectiveness of the proposed MCL. Compared with previous advanced systems, MCL achieved a robust advantage across various downstream tasks. Abundant ablation studies confirm that multi-perspective courses encourage models to learn the data in a sample-efficient way. Besides, a course soups trial is conducted to further interpret and dissect the core of multi-perspective learning, providing a novel approach to enhance the pre-training efficiency and performance.

heading: 2 Preliminary
text: In this work, we built our system based on the ELECTRA-style framework. Thus, the framework of ELECTRA is reviewed. Unlike BERT (Devlin et al., 2019), which uses only one transformer encoder trained with MLM, ELECTRA is trained with two transformer encoders: a generator G and a discriminator D. G is trained with MLM and used to generate ambiguous tokens to replace masked tokens in the input sequence. Then the modified input sequence is fed to D, which needs to determine if a corresponding token is either an original token or a token replaced by the generator.
Generator Training Formally, given an input sequence X = [x1, x2, ..., xn], a mask operation is conducted to randomly replace its tokens with [MASK] at the position set r.1 And the masked sentence Xmask = [x1, x2, ...,[MASK]i, ..., xn] is fed into the generator to produce the contextualized representations {hi}ni=1. G is trained via the following loss LMLM to predict the original tokens
1Typically the proportion is set as 15%, which means 15% of the tokens are masked out for each sentence.
from the vocabulary V at the masked positions:
pMLM(xt|hi) = exp(x⊤t hi)∑|V | t′=1 exp(x ⊤ t′hi) , (1)
LMLM = E ( − ∑
i∈r log pMLM
( xi ∣∣hi ) ) , (2)
where {xt}|V |t=1 are the embeddings of tokens that are replaced by [MASK]. Masked language modeling only conducts on the masked positions.
Discriminator Training. G tends to predict the original identities of the masked-out tokens and thus X rtd is created by replacing the masked-out tokens with generator samples:
xrtdi ∼ pMLM (x|hi) , if i ∈ r ; xrtdi = xi, else. (3)
D is trained to distinguish whether the tokens in X rtd have been replaced by G via the replaced token detection (RTD) loss LRTD:
pRTD(x rtd i = xi ∣∣hi) = exp(w⊤hi)
(1 + exp(w⊤hi)) , (4)
LRTD = E ( − ∑
xrtdi =xi
log pRTD ( xrtdi = xi ∣∣hi )
− ∑
xrtdi ̸=xi
log ( 1− pRTD ( xrtdi = xi ∣∣hi )) ) ,
(5)
where w is a learnable weight vector. This optimization is conducted on all tokens.
The overall pre-training objective is defined as:
LELECTRA = LMLM + λLRTD. (6)
where λ (typically 50) is a hyperparameter used to balance the training pace of G and D. Only D is fine-tuned on downstream tasks after pre-training.

heading: 3 Challenges
text: Biased Learning Though the ELECTRA training method is simple and effective, treating corpora from a single perspective could cause biased learning. As for the progress of MLM and RTD, there exists an inherent flaw that G might predict appropriate but not original token on the [MASK] position, and such appropriate expression still needs to be judged as substitution by D. For example, if the original sequence “Alan buys an apple” is
masked as “Alan [MASK] an apple”, there are too many candidate words like “eats, peels, cuts” that can replace [MASK] to form a harmonious context. Thus, to request D to continue to distinguish replaced tokens is a difficult even awkward task. Treating the same piece of data in a single way reduces the effectiveness of training. As for the distribution of generated corpus from G, the labelimbalance may gradually emerge with the MLM training of G, which could disturb the RTD training of D. As the semantic construction of G thrives with the pre-training, the pseudo token on [MASK] becomes more reasonable and even matches the original word. Thus, the proportion of replaced tokens in the training sentences of D collapses, which interferes with the binary classification task seriously. In this work, three self-supervision courses are tasked to train the model in a multi-perspective way, improving the learning efficiency of data and balancing the distribution of labels.
Deficient Interaction The core of selfsupervision training is to ingeniously design and construct labeled data from original corpora. Evolving from random masking as BERT does, ELECTRA provides more realistic corpora by generator samples, encouraging G and D to compete with each other. However, there is no explicit feedback loop from D to G, resulting that the pre-training of G is practically dominated by MLM as before. To bridge the chasm between these two components, in this work, we take advantage of discriminant results from D to create a “correction notebook” for both G and D, and propose two self-correction courses to provide secondary-supervision. Revision-training fully leverages the relationship and characteristics of the two encoders to increase the quality of training.

heading: 4 Methodology
text: In this section, we will start by formulating three self-supervision courses which encourage models to treat data in a multi-perspective way. Then two self-correction courses are elaborated, deriving from the course-like relationship between G and D. These various courses are weaved into the entirety of the multi-perspective course learning method.

heading: 4.1 Self-supervision Course
text: The essentiality of large-scale data pre-training, undoubtedly is to excogitate a way to take full advantage of the massive rude corpora. ELECTRA has
provided an applicable paradigm for models to construct semantic representations through ennoising and denosing. Based on this framework, we extend the perspective that models look at sequences and propose three binary classification tasks in order to improve training efficiency, alleviate biased learning, and balance label distributions.

heading: 4.1.1 Replaced Token Detection
text: On account of the compelling performance of pre-training language models with masked language modeling, we retain the replaced token detection task from ELECTRA. Following the previous symbol settings, given an original input sequence X = [x1, x2, ..., xn], we first mask out it into Xmask = [x1, x2, ...,[MASK]i, ..., xn], which is then fed into G to get the filling-out sequence X rtd = [x1, x2, ..., x rtd i , ..., xn] by generator samples. Finally, D is tasked to figure out which token is original or replaced. As illustrated in the Section 2, G and D are trained with LMLM and LRTD respectively. MLM endows G with fundamental contextual semantic construction by cloze test, and RTD is a higher level course for D to let the model drill down into context for seeking out dissonance in the pseudo sequence X rtd.

heading: 4.1.2 Swapped Token Detection
text: Intuitively, recombination tasks contribute to sequence-related learning. As mentioned in Section 3, information absence at the [MASK] position will cause the unreliability of generated pseudo words. Whether the filled sample is appropriate or not, biased learning occurs interfering training of D. Thus, to reserve primeval message for precise prediction, without slashing the degree of task difficulty, we present swapped token detection (STD)
course to sharpen the model’s structure perception capability through a word rearrangement task.
For an input sentence X = [x1, x2, ..., xn], a random position set s is chosen for swapping operation.2 Precisely, tokens at the chosen position are extracted, reordered and filled back into the sentence. G is required to restore the swapped sentence Xswap to X , and the adjacent D is tasked to discriminate which token is swapped in Xstd by generator samples. Note the contextualized representations from G as {hi}ni=1, the training process of swapped language modeling (SLM) is formulated below:
pSLM(xs|hi) = exp(x⊤s hi)∑|V |
s′=1 exp(x ⊤ s′hi)
, (7)
LSLM = E ( − ∑
i∈s log pSLM
( xi ∣∣hi ) ) , (8)
where {xs}|V |s=1 are the embeddings of tokens at the swapped positions. Note that the vocabulary V is still the same across all courses, because it helps the generation of G in a consistent and natural environment, even the correct answer is lying in the pending sequence during SLM. SLM is only conducted on tokens at the swapped positions.
SLM brings G to making reasonable even original predictions on the swapped positions, taking the attention of training from guessing of a single word to comprehensively understanding structure and logic of the whole sequence. The swapped token detection (STD) course of D is naturally formed as a deja vu binary classification. Xstd is created by replacing the swapped positions with generator samples:
xstdi ∼ pSLM (x|hi) , if i ∈ s ; xstdi = xi, else. (9)
D is trained to distinguish whether the tokens in Xstd is original or not via the swapped token detection (RTD) loss:
pSTD(x std i = xi ∣∣hi) = sigmoid(wTs hi), (10)
LSTD = E ( − ∑
xstdi =xi
log pSTD ( xstdi = xi ∣∣hi )
− ∑
xstdi ̸=xi
log ( 1− pSTD ( xstdi = xi ∣∣hi )) ) .
(11) 2In the absence of special instructions, the proportion is
set as 15%.
where ws is an independent trainable parameter from w since each of courses uses its own binary classification head.

heading: 4.1.3 Inserted Token Detection
text: With the pace of pre-training with MLM and SLM, G is inevitably able to produce much more harmonious sequences for the consummation of semantic learning. In the meanwhile, the label distribution of corrupted sentences provided by G becomes magically imbalanced, since almost all tokens exactly match the words in the original sentence. Thus, training of D faces serious interference and lack of efficiency. The propensity of the training labels leads to the propensity of D’s judgment.
To alleviate the issue of label-imbalance, and to seek another perspective of treating data, we propose the inserted token detection (ITD) course. For a given sentence X = [x1, x2, ..., xn], [MASK] is randomly inserted into the sequence at the inserted position set i. The extended sentence X in contains several illusory vacancies waiting for the prediction of G. Subsequently, D has to figure out which token should not be presented in the generated sentence X itd with the training of the following loss:
pITD(x itd i = x in i ∣∣hi) = sigmoid(wTi hi), (12)
LITD = E ( − ∑
xitdi =x in i
log pITD ( xitdi = x in i ∣∣hi )
− ∑
xitdi ̸=xini
log ( 1− pITD ( xitdi = x in i ∣∣hi )) ) .
(13)
On the one hand, the ratio of real and inserted words is fixed, solving the label-imbalance to some extent. On the other hand, training on void locations tones up the generation capability of models.
The overall structure of the proposed selfsupervision courses is presented in Figure 1. All courses are jointly conducted within the same data and computing steps.

heading: 4.2 Self-correction Course
text: According to the above self-supervision courses, a competition mechanism between G and D seems to shape up. Facing the same piece of data, G tries to reform the sequence in many ways, while D yearns to figure out all the jugglery caused previously. However, the shared embedding layer of these two encoders becomes the only bridge of
communication, which is apparently insufficient. To strengthen the link between the two components, and to provide more supervisory information on pre-training, we conduct an intimate dissection of the relationship between G and D.
Take the procedure of RTD for example. For each token xrtdi in the corrupted sentence X
rtd, whereafter fed into D, we identify and document its label by comparing with the token xi at the corresponding position in X . After the discrimination process of D, this token is binary classified as original or replaced. As shown in Table 1, there exist four situations of distinguish results for xi. pos1: where G predicts the correct answer on the [MASK] position and D successfully makes a good judgment, no additional operation needs to be conducted for this kind of token. pos2: where G fills an alternative to replace the original token and D inaccurately views it as original, it means G produces an appropriate expression to form a harmonious context as mentioned in Section 3, which makes it difficult for D to distinguish. pos3: where D makes a clumsy mistake of incorrectly annotating an original token as replaced. pos4: where G fills in an impertinent token at the [MASK] position and D easily figures it out.
To sum it up, on the one hand, G needs to regenerate tokens at pos4, since the initial alternatives are inappropriate and unchallenging for D. As shown in Figure 2, too much [MASK] are placed at important locations rich in information, leading to the erratic generation “thanked”. Considering that other [MASK] in the same sequence may interfere with the generation of tokens at pos4, we restore other [MASK] to the original tokens for convenience of the re-generation process. On the other hand, D is expected to re-discriminate tokens at pos2 and pos3. When there exist tokens at pos4 in a sequence, these inappropriate tokens may seriously disturb decisions of D on other tokens,
leading to the consequence of pos2 and pos3. Take the sentence in Figure 2 for example, serious distraction “thanked” makes D falsely judges “meal” as replaced. So we replace tokens at pos4 in X rtd to original tokens to alleviate this kind of interference, and conduct the re-discrimination training on D at pos2 and pos3.
By sorting out and analyzing errors, an “correction notebook” for G and D is built, guiding the re-generation and re-discrimination training. Note that it’s not just redoing the problem, however, we redesign the context for each kind of issue. Thus, Lre-MLM and Lre-RTD is designed as the learning objective for self-correction course of RTD. Likewise, Lre-SLM and Lre-STD presents the training loss selfcorrection course of STD.3 Cause there are no original tokens at the inserted [MASK] positions, no revision is conducted for ITD. Two proposed selfcorrection courses bridge the chasm between G and D through introspection and melioration, and provide a sample-efficient secondary-supervision for the same piece of data.
Finally, G and D are co-trained with three selfsupervision courses as well as two self-correction courses. The proposed MCL dissects one same sequence profoundly and comprehensively, without incurring any additional inference or memory costs.

heading: 5 Experiments

heading: 5.1 Setup
text: Pre-training Settings We implement the experiments on two settings: base and tiny. Base is the standard training configuration of BERTBase (Devlin et al., 2019). The model is pre-trained on English Wikipedia and BookCorpus (Zhu et al., 2015), containing 16 GB of text with 256 million samples. We set the maximum length of the input sequence
3The equation is not listed since the form is consistent with the previous text.
to 512, and the learning rates are 5e-4. Training lasts 125K steps with a 2048 batch size. We use the same corpus as with CoCo-LM (Meng et al., 2021) and 64K cased SentencePiece vocabulary (Kudo and Richardson, 2018). The details of the hyperparameter of pre-training is listed in Appendix A. Tiny conducts the ablation experiments on the same corpora with the same configuration as the base setting, except that the batch size is 512.
Model Architecture The layout of our model architecture maintains the same as (Meng et al., 2021) both on base and tiny settings. D consists of 12-layer Transformer, 768 hidden size, plus T5 relative position encoding (Raffel et al., 2020). G is a shallow 4-layer Transformer with the same hidden size and position encoding. After pre-training, we discard G and use D in the same way as BERT, with a classification layer for downstream tasks.
Downstream Tasks To verify the effectiveness of the proposed methods, we conduct evaluation experiments on various downstream tasks. We evaluate on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) and Stanford Question Answering 2.0 (SQuAD 2.0) dataset (Rajpurkar et al., 2018). As for the evaluation metrics of GLUE tasks, we adopt Spear-
man correlation for STS, Matthews correlation for CoLA, and accuracy for the other. For SQuAD 2.0, in which some questions are unanswerable by the passage, the standard evaluation metrics of ExactMatch (EM) and F1 scores are adopted. More details of the GLUE and SQuAD 2.0 benchmark are listed in Appendix B.
Baselines Various pre-trained models are listed and compared in the base setting. All numbers are from reported results in recent research. When multiple papers report different scores for the same method, we use the highest of them for comparison.
Implementation Details Our implementation builds upon the open-source implementation from fairseq (Ott et al., 2019). With 128 A100 (40 GB Memory), one pre-training run takes about 24 hours in base setting. The fine-tuning costs are the same with BERT plus relative positive encodings. More details of fine-tuning are listed in Appendix C.

heading: 5.2 Evaluation Results
text: We first pre-trained our model with the proposed MCL method, and then fine-tuned it with training sets of 8 single tasks in the GLUE benchmark. We conducted a hyperparameter search for all downstream tasks, and report the aver-
age scores among 5 random seeds. Results are elaborated in the top half of Table 2. The proposed MCL evidently enhances ELECTRA and achieves at least 1.1% absolute overall improvements against state-of-art pre-trained language models on the GLUE benchmark under the base setting. For the most widely reported task MNLI, our model achieves 88.5/88.6 points on the matched/mismatched (m/mm) set, which obtains 1.6/1.8 absolute improvements against ELECTRA. Take a broader look at all GLUE single tasks, MCL overshadows all previous models, except RTE tasks, where CoCo-LM takes a narrow lead.
We also evaluated the proposed MCL on the SQuAD 2.0 datasets, which is an important reading comprehension dataset that requires the machine to extract the answer span given a document along with a question. The results of Exact-Match (EM) and F1 score (F1) are displayed in the top half of Table 3. Consistently, our model significantly improves the ELECTRA baseline and achieves a banner score compared with other same-size models. Specifically, under the base setting, the proposed MCL improves the absolute performance over ELECTRA by 3.2 points (EM) and 3.3 points (F1). Also, our model outperforms all other previous models with an overt margin.
The compelling results demonstrate the effectiveness of the proposed MCL. With the equal amount of training corpus, plus slight comput-
ing cost of forward propagation, MCL tremendously advanced ELECTRA baseline, showing its property of sample-efficiency. In other words, multi-perspective course learning gives the model a deeper and more comprehensive insight into the underlying meaning of corpora, which provides more valuable information for the pre-training process.

heading: 5.3 Ablation Study
text: In order to dive into the role of each component in the proposed MCL, an ablation study is conducted under the tiny setting. Both the GLUE and SQuAD 2.0 datasets are utilized for evaluation, and the ablation results are listed in the bottom half of Table 2 and Table 3. Bolstered by several curve graphs regarding with loss and accuracy during pre-training, every course is discussed below. 4
RTD The most basic component, also represents the ELCETRA itself. Its performance would be employed as the baseline to compare with other additions. Not only the scores, but also the curves would be taken for important reference.
STD This course is tasked to help the model to obtain better structure perception capability through a more harmonious contextual understanding. STD improves ELECTRA on all tasks in GLUE and SQuAD 2.0 datasets. It is worth noting that the scores on CoLA task surprisingly stand out amongst the crowd. The Corpus of Linguistic Acceptability (CoLA) is used to predict whether an English sentence is linguistically acceptable or not. Apparently, pre-training on word rearrangement indeed lifts the global intellection of models, which makes it focus more on structure and logic rather than word prediction. Even the best CoLA result of 71.25 comes from the re-STD course, which further embodies the effectiveness of STD.
ITD This course is tasked to alleviate labelimbalance. As shown in Figure 5, replace rate reflects the prediction accuracy of G. Accompanied by MLM and SLM, G predicts more correct words on the [MASK] positions, causing the “replaced” labels to become scarce for the training of D. By adding inserted [MASK], the replace rate has a fixed lower limit corresponding to the inserted proportion, leading to a balanced distribution of labels. Besides, ITD shows great improvements over
4Due to the constraints of space, curve graphs of pretraining and concrete instructions are replaced at Appendix D
ELECTRA, especially on SST-2 datasets. The Stanford Sentiment Treebank (SST-2) provides a dataset for sentiment classification that needs to determine whether the sentiment of a sentence extracted from movie reviews is positive or negative. Predicting for the illusory [MASK] makes the model focus more on content comprehension, which may helpful for sentiment classification.
Self-correction Course Revision always acts as a difficult assignment, because of the challenge to reverse stereotypes. As shown in Figure 5, losses of G and D during self-correction training generally exceed that during self-supervision training, demonstrating the difficulties. However, the replace accuracy of re-RTD course goes higher than the baseline, certifying the effectiveness. Despite that self-correction training outperforms other components on all downstream tasks, the phenomena of “tug-of-war” dynamics is worth exploring. Scores listed in the last three rows of Table 2 almost touch each other, and optimal results of every single task do not always appear under the same model. It means multi-perspective courses may interfere with each other in attempts to pull parameters in different directions, which seems even more apparent under the self-correction course where secondarysamples are well designed for bootstrapping. To alleviate this situation and further improve the effectiveness of training, we found a feasible solution elaborated in Section 5.5.

heading: 5.4 Sample Efficiency Comparison
text: To demonstrate the proposed MCL is sampleefficient, we conduct a comparative trial between MCL and ELECTRA. As shown in Figure 3, the prevalent task MNLI is chosen for evaluation. For every 25K steps of pre-training, we reserved the model and fine-tuned it with the same configuration mentioned in Section 5.1. Obviously, MCL
preponderates over ELECTRA baseline on every training node, which obtains 87.8 points at 25K steps, demonstrating its enormous learning efficiency even on small pieces of corpora.

heading: 5.5 Course Soups Trial
text: Inspired by model soups (Wortsman et al., 2022), which averages many models in a hyperparameter sweep during fine-tuning, we find similarities and bring this idea to MCL in a task sweep during pretraining. Different courses lead the model to lie in a different low error basin, and co-training multiple courses may create the “tug-of-war” dynamics. To solve the training conflicts, and to further improve the learning efficiency of models in the later pretraining stage, we conduct a “course soups” trial.
For ingredients in soups, we arrange all combinations of 4 losses in self-correction courses, training them into 14 single models while retaining the structure of self-supervision courses. Then all ingredients are merged through uniform and weighted integration. Results lies in Figure 4. Optimal results obtained by weight soups, which improves the average GLUE score by 0.19 absolute points against our best model MCL. The results show that the course soups suggests a effective way to guide the later training of the model by separating multiple objectives and combining them at last. More details scores are listed in Appendix E.

heading: 6 Conclusion
text: This paper proposes the multi-perspective course learning method, containing three self-supervision courses to improve learning efficiency and balance label distributions, as well as two self-correction courses to create a “correction notebook” for revision training. Besides, the course soups method is
designed to figure out a novel approach for efficient pre-training. Experiments show that our method significantly improves ELECTRA’s performance and overshadows multiple advanced models under same settings, verifying the effectiveness of MCL.
Limitations
Although the proposed method has shown great performance to alleviate the issues of biased learning and deficient interaction, which are common problems among ELECTRA-style pre-training models, we should realize that the proposed method still can be further improved. For example, the inherent flaw of RTD mentioned in Section 3 could only be relieved rather than solved. More about mission design regarding with this issue is worth studying. Besides, although the results show great performance, more efforts are required to explore the hidden impact of each course, which will help the application of the proposed model in the future.

heading: A Hyperparameters for Pre-training
text: As shown in Table 4, we present the hyperparameters used for pre-training MCL on the base setting. We follow the optimization hyperparameters of CoCo-LM (Meng et al., 2021) for comparisons. Note that all losses conducted on D are multiplied by λ (set as 50), which is a hyperparameter used to balance the training pace of G and D.

heading: B Details of Downstream Tasks
text: GLUE contains a wide range of tasks covering textual entailment: RTE (Giampiccolo et al., 2007) and MNLI (Williams et al., 2018), question-answer entailment: QNLI (Rajpurkar et al., 2016), paraphrase: MRPC (Dolan and Brockett, 2005), question paraphrase: QQP (Iyer et al., 2017), textual similarity: STS (Cer et al., 2017), sentiment: SST (Socher et al., 2013), and linguistic acceptability CoLA (Warstadt et al., 2019).
Natural Language Inference involves reading a pair of sentences and judging the relationship between their meanings, such as entailment, neutral and contradiction. We evaluate on three diverse datasets, including Multi-Genre Natural Language Inference (MNLI), Question Natural Language Inference (QNLI) and Recognizing Textual Entailment (RTE).
Semantic similarity tasks aim to predict whether two sentences are semantically equivalent or not. The challenge lies in recognizing rephrasing of concepts, understanding negation, and handling syn-
tactic ambiguity. Three datasets are used, including Microsoft Paraphrase corpus (MRPC), Quora Question Pairs (QQP) dataset and Semantic Textual Similarity benchmark (STS-B).
Classification The Corpus of Linguistic Acceptability (CoLA) is used to predict whether an English sentence is linguistically acceptable or not. The Stanford Sentiment Treebank (SST-2) provides a dataset for sentiment classification that needs to determine whether the sentiment of a sentence extracted from movie reviews is positive or negative.
As a widely used MRC benchmark dataset, SQuAD 2.0 is a reading comprehension dataset that requires the machine to extract the answer span given a document along with a question. We select the v2.0 version to keep the focus on the performance of pure span extraction performance. Two official metrics are used to evaluate the model performance: Exact Match (EM) and a softer metric F1 score, which measures the average overlap between the prediction and ground truth answer at the token level.
The summary of the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) is shown in Table 5.

heading: C Hyperparameters for Fine-tuning
text: Table 6 presents the hyperparameters used for finetuning over SQuAD v2.0 (Rajpurkar et al., 2018), and the GLUE benchmark (Wang et al., 2019) following CoCo-LM for fair comparison. On the development sets, the hyperparameters are searched based on the average performance of five runs.

heading: D Curves for Ablation Study
text: As shown in Figure 5, three metrics are selected to evaluate the quality of pre-training.
Replace Rate This metric represents the ratio of replaced tokens in the corrupted sentence X rtd. The less of this ratio, the better of G’s pre-training, and the more uneven of the label distributions for D. From the first row in the diagram, we can see that the lower bound of the replace rate with ITD apparently exceeds that with RTD, demonstrating that ITD indeed alleviates the issue of label-imbalance.
Loss and Replace Accuracy Training loss reflects the pre-training assessment. One of the selfcorrection courses, re-RTD, holds a higher loss against that with RTD, showing the difficulty of the revision training. Replace accuracy denotes the prediction accuracy on replaced tokens. As shown in the third line of Figure 5, re-RTD achieves better replace accuracy by a significant margin compared with RTD, demonstrating the effectiveness of the self-correction course.

heading: E Detailed Scores of Course Soups
text: Table 7 lists all scores on the GLUE benchmark of the course soups trial. It seems that optimal results on single tasks are randomly distributed in the ingredients of soups. Through uniform and weighted integration, the best model emerges with the uppermost average GLUE scores.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Left blank.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Besides, although the results show great performance, more efforts are required to explore the hidden impact of each course, which will help the application of the proposed model in the future