Gradient Ascent Post-training Enhances Language Model Generalization
Year: 2023

Abstract:
In this work, we empirically show that updating pretrained LMs (350M, 1.3B, 2.7B) with just a few steps of Gradient Ascent Post-training (GAP) on random, unlabeled text corpora enhances its zero-shot generalization capabilities across diverse NLP tasks. Specifically, we show that GAP can allow LMs to become comparable to 2-3x times larger LMs across 12 different NLP tasks. We also show that applying GAP on out-of-distribution corpora leads to the most reliable performance improvements. Our findings indicate that GAP can be a promising method for improving the generalization capability of LMs without any task-specific finetuning 1.

heading: 1 Introduction
text: Recently, Language Models (LMs) pretrained on a vast amount of text corpora have shown to be capable of performing diverse downstream NLP tasks in a zero-shot manner (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2022; Zhang et al., 2022) or through in-context learning (Brown et al., 2020; Min et al., 2022) without any gradient updates. This paradigm has been preferred over task-specific fine-tuning (Devlin et al., 2019), which requires considerable amount of labeled data for the given target task.
Motivated by the positive effect of gradient ascent during fine-tuning (Foret et al., 2021), in this work, we explore whether adapting pretrained LMs with Gradient Ascent Post-training (GAP) on random, unlabeled text corpora can bring any benefits in terms of enhancing its generalization capabilities of performing diverse downstream NLP tasks in a zero-shot or few-shot manner without the need for task-specific training data.
Specifically, we apply just a few steps of gradient ascent to OPT LMs (Zhang et al., 2022) using
∗Equal Contribution 1Code and full results for individual GAP runs are avail-
able at https://github.com/kaist-lklab/GAP
randomly sampled text sequences from 3 different corpora from the Pile (Gao et al., 2021) with varying degree of familiarity between the LM and the corpus. Experimental results show that this simple approach achieves performance gains across 12 downstream NLP tasks: 4 dialogue tasks and 8 classification tasks. We observe that applying GAP with out-of-distribution data, specifically code data that OPT was not explicitly trained on, results in the most reliable performance gain.
Our main contributions can be summarized into two folds:
• We empirically show that GAP is a promising generalization enhancement technique as it is (1) effective, as evidenced by multiple benchmark results; (2) simple & efficient, requiring maximum 15 steps of parameter update; (3)
851
versatile, as it can be applied easily to any pretrained LMs and does not necessitate taskspecific fine-tuning.
• We show analysis of what makes GAP work by splitting the corpora into three groups according to the LMs’ degree of familiarity with the data. We observe that performing GAP with the most unfamiliar (out-of-distribution) data results in the most reliable performance gain.

heading: 2 Related Works
text: Task-Specific Gradient Ascent Deep neural network models exhibiting poor generalization due to converging at sharp local minima is a well-known phenomenon in literature (Keskar et al., 2017; Izmailov et al., 2018; Cha et al., 2021; Chen et al., 2022). To address this issue, Foret et al. (2021) introduce Sharpness-Aware Minimization (SAM), an algorithm that performs both gradient ascent as well as gradient descent during task-specific fine-tuning to avoid sharp local minima, improving performance. The effectiveness of SAM has motivated several studies to apply them to LMs and report meaningful improvements in performance.
Bahri et al. (2022) have shown that applying SAM when fine-tuning various scales of T5 LMs (Raffel et al., 2020) on multiple downstream tasks results in a substantial performance gains. Similarly, Kaddour et al. (2022) also explore SAM across computer vision, natural language processing, and graph representation learning tasks, further bolstering its efficiency.
While SAM was proposed as a robust fine-tuning methodology that targets convergence on supervised dataset, we instead explore the benefits gradient ascent can bring without task-specific labeled data for generic LMs.
Task-Agnostic Gradient Ascent In a recent study, Jang et al. (2022) investigate the use of gradient ascent for addressing privacy risks in LMs. The main objective of the work is utilizing gradient ascent to unlearn specific token sequences; surprisingly, they report unexpected performance gains in some cases. Our work can be seen as a direct extension of this phenomenon where our main objective is to enhance the generalization capabilities instead of forgetting specific data to ensure privacy.

heading: 3 Gradient Ascent Post-training (GAP)
text: In this section, we give a formal definition of GAP. Specifically, given an LM with parameters w and a sequence of tokens x = (x1, ..., xN ), GAP is defined as:
wt+1 = wt + α∇fwt(x) (1)
fwt(x) = − N∑
n=1
log(pwt(xn|x<n)) (2)
where t represents the gradient ascent iteration, α denotes the learning rate, x<n indicates the token sequence (x1, ..., xn−1) and pwt(xn|x<n) represents the likelihood of predicting the next token, xn, given the previous token sequence as an input to an LM with parameter wt.
Markedly, GAP solely utilizes gradient ascent and does not actively facilitate convergence, as it updates the model parameters to maximize (1) the language modeling loss function (2). We propose GAP as an unsupervised methodology that can bring significant performance gains even without curated fine-tuning data.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Baseline Models and Evaluation Datasets We use OPT (350M, 1.3B, 2.7B, 6.7B) LMs (Zhang et al., 2022) as the baseline LMs. We observe the effect GAP has on their generalization capabilities which is measured via evaluation on 12 different downstream NLP tasks; we use Wizard of Wikipedia (Dinan et al., 2019), Empathetic Dialogues (Rashkin et al., 2019), Blended Skill Talk (Smith et al., 2020) and WizInt (Komeili et al., 2022) to evaluate generative capabilities, Hellaswag (Zellers et al., 2019) to assess linguistic reasoning abilities, Winogrande (Sakaguchi et al., 2021) and COPA (Brassard et al., 2022) to measure commonsense reasoning abilities, and ARC-Easy (Clark et al., 2018), ARCChallenge (Clark et al., 2018), PIQA (Bisk et al., 2020), MathQA (Amini et al., 2019) and PubmedQA (Jin et al., 2019) to measure the scientific reasoning abilities. The exact prompts used for each task are provided in Appendix A.
Random Unlabeled Data We apply GAP on text snippets from three different corpora, which all originate from the Pile (Gao et al., 2021) training set: (1) Training Data Extraction Challenge
(TDEC)2, (2) Common Crawl (CC) and (3) Github (Git.). We choose these corpora in order to observe the effect of the LMs’ degree of familiarity with the data. Training Data Extraction Challenge includes examples from the Pile that are identified to be easy-to-extract from GPT-Neo LMs (Black et al., 2022), mainly due to high levels of duplication. We assume these examples are also relatively easier-toextract from OPT LMs as they were also pretrained on subset of the Pile, indicating the highest level of familiarity / memorization. We consider OPT LMs to be familiar (in-domain) to Common Crawl, as it was included in their pretraining corpora. As OPT LMs were not explicitly trained on the Github corpora we consider OPT to be unfamiliar (out-ofdistribution) with Github. Examples of the random unlabeled data are provided in Appendix D.
Configurations For each of the 3 LM sizes [350M, 1.3B, 2.7B], we sample a total of 300 text samples (each 200 token lengths long) for applying GAP, with 100 samples taken from each of the three corpora. For each run, a single text sample is used, ultimately resulting in 300 runs of GAP per LM size. Therefore, a single epoch of a GAP run comprises of a single gradient ascent step with batch size set to 1. The number of maximum epochs is set to 15 and we report the validation score from the best-performing epoch, as preliminary experiments showed gradient ascent past 15 steps mostly resulted in performance degradation. Due to computational constraints we sample the validation data to a maximum of 320 samples per dataset for all of the 12 evaluation datasets. For further exploration of GAP as a methodology, we use the checkpoints with the best validation scores and evaluate the LMs on the test datasets for the 4 dialogue tasks. We do not separately report the test evaluation results for classification datasets since most of them require direct submission to the task website. For a single run, we use one Nvidia 40GB A100 GPU. Further details regarding the experimental configurations (e.g. optimizer, learning rate, etc.) are provided in Appendix B.

heading: 4.2 Dialogue Tasks
text: Main Results As shown in Figure 1 in Section 1, GAP substantially enhances the average validation performance on the 4 dialogue tasks, with median F1-score of 1.3B LMs outperforming the
2https://github.com/google-research/lm-extractionbenchmark

heading: 6.7B 14.5 51.1 88.3 11.9
text: 2.7B LM baseline, and some 1.3B LMs even able to match the performance of the 6.7B LM baseline 3. We report the average test F1 score as well as MAUVE (Pillutla et al., 2021), diversity (Su et al., 2022), and generation length of our best validation checkpoints for each model size (excluding outliers) in comparison to the baseline LMs in Table 1 4.
Results show a substantial improvement in all of the metrics, F1 Score, MAUVE, and generation length, with our 1.3B and 2.7B LM checkpoints even outperforming the larger LM baselines. This result is significant considering that no task-specific dataset is used. Examples of text generation for the dialogue tasks are provided in Appendix E.
Human Evaluation We also evaluate and compare the qualitative quality of generated responses of the baseline LMs and the LMs adapted with GAP
3Detailed numerical data for the median values is available in C.
4Explanation of how MAUVE and diversity is measured is provided in Appendix B.
350M 1.3B 2.7B
side-by-side. For this, we sample 100 contexts from the WizInt (Komeili et al., 2022) dataset and generate the corresponding responses with the 2.7B LM baseline and 2.7B LM + GAP denoted as Ours. Then, we compare the generated response pairs from the LMs from the perspective of three metrics: coherence, fluency, and informativeness (Su et al., 2022). We ask human evaluators to select the better response from each pair with respect to each metrics 5. We find our GAP-enhanced LM shows significant strengths in all the metrics compared to its baseline (Table 2). Moreover, our LM shows comparable performance to human upper bounds (gold response) except for informativeness.

heading: 4.3 Classification Tasks
text: The average validation performances of the 8 classification tasks when performing GAP on the OPT LMs are shown in Figure 2. While GAP fails to provide consistent improvements for 350M LMs and 2.7B LMs, mostly resulting in a degradation of performance as shown by the median performance underperforming the baselines, the LMs show considerable performance gains in some cases for the larger LMs. This result suggests that although GAP does not show steady improvement of generalization for the classification tasks unlike the dialogue
5Further study details are in Appendix F.
tasks, it does show some potential for improvement considering that some runs did result in substantial improvements. We leave choosing the right text samples to perform GAP on for a consistent performance enhancement on classification tasks for future work.

heading: 4.4 Analysis of GAP
text: Figure 3 shows the average performance of the 300 GAP runs for the 350M LMs (zoomed-in version of Figure 1). To observe the effect of LMs’ familiarity to the unlabeled data, we plot the dots with different symbols with respect to the corpus. Interestingly, samples from the unfamiliar corpus (Github) results in significant improvements, mostly achieving higher scores than the median score. Consistent findings are also evident in Table 3, with Github achieving the highest median F1 scores across all model sizes. This suggests that future applications of GAP can be applied more efficiently by mostly
using unfamiliar (out-of-distribution) text. Additional figures for other LM sizes are available in Appendix C.

heading: 5 Conclusion
text: In this work, we introduce GAP, a novel method of improving the generalization capability of LMs without any task-specifc data by sampling random text and performing gradient ascent for a few steps. We show that our approach is (1) simple to use, (2) effective in making more robust LMs, and (3) has much room for improvements for future work when scaling the number of GAP runs (e.g. >300) and choosing specific text samples (e.g. out-ofdistribution text) to perform GAP on. Thus, we urge the community to consider GAP when prompting off-the-shelf pretrained LMs for performing diverse downstream NLP tasks.
Limitations
While we show that applying GAP can result in a significant improvement in the generalization capability of LMs, especially for dialogue tasks, we are only able to show 300 GAP runs for each LM size in this work. We leave scaling the number of GAP runs, as well as selecting specific text samples to perform GAP on for future work. Furthermore, a separate validation set of the tasks at interest are needed in order to choose the best checkpoint when performing GAP. Future work may look for other task-agonostic cues such as language modeling loss to determine the best checkpoint to use for inference.

heading: Acknowledgements
text: This work was partly supported by KAIST-NAVER Hypercreative AI Center (80%) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00113, Developing a Sustainable Collaborative Multi-modal Lifelong Learning Framework, 20%).

heading: A Task Prompts
text: Table 4 shows the prompts we use for each of the 12 benchmark dataset to enable zero-shot/few-shot learning. For dialogue tasks (Wizard of Wikipedia, Blended Skill Talks, Empathetic Dialogues, WizInt), we use the prompts used by Zhang et al. (2022).

heading: B Details of Experimental Configurations
text: In this section, we give further details of our main experimental setting of performing GAP. We use Adam optimizer (Kingma and Ba, 2014) with a constant learning rate of 5e-5 with no weight decay and no dropout.
For the dialogue tasks, we adopt the settings of Zhang et al. (2022) and prompt the LM with alternating "User 1:" and "User 2:" lines of dialogue (examples shown in Appendix A). To generate tokens, we employ greedy decoding method and set a maximum generation length of 32 tokens. For the classification tasks, we use a verbalizer method by selecting the output option with higher log-likelihood following Brown et al. (2020); Sanh et al. (2021). We use unigram F1 score as our main metric for the dialogue generation tasks and accuracy for the classification tasks.
For the diverse metrics used for evaluation on the test sets of the 4 dialogue tasks, MAUVE (Pillutla et al., 2021) compares the text representation of the LM generated-response to human-written text, higher values indicate greater similarity to human-written text. Diversity metric (Su et al., 2022) measures token-level repetition, with higher values indicating greater diversity and less repetition in the generated text.

heading: C Full Results
text: Tables 5 and 6 show the median validation score of all 300 GAP runs. For classification tasks, the
median values do not show significant improvements. However for dialogue tasks, GAP shows considerable improvements across all tasks.
Tables 7, 8, 9 and 10 show the individual test performance for each dialogue dataset. The four dialogue datasets are: Blended Skill Talks (BST), Empathetic Dialogues (ED), Wizard of Wikipedia (WoW) and WizInt. Our models demonstrate superior performance compared to their same sized baselines on every metrics in all four task.
Figures 4 and 5 represent the familiarity analysis results for 1.3B and 2.7B sized models, respectively. For both 1.3B and 2.7B models, data sampled from the out-of-domain corpora (Github) results in reliable performance gains. For the bigger sized models, in-domain corpora (CC) also results in competitive performance gains, suggesting larger sized morels are more robust to GAP data selection.

heading: D Examples of Random Data
text: Table 11 shows examples of the random data we use to apply GAP to OPT LMs. Specifically, they are the best performing data for each model size.

heading: E Examples of Dialogue Generation Outputs
text: Table 12 shows some examples of text generated by baseline models and our models trained with GAP. Notice that our models generate diverse and interesting text while also maintaining coherence to the given dialogue history.

heading: F Details of Human Evaluation
text: We conduct the human evaluation on Amazon Mechanical Turk (AMT). An example of the interface shown to the workers is shown in Figure 6. Specifically, we recruit three different annotators for each comparison pair with a compensation of 1$ per instance. We include brief instructions on the evaluation including descriptions of three metrics. Then, we ask the workers to compare each generated (or ground-truth for human baseline) response pair with the given dialogue context. We evaluate 200 samples in total, including 100 for the OPT baseline and 100 for the human upper bounds. The Fleiss kappa among the workers is calculated as 0.36, which indicates moderate-level agreements. We also test the significance between the comparing systems via a bootstrap test with 100,000 samplings.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6
3 A2. Did you discuss any potential risks of your work? Section 6
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.

heading: B1. Did you cite the creators of artifacts you used?
text: No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 4 and 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4 and 6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4 and 6
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 and 6
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 and 6
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4 and 6
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Section 6
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4 and 6
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section 4 and 6
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We weren’t able to obtain the information because Amazon Mechanical Turk does not provide the information.

heading: Future Work
text: We show that our approach is (1) simple to use, (2) effective in making more robust LMs, and (3) has much room for improvements for future work when scaling the number of GAP runs (e. g. >300) and choosing specific text samples (e. g. out-ofdistribution text) to perform GAP on. Thus, we urge the community to consider GAP when prompting off-the-shelf pretrained LMs for performing diverse downstream NLP tasks. Limitations
While we show that applying GAP can result in a significant improvement in the generalization capability of LMs, especially for dialogue tasks, we are only able to show 300 GAP runs for each LM size in this work. We leave scaling the number of GAP runs, as well as selecting specific text samples to perform GAP on for future work. Furthermore, a separate validation set of the tasks at interest are needed in order to choose the best checkpoint when performing GAP. Future work may look for other task-agonostic cues such as language modeling loss to determine the best checkpoint to use for inference