Meta-Tuning LLMs to Leverage Lexical Knowledge for Generalizable Language Style Understanding (2024)

Abstract: Language style is often used by writers to convey their intentions, identities, and mastery of language. In this paper, we show that current large language models struggle to capture some language styles without fine-tuning. To address this challenge, we investigate whether LLMs can be meta-trained based on representative lexicons to recognize new styles they have not been fine-tuned on. Experiments on 13 established style classification tasks, as well as 63 novel tasks generated using LLMs, demonstrate that meta-training with style lexicons consistently improves zero-shot transfer across styles. We release the code and data at https: //github.com/octaviaguo/Style-LLM.

heading: 1 Introduction
text: The style of a text refers to unique ways authors select words and grammar to express their message (Hovy, 1987), providing insights into social interactions and implicit communication. The open-ended and ever-evolving nature of style (Xu, 2017; Kang and Hovy, 2021) motivates the need for zero-shot classification, as it is costly to annotate data for every possible style in every language.
Recent large language models (LLMs) and their instruction-tuned variants have achieved notable success in zero-shot learning for diverse tasks using prompting strategies (Brown et al., 2020). Yet, their efficacy in style classification has not been thoroughly investigated. As we will show in this paper (§3.2), style classification remains a challenge for standard LLM prompting. On the other hand, before the paradigm in NLP shifted to pretrained language models, lexicons of words that are stylistically expressive were commonly used as important lexical knowledge (Verma and Srinivasan, 2019) in rule-based (Wilson et al., 2005; Taboada et al., 2011), feature-based (Mohammad et al., 2013; Eisenstein, 2017), and deep learning models (Teng et al., 2016; Maddela and Xu,
2018) for style identification. Many lexicons have been developed for varied styles, such as politeness (Danescu-Niculescu-Mizil et al., 2013), happiness (Dodds et al., 2015), emotions (Mohammad and Turney, 2010; Tausczik and Pennebaker, 2010), etc. This leads to a natural question: can we leverage lexicons during instruction fine-tuning of LLMs to improve their understanding of language style?
In this paper, we examine the effectiveness of fine-tuning LLMs to interpret lexicons that are provided as inputs to elicit latent knowledge (Kang et al., 2023) of language styles that were acquired during pre-training. We first compile a benchmark of 13 diverse writing styles with both annotated test sets and style-representative lexicons. Using this benchmark, we show that meta-tuning with lexicons enables different pre-trained LLMs to generalize better to new styles that have no labeled data. For example, meta-tuning LLaMA-2-7B (Touvron et al., 2023) on seven styles can improve the average F1 score on a separate set of six held-out styles by 12%, and by 8% over a general instruction-tuned model, LLaMA-2-Chat.
To further verify the capability of LLMs to generalize to novel styles using lexicons as the only source of supervision, we generated a diverse set of 63 unique writing styles with examples (§4) using an approach inspired by self-instruction (Wang et al., 2023). We demonstrate that using a small lexicon of as few as five words can effectively improve generalization to new styles. We found it helpful to replace class names with random identifiers when meta-training models with lexicons, which prevents models from ignoring the lexicons and simply memorizing source styles’ class names. In addition, we show that when combined with meta in-context learning (Min et al., 2022a; Chung et al., 2022), incorporating lexicons can significantly reduce variance.
In summary, our contributions are: (1) we introduce lexicon-based instructions (§2.1), a simple
13708

heading: Helpfulness
text: yet effective method for zero-shot style classification leveraging lexical knowledge in LLMs; (2) we show class randomization (§2.1) can improve generalization capability of lexicon-instructed models significantly (§3.2); (3) we provide a benchmark for zero-shot style classification, featuring 13 established tasks (§2.2) and a synthetic dataset of 63 new tasks (§4), complete with representative lexicons.

heading: 2 Meta-Tuning for Style Generalization
text: We investigate the capabilities of LLMs to interpret language styles using lexical knowledge, and identify text that is representative of the associated styles. We compare lexicon-based instructions with other methods in the zero-shot setting, and further explore a few-shot setting. To study the effectiveness of meta-tuning with lexicons, in generalizing to various writing styles, we first consider a set of thirteen styles, where high-quality annotated data is available. Later, in §4, we further demonstrate the ability of lexicon-instructed models to generalize using 63 novel LLM-generated styles.

heading: 2.1 Problem Definition and Approach
text: Given an input text and style with pre-defined classes C = {ck}|C|k=1, we present the language model with lexicon-based instructions, by instantiating lexicons (i.e., a list of words or phrases that are representative of each class ck) in a pre-defined instruction template (see templates in Table 20). A language model is expected to predict one of the classes ĉ ∈ C, given the lexicon-based instruction. These style-lexicons, are the only source of targetstyle supervision provided to the LLM, enabling it to make stylistic predictions using parametric
knowledge that was acquired during pre-training (Raffel et al., 2020; Brown et al., 2020), and elicited using lexicon-based instructions.
Meta-Tuning on Source Styles. Meta-tuning is a simple, but effective approach that directly optimizes the zero-shot learning objective through fine-tuning on a collection of datasets (Zhong et al., 2021). 1 In order to guide models to draw upon latent lexical knowledge for zero-shot style classification, we meta-tune LLMs to learn to understand style-lexicon relations. During preliminary experiments, we found that it is important to make use of class randomization (§3.2.1) during meta-tuning, e.g. using randomly selected identifiers to replace more meaningful style labels (e.g., “humorous”), in order to prevent models from simply memorizing the (source) styles used for fine-tuning. Without randomizing labels, memorization prevents the model from effectively generalizing to interpret lexicons for new styles. In §3.2, we conduct an analysis into the impact of randomization, and compare different types of identifier randomization.

heading: Zero-Shot Evaluation on Unseen Target Styles.
text: To make predictions, we provide the model with the target-style lexicon and use rank classification (Sanh et al., 2021), in which we compute the likelihood of each style label, and then pick the one with highest likelihood as the final prediction.

heading: 2.2 A Benchmark for Style Generalization
text: Style Datasets. We include thirteen language styles that have sentence-level annotated datasets
1In this work, we use the term meta-tuning to mean finetuning on diverse datasets and tasks. Instruction tuning further narrows the scope to datasets that include instructions.
:::::
45-54,
::::
55-74,
::::::::::::::
75YearsOrOlder) in Age dataset into two new classes (i.e., youthful,
::::::
mature).
available, covering a wide range of domains, as summarized in Table 1. These come from a variety of sources, including the XSLUE benchmark (Kang and Hovy, 2021), Subjectivity (Pang and Lee, 2004), Shakespeare (Xu et al., 2012) and Readability (Arase et al., 2022) - more details are available in Appendix A. In the cross-style zero-shot setting, a model is fine-tuned on a set of training styles, then evaluated on a separate set of held-out styles with no overlap. For each training style, its training set is used for fine-tuning, and the validation set is used for model selection (Chen and Ritter, 2021). We ensure evaluation style datasets do not share any examples with training styles. Given space limitations, we present results for one split, which includes Sentiment, Formality, Politeness, Hate/Offense, Readability, Politics, and Subjectivity in the training split, while the remaining six styles are included in evaluation split. Experiments on more style splits are shown in Appendix E.4.
Lexicon Collection. We use stylistic lexicons that have been created by other NLP researchers where possible (listed as “NLP” in Table 1). These lexicons were either manually annotated (Maddela and Xu, 2018) or automatically induced using corpus-based approaches (Danescu-NiculescuMizil et al., 2013). For styles where such lexicons are not readily available, we explore three methods to create lexicons: (i) prompting ChatGPT to generate words for each class of a style, e.g., the words for the “humorous” class are “funny, laugh-outloud, silly”; (ii) extracting the definition of each class from Google Dictionary,2 e.g., “being comical, amusing, witty" for the “humorous” class; (iii)
2An online service licensed from Oxford University Press: https://www.google.com/search?q=Dictionary
asking a native English speaker to write a list of words for each style. Creation details and more lexicon examples are provided in Appendix B.

heading: 2.3 Experimental Settings
text: To assess the effectiveness and generality of lexicon-based instructions, we compare our approach with other prompting methods in two learning settings.

heading: 2.3.1 Zero-Shot
text: A model is prompted to predict the evaluation styles without any labeled data. In this setting, we evaluate our Style-* models that are instruction-tuned on the training styles (introduced in §2.1). We also experiment with models fine-tuned on general instruction tuning data, including Flan-T5 and LLaMA-2-Chat. For each model, we compare the Standard instructions and lexicon-based instructions (i.e., + Lex) without demonstrations (i.e., example sentences for a evaluation style). Both methods utilize the same instruction template described in Appendix E.1, except that class names instead of lexicons are used in standard instructions. To construct a lexicon-based instruction, for each class (e.g., “polite” or “impolite”) of the style (e.g., politeness), we randomly select m words from the corresponding lexicon, then incorporate them into the instruction. We use m = 5 in the main paper and perform an analysis on varied values of m in Appendix E.3.

heading: 2.3.2 Few-Shot
text: We also investigate how different prompting methods perform in the few-shot setting, where a few training examples of the evaluation styles are available. These experiments are not necessarily in-
tended to improve upon the state-of-the-art on this benchmark, but rather to compare the impacts of using in-context examples versus lexicons in enhancing few-shot generalization capabilities.
MetaICL (Min et al., 2022a; Chung et al., 2022). We adapt MetaICL, a method that meta-tunes on a collection of tasks with an in-context learning objective, to align models to better learn from the incontext examples through meta-tuning on a set of source styles. During each iteration of fine-tuning, one source style is sampled, and K labeled examples are randomly selected from the train set of that style. Each prompt consists of K demonstrations followed by an input sentence for the model to predict the class. At inference time, the prompt is built similarly to the fine-tuning stage, except that the K demonstrations are sampled from the train set of target styles instead of source styles. Recently, Min et al. (2022b) have shown that ground-truth labels are not always needed in MetaICL. We re-examine this finding in the context of style classification, experimenting with both random and gold labels in the demonstrations. We follow Min et al. (2022b) to set K = 4 and K = 16.
MetaICL+Lex. For a more comprehensive comparison between the two sources of supervision (i.e., demonstrations vs. lexicons), we also modify MetaICL to incorporate lexicons. Specifically, we concatenate the name of each class with its corresponding lexicon words, and prepend this information to each labeled example to form a modified demonstration. Each prompt contains K modified demonstrations followed by an input sentence.

heading: 3 Results and Analysis
text: We report macro-average F1 for style classification tasks following the XSLUE benchmark (Kang and Hovy, 2021). Our experimental results show that lexicon-based instructions can improve the zeroshot style classification performance in all settings, especially when source style meta-tuning and class randomization are involved.

heading: 3.1 Pre-trained Language Models
text: We experiment with T5 (Raffel et al., 2020), GPTJ (Wang and Komatsuzaki, 2021), and LLaMA-2 (Touvron et al., 2023). We also include experiments with the instruction-tuned models Flan-T5 (Chung et al., 2022) and LLaMA-2-Chat (Touvron et al., 2023), as these models have demonstrated
the ability to effectively respond to instructions and generalize well to unseen tasks (Chung et al., 2022; Touvron et al., 2023).3 Implementation details are described in Appendix D.

heading: 3.2 Zero-shot Learning Results
text: Table 2 shows zero-shot learning results for different methods on various models. We compare them with four distinct baseline methods, which are described in the caption of Table 3.
Lexicon-based instructions outperform standard instructions. In the zero-shot setup, incorporating lexicons into instructions demonstrates a significant advantage over the standard instructions without lexicon information across all the experimented models. For example, after integrating lexicons into instructions and randomizing classes, + Lex improves upon the standard instructions by an average of 23.58 F1 points on Style-T5 and an average of 13.22 F1 points on Style-LLaMA. One possible explanation for this improvement is that fixing class names during source fine-tuning may lead the model to memorize these names, rather than meta-learning to classify target styles zeroshot. This is not ideal as our goal is to predict unseen styles and thus learning to use lexicons is important. By randomizing class names during meta-tuning, the model is forced to use lexicons, rather than relying on source-style identifiers (e.g., class names). Further experiments on class randomization are presented later in this section.
Furthermore, we find that lexicons seem to improve zero-shot style classification even without meta-tuning. For example, by simply integrating lexicons into instructions, LLaMA-2-Chat models improve their performance in most styles. Notably, F1 jumps from 43.84 to 51.01 for Humor style on LLaMA-2-Chat (7B).
Meta-tuning on style data with lexicons enhances the zero-shot performance compared to general instruction tuning. Both Style-T5 and Style-LLaMA demonstrate a significant performance improvement over their general instructiontuned counterparts, i.e., Flan-T5 and LLaMA-2Chat, when lexicons are included. For instance, Style-LLaMA (7B) outperforms LLaMA-2-Chat (7B) in five out of six styles, achieving an average increase of 4.28 F1 points. This suggests the
3We ensure the evaluation style datasets are excluded from the fine-tuning tasks of Flan-T5, but it is unclear if LLaMA-2Chat has been trained on these evaluation styles before.
benefits of lexicon-based instructions and the effectiveness of instruction tuning on training styles.
Class randomization matters in lexicon-based meta-tuning. We study the impact of natural language descriptions and class randomization in our approach by independently fine-tuning Style-T5 on training styles using nine lexicon-based instruction variants (see §3.2.1 below). Our experimental results in Figure 2 show that introducing class randomization can improve the zero-shot performance on the six unseen evaluation styles consistently. For example, the average F1 improves from 35.58 (minimal) to 50.54 (R#).

heading: 3.2.1 Details on Lexicon-based Instruction Variations with Class Randomization
text: Here we describe the lexicon-based instruction formats tested, including different types of class ran-
domization. All prompt variants are summarized in Table 5, while example prompts for each variant are shown in Figure 5 in the Appendix. “R#” represents randomizing class names with numerical indices, and “Rw” means using random words as class names in the instruction. We simply use the default English word list in Ubuntu4 for this randomization. “Rw-" and “Rw" differ only in the size of the fixed set from which a class name is sampled. “Rw” uses a much larger set (“vocab size”) compared to other variants, which reduces the chance of assigning the same word to the same class in different examples. This class randomization has pros and cons. On one hand, it may hurt performance because it prevents the model from inferring the meaning of classes from class names. On the other hand, it could enhance performance by
4/usr/share/dict/words
tuning with class Randomization shows advantages over those without. Instructions with natural language perform generally better than those without. We also compare fine-tuning with generic identifiers, a method that differs from the “Lang” variant by mapping class names of each style to a fixed set of generic names (e.g., Style A, Style B). This approach improves model generalization over “Lang”, but generally falls short of the performance achieved with our optimal randomized identifiers “Lang, Rw”.
encouraging the model to genuinely learn the inputclass mappings and make use of lexicons, rather than memorizing class names from training styles that are observed during meta-training. Figure 2 shows that class randomization is helpful, possibly because the latter factor outweighs the former.

heading: 3.3 Few-shot Learning Results
text: Table 4 shows results of few-shot learning methods.
Incorporating lexicons in few-shot learning reduces the sensitivity to example selection. Prior work has shown that the choice of examples selected for few-shot learning can lead to very different performance (Zhao et al., 2021; Liu et al., 2022). Hence how to reduce the sensitivity due to example selection has become an important question. In Table 4, we observe that after introducing lexicons into prompts, the standard deviation of
performance across five runs generally decreases. For example, MetaICL4 performs extremely unreliably on Romance with a high standard deviation of 27.30, while MetaICL4+Lex not only improves performance but also stabilizes inference with a standard deviation dropped to 6.52. This suggests using lexicons may reduce a model’s reliance on the selection of few-shot examples (Liu et al., 2022).
Introducing lexicons into in-context examples can be beneficial when gold labels are absent. When the examples of the evaluation style are randomly labeled (Min et al., 2022b), introducing lexicons into MetaICL is generally more useful than increasing the number of examples. In four out of six styles, MetaICL16 shows a lower average across five runs, compared to MetaICL4. The average F1 score (38.47) of MetaICL16 over six styles is lower than that (42.79) of MetaICL4. In contrast, MetaICL4+Lex outperforms MetaICL4, achieving an average F1 score of 45.29 over the six styles. Four styles show improved average performance across five runs with MetaICL4+Lex, compared to MetaICL4. For the other two styles (i.e., Shakespeare and Age) where MetaICL4+Lex underperforms, their standard deviation in MetaICL4+Lex is much smaller than in MetaICL4. When ground-
truth labels are accessible, MetaICL16 showcases a superior average performance, suggesting that increasing the number of demonstrations might be more effective in this case.

heading: 4 Generalization to Novel Styles
text: In prior sections, we demonstrated the effectiveness of lexicon meta-tuning on existing style datasets. To demonstrate that our method is able to generalize beyond styles that have been previously studied in the NLP community, we use LLMs to semiautomatically propose new styles, and then generate instances of text presenting each style (i.e., labeled examples). The new styles generated in this section are then used to evaluate our approach’s ability to generalize to styles that include but are not limited to niche literary genres, or rapidly evolving communication styles in social media (see examples in Table 14 and 15 in the Appendix).

heading: 4.1 A Diverse Collection of New Styles
text: Style Creation. We compiled a diverse collection of language styles by initiating the data generation based on the 13 styles listed in Table 1. This initial set served as seeds for prompting LLaMA-2-Chat 70B to generate new style classification tasks. We filtered out any LLM-generated tasks that did not align with our textual classification objective. To encourage diversity, a new task is added to the pool only when its ROUGE-L similarity with any existing task is less than 0.6. This process produced 58 new unique style classification tasks (full list in Appendix Table 14). We then randomly divided these tasks into the training and evaluation split, avoiding task overlap. To further enrich the diversity, we developed and added 5 additional tasks to the evaluation split, such as composite chatbot styles (e.g., a blend of empathetic, colloquial, and humorous responses), and writing styles of various authors. Please refer to Appendix C.1 for additional details on the style creation process.
Lexicon Creation. To ensure consistency and clarity in our approach to style identification, we developed a lexicon for each new style. This was achieved by prompting LLaMA-2-Chat 70B, as detailed in Appendix C.2, to generate a concise lexicon comprising up to five words or phrases for each style class. Depending on the construction method, these lexicons may vary in quality and size from a few words to thousands. Nevertheless, we demonstrate the benefits of our method with as
few as five words per style sampled from lexicons (Appendix E.3), and highlight the robustness of lexicon-based instructions across various lexicon creation methods, particularly when class randomization is applied.
Labeled Example Generation. We employed LLaMA-2-Chat 7B to generate 100 unique examples for each class in our training style split, which results in a training style dataset Dtrain. For the evaluation style dataset Deval, we used GPT-4 (OpenAI, 2023) to create high-quality stylistic examples. Through the OpenAI API, we generated 20 examples for each class at a total cost of $9.11. Details about this process are in Appendix C.3, together with examples of lexicons in Table 15.
Human Verification of Data Quality. To measure the reliability of Deval, we asked three human annotators5 to independently review a shared set of 500 randomly selected annotation examples. Annotators were instructed to assess the accuracy of labels for examples generated by GPT-4 and make necessary corrections, as detailed in Appendix C.4. We computed inter-annotator agreement using Krippendorff’s alpha. A high agreement score of 93.27% reflects strong reliability in the annotations (Krippendorff, 2004).
Corpus Statistics. Our data generation process produced a collection of 11,358 distinctive examples, spanning 63 varied style classification tasks. Table 6 describes the statistics of our data. The distribution of K-class tasks (where K is the number of distinct style classes to be distinguished) is illustrated in Figure 4 in the Appendix, showcasing the diverse range of styles included in our analysis.

heading: 4.2 Experiments
text: Experiment Setup. We evaluated the zero-shot performance of LLaMA-2-Chat (7B, 13B) and Style-LLaMA (7B) on Deval. Given the balanced class distribution in this test set, we report accuracy
5The three annotators include: one of the authors, a graduate student in CS, and a mathematician.
in Table 7. We also include Style-LLaMA+ (7B), which is a LLaMA-2 model fine-tuned on a mix of benchmark training styles and the training set Dtrain generated by LLaMA-2-Chat 7B. Note that the training set Dtrain and the evaluation set Deval were created by different language models (§4.1). Implementation details are described in Appendix D. A baseline was set by randomly assigning a class to each example, averaging the results over five seeds.
Results Table 7 demonstrates the advantages of lexicon-based instructions over the standard instructions. Notably, Style-LLaMA and StyleLLaMA+ show the most significant performance gains, with an average improvement of 12.46 and 8.85, respectively. This is likely because lexiconbased instruction-tuning enhances their adaptability to new styles through more effective lexicon usage. Furthermore, Style-LLaMA+ shows a substantial improvement over other models, suggesting that the inclusion of a diverse set of model-generated style training data can effectively enhance performance. The high score of Style-LLaMA+ with lexicon integration suggests that the combination of additional training data and lexicon-based instructions might be the most effective approach for generalization among the evaluated methods.

heading: 5 Related Work
text: Style classification. Research in NLP has studied various language styles. Kang and Hovy (2021) provided a benchmark for fully-supervised style classification that combines many existing datasets for style classification, such as formality (Rao and Tetreault, 2018), sarcasm (Khodak et al., 2018), Hate/Offense (i.e., toxicity) (Davidson et al., 2017), politeness (Danescu-Niculescu-Mizil et al., 2013), and sentiment (Socher et al., 2013; Wang et al., 2021). Other writing styles include but are not limited to readability (i.e., simplicity) (Arase et al., 2022), Shakespearean English (Xu et al., 2012),
subjectivity (Pang and Lee, 2004), and engagingness (Jin et al., 2020). Despite an extensive range of style classification tasks studied in prior research, zero-shot or cross-style classification is relatively underexplored (Puri and Catanzaro, 2019). In particular, much of the cross-style research thus far has focused on text generation tasks (Jin et al., 2022; Zhou et al., 2023), rather than classification. In this study, we aim to address this gap in the literature by concentrating on zero-shot style classification across a collection of diverse styles.
Language model prompting. Large language models, such as GPT-3 (Brown et al., 2020), exhibit impressive zero-shot learning abilities when conditioned on appropriate textual contexts, i.e., prompts, or natural language instructions. Since then, how to design appropriate prompts has become a popular line of research (Sanh et al., 2021; Chung et al., 2022). In this work, we propose to incorporate lexicons into instructions and teach the model to better utilize stylistic lexicon knowledge through instruction tuning. Recently, Zhou et al. (2023) specified styles in instructions as constraints to improve controlled text generation. Parallel to our study, Gao et al. (2023) investigated label descriptions to enhance zero-shot topic and sentiment classification. We focus on style classification, a challenging area in NLP characterized by its extensive scope and complexity, encompassing a wide range of stylistic expression across various domains of text. In order to improve the generalization capabilities of instruction-tuned models, we replace class names in instructions with entirely random words during fine-tuning on training styles. This is similar to Zhao et al. (2022), which indexes and shuffles slot descriptions in prompts for dialogue state tracking. Moreover, our work differs from the standard practice in previous studies (Min et al., 2022b; Zhao et al., 2022; Wei et al., 2023), where a pre-defined set of class names, is equal in size to the number of labels in the associated datasets.

heading: 6 Conclusion & Discussion
text: In this work, we study zero-shot style classification using large language models in combination with lexicon-based instructions. Experiments show that conventional instructions often struggle to generalize across diverse styles. However, our lexiconbased instruction approach demonstrates the potential to fine-tune models for improved zero-shot generalization to unseen styles. By utilizing a diverse
range of data sources, such as Wikipedia, captions, social media (Table 1), and LLM-generated data (§4), we ensure the robustness and generalizability of our findings across various contexts.
Furthermore, the wide range of tasks, together with the flexible and effective usage of lexicons highlights potential applications of our approach. For example, our approach can be used to identify harmful or toxic content, interpret the implicit meaning in communication (e.g., humor, sarcasm, friendliness, hostility, etc.), and assess text readability. More potential applications can be found in Appendix Table 14. In addition, our method may generalize to generation tasks (e.g., text style transfer), which we intend to explore in future work.

heading: Limitations
text: The ambiguity of style in language often poses a challenge for classification efforts. Individual interpretations of styles can vary widely based on personal, cultural, and contextual factors (Hovy, 1987), underscoring the nuanced nature of language and the difficulty of categorizing it into discrete styles. Our method, which utilizes lexicon-based instruction, mitigates this issue by focusing on stylistically expressive words or phrases rather than relying solely on style names. This approach offers a degree of flexibility in representing styles, although it does not fully resolve the underlying complexity. Future research should continue to explore these nuances, striving for definitions that are both precise and adaptable to the diverse ways in which style manifests.
Additionally, in our method, we leverage the lexicons we have collected (as detailed in Table 1). However, it is important to acknowledge that a potential limitation of our approach lies in the possibility of different performance outcomes when using lexicons of varying qualities. While we have conducted comparisons between lexicons from different sources in Appendix E.2, it is plausible that utilizing different lexicons could yield different results.
Another limitation is that we only include a limited set of styles in English for evaluation in §3, due to availability of high-quality style datasets and lexicons. While we attempted to assess the generalization capabilities of our method to a novel set of GPT-4 generated language styles in §4, it is important to recognize that they might not be representative of the styles that users wish to an-
alyze using our approach. Moreover, our artificially balanced datasets may not accurately reflect the imbalanced label distribution in real-world scenarios. Furthermore, utilizing LLMs to generate datasets can perpetuate the inherent biases within LLMs and introduce biases through the choice of prompts used (Wang et al., 2023), which complicates the task of creating unbiased and representative datasets. Future work could focus on curating human-written datasets to assess robustness and applicability of lexicon meta-tuning in practical applications.

heading: Ethical Considerations
text: Style classification is widely studied in the NLP research community. We strictly limit to using only the existing and commonly used datasets that are related to demographic information in our experiments. As a proof of concept, this research study was only conducted on English data, where human annotations for multiple styles are available for use in the evaluation. We also acknowledge that linguistic styles are not limited to what are included in this paper, and can be much more diverse. Future efforts in the NLP community could further extend research on stylistics to more languages and styles.

heading: Acknowledgements
text: We thank Tarek Naous, Michael Ryan, Fan Bai, Shuheng Liu, Junmo Kang, Duong Minh Le and anonymous reviewers for their helpful feedback on this work. We also would like to thank Azure’s Accelerate Foundation Models Research Program graciously providing access to API-based models, such as GPT-4. This research is supported in part by the NSF (IIS-2052498, IIS-2144493 and IIS2112633), ODNI and IARPA via the HIATUS program (2022-22072200004). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of NSF, ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A Benchmark Datasets Details
text: The XSLUE benchmark, designed for exploring cross-style language understanding, encompasses 15 styles (Kang and Hovy, 2021). We choose 10 writing styles from XSLUE based on their suitability for our task. Specifically, we consider the task type (i.e., whether the task is classification or not), task granularity (e.g., whether the annotated style is sentence-level or not), expressiveness at both the word and phrase level (i.e., the possibility of expressing a style with lexicons). For example, the TroFi dataset for style Metaphor is not used because it is focused on the literal usage of one specific verb in a sentence. Take the verb “drink" as an example, it is a literal expression in the sentence “‘I stayed home and drank for two years after that,’ he notes sadly”, whereas in “So the kids gave Mom a watch, said a couple of nice things, and drank a retirement toast in her honor”, “drink” is non-literal.

heading: B Benchmark Lexicons Details

heading: B.1 Lexicon Creation
text: ChatGPT-generated Lexicons. Prior work has used models, such as BERT to generate class vocabularies for topic classification (Meng et al., 2020). Inspired by this approach, we utilize the knowledge of LLMs by prompting them to generate a list of words that express the specific class of a style. In a preliminary study, we experimented with many LLMs, including BERT, GPT-J, GPT-NeoX, GPT3.5 and ChatGPT. Among all, ChatGPT performs the best, so we use it to generate the lexicons. Table 8 shows the prompts we used for ChatGPT. Figure 3 presents some examples of ChatGPT output.
Dictionary-based Lexicons. We also considered lexicons generated by extracting the definition of each style from Google Dictionary.

heading: B.2 Statistics and Examples of Lexicons
text: Table 9 provides the statistics of NLP and ChatGPT lexicons used in the experiments. Table 10 shows examples of lexicons from different sources.

heading: C Model-Generated Data For Generalization Experiments
text: Recall in §4 that in order to further evaluate the generalization capabilities of our porposed approach, we collected a diverse collection of styles using LLMs. Here we provide more details throughout
the data generation process, including style creation (§C.1), lexicon generation (§C.2), and labeled example (i.e., instance) generation (§C.3).

heading: C.1 Style Creation
text: We initiated the process of style classification task generation based on the thirteen styles outlined in our benchmark (refer to Table 1). We had one author write the style classification instruction for each of these thirteen styles. During the task generation process, we randomly selected eight incontext examples from our pool, including three seed tasks and five model-generated tasks. We employed LLaMA-2-Chat 70B for new task generation. The template used for prompting these new style classification tasks are detailed in Table 11. To ensure the diversity of the generated style classification tasks, a new task is added to the pool only when its ROUGE-L similarity with any existing task is less than 0.6. This process resulted in a total of 58 model-generated tasks, which we divided into 43 training tasks and 15 evaluation tasks. In order to further enrich the diversity of the evaluation task split, we designed 5 additional style classification tasks and incorporated them into the evaluation task split. Overall, this data generation process produces a total of 43 training style classification tasks and 20 evaluation style classification tasks. We present the full list of 63 generated style classification tasks in Table 14.

heading: C.2 Lexicon Creation
text: After creating the training and evaluation tasks, we employed LLaMA-2-Chat 70B to generate a concise lexicon for each class in the style classification tasks, using in-context examples. Our ablation studies, as detailed in §E.3, revealed that a lexicon consisting of just five words or phrases are sufficient for effective generalization to new styles. So we restricted the lexicon size for each style class to five words or phrases. The template used for prompting the generation of style class lexicons are displayed in Table 12.

heading: C.3 Labeled Example Generation
text: We prompted LLaMA-2-Chat 7B to generate labeled examples for our training style classification tasks, and GPT-4 to generate examples for our evaluation tasks. Both utilize the same prompting template presented in Table 13 for labeled example generation.
Style Classification Task Classes Identify the type of writing style used in a given text. narrative, descriptive, expository, persuasive
Determine whether the given text contains any errors in grammar, spelling, or punctuation. error-free, erroneous
Classify the style of a poem into one of the four types. sonnet, haiku, free verse, limerick
Categorize the emotion of the utterances. angry, disgusted, fearful, happy, sad
Determine the level of organization in the text. well-organized, disorganized
Classify the style of a text according to its structure. chronological, non-chronological
Classify the text according to its tone. friendly, hostile, neutral Define the writing style "Infotainment" as "merging informative writing with an entertaining approach". Define the writing style "Techeative" as "blending technical writing (e.g. precise descriptions of complex subjects) with creative elements to make it more engaging and understandable". Classify the style of a presentation into one of the above two categories. Infotainment, Techeative
Classify the style of a text according to its content and language use. rational, irrational
Evaluate the level of clarity in the text. clear, unclear
Classify the text style according to its tone and language use. nostalgic, reflective, analytical
Classify the style of a text according to its content and language use. creative, conventional
Identify the author’s voice style in a given text. authoritative, unreliable
Evaluate the level of emotional appeal in the text. low emotional appeal, high emotional appeal
Determine the level of originality in a story. original, somewhat original, not original
Evaluate the level of credibility in the text. credible, moderately credible, not credible Read a passage, and select the topic for this passage based on the content and text style. finance, politics, health, education,technology, entertainment
Read the summary of a book and categorize its genre. science fiction, romance, thriller, biography
Determine the primary intention behind the author’s writing of a specific text. persuasive, informative, entertaining, educational
Classify the text style according to its tone and language use. assertive, submissive
Classify the text style according to its tone and language use. strong, weak
Classify text style. conversational, academic
Determine the most likely author based on the writing style. Hemingway, Joyce, Kafka, Hurston, Christie
Classify the text style according to its tone and language use. monotonous, engaging
Classify the content of a piece of text. spam, ham
Read the text and classify its style. fictional, non-fictional
Evaluate the mood of a song based on its lyrics. relaxing, energizing, romantic, melancholic Identify the rhetorical devices used in a given text. onomatopoeia, alliteration, hyperbole,repetition, oxymoron
Classify the text as one of the following: journalistic, academic, or literary. journalistic, academic, literary
Assess how supportive the context is in response to a request for help. very supportive, moderately supportive, not supportive
Classify the text according to its tone and language use. realistic, idealistic
continued on next page

heading: C.4 Data Quality
text: To investigate the quality of our generated evaluation set Deval, we randomly sampled a shared set of 500 annotation examples. We asked each annotator to assess the quality of the examples and their labels. Specifically, they were instructed to consider these questions: (1) is it easy to identify a style (e.g., low emotional appeal vs. high emotional appeal) in a given text example? (2) is the label of the given example accurate? (3) If the label is incorrect, what would be the correct label?
We collected responses from three annotators. According to their quality reviews, about 97% of the examples are easily identifiable by their styles, with all annotators answering “Yes" to the first question. Gathering responses for the last two questions from three annotators provided three human annotations for each example. We followed prior works (Kang and Hovy, 2021; Artstein and Poesio, 2008) to compute the inter-annotator agreement using Krippendorff’s alpha (Krippendorff, 2004) for these annotation examples. A high agreement score of 93.27% indicates strong reliability in the annotations (Krippendorff, 2004). Moreover, our further analysis reveals that 87.4% of the evaluated examples received unanimous agreement the GPT-4 generated label was correct from all three annotators, and 99.2% reached a consensus among at least two annotators, highlighting a high level of agreement and consistency in the annotation process.

heading: C.5 Statistics and Examples of Generated Data
text: Figure 4 plots the distribution of 63 generated style classification tasks in this data generation process. We present examples of style annotation data and their lexicons in Table 15.
D Implementation Details
We use PyTorch (Paszke et al., 2019) and Huggingface Transformers (Wolf et al., 2020) in the experiments.
In our preliminary studies, we compare the performance of fully fine-tuned, partially (only the last two layers) fine-tuned, and parameter-efficiently fine-tuned GPT-J. We find that fine-tuning GPTJ with LoRA (Hu et al., 2021) achieves the best performance, so we use this method in our main experiments with GPT-J.
In the zero-shot learning experiments, we
prompted LLaMA-2-Chat (13B) to predict the target styles without any fine-tuning. We employed 4-bit inference due to our computing resource constraints (Dettmers and Zettlemoyer, 2023).
In the zero-shot cross-style experiments, we first fine-tuned a model on the training styles before evaluating it on the evaluation styles. We fine-tuned the LLaMA-2 (7B) model on 4 A40 GPUs using DeepSpeed. All the other models were fine-tuned on one single A40 GPU. Hyperparameters are selected following the common practices in previous research. Table 16 reports the hyperparameters for our instruction tuning.

heading: E Additional Experimental Results & Analyses
text: E.1 Impact of Instruction Templates
Prior works find that prompting an LLM on an unseen task is extremely sensitive to the prompt design, such as the wording of prompts (Sanh et al., 2021). To investigate the sensitivity of lexiconbased instructions, we experiment with four instruction templates t1, t2, t3, t4 (Table 20), each of which contains different natural language task instructions. For each template, we fine-tune a model on our benchmark training styles using lexiconbased instructions. Table 17 shows that without randomization during instruction tuning, lexiconbased instruction (i.e., the “Lang” variant) is sensitive to the choice of templates. However, after introducing class randomization, lexicon-based instruction (i.e., the “Lang, Rw” variant) improves the average F1 across the templates by a substantial margin, while reducing the standard deviation, indicating that it is more robust to the wordings of the prompts.

heading: Instruction Template in Main Experiments In
text: our main experiments (§3), we conduct a comparative analysis between the lexicon-based instruction and the standard instruction. Both utilize the template t2 in Table 20 except that the standard instruction does not incorporate any lexicon sampling. Instead, each slot for the lexicon words contains only the corresponding class name. Here is an example input of the standard instruction on Politeness: In this task, you are given sentences. The task is to classify a sentence as "polite" if the style of the sentence is similar to the words "polite" or as "impolite" if the style of the sentence is similar to the words "impolite". Here is the sentence: "I’ve just noticed I wrote... and smooth out the text?". Its output is polite.
E.2 Impact of Lexicon Source
We study the impact of lexicon choices in lexiconbased instruction that include: (1) dict: all lexicons are from dictionary; (2) nlp+chat: for classes that have NLP lexicons, we directly use them, whereas for those without, we create ones using ChatGPT; (3) class: each class lexicon contains only its class name, e.g., the “humorous” lexicon has a single word “humorous”; (4) human: we have a native speaker create a lexicon for each style class, by carefully choosing words or phrases that best capture the characteristics of each style class. Table 17 shows that without class randomization during instruction tuning with lexicon, the average F1 for nlp+chat across four templates is the highest at 40.54. With randomization, dict performs the best at 54.50. Randomizing classes with words
OutputVariant
Lexicon-Based Instruction Input
in lexicon-based instructions consistently improves the average F1 while reducing the standard deviation across four lexicon sources, regardless of the prompt templates used. The human-created lexicon is the most robust to the change of templates.
E.3 Varying Number of Lexicon Words (m) in Lexicon-Based Instructions
When predicting a style in the evaluation split zeroshot, the lexicon instruction-tuned model only has access to a subset of m lexicon words that express or imply the style classes rather than example sentences. To investigate the model’s dependence on the number of lexicon words, we take the variant of lexicon-based instruction with class randomization
(i.e., the “Lang, Rw” variant) and incrementally increase m from 0 to 30 in both fine-tuning and evaluation phases. Figure 7 shows a general trend that the average F1 of six targets initially increases with increasing m, but then either drops or stabilizes. On average, our method performs the best when m = 5.
Moreover, we fix the model fine-tuned with the “Lang, Rw" lexicon-based instruction variant at m = 5, and then gradually increase m while evaluating evaluation styles. A similar trend is noticed in Figure 7. It can also be seen that when target styles have no lexicon resources (m = 0), increasing the number of lexicon words in each prompt during source fine-tuning might be beneficial. For instance, “src-5, tgt-0” improves the performance of “src-0, tgt-0” by an average of 3.96 F1 points.
Figure 8 provides a detailed view of the performance change associated with an increase in m, broken down by each target style. It reveals that different styles reach their peak performance at different values of m.
over the six target styles. Light grey cells indicate the average (Avg.) and the standard deviation (SD.) scores over four lexicon sources. Dark grey cells represent Avg. and SD. over four templates.

heading: E.4 More Experiments on Style Splits
text: This section presents additional experimental results of our approach, utilizing various style splits outlined in Table 18. Results are presented in Table 19. It is observed that lexicon-based instruction tuning consistently outperforms standard instruction tuning across various style splits in both T5 and GPT-J models.

heading: E.5 Comparisons of MetaICL and
text: Lexicon-Based Instructions in Few-Shot Learning
To compare lexicon-based instructions and MetaICL fairly, it is necessary to incorporate supervision from K demonstrations in evaluation style
into our approach. We thus introduce a modification to lexicon-based instructions called +Lex +K. Specifically, for each evaluation style, we randomly select K examples from its train set and assign a label to each. Next, a model that was previously fine-tuned on the training styles using the ’Lang, Rw’ lexicon-based instructions, is further fine-tuned on these K demonstrations. Finally, we evaluate the model on the evaluation style using lexicon-based instructions without demonstrations.
The results are reported in Table 21. It is observed that with random labels, +Lex +K generally outperforms other methods. These may suggest that lexicons can provide a useful signal for the prediction of unseen styles when the gold labels of examples are absent.
E.6 Varying Number of Training Examples (K) used in Few-Shot Learning
We investigate the impact of the number of examples (K) that are used in the few-shot learning methods MetaICLK and +Lex +K. Results are reported in Figure 9. The performance of both methods deteriorates with an increase in K when using random labels. However, when gold labels are utilized for the target-style training examples, the performance improves with larger K, particularly showing significant improvement from K = 8 to K = 16. Moreover, as K increases, the performance disparity between utilizing ground-truth labels and random labels further expands. These observations show that the ground-truth input-label mapping is important in our case.

heading: F More Prompting Examples
text: Figure 5 shows the example input and output for all lexicon-based instruction variants. In MetaICLK+Lex, one prompt consists of K demonstrations and an input sentence. Figure 6 provides an example demonstration.

heading: Future Work
text: Future efforts in the NLP community could further extend research on stylistics to more languages and styles