Venue: NAACL 2025

# StyleDistance: Stronger Content-Independent Style Embeddings

with Synthetic Parallel Examples

 Ajay Patel \({}^{\dagger}\)1 Jiacheng Zhu\({}^{\dagger}\)2 Justin Qiu\({}^{\dagger}\)&Zachary Horvitz\({}^{\ddagger}\)

Marianna Apidianaki\({}^{\dagger}\)Kathleen McKeown\({}^{\ddagger}\)Chris Callison-Burch\({}^{\dagger}\)

University of Pennsylvania\({}^{\dagger}\)Columbia University\({}^{\ddagger}\)

Footnote 1: Denotes equal contribution; direct correspondence to: ajayp@upenn.edu

###### Abstract

Style representations aim to embed texts with similar writing styles closely and texts with different styles far apart, regardless of content. However, the contrastive triplets often used for training these representations may vary in both style and content, leading to potential content leakage in the representations. We introduce StyleDistance, a novel approach to training stronger content-independent style embeddings. We use a large language model to create a synthetic dataset of near-exact paraphrases with controlled style variations, and produce positive and negative examples across 40 distinct style features for precise contrastive learning. We assess the quality of our synthetic data and embeddings through human and automatic evaluations. StyleDistance enhances the content-independence of style embeddings, which generalize to real-world benchmarks and outperform leading style representations in downstream applications. Our model can be found at [https://huggingface.co/StyleDi](https://huggingface.co/StyleDi) stance/styledistance.

## 1 Introduction

The most common objective when training text embeddings is to place texts with similar semantics close together in the embedding space Reimers and Gurevych (2019). Style representations, by contrast, aim to embed texts with similar writing styles near each other and texts with different styles far apart, regardless of their semantic content Wegmann et al. (2022). Embeddings are usually trained via contrastive learning, with triplets consisting of an anchor text, a positive text (which should be embedded closely to the anchor), and a negative text (which should be embedded far from the anchor) Goldberger et al. (2004); Khosla et al. (2020); Schroff et al. (2015). Existing approaches often use social media datasets with the assumption that all writing by the same author shares a similar style and that texts by different authors exhibit dissimilar styles Wegmann et al. (2022). These methods also attempt to minimize content representation in the resulting embeddings. They select a text from the same author on a _different topic_ as the positive example, and a text from a different author on the social media datasets with the assumption that all writing by the same author shares a similar style and that texts by different authors exhibit dissimilar styles Wegmann et al. (2022). These methods also attempt to minimize content representation in the resulting embeddings. They select a text from the same author on a _different topic_ as the positive example, and a text from a different author on the

Figure 1: StyleDistance embeddings are trained using contrastive learning from synthetic parallel (positive and negative) examples representing 40 style features. The illustrated example is for the “Usage of Active Voice” feature.



same topic_ as the negative example, approximating topic similarity using subreddit or conversation metadata. However, these methods are limited by the imperfect nature of data acquired under such assumptions. For example, the same author may write about the same topic even in different subreddits. As a result, these imperfect contrastive triplets do not explicitly control for content, leading to style embeddings with weak content-independence. The "content leakage" caused by such proxy objectives (illustrated in Figure 2) can undermine the effectiveness of style representations in tasks that require strict separation between style and content, such as stylistic analysis, authorship tasks, style transfer steering, and automatic style transfer evaluation. To overcome this, a more controlled approach to style contrastive learning is necessary.

In this paper, we introduce StyleDistance, a novel method for training stronger, content-independent style embeddings which leverages synthetic parallel text examples generated by a large language model (LLM) [14]. By creating near-exact paraphrases with controlled stylistic variations, we produce positive and negative examples across 40 distinct style features. This synthetic dataset, which we call SynthStel, enables more precise contrastive learning (visualized in Figure 1) and is more robust to the content leakage inherent in existing datasets. We evaluate our method on both human and automated benchmarks, measuring the content-independence, quality, and utility of StyleDistance embeddings.

In summary, our primary contributions are:

1. We generate and release SynthStel, a dataset of near-exact paraphrases across 40 distinct style features.
2. We introduceStyleDistance, a new approach to style representation learning which uses synthetic parallel examples with controlled stylistic variations. We release the embedding model as a resource.
3. We demonstrate that StyleDistance significantly improves the content-independence of style embeddings, generalizing effectively to real-world benchmarks of style representation quality and outperforming existing style representations in downstream applications.

## 2 Related Work

Style RepresentationsIn previous work, style representations were learned from unlabeled texts [56, 79, 80, 81]. Due to the lack of parallel datasets covering diverse style features, Hay et al. (2020) and Wegmann et al. (2022) have both recently used authorship as a proxy for style as we discussed previously. Rivera-Soto et al. (2021) trained embeddings to uniquely represent different authors. Although these embeddings capture features representative of authors' style, they also capture content features due to the lack of control for content-related aspects. Patel et al. (2023) trained LISA, a style vector created by annotating texts with their stylistic features using LLMs. LISA, however, trades off performance to create a vector with interpretable dimensions. The highest quality style representations to date come from approaches that employ a contrastive learning objective [16]. In this paper, we propose leveraging the strengths of generative LLMs to build a dataset that will serve to train strong content-independent style embeddings using a contrastive learning objective. From a practical perspective, style representations are useful in downstream applications such as arbitrary text style transfer [15, 14] where they help steer and guide transfer, and content-independence is important for reducing hallucinations in generations.

LLMs and Text StyleLLMs are commonly used for style transfer [13, 80, 81, 82] and style analysis [19, 80]. Their strength with style-related tasks has been demonstrated, yet leveraging this knowledge to build strong content-independent style representations has not yet been explored.

## 3 Data Generation

A core component of our proposed StyleDistance approach is a LLM that generates a synthetic dataset with controlled content and style. The dataset is composed of pairs of sentences which are paraphrases of each other (i.e. their content is similar) but differ in style. Each pair includes a positive example that showcases a specific stylistic feature (e.g., usage of active voice or emojis) and a corresponding negative example that lacks this feature. In this section, we outline the selected 

style features and provide implementation details for our synthetic data generation procedure. Additionally, we present evaluations conducted to assess the quality of the synthetic dataset prior to its use in training our style embeddings.

Style Feature SelectionThere is no predefined set of style features, and what features are considered to describe style vs. content can vary across different studies (Jin et al., 2022). For this work, we select 40 style features across 7 broad categories (visualized in Figure 3) which have been addressed in different works on text style (Tausczik and Pennebaker, 2010; Kang and Hovy, 2019; Wegmann and Nguyen, 2021; Jin et al., 2022; Patel et al., 2023). Specifically, we select features for which it is possible to generate both positive and negative examples (e.g., formal/informal, passive/active voice). Since some features can blur the line between style and content (e.g., usage of sarcasm), it might be difficult to generate perfectly parallel positive and negative pairs, with the same content. For these features, we control the generation as much as possible with the aim to obtain near-exact paraphrases. Furthermore, some style features may be impossible to fully remove from a sentence in order to generate a negative example (e.g., usage of articles). For these, we aim for the positive example to contain the feature with higher frequency than the negative example. For more details on all the selected style features, see Appendix A. While these 40 features may not cover the infinite number of styles that may exist, we believe they can serve to learn more primitive features (e.g., use of contractions, use of long words, use of formal style) which may help generalize to more complex styles involving these features (e.g., "professorial style"). We discuss and test this generalization assumption in Section 5.1.

GenerationFor each of the selected features, we generate 100 pairs of positive and negative examples by prompting GPT-4 (OpenAI et al., 2024) with the DataDreamer library (Patel et al., 2024). Each generated pair contains a sentence where the feature is present (positive example), and a paraphrase where the feature is absent or less present (negative example).

Yu et al. (2023) found that LLMs struggle with diversity when prompted to generate text examples. We use their proposed attributed prompt (AttrPrompt) method to ensure generations are sufficiently diverse and varied acro

\begin{table}
\begin{tabular}{l l p{186.7pt}} \hline \hline
**Anchor:** & **Awesome game.** Took off from work to play! Halway through the week playing monstop. Such a quality product. The SpiderMan game we’ve needed for years. \\ \cline{2-3}  & Such a **great** SpiderMan game.** Same & Same Style Some Content _(loads to content leakage in style representations)_ & **I observed** the impact rising temperatures had on tropical diseases. \\ \cline{2-3}
**Positive:** & **No Android version?** I guess **I’m not getting it** & Different Style Different Content _(leads to content leakage in style representations)_ & The Dolce \& Gabbana mave leather creation with a signature lock was **adored by me**. \\ \hline \hline \end{tabular}
\end{table}
Table 1: Attributes sampled for the attributed prompt string generation.

Figure 2: In the training triplets used by Wegmann et al. (2022) (left), an anchor text is paired with a positive instance written by the same author, and a negative instance written by a different author, assuming content can be controlled via subreddit/conversation metadata. However, this assumption can fail, leading to uncontrolled content as illustrated. In our dataset used to train StyleDistance (right), we control for both style and content.

 

such as "Sentence Length" and "Type of Sentence". The method randomly selects values from a defined set of attributes to be included in the prompt, which serve as conditioning for the text generation. In Table 1, we showcase the attributes we sample from in our attributed prompt in order to vary our generations. See Appendix B.2 for details on our full attributed prompt and inference parameters.

For the "Topic" attribute, we sample fine-grained distinct topics for each generation from the C4 corpus (Raffel et al., 2020). We do this by extracting a random sentence from a random document in C4, and we then use a zero-shot prompt (given in Appendix B.1) with GPT-4 to identify the fine-grained topic of that sentence. We employ several heuristics to select sentences from C4 that have desired characteristics: written in English, sufficiently long (greater than 32 words), and consist of natural text rather than formatting text found in some C4 documents. We provide an implementation of these heuristics in our supplementary materials.

We call our final synthetically generated dataset SynthStel, and create train and test splits using a 90%/10% split stratified by style feature.

Dataset EvaluationWe conduct a human and an automatic evaluation of the quality of our synthetic dataset for a number of different properties using the test split.

First, we measured the extent to which humans judge our positive examples do contain the desired style feature and our negative examples do not. Note that the appreciation of some style features (such as "Incorporation of Humor") can be subjective, and some other features (e.g., "Usage of Articles") are not fully removed in negative examples but might appear less frequently therein than in positive examples. In spite of these intricacies which made the annotators' task more difficult, our human evaluation results were strong: 92% of the time, annotators judged the positive and negative labels correct (random chance is 50%). Each instance was annotated by 10 different annotators from a pool of 73 graduate students in a NLP class. We also assessed inter-annotator agreement with Krippendorf's Alpha (Krippendorff, 2011) and achieved a reliability score of 0.55. For more details about the human annotation, see Appendix C.

We also run automatic evaluations to assess other properties of the dataset:

* **Content Similarity** measures the average semantic similarity between the positive and negative parallel examples (Reimers and Gurevych, 2019).1 Footnote 1: For evaluations using semantic similarity, we use the all-mpnet-base-v2 model.
* **Fluency** measures the average fluency of our examples2 using a classifier trained on CoLA Warstadt et al. (2019). Footnote 2: We exclude generations for style features that specifically address disfluency.
* **Diversity** uses the score proposed by Yang et al. (2024) to measure how different each generated text is from every other in terms of content/topic using semantic similarity.

We compute a baseline for these scores with natural data from the dataset of sentence pairs in Wegmann and Nguyen (2021). The results of these

Figure 3: We generate synthetic parallel examples to train StyleDistance for a wide range of style features in seven linguistic and stylistic categories. Further details on these features can be found in Appendix A.

 

evaluations can be found in Table 2. Our generated examples fare well in all these aspects: they are topically diverse and fluent, and the similarity inside each pair of positive/negative examples is high. An additional (less direct) evaluation of the quality of our synthetic dataset is proposed in Section 5, where we evaluate the StyleDistance embeddings that we train on this dataset.

Footnote 1: [https://github.com/](https://github.com/) 

both content and style features in an attempt to represent different authors. LUAR is sometimes used for automatic style transfer evaluation, where we believe a strong style representation would be better suited. We thus choose this task for evaluation.

### STEL and STEL-or-Content Evaluation

We first benchmark our embeddings on the STEL and STEL-or-Content tasks that allow for a direct evaluation of the quality of style representations. We briefly describe these tasks below and illustrate examples of these tasks in Appendix F:

* **STEL:** Given two anchor sentences (A1, A2) and two test sentences (S1, S2), STEL measures the ability of an embedding model to pair each test sentence with the anchor sentence that shares the same style based on the cosine similarity of the embeddings of the sentences.
* **STEL-or-Content:** The STEL-or-Content task is similar to STEL but more adversarially challenging, hence better for testing content-independence. In this task, there are again two test sentences (S1, S2) but only a single anchor sentence (A). The test sentence that best matches the _style_ of the anchor must be selected; but the incorrect test sentence--which is written in a different style--is a paraphrase of the anchor with similar _content_. Therefore, in order to succeed on the STEL-or-Content task, a model needs to represent style features stronger than content features.

In their paper, Wegmann and Nguyen (2021) provide a STEL and STEL-or-Content evaluation benchmark over five features with curated natural data. We test our models on this benchmark and present the results in Table 3. Our results are consistent with results reported by Wegmann and Nguyen (2021), who showed that even untrained models like roberta-base can capture style information well, resulting in stronger STEL performance than any of their fine-tuned models. However, the more challenging STEL-or-Content task, which better tests content-independence, shows that only models specifically trained for content-independence are able to capture style features better than content features. Our results indicate that the embeddings generated with our StyleDistance approach lead over other style representations on both the STEL and STEL-or-Content tasks. Interestingly, we find thatStyleDistanceSynth captures style remarkably well, and manages to generalize to the natural text examples in the evaluation benchmark despite only being trained on our synthetic contrastive triplets. We conclude using synthetic parallel examples during training makes the model more content-independent and helps better capture style.

\begin{table}
\begin{tabular}{l c|c c c c c c c c c c c c|c} \hline \multirow{2}{*}{**Features**} & \multirow{2}{*}{**Features**} & \multirow{2}{*}{**Features**} & \multicolumn{2}{c}{**Formal**} & \multicolumn{2}{c}{**Complex**} & \multicolumn{2}{c}{**Numb3r**} & \multicolumn{2}{c}{**C’tion**} & \multicolumn{2}{c}{**Emoji**} & \multicolumn{2}{c}{**Avg**} & \multirow{2}{*}{**Retained**} \\  & & & **STEL** & **S-o-C** & **STEL** & **S-o-C** & **STEL** & **S-o-C** & **STEL** & **S-o-C** & **STEL** & **S-o-C** & **STEL** & **S-o-C** \\ \hline In-Domain & 40 out of 40 & 0.85 & 0.73 & 0.60 & 0.27 & 0.71 & 0.28 & 0.99 & 0.22 & 0.63 & 0.07 & 0.76 & 0.31 & 100\% \\ Out-of-Domain & 34 out of 40 & 0.85 & 0.67 & 0.62 & 0.28 & 0.56 & 0.15 & 0.80 & 0.00 & 0.60 & 0.01 & 0.68 & 0.22 & 65\% \\ Out-of-Distribution & 25 out of 40 & 0.64 & 0.49 & 0.65 & 0.26 & 0.57 & 0.11 & 0.63 & 0.02 & 0.68 & 0.02 & 0.63 & 0.18 & 50\% \\ \hline \end{tabular}
\end{table}
Table 4: We evaluate how well StyleDistanceSynth embeddings generalize to unseen style features by ablating features from the synthetic training dataset under three conditions: In-Domain, Out-of-Domain, Out-of-Distribution. We evaluate their performance on the STEL and STEL-or-Content (S-o-C) tasks.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline
**Model** & \multicolumn{2}{c}{**Formal**} & \multicolumn{2}{c}{**Complex**} & \multicolumn{2}{c}{**Numb3r**} & \multicolumn{2}{c}{**C’tion**} & \multicolumn{2}{c}{**Emoli**} & \multicolumn{2}{c}{**Avg**} \\ \cline{2-13}  & **STEL** & **S-o-C** & **STEL** & **S-o-C** & **STEL** & **S-o-C** & **STEL** & **S-o-C** & **STEL** & **S-o-C** & **STEL** & **S-o-C** \\ \hline \multicolumn{13}{l}{_Content-Aware Representations_} \\ \hline roberta-base & 0.83 & 0.09 & 0.73 & 0.01 & 0.94 & 0.13 & 1.00 & 0.00 & 0.98 & 0.03 & **0.90** & **0.05** \\ LUAR & 0.80 & 0.14 & 0.67 & 0.00 & 0.74 & 0.03 & 0.77 & 0.00 & 0.99 & 0.02 & 0.86 & 0.03 \\ \hline \multicolumn{13}{l}{_Content-Independent Style Representations_} \\ \hline LISA & 0.73 & 0.05 & 0.65 & 0.00 & 0.85 & 0.03 & 0.92 & 0.00 & 0.77 & 0.02 & 0.71 & 0.03 \\ Wegmann et al. (2022) & 0.83 & 0.70 & 0.58 & 0.27 & 0.56 & 0.03 & 0.96 & 0.02 & 0.95 & 0.07 & 0.77 & 0.22 \\ StyleDistance & 0.89 & 0.72 & 0.64 & 0.25 & 0.84 & 0.12 & 1.00 & 0.20 & 0.99 & 0.15 & **0.87** & 0.29 \\ StyleDistanceSynth & 0.85 & 0.73 & 0.60 & 0.27 & 0.71 & 0.28 & 0.99 & 0.22 & 0.63 & 0.07 & 0.76 & **0.31** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracy on the STEL/STEL-or-Content (S-o-C) tasks. StyleDistance leads on both tasks among representations trained for content-independence, and StyleDistanceSynth generalizes remarkably well to real text data despite being trained only on synthetic data.

 



### Generalization Experiment

In an ablation study, we test the ability of our training approach to produce embeddings that can generalize to unseen style features which are not present in the synthetic dataset. We conduct this evaluation by ablating features from the data used to train StyleDistance\({}_{\text{synth}}\) under three conditions and show the results in Table 4. In the **In-Domain** condition, all 40 style features are included. In the **Out-of-Domain** condition, we exclude synthetic examples corresponding to the five style features in the STEL/STEL-or-Content benchmark.4 In the **Out-of-Distribution** condition, we further exclude examples for any features similar or indirectly related to the five evaluated features. Details on the exact 15 style features ablated can be found in Appendix H. We compare the Out-of-Domain and Out-of-Distribution performance of StyleDistance\({}_{\text{Synth}}\) on the two tasks to its In-Domain performance, obtained when it was trained on data for all 40 style features. Even in the challenging Out-of-Distribution condition, StyleDistance\({}_{\text{synth}}\) retains 50% of its performance on the challenging STEL-or-Content task (see the "Retained Perf." column). This study indicates our training approach generalizes reasonably well to out-of-domain style features which can be composed from style features selected for generation and, to some extent, even to out-of-distribution style features fully outside the selected set.

Footnote 4: In our 40 features, there are two separate features for emoji and text emoticons (:-D) so we exclude 6 total features for this condition instead of 5 features, resulting in 34 features.

### Synthetic Data for Probing

Our previous experiments demonstrate that training on our synthetic dataset yields strong style embeddings. Next, we investigate whether the synthetic dataset can be used for an entirely different purpose: to probe which specific style features are captured by existing style representations. Our SynthStel dataset allows for the creation of synthetic STEL and STEL-or-Content task instances across a range of 40 style features--much broader than the Wegmann and Nguyen (2021) benchmark where five features were addressed. We use the test split of SynthStel to generate task instances for probing. We examine whether LISA vectors and the Wegmann et al. (2022) style embeddings capture these 40 style features, whichStyleDistance models are directly trained to represent. We show average results over all 40 features in Table 5. Per feature results are provided in Appendix G. Our findings reveal only moderate coverage by LISA and Wegmann et al. (2022), with high variance depending on the evaluated feature (e.g., "Usage of Nominalizations" is poorly captured, with a near-zero STEL-or-Content score for both models). We calculate the mean squared error (MSE) between the STEL and STEL-or-Content scores for the real and synthetic task instances across the five features in the real benchmark, finding an average MSE of 0.039. This small MSE value shows that using synthetic data for probing can reasonably serve to assess which style features are represented by a model without need for manual example curation.

### Downstream Evaluation

We next evaluate and/or demonstrate our style embeddings in three downstream applications.

Authorship VerificationWe first test our style embeddings on the authorship verification (AV) task Koppel and Winter (2014). Given two documents by unknown authors, the goal of the authorship verification (AV) task is to determine whether they were written by the same author, based on their stylistic similarities and differences Kocher and Savoy (2017). We use a series of AV shared task datasets released by PAN in 2011-2015 Argarom and Juola (2011); Juola and Stamatatos (2013); Stamatatos et al. (2014, 2015).5 Since the two documents may be about different topics, good content-independent style representations would be expected to perform better in the AV task than embeddings that capture content. We calculate the cosine similarity of the two documents using our tested style embedding models with no fine-tuning, to measure their off-the-shelf ability to identify whether two documents were written by the same author and report results using the standard ROCAUC metric used in AV. In Table 6, we compare

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **STEL** & **S-o-C** \\ \hline LISA & 0.79 & 0.06 \\ Wegmann et al. (2022) & 0.76 & 0.25 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results obtained by LISA and the Wegmann et al. (2022) embeddings on STEL and STEL-or-Content instances created from the test split of our SynthStel dataset. See Appendix G for full results.



the performance of StyleDistance embeddings against LISA and the Wegmann et al. (2022) style embeddings. On average, StyleDistance outperforms the other representations on AV, demonstrating its effectiveness in representing style.

Automatic Style Transfer EvaluationPatel et al. (2022) proposed the LUAR embedding model as an automatic measure for "style transfer accuracy". This approach was effective and was subsequently adopted for style transfer evaluation Liu et al. (2024); Horvitz et al. (2023, 2024). However, the LUAR model considers both style and content, hence confounding two aspects of style transfer evaluation--accuracy and meaning preservation--which are typically measured separately. Content-independent style representations would be a better measure for this task. We use the same evaluation dataset of 675 task instances used by Patel et al. (2022) where given an example text of a target author's style, the task is to discriminate which of two texts (a style transfer output and another actual text by the target author) is written by the target author. We show our results on this task in Table 7.StyleDistance proves to be a more effective discriminator than all models, including LUAR. All models surpass human performance in distinguishing style transfer outputs. Since automatic style transfer evaluation typically includes a separate score for meaning preservation, using a model like LUAD (which is not content-independent) for measuring style transfer accuracy undermines the rigor of the style transfer accuracy metric. We find that a robust content-independent model like StyleDistance may enhance automatic style transfer evaluation by: (1) acting as a stronger discriminator, and (2) ensuring style transfer accuracy is assessed independently of meaning preservation.

Style Transfer SteeringPrevious systems, like TinyStyler, have leveraged style embeddings to steer style transfer Horvitz et al. (2024). While the original TinyStyler system rewrites text by conditioning on Wegmann et al. (2022) embeddings, we demonstrate that StyleDistance provides an alternative, and reproduce TinyStyler with StyleDistance embeddings. We showcase an example of an output in Table 8 with more details and results in Appendix I. We will make this version of TinyStyler available as a resource. With this result, we demonstrate StyleDistance can be used as a simple drop-in replacement for downstream applications in systems where weaker style representations have been previously used.

## 6 Conclusion

We introduced StyleDistance, a novel method for training content-independent style embeddings using synthetic parallel examples. By employing a large language model to generate a dataset of near-exact paraphrases with controlled style variations, we overcome limitations associated with content leakage and imperfect parallel examples in existing style embedding methods. Evaluations using the STEL and STEL-or-Content tasks demonstrate that embeddings trained solely on synthetic examples can capture style extremely well. Additionally, the technique's ability to generalize to unseen style features indicates its potential to represent a broader range of style attributes beyond those addressed in synthetic data generation. Notably, our results highlight the efficiency of large language models in creating task-specific representations. This approach circumvents the need for manual dataset collection and the reliance on weak implicit assumptions over data, offering a more direct and accurate method for training on the precise task-specific features of interest.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Model** & **PAN’11** & **PAN’13** & **PAN’14** & **PAN’15** & **Avg** \\ \hline LISA & 0.55 & 0.66 & 0.43 & 0.64 & 0.57 \\ Wegmann et al. & 0.65 & 0.39 & 0.57 & 0.63 & 0.56 \\ (2022) & & 0.62 & 0.68 & 0.58 & 0.67 & **0.64** \\ \hline \hline \end{tabular}
\end{table}
Table 6: ROC-AUC results on the PAN 2011-2015 Authorship Verification (AV) shared tasks.

\begin{table}
\begin{tabular}{p{113.8pt} p{284.5pt}} \hline \hline
**Source Text (Informal)** & _“its keeping me up at nite, i_ _have to know what it is”_ \\ \hline Wegmann et al. (2022) (\(\rightarrow\)**Formal)** & _“Have to know what this is, keping me up at night.”_ \\
**StyleDistance** & _“What is it? It is keeping me up at night.”_ \\ \hline \hline \end{tabular}
\end{table}
Table 7: Accuracy results on style transfer evaluation.

 

Model, Data, and CodeWe release the StyleDistance models, the SynthStel dataset, and our code for other researchers to use at: [https://huggingface.co/StyleDistance/](https://huggingface.co/StyleDistance/).

### Limitations

Our approach shows strong results using 40 style features across 7 categories, though it does not fully cover the near-infinite range of possible style variations. The synthetic data may introduce some systematic biases, and we observe occasional repetitive patterns in the generated examples. Nonetheless, our method outperforms existing style representations, and we find that training on our selected 40 features offers strong generalization to unseen styles. Expanding this feature set could further enhance performance. While generating perfectly parallel examples for all style features is challenging--particularly for certain style features that may have overlap with content Jin et al. (2022)--our model effectively leverages synthetic data to improve content-independence. Additionally, our current focus on sentence-level generations leaves room for future work to explore varying text lengths and multi-sentence style variations, which could further strengthen our approach.

## Ethical Considerations

This work demonstrates the potential of synthetic data for enhancing style embeddings. However, it is important to recognize that the synthetic data generated by large language models may reflect and reinforce existing biases inherent in these models Patel et al. (2023). While our approach shows significant promise, ongoing efforts should ensure that such synthetic datasets are evaluated for fairness and bias to promote more equitable outcomes.

## Acknowledgements

This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200005. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

## References

* Argamon and Juola (2011) Shlomo Argamon and Patrick Juola. 2011. PAN11 Author Identification: Attribution. Zenodo. DOI: [https://doi.org/10.5281/zenodo.3713245](https://doi.org/10.5281/zenodo.3713245).
* Bird and Loper (2004) Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In _Proceedings of the ACL Interactive Poster and Demonstration Sessions_, pages 214-217, Barcelona, Spain. Association for Computational Linguistics.
* Dai et al. (2019) Ning Dai, Jianze Liang, Xipeng Qiu, and Xuan-Jing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5997-6007.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186.
* Fisher et al. (2024) Jillian Fisher, Skyler Hallinan, Ximing Lu, Mitchell Gordon, Zaid Harchaoui, and Yejin Choi. 2024. Styleremix: Interpretable authorship obfuscation via distillation and perturbation of style elements. _Preprint_, arXiv:2408.15666.
* Goldberger et al. (2004) Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. 2004. Neighbourhood components analysis. _Advances in neural information processing systems_, 17.
* Hay et al. (2020) Julien Hay, Bich-Lien Doan, Fabrice Popineau, and Ouassim Ait Elhara. 2020. Representation learning of writing style. In _Proceedings of the 6th Workshop on Noisy User-generated Text (W-NUT 2020)_.
* Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. _arXiv preprint arXiv:1503.02531_.
* Horvitz et al. (2023) Zachary Horvitz, Ajay Patel, Chris Callison-Burch, Zhou Yu, and Kathleen McKeown. 2023. Paraguide: Guided diffusion paraphrasers for plug-and-play textual style transfer. _ArXiv_, abs/2308.15459.
* Horvitz et al. (2024) Zachary Horvitz, Ajay Patel, Kanishk Singh, Chris Callison-Burch, Kathleen McKeown, and Zhou Yu. 2024. Tinrystler: Efficient few-shot text style transfer with authorship embeddings. _Preprint_, arXiv:2406.15586.
* Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. _Preprint_, arXiv:2106.09685.
* Jin et al. (2022) Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022. Deep Learning for Text Style Transfer: A Survey. _Computational Linguistics_, 48(1):155-205 

Patrick Juola and Efstathios Stamatatos. 2013. PAN13 Author Identification: Verification. Zenodo. DOI: [https://doi.org/10.5281/zenodo.3715998](https://doi.org/10.5281/zenodo.3715998).
* Kang and Hovy (2019) Dongyeop Kang and Eduard H. Hovy. 2019. xs-lue: A benchmark and analysis platform for cross-style language understanding and evaluation. _ArXiv_, abs/1911.03663.
* Khan et al. (2021) Aleem Khan, Elizabeth Fleming, Noah Schofield, Marcus Bishop, and Nicholas Andrews. 2021. A deep metric learning approach to account linking. _Preprint_, arXiv:2105.07263.
* Khan et al. (2023) Aleem Khan, Andrew Wang, Sophia Hager, and Nicholas Andrews. 2023. Learning to generate text in arbitrary writing styles. _arXiv preprint arXiv:2312.17242_.
* Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. _Advances in neural information processing systems_, 33:18661-18673.
* Kocher and Savoy (2017) Mirco Kocher and Jacques Savoy. 2017. A simple and efficient algorithm for authorship verification. _Journal of the Association for Information Science and Technology_, 68(1):259-269.
* Koppel and Winter (2014) Moshe Koppel and Yaron Winter. 2014. Determining if two documents are written by the same author. _Journal of the Association for Information Science and Technology_, 65(1):178-187.
* Krippendorff (2011) Klaus Krippendorff. 2011. Computing krippendorff's alpha-reliability.
* Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Sasko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clement Delangue, Theo Matsuisere, Lysandre Debutt, Stas Bekman, Pierric Cistate, Thibault Goehringer, Victor Mustara, Francois Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Liu et al. (2024) Shuai Liu, Shantanu Agarwal, and Jonathan May. 2024. Authorship style transfer with policy optimization. _Preprint_, arXiv:2403.08043.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Mangrukar et al. (2022) Sourbag Mangrukar, Sylvain Gugger, Lysandre Debutt, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Perf: State-of-the-art parameter-efficient fine-tuning methods. [https://github.com/huggingfa/ce/perft](https://github.com/huggingfa/ce/perft).
* McInnes et al. (2018) Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. 2018. Umap: Uniform manifold approximation and projection. _Journal of Open Source Software_, 3(29):861.
* Openi et al. (2020) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altaman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcon, Paul Baltcescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jakef Berdine, Gabriel Bernadet-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madeline Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Britany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Foits Cantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dolhan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyha Eloundou, David Farhi, Liam Fedus, Niko Felix, Simon Posada Fishman, Juston Forte, Isabella Fullford, Leo Gao, Eile Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijoo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heledecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoosechele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Hewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongijk Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Luc, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Wayne, Bob McGrew, Scott Mayer McKinnney, Christine McLeavey, Paul McMillan, Jake McNell, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnine Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mely, Ashvin Nair, Reichiro Nakano, Rajeeev Nayak, Arvind Neelakantan, Richard Ngo, 

Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipte de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokornry, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Raez, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Seslam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Syzmon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolovsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalic Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeeline B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Feclipe Ceron Uribe, Andrea Vallone, Arun Vijayeriya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zelters, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. Gpt-4 technical report. _Preprint_, arXiv:2303.08774.
* Patel et al. (2022) Ajay Patel, Nicholas Andrews, and Chris Callison-Burch. 2022. Low-resource authorship style transfer with in-context learning. _arXiv preprint arXiv:2212.08986_.
* Patel et al. (2024) Ajay Patel, Colin Raffel, and Chris Callison-Burch. 2024. DataDreamer: A tool for synthetic data generation and reproducible LLM workflows. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 3781-3799, Bangokok, Thailand. Association for Computational Linguistics.
* Patel et al. (2023) Ajay Patel, Delip Rao, Ansh Kothyary, Kathleen McKeewn, and Chris Callison-Burch. 2023. Learning interpretable style embeddings via prompting LLMs. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 15270-15290, Singapore. Association for Computational Linguistics.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21:1-6.
* Rao and Tetreault (2018) Sudha Rao and Joel Tetreault. 2018. Dear sir or madam, may I introduce the GYAFC dataset: Corpus, benchmarks and metrics for formality style transfer. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Volume 1 (Long Papers)_, pages 129-140, New Orleans, Louisiana. Association for Computational Linguistics.
* Reif et al. (2022) Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coonen, Chris Callison-Burch, and Jason Wei. 2022. A recipe for arbitrary text style transfer with large language models. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 837-848, Dublin, Ireland. Association for Computational Linguistics.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3982-3992.
* Riley et al. (2021) Parker Riley, Noah Constant, Mandy Guo, Girish Kumar, David C Uthus, and Zarana Parekh. 2021. Textsettr: Few-shot text style extraction and tunable targeted restlying. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3786-3800.
* Rivera-Soto et al. (2021) Rafael A Rivera-Soto, Olivia Elizabeth Miano, Juanita Ordonez, Barry Y Chen, Aleem Khan, Marcus Bishop, and Nicholas Andrews. 2021. Learning universal authorship representations. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 913-919.
* Saakyan and Muresan (2023) Arkadiya Saakyan and Smaranda Muresan. 2023. Iclef: In-context learning with expert feedback for explainable style transfer. _ArXiv_, abs/2309.08583.
* Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 815-823.
* Stamatatos et al. (2014) Efstathios Stamatatos, Walter Daelemans, Ben Verhoeven, Martin Potthast, Benno Stein, Patrick Juola, Miguel A. Sanchez-Perez, and Alberto Barron-Cedeno. 2014. PAN14 Author Identification: Verification. Zenodo. DOI: [https://doi.org/10.5281/zenodo.3716032](https://doi.org/10.5281/zenodo.3716032).
* Stamatatos et al. (2015) Efstathios Stamatatos, Walter Daelemans Daelemans amd Ben Verhoeven, Patrick Juola, Aurelio Lopez-Lopez, Martin Potthast, and Benno Stein. 2015. PAN15 Author Identification: Verification. Zenodo. DOI: [https://doi.org/10.5281/zenodo.3737563](https://doi.org/10.5281/zenodo.3737563).
* Suzgun et al. (2022) Mirac Suzgun, Luke Melas-Kyriazi, and Dan Jurafsky. 2022. Prompt-and-rerank: A method for zero-shot and few-shot arbitrary textual style transfer with 

small language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, pp. 2195-2222. External Links: Link Cited by: SS4.
* Y. R Tausczik and J. W Pennebaker (2010)The psychological meaning of words: liwc and computerized text analysis methods. Journal of language and social psychology29 (1), pp. 24-54. Cited by: SS3.
* A. Warstadt, A. Singh, and S. R Bowman (2019)Neural network acceptability judgments. Transactions of the Association for Computational Linguistics7, pp. 625-641. Cited by: SS4.
* A. Wegmann and D. Nguyen (2021)Does it capture stel? a modular, similarity-based linguistic style evaluation framework. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7109-7130. Cited by: SS4.
* A. Wegmann, M. Schraagen, D. Nguyen, et al. (2022)Same author or just same topic? towards content-independent style representations. In Proceedings of the 7th Workshop on Representation Learning for NLP, pp. 249. Cited by: SS4.
* T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew (2019)Huggingface's transformers: state-of-the-art natural language processing. CoRRabs/1910.03771. Cited by: SS4.
* Y. Yang, M. Gandhi, Y. Wang, Y. Wu, M. S. Yao, C. Callison-Burch, J. C. Gee, and M. Yatskar (2024)A textbook remedy for domain shifts: knowledge priors for medical image analysis.. arXiv preprint arXiv:2405.14839. Cited by: SS4.
* Y. Yu, Y. Zhuang, J. Zhang, Y. Meng, A. J. Ratner, R. Krishna, J. Shen, and C. Zhang (2023)Large language model as attributed training data generator: a tale of diversity and bias. In Advances in Neural Information Processing Systems, Vol. 36, pp. 55734-55784. Cited by: SS4.
* K. Zhu, Z. Tian, R. Luo, and X. Mao (2022)StyleFlow: disentangle latent representations via normalizing flow for unsupervised text style transfer. In International Conference on Language Resources and Evaluation, 

Style Features and Definitions

We list all style features selected for our synthetic dataset below along with the positive and negative prompts (used for constructing a full prompt for generating positive and negative examples as shown in Appendix B.2) and definitions (used to help define the style feature to human annotators in the annotation interface in Appendix C).

 

* [108] M. C. B. K., and R. A. S. G. Wilson, "Theoretical Methods for the Study of Computer Science," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 235-246, 1979.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l l p{142.3pt}} \hline
**Sole Feature Name** & **Positive and Negative Prompts** & **Sole Feature Definition** \\ \hline Usage of Self-Focused Language & Positive: Self-focused & The “Usage of Self-Focused Language” test style feature refers to the use of words or phrase that focus on the speaker’s or writer themselves. This includes the use of first-person processing time; ” ”, ”, ”, ”, ”, ”, ”, and ” myself. \\  & \\ Uage of Certain Tree & Positive: Certain & This text style feature refers to the use of a conflict tone in writing, where the author avoids using uncertain words or phrases such as ’think’, ‘might’, or ‘scosm”. This results is a textual theme more assertive and arise of the information being presented. \\ Uage of Present-Focused Tense and Words & Positive: Present-focused & The “Usage of Presence: Focused Tense and Word" test style feature refers to the use of both the present content and work from the current moment or situation. This means the text is primarily discussing events, actions, or states that are happening right now or generally true. \\ Uage of Suracam & Positive: With sarcem & The “Usage of Stream” test style feature refers to the presence of statements or expressions in the text than the opposite of what they literally stop, often used to mock or show irritation. This style is often characterized by irony, indiccide, or smokers, and is not express countertop or to criticize something or someone in a humorous way. \\ Uage of Self-Focused Perspective or Words & Positive: Self-focused & The “Usage of Self-Focused Experience or Work” test style feature refers to the use of words or phrases that focus on the speaker’s owntherness. This include the use of first-person pronouns like ”, ”mee', ”myr’, ”name’, and “nsgiver!, or statements that express the speaker’s personal thought, feelings, or experiences. \\ Frequent Usage of Paucutation & Positive: With frequent punctuation & The text style feature “Frrequent Usage of Punctuations” refers to the regular and substitute of punctuation marks such as commons, periods, explanation points, question marks, etc., in a piece of text. This style feature is present when the writer often uses these symbols to structure their sentences, express emotions, or emphasize certain points. \\ Uage of Public Tone & Positive: Police & The “Usage of Public Tone” test style feature refers to the use of respectful and consider language in a text. This can include using word like “_please_”, that you?, we phrases thus allow reference to report for the reader. It’s also making the text sound comments and interpret, rather than demanding or rule. \\ Uage of Contractions & Positive: With contractions & The “Usage of Cattations” test style feature refers to the use of shortened terms of words or phrases in a text. These are typically formed by omitting context items relevant or sounds and replacing them with an assumption, such as “a “b/or “d/o” or “f/o” or “f/o”. If a text frequently uses such shortened forms, it has this style feature. \\ Frequent Usage of Determines & Positive: With determiners & The text style feature “Trrequent Usage of Determinants” refers to the regular use of words that introduce nouns and one key infenormation about its quantity, premiums, definitions, or “free wordsirable” (‘v’, “vi''), “the”, “these’, “my’, ‘year’, “ls’, here’, “in’, “var’, “their”. If a text often uses such words, it has this style feature. \\ \hline \end{tabular}
\end{table}
Table 9: The style features selected for synthetic data generation in this work.



Generation Prompts and Details

Below we detail the structure of our prompts and inference parameters used for synthetic data generation.

### Extracting Topics from C4

We select random sentences from C4 (Raffel et al., 2020), and extract the fine-grained topic of the sentence using GPT-4 (OpenAI et al., 2024) and the zero-shot prompt shown below. We perform sampling with temperature = 1.0 and top_p = 0.0.

What is the fine-grained topic of the following text: (sentence) Only return the topic.

The fine-grained topic is then used as part of the attributed prompt described in Section B.2 to ensure diversity in the generations.

### Generating Positive and Negative Example Sentences for Each Style Feature

For each style feature, we generate positive and negative parallel examples using a zero-shot prompt and the attributed prompt (AttrPrompt) method (Yu et al., 2023) with GPT-4 to create diverse and realistic synthetic examples. To generate many examples per style feature, we use a new unused topic extracted from C4 in the prompt each time, we randomly sample a new permutation of the attributes in the prompt, and we perform sampling with temperature = 1.0 and top_p = 1.0. We demonstrate an example below for the "Usage of Active Voice" style feature.

Generate a pair of active and passive sentences with the following attributes:  1. Topic: (topic)  2. Length: (sentence_length)  3. Point of view: (point_of_view)  4. Tense: (tense)  5. Type of Sentence: (sentence_type) Ensure that the generated sentences meet the following conditions:  1. There is no extra information in one sentence that is not in the other.  2. The difference between the two sentences is subtle.  3. The two sentences have the same length.  (special_conditions_for_style_feature) Use Format:  Active: [sentence] Passive: [sentence] Your response should only consist of the two sentences, without quotation marks. For the exact prompts for each style feature, see the code in our supplementary materials for this work.

 

Human Annotation Details

We provide an example of a task instance in our annotation interface. Human annotators were asked to rate whether the style feature was present or not in the sentence, with the option to also select "Possibly" if the annotator was unsure (instructed to use sparingly). We provide annotators with a definition of each style feature as well.

We used a population of graduate students taking a class on natural language processing as the annotators. Each task instance was annotated by 10 distinct human annotators. We assign a score of 0 to "No", 0.5 to "Possibly", and 1 to "Yes". We average the scores from all 10 annotators assigned to each task instance. We consider to have agreement for a positive example if the average score is >=0.5, and for a negative example if the average score is < 0.5.

We measure inter-annotator agreement using Krippendorf's Alpha [26] which indicates moderate agreement of 0.55. As a more easily interpretable measure of agreement between annotators, for each task instance, we also find, on average, around 8 out of the 10 annotators annotated in agreement on whether a style feature was present or not in the text.

Figure 4: The annotation interface used for human annotation.

 



## Appendix D Training Details

More exact training details can be found in the source code provided in the supplementary materials for this work.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Hyperparameter** & **Value** \\ \hline Model & Facebook/roberta-base \\ Hardware & 4x or 8x NVIDIA RTX A6000 \\ Distributed Protocol & PyTorch FSDP \\ Data Type & torch.bfloat16 \\ Loss Function & TripletLoss (Schroff et al., 2015) \\ Triplet Loss Margin & 0.1 \\ LoRA (Hu et al., 2021) & all-linear, r=8 \\ Optimizer & lora_alpha=8 \\ Learning Rate & le-4 \\ Weight Decay & 0.01 \\ Learning Rate Scheduler & linear \\ Warmup Steps & 0 \\ Batch Size & 512 \\ Train-Validation Split & 90/10\% \\ Early Stopping Threshold & 0.0 \\ Early Stopping Patience & 1 epoch \\ \hline \hline \end{tabular}
\end{table}
Table 10: Hyperparameters selected for contrastive learning training experiments.



Visualization of StyleDistance Embedding Space

We compare the embedding space of Wegmann et al. (2022) and StyleDistance on informal/formal texts from GYAFC6Rao and Tetreault, 2018) in Figure 5 below.

Footnote 6: Grammarly’s Yahoo Answers Formality Corpus which contains 110K informal/formal sentence pairs.

Figure 5: UMAP visualizations of style embeddings from Wegmann et al. (2022) and StyleDistance on \(n=100\) random parallel formal/informal examples. Wegmann et al. (2022) forms two distinct clusters of informal texts, making many informal examples distant in the embedding space despite sharing the same style.

 

STEL and STEL-or-Content Task Visualization

Figure 6: A visualization of the STEL and STEL-or-Content task evaluation we describe in Section 5.1.

 



## Appendix G STEL and STEL-or-Content Results on SynthStel

\begin{table}
\begin{tabular}{l c c c c|c c c c} \hline \hline

**Average** & 0.79 & 0.06 & 0.76 & 0.25 & 0.88 & 0.97 & 0.74 & 0.89 \\ \hline \hline \end{tabular}
\end{table}
Table 11: STEL and STEL-or-Content results for top-performing models on the SynthStel test split. This table shows the performance variations and coverage of LISA and Wegmann et al. (2022) embeddings for the 40 different style features found in the dataset. After training StyleDistance models on the SynthStel train split, we observe strong coverage of these 40 style features as expected, demonstrating the successful distillation of the LLM’s strong style knowledge into a more efficient representation model Hinton et al. (2015).



Style Feature Ablation Details

For the generalization experiment and ablation results we demonstrate in Table 4, we list the style features ablated (removed from the training data) for the Out-of-Domain and Out-of-Distribution conditions.

**Out-of-Domain**:

1. Usage of Formal Tone
2. Usage of Contractions
3. Usage of Numerical Substitution
4. Complex Sentence Structure
5. Usage of Text Emojis
6. Usage of Emojis

**Out-of-Distribution**:

1. Usage of Formal Tone
2. Usage of Polite Tone
3. Fluency in Sentence Construction
4. Usage of Only Uppercase Letters
5. Usage of Only Lowercase Letters
6. Incorporation of Humor
7. Usage of Sarcasm
8. Usage of Contractions
9. Usage of Numerical Substitution
10. Usage of Numerical Digits
11. Complex Sentence Structure
12. Usage of Long Words
13. Usage of Lost Emojis
14. Usage of Emojis
15. Presence of Misspelled Words 

Style Transfer Performance

TinyStyler is a style transfer system that reconstructs texts by conditioning on style embeddings Horvitz et al. (2024). The original system is trained on the Reddit Million User Dataset (MUD) Khan et al. (2021) with a **four step** procedure:

1. A modified T5 model Raffel et al. (2020) is trained to reconstruct texts from paraphrases and style embeddings (Wegmann).
2. This unsupervised model is then used to generate style transfer outputs between multiple authors from the original corpus.
3. The resulting synthetic data is then filtered using style embedding (Wegmann) distance and meaning preservation metrics.
4. Finally, a model is then fine-tuned on the resulting filtered dataset.

We reproduce the TinyStyler procedure with the exact dataset and hyperparameters in the original paper. We make only one modification: replacing Wegmann embeddings with StyleDistance in the generation and filtering steps. We include the formality transfer evaluation results in Table 12. In these automatic evaluations, the StyleDistance conditioned model performs comparably. We additionally include examples comparing model output in Table 13.

\begin{table}
\begin{tabular}{l l c} \hline \hline
**Source Text (Informal)** & **Wegmann (\(\rightarrow\) Formal)** & **StyleDistance (\(\rightarrow\) Formal)** \\ \hline _“its keeping me up at nite, i have to know what it is”_ & _“Have to know what this is, keeping me up at night.”_ & _"What is it? It is keeping me up at night.”_ \\ _“i like journey’s open arms... but i like marial carey’s version of blurney, Open Arms.”_ & _ “I think marial carey’s version of that song is better.”_ \\ _“And you can’t rely on rumors.”_ & _“You can never trust rumors about it.”_ & _“I would say that you can’t rely on rumors.”_ \\ \hline
**Source Text (Formal)** & **Wegmann (\(\rightarrow\) Informal)** & **StyleDistance (\(\rightarrow\) Informal)** \\ \hline _“If favor the man as he is humorous and grounded.”_ & _“i love tahm as he is humorous and grounded...”_ & _“i'm gonna go with the man...hes humorous and grounded.”_ \\ _“I am sure you will both enjoy it.”_ & _“ok i am sure you both will enjoy it.”_ & _“oh yea...you’ll both enjoy it”_ \\ _“I very much enjoy this song.”_ & _“i love this song so much, i’m a fan of it.”_ & _“i like this song so much man...”_ \\ \hline \hline \end{tabular}
\end{table}
Table 13: Outputs from TinyStyler, conditioned on Wegmann embeddings and StyleDistance embeddings.

 

Resources

We provide links and citations to resources used in this paper which provide license information, documentation, and their intended use. Our usage follows the intended usage of all resources.

#### We utilize the following models:

* GPT-4 (full model, accessed June, 2024) (OpenAI et al., 2024)
* RoBERTa (roberta-base) (Liu et al., 2019; Devlin et al., 2019; Reimers and Gurevych, 2019)
* Learning Universal Authorship Representations (LUAR) Embedding model (Rivera-Soto et al., 2021)
* Style embedding model from Wegmann et al. (2022)
* LISA model (Patel et al., 2023)
* CoLA model (textattack/roberta-base-CoLA) (Warstadt et al., 2019)
* all-mpnet-base-v2 Sentence Similarity model (Reimers and Gurevych, 2019) ([https://huggingface.co/sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2))

We utilize the following datasets and resources:

* STEL dataset (Wegmann and Nguyen, 2021)
* Contrastive Authorship Verification dataset (Wegmann et al., 2022)
* C4 (Raffel et al., 2020)
* LIWC (Tausczik and Pennebaker, 2010)
* 2015 Authorship Verification Datasets (Argamon and Juola, 2011; Juola and Stamatatos, 2013; Stamatatos et al., 2014, 2015)
- Curated at: [https://huggingface.co/datasets/swan07/authorship-verification](https://huggingface.co/datasets/swan07/authorship-verification)
* GYAFC (Rao and Tetreault, 2018)

We utilize the following software:

* DataDreamer (Patel et al., 2024)
* Transformers (Wolf et al., 2019)
* Datasets (Lhoest et al., 2021)
* PEFT (Mangrulkar et al., 2022)
* Sentence-Transformers (Reimers and Gurevych, 2019)
* NLTK (Bird and Loper, 2004)

We estimate the total compute budget and detail computing infrastructure used to run the computational experiments found in this paper below:

* 80 hours