mStyleDistance: Multilingual Style Embeddings and their Evaluation
Authors: Justin Qiu, Jiacheng Zhu, Ajay Patel, Marianna Apidianaki, Chris Callison-Burch
Accepted at: acl25

mSTYLEDISTANCE: Multilingual Style Embeddings and their Evaluation

Justin Qiu*    Jiacheng Zhu*    Ajay Patel
Marianna Apidianaki    Chris Callison-Burch
University of Pennsylvania
{jsq, jiachzhu, ajayp, marapi, ccb}@seas.upenn.edu

Abstract

Style embeddings are useful for stylistic analysis and style transfer; however, only English style embeddings have been made available. We introduce Multilingual STYLEDISTANCE (mSTYLEDISTANCE), a multilingual style embedding model trained using synthetic data and contrastive learning. We train the model on data from nine languages and create a multilingual STEL-or-Content benchmark (Wegmann et al., 2022) that serves to assess the embeddings’ quality. We also employ our embeddings in an authorship verification task involving different languages. Our results show that mSTYLEDISTANCE embeddings outperform existing models on these multilingual style benchmarks and generalize well to unseen features and languages. We make our model publicly available at https://huggingface.co/StyleDistance/mstyledistance.

1 Introduction

Style embedding models seek to embed texts with similar style closer in the embedding space regardless of their content. Style embeddings are useful for tasks like style transfer and authorship attribution, but only exist for English (Wegmann et al., 2022; Patel et al., 2024b). Multilingual style embeddings could also serve to automatically evaluate style preservation in machine translation. Models like XLM-RoBERTa (Conneau et al., 2019) and ES (Wang et al., 2024) create multilingual representations for semantic tasks, but have not addressed style mainly due to the scarcity of style datasets.

We propose a procedure, called multilingual STYLEDISTANCE (mSTYLEDISTANCE), to train style embeddings using contrastive learning with synthetic data in multiple languages. Early work on style representations learning often involved unlabeled social media data (Hay et al., 2020; Wegmann et al., 2022; Rivera-Soto et al., 2021; Patel et al., 2023), but Patel et al. (2024b) showed that a contrastive learning objective with synthetic examples (sentence pairs with similar content and different style) can generate high quality style representations for English. We create mSYNTHSTEL, a synthetic dataset of paraphrases addressing various style features in nine languages, and use it to create our multilingual style embeddings.

In order to evaluate their quality, we contribute a new multilingual and cross-lingual STEL-or-Content (SoC) evaluation benchmark which, following the original SoC evaluation task (Wegmann et al., 2022), measures the ability of a model to embed sentences with the same style closer in the embedding space than sentences with the same

Figure 1: mSTYLEDISTANCE is trained using contrastive learning with synthetic examples created for ~40 style features in 9 languages. These positive and negative examples serve to form multilingual and cross-lingual training triplets.
content. We show that mSTYLEDISTANCE embeddings outperform other representations in these evaluations, and demonstrate their usefulness in a downstream setting addressing a multilingual authorship verification task. We publicly release our model, data, and evaluation benchmarks.

2 Multilingual Synthetic Data

We extend the Patel et al. (2024b) dataset to nine languages (\( L \)): Arabic, German, Spanish, French, Hindi, Japanese, Korean, Russian, and Chinese.\footnote{These languages were chosen because they align with the linguistic background of our annotators.}

Style Feature Selection  We use the set of 40 features addressed in Patel et al. (2024b), leaving out features not applicable to specific languages. For example, articles are not relevant for Chinese and Japanese so the corresponding features have been discarded. Our set of features (\( F \)) includes syntactic features (e.g., active/passive voice, contractions, frequent use of function words), emotional and cognitive features (e.g., words indicating sentiment or cognitive processes), stylistic and aesthetic features (e.g., metaphors, formal tone), social and interpersonal features (e.g., polite or offensive tone), graphical and digital features (e.g., capitalization, emojis, numerical digits), temporal and aspectual features (e.g., focus on present or future). A full list of the features that were considered for each language and details about the features that were omitted are given in Appendix A.

Data Generation  For each retained feature \( f \in F \) for a language \( l \in L \), we generate 100 pairs of positive (pos) and negative (neg) examples (paraphrases). In each pair, pos is a sentence that contains the style feature (e.g., a formal sentence or a metaphorical one) while neg does not. This is illustrated in Figure 1 for the feature “Active Voice”. Features that cannot be removed completely (e.g., “usage of articles”) are present with higher frequency in pos than in neg examples.

Using the same prompting workflow as Patel et al. (2024b), we generate sentence pairs by prompting GPT-4 with the DataDreamer library and an attributed prompt (Yu et al., 2023) illustrated in Figure 2 (OpenAI et al., 2024; Patel et al., 2024a). For diversity, a “Topic” for each generation is sampled by extracting a random sentence from a random document in the C4 corpus (Raffel et al., 2020), and prompting GPT-4 to identify the topic. Our training dataset also includes English-only data from SYNTHSTEL (Patel et al., 2024b). For further details on all prompts, see Appendix B.

We experiment with two approaches to multilingual data generation. In our first approach, sentence pairs are directly generated in each \( l \in L \) using a language-specific instruction in the prompt, as illustrated in Figure 2. In our second approach, English sentence pairs are generated using the prompting workflow and then translated into each target language \( l \in L \). We use DeepL for all languages except for Hindi which is not supported, where we use Google Translate instead. We generate data using both methods and conduct human validation on a random 10% split of the training data in order to determine the best approach for each language.\footnote{Our annotators were undergraduate and graduate students native speakers of a language, who were offered extra credit for participation.}

<table>
  <tr>
    <th colspan="2">Generate a pair of active and passive Russian sentences with the following attributes:</th>
  </tr>
  <tr>
    <td>1.</td>
    <td>Topic: Brake parts and components</td>
  </tr>
  <tr>
    <td>2.</td>
    <td>Length: 15-20 words</td>
  </tr>
  <tr>
    <td>3.</td>
    <td>Point of view: second-person</td>
  </tr>
  <tr>
    <td>4.</td>
    <td>Tense: past</td>
  </tr>
  <tr>
    <td>5.</td>
    <td>Type of Sentence: Exclamation</td>
  </tr>
</table>

Figure 2: Example prompt for generating a pair of sentences in Russian.

Human validation  Each sentence pair for a style feature creates two annotation instances, one for the positive and one for the negative sentence. For each instance, we asked the annotators to judge whether the style feature is present in the sentence by selecting an answer among “Yes”, “Possibly”, and “No”. We also provided a definition for each style feature to help annotators in their decision. We then asked the annotators to rate the fluency of the sentence by selecting one among “Fluent”, “Mostly Fluent”, “Mostly Disfluent”, “Disfluent”. We collected a total of 13,651 annotations for the 9 languages in \( L \). When considering only instances annotated by at least three annotators, the inter-annotator agreement was \( \alpha = 0.4247 \) (Krippendorff, 2011). More details on the annotation task and the exact number of collected annotations per language can be found in Appendix C.

We calculate an aggregate “feature presence” accuracy score for each \( l \in L \) based on whether the average feature presence score over all annotations
is higher for the positive sentence than for the negative sentence in a pair. We assign a score of 1 to “Yes”, 0.5 to “Possibly”, and 0 to “No”. We also calculate an aggregate fluency score by taking an average of the fluency scores that each annotator gave to each text. We assign a score of 1 to “Fluent”, 0.67 to “Mostly Fluent”, 0.33 to “Mostly Disfluent”, and 0 to “Disfluent”. We selected the best approach (direct generation v.s. English → MT) for each \( l \in L \) as the one that produced the most fluent sentences, or the highest feature presence score if both methods produced similarly fluent sentences. The direct approach was selected for all languages in \( L \) except for Japanese and Hindi, where the translation approach produced higher quality data. Our final average feature presence and fluency scores over all \( l \in L \), with the best generation approach selected for each \( l \), are 0.79 and 0.93, respectively, both above random chance (0.5). Detailed results by language are given in Appendix D.

Automatic Data Validation Following Patel et al. (2024b), we also perform automatic validations of the generated data. We examine whether our parallel examples are indeed paraphrases by computing their average cosine similarity\(^3\) (Reimers and Gurevych, 2019). For comparison, we calculate the similarity of 100 gold-standard paraphrases sampled from the multilingual dataset compiled by Scherrer (2020) for each language. The average similarity of our parallel examples is 0.91 which is comparable to that calculated on the Scherrer (2020) natural data (0.88), indicating that our pairs are reasonable paraphrases.

We also measure topic diversity across generated sentences for a \( l \in L \) using the metric proposed by Yang et al. (2024) which relies on cosine distance. In this case, we only use the pos sentence which contains the style feature for each pair. For comparison, we also compute the diversity score for texts from Scherrer (2020). Again, the two scores are comparable (0.83 vs. 0.85), showing that our examples are similar to natural data in terms of diversity. Detailed results of the automatic evaluations are given in Appendix D.

3 Training mStyleDistance

Following the contrastive training approach of Patel et al. (2024b), we construct feature-specific triples for each language \( l \in L \) which contain: an anchor text (\( a \)); a text with the same style as \( a \) but different content (pos); a distractor text (neg) which is a paraphrase of \( a \) or pos, but different in style from \( a \). We use the multilingual xlm-roberta-base as our base model and train with a triplet loss (Conneau et al., 2019; Schroff et al., 2015). We ensure half of our triplets are cross-lingual, i.e. the pos and neg texts are randomly sampled from a different language than the anchor text. Our triplet creation process ensures equal coverage of the languages we support. Full training details can be found in Appendix E.

4 Evaluation

STEL-or-Content (SoC) Benchmark In order to evaluate our style embeddings, we construct a multilingual version of the SoC benchmark (Wegmann et al., 2022).\(^4\) SoC measures the ability of a model to embed sentences with the same style closer in the embedding space than sentences with the same content. We construct our multilingual SoC benchmark by sampling 100 pairs of parallel pos-neg examples for each language from four ground-truth datasets covering four style features and 22 languages: simplicity (Ryan et al., 2023), formality (Briakou et al., 2021), toxicity (Dementieva et al., 2024), and positivity (Mukherjee et al., 2024).\(^5\) Each instance in our multilingual SoC benchmark consists of a triplet (\( a, \) pos, neg) constructed as explained in Section 3. However, following Wegmann et al. (2022), the distractor text in our SoC benchmark is always a paraphrase of pos. A model tested on this benchmark is expected to embed \( a \) and pos closer than \( a \) and neg. We rate a model by computing the percentage of instances it achieves this goal for across all instances. We form test instances for each feature \( f \in F \) available for a language corresponding to all possible triplets, resulting in 4,950 instances for each language-feature style combination.

We also construct a cross-lingual SoC benchmark that addresses embeddings’ ability to capture style similarity *across languages*. This can be useful, for example, to evaluate style preservation in translations. We construct the benchmark with the XFormal dataset (Briakou et al., 2021),

\(^3\)We use paraphrase-multilingual-mpnet-base-v2.

\(^4\)The English SoC benchmark covered formality, complexity, number usage, contraction usage, and emoji usage.

\(^5\)Combined, these datasets cover the following languages: Amharic, Arabic, Bengali, German, English, Spanish, French, Hindi, Italian, Japanese, Magahi, Malayalam, Marathi, Odia, Punjabi, Portuguese (Brazil), Russian, Slovenian, Telugu, Ukrainian, Urdu, and Chinese.
<table>
  <tr>
    <th>Model</th>
    <th colspan="2">Test Set</th>
    <th>Simplicity</th>
    <th>Formality</th>
    <th>Toxicity</th>
    <th>Positivity</th>
    <th>Formality (cross-lingual)</th>
  </tr>
  <tr>
    <td><b>Wegmann et al. (2022)</b></td>
    <td>Original</td>
    <td></td>
    <td>0.23</td>
    <td>0.63</td>
    <td>0.19</td>
    <td>0.23</td>
    <td>0.45</td>
  </tr>
  <tr>
    <td><b>STYLEDISTANCE</b></td>
    <td>Original</td>
    <td></td>
    <td>0.21</td>
    <td>0.67</td>
    <td>0.15</td>
    <td>0.18</td>
    <td>0.49</td>
  </tr>
  <tr>
    <td>xlm-roberta-base</td>
    <td>Original</td>
    <td></td>
    <td>0.12</td>
    <td>0.16</td>
    <td>0.09</td>
    <td>0.07</td>
    <td>0.19</td>
  </tr>
  <tr>
    <td>LISA</td>
    <td>Original</td>
    <td></td>
    <td>0.15</td>
    <td>0.09</td>
    <td>0.09</td>
    <td>0.21</td>
    <td>0.27</td>
  </tr>
  <tr>
    <td><b>mSTYLEDISTANCE</b></td>
    <td>Original</td>
    <td></td>
    <td><b>0.36</b></td>
    <td><b>0.71</b></td>
    <td><b>0.37</b></td>
    <td><b>0.30</b></td>
    <td><b>0.53</b></td>
  </tr>
  <tr>
    <td><b>Wegmann et al. (2022)</b></td>
    <td>Translated</td>
    <td></td>
    <td>0.25</td>
    <td>0.50</td>
    <td>0.21</td>
    <td>0.19</td>
    <td>0.36</td>
  </tr>
  <tr>
    <td><b>STYLEDISTANCE</b></td>
    <td>Translated</td>
    <td></td>
    <td>0.25</td>
    <td>0.44</td>
    <td>0.23</td>
    <td>0.27</td>
    <td>0.35</td>
  </tr>
  <tr>
    <td>xlm-roberta-base</td>
    <td>Translated</td>
    <td></td>
    <td>0.12</td>
    <td>0.07</td>
    <td>0.08</td>
    <td>0.07</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>LISA</td>
    <td>Translated</td>
    <td></td>
    <td>0.04</td>
    <td>0.02</td>
    <td>0.07</td>
    <td>0.13</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td><b>mSTYLEDISTANCE</b></td>
    <td>Translated</td>
    <td></td>
    <td>0.33</td>
    <td>0.51</td>
    <td><b>0.41</b></td>
    <td>0.28</td>
    <td>0.43</td>
  </tr>
</table>

Table 1: Performance of multilingual and English embeddings on the original multilingual and cross-lingual SoC benchmarks (rows 1-5), averaged across languages for each style feature. mSTYLEDISTANCE leads in cross-lingual and overall performance. For comparison, we also report results obtained on the test set translated into English (rows 6-10).

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">PAN 2013</th>
    <th colspan="2">PAN 2014</th>
    <th colspan="2">PAN 2015</th>
    <th colspan="2">Average</th>
  </tr>
  <tr>
    <th>Greek</th>
    <th>Spanish</th>
    <th>Greek</th>
    <th>Spanish</th>
    <th>Dutch</th>
    <th>Greek</th>
    <th>Spanish</th>
    <th>Dutch</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td><b>Wegmann et al. (2022)</b></td>
    <td>0.66</td>
    <td>0.87</td>
    <td>0.56</td>
    <td>0.54</td>
    <td>0.59</td>
    <td>0.47</td>
    <td>0.61</td>
    <td>0.59</td>
    <td>0.56</td>
    <td>0.67</td>
    <td>0.59</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td><b>STYLEDISTANCE</b></td>
    <td>0.61</td>
    <td>0.62</td>
    <td>0.48</td>
    <td>0.51</td>
    <td>0.65</td>
    <td>0.47</td>
    <td>0.73</td>
    <td>0.59</td>
    <td>0.52</td>
    <td>0.62</td>
    <td>0.62</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>LISA</td>
    <td>0.51</td>
    <td>0.64</td>
    <td>0.46</td>
    <td>0.56</td>
    <td>0.62</td>
    <td>0.48</td>
    <td>0.66</td>
    <td>0.48</td>
    <td>0.48</td>
    <td>0.62</td>
    <td>0.55</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td><b>mSTYLEDISTANCE</b></td>
    <td>0.48</td>
    <td>0.78</td>
    <td>0.71</td>
    <td>0.68</td>
    <td>0.72</td>
    <td>0.73</td>
    <td>0.74</td>
    <td>0.48</td>
    <td><b>0.64</b></td>
    <td><b>0.73</b></td>
    <td><b>0.60</b></td>
    <td><b>0.66</b></td>
  </tr>
</table>

Table 2: Results on the PAN 2013-2015 Authorship Verification shared task for Greek, Spanish, and Dutch. We report performance separately on each PAN dataset and average performance across datasets for the same language. We use the standard ROC-AUC metric for authorship verification.

which includes parallel data in French, Italian and Portuguese. We again create triplets as described above, but instead of using pos and neg texts from the same language as the anchor (\( a \)), we sample them from a different language than \( a \). We end up with 19,800 instances for each style in each language. Appendix F contains illustrative examples from each benchmark.

SoC Evaluation Results The results obtained by mSTYLEDISTANCE on the multilingual and cross-lingual SoC benchmarks are presented in Table 1. Since no general multilingual style embeddings are currently available, we compare with a base multilingual encoder model xlm-roberta-base (Conneau et al., 2019) as well as with a number of English-trained style embedding models applied in zero-shot fashion to multilingual text: the Wegmann et al. (2022) style embeddings, STYLEDISTANCE embeddings (Patel et al., 2024b), and LISA embeddings (Patel et al., 2023). The results of the models on the original cross-lingual and multilingual test set are given in the top part of Table 1 (rows 1-5). mSTYLEDISTANCE embeddings outperform the other models on this dataset, indicating their suitability for multilingual applications. The other models perform slightly better than the untrained xlm-roberta-base but still worse than mSTYLEDISTANCE.

We also compare mSTYLEDISTANCE with the English-only models on test data translated into English.\footnote{We use DeepL for languages it supports and Google Translate for the other languages.} The results on this translated test set, which are given in the lower part of Table 1 (rows 6 to 10), are often lower than those obtained on the original multilingual test set. This might look surprising but can be explained by the inherent limitations of machine translation, which can cause certain stylistic nuances to be diminished or lost during the translation process. This is illustrated in the example below where the Chinese formal (您) and informal (你) pronouns are both translated as “you” in English, and the original formality nuances are not present in the translation.

Formal: 您是否对自己过去对于膳食补充剂立场的批评有所回应?
Translation: Have you responded to criticisms of your past position on dietary supplements?
Informal: 你回应过对你过去膳食补充剂立场的批评了吗?
Translation: Have you responded to the criticisms of your past stance on dietary supplements?

Breakdown of results by feature and language in
<table>
  <tr>
    <th>Features Tested</th>
    <th>m avg</th>
    <th>c avg</th>
    <th colspan="2">Retained Perf (%)</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th>m</th>
    <th>c</th>
  </tr>
  <tr>
    <td>In-Domain</td>
    <td>0.38</td>
    <td>0.53</td>
    <td>100%</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>Out of Domain</td>
    <td>0.31</td>
    <td>0.44</td>
    <td>75%</td>
    <td>74%</td>
  </tr>
  <tr>
    <td>Out of Distribution</td>
    <td>0.31</td>
    <td>0.40</td>
    <td>75%</td>
    <td>62%</td>
  </tr>
  <tr>
    <td>No Language Overlap</td>
    <td>0.35</td>
    <td>0.52</td>
    <td>89%</td>
    <td>97%</td>
  </tr>
</table>

Table 3: mSTYLEDISTANCE embeddings under three generalization conditions on the multilingual (m avg) and cross-lingual (c avg) STEL-or-Content tasks.

the multilingual and cross-lingual STEL-or-content benchmarks can be found in Appendix H.

Ablation Experiments Following (Patel et al., 2024b), we run several ablation experiments to evaluate how well our model generalizes to unseen style features and languages. In the In-Domain condition, all style features are included in the training data for every language. To test generalization to unseen style features, in the Out of Domain condition, any style features directly equivalent to those features tested in the Multilingual and Cross-lingual SoC benchmarks are excluded from the training data. Out of Distribution further removes any style features indirectly similar or related to those tested in the benchmarks. Finally, No Language Overlap removes the languages present in the benchmark from the training data, in order to test generalization to new languages. Our results are given in Table 3 where we measure how much of the performance increase on SoC benchmarks over the base model is retained, despite ablating training data. The results indicate that our method generalizes reasonably well to both “out of domain”, and “out of distribution” style features, and very well to languages not in the training data. Further details on features and languages ablated and full results are provided in Appendix G.

Downstream Task Following Patel et al. (2024b), we also evaluate our mSTYLEDISTANCE embeddings in the authorship verification task, where the goal is to determine if two documents were written by the same author using stylistic features (Koppel and Winter, 2014). We use the datasets released by PAN7 between 2013 and 2015 in Greek, Spanish, and Dutch. Our results are given in Table 2. mSTYLEDISTANCE vectors outperform existing English style embedding models on Spanish and Greek, while Dutch shows similar performance to English STYLEDISTANCE.

We hypothesize that the linguistic proximity (West Germanic roots) of the two languages helps STYLEDISTANCE to generalize to Dutch.

5 Conclusion

We introduced a novel approach to learning multilingual style embeddings from synthetic examples, and contribute a two benchmarks for evaluating the quality of multilingual style representations. We show that mSTYLEDISTANCE embeddings are able to distinguish style from content better than other English and multilingual embeddings, and generalize well to unseen features and languages. The authorship verification evaluation shows that mSTYLEDISTANCE embeddings also offer strong performance on multilingual downstream tasks.

Limitations

Our synthetic data generation approaches rely on direct generation or machine translation techniques, both of which have limitations for languages other than English. Most of the languages included in our multilingual and cross-lingual STEL-or-Content and authorship evaluations are not really low-resource, so our evaluations may not reflect performance in languages with less resources. Furthermore, our approach only targets the 33-40 style features (depending on the language) we generated data for, and cannot account for the wide range of possible style variations. We generated data for features that are applicable to some extent to a given language, even if their expression is relatively weaker than in another language. Language compatibility in style space is a question worth exploring in depth in future work. While these constraints may limit our approach, our ablation experiments show strong generalization capabilities to unseen languages and style features indicating promising generalized performance.

Ethical Considerations

This work demonstrates the potential of using synthetic data for creating style embeddings in languages lacking such resources, increasing access to broader communities. However, it is important to recognize that the synthetic data generated by large language models may reflect and reinforce existing biases inherent in these models (Patel et al., 2023). While our approach shows significant promise, ongoing efforts should ensure that such synthetic
datasets are evaluated for fairness and bias to promote more equitable outcomes.

Contribution Statement

Justin Qiu created the SoC benchmarks and performed most of the evaluations. Jiacheng Zhu created mSYNTHSTEL by carrying out data generation and human annotation collection. Ajay Patel trained mSTYLEDISTANCE. Marianna Apidianaki helped greatly with advising and the final version of the manuscript. Chris Callison-Burch also gave us valuable advice as our advisor. All authors contributed to the final manuscript.

Acknowledgements

This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200005. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

References

Eleftheria Briakou, Di Lu, Ke Zhang, and Joel Tetreault. 2021. Olá, bonjour, salve! XFORMAL: A benchmark for multilingual formality style transfer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3199–3216, Online. Association for Computational Linguistics.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguistics.

Darya Dementieva, Daniil Moskovskiy, Nikolay Babakov, Abinev Ali Ayele, Naquee Rizwan, Florian Schneider, Xintog Wang, Seid Muhie Yimam, Dmitry Ustalov, Elisei Stakovskii, Alisa Smirnova, Ashraf Elnagar, Animesh Mukherjee, and Alexander Panchenko. 2024. Overview of the multilingual text detoxification task at pan 2024. In Working Notes of CLEF 2024 - Conference and Labs of the Evaluation Forum. CEUR-WS.org.

Julien Hay, Bich-Liên Doan, Fabrice Popineau, and Ouassim Ait Elhara. 2020. Representation learning of writing style. In Proceedings of the 6th Workshop on Noisy User-generated Text (W-NUT 2020).

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.

Moshe Koppel and Yaron Winter. 2014. Determining if two documents are written by the same author. Journal of the Association for Information Science and Technology, 65(1):178–187.

Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability.

Sourabrata Mukherjee, Atul Kr. Ojha, Akanksha Bansal, Deepak Alok, John P. McCrae, and Ondrej Dušek. 2024. Multilingual text style transfer: Datasets i& models for indian languages. Preprint, arXiv:2405.20805.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Justin Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Hee-woo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li,
Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovsky, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mélé, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emily Parpartia, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussee, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillett, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayaraghavan, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolfrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qingming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Ajay Patel, Colin Raffel, and Chris Callison-Burch. 2024a. DataDreamer: A tool for synthetic data generation and reproducible LLM workflows. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3781–3799, Bangkok, Thailand. Association for Computational Linguistics.

Ajay Patel, Delip Rao, Ansh Kothary, Kathleen McKeown, and Chris Callison-Burch. 2023. Learning interpretable style embeddings via prompting LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15270–15290, Singapore. Association for Computational Linguistics.

Ajay Patel, Jiacheng Zhu, Justin Qiu, Zachary Horvitz, Marianna Apidianaki, Kathleen McKeown, and Chris Callison-Burch. 2024b. Styledistance: Stronger content-independent style embeddings with synthetic parallel examples. Preprint, arXiv:2410.12757.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–6.

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.

Rafael A Rivera-Soto, Olivia Elizabeth Miano, Juanita Ordonez, Barry Y Chen, Aleem Khan, Marcus Bishop, and Nicholas Andrews. 2021. Learning universal authorship representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 913–919.

Michael Ryan, Tarek Naous, and Wei Xu. 2023. Revisiting non-English text simplification: A unified multilingual benchmark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4898–4927, Toronto, Canada. Association for Computational Linguistics.

Yves Scherrer. 2020. TaPaCo: A corpus of sentential paraphrases for 73 languages. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6868–6873, Marseille, France. European Language Resources Association.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. Preprint, arXiv:2402.05672.

Anna Wegmann, Marijn Schraagen, Dong Nguyen, et al. 2022. Same author or just same topic? towards content-independent style representations. In Proceedings of the 7th Workshop on Representation Learning for NLP, page 249. Association for Computational Linguistics.

Yue Yang, Mona Gandhi, Yufei Wang, Yifan Wu, Michael S Yao, Chris Callison-Burch, James C Gee, and Mark Yatskar. 2024. A textbook remedy for domain shifts: Knowledge priors for medical image analysis. arXiv preprint arXiv:2405.14839.

Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023. Large language model as
attributed training data generator: A tale of diversity and bias. In Advances in Neural Information Processing Systems, volume 36, pages 55734–55784. Curran Associates, Inc.
A  Style Features and Definitions

The style features addressed in our experiments included most of the 40 style features in the Patel et al. (2024b) dataset. In Table 9, we list the 40 style features with an ‘Excluded in’ column indicating the languages where each feature is not applicable and was therefore omitted from our dataset.

B  Generation Prompts and Details

Below we detail the structure of the prompts and the inference parameters used for our two multilingual synthetic data generation methods.

B.1  Topic Extraction from C4

We use the same topic extraction method as Patel et al. (2024b), which is derived from the C4 dataset (Raffel et al., 2020), to identify 50,000 topics through zero-shot prompting with GPT-4 (OpenAI et al., 2024). These 50,000 fine-grained, unique topics ensure that each sentence pair has a distinct context across various features and languages. We perform topic sampling with a temperature setting of 1.0 and top_p = 0.0.

What is the fine-grained topic of the following text:
{sentence} Only return the topic.

The fine-grained topic is used as part of the attributed prompt in Section B.2 to ensure diverse generations.

B.2  Generation of Positive and Negative Example Sentences

We use the same prompt as Patel et al. (2024b) to generate positive and negative example sentences in English. We then translate these sentence pairs into the target languages using the DeepL API. The only exception is Hindi, which we translate using Google Translate API due to DeepL’s limited language support.

For our second method, where example sentences are directly generated in the target language, we use the following prompt with temperature setting of 1.0 and top_p = 1.0

Generate a pair of {target language} sentences with and without sarcasm with the following attributes:
1. Topic: {topic}
2. Length: {sentence_length}
3. Point of view: {point_of_view}
4. Tense: {tense}
5. Type of Sentence: {sentence_type}

Ensure that the generated sentences meet the following conditions:
1. There is no extra information in one sentence that is not in the other.
2. The difference between the two sentences is subtle.
3. The two sentences have the same length.
{special_conditions_for_style_feature}

Use Format:
With sarcasm: [sentence in {target language}]
Without sarcasm: [sentence in {target language}]

Your response should only consist of the two sentences, without quotation marks.

C  Human Annotation Details

Text Style Feature: Usage of Active Voice

Sentence: 我简直不敢相信，我竟然在废品收购站里拍到了 Allantés!

Definition of the Text Style Feature: The usage of active voice in a text style feature refers to sentences where the subject performs the action stated by the verb. In other words, the subject is active and directly involved in the action. For example, in the sentence "The cat chased the mouse", 'the cat' is the subject that is actively doing the chasing.

Question 1: Is the text style feature present in the sentence?
  ○ Yes
  ○ Possibly
  ○ No

Answer Yes or No. Use Possibly if you are on the fence, but use it sparingly for true edge cases.

Question 2: Please rate the fluency of this sentence in the language it is written in:
  ○ Fluent
  ○ Mostly Fluent
  ○ Mostly Disfluent
  ○ Disfluent

1 - Fluent: Natural and clear; 4 - Disfluent: Unnatural and difficult to understand.

Figure 3: Instances from the annotation interface.

In Figure 3, we show an instance from the human annotation interface. We first asked the annotators
<table>
  <tr>
    <th>Language</th>
    <th>Baseline Feature Presence</th>
    <th>Feature Presence</th>
    <th>Baseline Fluency</th>
    <th>Fluency</th>
    <th>Baseline Diversity</th>
    <th>Diversity</th>
    <th>Baseline Similarity</th>
    <th>Similarity</th>
  </tr>
  <tr>
    <td>ar</td>
    <td>0.5</td>
    <td>0.7475</td>
    <td>0.5</td>
    <td>0.9526</td>
    <td>0.8278</td>
    <td>0.8245</td>
    <td>0.9232</td>
    <td>0.9156</td>
  </tr>
  <tr>
    <td>de</td>
    <td>0.5</td>
    <td>1.0000</td>
    <td>0.5</td>
    <td>0.7708</td>
    <td>0.8345</td>
    <td>0.8341</td>
    <td>0.8799</td>
    <td>0.9171</td>
  </tr>
  <tr>
    <td>es</td>
    <td>0.5</td>
    <td>0.8125</td>
    <td>0.5</td>
    <td>0.9853</td>
    <td>0.8449</td>
    <td>0.8478</td>
    <td>0.8567</td>
    <td>0.9298</td>
  </tr>
  <tr>
    <td>fr</td>
    <td>0.5</td>
    <td>0.7391</td>
    <td>0.5</td>
    <td>0.9855</td>
    <td>0.8483</td>
    <td>0.8404</td>
    <td>0.8573</td>
    <td>0.9224</td>
  </tr>
  <tr>
    <td>hi</td>
    <td>0.5</td>
    <td>0.7595</td>
    <td>0.5</td>
    <td>0.9958</td>
    <td>0.8588</td>
    <td>0.8253</td>
    <td>0.9468</td>
    <td>0.8903</td>
  </tr>
  <tr>
    <td>ja</td>
    <td>0.5</td>
    <td>0.6667</td>
    <td>0.5</td>
    <td>0.8889</td>
    <td>0.8528</td>
    <td>0.8321</td>
    <td>0.8514</td>
    <td>0.8761</td>
  </tr>
  <tr>
    <td>ko</td>
    <td>0.5</td>
    <td>0.8000</td>
    <td>0.5</td>
    <td>0.8972</td>
    <td>0.8540</td>
    <td>0.8214</td>
    <td>0.8652</td>
    <td>0.9286</td>
  </tr>
  <tr>
    <td>ru</td>
    <td>0.5</td>
    <td>0.8000</td>
    <td>0.5</td>
    <td>0.8972</td>
    <td>0.8542</td>
    <td>0.8097</td>
    <td>0.8713</td>
    <td>0.9171</td>
  </tr>
  <tr>
    <td>zh-hans</td>
    <td>0.5</td>
    <td>0.7475</td>
    <td>0.5</td>
    <td>0.9526</td>
    <td>0.8571</td>
    <td>0.8220</td>
    <td>0.8729</td>
    <td>0.9322</td>
  </tr>
  <tr>
    <th>Average</th>
    <th>0.5</th>
    <th>0.7859</th>
    <th>0.5</th>
    <th>0.9251</th>
    <th>0.8480</th>
    <th>0.8286</th>
    <th>0.8805</th>
    <th>0.9144</th>
  </tr>
</table>

Table 4: Human and automatic evaluations on our synthetic dataset.

whether a given style feature was present in a sentence in their chosen language. We also provided a definition for each style feature to help the annotators in their decision. The annotators then had to rate the fluency of the sentence by selecting one answer among: “Fluent”, “Mostly Fluent”, “Mostly Disfluent”, “Disfluent”.

Our annotators were undergraduate and graduate students from a NLP class and were offered extra credit for their participation in the study. Each instance was annotated by at least three annotators; three for languages with fewer native speakers such as Arabic and Russian; over ten for languages with a large number of native speakers, such as Chinese. We used Krippendorff’s Alpha (Krippendorff, 2011) to measure inter-annotator agreement, which indicated moderate agreement of \(0.4247 \pm 0.1719\).

A breakdown of the 13,651 annotated instances by language:
• Chinese: 6,553
• Spanish: 1,420
• Hindi: 1,186
• French: 1,003
• Korean: 982
• Japanese: 524
• German: 357
• Arabic: 344
• Russian: 282

D Dataset Evaluation

In Table 4, we show the per language results of the human and automatic evaluations for our synthetic dataset. Our synthetic dataset is comparable to a reference dataset compiled by Scherrer (2020) in terms of feature presence, fluency, diversity, and similarity. Note that baselines shown for feature presence and fluency are just 0.5 to represent random guessing.

E Training Details

Table 5 contains details about the hyperparameters used for training. More exact training details can be found in the source code provided in the supplementary materials for this work.

<table>
  <tr>
    <th>Hyperparameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>Model</td>
    <td>xlm-roberta-base</td>
  </tr>
  <tr>
    <td>Hardware</td>
    <td>4x or 8x NVIDIA RTX A6000</td>
  </tr>
  <tr>
    <td>Distributed Protocol</td>
    <td>PyTorch FSDP</td>
  </tr>
  <tr>
    <td>Data Type</td>
    <td>torch.bfloat16</td>
  </tr>
  <tr>
    <td>Loss Function</td>
    <td>TripletLoss (Schroff et al., 2015)</td>
  </tr>
  <tr>
    <td>Triplet Loss Margin</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>LoRA (Hu et al., 2021)</td>
    <td>all-linear, r=8, lora_alpha=8, lora_dropout=0.0</td>
  </tr>
  <tr>
    <td>Optimizer</td>
    <td>adamw_torch</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>1e-4</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>Learning Rate Scheduler</td>
    <td>linear</td>
  </tr>
  <tr>
    <td>Warmup Steps</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Batch Size</td>
    <td>384</td>
  </tr>
  <tr>
    <td>Train-Validation Split</td>
    <td>90/10%</td>
  </tr>
  <tr>
    <td>Early Stopping Threshold</td>
    <td>0.0</td>
  </tr>
  <tr>
    <td>Early Stopping Patience</td>
    <td>1 epoch</td>
  </tr>
</table>

Table 5: Hyperparameters selected for contrastive learning training experiments.

F Instances from the Multilingual and Cross-lingual SoC Benchmarks

In our multilingual SoC benchmark, anchor (\(a\)) has the same style and different content from a positive
<table>
  <tr>
    <th>Ablation Condition</th>
    <th colspan="2">Ablated Features and Languages</th>
  </tr>
  <tr>
    <td rowspan="2">Out-of-Domain</td>
    <th>Ablated Style Features:</th>
    <td>Usage of Formal Tone, Usage of Contractions, Usage of Numerical Substitution, Complex Sentence Structure, Usage of Positive Tone, Usage of Offensive Tone, Usage of Polite Tone</td>
  </tr>
  <tr>
    <th>Ablated Style Features:</th>
    <td>Usage of Formal Tone, Usage of Polite Tone, Fluency in Sentence Construction, Usage of Only Uppercase Letters, Usage of Only Lowercase Letters, Incorporation of Humor, Usage of Sarcasm, Usage of Contractions, Usage of Numerical Substitution, Usage of Text Emojis, Usage of Emojis, Presence of Misspelled Words, Complex Sentence Structure, Usage of Long Words, Usage of Polite Tone, Usage of Offensive Tone</td>
  </tr>
  <tr>
    <td>Out-of-Distribution</td>
    <th>Ablated Style Features:</th>
    <td>Usage of Formal Tone, Usage of Polite Tone, Fluency in Sentence Construction, Usage of Only Uppercase Letters, Usage of Only Lowercase Letters, Incorporation of Humor, Usage of Sarcasm, Usage of Contractions, Usage of Numerical Substitution, Usage of Text Emojis, Usage of Emojis, Presence of Misspelled Words, Complex Sentence Structure, Usage of Long Words, Usage of Polite Tone, Usage of Offensive Tone</td>
  </tr>
  <tr>
    <td>No Language Overlap</td>
    <th>Ablated Languages:</th>
    <td>ar (Arabic), de (German), es (Spanish), fr (French), hi (Hindi), ja (Japanese), ru (Russian)</td>
  </tr>
</table>

Table 6: Style features and languages ablated for Out-of-Domain, Out-of-Distribution, and No Language Overlap, the three ablation conditions in our ablation study.

<table>
  <tr>
    <th rowspan="2">Features Tested</th>
    <th colspan="4">Multi-lingual SoC</th>
    <th colspan="2">Cross-lingual SoC</th>
  </tr>
  <tr>
    <th>Simplicity</th>
    <th>Formality</th>
    <th>Toxicity</th>
    <th>Positivity</th>
    <th>Formality</th>
    <th>Retained Perf (%)<br>m<br>c</th>
  </tr>
  <tr>
    <td>In-Domain</td>
    <td>0.36</td>
    <td>0.71</td>
    <td>0.37</td>
    <td>0.30</td>
    <td>0.53</td>
    <td>100%<br>100%</td>
  </tr>
  <tr>
    <td>Out of Domain</td>
    <td>0.29</td>
    <td>0.63</td>
    <td>0.31</td>
    <td>0.23</td>
    <td>0.44</td>
    <td>75%<br>74%</td>
  </tr>
  <tr>
    <td>Out of Distribution</td>
    <td>0.33</td>
    <td>0.39</td>
    <td>0.26</td>
    <td>0.32</td>
    <td>0.40</td>
    <td>75%<br>62%</td>
  </tr>
  <tr>
    <td>No Language Overlap</td>
    <td>0.27</td>
    <td>0.51</td>
    <td>0.41</td>
    <td>0.32</td>
    <td>0.52</td>
    <td>89%<br>97%</td>
  </tr>
</table>

Table 7: Results of the ablation study for mSTYLEDISTANCE embeddings on the SoC benchmarks.

example (pos), and the same content but different style from a negative example (neg). The anchor and the pos and neg sentences are in the same language. The tested model needs to successfully pair \( a \) with pos (rather than \( a \) and neg). Cross-lingual SoC has the same setup as multilingual SoC, except that the pos and neg examples are in a different language than the anchor. Figure 4 contains instances of each benchmark.

G Ablation Details and Results

Details about the ablated features and languages can be found in Table 6. Table 7 contains the results of the ablation study for mSTYLEDISTANCE embeddings on the SoC benchmarks under three generalization conditions: Out of Domain, Out of Distribution, and No Language Overlap. For multilingual SoC, we use all four style features; simplicity, formality, toxicity, and positivity. For the cross-lingual variant, we only use formality.

![Multilingual STEL-or-Content Task](page_1012_1202_377_180.png)

![Crosslingual STEL-or-Content Task](page_1012_1402_377_180.png)

Figure 4: Instances from our multilingual and cross-lingual SoC benchmarks. For multilingual SoC, the anchor is in the same language as the pos and neg sentences. For cross-lingual SoC, the anchor is in a different language from the pos and neg sentences.
H   Full SoC Results

<table>
  <tr>
    <th>Language</th>
    <th>Wegmann et al. (2022)</th>
    <th>STYLEDISTANCE</th>
    <th>xlm-roberta-base</th>
    <th>LISA</th>
    <th>mSTYLEDISTANCE</th>
  </tr>
  <tr>
    <th colspan="6">Simplicity</th>
  </tr>
  <tr>
    <td>de</td><td>0.23</td><td>0.06</td><td>0.00</td><td>0.00</td><td>0.24</td>
  </tr>
  <tr>
    <td>en</td><td>0.26</td><td>0.32</td><td>0.05</td><td>0.00</td><td>0.36</td>
  </tr>
  <tr>
    <td>fr</td><td>0.29</td><td>0.33</td><td>0.22</td><td>0.12</td><td>0.46</td>
  </tr>
  <tr>
    <td>it</td><td>0.21</td><td>0.15</td><td>0.08</td><td>0.03</td><td>0.48</td>
  </tr>
  <tr>
    <td>ja</td><td>0.09</td><td>0.05</td><td>0.01</td><td>0.48</td><td>0.14</td>
  </tr>
  <tr>
    <td>pt-br</td><td>0.10</td><td>0.07</td><td>0.04</td><td>0.03</td><td>0.15</td>
  </tr>
  <tr>
    <td>ru</td><td>0.26</td><td>0.24</td><td>0.07</td><td>0.15</td><td>0.38</td>
  </tr>
  <tr>
    <td>sl</td><td>0.43</td><td>0.43</td><td>0.46</td><td>0.39</td><td>0.69</td>
  </tr>
  <tr>
    <td>average</td><td>0.23</td><td>0.21</td><td>0.12</td><td>0.15</td><td>0.36</td>
  </tr>
  <tr>
    <th colspan="6">Formality</th>
  </tr>
  <tr>
    <td>fr</td><td>0.70</td><td>0.81</td><td>0.16</td><td>0.06</td><td>0.82</td>
  </tr>
  <tr>
    <td>it</td><td>0.64</td><td>0.63</td><td>0.18</td><td>0.10</td><td>0.69</td>
  </tr>
  <tr>
    <td>pt-br</td><td>0.56</td><td>0.57</td><td>0.15</td><td>0.11</td><td>0.62</td>
  </tr>
  <tr>
    <td>average</td><td>0.63</td><td>0.67</td><td>0.16</td><td>0.09</td><td>0.71</td>
  </tr>
  <tr>
    <th colspan="6">Toxicity</th>
  </tr>
  <tr>
    <td>am</td><td>0.35</td><td>0.29</td><td>0.24</td><td>0.21</td><td>0.53</td>
  </tr>
  <tr>
    <td>ar</td><td>0.05</td><td>0.04</td><td>0.02</td><td>0.10</td><td>0.18</td>
  </tr>
  <tr>
    <td>de</td><td>0.01</td><td>0.02</td><td>0.01</td><td>0.00</td><td>0.28</td>
  </tr>
  <tr>
    <td>en</td><td>0.56</td><td>0.48</td><td>0.09</td><td>0.08</td><td>0.51</td>
  </tr>
  <tr>
    <td>es</td><td>0.26</td><td>0.20</td><td>0.13</td><td>0.05</td><td>0.35</td>
  </tr>
  <tr>
    <td>hi</td><td>0.15</td><td>0.09</td><td>0.09</td><td>0.15</td><td>0.37</td>
  </tr>
  <tr>
    <td>ru</td><td>0.18</td><td>0.16</td><td>0.13</td><td>0.09</td><td>0.61</td>
  </tr>
  <tr>
    <td>uk</td><td>0.07</td><td>0.05</td><td>0.04</td><td>0.02</td><td>0.25</td>
  </tr>
  <tr>
    <td>zh</td><td>0.05</td><td>0.02</td><td>0.04</td><td>0.07</td><td>0.23</td>
  </tr>
  <tr>
    <td>average</td><td>0.19</td><td>0.15</td><td>0.09</td><td>0.09</td><td>0.37</td>
  </tr>
  <tr>
    <th colspan="6">Positivity</th>
  </tr>
  <tr>
    <td>bn</td><td>0.27</td><td>0.13</td><td>0.04</td><td>0.23</td><td>0.32</td>
  </tr>
  <tr>
    <td>en</td><td>0.21</td><td>0.20</td><td>0.03</td><td>0.19</td><td>0.18</td>
  </tr>
  <tr>
    <td>hi</td><td>0.11</td><td>0.10</td><td>0.04</td><td>0.14</td><td>0.22</td>
  </tr>
  <tr>
    <td>mag</td><td>0.09</td><td>0.08</td><td>0.08</td><td>0.13</td><td>0.41</td>
  </tr>
  <tr>
    <td>ml</td><td>0.32</td><td>0.28</td><td>0.10</td><td>0.27</td><td>0.39</td>
  </tr>
  <tr>
    <td>mr</td><td>0.19</td><td>0.18</td><td>0.03</td><td>0.17</td><td>0.22</td>
  </tr>
  <tr>
    <td>or</td><td>0.27</td><td>0.19</td><td>0.08</td><td>0.24</td><td>0.35</td>
  </tr>
  <tr>
    <td>pa</td><td>0.18</td><td>0.15</td><td>0.06</td><td>0.17</td><td>0.23</td>
  </tr>
  <tr>
    <td>te</td><td>0.39</td><td>0.34</td><td>0.20</td><td>0.29</td><td>0.40</td>
  </tr>
  <tr>
    <td>ur</td><td>0.24</td><td>0.20</td><td>0.08</td><td>0.28</td><td>0.26</td>
  </tr>
  <tr>
    <td>average</td><td>0.23</td><td>0.18</td><td>0.07</td><td>0.21</td><td>0.30</td>
  </tr>
  <tr>
    <th colspan="6">Formality (cross-lingual)</th>
  </tr>
  <tr>
    <td>fr-it</td><td>0.47</td><td>0.51</td><td>0.22</td><td>0.28</td><td>0.53</td>
  </tr>
  <tr>
    <td>fr-pt</td><td>0.45</td><td>0.48</td><td>0.19</td><td>0.29</td><td>0.52</td>
  </tr>
  <tr>
    <td>it-fr</td><td>0.48</td><td>0.53</td><td>0.18</td><td>0.26</td><td>0.53</td>
  </tr>
  <tr>
    <td>it-pt</td><td>0.41</td><td>0.45</td><td>0.19</td><td>0.27</td><td>0.52</td>
  </tr>
  <tr>
    <td>pt-fr</td><td>0.46</td><td>0.53</td><td>0.17</td><td>0.27</td><td>0.53</td>
  </tr>
  <tr>
    <td>pt-it</td><td>0.42</td><td>0.47</td><td>0.21</td><td>0.27</td><td>0.52</td>
  </tr>
  <tr>
    <td>average</td><td>0.45</td><td>0.49</td><td>0.19</td><td>0.27</td><td>0.53</td>
  </tr>
</table>

Table 8: Performance obtained for different features and languages in the multilingual and cross-lingual STEL-or-content benchmarks. For the cross-lingual SoC evaluation, "a-b" means that the anchor sentences were all in language a and alternative sentences were all in language b. mSTYLEDISTANCE leads in cross-lingual and overall performance.
<table>
  <tr>
    <th>Style Feature</th>
    <th>Positive and Negative Prompts</th>
    <th>Style Feature Definition</th>
    <th>Excluded In</th>
  </tr>
  <tr>
    <td>Usage of Conjunctions</td>
    <td>Positive: With conjunctions<br>Negative: Less frequent conjunctions</td>
    <td>The "Usage of Conjunctions" text style feature refers to the use of words that connect clauses or sentences. Conjunctions are words like "and", "but", "or", "so", "because", etc. They are used to make sentences longer, more complex, or to show the relationship between different parts of a sentence.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Numerical Substitution</td>
    <td>Positive: With number substitution<br>Negative: Without number substitution</td>
    <td>Numerical substitution refers to the practice of replacing certain letters in words with numbers that visually resemble those letters. For example, replacing the letter 'e' with the number '3' in the word 'hello' to make it 'h3llo'. This is a common feature in internet slang and informal digital communication.</td>
    <td>Arabic, Hindi, Japanese, Korean, Chinese</td>
  </tr>
  <tr>
    <td>Usage of Words Indicating Affective Processes</td>
    <td>Positive: Affective processes<br>Negative: Cognitive processes</td>
    <td>The text style feature 'Usage of Words Indicating Affective Processes' refers to the use of words that express emotions, feelings, or attitudes. These could be words that show happiness, sadness, anger, fear, surprise, or any other emotional state. The presence of such words in a text indicates that the writer is expressing some form of emotional reaction or sentiment.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Metaphors</td>
    <td>Positive: With metaphor<br>Negative: Without metaphor</td>
    <td>The "Usage of Metaphors" text style feature refers to the presence of phrases or sentences in the text that describe something by comparing it indirectly to something else. Metaphors often don’t make a direct comparison at all and to explain complex ideas in a more understandable way. For example, saying “time is a thief” is a metaphor because it’s not literally true but it helps to convey the idea that time passes quickly and can’t be regained.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Long Words</td>
    <td>Positive: Long average word length<br>Negative: Short average word length</td>
    <td>The "Usage of Long Words" text style feature refers to the frequency or prevalence of long words, typically those with more than six or seven letters, in a given text. This style feature is often used to measure the complexity or sophistication of a text. If a text has many long words, it is said to have a high usage of long words.</td>
    <td>Arabic, Japanese, Korean, Chinese</td>
  </tr>
  <tr>
    <td>Usage of Uppercase Letters</td>
    <td>Positive: With uppercase letters<br>Negative: Without uppercase letters</td>
    <td>The usage of uppercase letters as a text style feature refers to the frequency or manner in which capital letters are used in a text. This could be for emphasis, to denote shouting or strong emotions, or to highlight specific words or phrases. It’s not just about the start of sentences or proper nouns, but also about other uses of capital letters in the text.</td>
    <td>Arabic, Hindi, Japanese, Korean, Chinese</td>
  </tr>
  <tr>
    <td>Usage of Articles</td>
    <td>Positive: With articles<br>Negative: Less frequent articles</td>
    <td>The "Usage of Articles" text style feature refers to how often a text uses words like "a", "an", and "the". These words are called articles and they are used before nouns. This feature measures the frequency of these articles in a given text.</td>
    <td>Arabic, Hindi, Japanese, Korean, Russian, Chinese</td>
  </tr>
  <tr>
    <td>Usage of Text Emojis</td>
    <td>Positive: Text Emojis<br>Negative: No Emojis</td>
    <td>The text style feature "Usage of Text Emojis" refers to the inclusion of emoticons or smileys in the text. These are combinations of keyboard characters that represent facial expressions or emotions, such as -D for a big grin or happy face. The presence of these symbols in a text indicates the use of this style feature.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Nominalizations</td>
    <td>Positive: With nominalizations<br>Negative: Without nominalizations</td>
    <td>Nominalizations refer to the use of verbs, adjectives, or adverbs as nouns in a sentence. This style feature is often used to make sentences more concise or formal. For example, "the investigation of the crime" is a nominalization of "investigate the crime".</td>
    <td></td>
  </tr>
  <tr>
    <td>Frequent Usage of Function Words</td>
    <td>Positive: With function words<br>Negative: Less frequent function words</td>
    <td>The text style feature "Frequent Usage of Function Words" refers to the regular use of words that have little meaning on their own but work in combination with other words to express grammatical relationships. These words include prepositions (like 'in', 'at', 'on'), conjunctions (like 'and', 'but', 'or'), articles (like 'a', 'an', 'the'), and pronouns (like 'he', 'they', 'it').</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Self-Focused Perspective or Words</td>
    <td>Positive: Self-focused<br>Negative: Third-person singular</td>
    <td>The "Usage of Self-Focused Perspective or Words" text style feature refers to the use of words or phrases that focus on the speaker or writer themselves. This includes the use of first-person pronouns like I, me, my, mine, and myself, or statements that express the speaker's personal thoughts, feelings, or experiences.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Formal Tone</td>
    <td>Positive: Formal<br>Negative: Informal</td>
    <td>The "Usage of Formal Tone" text style feature refers to the use of language that is polite, impersonal and adheres to established conventions in grammar and syntax. It avoids slang, contractions, colloquialisms, and often uses more complex sentence structures. This style is typically used in professional, academic, or official communications.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Emojis</td>
    <td>Positive: With Emojis<br>Negative: No Emojis</td>
    <td>The "Usage of Emojis" text style feature refers to the inclusion of emojis, or digital icons, in a text. Emojis are often used to express emotions, ideas, or objects without using words. If a text contains emojis, it has this style feature.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Offensive Language</td>
    <td>Positive: Offensive<br>Negative: Non-Offensive</td>
    <td>The "Usage of Offensive Language" text style feature refers to the presence of words or phrases in the text that are considered rude, disrespectful, or inappropriate. These can include swear words, slurs, or any language that could be seen as insulting or derogatory.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Present Tense and Present-Focused Words</td>
    <td>Positive: Present-focused<br>Negative: Future-focused</td>
    <td>The text style feature 'Usage of Present Tense and Present-Focused Words' refers to the use of verbs in the present tense and words that focus on the current situation or action. This means the text is primarily discussing events, actions, or states that are happening now or factual truths. It’s like the text is talking about what is happening in the present time.</td>
    <td></td>
  </tr>
  <tr>
    <td>Presence of Misspelled Words</td>
    <td>Positive: Sentence With a Few Misspelled Words<br>Negative: Normal Sentence</td>
    <td>The text style feature 'Presence of Misspelled Words' refers to the occurrence of words in a text that are not spelled correctly according to standard dictionary spelling. This could be due to typing errors, lack of knowledge about the correct spelling, or intentional for stylistic or informal communication purposes.</td>
    <td></td>
  </tr>
  <tr>
    <td>Incorporation of Humor</td>
    <td>Positive: With Humor<br>Negative: Without Humor</td>
    <td>The "Incorporation of Humor" text style feature refers to the use of language, phrases, or expressions in a text that are intended to make the reader laugh or feel amused. This could include jokes, puns, funny anecdotes, or witty remarks. It’s all about adding a touch of comedy or light-heartedness to the text.</td>
    <td></td>
  </tr>
</table>
<table>
  <tr>
    <th>Style Feature Name</th>
    <th>Positive and Negative Prompts</th>
    <th>Style Feature Definition</th>
    <th>Excluded In</th>
  </tr>
  <tr>
    <td>Usage of Personal Pronouns</td>
    <td>Positive: With personal pronouns<br>Negative: Less frequent pronouns</td>
    <td>The "Usage of Personal Pronouns" text style feature refers to the use of words in a text that refer to a specific person or group of people. These words include: "I", "you", "he", "she", "it", "we", and "they". The presence of these words in a text can indicate a more personal or direct style of communication.</td>
    <td></td>
  </tr>
  <tr>
    <td>Fluency in Sentence Construction</td>
    <td>Positive: Fluent sentence<br>Negative: Disfluent sentence</td>
    <td>"Fluency in Sentence Construction" refers to the smoothness and ease with which sentences are formed and flow together. It involves using correct grammar, appropriate vocabulary, and logical connections between ideas. A text with this feature would read smoothly, without abrupt changes or awkward phrasing.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Only Uppercase Letters</td>
    <td>Positive: All Upper Case<br>Negative: Proper Capitalization</td>
    <td>The usage of only uppercase letters style feature refers to the practice of writing all the letters in a text in capital letters. This means that every single letter in the text, whether at the beginning, middle, or end of a sentence, is capitalized. It's like the 'Caps Lock' key on your keyboard is always turned on while typing the text.</td>
    <td>Arabic, Hindi, Japanese, Korean, Chinese</td>
  </tr>
  <tr>
    <td>Usage of Self-Focused Perspective or Words</td>
    <td>Positive: Self-focused<br>Negative: Inclusive-focused</td>
    <td>The "Usage of Self-Focused Perspective or Words" text style feature refers to the use of words that focus on the speaker or writer themselves. This includes the use of first-person pronouns like "I", "me", "my", "mine", and "myself", or statements that express the speaker's personal thoughts, feelings, or experiences.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Pronouns</td>
    <td>Positive: With pronouns<br>Negative: Less frequent pronouns</td>
    <td>The "Usage of Pronouns" text style feature refers to the frequency and types of pronouns used in a text. Pronouns are words like "he", "she", "it", "they", "we", "you", "I", etc., that stand in place of names or nouns in sentences. This feature can indicate the level of personalization, formality, or perspective in a text.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Words Indicating Cognitive Processes</td>
    <td>Positive: Cognitive process<br>Negative: Perceptual process</td>
    <td>The text style feature "Usage of Words Indicating Cognitive Processes" refers to the use of words that show thinking or mental processes. These words can express understanding, knowledge, belief or doubt. For example, words like "think", "know", "believe", "understand" are used to indicate cognitive processes.</td>
    <td></td>
  </tr>
  <tr>
    <td>Complex Sentence Structure</td>
    <td>Positive: Complex<br>Negative: Simple</td>
    <td>The "Complex Sentence Structure" text style feature refers to sentences that contain multiple ideas or points, often connected by conjunctions (like 'and', 'but', 'or') or punctuation (like commas, semicolons). These sentences often include dependent clauses, which are parts of the sentence that can't stand alone as a complete thought. A complex independent clause, which can stand alone, but a complex sentence has more than one part and these parts are linked together in a way that they give more detailed information or express multiple thoughts, it has a complex sentence structure.</td>
    <td></td>
  </tr>
  <tr>
    <td>Positive Sentiment Expression</td>
    <td>Positive: Positive<br>Negative: Negative</td>
    <td>Positive Sentiment Expression is a text style feature that refers to the use of words, phrases, or expressions that convey a positive or optimistic viewpoint or emotion. This could include expressions of happiness, joy, excitement, love, or any other positive feeling. The text is considered to have this feature if it makes the reader feel good or positive after reading it.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Numerical Digits</td>
    <td>Positive: With digits<br>Negative: Less frequent digits</td>
    <td>The "Usage of Numerical Digits" text style feature refers to the presence and use of numbers in a text. This includes any digit from 0-9 used alone or in combination to represent quantities, dates, times, or any other numerical information.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Words Indicating Affective Process</td>
    <td>Positive: Affective process<br>Negative: Perceptual process</td>
    <td>The "Usage of Words Indicating Affective-Process" text style feature refers to the use of words that express emotions, feelings, or attitudes. These words can show positive or negative sentiments, like happiness, anger, love, or hate. If a text uses a lot of these words, it means the writer is expressing a lot of emotion or personal feelings.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Active Voice</td>
    <td>Positive: Active<br>Negative: Passive</td>
    <td>The usage of active voice in a text style feature refers to sentences where the subject performs the action stated by the verb. In other words, the subject is active and directly involved in the action. For example, in the sentence "The cat chased the mouse", 'the cat' is the subject that is actively doing the chasing.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Only Lowercase Letters</td>
    <td>Positive: All Lower Case<br>Negative: Proper Capitalization</td>
    <td>The style feature "usage of only lowercase letters" refers to the practice of writing all words in a text with small letters only, without using any capital letters. This means that even the first word of a sentence, proper nouns, or the pronoun 'I' are not capitalized. It is like writing a whole text without ever pressing the shift key on your keyboard.</td>
    <td>Arabic, Hindi, Japanese, Korean, Chinese</td>
  </tr>
  <tr>
    <td>Frequent Usage of Common Verbs</td>
    <td>Positive: With common verbs<br>Negative: Less frequent common verbs</td>
    <td>The text style feature "Frequent Usage of Common Verbs" refers to the regular use of basic action words in a text. These are often simple, everyday verbs that are widely used in language, such as 'is', 'have', 'do', 'say', 'go' etc. If a text frequently uses these common verbs, it has this style feature.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Prepositions</td>
    <td>Positive: With prepositions<br>Negative: Less frequent prepositions</td>
    <td>The "Usage of Prepositions" text style feature refers to the use of words that link nouns, pronouns, or phrases to other words within a sentence. These words often indicate location, direction, time, or manner. Examples of prepositions include words like "in", "at", "on", "over", "under", "after", and "before".</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Self-Focused Language</td>
    <td>Positive: Self-focused<br>Negative: Audience-focused</td>
    <td>The "Usage of Self-Focused Language" text style feature refers to the use of words or phrases that focus on the speaker or writer themselves. This includes the use of first-person pronouns like "I", "me", "my", "mine", and "myself". It's a way of writing or speaking where the person is often referring to their own thoughts, feelings, or experiences.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Certain Tone</td>
    <td>Positive: Certain<br>Negative: Uncertain</td>
    <td>This text style feature refers to the use of a confident tone in writing, where the author sounds using certain words or phrases such as "I think", "might", or "seems". This means in a text that appears more assertive and sure of the information being presented.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Present-Focused Tense and Words</td>
    <td>Positive: Present-focused<br>Negative: Past-focused</td>
    <td>The "Usage of Present-Focused Tense and Words" text style feature refers to the use of verbs in the present tense and words that focus on the current moment or situation. This means the text is primarily discussing events, actions, or states that are happening right now or generally true.</td>
    <td></td>
  </tr>
</table>
<table>
  <tr>
    <th>Style Feature Name</th>
    <th>Positive and Negative Prompts</th>
    <th>Style Feature Definition</th>
    <th>Excluded In</th>
  </tr>
  <tr>
    <td>Usage of Sarcasm</td>
    <td>Positive: With sarcasm<br>Negative: Without sarcasm</td>
    <td>The "Usage of Sarcasm" text style feature refers to the presence of statements or expressions in the text that mean the opposite of what they literally say, often used to mock or show irritation. This style is often characterized by irony, ridicule, or mockery, and is used to express contempt or to criticize something or someone in a humorous way.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Self-Focused Perspective or Words</td>
    <td>Positive: Self-focused<br>Negative: You-focused</td>
    <td>The "Usage of Self-Focused Perspective or Words" text style feature refers to the use of words or phrases that focus on the speaker or writer themselves. This includes the use of first-person pronouns like "I", "me", "my", "mine", and "myself", or statements that express the speaker's personal thoughts, feelings, or experiences.</td>
    <td></td>
  </tr>
  <tr>
    <td>Frequent Usage of Punctuation</td>
    <td>Positive: With frequent punctuation<br>Negative: Less Frequent punctuation</td>
    <td>The text style feature "Frequent Usage of Punctuation" refers to the regular and abundant use of punctuation marks such as commas, periods, exclamation points, question marks, etc., in a piece of text. This style feature is present when the writer often uses these symbols to structure their sentences, express emotions, or emphasize certain points.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Polite Tone</td>
    <td>Positive: Polite<br>Negative: Impolite</td>
    <td>The "Usage of Polite Tone" text style feature refers to the use of respectful and considerate language in a text. This can include using words like 'please', 'thank you', or phrases that show deference or respect to the reader. It's about making the text sound courteous and respectful, rather than demanding or rude.</td>
    <td></td>
  </tr>
  <tr>
    <td>Usage of Contractions</td>
    <td>Positive: With contractions<br>Negative: Without contractions</td>
    <td>The "Usage of Contractions" text style feature refers to the use of shortened forms of words or phrases in a text. These are typically formed by omitting certain letters or sounds and replacing them with an apostrophe, such as "don't" for "do not" or "I'm" for "I am". If a text frequently uses such shortened forms, it has this style feature.</td>
    <td>Arabic, Hindi, Japanese, Korean, Russian, Chinese</td>
  </tr>
  <tr>
    <td>Frequent Usage of Determiners</td>
    <td>Positive: With determiners<br>Negative: Less frequent determiners</td>
    <td>The text style feature "Frequent Usage of Determiners" refers to the regular use of words that introduce a noun and give information about its quantity, proximity, definiteness, etc. These words include 'the', 'a', 'an', 'this', 'that', 'these', 'those', 'my', 'your', 'his', 'her', 'its', 'our', 'their'. If a text often uses such words, it has this style feature.</td>
    <td></td>
  </tr>
</table>

Table 9: The 40 style features addressed in our experiments. The 'Excluded in' column indicates that a particular feature was omitted from our dataset due to its inapplicability to a specific language.

Reviewer paper summaries: This paper extends the work of Patel et al. (2024b) from an English-only style embedding to a multilingual style embedding. They do so by constructing new data examples using generation and/or machine translation based on the original data and fine-tuning XLM-R base (an encoder model that also is used as one of the baselines).  The training data is evaluated for similarity and diversity of topics using automated metrics, but also human annotation of data quality (was the expected style feature present and is the sentence fluent).  By constructing new multilingual and cross-lingual benchmarks, the authors show that the model performs better than current English-only embeddings, but there is still plenty of room for improvement.  The model is also evaluated on authorship verification benchmarks in Spanish, Dutch, and Greek (PAN 2013-2015) and outperforms the English baselines for Spanish and Greek (but is beat by one other model for Dutch).  The authors provide an ablation to understand how well their model would extend to unseen style features and languages, and it does both, but is stronger across unseen languages than style features.
Summary of strengths: - Ablation results make a good case for the usefulness of the approach to other languages
- Well presented for a short paper, good use of Appendices for more details
- Results are not strong, but a good starting point for future work
Summary of weaknesses: - Results are not strong, often not much better than the English-only models, but this is not mentioned/addressed in the paper (in particular, how do their unseen language results compare to the English-only models)
- Some lack of clarity on exactly what data is included: is all training data human validated or just 10%?  Is the original English data from Patel et al. (2024b) included in the training?
- Possible missing baseline: Do the English-only baselines involve translating data into English first?  If not, isn't this a reasonable and expected baseline?
Comments, suggestions and typos: - Did the annotators assess every instance of training data or just 10% of the data to evaluate the best method of annotating?
- Did the authors include the English-only data in the training data? If the English-only models are strong enough to be close-competitors, perhaps training on available English training data would be helpful.
- Do the toxicity and positivity datasets include merely stylistic changes?
- Do the English-only baselines involve translating data into English first?  If not, isn't that also a useful baseline?  The "translate-test" way of using English-only models for multilingual use cases is now standard practice -- though I admit it may not work well for this use case.
- Is the mStyleDistance model intended for English style embedding?  It is multilingual so it is reasonable to use it.  Why not compare mStyleDistance on the English SoC as well or at least mention this use case?
- Table 3 should include the best competitive system from Table 1, for comparison with No Language Overlap.  I assume that the ablated model is still better than Style Distance or Wegmann et al. (2022).  In fact, I believe the "c avg" is directly comparable to the numbers in Table 1's formality column, so it is true for that case.
Overall assessment scores: 3.5