RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Problem Formulation: Few-Shot Conversation Style Transfer
In this section, we propose the novel task of conversation style transfer, based on a few non-parallel examples, that does not rely on style attribute definitions (an example is illustrated in Figure 1 ). Given a conversation in source style A and a few shot non-parallel conversations in target style B, the task is to transfer the style of the conversation in source style A to style B. We address the following limitations of the state-of-the-art models in this task.
In-Context Learning for Conversation Style Transfer
In this section, we propose a novel in-context learning based method using large language models (LLMs) for few-shot conversation style transfer. The method is illustrated in Figure 2 .
In-context learning with non-parallel examples in source and target styles
To tackle the problem of the unavailability of parallel conversations in source and target styles (as described in Section 2 ), in this paper, we propose a cheaper alternative solution, which prompts the language models with dialogues in one style and their style-free versions.
Previous work by Madaan et al. ( 2020 ) showed the effectiveness of style transfer after reducing the source text to a style-free format and then converting the style-free format to the target style (although they relied on large amount of training data for the purpose). Inspired from these intuitions we break down the task of style transfer in the following two steps.

Style Reduction: In this step, we use an in-context learning method using LLMs to reduce the source conversation to a style-free form.
As a result, we need parallel examples only in the form ( C_{A} , C^{\prime} ) for prompting LLMs, where C_{A} is a conversation in source style A and C^{\prime} is the style free form of C_{A} .

Transfer to the Target Style: In this step, we use another in-context learning step where we convert the style-free input conversation to the target style. This step also requires parallel examples only in the form ( C^{\prime} , C_{B} ), where C_{B} is a conversation in target style B and C^{\prime} is the style free form of C_{B} .

We use human supervision to construct the parallel ( C_{A/B} , C^{\prime} ) examples. Note that, it is easier for humans to rewrite a conversation in a style-free format as it omits the requirement of having knowledge about the target style. Prompt structures and examples for in-context learning for the above two steps can be found in Appendix A .
Dynamic Prompt Selection
Conversations are dynamic and the style of a response depends on the situation as shown in Figure 1 . Hence, the same set of few-shot examples may not work best as training examples for all test conversations as the situations and respective styles may be different (e.g., different styles are expected when responding to someone distressed vs happy). To resolve this, we propose a dynamic prompt selection technique (Reif et al., 2022 ; Han et al., 2022 ) for style transfer where semantically similar examples to a test conversation are retrieved and used as few-shot training examples in the prompt. We first concatenate all utterances of a participant in a conversation sequentially. Then we use a sentence transformer Reimers and Gurevych ( 2019 ) designed for semantic search to encode the concatenated utterances to get a semantic meaning-based embedding. For each test conversation, we measure the cosine similarity between the embedding of the test conversation and all of the available few-shot training conversations. We select the top-k semantically similar few-shot examples for the test conversation during prompting. The more semantically similar conversation appears later in the prompt to place it closer to the test conversation. The effectiveness of this approach is examined by comparing it with random prompt example selection method in Section 4 .
Baseline: Utterance level style transfer
Existing works study style transfer at the utterance level with in-context learning (Reif et al., 2022 ; Suzgun et al., 2022 ) , hence, we use utterance-level style transfer as a baseline. We transfer the style of the utterances of one party in a dialogue utterance by utterance using the same procedure described above. For dynamic prompt selection, we measure semantic similarity between single utterances instead of concatenating all utterances of a participant in a dialogue. As existing models are either applicable to utterance level only (Riley et al., 2021 ) or require a lot of training data (Madaan et al., 2020 ) for style transfer, they are not applicable in conversation style transfer in a few-shot setting.
Experiments
In this section, we present the evaluation setup and the results of the proposed approaches on style transfer quality including style strength, appropriateness, and semantic correctness. Then, we show the evaluation results of applying the approach on a downstream task, namely intent classification.
Results
We show human evaluation results on utterance-level and conversation-level style transfer in Table 8 . Models were run on test data (Table 5 ) using the best hyper-parameters and prompt selection method obtained in the ablation step. We first observe that the highest style strength rank score is achieved when performing utterance-level style transfer, however, this results in a lower appropriateness score. This observation shows that performing conversation style transfer without the dialogue context has a significant risk of resulting in inappropriate agent utterances (i.e., utterances do not fit in the context). We can also observe in Table 8 that the smaller LLM GPT-NeoX suffers more from the problem of generation of inappropriate responses compared to the larger LLM Bloom. Next, we observe that if we increase context (4/5 turns) in the conversation style transfer, the style strength decreases but appropriateness is preserved. Interestingly, for the larger LLM Bloom, the semantic similarity decreases with the increase of context. We found out that sometimes Bloom generates new agent utterances different from the source utterances or swaps the agent utterance with the customer utterance when performing 4-5 turns conversation-level style transfer (examples are shown in Appendix I ). Hence, resulting in semantically dissimilar utterances.

Therefore, we conclude that the LLMs are still not successful in conditioning on a larger context when performing style transfer, hence, a limited context consisting of 2 utterances is the optimal setting for style transfer in our study. Automatic evaluation results on the test set resulted in the same pattern (shown in Appendix G ). Examples of style-transferred conversations in all style directions by various versions of our model are shown in Appendix H and the effects of style transfer on the observed style properties in Table 4 are discussed in Appendix K . We present examples of errors by various versions of the models in Table 9 .
Evaluation on Downstream Task
Downstream applications of conversation style transfer are understudied. In this paper, we apply conversation style transfer to intent classification.
We evaluate the setting where we have abundant of training data in one style and the test data is in a different style. We test our approach on three domains in the DSTC11 intent induction dataset: insurance, banking, and finance. Here, the training data is in human-to-human (h2h) style and the test data is in human-to-bot (h2b) style. We transfer the training data from h2h style to h2b style before training a RoBERTa-based intent classifier.

We run an ablation (using data from banking and finance domains) with utterance-level style transfer and short-conversation-level style transfer using GPT-NeoX and observe that training data transferred to h2b style using utterance-level style transfer results in higher intent classification F1 scores. We conjecture the reason is that utterance-level style transfer has the strongest style strength score, benefiting the application of domain adaptation. We report results with this method on all three domains in Table 10 . The intent classification results show statistically significant improvement in insurance and banking, and non-significant improvement in finance, compared to the baseline where the training data has h2h style.
Data statistics, experimental details, and ablation studies can be found in Appendix E .
Conclusion
In this paper, we study a novel problem of conversation style transfer using few-shot non-parallel examples. To solve this problem we propose a novel in-context learning approach that transfers the style of a source conversation to a target style using style-free conversations as pivots. Only a few non-parallel examples in source and target styles are needed for the purpose. We perform human and automatic evaluations to evaluate the style transfer quality for task-oriented dialogues on style strength, appropriateness, and semantic correctness. Quantitative and qualitative evaluations show that conversation style transfer yields more appropriate and semantically correct responses compared to utterance-level style transfer, which is crucial when applying to chatbot personalization. Finally, the usage of conversation style transfer for domain adaptation of training data for downstream intent classification task showed improvement in F1 score.
Limitations
We construct styled-to-style-free parallel conversations manually using human supervision. This may be expensive to do when there are a large number of style domains.
An automatic measure would be ideal for this purpose and this can be an interesting future work.

We ran our experiments only on one language, English. Various steps of the approach may be difficult to perform if style transfer is done in other languages as styles in different languages depend highly on social cultures and norms. That is mostly because most of the Large Language Models are pre-trained only on English text and may not perform well in other languages. Replicating this study in other languages may be an interesting future work.

New LLMs of different parameter sizes have been proposed in recent times. Replicating our study with other available LLMs of different parameter sizes can be an interesting future work.
Ethics Statement
In this paper, we did not annotate any new dataset rather we ran our models on publicly available datasets. The DSTC11 dataset is licensed under the Apache-2.0 License and the TWCS dataset is licensed under CC BY-NC-SA 4.0, both allow non-commercial use and distribution. The dataset references are cited and we provide detailed statistics of the dataset used.

The examples shown in Table 1 are from real customer care agents from different companies and are taken from the TWCS dataset. The examples from these companies were selected only for studying the problem using real data, the authors in this paper have no connection to these companies. Note that, the identity of the individual agents is hidden in the original dataset. Hence, it does not contain any personal identification information. The signatures of names at the end of the response by the Chipotle agents from the TWCS dataset are already altered to hide the actual identity of the agents.

We performed a human evaluation of our proposed models in this paper. We made sure that the human evaluation UIs do not impose any cognitive bias towards a specific model. We ensured that by hiding model names, shuffling orders of model outputs, and so on. We provide inter-annotator agreement scores and the detailed human evaluation process in the paper and in the appendix. Corresponding appendices are appropriately referred to in the paper.

The model descriptions and all hyper-parameter details are provided in the paper. Hence, we believe our results are reproducible.

Any generated texts that are reported as examples in this paper are the outputs of machine learning models and do not represent the authors’ or the organization’s viewpoints in any way.

Language models are pre-trained on large amounts of human-generated text. Hence, recent studies (Blodgett et al., 2020 ; Brown et al., 2020 ) have discussed that there may be inherent social and human biases in these models. However, probing the increasing number of Large Language Models for biases is a separate and broad research area and falls outside the scope of our study in this paper.
Acknowledgements
We gratefully acknowledge Justin Sun for his help in setting up the LLMs and the members of the AWS AI Labs for providing valuable feedback on the project. We express our gratitude to the AWS AI Data Team for supporting us with the human evaluation. We would also like the thank the anonymous reviewers for their insightful comments.
A.1 Prompt Structure
The structure of prompts for various versions of our model for converting a source conversation to style free conversation are shown in Figure 3 . The prompt structures for converting style free conversation to the target style are shown in Figure 4 .
A.2 Prompt Example
Examples for all types of prompt structures (as shown in Figure 3 and Figure 4 ) are shown in Figures 5 and 6 .
B.1 PMI-based Style Indicator Lemma Identification
For the identification of style indicator lemmas for each style domain, we use a Pointwise Mutual Information (PMI) Church and Hanks ( 1990 ) based approach. We first take all of the agent utterances from each style domain and lemmatize each word used by the agents using the spaCy Python library. We ignore all punctuations and stopwords. Then for a lemma, w we calculate the pointwise mutual information (PMI) with a style domain t , I(w,t) using the following formula.

 I(w,t)=\operatorname{log}\frac{P(w|t)}{P(w)}

Where P(w|t) is computed by taking all lemmas used in style t and computing \frac{count(w)}{count(all-lemmas)} and similarly, P(w) is computed by counting lemma w over the set of utterances in all styles. Now, we rank lemmas for each style domain based on their PMI scores. To remove topic-specific lemmas and rarely used lemmas, we ignore lemmas that are used in more than 10\% of the agent utterances in each style domain and used less than 0.5\%,0.3\%,0.3\% of the time in case of styles H_{1},B,H_{2} , respectively. The top 300 high PMI lemmas for each style domain are reported in Table 11 . Hand-picked style indicator lemmas from this top 300 list are shown in Table 4 .
B.2 Construction of parallel style free conversations using human supervision
A human annotator was presented with 5-7 conversations from each of the style domains ( B,H_{1},H_{2} ) and they were asked to rewrite those conversations in a style-free form. One parallel style-free example per style domain written by the human annotator is shown on the right-hand side of Figure 7 . The human annotator is a researcher in NLP and it took approximately 5 minutes for them to rewrite a 10-12 turns conversation in a style-free format. These style-free parallel conversations are used for in-context learning as described in Section 3 . The statistics of the annotated few shot examples per style domain are shown in Table 12 .
D.1 Data Selection for Human Evaluation
Our goal with human evaluation is to compare different models. We used the test dataset described in Table 5 for human evaluation. Note that the same conversation segments are used to evaluate various versions of our model and the baseline using GPT-NeoX and Bloom as LLMs. We evaluate only agent responses and we apply two types of filtering step on these datasets before human evaluation.

Filtering Step 1: When doing style transfer at 4/5-turn conversation level, it may result in non-parallel conversation compared to the source conversation because of turn reduction by the model. To match the non-parallel utterances with the source utterances, we rank the style transferred utterances based on their semantic similarity with the source utterances and pick the one with the highest similarity. We discard any style transferred utterance that has the highest semantic similarity of less than 0.2. Looking manually at those utterances it was observed that those were unrelated utterances generated by the LLMs.

Filtering Step 2: We filtered out all agent responses where none of the models (including the baseline) changes the source agent utterances or when the style transferred versions were the same from all models.

Application of the above two filtering steps resulted in 100 + agent utterances in each style direction. We perform the human evaluation on this filtered set. The statistics of the data after each filtering step is shown in Table 14 .
D.2 Human Evaluation Settings
Each data point was evaluated by three human evaluators. We worked with professional data linguists who are fluent in English. They were compensated at hourly basis which was in accordance with the standard compensation rate in the United States. They were first trained on the tasks. Specifically, they were briefed on what we mean by style strength, appropriateness, and semantic correctness. Worked-out examples were provided to them. The model names were hidden from the annotators and the four versions were presented in a randomly shuffled order for each example. For ranking-based evaluation in style strength and appropriateness, the human evaluators were instructed to rank the various style-transferred versions from various models based on their style strength and appropriateness. For example, when evaluating among 3 models, a rank of 1 means it has the highest style strength or appropriateness and a rank of 3 means the lowest style strength or appropriateness. The annotators were instructed to provide two style-transferred versions the same rank if they were equal in style strength or appropriateness. For the evaluation of semantic correctness, the human evaluators were presented with the source utterance and the style-transferred versions of the source utterance by each of the models. Then we asked them for each style transferred version if it is semantically similar, partially similar, or dissimilar to the source utterance. Each data point in all of the evaluation metrics is evaluated by three human evaluators. The annotation UIs for style strength, appropriateness, and semantic correctness evaluation tasks are shown in Figures 8 , 9 , 10 , respectively.
D.3.1 Style Strength and Appropriateness
For measuring the inter-annotator agreement in ranking evaluations for style strength and appropriateness, we use Spearman’s Rank Correlation Coefficient Zar ( 2005 ) . We take the average Spearman’s Rank Correlation Coefficient between each pair of human annotators for each data point as an agreement measure. It ranges from -1 to +1 where -1 means absolute disagreement and +1 means absolute agreement.
D.3.2 Semantic Correctness
For measuring inter-annotator agreement in semantic correctness evaluation task which is categorical, we use Krippendorff’s \alpha Krippendorff ( 2004 ) . It ranges from -1 to +1 where \alpha = +1 means perfect agreement, and \alpha = -1 means no agreement.
D.3.3 Agreement Scores
The inter-annotator agreement in all of the tasks are shown in Table 6 . Note that, for calculating agreement in the semantic correctness evaluation task, all of the data points are aggregated to measure the agreement score as they represent categorical evaluation measures. On the other hand, that is not possible in case of ranking based evaluations for style strength and appropriateness. So, we measure the agreement for each data point and take the average agreement over all data points. We can see in the Table 6 that in all of the cases we get strong agreement ( >0.70 ) among the annotators for the style strength and appropriateness evaluation. The only exception is the case of style strength evaluation task in the direction of H_{1} \rightarrow H_{2} , using the GPT-NeoX model. The agreement score is slightly lower ( 0.69 ) in this case. Our insight is that these two directions are basically human styles and difference between them is very subtle. As a result, it is difficult for humans as well to differentiate among them. This pattern is observed when doing the automatic evaluation as well.

In case of semantic correctness evaluation task, we always get strong agreement among annotators ( >0.75 ).
D.4 Scaling Ranking Scores
In the style strength and appropriateness evaluation tasks we use ranking based measure among the output from various models. For example, when evaluating among 3 models, a rank of 1 means it has highest style strength or appropriateness and a rank of 3 means the lowest style strength or appropriateness. We scale these rank scores in the range between 0 to 1 where a higher score means higher style strength or appropriateness. The ranking were scaled for each data point using the following formula.

For each data point, if the number of versions to be ranked is k and ranking of a version i (i\in{1,...,k}) is r_{i} , then the reverse rank score, r_{i}^{rev}=k-r_{i}+1 . Now, the scaled rank score, r_{i}^{scaled}=\frac{r_{i}^{rev}-\min_{j\in 1,...,k}r_{j}^{rev}}{\max_{j\in 1,...,k}r_{j}^{rev}-\min_{j\in 1,...,k}r_{j}^{rev}} . We average over all human evaluators’ scaled ranking score to get the final scaled ranking score for a data point.
D.5 Pairwise Comparison Among Models
The pairwise comparison among various versions of the models for style strength and appropriateness are shown in Table 15 . This table represents the statistics on the percentage of time a model is ranked higher in the style strength and appropriateness evaluation by humans, than the other in a pair-wise manner.
E.1 Dataset
We take the three domains of DSTC11 dataset namely, Insurance, Banking and Finance for this task. In this dataset, mostly the customer utterances are annotated for intents. We take the human-to-human conversations as training data and human-to-bot conversations as test data. We consider intent classes having at least 20 training utterances for this study. Then we randomly select 90\% of training data from each intent class as training set and select rest of the 10\% as validation set. The training, test and validation data statistics for each of the domains are shown in Table 16 .
E.2 Few-Shot Style Transfer of Training Data
In this dataset, mostly customer utterances are annotated for intent classes. So, we perform few-shot style transfer of the customer utterances only, using the same procedure that we followed for agent utterances style transfer. We found out that customers are more conversational when talking to a human agent compared to when talking to a chatbot agent. So, we use few-shot customer utterances from the human-to-bot conversations to transfer the style of customers in human-to-human conversations. Then use this style transferred data for training an intent classifier. We use a 10-shot setting with dynamic prompt selection based on semantic similarity.
E.3 Intent Classification Results
We compare the performance of the intent classifier when trained on human-to-human conversations vs. training on human-to-human conversations that are transferred to human-to-bot style. We ran an ablation where we experimented with utterance level style transfer and 2-turns conversation level style transfer as these two methods yielded better style strength in our studies. We ran this ablation using only banking and finance domains out of the thee domains. The classification was done 10 times with 10 random seeds for each domain. A RoBERTa-based Liu et al. ( 2019 ) text classifier was used to perform the intent classification task. We encoded each utterance using RoBERTa where the embedding of the [CLS] token of the last layer was used as a representation of the utterance. This representation was used for intent classification. The average classification results are shown in Table 17 . Overall, the utterance level style transfer yields the best intent classification results as it achieves the best style strength of the test domain (human-to-bot style).