RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Figure 1: Transferring the style of an utterance from chatbot to human agent style based on three utterance-level (single-turn) and conversation-level (multi-turn) examples as input. Additional conversational context helps the style transfer model to yield a more appropriate response as the dialogue contains useful information that can be leveraged during the generation process.; Transferring the style of an utterance from chatbot to human agent style based on three utterance-level (single-turn) and conversation-level (multi-turn) examples as input. Additional conversational context helps the style transfer model to yield a more appropriate response as the dialogue contains useful information that can be leveraged during the generation process.
Figure 2: The proposed two-step in-context learning-based approach for conversation style transfer: (Step 1) The style in the source conversation is reduced and it is converted to a style-free conversation. (Step 2) The style-free conversation is converted to the target style. Both conversion steps are learned in context.; The proposed two-step in-context learning-based approach for conversation style transfer: (Step 1) The style in the source conversation is reduced and it is converted to a style-free conversation. (Step 2) The style-free conversation is converted to the target style. Both conversion steps are learned in context.
Figure 3: Prompt structure for transferring a source conversation to a style free conversation.; Prompt structure for transferring a source conversation to a style free conversation.
Figure 4: Prompt structure for transferring a style free conversation to the target style.; Prompt structure for transferring a style free conversation to the target style.
Figure 5: Prompt examples for transferring a source conversation (in chatbot agent style, B ) to a style free conversation using various versions of our model. For simplicity, 3-shot, 3-shot, and 2-shot prompts are shown in case of utterance level style transfer, 2-turns conversation level style transfer and 4/5-turns conversation level style transfer, respectively.; Prompt examples for transferring a source conversation (in chatbot agent style, B ) to a style free conversation using various versions of our model. For simplicity, 3-shot, 3-shot, and 2-shot prompts are shown in case of utterance level style transfer, 2-turns conversation level style transfer and 4/5-turns conversation level style transfer, respectively.
Figure 6: Prompt structure for transferring a style free conversation to the target style (in human agent style, H_{1} ) using various versions of our model. For simplicity, 3, 2, 1 shots prompts are shown in case of utterance level style transfer, 2-turns conversation level style transfer and 4/5-turns conversation level style transfer, respectively.; Prompt structure for transferring a style free conversation to the target style (in human agent style, H_{1} ) using various versions of our model. For simplicity, 3, 2, 1 shots prompts are shown in case of utterance level style transfer, 2-turns conversation level style transfer and 4/5-turns conversation level style transfer, respectively.
Figure 7: Example conversations from three domains ( B,H_{1},H_{2} ) are shown in the left hand side. Human annotated style-free versions of the corresponding conversations are shown in the right hand side. This parallel data is used for in-context learning. Here, H_{1},H_{2},B refer to human style from DSTC11 dataset, style of Chipotle agents (from TWCS dataset), and Chatbot style from DSTC11 dataset, respectively.; Example conversations from three domains ( B,H_{1},H_{2} ) are shown in the left hand side. Human annotated style-free versions of the corresponding conversations are shown in the right hand side. This parallel data is used for in-context learning. Here, H_{1},H_{2},B refer to human style from DSTC11 dataset, style of Chipotle agents (from TWCS dataset), and Chatbot style from DSTC11 dataset, respectively.
Figure 8: Human evaluation UI for the evaluation of style strength.; Human evaluation UI for the evaluation of style strength.
Figure 9: Human evaluation UI for the evaluation of appropriateness.; Human evaluation UI for the evaluation of appropriateness.
Figure 10: Human evaluation UI for the evaluation of semantic correctness.; Human evaluation UI for the evaluation of semantic correctness.
(a) Style transfer from human style ( H_{1} ) to chatbot style ( B ).; (b) Style transfer from one human style ( H_{1} ) to another human style ( H_{2} ).; (c) Style transfer from one chatbot style ( B ) to human style ( H_{1} ).; (d) Style transfer from one chatbot style ( B ) to human style ( H_{2} ).; Style transfer examples in all directions by various versions of our model. Here, H_{1},H_{2},B refer to human style from DSTC11 dataset, style of Chipotle agents (from TWCS dataset), and Chatbot style from DSTC11 dataset, respectively.
Figure 12: Example conversations in styles H_{1},B,H_{2} . These three styles are holistically different. Some properties of the human styles are that they are conversational, sometimes formal, and sometimes casual and friendly. For example, the human style, H_{2} is informal and friendly while the other human style, H_{1} is formal while both of these two human styles are conversational. In human style H_{2} , agents sign their names at the end of a response preceded by a hyphen. In the other human style, H_{1} , this stylistic property is not observed. Some observed properties of the bot style are crispness and to-the-point while not being informal. The conversations in human style H_{1} and bot style B presented in this table, are on the same situation showing the holistic difference between these two styles. Note that, the other human style H_{2} is from a different domain, hence, a conversation on a similar situation could not be found.; Example conversations in styles H_{1},B,H_{2} . These three styles are holistically different. Some properties of the human styles are that they are conversational, sometimes formal, and sometimes casual and friendly. For example, the human style, H_{2} is informal and friendly while the other human style, H_{1} is formal while both of these two human styles are conversational. In human style H_{2} , agents sign their names at the end of a response preceded by a hyphen. In the other human style, H_{1} , this stylistic property is not observed. Some observed properties of the bot style are crispness and to-the-point while not being informal. The conversations in human style H_{1} and bot style B presented in this table, are on the same situation showing the holistic difference between these two styles. Note that, the other human style H_{2} is from a different domain, hence, a conversation on a similar situation could not be found.
Example of conversations of customer care agents from the TWCS dataset that show the limitation of style definitions using fixed attributes, here Politeness. Chipotle and VirginTrains customer care agents get roughly the same politeness score by an off-the-shelf politeness classifier Danescu-Niculescu-Mizil et al. ( 2013 ) , however, intuitively their style attributes are different as shown in the third column.
Example of style transfer evaluation metrics - style strength, appropriateness , and semantic correctness , by comparing three style transferred versions of the same agent utterance.
Inappropriate and semantically incorrect segments of the generated utterances are marked in red.
Quantitative differences among styles H_{1} , H_{2} , and B . Human agents ( H_{1} , H_{2} ) are more conversational and use diverse words compared to bots ( B ). The bot style is very crisp and to-the-point. Apart from these properties, in human style, H_{2} agents sign their names at the end of the response 98\% of the time.
Validation and test data statistics. Long conversations are divided into small segments consisting of 4-5 turns. We cover four style transfer directions to/from two human styles ( H_{1} , H_{2} ) and bot style ( B ).
Inter-annotator agreement scores for the three human evaluation tasks. Standard deviations over all data points are shown in brackets for the style strength and appropriateness evaluation tasks. The detailed procedure for calculating the agreement scores can be found in Appendix D .
Comparison between dynamic and random prompt selection on target style strength across utterance and conversation level style transfers.
Human evaluation results for utterance level (baseline) and conversation level style transfer with GPT-NeoX and Bloom LLMs using our model. The best average scores over all style dimensions are marked in bold. Utterance level style transfer achieves higher style strength but conversation level style transfers yield more appropriate and semantically correct responses. Statistics with standard deviations can be found in Appendix G .
Qualitative error analysis on hand-picked examples. Human evaluation scores associated with the output of various versions of the model (using Bigscience-Bloom as the LLM) are shown in the three rightmost columns. A score of 1 means the most appropriate or most target style strength.
Intent classification results in terms of F1 score. Transferring the training data (human-to-human style) to test data style (human-to-bot style) improves the test F1 score in three domains: Insurance, Banking, and Finance. The significance of difference, p -values for Insurance and Banking are p<0.05 and p<0.01 , respectively. For Finance the improvement is non-significant.
High PMI lemmas for each style domain. Bots ( B ) do not use many non-topic-specific words. Mostly formal words are used in human style H_{1} and many informal and friendly words (e.g., bummer) are used in human style H_{2} .
Manually created few-shot examples summary. The data is used for in-context learning as described in Section 3 . Here, H_{1},H_{2},B refer to human style from DSTC11 dataset, style of Chipotle agents (from TWCS dataset), and Chatbot style from DSTC11 dataset, respectively.
Ablation study for selecting number of shots and prompt selection method. Here, "N/S" means "Not Supported" because of token limit in prompt. GPT-NeoX was used as the base LLM in this ablation study. Dynamic prompt selection technique outperforms random prompt selection in all of the cases. The optimal number of shots for utterance level style transfer, 2-turns conversation level style transfer, and 4/5-turns conversation level style transfer are 10, 10, and 8 respectively.
Dataset statistics for human evaluation.
Human evaluation results on style strength and appropriateness. The table presents a pair-wise comparison among three versions of our model - utterance level style transfer (denoted as U ), 2-turns conversation level style transfer (denoted as C_{1} ), and 4/5-turns conversation level style transfer (denoted as C_{2} ). Each cell represents the \% of time a model is ranked higher than the other by the human evaluators. For example, column U>C_{1} represents the \% of time the utterance level style transfer model is ranked higher than the 2-turns conversation level style transfer.
Intent classification dataset statistics.
Detailed Intent classification results. The ablation between two types of models - utterance level style transfer and 2-turns conversation level style transfer was performed on two domains - banking and finance. Overall, utterance level style transfer yields the best intent classification F1 scores as it achieves the highest style strength score as the test domain (human-to-bot).
Human evaluation results for utterance level (baseline) and conversation level style transfer with GPT-NeoX and Bigscience-Bloom LLMs using our model. The best average score over all style dimensions are marked in bold. Utterance level style transfer achieves higher style strength but conversation level style transfers yield more appropriate and semantically correct responses.
Automatic style strength and semantic correctness evaluation results for utterance level (baseline) and conversation level style transfer with GPT-NeoX and Bigscience-Bloom LLMs using our model. Utterance level style transfer achieves higher style strength and conversation level style transfers yield more semantically similar responses.
Error analysis of Bigscience-Bloom when performing 4/5-turns conversation level style transfer. Sometimes customer and agent turns are swapped as shown in the upper conversation and sometimes unrelated agent responses are generated as shown in the lower conversation. Here, H_{1},H_{2},B refer to human style from DSTC11 dataset, style of Chipotle agents (from TWCS dataset), and Chatbot style from DSTC11 dataset, respectively.
Effects of style transfer on the observed style properties such as crispness, diversity in vocabulary, and signature at the end of responses (as described in Table 4 ). Note that, signing names at the end of a response is a unique structural style property of the style H_{2} , hence, this style property is obtained by the models only when transferring a source style to H_{2} . The statistics are obtained on the test set as described in Table 5 .