Conditioning on Dialog Acts improves Empathy Style Transfer
Authors: Renyi Qu
Lyle Ungar
João Sedoc
Venue: EMNLP 2023 Findings (2023)
Keywords: empathy style transfer
text style transfer
empathy
GPT-4
large language models
dialog acts
pragmatics
prompt engineering
in-context learning
few-shot prompting
TL;DR: In empathy style transfer with large language models, conditioning few-shot examples on dialog acts improves style transfer strength and content preservation.

Abstract: We explore the role of dialog acts in style transfer, specifically empathy style transfer -- rewriting a sentence to make it more empathetic without changing its meaning. Specifically, we use two novel few-shot prompting strategies: target prompting, which only uses examples of the target style (unlike traditional prompting with source/target pairs), and dialog-act-conditioned prompting, which first estimates the dialog act of the source sentence and then makes it more empathetic using few-shot examples of the same dialog act. Our study yields two key findings: (1) Target prompting typically improves empathy more effectively while maintaining the same level of semantic similarity; (2) Dialog acts matter. Dialog-act-conditioned prompting enhances empathy while preserving both semantics and the dialog-act type. Different dialog acts benefit differently from different prompting methods, highlighting the need for further investigation of the role of dialog acts in style transfer.

Conditioning on Dialog Acts Improves Empathy Style Transfer
Renyi Qu', Lyle Ungar', João Sedoc', Uniiversity of Pennsylvania, "New York University (requ, @upenn. edu, jsedoc@stern, nyu edu
Abstract
I've had hard time trying to cape wi everydhing that's been haopening.
can you tell what happened?
We explore the role of dialog acts in style trans- fer, specifically empathy style transfer rewrit- ing sentence make more empathetic with- out changing its meaning. Specifically, we use two novel few-shot prompting strategies: targer prompting, which only uses examples the target style (unlike traditional prompt- ing with source/target pairs); and dialog-uu act- conditioned prompting, which first estimates the dialog act of the source sentence and then makes it more empathetic using few-shot amples of the same dialog act. Our study yields two key findings: (1) Target prompting typically improves empathy more effectively than pairwise prompting, while maintaining the same level of semantic similarity; (2) Dialog acts matter. Dialog-act-conddttonedd prompt- ing enhances empathy while preserving both semantics and the dialog-act type. Different dialog acts benefit differently from different prompting methods, highlighting need further investigation of the role of dialog acts in style transfer.
Ohiginal Resporse
User
Your input empathetic possible exarpless
change
happaned.
axarples)
Wiena dialog act Loss ampathatic.
-
System
GPT-1
(a) Empathy transfer with pairwise prompting
I' vu had u hard timn trying to cope wf everyching that's been happentng.
Can you tell me what happened?
8 User
Crigiaal Respanse
siahno queitoo
nowr the Input yes/no question the espachatic possible. not change the content. yes/no question rewritingg
Clasiifier
would you sharing
what hasoened?
highly
1 Introduction
yes/no quest
(3x yas/no questions)
Conrect Mar empallatiic
Expressing empathy important for communica- tion in applications ranging from customer service to mental health. Although there no universal definition of empathy, most definitions share a core having the ability to understand and experience the feelings of others (Cuff et al., 2016). Recent research has explored empathetic response gener- ation, where the language model generates re- sponse that conveys both the intended message and a sense of empathy given the input context (Ma- jumder et al., 2020; Liu et al., 2022; Zheng al., 2021; Sabour et al., 2022). However, empathetic response generation is not applicable to more sensi- tive cases such as medicine and therapy, where the content of the responses requires the supervision
Byskan
(b) Empathy transfer with DA-target prompting (explicit)
Figure 1: Example of conditioning few-shot examples and prompt format on the source dialog act. (a) illus- trates pairwise prompting, which few-shot example pairs for all inputs; (b) illustrates DA- target prompting (explicit), which identifies the source dialog act and changes the prompt accordingly, shown in red, resulting in better performance in both empathy transfer strength and content preservation.
common set of
of medical experts. A key issue for state-of-the-art conversational models hallucination (Dziri al., 2022; Azamfirei et al., 2023), where the models provide fake or incorrect information in their re-
'Of course, empathic chatbots do not actually experience feclings; they merely speak as if they did.

---

sponses, Physicians often (reasonably) do not trust the content of compuler-generaal responses but use large language models (LLM) such as GPT- 4 (OpenAI, 2023) to rephrase their responses to express more empathy.
act-conditioned prompting: (a) selecting few-shot examples based on the dialog act of the source sen- tence and (b) explicitly specifying the dialog act in the prompt format in addition to (a). We test our prompting strategies and focus our evaluations on empathy transfer strength and content preservation. In summary, we find that:
This is style transfer: paraphrasing a sentence from style to another without changing its content (Table 1). The input sentence is typically named the source sentence or "prompt", and the output sentence is typically named the target sen- tence. Style is, unfortunately, often entangled with content: being polite, rude, or angry is hard to sep- arate from c.g. the substance of product review. Many researchers have worked on this task, but it remains challenging because of many unsolved is- sues, including the lack of parallel data and the lack of reliable evaluation metries for different styles (Toshevska and Gievska, 2021; Hu al., 2022; Jin et al., 2022; Lample et al., 2019).
1. Target prompting generates higher empathy than pairwise prompting while preserving a similar level of content
2. Dialog acts matter: conditioning the prompt on them when doing style transfer increases empathy while better preserving source dialog acts. Different dialog acts benefit differently from different prompting methods
Related Work
Fhw-shot prompting, also called "in-context learning", has proven remarkably good at style transfer, giving performance competitive with more data-intensive fine-tuning (Reif et al., 2022). De- spite having some limitations (Webson and Pavlick, 2022), prompting attractive in eliminating the need for extensive curated data, Previous work compares zero-shot and pairwise few-shot prompt- ing (where there are, for example, content-matched pairs of high and low empathy responses) on text style transfer (Reif et al., 2022). Zero-shot prompt- ing is prone generating illogical nonsensi- cal outputs, while pairwise few-shot prompting re- quires optimal choices of sampled sentence pairs to maximize its performance (Suzgun et al., 2022).
Empathetic Response Generation Many re- searchers developed empathetic dialogue systems as this may improve the overall usefulness of AI systems (Czerwinski et al., 2021). These generally rely on datasets of empathetic dialogs, with early datasets (Rashkin et al., 2019) being followed by large-scale (1M empathetic dialogues) (Welivita et al., 2021) and multi-level datasets of conversa- tions (Omitaomu al., 2022). Recently, Welivita et al. (2023) created an empathetic response dataset for distress-related topics with comprehensive eval- uation metrics. The availability of spurcsource datasets allowed the creation of many transformer- based empathetic response generation models (Li et al., 2020b; Majumder et al., 2020; Liu et al., 2022; Zheng et al., 2021; Sabour et al., 2022), all of which directly generate empathetic responses from the input prompt. Qian al. (2023) discov- ered that a two-stage system (response generation and style transfer) can yield better performance than one-stage models. However, a two-stage sys- tem requires two language models to be separately trained on different datasets. More recently, Lee et al. (2022) confirmed that LLMs are capable of generating empathetic sentences through zero-shot prompting, reducing the burden of training and shifting the focus to prompt engineering.
In this paper, we investigate empathy style transfer using prompt-based inference with GPT- 4. We propose two sets of novel prompting approaches: target prompting and dialog-act- conditioned prompting. Target prompting is in- spired by the difficulty of collecting counterparts of empathetic sentences, and uses sentences of only the target style as its examples (i.e., highly empathetic sentences). It contrasts with the more common pairwise prompting, where the prompts contain semantically matched pairs of high and low empathy responses. Dialog-act caaloggact-conditionee prompting is inspired by the fact that dialog acts play a crucial role in pragmatics and convey es- sential information about our cognitive processes, which are closely related to our empathy expres- sions (Bunt, 201 1). We propose two types of dialog-
Text Style Transfer One conventional way do text style transfer was style-content disentangle- ment, where encoder-decoder structures such as Variational Autoencoders (Bao et al., 2019; Liu

---

Entity
Empathy
Content
input sentence
2.74
Yes I'm worried that things like that will continue to happen in the US also.
zero-shot response
1.97
I understand that such events will persist in the US as well.
few-shot target
2.84 I share your concern that such events may persist in the US, causing distress and unease among its people.
prompting response
few-shot DA-target
3.23
I can genuinely understand the concern that events like these may persist in the US, causing sense of unease and apprehension in our hearts.
prompting response
Table 1: Example of input and responses generated by GPT-4 in three different conditions. The computed empathy scores show that the responses have higher empathy when using target prompting. The exact methods are described later in the paper, but "DA-target prompting" uses the dialog act of the prompt.
et al., 2020; Yi et al., 2021) and transformers (Sud- hakar et al., 2019; Dai et al., 2019; Lee, 2020) were used to alternate the style embeddings in order to do style transfer. Following the rapid development of transformer models, many rescarchers conducted experiments on more specific styles such as senti- ment (Wu et al., 2019; Pant et al., 2020), formal- ity (Yao and Yu, 2021; ;; Lai et al., 2021), and polite- ness (Madaan et al., 2020). The most recent work on empathy style transfer is Zhang et al. (2022), where they used empathy transformer and dis- criminator to tune the empathy level. However, all the methods above require data availability and training, making them less efficient.
between dialog acts and sentiments. Later, Saha et al. (2020) incorporated emotions into dialog act classification and confirmed that emotions affect our dialog act choices; and Bothe al. (2020) able build neural annotators jointly annotate emotions and dialog acts. More recently, Li et al. (2022) found out that dialog act prediction signif- icantly improved depression detection, hinting at the strong potential of dialog acts in understanding the emotional and mental aspects of discourse.
Dialog acts are rarely, if ever, used text style transfer, but their use has enhanced performance in other natural language generation tasks (Wang et al., 2020; Peng et al., 2020; Gella et al., 2022). Motivated by these papers, we believe that using dialog acts will benefit empathy style transfer.
More recently, prompt-basee based inference with LLMs was significantly more efficient than fine- tuning, and it eliminated the issue of lack of data availability in text style transfer. Reif al. (2022) used the prevalent zero-shot and few-shot prompt- ing with GPT-3 (Brown et al., 2020) for arbitrary style transfer. Suzgun et al. (2022) prompts the large language model multiple times to generate multiple responses and reranks them based on three evaluation metrics: transfer strength, semantic sim- ilarity, and fluency. However, our knowledge, none of the text style transfer work has considered dialog acts so far.
Methods
3.1 Problem Setting
Prior work on empathy-and most other-style trans- fer has considered the source and target styles two different styles in binary setting, focusing on the two opposite ends of the style scale, while ig- noring the middle area. In contrast, we assume that source and target styles are of the same type but at different positions on a continuous, real-valued scale. Discrete style labels miss the continuous nature of the empathy style (Cuff et al., 2016; Lah- nala et al., 2022). Hence, we formulate our empa- thy transfer task as follows. Given a target style s and sentence z with degree of style polarity p. empathy style transfer outputs sentence y with the highest attainable polarity p' of the same style while adequately preserving the sentence seman- tics. this paper, we are transferring from lower to higher empathy.
Dialog Act Dialog act is an indicator of the intent purpose of a speaker's utterance within a conver- sation. It has long been acknowledged that under- standing dialog acts is crucial for discourse analy- sis, especially for pragmatic aspects like sentiments and emotions. Cerisara et al. (2018) first attempted to model the joint distribution of dialog acts and sentiments, and Li et al. (2020a) further enhanced the model and acknowledged the strong correlation

---

3.2 Prompting Strategies
Category
Dialog Act Forward Communication
Table 2 summarizes the few-shot prompting meth- ods used in this paper. For more details on the prompt formats, please refer to Table the Ap- pendix. We are interested in two branches of com- parison: target vs pairwise prompting and dialog- act-conditioned vs direct prompting. The first branch differs in the structure of few-shot examples, while the second branch differs in the conditional- ity of few-shot examples the dialog act of the source sentence.
Statement
sd, sv
Question
qo. qr, qw, qy fa, fc, fp. ft, fx, qh
Other Forward
Backward Communication
Agreement Understanding b, ba, bf, bh, br, by, "2 Answer
aa,
na, ng, nn, no
Other
Other
"q. h, t, %. 13, 'none' (unclassified)
Table 3: Categorization of dialog acts in order to reduce complexity in our analyses. These categories are mostly the same as Jurafsky et al. (1997). Only the dialog acts involved in the EC dataset are included. Acronyms are defined in Appendix Table 5.
Target prompting Inspired by the challenges in pairwise prompting, we use only the target sen- tences as the few-shot examples without any pair- wise comparison. Since non-parallel data is abun- dant, this method eliminates the effort of searching for or manually creating high-quality counterexam- ples for style transfer.
4
Experiments
4.1 Data
We evaluate our prompting methods on the Em- pathic Conversations (EC) dataset (Omitaomu et al., 2022), which consists of 500 conversations (1 (11,778 utterances) between Amazon MTurkers on 100 articles various tragic events. Each arti- cle contributes total of 5 conversations. Each conversation consists of 20-25 utterances between MTurkers with its topic strictly focused on the assigned article. This data mimics our daily interac- tions with empathetic scenarios extremely well and is optimal for our empathy transfer experiment.
DA DA-connitioned prompting We expand pairwise and target strategies two-stage process by con- ditioning the prompt dialog acts. First, iden- tify the dialog acts of the source sentence using a dialog act classifier. Then, we map the identified dialog act to its corresponding few-shot examples. Since the examples are of the same dialog act, we expect the model to perform empathy transfer bet- ter in terms of naturalness, coherence, and content preservation of the source sentence. Because do not tell the LLM what the source dialog act is, this approach is considered "implicit".
use a pretrained Before the experiment, RoBERTa (Liu et al., 2019) evaluator to calculate the empathy scores of all utterances a continuous range of 0-4, and we use a pretrained RoBERTa multilabel classifier to predict the dialog acts of all utterances from the set of 42 labels from Switch- board (Jurafsky et al., 1997). When an utterance is classified with multiple dialog acts, we select the one with the highest probability. When an utterance is classified with no dialog act due to ambiguity, put it into the none category.
As an extension to the implicit approach, we ex- plicitly specify the source dialog act in the prompt (example shown in Figure I). Specifically, we change the identifier words in the original prompt into the dialog act name, and add one extra sen- tence to emphasize the preservation of dialog acts. This approach is considered "explicit".
3.3 Dialog act categorization
4.2 Conditionality of Empathy on Dialog Acts
We present the categorization for dialog acts in Ta- ble 3, similar to Jurafsky et al. (1997) with some modifications. Rhetorical questions (qh) are ex- cluded from the Question category because they function differently from normal questions in more similar way to other forward expressions. Not all dialog acts are included in the EC dataset, so we excluded the nonexistent ones from the table.
Our exploratory analysis of the EC dataset indicates strong correlation between empathy strength and dialog acts (Figure 4) and demonstrates the follow- ing:
Opinions, Facts, and Agreements have near-
Pleaae refer to (Omitaomu et al., 2022) for the perfor- mance of the classifiers.

---

Prompting Method
Description
Pairwise
Each example consists of one unempattheii---eppahheiiic sentence pair of high
semantic similarity.
DA- -A-Pairwise (implicit)
Pairwise prompting where each example shares the source dialog act.
DA- PA-Pairwise (explicit)
Pairwise prompting where each example shares the source dialog act and the prompt explicitly specifies the source dialog act.
Target
Each example consists of one empathetic sentence.
DA- Target (implicit)
Target prompting where each example shares the source dialog act.
DA DArTattet (explicit)
Target prompting where each example shares the source dialog act and the prompt explicitly specifies the source dialog act.
Table Few-shot prompting methods. Pairwise prompting the conventional prompting method used in text style transfer, while the other are novel variations.
Normal empathy score distributions
to highly empathetic samples. Appendix Figure 3 shows the empathy score distribution and dialog act histogram for the selected test samples.
Wh-Questions Yes/No Questions, and Open Questions have empathy scores skewed to the lower end, while Rhetorical Questions have empathy scores centered in the middle
Forward Closings have empathy scores heav- ily skewed to the lower end, while Backward Sympathy expressions have empathy scores heavily skewed to the higher end
40
share (%)
data
In addition, low empathy sentences are mainly Forward Closing sentences (fc), while high em- pathy sentences mainly Opinions (sv), Facts (sd), and Agreements (aa) (Figure 2). Different di- alog acts have different empathy score distributions (Appendix B).
low
dialeg act
4.3 Sample Selection
Figure 2: Shares of most frequent dialog acts in the 2000 utterances of the highest empathy (blue) and the lowest empathy (orange) in the EC dataset. Opinions (sv), Facts (sd), Agreements (aa), and Backward Sym- pathy (by) appear much more often highly empathetic sentences, while Questions (go, qy) and Forward Clos- ings (fc) appear much more often lowly empathetic sentences, indicating the dependency of empathy on dialog acts.
Test sample selection If we conduct our experi- ment only on the utterances with the lowest empa- thy scores, the results are largely biased towards Forward Closing sentences (Figure 2). To miti- gate this bias, we sort all conversations based the mean empathy score over all utterances within each conversation. We then select the 41 conversa- tions (1016 uttcrances) with the lowest mean em- pathy scores. This is beneficial for two reasons. First, each conversation is long enough to include reasonable variety of dialog acts, thus mitigating sample biases towards any specific dialog act. Sec- ondly, still obtain reasonable set of utterances with low empathy scores, which is more suitable for evaluating empathy transfer strength compared
Few-shot example selection We sample 3-shot examples from GPT-4 by prompting to generate high and low empathy sentences. A sentence with an empathy score higher than 3 (75% of total score) is accepted as highly empathetic example, and sentence with an empathy score lower than (25% of total score) is accepted as a low empathy example.
Foor simplicity, we refer to Opinion Statements Opin- ions and Non-opinion Statements as Facts.
For Target prompting, we directly use the quali-

---

fied highly empathetic examples as the few-shot ex- amples. For Pairwise prompting, we take the same highly empathetic examples from Target prompt- ing and ask GPT-4 to rewrite them into an unem- pathetic version. We repeat this until the rewritten version satisfies the score constraint. Then, we pair the original empathetic sentences and the rewritten unempathetic sentences together our few-shot examples.
accuracy. The basic idea of empathy style transfer is to make sentence more empa- thetic, so a decrease in empathy score con- sidered a failure.
Content Preservation While transfer strength is crucial, we do not want to overly change the con- tent of the source sentence. We use the following metrics to assess content preservation:
For direct prompting, we use the pretrained RoBERTa classifier to ensure that the selected few- shot examples have different dialog acts from each other to avoid possible conflict with the examples used in DA-conditioned prompting, We use the same qualified examples across all input dialog acts. For DA-conditioned prompting, explicitly instruct GPT-4 to generate sentences of the selected dialog and validate their dialog acts using the classifier.
BERTScore: We obtain the Fl score from BERTScore (Zhang al., 2019) for each pair of generated and source sentences, and we report the average over all test samples. BERIScore calculates text similarity token by token based on contextual embeddings and outperforms traditional semantic similar- ity metries. We use deberta-xlarge-minn (He al., 2020) as the best-performing variation.
4.4 Model
We calculate BLEURT for each pair of generated and source sentences in the range [0, I]. and we report the average over all test samples. BLEU scores (Papineni et al., 2002) notorious for not resembling human judgments on semantic similarity (Callison- Burch et al., 2006), so we use BLEURT-20, a BERT-based BLEU variation trained on many synthetic examples (Sellam et al., 2020), to get similarity scores closer to human judg- ments.
BLEURT:
We use GPT-4 (OpenAI, 2023) to perform empathy style transfer. Each test sample is combined with the predefined prompt formats shown in Figure I as the full prompt, and GPT-4 generates transferred sentence with a temperature of 0 and maximum length of 32. In addition to the few-shot prompting methods in Table baseline, where we directly ask GPT-4 to rewrite the source sentence with no example.
zero-shot prompting as
4.5 Evaluation
We follow the three categorics of evaluation crite- ria specified in Hu et al. (2022y: Transfer Strength, Content Preservation, and Fluency. We also intro- duce a new metric to monitor GPT-4's tendency to add tokens during style transfer. All of the follow- ing evaluation metrics are automatic.
DA-Acc: We calculate the share of samples with dialog act unchanged after style transfer as the accuracy for dialog act preservation.
Other attributes We use the following metrics for additional evaluation:
Transfer Strength Transfer strength directly shows how good the style transfer method is. We pretrained RoBERTa evaluator to calculate the empathy scores of the generated sentences and compare them with the source sentences. We use the following metrics assess transfer strength:
We calculate the difference in the number of words for each pair of generated and source sentences and report the average over all test samples. During the experiment, we noticed that GPT-4 tends to add many to- kens to reach the desired outcome for style transfer, so we included this metric to monitor such behavior.
ALength:
Ampp:
We calculate the empathy score dif- ference within each pair of generated and source sentences in the range [-4, 4), and we report the average over all test samples.
Perplexity: We use GPT-2-Large (Radford al., 2019) to measure the perplexity of each generated sentence, and we report the average over all test samples.
Ace: We calculate the share of samples with increased empathy scores after style transfer

---

Transfer Strength
Content Preservation
Others
Method
AEmp
Acc (96)
BERTScore
BLEURT
DA-Acc (%)
Aenggh
PPL
Zero-shot
0.23
75.98
0.72
0.63
40.78
5.24
69
Pairwise
0.59
90.85
0.68
0.59
49.80
12.34
30
DA PA-Pairwise (implicit)
0.65
93.90
0.68
0,58
54.81
11.71
33
DA- DA-Pairwise (explicit)
0.70
92.81
0.68
0.59
55.56
12.08
34
Target
0.67
93.41
0.67
0,58
46.46
15.18
29
DA Target (implicit)
0.80
96.56
0.68
0.58
53.05
13.04
31
DA- -A-Tare (explicit)
0.77
95.96
0.69
0.59
56.50
11.23
28
Table Evaluation results. The bolded items are the best values for each metric. The arrows show the direction of better performance of the corresponding metric. Row is the baseline; Row 2-4 are pairwise prompting methods; Row 5-7 are target prompting methods. Target methods transfer better empathy than pairwise methods, and DA-conditioned methods transfer better empathy and preserve dialog acts better than direct methods. No visible difference in semantic similarity is observed among the few-shot prompting methods.
5
Results
plicitly specifying the source dialog act in the prompt does not necessarily improve the perfor- mance of DA-conditioned prompting. For DA- Pairwise prompting, explicit specification transfers higher empathy average, but implicit prompt- ing achieves higher accuracy, meaning that more samples are successfully converted into their more empathetic versions. For DA-Targe prompting, im- plicit prompting outperforms explicit specification in both the amount of transferred empathy and the success rate of empathy style transfer. This might be a result of how the prompt is formatted. In the explicit version, we inject tokens related to the source dialog act into the prompt, which inevitably diverts some attention weight from the empathy style transfer instruction the dialog act preserva- tion instruction. As result, explicit specification leads higher accuracy for dialog act preserva- tion than the implicit version. For this case, neither target nor pairwise are statistically significantly dif- ferent.
Target vs Pairwise As shown in Table 4, regard- less whether one conditions the prompt on the source dialog act, target prompting always outper- forms pairwise prompting in terms of both em- pathy score and accuracy. This difference is sta- tistically significantly higher for both direct and DA-conditioned methods. Without conditioning on dialog act, sentences generated by target prompt- ing are slightly less similar the source sentences compared to pairwise prompting, but the difference is nearly negligible. With conditioning on dialog act, the difference in semantic similarity disappears. This is expected since we always explicitly instruct GPT-4 not to change the content (Figure 1). This result suggests that it is unnecessary collect pair- wise samples when doing empathy style transfer through prompting.
DA-conditioned Direct Regardless of whether we use target or pairwise prompting, conditioning the prompt on the dialog act of the source sentences improved empathy transfer strength while maintain- ing a similar level of semantic similarity. The dif- ference in empathy transfer strength is statistically significantly higher for both target and pairwise methods. Also, with DA-condie DA-conditioned prompting, samples have their dialog acts unchanged af- ter empathy style transfer. However, the difference is as large as might be expected, as shown by DA-Acc. This could potentially result from the failure modes of the dialog act classifier, as dis- cussed in Section 4.1 and further addressed below in Section 6.
Zero-shot vs Few-shot Although Reif et al. (2022) showed that few-shot prompting generally performs better than zero-shot on GPT-3, the stun- ning zero-shot capabilities of GPT-4 on various generation tasks made people question whether few-shot prompting is still needed (OpenAI, 2023; Peng et al., 2023; Bubeck et al., 2023). We find that zero-shot prompting with GPT-4 performs much worse than few-shot prompting. All few-shot meth- ods better at empathy transfer. All few-shot methods are more likely to preserve the source di- alog act after style transfer than zero-shot prompt- ing was. Zero-shot prompting produces highest text similarity scores according to BERTScore and BLEURT, but there are two reasons why this does
Implicit vs Explicit While DA-conditioned meth- ods outperformed direct methods in general, ex-

---

not necessarily matter. First, automatic similarity measures are not reliable for measuring content preservation (Shimorina, 2018). Human evalua- tions are necessary to validate this. Second, zero- shot prompting does preserve content better than few-shot prompting, the significant lack of transfer strength is a direct drawback for using zeoo prompting, This is also shown by the difference in sentence lengths. Zero-shot prompting tends add much fewer words than few-shot prompt- ing does, suggesting that zero-shot prompting does not change the sentences by much, thus yielding higher similarity. The lower similarity scores of few-shot methods may also be caused by the lower similarity between the few-shot examples and the input source sentence since dialog act was the only controlled sentence attribute.
Relative to other categories. the BLEURT scores for Other forward expressions (includ- ing forward openings and closings) are par- ticularly low, indicating that may not be a good idea to do empathy style transfer on such dialog acts due to changes in content
Changes in dialog acts after empathy style trans- fer for forward communication and backward com- munication are shown in Appendix Figures 6 and 7.
For forward communications, facts are prone be converted into opinions, and this tendency is surprisingly much more visible in explicit versions of DA-conditioned prompting. Hence, seems unwise to tell GPT-4 the source dialog act is fact when doing empathy style transfer, and it may not be good idea to use empathy style transfer on facts in the first place. On the other hand, DA- conditioned methods work particularly well with maintaining the dialog acts of Questions and Other forward expressions, but direct methods are slightly better at keeping Opinions as opinions.
5.1 DA-specific results
The performance of different prompting methods varies across the source dialog acts. We cluster the 31 dialog acts in the EC dataset into higher-level categories based on Switchboard (Jurafsky et al., 1997) (Table 3). There are two main categories: forward communication and backward communi- cation, each including three subcategories. The "Statement" category is further separated into "Non- opinion" and "Opinion" during analysis since they comprise most the sentences involved in the test samples.
For backward communications, the performance is vastly different. DA-conditioned prompting works very well with Agreements, and explicit specifications of Agreements help preserve this dialog act. Performance on Answers and Other backward expressions is problematic, but note that the sample sizes of Answers and Other backward expressions are small, so the results may not be rep- resentative of the overall samples. However, these results may indicate that it not a good idea to do empathy style transfer on such backward dialog acts.
We present the results of all seven prompting methods for each dialog act category in Figures 5, 6, and 7 in Appendix E, and describe the major findings here. We conclude that different methods work differently on different dialog act categories.
We find the following DA-specific evaluation results for AEmp, style transfer accuracy, and BLEURT (Figure 5):
Conclusion
DA-Target prompting generally transfers bet- ter empathy at a higher accuracy compared to other methods, with the explicit version performing particularly well for facts and the implicit version for others
In this paper, we propose two novel few-shot prompting methods for empathy style transfer: tar- get prompting and dialog-act-conditionee prompt- ing. We examine and confirm the conditionality of empathy scores on the dialog acts of the source sen- tences. Target prompting is better at increasing em- pathy than prompting with traditional source/target pairs. Conditioning the few-shot prompt on the dia- log of the source sentence leads to stronger em- pathy transfer strength and better dialog act preser- vation. We expect that accounting for dialog acts will prove important for other style transfer tasks.
Direct target prompting performs better than all pairwise methods for most forward expres- sions except Facts
Relative other categories, empathy style transfer does not boost the empathy strength of Opinions and Questions

---

Limitations
mental health. It can similarly potentially be used to increase the efficacy of manipulative behaviors. It should also be noted that empathetic communi- cation can be highly context- dereexdddeeeeeet. The work here ignores the many potentially important con- texts: who is talking whom about what. What is their relationship? Their identities? Their cultures? Their power relations? Communications that are perceived as empathetic in one context may be of- fensive or creepy in another. People also differ widely in their response to "empathetic" communi- cations from chatbots in some contexts, with some people finding it offensive to have chatbots, rather than humans expressing empathy. Again, there are strong individual and cultural differences.
Imbalanced and unrepresentative data There are two imbalances in the usage of the EC dataset: dialog act imbalance and sentiment imbalance. De- spite using conversation-level mean empathy score as a selection criterion for test data, the distribu- tion of dialog acts in the test set is still imbalanced (Figure 3b). The results in Table 4 may be bi- ased towards the most frequent dialog acts, namely Statements, Questions, Forward Closings, and none texts. There are many other scenarios in the broader world where empathy style transfer is needed, such as medical and therapeutic conversations. Our re- sults on tragic events are not representative of such scenarios, especially scenarios involving positive empathy. Additional data including more scenarios and positive empathy could expand on this work.
Acknowledgements
We would like to thank the reviewers for their fruit- ful discussion with us. We would also like to thank Claire Daniele for editorial support.
Multiple dialog acts in one sentence Long texts tend to have multiple dialog acts because they often consist of multiple subsentences. In our analyses, we only use the dialog act label of the highest prob- ability as the dialog act for each sentence. It is difficult to rewrite a sentence of multiple dialog acts while retaining its semantics. Though the sen- tences of multiple dialog acts only make up 4.5% of the test samples, 12.0% of the sentences in the EC dataset have multiple dialog acts. It is still de- batable whether it is optimal to treat such sentences single dialog acts and perform DA prompting in the same way as discussed in this paper, or whether it would be better to incorporate multiple dialog acts into few-shot examples instead.
References
Razvan Azamfirei, Sapna R Kudchadkar, and James Fackler. 2023, Large language models and the perils of their hallucinations. Critical Care, 27(1):1-2.
Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xinyu Dai, and Jiajun Chen. 2019. Generating sentences from disentangled syntactic and semantic spaces. Proceedings the 57th An- nual Meeting of the Association for Computational Linguistics, pages 6008-6019.
Chandrakant Bothe. Cornelius Weber, Sven Magg. and Stefan Wermter. 2020. Eda: Enriching emotional dialogue acts using an ensemble of neural annotators. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 620-62 220-627.
Ambiguous dialog acts Another bottleneck in conditioning style transfer on dialog acts is the existence of sentences with ambiguous dialog acts. Such sentences are classified with no dialog act label, and this might be more problematic than having multiple dialog acts since they make up a larger proportion of our data (16.63% in text data and 13.10% in the EC dataset). Our current treatment of such sentences to give them none label and feed them the same few-shot examples used for non-DA methods, but this may not be optimal since DA methods improve performance.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances neural information processing systems, 13:1877-1901.
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar. Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303. 12712.
Ethical Considerations
Style transfer to increase empathy can help improve the quality of human and chatbot conversations in applications ranging from customer support to
Harry Bunt. 2011 1. The semantics of dialogue acts. In Proceedings of the Ninth International Conference on Computational Semantics (IWCS 2011).

---

Chris Callison-Bureds Miles Osbome, and Philipp Koehn. 2006. Re-evaluating the role of bleu in ma- chine translation research. In 11th conference of the european chapter of the association for computa- lional linguistics, pages 249-256.
Huiyuan Lai, Antonio Toral, and Malvina Nissim. 2021. Thank you bart! rewarding pre- pre-trained models im- proves formality style transfer. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 2: Short Papers), pages 484-494.
Christophe Cerisara, Somayeh Jafaritazehjani, Adedayo Oluokun, and Hoa T Lc. 2018. Multi-task dialog act and sentiment recognition on mastodon. In Pro- ceedings of the 27th International Conference on Computational Linguistics, pages 745-754.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc' Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attrionn text rewriting. In International Conference on Learning Reprexenta- tions.
Benjamin MP Cuff, Sarah Brown, Laura Taylor, and Douglas J Howat. 2016. Empathy: A review of the concept. Emotion review, 842):14
Joosung Lee. 2020. Stable style transformer: Delete and generate approach with encoder-decoder for text style transfer. In Proceedings of the 13th International Conference on Natural Language Generation, pages 195-2044
Mary Czerwinski, Javier Hernandez, and Daniel Mc- Duff. 2021. Building an ai that feels: Ai systems with emotional intelligence could learn faster and be more helpful. IEEE Spectrum, 58(5):22-38.
Young-Jun Lee, Chac-Gyun Lim, and Ho-Jin Choi. 2022. Does gpt-3 generate empathetic dialogues? novel in-context example selection method and tomatic evaluation metric for empathetic dialogue generation. In Proceedings of the 29th International Conference Computational Linguistics, pages 669- 683.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuan-Jing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 5997- 6007.
Chuyuan Li, Chloé Braud, and Maxime Amblard. 2022. Multi-task learning for depression detection in di- alogs. arXiv preprint arXiv:2208. 10250.
Nouha Dziri, Sivan Milton. Mo Yu, Osmar R Zaiane. and Siva Reddy. 2022. On the origin of hallucinations in conversational models: Is it the datasets or the models? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisticy: Human Language Technologies, pages 5271-528S
Jingye Li, Hao Fei, and Donghong Ji. 2020a. Modeling local contexts for joint dialogue act recognition and sentiment classification with bi-channel dynamic con- volutions. In Proceedings of the 28th International Conference on Computational Linguistics, pages 616- 626.
Spandana Gella, Aishwarya Padmakumar, Patrick L Lange, and Dilek Hakkani- Tur. 2022. Dialog acts for task driven embodied agents. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 111-123.
Qintong Li, Hongshen Chen, Zhaochun Ren, Pengjie Ren, Zhaopeng Tu, and Zhumin Chen. 2020b. Em- pdg: Multi-resolution interactive empathetic dia- logue generation. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 4454 4466.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.
Dayiheng Liu, Jie Fu, Yidan Zhang, Chris Pal, and Jiancheng Lv. 2020. Revision in continuous space: Unsupervised text style transfer without adversarial learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34(10, pages 8376- 8383.
Zhiqiang Hu, Roy Ka-Wei Lee, Charu C Aggarwal, and Aston Zhang. 2022. Text style transfer: A review and experimental evaluation. ACM SIGKDD Explo- rations Newsletter, 24(1):14-45
Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022. Deep learning for text style transfer: A survey. Computational Lingustiess 48(1):155-205 205.
Ye Liu, Wolfgang Maier, Wolfgang Minker, and Stefan Ultes. 2022. Empathetic dialogue generation with pre-trained roherta-gpt2 and external knowledge. In Conversational Al for Natural Human-Cerne teraction: 12th International Workshop on Spoken Dialogue System Technology, IWSDS 2021, Singa- pore, pages 67-81. Springer.
In-
Daniel Jurafsky, Elizabeth Shriberg, and Debra Biasca. 1997. Switchboard-dann damsl labeling project coder's manual.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini arXiv: 1907.116199
Allison Lahnala, Charles Welch, David Jurgens, and Lucie Flek. 2022. A critical reflection and forward perspective on empathy and natural language process- ing. arXiv preprint 2rXiv:2210. 16604.

---

Aman Madaan, Amrith Setlur, Tanmay Parekh, Barn- abás Poczós, Graham Neubig, Yiming Yang, Ruslan Salakhutdinov, Alan W Black, and Shrimai Prabhu- moye. 2020. Politeness transfer: A tag and generate approach. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 1869-1881.
dataset. Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 5370-5381.
Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris callison-Burchs and Jason Wei. 2022. A recipe for arbitrary text style transfer with large language models. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 837-848.
Navonil Majumder, Pengfei Hong. Shanshan Peng. Jiankun Lu, Deepanway Ghosal, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020. Mime: Mimicking emotions for empathetic response gen- eration. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8968-8979.
Sahand Sabour, Chujie Zheng, and Minlie Huang. 2022. Cem: Commonsense- aware empathetic response gen- eration. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36(10), pages 11229- 11237.
Damilola Omitaomu, Shabnam Tafreshi, Tingting Liu, Sven Buechel, Chris Callison-Burch, Johannes Fichstaedt, Lyle Ungar, and João Sedoc. 2022. Empathic conversations: A multi-level dataset of contextualized conversations. arXiv preprint arXiv:2205. 12698.
Tulika Saha, Aditya Patra, Sriparna Saha, and Pushpak Bhattacharyya. 2020. Towards emotion-aided multi- modal dialogue act classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages
OpenAl. 2023. Gpt-4 technical report. arXiv, pages 2303-08774.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 7881- 7892.
Kartikey Pant, Yash Verma, and Radhika Mamidi, 2020. Sentiine: Incorporating sentiment information into sentiment transfer without parallel data. In Advances Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part II, pages 312-319. Springer.
Anastasia Shimorina. 2018. Human vs automatic met- ries: on the importance of correlation design. arXiv preprint arXiv: 11774.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- rional Linguistics, pages 311-318.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Ma- heswaran. 2019. "transforming" delete, retrieve, gen- crate approach for controlled text style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3269-3279.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.022 032777.
Mirac Suzgun, Luke Melas- Kyriazi, and Dan Juraf- sky. 2022. Promat-and-rerank: A method for zero-shot and few-shot arbitrary textual style trans- fer with small language models. arXiv preprint arXiv:2205.11500
Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li. Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020. Few-shot natural language generation for task- oriented dialog. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 172-182.
Martina Toshevska and Sonja Gievska. 2021. A view of text style transfer using deep learning. JEEE Transactions on Artificial Intelligence.
Yushan Qian, Bo Wang, Shangzhao Ma, Wu Bin, Shuo Zhang, Dongming Zhao, Kun Huang, and Yuexian Hou. 2023. Think twice: A human-like two-stage conversational agent for emotional response genera- tion. arXiv preprint arXiv:2301.044 04907.
Kai Wang. Junfeng Tian, Rui Wang, Xiaojun Quan, and Jianxing Yu. 2020. Multi-domal dialogue acts and response co-gemeration. Anmual Meeting of the Association for Computational Linguistics, pages 7125-7134.
Proceedings of the 58th
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2300-2344.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open- domain conversation models: A new benchmark and

---

Anuradha Welivita, Yubo Xie, and Pearl Pu. 2021. large-scale dataset for empathetic response gener- ation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1251-1264.
Anuradha Welivita, Chun-Hung Yeh, and Pearl Pu. 2023. Empathetic response generation for distress support. Proceedings of the 24th Meeting of the Special Interest Group on Discourse and Dialogue, pages 632-644, Prague, Czechia. Association for Computa- tional Linguistics.
Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. mask and infill": Applying masked language model to sentiment transfer. arXiv preprint
Zonghai Yao and Hong Yu. 2021. Improving formal- ity style transfer with context-aware rule injection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and 1 1thh International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 1561-
Xiaoyuan Yi, Zhenghao Liu, Wenhao Li, and Maosong Sun. 2021. Text style transfer via learning style in- stance supported latent space. In Proceedings of the Twenty-Ninth International Conference on Interna- tional Joint Conferences on Artificial Intelligence, pages 3801-3807.
Mingzhe Zhang, Lin Yue, and Miao Xu. 2022. Estd: Empathy style transformer with discriminative mech- anism. In Advanced Data Mining and Applications: 18th International Conference, ADMA 2022, Bris- bane, QLD, Australia, November 28-30, 2022, Pro- ceedings, Part 11, pages 58-72. Springer.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.09675.
Chujie Zheng, Yong Liu, Wei Chen, Yongcai Leng, and Minlie Huang. 2021. Comae: A multi-factor hierar- chical framework for empathetic response generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2027, pages 813-824.

---

bh A Dialog acts used
The naming used in the explicit DA methods is shown in Table 5. The names were obtained by rewriting the DAMSL codes of Jurafsky et al. (1997) into their noun versions (See https: //web stanford.
300
ws 97 manual augustl html for details).
Dialog Act
Name
sd
non-opinion statement (fact)
sv
opinion statement (statement)
qh
rhetorical question
0.5
1.0
1.5
2.3
2.5
3.0
3.5
4.3
compatr
qo
open question
(a) Eippathy score distribution for selected test samples
qr
or-question
qw
wh-question
qy
yes-no question
b
acknowledgement
36
ba
appreciation
paraphrase
rhetorical question continuer
br
request for repeat
8.
by
expression of sympathy
fo
forward closing
fp
forward opening
ft expression of gratitude
fx expliitt-eefoormaiie
apology
103
aa agreement
count
ad
action directive
(b) Dialog act histogram selected test samples
na
affirmative answer
Figure 3: Empathy score distribution and dialog act his- togram the selected 1016 test samples. 63% of them have empathy score below 2.0 (50% score threshold), total of 25 dialog acts are included, with Opinions (sv), Facts (sd), and "none" taking the largest share of the test samples.
ng
negative answer no-answer
nn
no
non-yes-no answer
h
evasive statement interruption
%
+
sentence continuer
13
3rd-party-talk
2
collaborative completion
q
quotation
none
text
C Empathy distributions for dialog acts
Table 5: Full names of the dialog acts appeared in the BC dataset, obtained and modified from Jurafsky et al. (1997).
We present the histograms of empathy scores for the top most frequently occurring dialog acts the entire EC dataset in Figure 4. While statements and agreements have similar distributions that look like normal distribution slightly skewed on the right, other dialog acts have very distinctive distri- butions. Some of the dialog acts (by and qh) have much smaller proportions in the EC dataset com- pared to the top 3 most frequently occurring ones, so further data collection of these dialog acts will be necessary to draw conclusions on their popula- tion distribution of empathy scores.
B Empathy and Dialog Act Distribution
We present the empathy score distribution and di- alog act histogram for the selected test samples in Figure 3. Though there are still dominant dialog acts in terms of proportion, the overall distribution is relatively less biased than taking the least empa- thetic sentences from the EC dataset, which mostly consists of Forward Closings according to Figure 2.

---

Spinion (sv)
Agrecment (aa)
Fact (sd)
0.35
0.30
probabiitt
0.20
0.15
0.10
0.00
Wh-Qucstion (qw)
Y/N Question (qy)
Open Question (qo)
0.35
0.30
0.25
probablity
0.20
0.15
0.05
0.00
Forward Closing (fc)
Backward Sympathy (by)
Rhetorical Question (qh)
0.30
0.25
probabiityy
0,20
0.10
0.05
0.00
0.5
1.0
1.5
2.0 2.5 empathy
3.0
4.0
0.5
1.0
1.5
2.5
3.5
4.0
0.5
1.0
2.0
3.0
4.0
empathy
empathy
Figure 4: Histograms of empathy scores for the top 9 most frequent dialog acts the EC dataset. These figures illustrate that Opinions, Facts, and Agreements share similar empathy score distributions while other different dialog acts have very different empathy score distributions.
D Prompt formats and examples
after empathy style transfer for forward communi- cation and backward communication, respectively. Each subplot stands for dialog category of the source sentences mentioned in Table 3 where the Statement category further separated into the Fact and Opinion categorics because of their large shares in the dataset.
We present the prompt formats and some samples few-shot examples for Pairwise prompting and DA-Target prompting in Table 6, to highlight their differences.
E DA-specific analyses
We present the results of all prompting methods for each dialog category in Figure 5, 6, and See the main text for descriptions of these figures. Figure 5 shows DA-specific evaluation results for AEmp, style transfer accuracy, and BLEURT. Figures and 7 show changes in dialog acts

---

Component
Format
You are an excellent linguist. Your task is to rewrite the input (DA) in the most empathetic way possible. Make sure the input and output have the same content. Here some examples:
Common Prefix
You are an excellent linguist. Your task is to rewrite the input (DA) J in the most empathetic way possible. Make sure the text is still a (DA} after rewriting. Make sure the input and output have the same content. Here are some examples of empathetic (DA) pairs (pairwise) highly empathetic [DA] (target):
Explicit Specification
Input: think these children went through bad time. Output: My heart goes out to these innocent children who have endured such trying times, bearing the heavy weight hardship and adversity.
Input: The Jewish people are experiencing something diffi- cult.
Output: The Jewish community is navigating through deeply challenging circumstances, grappling with pain and emo- tional turmoil.
Few-shot examples (Pairwise)
Input: Giving a child away is sometimes the right decision, though difficult. It can be hard, but it is the best choice for the child.
Output: It's an excruciatingly complex decision to entrust child to another's care. Despite the heart- emo- tions, it's truly the most compassionate and selfless choice for the child's well-being and future.
What transpired in the lives those precious, helpless infants that they had to undergo such experiences?
What heart- wrenchig circumstances might have led the family to take the agonizing decision to leave behind their beloved pets?
Few-shot examples (DA-Target, Wh-Questions)
How pervasive this heartrending issue, in which these innocent children are struggling with hunger and fatigue, desperately trying to stay awake?
Table 6: Prompt components. For direct prompting methods, the (DA) placeholder is replaced with "text". For DA-conditioned prompting methods, (DA) is replaced with the source dialog act name specified in Table 5.

---

Mean Empathy Boost Scores for each dialog category
method
08
target
darcarr timp)
devnerget (imp
da sair temp!
0.6
0.0
fact
opinion
question
forward
agreement
answer
backward
Empathy Transfer Accuracy dialog act category
0.0
0.4
fact
opinion
question
forward
agreement
answer
backward
BLEURT Scores dialog act category
0.0
0.5
8.4
fact
opinion
question
forward dialog act of input text
agreement
answer
backward
Figure 5: Barplots of different metries different methods on each dialog act category. Each subplot represents the results for evaluation metric suggested by the titles. The x-axis represents the 7 dialog act categories of the source sentences, excluding none category since its results do not contribute to the analysis. The y-axis represents the values on the metric. Each color stands each prompting method. DA-Target prompting generally transfers better empathy at higher accuracy compared to other methods, with the explicit version performing particularly well for facts and the implicit version for others. There minimal difference in semantic similarity between the few-shot prompting methods.

---

input sentences facts
Var aEAR
180
trrget
Se pair timp
do-Earge timpl
60
pair (oou))
leep!
fact
opinion
question
forward
agreement
backward
answer
others
Input sentences opinions
60
fact
opinion
question
forward
agreement
backward
answer
others
Input sentences questions
100
100
80
40
20
fact
opinion
question
forward
agreement
backward
others
input sentences = other forward expressions
80
20
fact
opinion
question
forward
agreement
backward
others
Figure 6: Countplots of dialog acts of generated sentences for forward source dialog act categories. Each subplot represents the count plot in the case where the input sentences are of the dialog act category specified in the title. The x-axis represents the dialog act categories of the generated sentences (different from Figure 5), including none category since it contributes to the analysis of changes in dialog act. Each bar color stands for each prompting method. Facts are prone to be converted into opinions after empathy style transfer, while Questions and Other forward expressions are unlikely be converted into different dialog act categories.

---

input sentences agreements
Vanaale
zeno
89
pair (mmpl
target limpl
da pair (oou))
tanget texp!
213
fact
apinion
question
forward
agreement
backward
others
Input sentences
answers
fact
opinion
question
forward
agreement
backward
answer
others
input sentences other backward expressions
fact
opinion
question
forward
agreemment
backward
answer
others
Figure 7: Countplots dialog acts generated sentences for backward source dialog act categories. The formats are the same Figure 6. Unlike forward expressions, backward expressions highly prone to changing after empathy transfer. Explicit methods show their ad advantage in keeping the dialog act of Agreements, but they did not perform well with Answers and Other backward expressions either.


Reviews:
[{"review_id": "lgxvKEUrIN", "reviewer": "Reviewer_kZET", "summary": "", "questions": "* How about comparing the performance of giving emotion information vs. dialogue acts? I suspect if you give any sorts of related info with the source sentence or context can lead to performance improvements.\n* It would be more informative to clearly flesh out what the authors want to highlight as their main contribution. Simply saying “Dialogue acts matters” is ambiguous and this information isn't something that readers in the NLP field will find novel. What is the most important finding of this paper?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We reply to each of your concerns below.\n\nReasons to Reject:\n- We agree that human evaluation is important; however, Omitaomu et al. (2022) showed that there is a correlation of 0.771 using 5-fold cross-validation between the empathy evaluator and human judgments. This was further validated in the WASSA 2023 shared task. While the difference in semantic similarity scores is small due to the high performance of GPT-4, this is not the case for transfer strength. Figure 5 in the Appendix shows a clearer distinction in empathy score boost between different prompting methods on different source dialog acts. While the difference in accuracy may look small, we scaled the y-axis of the graph to the range [0,1] instead of using our baseline (zero-shot GPT-4, the blue bar) as the reference point, which occupies 75.98% of the y-axis on average as stated in Table 4. We will fix this when we resubmit.\n- Our main contribution is to highlight the importance of \"dialog act\"  in empathy style transfer (line 106-110, 231-242, 580-583), instead of \"dialog act labels”. This is the comparison “DA-conditioned vs Direct” (line 442-453), meaning whether or not the few-shot examples in the prompt have the same dialog act as the source sentence. We will clarify this in future iterations of the paper.\n- We stated that the comparison “Implicit vs Explicit\" is not our main contribution because no clear conclusion can be drawn from this (line 243, 454-458). The significance test results in the rebuttal for Reviewer rbqx indicate this as well.\n- We believe the comparison between Target and DA-Pairwise prompting is unrealistic because it messes up the control of variables when comparing methods in experiments. We have 2 branches of comparison: Target vs Pairwise, and DA-conditioned vs plain. When the second branch is fixed, we find that target methods transfer empathy better than pairwise methods. That is our contribution.\n\nQuestions for the Authors:\n- While other related information may lead to performance improvements, this is beyond the scope of our research. As indicated in Section 2, our research links the following 3 aspects of NLP altogether, which no published paper has done to our knowledge:\n    - Empathy: a major emotional or cognitive response in conversations\n    - Text style transfer: a challenging subfield of natural language generation, unsolved by zero-shot prompting with LLMs\n    - Dialog act: a key factor of a sentence's functionality in a given conversation\n\n    We are more than excited to explore other related information that helps with empathy transfer in the future, such as personality traits, other emotions, and other sentence attributes as well. Nonetheless, this is outside the scope of our paper.\n\n- Our main contributions are fleshed out in line 103-110, 213-242, 285-307, 426-453, and 573-585. While the other results in Section 5 and Appendix may not appear related to our main contributions on the surface, they are closely aligned with our main contribution (line 528-535, Figure 5-7). In line 106-110, we stated that conditioning the prompt (input to GPT-4) on the dialog act of the source sentence would improve GPT-4’s performance in empathy style transfer. This is what we mean by “Dialog acts matter”.\n\nTypos Grammar Style And Presentation Improvements:\n- Many past papers on text style transfer assumed binary or categorical style labels (Toshevska and Gievska, 2021; Hu et al., 2022; Jin et al., 2022). In line 198 we attempted to explain that we assume numerical empathy style scores instead. We will clarify this in our revision.\n- Thank you for pointing out the typos we will fix them.\n- We have clearly indicated the method names in Table 2 and 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Rephrasing sentences to have more empathy can impact real-world applications.", "reasons_to_reject": "* Empathy is very difficult to be captured by automatic metrics, hence human evaluation is a must to verify the improvements. However, the authors only report automatic metrics. Moreover, the difference of the automatic scores between approaches are small. Therefore it is hard to tell whether the approach in the paper is effective or not.\n\n* Although the authors highlight dialogue act labels to be their main contribution, the results do not favor their claim, because the scores are better when the dialogue act label for the source target is not given (implicit vs. explicit). Moreover, the plain Target prompting, which do not include dialogue act labels, is comparable to DA-Pairwise. Again, further human evaluation is needed to compare the true effectiveness of the authors’ approach.", "questions_to_authors": "* How about comparing the performance of giving emotion information vs. dialogue acts? I suspect if you give any sorts of related info with the source sentence or context can lead to performance improvements.\n* It would be more informative to clearly flesh out what the authors want to highlight as their main contribution. Simply saying “Dialogue acts matters” is ambiguous and this information isn't something that readers in the NLP field will find novel. What is the most important finding of this paper?", "paper_topic_and_main_contributions": "This paper attempts to make utterances more empathetic by prompting GPT-4 with few-shot samples and dialogue act labels. The authors show that it is unnecessary to provide source-target pairs when prompting GPT-4 and dialogue act labels might be a help.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Some sentences were hard to follow. Why do you need line 198?\n- Many typos (e.g., wed use, capitalization errors) and citation formatting errors in sentences.\n- It would be much better to clearly indicate which approach you are referring to in the results section (e.g., explicitly mention the method name in the table)."}}, {"review_id": "xkbkNkVzGT", "reviewer": "Reviewer_89A6", "summary": "", "questions": "1. Are you use three examples for all few-shot cases? Maybe I missed some content, but I saw there are three examples in the appendix Table 6.\n2.  line 335-335, what do you mean by qualified examples for targeting prompting here? all highly empathic examples. Are you always using the same few-shot examples for all prompts for all inputs? Could you clarify more? \n3. It makes sense that the qualified examples in the target prompting are from different dialogue acts. But i am curious what if you didn’t explicitly give the dialogue act, but only list a few examples all in target dialogue acts? What will happen? Do we still need the explicit dialogue act in that cases?\n4. For each method, could please also clarify the effort used for prompting engineering? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We reply to each of your concerns below.\n\nReasons to Reject:\n- While it is difficult to include all examples in the paper because of the number of dialog acts involved, we will add more to the appendix in the revision. We apologize for the confusion in describing our experimental controls, and we will clarify this next revision.\n\nQuestions for the Authors:\n1. Yes. We apologize for the confusion. We will clarify this.\n2. For plain target prompting, the qualified examples (i.e., highly empathetic examples) are always the same for all inputs. However, for DA-Target prompting, the qualified examples are only the same for the same source dialog act (e.g., if two inputs are both opinions, they share the same examples. If one is an opinion and the other is a question, they use different examples.) We will elaborate further in the revision.\n3. This is the “implicit” method, and the “explicit” specification of dialog act labels is an extension of our DA-conditioned methods (line 231-249). No conclusion could be drawn from the comparison “implicit vs explicit” (line 454-474), meaning that dialog act label specification does not necessarily improve the performance when we condition the few-shot examples on the dialog act of the input sentence. The significance test results in the rebuttal for Reviewer rbqx indicate this as well.\n4. We clarified the effort used for prompt engineering in line 308-351 and Table 6. We will attempt to improve the clarity in the revision.\n\nTypos Grammar Style And Presentation Improvements:\n- We apologize for the typos and will fix them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It studies a novel direction on bringing dialogue acts for empathy style transfer. The proposed targeting prompting and conditioning on the dialogue act of the source sentence works great. The overall evaluation is extensive. ", "reasons_to_reject": "It is unclear what are the qualified few-shot examples selected for target prompting(see more details in the questions).  It seems many factors here impact the performance: few-shot example selection(number of examples, diversity, distribution of empathy scores), prompt templates etc. The discussion to control those factors is not well-organized,  it is not convincingly clear that claim the proposed prompting method is better than others. ", "questions_to_authors": "1. Are you use three examples for all few-shot cases? Maybe I missed some content, but I saw there are three examples in the appendix Table 6.\n2.  line 335-335, what do you mean by qualified examples for targeting prompting here? all highly empathic examples. Are you always using the same few-shot examples for all prompts for all inputs? Could you clarify more? \n3. It makes sense that the qualified examples in the target prompting are from different dialogue acts. But i am curious what if you didn’t explicitly give the dialogue act, but only list a few examples all in target dialogue acts? What will happen? Do we still need the explicit dialogue act in that cases?\n4. For each method, could please also clarify the effort used for prompting engineering? ", "paper_topic_and_main_contributions": "This paper investigates the prompting methods for empathy style transfer(incremental).  Based on GPT-4 model, It mainly proposed two few-shot methods for prompting: target prompting and dialog-act conditioned prompting.  The experiments are based on the Empathetic Conversation dataset and evaluated with three categories of metrics: Transfer Length, Content Preservation, and Fluency.  Extensive experiments and comparative study shows that target prompting is better than pairwise prompting on increasing empathy, and conditioning on the dialogue act of source sentence leads to stronger empathy and better dialogue act preservation. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 443： wed => we"}, "extra_text": {}}, {"review_id": "X2Ex7cvmQf", "reviewer": "Reviewer_rbqx", "summary": "", "questions": "1. Could you provide the theoretical support for the idea in sec 3.1?\n2. How do you prove that the comparisons in the experiment are complete?\n3. In the experiment section, is that possible to provide the variance of the experiments as well as the significance tests?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. We reply to each of your concerns below.\n\nReasons to Reject:\n1. We believe this is a great point and will clarify this further in the revision. Many past papers on text style transfer assumed binary or categorical style labels (Toshevska and Gievska, 2021; Hu et al., 2022; Jin et al., 2022), and we attempted to explain that we assume numerical empathy style scores instead because empathy is too complicated to be discretized as discussed by the papers in line 695-697 and 731-734. We will delete line 201-203 and modify this section to reflect our assumption with more support.\n\n2. In general, with prompting it is quite hard to show that we are able to have good coverage and conversely with open-source LLMs it is hard to claim they generalize to larger closed-source models. As a result, any paper (generally from academia) without access to large closed-source models has the drawback of being unable to show full prompt coverage. This having been said, we tried variations of zero/few-shot prompting together with more complex prompting methods (e.g., self-critique and chain-of-thought) during our initial experiments. We found that few-shot prompting provides a better representation of the text style transfer task because it clearly specifies how the input and output should look.\n\nQuestions for the Authors:\n1. We will clarify this further in the revision as discussed above.\n\n2. We have 2 main branches of comparison: Target vs Pairwise, and DA-conditioned vs direct, and we also included Zero-shot prompting as our baseline. We assume by “complete” you mean the set of choices in each comparison is complete. Zero-shot prompting provides no input or output example, Target prompting provides only output examples, Pairwise prompting provides both input and output examples, and it makes no sense to use a method with only input examples, so the first comparison should be complete in choice. The second comparison is a binary comparison of whether to condition the examples on the input dialog act, thus complete in choice. The addition extension of “implicit vs explicit” is also a binary comparison of whether to specify the input dialog act label in the prompt, thus complete in choice. If you meant “completeness” in other aspects, we are happy to discuss further.\n\n3. Yes. The t-test results (p-values) across all inputs are:\n\n- Target vs Pairwise comparison (5% significance level):\n    - Plain: 0.017\n    - DA-conditioned (implicit): 0.000 (3.856e-6)\n    - DA-conditioned (explicit): 0.016\n\n- DA-conditioned vs Plain comparison (1% significance level):\n    - Target: 0.000 (4.026e-5)\n    - Pairwise: 0.008\n\t\n- Implicit vs Explicit comparison (insignificant):\n    - Target: 0.434\n    - Pairwise: 0.142\n\nAs shown above, our results for the two main contributions are statistically significant, while the extensions (of explicitly specifying dialog act labels) are not. The results align with our findings in line 454-458. We will run the significance tests specific to each dialog act category in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. How to execute empathy style transfer via prompt-based inference is a critical research subject in natural language processing, which is addressed in this work.\n2. Target prompting and dialog-act-conditioned prompting, two innovative prompting methodologies that the study suggests, may prove helpful for future NLP style transfer research.", "reasons_to_reject": "1. The idea that the transfer between \"apathetic\" and \"empathetic\" is distinct from the transfer between \"unempathetic\" and \"empathetic\" with regard to the issue setting in sec. 3.1 may be based on intuition and prior understanding of empathy and related styles. Theoretical support for this supposition and an explanation of how empathy styles are arranged on a continuous, real-valued scale would be helpful, though. Without more support, this idea may be questioned and criticized.\n\n2. While the comparisons of target vs. pairwise prompting and dialog-act-conditioned vs. direct prompting offer insights into the efficacy of various few-shot prompting strategies, it is unclear from these comparisons whether they represent a thorough analysis of all potential prompting methods or variations.", "questions_to_authors": "1. Could you provide the theoretical support for the idea in sec 3.1?\n2. How do you prove that the comparisons in the experiment are complete?\n3. In the experiment section, is that possible to provide the variance of the experiments as well as the significance tests?", "paper_topic_and_main_contributions": "In particular, the research examines the function of dialog actions in empathy style transfer, which entails altering a statement to make it more sympathetic while preserving its original meaning.\n\nThe major contribution of this study is the suggestion and assessment of dialog-act-conditioned prompting and target prompting as two innovative prompting techniques for empathy style transfer using GPT-4. While dialog-act-conditioned prompting guesses the dialog act of the source phrase and gives few-shot examples of the same dialog act to foster empathy, target prompting simply utilizes examples of the target style.\n\nThe study discovered that conditioning the prompt on dialog acts enhances the empathy transfer while better retaining source dialog acts, and target prompting creates stronger empathy than pairwise prompting while maintaining a similar degree of content. The study also emphasizes the significance of dialog actions in empathy style transmission and urges additional research into their function in style transfer.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper attempts to make utterances more empathetic by prompting GPT-4 with few-shot samples and dialogue act labels. The authors show that it is unnecessary to provide source-target pairs when prompting GPT-4 and dialogue act labels might be a help.

pros:
- It studies a novel direction on bringing dialogue acts for empathy style transfer. The proposed targeting prompting and conditioning on the dialogue act of the source sentence works great. The overall evaluation is extensive.

cons:
- Empathy is very difficult to be captured by automatic metrics, hence human evaluation is a must to verify the improvements. However, the authors only report automatic metrics. Moreover, the difference of the automatic scores between approaches are small. Therefore it is hard to tell whether the approach in the paper is effective or not.
- It is unclear what are the qualified few-shot examples selected for target prompting(see more details in the questions). It seems many factors here impact the performance: few-shot example selection(number of examples, diversity, distribution of empathy scores), prompt templates etc. The discussion to control those factors is not well-organized, it is not convincingly clear that claim the proposed prompting method is better than others.
-  The idea that the transfer between "apathetic" and "empathetic" is distinct from the transfer between "unempathetic" and "empathetic" with regard to the issue setting in sec. 3.1 may be based on intuition and prior understanding of empathy and related styles. Theoretical support for this supposition and an explanation of how empathy styles are arranged on a continuous, real-valued scale would be helpful, though. Without more support, this idea may be questioned and criticized.
Decision: Accept-Findings