CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems (2024)

Abstract: An important unexplored aspect in previous work on user satisfaction estimation for TaskOriented Dialogue (TOD) systems is their evaluation in terms of robustness for the identification of user dissatisfaction: current benchmarks for user satisfaction estimation in TOD systems are highly skewed towards dialogues for which the user is satisfied. The effect of having a more balanced set of satisfaction labels on performance is unknown. However, balancing the data with more dissatisfactory dialogue samples requires further data collection and human annotation, which is costly and time-consuming. In this work, we leverage large language models (LLMs) and unlock their ability to generate satisfaction-focused counterfactual dialogues to augment the set of original dialogues of a test collection. We gather human annotations to ensure the reliability of the generated samples. We evaluate two opensource LLMs as user satisfaction estimators on our augmented collection against state-of-theart fine-tuned models. Our experiments show that when used as few-shot user satisfaction estimators, open-source LLMs show higher robustness to the increase in the number of dissatisfaction labels in the test collection than the fine-tuned state-of-the-art models. Our results shed light on the need for data augmentation approaches for user satisfaction estimation in TOD systems. We release our aligned counterfactual dialogues, which are curated by human annotation, to facilitate further research on this topic.1

heading: 1 Introduction
text: Task-oriented dialogue (TOD) systems help users complete specific tasks, e.g., booking a hotel or restaurant, through conversations (Feng et al., 2022; Sen et al., 2023; Wang et al., 2022; Zeng et al., 2023). User satisfaction estimation (USE) is a key task in TOD systems, aiming to measure the extent
1Available at https://github.com/aminvenv/use
to which users are satisfied with the dialogue they are having with the system (see Figure 1). USE has various applications as it can be viewed as a continuous approximation of human feedback for the quality of the dialogue. Such feedback enables human intervention for users who are having a dissatisfactory dialogue with the system. Furthermore, it serves as a scalable method for the automatic evaluation of dialogue systems and helps identify and optimize a dialogue system’s shortcomings (Song et al., 2023; Ye et al., 2023).
Prior work has studied user satisfaction estimation in TOD systems (Deng et al., 2022; Hu et al., 2023; Sun et al., 2021; Ye et al., 2023) based on the user satisfaction simulation (USS) benchmark, which consists of several datasets annotated with user satisfaction labels by Sun et al. (2021). However, the robustness of user satisfaction estimators for the identification of user dissatisfaction is an unexplored aspect in these works as most of the datasets are highly skewed towards the dialogues for which the user is satisfied. Put another way, the impact of a more balanced set of satisfaction labels on the performance of the USE models remains unknown. Nevertheless, balancing the data with more dissatisfactory dialogue samples demands further dialogue collection and human annotation which is costly and time-consuming.
To begin to address the issues raised above, we aim to expand the current imbalanced benchmarks
14623
of TOD systems with more dissatisfactory dialogues. To this aim, we leverage large language models (LLMs) and unlock their ability to generate counterfactual task-oriented dialogue samples. We use counterfactual utterance generation to generate counterpart dialogue samples with an opposite satisfaction score for a given input dialogue sample, thereby increasing the number of dissatisfactionlabeled samples in the test collections. Following the definition of user satisfaction and the annotation guidelines from the original work in which MultiWOZ (Eric et al., 2020) and SGD (Rastogi et al., 2020) were annotated for user satisfaction levels,2 we conduct human annotation on the counterfactual dialogues to ensure the quality and reliability of the generated utterances. By doing so, we introduce two augmented versions of the test collections for MultiWOZ and SGD benchmarks.
We focus on binary satisfaction levels, i.e., dissatisfaction and satisfaction. We argue that (i) binary labels reduce the subjectivity of annotators in labeling the dialogue, and (ii) binary satisfaction could be more relevant in some TOD system contexts, since in real-world use cases, e.g., posthoc analysis of dialogue systems, one might only look for identification of the cases where the user is dissatisfied with the dialogue and discard the cases where the dialogue proceeds smoothly and normally. In other words, for our purposes classifying whether a dialogue is dissatisfactory or not is of more importance than classifying a normal (rating 3 in a five-point scale satisfaction levels) or satisfying (rate 4) from a very satisfying dialogue (rate 5). Table 1 shows both the five-point scale and the binary-level mapping of the MultiWOZ and SGD datasets used by Sun et al. (2021). As Table 1 indicates, the current evaluation test collections for user satisfaction estimation in TOD systems are highly imbalanced towards the normal satisfaction label (3). In the binary-level satisfaction setting, this imbalance results in most dialogue samples being annotated with satisfaction labels, while the remaining samples are labeled as dissatisfaction.
Recently, Hu et al. (2023) have shown that ChatGPT’s ability to predict user satisfaction scores is comparable to that of fine-tuned state-of-the-art models. This comparable performance was only based on in-context few-shot learning (i.e., without fine-tuning) (Brown et al., 2020; Min et al.,
2We contacted the authors of (Sun et al., 2021) in which the datasets were originally annotated with satisfaction scores.
2022; Perez et al., 2021; Zhao et al., 2021). We examine to what extent this finding on estimating user satisfaction generalizes to open-source LLMs. We use two open-source LLMs, namely, Zephyr-7b-beta3 and Mistral-7B-Instruct4 (to which we refer as Zephyr and MistralIF, respectively), and evaluate their performance on user satisfaction estimation on the MultiWOZ and SGD datasets.
Our experiments show that when we incorporate more dissatisfactory dialogue samples in the test collections with our methodology for generating counterfactual dissatisfying utterances, LLMs can significantly outperform the state-of-the-art finetuned models. We argue that this discrepancy in the performance of models across more balanced test sets is due to the imbalanced training sets with plentiful dialogue samples with satisfaction labels.
We summarize our contributions as follows:
• We show and unlock the power of LLMs in generating satisfaction-focused counterfactual dialogues in TOD systems, paving the way for data augmentation in USE for TOD systems. • We conduct human evaluations on our generated counterfactual dialogue samples and augment the test collections of MultiWOZ and SGD benchmarks. • Through the robustness study of USE, we find that the performance of fine-tuned state-of-theart estimators drastically decreases with an increase in dissatisfaction-labeled dialogues in test collections. • We show that open-source LLMs, when used in few-shot USE, maintain higher robustness in identifying user dissatisfaction in TOD systems than state-of-the-art fine-tuned estimators.
3https://huggingface.co/HuggingFaceH4/ zephyr-7b-beta
4https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.2

heading: 2 Related Work

heading: 2.1 User Satisfaction Estimation in TODSs
text: User satisfaction estimation has been studied in the context of various information retrieval and natural language processing tasks, including conversational recommender systems (Siro et al., 2022, 2023) and TOD systems (Deng et al., 2022; Pan et al., 2022; Ye et al., 2023). In TOD systems, the goal of the user is to complete a specific task, e.g., booking a hotel, reserving a ticket. Depending on the flow of conversation between the user and the TOD system, user satisfaction can vary throughout the dialogue (Sun et al., 2023a). Predicting the extent to which the user is satisfied with the dialogue is defined as user satisfaction estimation. Sun et al. (2021) study user satisfaction estimation in TOD systems and propose a benchmark for the task consisting of several datasets. They find that the core reason for user dissatisfaction is the system’s failure to accurately understand the user’s requests or manage their requirements effectively. Kim and Lipani (2022) propose a multi-task framework and show that user satisfaction estimation, action prediction, and utterance generation tasks can benefit from each other via positive transfer across tasks. Ye et al. (2023) model user satisfaction across turns as an event sequence and use the dynamics in this sequence to predict user satisfaction for a current turn in the dialogue. Hu et al. (2023) leverage ChatGPT as a user satisfaction estimator and use the satisfaction scores as feedback for training a dialogue utterance generation model.

heading: 2.2 Counterfactual Data Generation
text: Generating counterfactual data samples has been studied across various natural language processing tasks (Abolghasemi et al., 2024; Miao et al., 2023; Wen et al., 2022; Zhang et al., 2023). Specifically, there is a body of prior work on generating counterfactual dialogues. Li et al. (2020) and Huang et al. (2021) explore counterfactual dialogue generation in the context of dialogue state tracking (DTS) task. Calderon et al. (2022) focus on the multi-label intent prediction of utterances from information-seeking dialogues and produce domain-counterfactual samples. These samples are similar to the original samples in every aspect, including the task label, yet their domain is altered to a specified one. Ben-David et al. (2021) study counterfactual data generation in the context of intent prediction; they address counterfactual generation,
not for generating a system utterance, but for a user utterance, in contrast to the approach we take in this paper.
There is also prior work on counterfactual data generation using LLMs, as they have shown to be highly capable in natural language generation tasks (Asai et al., 2023; Askari et al., 2024). For instance, Li et al. (2023) explore the strengths and weaknesses of LLMs in generating counterfactual data samples. However, to the best of our knowledge, there is no prior work on satisfaction-focused counterfactual dialogue generation, which we study in this work.

heading: 3 User Satisfaction Estimation
text: We formulate the task of user satisfaction estimation (USE) as follows. Given dialogue context D with T turns as D = {(U1, R1), (U2, R2), . . . , (UT , RT )}, where Ut and Rt stand for the t-th user utterance and system response, respectively, the goal is to estimate the user satisfaction s at the turn T . Therefore, the task objective is to learn a prediction model P (sT |D), where sT is the user satisfaction at the T -th turn.

heading: 4 Methodology

heading: 4.1 Counterfactual Utterance Generation
text: Annotated dialogues with user satisfaction labels are not necessary available upon deploying TOD systems. Moreover, obtaining annotations with user satisfaction labels is both expensive and laborintensive. However, LLMs have enabled quality text generation across various tasks (Bonifacio et al., 2022; Sun et al., 2023b; Yin et al., 2023; Yu et al., 2022). We take advantage of these models in order to generate new dialogue samples with a presumed satisfaction label in order to make up for the imbalance that exists in the benchmarks used for the evaluation of user satisfaction estimation. Utterance Generation Task Formulation. Given a dialogue context D = {(U1, R1), (U2, R2), . . . , (UT , RT )} with T turns, the goal is to generate R̂T in order to obtain D̂ = {(U1, R1), (U2, R2), . . . , (UT , R̂T )}, where the user satisfaction label for the T -th turn for dialogue D̂ is the opposite of user satisfaction label for D. Our definition of counterfactual utterance is based on the annotation guidelines in (Sun et al., 2021), in which MultiWOZ and SGD with user satisfaction labels are introduced.
In order to generate a counterfactual response R̂T for a given system response R, we use few-shot in-context learning (ICL) with LLMs (Brown et al., 2020; Perez et al., 2021). Here, we provide the LLM GPT-4 with an instruction regarding what a counterfactual system utterance means. We do that both when we have a satisfaction-labeled dialogue sample or a dissatifaction-labeled one. Figure 6 in the Appendix shows the prompt used for generating counterfactual system utterances using GPT-4. Clearly, we perform the generation in a dialogueaware manner, i.e., the generation of counterfactual system utterance R̂ is conditioned on the history of the dialogue between the user and the system.
Figure 2 shows two samples of counterfactual utterance generation. As the figure (left) shows, the counterfactual generation process is context-aware, meaning that the generated counterfactual system utterance includes information from the previous turns (i.e., context) of dialogue.

heading: 4.2 User Satisfaction Estimation using LLMs
text: Enabling zero-shot/few-shot (in-context learning) user satisfaction estimation could be of great use for the development and evaluation of dialogue systems. Such an in-context learning setup for the inference of user satisfaction labels facilitates the deployment of such systems as zero-shot/fewshot learning and removes the need for training samples which are costly to obtain. For instance, Hu et al. (2023) show that ChatGPT can provide a comparable performance to supervised methods. They employ ChatGPT as a user simulator to obtain user feedback on the generated utterances. While Hu et al. (2023) use zero-shot/few-shot in-context learning with a proprietary language model for user satisfaction estimation, we evaluate the perfor-
mance of open-source models. Few-shot In-context Learning. In order to estimate user satisfaction for a given dialogue, we use few-shot in-context learning (Brown et al., 2020; Perez et al., 2021). Figure 3 shows the prompt used for estimating user satisfaction using few-shot incontext learning with the two LLMs Zephyr (Tunstall et al., 2023) and MistralIF (Jiang et al., 2023).

heading: 5 Experimental Setup

heading: 5.1 Benchmarks
text: We evaluate the models on the Multi-Domain Wizard-of-Oz (MultiWOZ) (Eric et al., 2020) and Schema Guided Dialogue (SGD) (Rastogi et al., 2020) benchmarks in our experimental setup. MultiWOZ and SGD are two commonly-used multidomain task-oriented dialogue datasets and were initially annotated with user satisfaction scores by
Sun et al. (2021). We leverage the data splits used in prior work (Deng et al., 2022; Ye et al., 2023). Table 2 shows the statistics of train/validation/test splits in the MultiWOZ and SGD benchmarks.
We also note that in this paper we only work on turn-level satisfaction labeling. Generating a counterfactual sample for a complete dialogue requires more stratified and complicated dialogue generation methods that are beyond the scope of this paper.

heading: 5.2 Evaluation Metrics
text: Following (Hu et al., 2023; Sun et al., 2021; Ye et al., 2023), we use Accuracy, Precision (the proportion of the predicted correct labels over the number of predicted labels), Recall (the proportion of the predicted correct labels over the number of actual labels), and the F1-score (the harmonic mean of precision and recall) as our evaluation metrics.

heading: 5.3 Baselines
text: BERT. BERT (Devlin et al., 2019) is a widely-used baseline as satisfaction label classifier in prior work (Deng et al., 2022; Kim and Lipani, 2022; Sun et al., 2021; Ye et al., 2023). BERT achieves state-of-theart performance in (Sun et al., 2021) and Hu et al. (2023) shows that it outperforms ChatGPT in fewshot setting. We replicate the implementation from (Sun et al., 2021) for this baseline. In addition, we up-sample the dissatisfaction class by orders of 10x up to 50x and include the models with the best and the second best performance in our results. ASAP. ASAP is our second baseline for the evaluation against LLMs for user satisfaction estimation. Ye et al. (2023) propose ASAP as user satisfaction estimator in which they leverage Hawkes processes (Mei and Eisner, 2017) to capture the dynamics of user satisfaction across turns within a dialogue. Ye et al. (2023) show that ASAP achieves state-of-the-art performance over a variety of baselines. We conduct the same aforementioned upsampling approach of BERT for ASAP.

heading: 5.4 Human Annotation
text: To evaluate the quality of the generated counterfactual dialogues we conduct human evaluation on the samples for both MultiWOZ and SGD benchmarks. We use two human annotators (and a third in the case of disagreement) and annotate the counterfactual dialogues in terms of “user satisfaction,” and “dialogue coherence.” Dialogue Coherence (DC). DC refers to the degree to which a generated counterfactual is relevant (fitting) to the previous turns in the dialogue, i.e., if the counterfactual system utterance is coherent with the dialogue history. An example of a non-coherent counterfactual system utterance is a case where the system answers a request for booking a hotel in a city with a response regarding the reservation of a restaurant in that city. User Satisfaction Labeling. In the counterfactual dialogues, we only replace the last system utterance with a counterfactual one. To verify the effect of this change, we ask our annotators to label the whole dialogue in terms of user satisfaction. In the annotation pool, we mix the counterfactual dialogues with actual dialogues to prevent any learning bias. We use the same guidelines as Sun et al. (2021) with a slight difference where we exchange the five-point scale rating with a binarylevel satisfaction rating. We also note that, following Sun et al. (2021), we use before-utterance (BU) prediction of user satisfaction scores (Kim and Lipani, 2022). In this approach, user satisfaction is estimated after a system utterance and before the next user utterance. This is in contrast to after-utterance (AU) prediction (Bodigutla et al., 2020; Cai and Chen, 2020), in which the satisfaction score prediction is conducted after each user utterance, and therefore, user expressions in their utterance can be used as an indicator of their satisfaction level. While being more difficult, BU prediction enables the dialogue system to prevent potential negative user experiences by steering the conversation away from directions that might lead to dissatisfaction (Kim and Lipani, 2022).

heading: 6 Experimental Results

heading: 6.1 Data Quality
text: We first assess the quality of the data that we have collected. We measure the inter-annotator agreement (IAA) between our annotators. Table 3 shows the agreement between the annotators on the sat-
isfaction labels measure by Cohen’s Kappa. As for DC, most of the data falls into one category (agreement on the coherence of the generated system utterance), making Kappa not a reliable metric. Instead, we use Percent Agreement which is the percentage of agreement between the two annotators.
Additionally, Table 4 shows the ratio of correctly flipping the satisfaction status of the last system utterance, which we refer to as Counter Satisfaction Status (CSS). As the overall CSS values show, not all generated system utterances are satisfactionfocused counterfactuals of the original system utterances, i.e., 63.8 success rate for MultiWOZ and 80.3 for SGD. We only keep the samples in the CF set that are confirmed to be counterfactual by the human annotators.
Moreover, from the user evaluation in Table 4 we infer that GPT-4 is better at generating dissatisfying system utterances (the CSS values in the Satisfaction row in Table 4) than at generating satisfying system utterances (the CSS values in the Dissatisfaction row).
Based on the labeling obtained using the three annotators, Table 5 shows the number of test samples for both counterfactual and non-counterfactual (i.e., original samples) for the two classes of Satisfaction and Dissatisfaction.

heading: 6.2 User Satisfaction Estimation Results
text: Table 6 shows the results of user satisfaction estimation using BERT and ASAP as the state-of-the-art models (Hu et al., 2023; Ye et al., 2023), as well as two LLMs, Zephyr and MistralIF. BERT and
ASAP models are fine-tuned using the training samples indicated in Table 2. The two LLMs, however, are used in a few-shot manner as described in Section 4.2. We evaluate these models using different test sets. The Main group of results (at the top of Table 6) refers to the original test set from (Sun et al., 2021); CF refers to the counterfactual version of Main, which is generated as described in Section 4.1; and Mix is the aggregation over both Main and CF.
As the table suggests, while on the original data (Main), which is highly imbalanced across satisfaction and dissatisfaction labels, BERT and ASAP outperform the two LLMs, in the rest of the test sets (CF, Mix), it is the LLMs that achieve higher performance than BERT and ASAP by a large margin. Moreover, while we can see a drastic drop in the performance of BERT and ASAP on CF in comparison to their performance on the Main set, the performance of LLMs on the two sets of Main and CF is comparable. These results show the robustness of few-shot in-context learning for user satisfaction estimation under different distributions of labels in the test data. In addition, we can see from the results on the CF test data that while increasing the ratio of up-sampling dissatisfaction training samples from 10x to 20x increases the performance of the BERT and ASAP estimators on the MultiWOZ dataset, this way of augmenting training samples does not have the same effect on the SGD test set. This may indicate the lack of proper training data and the necessity for augmenting the training data for fine-tuning user satisfaction estimators. Furthermore, it highlights the need for more sophisticated data augmentation approaches rather than simply up-sampling the data. It is noteworthy that we also conducted our experiments using under-sampling of the satisfactory class; however, the results corresponding to this approach are not included since it led to a weak performance. Robustness results. The Main and CF test collections (Table 5) are the two extremes in case of
imbalance in the test data for the number of satisfaction and dissatisfaction test samples. To better explore the robustness of models with varying numbers of test samples from the two classes of Satisfaction and Dissatisfaction, we evaluate the models using different proportions of these classes. To this aim, we start with the Main test set with an approximate 95:5 ratio for satisfaction:dissatisfaction labels. We then increase the number of dissatisfaction labels in the Main condition using the dissatisfaction dialogue samples from the CF condition. We evaluate models while increasing the dissatisfaction fraction in steps of 5%. Figure 4 depicts the performance of all models on the MultiWOZ and SGD benchmarks. We see that the performance of the fine-tuned state-of-the-art models (BERT and ASAP) drastically drops when more Dissatisfaction samples are included in the evaluation. Moreover, Figure 5 shows the sensitivity (recall) for only the Dissatisfaction class. As we can see, few-shot in-context learning with LLMs provides an increased ability to identify user dissatisfaction in the dialogues, which is a crucial factor in the deployment of dialogue systems. This is particularly important as we can see the higher performance of fine-tuned state-of-the-art models (BERT and ASAP) in comparison to LLMs on the original test set (Main in Table 6), which includes about 5% dissatisfaction samples. However, the sensitivity of these fine-tuned state-of-the-art models (BERT and
ASAP) for the identification of user dissatisfaction is either lower than LLMs (BERT versus LLMs on MultiWOZ in Figure 5) or becomes comparable with them with a slight increase in the number of Dissatisfaction samples, e.g., change in results from 5% to 10% dissatisfaction ratio in Table 5.
Shared-context results. The counterfactual dialogue samples in the CF test set differ from the corresponding original samples in the Main test set in terms of the last system response (see Figure 2). To measure the success rate of estimators in predicting the user satisfaction label for both a dialogue and its corresponding counterfactual sample,
i.e., two samples with the same context (dialogue history), we use the Jaccard similarity index (JSI) |M∩C| |M∪C| , where M and C are the correctly predicted samples of the Main and CF test collections respectively. Table 7 shows the JSI for different user satisfaction estimators. The best performing BERT and ASAP setups from Table 6 are selected for this purpose. As the table shows, BERT and ASAP have a very low JSI in comparison to the LLM-based satisfaction estimators which is in line with the result of these models on the Main and CF test sets in Table 6. Furthermore, we can see that on both the MultiWOZ and SGD test sets, Zephyr has a higher JSI than MistralIF, even though MistralIF outperforms Zephyr on the Main test set (top-rows in Table 6).

heading: 7 Conclusion
text: We have studied the task of user satisfaction estimation and specifically focused on the robustness of estimators for TOD systems. We augment two previously introduced benchmarks using satisfactionfocused counterfactual utterance generation and conduct human evaluation on the generated dialogues. Using our augmented test collections, we show that there is a discrepancy between the performance of estimators on the original test sets and the test sets with a higher ratio of dissatisfaction dialogue samples.
Our experiments highlight an important missing aspect in previous studies: the robustness of satisfaction estimators for the identification of user dissatisfaction. Moreover, our work sheds light on the need for further research on data augmentation for training user satisfaction estimators. We hypothesize that training models with more balanced data is beneficial for the robustness of these models. In this work, we also unlock the power of LLMs in generating quality counterfactual dialogue samples which seems to be a promising direction for augmenting the training set of user satisfaction estimators. In future work, we plan to leverage LLMs for such satisfaction-oriented data augmentation in TOD systems. Furthermore, in this paper, we only work on turn-level satisfaction estimation and leave the dialogue-level setting for future work as generating dialogue-level counterfactual data requires more sophisticated methods. Finally, we have explored user satisfaction estimation only in task-oriented dialogue systems. User satisfaction estimation has also been studied for other tasks including conversation recommender systems (Siro et al., 2022; Sun et al., 2021). Also, we plan to study counterfactual utterance generation for a more broad application of USE in dialogue systems.
Limitations
While we employ proprietary model GPT-4 for the generation of counterfactual samples, we also point out the limitation in this approach in the sense that it still requires to leverage of a proprietary LLM. Here, we should note that we use GPT-4 to create counterfactual data samples in order to enhance the existing benchmarks. This is a one-off usage of proprietary models that enables future research on the evaluation of user satisfaction estimation for task-oriented dialogue systems.
In addition, it should be noted that our current research is exclusively on datasets in English. Therefore, we highlight the necessity of extending our work to include datasets in languages other than English. This expansion is of importance to ensure the applicability of our findings across a broader linguistic spectrum.

heading: Acknowledgements
text: This work was supported by the DoSSIER project under European Union’s Horizon 2020 research and innovation program, Marie Skłodowska-Curie grant agreement No. 860721, the Hybrid Intelligence Center, a 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, and project LESSEN with project number NWA.1389.20.183 of the research program NWA ORC 2020/21, which is (partly) financed by the Dutch Research Council (NWO). All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.

heading: A Appendix
text: A.1 Counterfactual Response Generation Prompt
Figure 6 shows the prompt used to generate a counterfactual system utterance for a new input dialogue, using few-shot in-context learning with GPT-4.
A.2 Full-dialogue Counterfactual Examples Figure 7 illustrates two dialogue samples with both their original last system utterance and the counterfactual last system utterance.

heading: Future Work
text: In future work, we plan to leverage LLMs for such satisfaction-oriented data augmentation in TOD systems. Furthermore, in this paper, we only work on turn-level satisfaction estimation and leave the dialogue-level setting for future work as generating dialogue-level counterfactual data requires more sophisticated methods. Finally, we have explored user satisfaction estimation only in task-oriented dialogue systems. User satisfaction estimation has also been studied for other tasks including conversation recommender systems (Siro et al. , 2022; Sun et al. , 2021). Also, we plan to study counterfactual utterance generation for a more broad application of USE in dialogue systems. Limitations
While we employ proprietary model GPT-4 for the generation of counterfactual samples, we also point out the limitation in this approach in the sense that it still requires to leverage of a proprietary LLM. Here, we should note that we use GPT-4 to create counterfactual data samples in order to enhance the existing benchmarks. This is a one-off usage of proprietary models that enables future research on the evaluation of user satisfaction estimation for task-oriented dialogue systems. In addition, it should be noted that our current research is exclusively on datasets in English. Therefore, we highlight the necessity of extending our work to include datasets in languages other than English. This expansion is of importance to ensure the applicability of our findings across a broader linguistic spectrum