Abstract: Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understand trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we develop a novel taxonomy of fine-grained"dialogue acts"frequently observed in customer service, showcasing acts that are more suited to the domain than the more generic existing taxonomies. Using a sequential SVM-HMM model, we model conversation flow, predicting the dialogue act of a given turn in real-time. We characterize differences between customer and agent behavior in Twitter customer service conversations, and investigate the effect of testing our system on different customer service industries. Finally, we use a data-driven approach to predict important conversation outcomes: customer satisfaction, customer frustration, and overall problem resolution. We show that the type and location of certain dialogue acts in a conversation have a significant effect on the probability of desirable and undesirable outcomes, and present actionable rules based on our findings. The patterns and rules we derive can be used as guidelines for outcome-driven automated customer service platforms.

Introduction: The need for real-time, efficient, and reliable customer service has grown in recent years. Twitter has emerged as a popular medium for customer service dialogue, allowing customers to make inquiries and receive instant live support in the public domain. In order to provide useful information to customers, agents must first understand the requirements of the conversation, and offer customers the appropriate feedback. While this may be feasible at the level of a single conversation for a human agent, automatic analysis of conversations is essential for data-driven approaches towards the design of automated customer support agents and systems. Analyzing the dialogic structure of a conversation in terms of the "dialogue acts" used, such as statements or questions, can give important meta-information about conversation flow and content, and can be used as a first step to developing automated agents. Traditional dialogue act taxonomies used to label turns in a conversation are very generic, in order to allow for broad coverage of the majority of dialogue acts possible in a conversation BIBREF0 , BIBREF1 , BIBREF2 . However, for the purpose of understanding and analyzing customer service conversations, generic taxonomies fall short. Table TABREF1 shows a sample customer service conversation between a human agent and customer on Twitter, where the customer and agent take alternating "turns" to discuss the problem. As shown from the dialogue acts used at each turn, simply knowing that a turn is a Statement or Request, as is possible with generic taxonomies, is not enough information to allow for automated handling or response to a problem. We need more fine-grained dialogue acts, such as Informative Statement, Complaint, or Request for Information to capture the speaker's intent, and act accordingly. Likewise, turns often include multiple overlapping dialogue acts, such that a multi-label approach to classification is often more informative than a single-label approach. Dialogue act prediction can be used to guide automatic response generation, and to develop diagnostic tools for the fine-tuning of automatic agents. For example, in Table TABREF1 , the customer's first turn (Turn 1) is categorized as a Complaint, Negative Expressive Statement, and Sarcasm, and the agent's response (Turn 2) is tagged as a Request for Information, Yes-No Question, and Apology. Prediction of these dialogue acts in a real-time setting can be leveraged to generate appropriate automated agent responses to similar situations. Additionally, important patterns can emerge from analysis of the fine-grained acts in a dialogue in a post-prediction setting. For example, if an agent does not follow-up with certain actions in response to a customer's question dialogue act, this could be found to be a violation of a best practice pattern. By analyzing large numbers of dialogue act sequences correlated with specific outcomes, various rules can be derived, i.e. "Continuing to request information late in a conversation often leads to customer dissatisfaction." This can then be codified into a best practice pattern rules for automated systems, such as "A request for information act should be issued early in a conversation, followed by an answer, informative statement, or apology towards the end of the conversation." In this work, we are motivated to predict the dialogue acts in conversations with the intent of identifying problem spots that can be addressed in real-time, and to allow for post-conversation analysis to derive rules about conversation outcomes indicating successful/unsuccessful interactions, namely, customer satisfaction, customer frustration, and problem resolution. We focus on analysis of the dialogue acts used in customer service conversations as a first step to fully automating the interaction. We address various different challenges: dialogue act annotated data is not available for customer service on Twitter, the task of dialogue act annotation is subjective, existing taxonomies do not capture the fine-grained information we believe is valuable to our task, and tweets, although concise in nature, often consist of overlapping dialogue acts to characterize their full intent. The novelty of our work comes from the development of our fine-grained dialogue act taxonomy and multi-label approach for act prediction, as well as our analysis of the customer service domain on Twitter. Our goal is to offer useful analytics to improve outcome-oriented conversational systems. We first expand upon previous work and generic dialogue act taxonomies, developing a fine-grained set of dialogue acts for customer service, and conducting a systematic user study to identify these acts in a dataset of 800 conversations from four Twitter customer service accounts (i.e. four different companies in the telecommunication, electronics, and insurance industries). We then aim to understand the conversation flow between customers and agents using our taxonomy, so we develop a real-time sequential SVM-HMM model to predict our fine-grained dialogue acts while a conversation is in progress, using a novel multi-label scheme to classify each turn. Finally, using our dialogue act predictions, we classify conversations based on the outcomes of customer satisfaction, frustration, and overall problem resolution, then provide actionable guidelines for the development of automated customer service systems and intelligent agents aimed at desired customer outcomes BIBREF3 , BIBREF4 . We begin with a discussion of related work, followed by an overview of our methodology. Next, we describe our conversation modeling framework, and explain our outcome analysis experiments, to show how we derive useful patterns for designing automated customer service agents. Finally, we present conclusions and directions for future work.

Question: How are customer satisfaction, customer frustration and overall problem resolution data collected?
Answer: By annotators on Amazon Mechanical Turk.