Abstract: In (Yang et al. 2016), a hierarchical attention network (HAN) is created for document classification. The attention layer can be used to visualize text influential in classifying the document, thereby explaining the model's prediction. We successfully applied HAN to a sequential analysis task in the form of real-time monitoring of turn taking in conversations. However, we discovered instances where the attention weights were uniform at the stopping point (indicating all turns were equivalently influential to the classifier), preventing meaningful visualization for real-time human review or classifier improvement. We observed that attention weights for turns fluctuated as the conversations progressed, indicating turns had varying influence based on conversation state. Leveraging this observation, we develop a method to create more informative real-time visuals (as confirmed by human reviewers) in cases of uniform attention weights using the changes in turn importance as a conversation progresses over time.

Introduction: The attention mechanism BIBREF1 in neural networks can be used to interpret and visualize model behavior by selecting the most pertinent pieces of information instead of all available information. For example, in BIBREF0 , a hierarchical attention network (Han) is created and tested on the classification of product and movie reviews. As a side effect of employing the attention mechanism, sentences (and words) that are considered important to the model can be highlighted, and color intensity corresponds to the level of importance (darker color indicates higher importance). Our application is the escalation of Internet chats. To maintain quality of service, users are transferred to human representatives when their conversations with an intelligent virtual assistant (IVA) fail to progress. These transfers are known as escalations. We apply Han to such conversations in a sequential manner by feeding each user turn to Han as they occur, to determine if the conversation should escalate. If so, the user will be transferred to a live chat representative to continue the conversation. To help the human representative quickly determine the cause of the escalation, we generate a visualization of the user's turns using the attention weights to highlight the turns influential in the escalation decision. This helps the representative quickly scan the conversation history and determine the best course of action based on problematic turns. Unfortunately, there are instances where the attention weights for every turn at the point of escalation are nearly equal, requiring the representative to carefully read the history to determine the cause of escalation unassisted. Table TABREF1 shows one such example with uniform attention weights at the point of escalation. Our application requires that the visualizations be generated in real-time at the point of escalation. The user must wait for the human representative to review the IVA chat history and resume the failed task. Therefore, we seek visualization methods that do not add significant latency to the escalation transfer. Using the attention weights for turn influence is fast as they were already computed at the time of classification. However, these weights will not generate useful visualizations for the representatives when their values are similar across all turns (see Han Weight in Table TABREF1 ). To overcome this problem, we develop a visualization method to be applied in the instances where the attention weights are uniform. Our method produces informative visuals for determining influential samples in a sequence by observing the changes in sample importance over the cumulative sequence (see Our Weight in Table TABREF1 ). Note that we present a technique that only serves to resolve situations when the existing attention weights are ambiguous; we are not developing a new attention mechanism, and, as our method is external, it does not require any changes to the existing model to apply. To determine when the turn weights are uniform, we use perplexity BIBREF2 (see more details in subsection SECREF4 ). If a conversation INLINEFORM0 escalates on turn INLINEFORM1 with attention weights INLINEFORM2 , let INLINEFORM3 . Intuitively, INLINEFORM4 should be low when uniformity is high. We measure the INLINEFORM5 of every escalated conversation and provide a user-chosen uniformity threshold for INLINEFORM6 (Figure FIGREF2 ). For example, if the INLINEFORM7 threshold for uniformity is INLINEFORM8 , 20% of conversations in our dataset will result in Han visuals where all turns have similar weight; thus, no meaningful visualization can be produced. Companies that deploy IVA solutions for customer service report escalated conversation volumes of INLINEFORM9 per day for one IVA BIBREF3 . Therefore, even at 20%, contact centers handling multiple companies may see hundreds or thousands of conversations per day with no visualizations. If we apply our method in instances where Han weights are uniform, all conversations become non-uniform using the same INLINEFORM10 threshold for INLINEFORM11 , enabling visualization to reduce human effort.

Question: What is the average length of the recordings?
Answer: 40 minutes