Facilitating automatic detection of relationships between sentences in conversations

Techniques are provided for training, by a system operatively coupled to a processor, an attention weighted recurrent neural network encoder-decoder (AWRNNED) using an iterative process based on one or more paragraphs of agent sentences from respective transcripts of one or more conversations between one or more agents and one or more customers, and based on one or more customer response sentences from the respective transcripts, and generating, by the system, one or more groups respectively comprising one or more agent sentences and one or more customer response sentences selected based on attention weights of the AWRNNED.

BACKGROUND

The subject disclosure relates generally to automatically detecting relationships between specific agent sentences and specific customer response sentences in an agent-customer conversation.

SUMMARY

The following presents a summary to provide a basic understanding of one or more embodiments of the invention. This summary is not intended to identify key or critical elements, or delineate any scope of the particular embodiments or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a prelude to the more detailed description that is presented later. One or more embodiments described herein include a system, computer-implemented method, and/or computer program product, in accordance with the present invention.

According to an embodiment, a system is provided. The system comprises a memory that stores computer executable components; and a processor that executes the computer executable components stored in the memory. The computer executable components can comprise: a model generation component that trains an attention weighted recurrent neural network encoder-decoder (AWRNNED) using an iterative process based on one or more paragraphs of agent sentences from respective transcripts of one or more conversations between one or more agents and one or more customers, and based on one or more customer response sentences from the respective transcripts; and a grouping component that generates one or more groups respectively comprising one or more agent sentences and one or more customer response sentences selected based on attention weights of the AWRNNED.

The computer executable components can also comprise a performance component that determines an effectiveness of an agent based upon an analysis of one or more conversations of the agent using the trained AWRNNED. This provides a benefit over prior art in that relationships between agent sentences and customer response sentences can be automatically determined and employed to evaluate effectiveness of an agent.

The computer executable components can also comprise a recommendation component that generates, during an ongoing conversation between an agent and a customer, a recommendation to the agent of an agent sentence to employ to elicit a particular customer response sentence based upon an analysis of the ongoing conversation using the trained AWRNNED. This provides a benefit over prior art in that an agent can be provided real-time recommendation of agent sentences to employ that can direct the conversation in a direction to satisfy a goal of the conversation.

In another embodiment, a computer-implemented method is provided. The computer-implemented method can include training, by a system operatively coupled to a processor, an attention weighted recurrent neural network encoder-decoder (AWRNNED) using an iterative process based on one or more paragraphs of agent sentences from respective transcripts of one or more conversations between one or more agents and one or more customers, and based on one or more customer response sentences from the respective transcripts, and generating, by the system, one or more groups respectively comprising one or more agent sentences and one or more customer response sentences selected based on attention weights of the AWRNNED.

In another embodiment, a computer program product for training an attention weighted recurrent neural network encoder-decoder (AWRNNED) is provided. The computer program product can include a computer readable storage medium having program instructions embodied therewith. The program instructions can be executable by a processer to cause the processer to: train the AWRNNED using an iterative process based on one or more paragraphs of agent sentences from respective transcripts of one or more conversations between one or more agents and one or more customers, and based on one or more customer response sentences from the respective transcripts, and generate one or more groups respectively comprising one or more agent sentences and one or more customer response sentences selected based on attention weights of the AWRNNED.

DETAILED DESCRIPTION

Conversations between an agent and a customer are often scripted with a manager evaluating the agent based upon their ability to follow the script or employ specific keywords. The agent receives training by listening to recorded calls of other agents which a manager felt were good examples of an interaction with the customer. These mechanisms for training an agent are subjective and time consuming.

To address the challenges in agent training as described herein, one or more embodiments of the invention can employ an Attention Weighted Recurrent Neural Network (AWRNN) model to automatically analyze conversations between an agent and a customer to identify relationships between agent sentences and customer response sentences. For example, agent sentences that are effective at eliciting specific response sentences from the customer can be identified using an AWRNNED. In an example, agent sentences that trigger positive responses can be identified, as well as, agent sentences that trigger negative responses can be identified. Using these identifications, agents can be trained on what to say and what not to say. That is, one or more embodiments herein can improve training of an agent using analysis of other agents' conversations, as well as, conversations of the agent being trained. In addition, an agent's conversation can be monitored in real-time and recommendations on sentences to say and sentences to avoid can be provided to the agent in real-time during the conversation to steer the conversation in a particular direction with respect to customer responses. In another non-limiting example, an agent's performance can be determined by automatically analyzing the agent's conversations using an AWRNNED.

One or more embodiments of the subject disclosure is directed to computer processing systems, computer-implemented methods, apparatus and/or computer program products that facilitate efficiently, effectively, and automatically (e.g., without direct human involvement) analyzing conversations between agents and customers to identify relationships between agent sentences and customer response sentences, train agents, evaluate performance of agents, and provide real-time recommendations of sentences to use and not use during an ongoing conversation. The computer processing systems, computer-implemented methods, apparatus and/or computer program products can employ hardware and/or software to solve problems that are highly technical in nature (e.g., adapted to generate and/or employ one or more different detailed, specific and highly-complex attention weighted recurrent neural network encoder-decoder models that can automatically analyze an agent-customer conversation to detect relationships between agent sentences and customer response sentences) that are not abstract and that cannot be performed as a set of mental acts by a human. For example, a human, or even thousands of humans, cannot efficiently, accurately and effectively manually gather and analyze thousands of data elements related to a variety of observations in a real-time network based computing environment to analyze conversations between agents and customers to identify relationships between agent sentences and customer response sentences, train agents, evaluate performance of agents, and provide real-time recommendations of sentences to use and not use during an ongoing conversation. One or more embodiments of the subject computer processing systems, methods, apparatuses and/or computer program products can enable the automated analysis of conversations between agents and customers to identify relationships between agent sentences and customer response sentences, training of agents, evaluations of performance of agents, and providing of real-time recommendations of sentences to use and not use during an ongoing conversation using artificial intelligence in a highly accurate and efficient manner to achieve one or more goals. By employing an Attention Weighted Recurrent Neural Network Encoder-Decoder (AWRNNED) model and artificial intelligence, the processing time and/or accuracy associated with the automated conversation analysis, agent training, agent performance evaluation, and real-time recommendation systems is substantially improved. Additionally, the nature of the problem solved is inherently related to technological advancements in artificial intelligence based conversation analysis that have not been previously addressed in this manner. Further, one or more embodiments of the subject techniques can facilitate improved performance of automated conversation analysis, agent training, agent performance evaluation, and real-time recommendations that provides for more efficient usage of storage resources, processing resources, and network bandwidth resources to provide highly granular and accurate conversation analysis, agent training, agent performance evaluation, and real-time recommendation using artificial intelligence. For example, by allowing for automated conversation analysis, agent training, agent performance evaluation, and real-time recommendations, agent efficiency and effectiveness is improved, and wasted usage of processing, storage, and network bandwidth resources can be avoided by decreasing the length of conversations between agents and customers.

An agent can be a person that is trying to influence a customer during a conversation. In a non-limiting example, an agent can be a customer support person, a salesperson, or any other suitable person that has a conversation in with a customer. In a non-limiting example, a conversation can be an audio conversation (e.g., in-person, telephonic, or any other suitable audio conversation), a text conversation (e.g., email, chat, text message, or any other suitable text conversation), or any other suitable conversation between an agent and a customer. In another non-limiting example, a sentence can be a sentence, a phrase, a clause, a statement, or any other suitable grouping of words from a person of a conversation. In a further non-limiting example, a paragraph can be a group of one or more sentences from a person of a conversation.

By way of overview, aspects of systems, apparatuses, or processes in accordance with the present invention can be implemented as machine-executable component(s) embodied within machine(s), e.g., embodied in one or more computer readable mediums (or media) associated with one or more machines. Such component(s), when executed by the one or more machines, e.g., computer(s), computing device(s), virtual machine(s), etc. can cause the machine(s) to perform the operations described.

FIG. 1illustrates a block diagram of an example, non-limiting system100that facilitates automatically analyzing conversation between agents and customers using an AWRNNED in accordance with one or more embodiments described herein.FIG. 2illustrates a block diagram of an example, non-limiting agent conversation component104in accordance with one or more embodiments described herein. Repetitive description of like elements employed in one or more embodiments described herein is omitted for sake of brevity.

With reference toFIGS. 1 and 2, and as shown inFIG. 1, the system100can include a computing device102, one or more networks112, and one or more data sources114. Computing device102can include an agent conversation component104that can facilitate automatically analyzing conversations between agents and customers using an AWRNNED as discussed in more detail below. Agent conversation component104can include conversation analysis component202that can automatically analyze conversations between agents and customers, train an AWRNNED based on the analysis, and identify relationships between agent sentences and customer response sentences as discussed in more detail below.

Computing device102can also include or otherwise be associated with at least one included (or operatively coupled) memory108that can store computer executable components (e.g., computer executable components can include, but are not limited to, the agent conversation component104and associated components), and can store any data generated by agent conversation component104and associated components. Computing device102can also include or otherwise be associated with at least one processor106that executes the computer executable components stored in memory108. Computing device102can further include a system bus110that can couple the various server components including, but not limited to, the agent conversation component104, memory108and/or processor106.

Computing device102can be any computing device that can be communicatively coupled to one or more data sources114, non-limiting examples of which can include, but are not limited to, include a wearable device or a non-wearable device Wearable device can include, for example, heads-up display glasses, a monocle, eyeglasses, contact lens, sunglasses, a headset, a visor, a cap, a mask, a headband, clothing, or any other suitable device that can be worn by a human or non-human user. Non-wearable device can include, for example, a mobile device, a mobile phone, a camera, a camcorder, a video camera, laptop computer, tablet device, desktop computer, server system, cable set top box, satellite set top box, cable modem, television set, monitor, media extender device, blu-ray device, digital versatile disc or digital video disc (DVD) device, compact disc device, video game system, portable video game console, audio/video receiver, radio device, portable music player, navigation system, car stereo, a mainframe computer, a robotic device, a wearable computer, an artificial intelligence system, a network storage device, a communication device, a web server device, a network switching device, a network routing device, a gateway device, a network hub device, a network bridge device, a control system, or any other suitable computing device102. A data source114can be any device that can communicate with computing device102and that can provide information to computing device102or receive information provided by computing device102. For example, data source114can be a device that an agent employs to communicate with a customer. Computing device102can obtain information about the conversation from data source114. In another example, data source114can be a server that stores conversations between agents and customers. It is to be appreciated that computing device102and data source114can be equipped with communication components (not shown) that enable communication between computing device102, and data source114over one or more networks112.

The various devices (e.g., computing device102, and data source114) and components (e.g., agent conversation component104, memory108, processor106and/or other components) of system100can be connected either directly or via one or more networks112. Such networks112can include wired and wireless networks, including, but not limited to, a cellular network, a wide area network (WAN) (e.g., the Internet), or a local area network (LAN), non-limiting examples of which include cellular, WAN, wireless fidelity (Wi-Fi), Wi-Max, WLAN, radio communication, microwave communication, satellite communication, optical communication, sonic communication, or any other suitable communication technology.

FIG. 4illustrates a block diagram of an example, non-limiting agent-customer conversation402in accordance with one or more embodiments described herein. Repetitive description of like elements employed in one or more embodiments described herein is omitted for sake of brevity.

In this non-limiting example, the agent initiates the conversation with agent sentences A1, A2, A3, A4, A5, and A6. Agent sentences A1, A2, A3, A4, A5, and A6can be a first communication of the agent. The customer responds to the agent's first communication with customer response sentences R1and R2. Agent sentences A1, A2, A3, A4, A5, and A6and customer response sentences R1and R2can form a first communication exchange between the agent and customer. The agent can then communicate agent sentences A7, A8, A9, A10, and A11. Agent sentences A7, A8, A9, A10, and A11can be a second communication of the agent. The customer responds to the agent's second communication with customer response sentence R3. Agent sentences A7, A8, A9, A10, and A11and customer response sentences R3can form a second communication exchange between the agent and customer. It is to be appreciated that a conversation can comprise any suitable number of agent sentences, customer response sentences, and/or communication exchanges.

With reference toFIGS. 2 and 4, conversation analysis component202can analyze conversation402to determine customer response sentences that meet defined labeling criteria and which agent sentences lead to the labeled customer response sentences. For example, the defined labeling criteria can be indicative of an effective customer response sentence. Referring toFIG. 5, a block diagram of an example, non-limiting agent-customer conversation402after an analysis by conversation analysis component202in accordance with one or more embodiments described herein. Conversation analysis component202can determine that customer sentences R1and R3meet a defined labeling criterion. For example, customer sentences R1and R3can be determined to be desired responses from the customer according to a goal of conversation402. Conversation analysis component202can also determine that agent sentences A2and A3meet a defined criterion indicative of having a level of influence in eliciting customer sentence R1, and agent sentences A6and A10meet the defined criterion indicative of having a level of influence in eliciting customer sentence R3. One or more embodiments below describe in greater detail operations of conversation analysis component202.

FIG. 3illustrates a block diagram of an example, non-limiting conversation analysis component202in accordance with one or more embodiments described herein. Repetitive description of like elements employed in one or more embodiments described herein is omitted for sake of brevity.

Conversation analysis component202can include sequence analysis component302that prepares a conversation for employment by the AWRNNED. Sequence analysis component302can generate a text transcript of a conversation. For example, sequence analysis component302can employ speech to text recognition to convert an audio conversation into a text transcript. In another example, sequence analysis component302can employ optical character recognition to convert an image of a conversation into a text transcript.

Sequence analysis component302can filter a text transcript to remove noise from the conversation. In a non-limiting example, noise can be any irrelevant words and/or sentences in the conversation. For example, filler words such as “um”, “ah”, “you know”, “basically”, “actually”, or any other suitable filler words can be filtered out of the transcript. It is to be appreciated that irrelevant words and/or sentences can be determined based on the information describing the conversation, such as in the non-limiting example, purpose of conversation, type of agent, type of customer, type of relationship between agent and customer, industry of agent and/or customer, or any other suitable information that can be employed to discern irrelevant words and/or phrases in a conversation. Sequence analysis component302can also correct spelling errors and/or grammatical errors in the transcript.

Sequence analysis component302can label sentences from that agent with an identification as agent sentences, and sentences from the customer with an identification as customer response sentences. It is to be appreciated that any suitable identification can be employed for differentiation between agent sentences and customer response sentences.

Conversation analysis component202can include response labeling component304that obtains the prepared transcript and labels the customer response sentences and/or words with metadata describing sentiment, emotion, and information-types describing the sentences and/or words. Response labeling component304can employ one or more artificial intelligence functions to analyze customer response statements and make determinations regarding sentiment, emotion, and/or information-types being conveyed by the customer response statements. For example, response labeling component304can employ a psycholinguistic function to determine metadata related to sentiment, emotion, and/or information-types being conveyed by the customer response statements. For example, a psycholinguistic function can analyze text and make determinations with respect to psychological factors of the author of the text. In a non-limiting example, sentiment provides in an indication of a sentiment (e.g. level of positive and/or level of negative) that the customer is presenting with the customer response sentence, such as positive or negative sentiment. It is to be appreciated that sentiment can be a binary indication, an n-ary indication, or any other suitable indicator of sentiment. In another non-limiting example, emotion provides an indication of emotion that the customer is presenting with the customer response sentence, such as happy, sad, anger, frustrated, trust, surprise, or any other suitable indication of emotion. In a non-limiting example, any suitable standards for emotion indications can be employed, non-limiting examples of which can include Plutchik's wheel of emotion, Eckman's six universal emotions, Cowen and Keltner model, Lovheim cube of emotion, or any other suitable emotion categorization. Information type is a categorization of a word and/or sentence based on the particular domain (e.g. industry, topic, technology, or any other suitable domain) with which the conversation is related. It is to be appreciated that any suitable categorization for information type can be employed. For example, a technology company can have a particular set of categories for information type that is appropriate for the technology in which the company operates, while a retail store can have a different set of categories for information type that is appropriate for the retail space in which they operate. In another example, an insurance company can have a set of categories for information type that is appropriate for the insurance industry. It is to be appreciated that any suitable categorization for information type can be employed based on the domain of the conversation.

Conversation analysis component202can include model generation component306that generates (e.g. creates, trains, models, etc.) an Attention Weighted Recurrent Neural Network Encoder-Decoder (AWRNNED) model based on analysis of the labeled transcripts of conversations between agents and customers. Model generation component306employs an iterative process that concurrently trains encoder recurrent neural networks (RNNs) with attention weights and a decoder RNN of the AWRNNED using sentences of the labeled transcripts as discussed below in more detail with respect toFIGS. 6-9 and 11.

FIG. 6illustrates a block diagram of an example, non-limiting an AWRNNED614model training in accordance with one or more embodiments described herein. Repetitive description of like elements employed in one or more embodiments described herein is omitted for sake of brevity.

As discusses above, one or more conversations602can be obtained by sequence analysis component to produce one or more transcripts604. The one or more transcripts604can be obtained by response labeling component304to produce one or more prepared transcripts606, which can comprise labeled agent sentences (A1. . . Am)610and labeled customer response sentences (R1. . . Rp)608. Model generation component306can obtain the one or more prepared transcripts606and provides agent sentences (A1. . . Am) and customer response sentences (R1. . . Rp)608to AWRNNED614, as well as setting and/or adjusting parameters612of AWRNNED614. AWRNNED614can produce predicted customer response sentences (E1. . . Eu)616which can be compared to customer response sentences (R1. . . Rp)608by model generation component306to determine an error between predicted customer response sentences (G1. . . Gn)616and customer response sentences (R1. . . Rp)608. Model generation component306can employ an iterative process to adjust parameters612and determine the error until the error is within a threshold error criterion (e.g., convergence or any suitable error criterion) which represents a satisfactory training of the AWRNNED. It is to be appreciated that m is an integer that represents the number of agent sentences, p is an integer that represents the number of customer response sentences, and u is an integer that represents the number of predicted customer response sentences.

The iterative process that model generation component306employs can comprise:using a sentence encoder RNN (S-RNN) of AWRNNED to encode agent sentences into respective sentence vectors.using a paragraph encoder RNN (P-RNN) of AWRNNED to encode agent sentence vectors associated with agent sentences of one or more paragraphs.generating attention weighted vectors for each word of a customer response sentence based on hidden states from P-RNN using an attention weighting component310of AWRNNED to generate attention weighted vectors.using the attention weighted vectors and a decoder RNN (D-RNN) of AWRNNED to predict customer response sentences.determine an error value for the predicted customer response sentences as compared to the customer response sentences.if the error value does not meet an error criterion, then employ back propagation to modify one or more parameters of the S-RNN, P-RNN, D-RNN, and/or attention weighting component310, and repeat the process; otherwise, if the error value meets the error criterion, then the AWRNNED is trained.

FIG. 7illustrates a block diagram of an example, non-limiting operation of a sentence encoder RNN (S-RNN)702in accordance with one or more embodiments described herein. Repetitive description of like elements employed in one or more embodiments described herein is omitted for sake of brevity.

S-RNN702can obtain a sentence S(w1, w2, . . . , wt-1, wt)704. It is to be appreciated that sentence S can be an agent sentence or a customer response sentence, and t is an integer that represents the number of words in sentence S704. S-RNN702can employ an iterative process that goes word by word of sentence S704to generate a sentence vector V for sentence S704. In a non-limiting example, sentence vector V can be determined based on Equation 1.
V=ht=fw(wt,ht-1),  Equation (1)
where fwis any suitable function for encoding one or more words into a vector.

For example, S-RNN702can obtain word w1and generate hidden state h1. S-RNN702can then obtain word w2which can be used with hidden state h1to generate hidden state h2. S-RNN702can then obtain word wt-1which can be used with hidden state ht-2to generate hidden state ht-1. S-RNN702can then obtain word wtwhich can be used with hidden state ht-1to generate hidden state ht. S-RNN702can output sentence vector V based on hidden state ht. S-RNN702can be employed to encode agent sentences (A1. . . Am)610and/or customer response sentences (R1. . . Rp)608.

FIG. 8illustrates a block diagram of an example, non-limiting operation of a paragraph encoder RNN (P-RNN)802in accordance with one or more embodiments described herein. Repetitive description of like elements employed in one or more embodiments described herein is omitted for sake of brevity.

S-RNN702can obtain a paragraph P(S1, S2, . . . , Sn-1, Sn)804, where n is an integer that represents the number of sentences in paragraph P804. S-RNN702generate respective sentence vectors V1806a, V2806b, Vn-1806c, and Vn806dfor sentences S1, S2, . . . , Sn-1, Sn. P-RNN802can employ an iterative process that goes sentence by sentence of paragraph P804to encode the sentences of paragraph P804and generate hidden states b1. . . bn. In a non-limiting example, encoding by P-RNN802can be performed according to Equation 2.
bn=fs(Vn,bn-1)  Equation (2)
where fsis any suitable function for encoding one or more sentences into a vector.

For example, P-RNN802can obtain sentence vector V1806aand generate hidden state b1. P-RNN802can then obtain sentence vector V2806bwhich can be used with hidden state b1to generate hidden state b2. P-RNN802can then obtain sentence vector Vn-1806cwhich can be used with hidden state bn-2to generate hidden state bn-1. P-RNN802can then obtain sentence vector Vn806dwhich can be used with hidden state bn-1to generate hidden state bn. P-RNN802can be employed to encode agent sentences (A1. . . Am)610based on groupings of respective agent sentences into paragraphs P of a conversation.

In a non-limiting example, a paragraph can be employed during training of AWRNNED to predict a customer response sentence R. In an example, a paragraph can comprise only agent sentences preceding a customer response sentence within a single communication exchange. For example, referring back toFIG. 4, agent sentences A1, A2, A3, A4, A5, and A6can be can be employed to predict customer response sentences R1and/or R2, and agent sentences A7, A8, A9, A10, and A11can be used to predict customer response sentence R3.

In another example, a paragraph can comprise agent sentences preceding a customer response sentence within a plurality of communication exchanges. For example, referring again toFIG. 4, agent sentences A1, A2, A3, A4, A5, A6, A7, A8, A8, A10, and A11can be used to predict customer response sentence R3.

It is to be appreciated that a paragraph can comprise any suitable selection of agent sentences that precede a customer response sentence in a conversation.

FIG. 9illustrates a block diagram of an example, non-limiting operation of a sentence decoder RNN (D-RNN)904in accordance with one or more embodiments described herein. Repetitive description of like elements employed in one or more embodiments described herein is omitted for sake of brevity. D-RNN904can obtain one or more attention weighted vectors C and employ an iterative process that generates predicted words (r1, r2, . . . , re) one by one to generate predicted customer response sentence G(r1, r2, . . . , re), where e is any suitable integer indicating the number of words in predicted customer response sentence G.

Hidden states b1. . . bnare generated by P-RNN802from paragraph P(S1, S2, . . . , Sn-1, Sn)804. It is to be appreciated that S1, S2, . . . , Sn-1, Sncan be agent sentences that precede customer response sentence R(d1, . . . , de) in a training conversation, where d1, . . . , deare words of customer response sentence R. It is to be noted that customer response sentence R and predicted customer response sentence G have the same number of words e, and P(S1, S2, . . . , Sn-1, Sn)804is being employed to train the AWRNNED to predict customer response sentence R.

Model generation component306can comprise attention weighting component310that employs hidden states b1. . . bnand customer response sentence R(d1, . . . , de) to generate respective attention weighted vectors for words d1, . . . , deof customer response sentence R. For respective words dyin customer response sentence R, attention weighting component310can assign respective attention weights ayzto agent sentences Szof paragraph P, where y is an integer from 1 to e, and z is an integer from 1 to n. For respective words dyin customer response sentence R, attention weighting component310can determine attention weighted vectors Cybased on the sum of ayzhzfor j equals 1 to n. In a non-limiting example, attention weighted vector Cyfor word dyof customer response sentence R can be determined according to Equation 3.
Cy=Σj=1nayzbyEquation (3)

Model generation component306can employ one or more attention weighted vectors C1. . . Ceas input to D-RNN904of AWRNNED to produce predicted customer response sentences (G1. . . Gu)616, where u is an integer that represents the number of predicted customer response sentence G(r1, r2, . . . , re). Model generation component306can determine an error E between predicted customer response sentence G and customer response sentence R as described in more detail below.

For example, D-RNN904can obtain attention weighted vector CI908aand generate hidden state k1and predicted word r1. D-RNN904can then obtain attention weighted vector C2908bwhich can be used with hidden state k1and predicted word r1to generate hidden state k2and predicted word r2. D-RNN904can then obtain attention weighted vector Ce908cwhich can be used with hidden state ke-1and predicted word re-1to generate hidden state keand predicted word re. In this manner, going word by word, D-RNN904can generate predicted customer response sentence G(r1, r2, . . . , re). In a non-limiting example, decoding by D-RNN904can generate hidden states according to Equation 4.
ke=fd(Ce,ke-1,re-1)  Equation (4)
where fdis any suitable function for decoding one or more attention weighted vectors.

Model generation component306can generated predicted customer response sentences G for any suitable numbers of paragraphs P and customer response sentences R in a set of training conversations.

For example, referring back toFIG. 6, model generation component306can employ attention weighting component310to generate one or more attention weighted vectors C from hidden states generated from agent sentences (A1. . . Am)610with respect to customer response sentences (R1. . . Rp)608using AWRNNED. Model generation component306can employ one or more attention weighted vectors C as input to D-RNN of AWRNNED to produce predicted customer response sentences (G1. . . Gu)616. Model generation component306can compare predicted customer response sentences (G1. . . Gu)616to corresponding customer response sentences (R1. . . Rp)608, and determine an error E indicative of a difference between predicted customer response sentences (G1. . . Gu)616and customer response sentences (R1. . . Rp)608using any suitable error formula. It is to be appreciated that u and p can have equal values in training AWRNNED. For example, a textual comparison of predicted customer response sentences (G1. . . Gu)616with customer response sentences (R1. . . Rp)608can be employed to determine error E. In another example, respective sentence vectors associated with predicted customer response sentences (G1. . . Gu)616can be compared to respective sentence vectors associated with customer response sentences (R1. . . Rp)608to determine distances between corresponding sentence vectors. It is to be appreciated that any suitable mechanism can be employed to determine error E based on predicted customer response sentences (G1. . . Gu)616and customer response sentences (R1. . . Rp)608.

Model generation component306can compare E to an error criterion Ec. It is to be appreciated that error criterion Ecbe any suitable criterion indicative of AWRNNED being trained. If E does not meet error criterion Ec, model generation component306can employ back propagation to modify one or more parameters of the S-RNN, P-RNN, D-RNN, and/or attention weights, and repeat the iterative process. If E meets error criterion Ec, model generation component306can determine that AWRNNED is trained.

The learned attention weights a of the trained AWRNNED provide indications of the strength of respective relationships between agent sentences and customer response sentences and/or words of the customer response sentences.

For example, as discussed above model generation component306can learn attention weights ayzfor agent sentences Szof paragraph P relative to words dyof customer response sentence R. For respective agent sentences Szof paragraph P relative to customer response sentence R, attention weighting component310can determine respective attention weights azRbased on the sum of ayzfor y equals 1 to e. In a non-limiting example, attention weight azRfor agent sentence Szwith respect to customer response sentence R can be determined according to Equation 5.
azR=Σy=1eayzEquation (5)

In another example, for respective words dyin customer response sentence R, model generation component306can also assign significance weights gybased on the metadata describing sentiment, emotion, and/or information-types of the customer response sentence R and/or words dyin the labeled prepared transcript606. The significance weights gycan provide an indication of the importance of the customer response sentence R and/or words dy. For respective agent sentences Szof paragraph P relative to customer response sentence R, attention weighting component310can determine respective attention weights azRbased on the sum of ayzgyfor y equals 1 to e. In a non-limiting example, attention weight azRfor agent sentence SZwith respect to customer response sentence R can be determined according to Equation 6.
azR=Σy=1eayzgyEquation (6)

Conversation analysis component202can include grouping component308that can group pairs of agent sentences and customer response sentences based on the learned attention weights and/or semantic distance. For example, grouping component308can group agent sentences that are close to a single customer response sentence based on a threshold with respect to attention weight and/or semantic distance. In another non-limiting example, semantic distance can be determined based on a comparison of sentence vectors generated by S-RNN702for customer response sentences and agent sentences. It is to be appreciated that any suitable mechanism can be employed for determining semantic distance between agent sentences and customer response sentences. This can provide an indication of one or more agent sentences that can lead to a particular customer response sentence. In another example, grouping component308can group customer response sentences that are close to a single agent sentence based on a threshold with respect to attention weight and/or semantic distance. This can provide an indication of one or more customer response sentences that can result from a particular agent sentence. In a further example, grouping component308can create, from a group of group customer response sentence, a subgroup of customer response sentences that fall into a particular category with respect to sentiment, emotion, and/or information-type that are close to a single agent sentence based on a threshold with respect to attention weight and/or semantic distance. This can provide an indication of one or more customer response sentences that fall into particular category with respect to sentiment, emotion, and/or information-type that can result from a particular agent sentence. It is to be appreciated the grouping component can group pairs of agent sentences and customer response sentences into any suitable groups according to any suitable criterion.

Agent conversation component104can employ the groupings to automatically generate an output (e.g. a user interface on a display, a report, a document, a chat bot, or any other suitable output) that be employed to train an agent. For example, an agent can be trained using a computer where a display can indicate one or more agent sentences that can lead to a particular customer response sentence, or indicate one or more customer response sentences that can result from a particular agent sentence, or one or more agent sentences that can lead to one or more customer responses that fall into particular category with respect to sentiment, emotion, and/or information-type. In another example, an interactive chat-bot can engage with an agent in training that employs the groupings to simulate customer response sentences based on sentences the agent provides.

Referring back toFIG. 2, agent conversation component104can also include recommendation component204that can automatically analyze an ongoing conversation between an agent and a customer in real-time using an AWRNNED614, and provide one or more recommendations to the agent with one or more agent sentences to use and/or avoid in order to steer the conversation in a particular direction with respect to eliciting one or more particular customer response sentences. For example, during the ongoing conversation, agent sentences and customer response sentences can be analyzed using AWRNNED614in real-time and agent conversation component104can provide an output (e.g., display, audio, or any other suitable output) comprising one or more agent sentences to use and/or avoid in order to steer the ongoing conversation in a particular direction with respect to eliciting one or more particular customer response sentences associated with a goal of the ongoing conversation.

Agent conversation component104can also include performance analysis component206that can automatically analyze an agent's conversations over a period of time using an AWRNNED614to determine the agent's performance. For example, the agent's conversations can be input to AWRNNED614to determine relationships between the agent's sentences and the customer response sentences in the agent's conversation. The determined relationship can provide an indication of how effective the agent is at eliciting particular customer response sentences based on the agent's sentences. For example, effectiveness can be based on the quantity of agent sentences needed to elicit a particular customer response sentence, time taken to elicit a particular customer response sentence, usage of the closest one or more agent sentences to the particular customer response sentence according to attention weight and/or semantic distance, or any other suitable measure of effectiveness.

WhileFIGS. 1, 2, 3, 4, 5, 6, 7, 8, and 9depict separate components in computing device102, it is to be appreciated that two or more components can be implemented in a common component. Further, it is to be appreciated that the design of the computing device102can include other component selections, component placements, etc., to facilitate automatically analyzing conversation between agents and customers using an AWRNNED in accordance with one or more embodiments described herein. Moreover, the aforementioned systems and/or devices have been described with respect to interaction between several components. It should be appreciated that such systems and components can include those components or sub-components specified therein, some of the specified components or sub-components, and/or additional components. Sub-components could also be implemented as components communicatively coupled to other components rather than included within parent components. Further yet, one or more components and/or sub-components can be combined into a single component providing aggregate functionality. The components can also interact with one or more other components not specifically described herein for the sake of brevity, but known by those of skill in the art.

Further, some of the processes performed can be performed by specialized computers for carrying out defined tasks related to automatically analyzing conversation between agents and customers using an AWRNNED. The subject computer processing systems, methods apparatuses and/or computer program products can be employed to solve new problems that arise through advancements in technology, computer networks, the Internet and the like. The subject computer processing systems, methods apparatuses and/or computer program products can provide technical improvements to systems automatically analyzing conversation between agents and customers using an AWRNNED in a live environment by improving processing efficiency among processing components in these systems, reducing delay in processing performed by the processing components, and/or improving the accuracy in which the processing systems automatically analyzing conversation between agents and customers using an AWRNNED.

The embodiments of devices described herein can employ artificial intelligence (AI) to facilitate automating one or more features described herein. The components can employ various AI-based schemes for carrying out various embodiments/examples disclosed herein. In order to provide for or aid in the numerous determinations (e.g., determine, ascertain, infer, calculate, predict, prognose, estimate, derive, forecast, detect, compute) described herein, components described herein can examine the entirety or a subset of the data to which it is granted access and can provide for reasoning about or determine states of the system, environment, etc. from a set of observations as captured via events and/or data. Determinations can be employed to identify a specific context or action, or can generate a probability distribution over states, for example. The determinations can be probabilistic—that is, the computation of a probability distribution over states of interest based on a consideration of data and events. Determinations can also refer to techniques employed for composing higher-level events from a set of events and/or data.

Such determinations can result in the construction of new events or actions from a set of observed events and/or stored event data, whether or not the events are correlated in close temporal proximity, and whether the events and data come from one or several event and data sources. Components disclosed herein can employ various classification (explicitly trained (e.g., via training data) as well as implicitly trained (e.g., via observing behavior, preferences, historical information, receiving extrinsic information, etc.)) schemes and/or systems (e.g., support vector machines, neural networks, expert systems, Bayesian belief networks, fuzzy logic, data fusion engines, etc.) in connection with performing automatic and/or determined action in connection with the claimed subject matter. Thus, classification schemes and/or systems can be used to automatically learn and perform a number of functions, actions, and/or determination.

FIG. 10illustrates a flow diagram of an example, non-limiting computer-implemented method1000that facilitates automatically analyzing one or more conversations between agents and customers using an AWRNNED are provided in accordance with one or more embodiments described herein. Repetitive description of like elements employed in other embodiments described herein is omitted for sake of brevity.

At1002, method1000can comprise obtaining, by a system operatively coupled to a processor, one or more agent-customer conversations (e.g., via a sequence analysis component302, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1004, computer-implemented method1000can comprise generating, by the system, respective prepared transcripts of the agent-customer conversations (e.g., via a sequence analysis component302, a response labeling component304, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1006, computer-implemented method1000can comprise training, by the system, an attention weighted recurrent neural network encoder-decoder (AWRNNED) model using the respective prepared transcripts (e.g., via a model generation component306, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1008, computer-implemented method1000can comprise generating, by the system, groupings of relationships between agent sentences and customer response sentences of the one or more agent-customer conversations based on learned attention weights of the AWRNNED and/or semantic distance (e.g., via a grouping component308, a conversation analysis component202, an agent conversation component104, and/or a computing device102).

FIG. 11illustrates a flow diagram of an example, non-limiting computer-implemented method1100that facilitates automatically generating a prepared transcript in accordance with one or more embodiments described herein. Repetitive description of like elements employed in other embodiments described herein is omitted for sake of brevity.

At1102, computer-implemented method1100can comprise obtaining, by a system operatively coupled to a processor, an agent-customer conversation (e.g., via a sequence analysis component302, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1104, computer-implemented method1100can comprise generating, by the system, a transcript of the agent-customer conversation (e.g., via a sequence analysis component302, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1106, computer-implemented method1100can comprise filtering, by the system, noise from the transcript of the agent-customer conversation (e.g., via a sequence analysis component302, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1108, computer-implemented method1100can comprise labeling, by the system, agent sentences in the transcript with indications of being an agent sentence (e.g., via a sequence analysis component302, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1110, computer-implemented method1100can comprise labeling, by the system, customer response sentences in the transcript with indications of being a customer response sentence (e.g., via a sequence analysis component302, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1112, computer-implemented method1100can comprise labeling, by the system, customer response sentences in the transcript with metadata indicative of sentiment, emotion, and/or information-type (e.g., via a response labeling component304, a conversation analysis component202, an agent conversation component104, and/or a computing device102).

FIG. 12illustrates a flow diagram of an example, non-limiting computer-implemented method1200that facilitates automatically training an attention weighted recurrent neural network encoder-decoder (AWRNNED) model in accordance with one or more embodiments described herein. Repetitive description of like elements employed in other embodiments described herein is omitted for sake of brevity.

At1202, computer-implemented method1200can comprise obtaining, by a system operatively coupled to a processor, one or more prepared transcripts of one or more agent-customer conversations (e.g., via a model generation component306, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1204, computer-implemented method1200can comprise encoding, by the system, agent sentences from the transcript into respective sentence vectors using a S-RNN (e.g., via a model generation component306, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1206, computer-implemented method1200can comprise encoding, by the system, one or more paragraphs of agent sentences from the transcript using a P-RNN to generate hidden states corresponding to the agent sentences (e.g., via a model generation component306, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1208, computer-implemented method1200can comprise generating, by the system, respective attention weighted vectors from the hidden states (e.g., via an attention weighting component310, a model generation component306, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1210, computer-implemented method1200can comprise generating, by the system, one or more predicted customer response sentences from the attention weighted vectors using a D-RNN (e.g., via a model generation component306, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1212, computer-implemented method1200can comprise determining, by the system, an error for the one or more predicted customer response sentences as compared to corresponding one or more customer response sentences (e.g., via a model generation component306, a conversation analysis component202, an agent conversation component104, and/or a computing device102). At1214, computer-implemented method1200can comprise determining, by the system, whether the error meets an error criterion (e.g., via a model generation component306, a conversation analysis component202, an agent conversation component104, and/or a computing device102). If the determination at1214is “NO” meaning that the error does not meet the error criterion, then the computer-implemented method proceeds to1216. If the determination at1214is “YES” meaning that the error does meet the error criterion, then the computer-implemented method ends and the AWRNNED is trained. At1216, computer-implemented method1200can comprise adjusting, by the system, one or more parameters of the S-RNN, P-RNN, D-RNN, and/or attention weights (e.g., via a model generation component306, a conversation analysis component202, an agent conversation component104, and/or a computing device102).

In order to provide a context for the various aspects of the disclosed subject matter,FIG. 13as well as the following discussion are intended to provide a general description of a suitable environment in which the various aspects of the disclosed subject matter can be implemented.FIG. 13illustrates a block diagram of an example, non-limiting operating environment in which one or more embodiments described herein can be facilitated. Repetitive description of like elements employed in other embodiments described herein is omitted for sake of brevity.

With reference toFIG. 13, a suitable operating environment1300for implementing various aspects of this disclosure can also include a computer1312. The computer1312can also include a processing unit1314, a system memory1316, and a system bus1318. The system bus1318couples system components including, but not limited to, the system memory1316to the processing unit1314. The processing unit1314can be any of various available processors. Dual microprocessors and other multiprocessor architectures also can be employed as the processing unit1314. The system bus1318can be any of several types of bus structure(s) including the memory bus or memory controller, a peripheral bus or external bus, and/or a local bus using any variety of available bus architectures including, but not limited to, Industrial Standard Architecture (ISA), Micro-Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Firewire (IEEE 1494), and Small Computer Systems Interface (SCSI). The system memory1316can also include volatile memory1320and nonvolatile memory1322. The basic input/output system (BIOS), containing the basic routines to transfer information between elements within the computer1312, such as during start-up, is stored in nonvolatile memory1322. By way of illustration, and not limitation, nonvolatile memory1322can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, or nonvolatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM). Volatile memory1320can also include random access memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM.

Computer1312can also include removable/non-removable, volatile/non-volatile computer storage media.FIG. 13illustrates, for example, a disk storage1324. Disk storage1324can also include, but is not limited to, devices like a magnetic disk drive, floppy disk drive, tape drive, Jaz drive, Zip drive, LS-100 drive, flash memory card, or memory stick. The disk storage1324also can include storage media separately or in combination with other storage media including, but not limited to, an optical disk drive such as a compact disk ROM device (CD-ROM), CD recordable drive (CD-R Drive), CD rewritable drive (CD-RW Drive) or a digital versatile disk ROM drive (DVD-ROM). To facilitate connection of the disk storage1324to the system bus1318, a removable or non-removable interface is typically used, such as interface1326.FIG. 13also depicts software that acts as an intermediary between users and the basic computer resources described in the suitable operating environment1300. Such software can also include, for example, an operating system1328. Operating system1328, which can be stored on disk storage1324, acts to control and allocate resources of the computer1312. System applications1330take advantage of the management of resources by operating system1328through program modules1332and program data1334, e.g., stored either in system memory1316or on disk storage1324. It is to be appreciated that this disclosure can be implemented with various operating systems or combinations of operating systems. A user enters commands or information into the computer1312through input device(s)1336. Input devices1336include, but are not limited to, a pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, web camera, and the like. These and other input devices connect to the processing unit1314through the system bus1318via interface port(s)1338. Interface port(s)1338include, for example, a serial port, a parallel port, a game port, and a universal serial bus (USB). Output device(s)1340use some of the same type of ports as input device(s)1336. Thus, for example, a USB port can be used to provide input to computer1312, and to output information from computer1312to an output device1340. Output adapter1342is provided to illustrate that there are some output devices1340like monitors, speakers, and printers, among other output devices1340, which require special adapters. The output adapters1342include, by way of illustration and not limitation, video and sound cards that provide a means of connection between the output device1340and the system bus1318. It should be noted that other devices and/or systems of devices provide both input and output capabilities such as remote computer(s)1344.

Computer1312can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s)1344. The remote computer(s)1344can be a computer, a server, a router, a network PC, a workstation, a microprocessor based appliance, a peer device or other common network node and the like, and typically can also include many or all of the elements described relative to computer1312. For purposes of brevity, only a memory storage device1346is illustrated with remote computer(s)1344. Remote computer(s)1344is logically connected to computer1312through a network interface1348and then physically connected via communication connection1350. Network interface1348encompasses wire and/or wireless communication networks such as local-area networks (LAN), wide-area networks (WAN), cellular networks, etc. LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet, Token Ring and the like. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks like Integrated Services Digital Networks (ISDN) and variations thereon, packet switching networks, and Digital Subscriber Lines (DSL). Communication connection(s)1350refers to the hardware/software employed to connect the network interface1348to the system bus1318. While communication connection1350is shown for illustrative clarity inside computer1312, it can also be external to computer1312. The hardware/software for connection to the network interface1348can also include, for exemplary purposes only, internal and external technologies such as, modems including regular telephone grade modems, cable modems and DSL modems, ISDN adapters, and Ethernet cards.

In an embodiment, for example, computer1312can perform operations comprising: in response to receiving a query, selecting, by a system, a coarse cluster of corpus terms having a defined relatedness to the query associated with a plurality of coarse clusters of corpus terms; determining, by the system, a plurality of candidate terms from search results associated with the query; determining, by the system, at least one recommended query term based on refined clusters of the coarse cluster, the plurality of candidate terms, and the query; and communicating at least one recommended query term to a device associated with the query.

It is to further be appreciated that operations of embodiments disclosed herein can be distributed across multiple (local and/or remote) systems.