Automatic generation of playlists from conversations

A moment identification system automatically generates a playlist of conversations having a specified moment. A moment can be occurrence of a specific event or a specific characteristic in a conversation, or any event that is of specific interest for an application for which the playlist is being generated. For example, a moment can include laughter, fast-talking, objections, response to questions, a discussion on a particular topic such as budget, behavior of a speaker, intent to buy, etc., in a conversation. The moment identification system analyzes each of the conversations to determine if one or more features of a conversation correspond to a specified moment, and includes those of the conversations in the playlist having one or more features that correspond to the specified moment. The playlist may include a portion of a conversation that has the specified moment rather than the entire conversation.

BACKGROUND

With over 2.4 million non-retail inside sales representatives in the United States (U.S.) alone, millions of sales phone conversations are made on a daily basis.iHowever, except for rudimentary statistics concerning e.g., call length and spotted keywords and phrases, sales conversations are left largely unanalyzed, rendering their content inaccessible to modeling, and precluding the ability to optimize them for desired outcomes. Recent advances in automatic speech recognition (ASR) technologies, and specifically in large vocabulary continuous speech recognition (LVCSR), are for the first time enabling high-accuracy automatic transcription of conversations. At the same time, natural language processing (NLP) approaches to both topic modeling and world-knowledge modeling, have become much more efficient due to the availability of large, freely accessible natural language corpora (e.g., CommonCrawl), as well as freely available ontologies or “knowledge graphs” (e.g., DBpedia). Finally, recent research on affect identification applying machine learning (ML) has been able to successfully model subjective aspects of emotion and personality traits as perceived by listeners.iInsidesales.com “Market size 2013” study

DETAILED DESCRIPTION

Embodiments are disclosed for a call-modeling system for modeling conversations, e.g., voice conversations, in real time, with the goal of helping users, e.g., sales representatives and/or their managers, to improve and/or guide the outcome of conversations with other users, e.g., customers. One such embodiment can model the calls based on characteristics of the conversation, e.g., voice of the representatives and/or the customers, and content of the conversation, with the goal of positively influencing the outcome of the call. The call-modeling system can generate real-time probabilities for possible outcomes of a real-time conversation, e.g., an ongoing conversation between a specific representative and a customer, and generate specific on-call guidance, which may be either conducive or detrimental to a desired conversation outcome. The generated probabilities and on-call guidance may be used by the representatives and/or their managers to either increase the probability of a desired outcome and/or optimize the conversation for a specified duration if the predicted outcome is not going to be a desired outcome. For example, for renewing a magazine subscription, the call-modeling system can generate an on-call guidance suggesting a representative to engage in a rapport building exercise with the customer if it is determined that doing so increases the chances of the customer renewing the membership by 45%. On the other hand, if the call-modeling system predicts from the on-going conversation that the customer is not going to renew the subscription membership, then the call-modeling system can suggest the representative to wrap up the conversation as soon as possible.

The call-modeling system can include (i) an offline analysis component and (ii) a real-time analysis component. The offline analysis component can take as input conversations between a calling party, e.g., a customer, and a called party, e.g., a representative, and process the conversations using multiple distinct components to generate multiple features of the conversations. In some embodiments, the conversations can be audio recordings of calls between called party and the calling party (collectively referred to as “participants”). The features can include transcripts of audio recordings, vocabulary, semantic information of conversations, summarizations of utterances and various natural language entailments, summarization of a call, voice signal associated features (e.g., a speech rate, a speech volume, a tone, and a timber), emotions (e.g., fear, anger, happiness, timidity, fatigue), personality traits (e.g., trustworthiness, engagement, likeability, dominance, etc.), personal attributes (e.g., an age, an accent, and a gender), customer-representative pair attributes that indicate specific attributes associated with both the speakers that contribute to a specified outcome (e.g., similarity of speech rate between the representative and the user, extrovert/introvert matching, or gender or age agreement).

Note that a recording of the conversation can be of a conversation that is any of telephone based, Voice over Internet Protocol (VoIP) based, video conference based, Virtual Reality (VR) based, Augmented Reality (AR) based, or based on any online meetings, collaborations or interactions, electronic mail (e-mail). The recording can also be of a conversation that has happened between two or more speakers physically located in the same room. In some embodiments, a recording based on any online meetings, collaborations or interactions, or email can be a transcript of the corresponding interaction.

Further, the features can include not only aural features, but also non-aural features, e.g., visual features such as body language of a participant, and facial expressions of the participant, or any combination of aural and non-aural features. The features could also be generated from the transcripts of any of emails, online messages, and online meetings. In some embodiments, any of a word, a phrase, a text, emoji, symbols, or a combination thereof can be used to determine a particular feature. For example, it can be determined that a text such as “Ha Ha” or “rofl” in the transcript can indicate laughter.

In some embodiments, the audio recordings can be tagged with various tags, e.g., a tag that indicates a trait (e.g., “extrovert”, “trustworthy voice”, “anxious”, etc.) of one or more of the participants, a tag that indicates a call outcome (e.g., “sales closed”, “sales failed”, or “follow-up call scheduled”), and/or a tag that indicates “key moments” of a conversation. A “key moment” or a “moment” can be a specific event or a specific characteristic which occurs in the call. The event can be any event that is of specific interest for a specific application for which the call-modeling system is being implemented. An administrator of the call-modeling system can configure what events in a call have to be identified as a moment. For example, a moment can be laughter, engagement, fast-talking, open-ended questions, objections, or any combination thereof in a conversation. In some embodiments, the moments are identified automatically by a moment identification system. The tags can be generated automatically by the call-modeling system, manually, e.g., by human judgment, or both. In some embodiments, the tags are generated automatically by the call-modeling system. The tag can include various details, e.g., information regarding a moment, a time interval at which the moment occurred in the call, duration for which the moment lasted, information regarding the participants of the call, etc.

The moments can also be notified to and/or shared between the participants during an on-going conversation and/or after the conversation. For example, during a call between a user and a representative, the call-modeling system can analyze the call, identify the moments in the conversation, and tag, notify and/or share the moments with the representative's manager, peers or other people. The shared moments can be used for various purposes, e.g., for coaching the representatives in handling the calls to improve outcomes of the calls based on various situations. The moments can be shared using various means, e.g., via email, a chat application, or a file sharing application.

In some embodiments, the offline analysis component uses distinct components to extract the features. The components can include an automatic speech recognition (ASR) component, which can produce a transcription of the conversation, a natural language processing (NLP) component, which can extract semantic information (such as open-ended questions asked, key objections, etc.) from the conversation, an affect component, which can analyze the recording for emotional signals and personality traits (e.g., likeability and trustworthiness), and a metadata component, which can extract data regarding conversation flow (i.e., who spoke when, and how much silence and overlap occurred).

The offline analysis component can analyze the features to generate one or more classifiers that indicate conversation outcomes, e.g., “sales closed”, “sales failed.” Each of the classifiers indicates a specific outcome and can include a set of features that contribute to the specific outcome. The offline analysis component can generate multiple classifiers for the same outcome; however, the multiple classifiers can have distinct sets of features. In some embodiments, the offline analysis component can analyze the features using a machine learning algorithm (e.g., a linear classifier, such as a support vector machine (SVM), or a non-linear algorithm, such as a deep neural network (DNN) or one of its variants) to generate the classifiers.

In some embodiments, the offline analysis component generates a classifier for different time intervals or time windows of the conversations. For example, the offline analysis component can analyze the extracted features for seconds 00:05-00:10 of a conversation, seconds 00:20-00:30, and minutes 1:00-2:00, and generate a classifier for each of those time windows. The offline analysis component feeds the extracted features into the machine-learning algorithm to produce multiple classifiers corresponding to the time windows. The time windows can be of varying lengths or fixed lengths. In some embodiments, the offline analysis component can generate classifiers for other relative positions of a conversation. For example, the offline analysis component can generate a classifier corresponding to an instance in the conversation, e.g., when a customer spoke for the first time in the conversation, and analyze features such as a pitch of the voice, a topic the customer spoke about first, and the length of the customer's first talk, to generate the classifier.

The real-time analysis component uses the classifiers to model a real-time conversation, e.g., an ongoing call between a representative and a customer, that helps the representative to increase a probability of a desired outcome of the conversation or optimize the conversation duration in case the real-time analysis component does not predict the desired outcome. The real time analysis component receives real-time call data of an ongoing conversation between the customer and a representative and analyzes the real-time call data to generate a set of features, e.g., using the offline analysis component as described above. The real-time analysis component can then feed the features to the classifiers to generate probabilities of potential outcomes of the call. The real-time analysis component can use the classifiers with highest prediction powers to generate the probabilities of various potential outcomes. In some embodiments, the real-time analysis component measures the prediction powers of the classifiers using an F-score, which, in statistical analysis, is a (possibly weighted) harmonic mean of precision and recall.

The real-time analysis component feeds the extracted features into the classifiers with high F-scores to generate probabilities of possible outcomes. Based on the probabilities, the real-time analysis component can also generate on-call guidance, which encourages the representative and/or their managers to modify, desist or persist with a specified on-call behavior to increase or decrease the probability of one of the possible outcomes, e.g., a desired outcome such as closing a sale. In some embodiments, the on-call guidance includes a set of suggested features and their values to be adopted, desisted or persisted with by the representative. For example, the on-call guidance can include instructions for the representative to change the rate of speech (e.g., speak slower), use specific key words, or pose more open-ended questions to the customer.

In some embodiments, the on-call guidance can change as the call progresses, e.g., based on the classifiers that are relevant to the call at that particular time of the conversation. For example, during the first two minutes of the call, a classifier that corresponds to the first two minutes of the call may be used to generate the on-call guidance such as instructing the representative to pose open-ended questions to the customer, and then in the third minute, a classifier that corresponds to the third minute of the call may be used to revise the on-call guidance, e.g., suggest to the representative to adjust the speech rate to match with that of the customer.

Additionally, if according to the classifiers, the real-time analysis component predicts the conversation to fail, the on-call guidance may suggest to the representative to quickly wrap up the call in order to spare the representative's time. The on-call guidance of the real-time analysis module may be presented on-screen or via any other interface (e.g., voice instructions given through an ear piece) to the representative and/or the manager. The embodiments can produce real-time probabilities of various outcomes of the conversations, enabling live coaching that can help the representatives in improving the outcomes of the conversations in real-time.

Embodiments are also disclosed for automatically generating a playlist of conversations having a specified moment. As described above, a moment can be a specific event or a specific characteristic which occurs in the conversation. The moment can be any event that is of specific interest for a specific application for which the call-modeling system is being implemented. An administrator of the call-modeling system can configure what events in a conversation have to be identified as a moment. For example, a moment can be laughter, fast-talking, open-ended questions, objections, response to questions, a discussion on a particular topic such as “budget,” behavior of a speaker, psychological reaction such as engagement, boredom, intent to buy, etc., or any combination thereof, in a conversation.

A moment identification system can analyze the extracted features of the conversation and identify the specified moment based on the features. For example, if the specified moment to be identified in the conversation is laughter, the moment identification system can analyze the features of the conversation to determine if the conversation had any laughter by any of the speakers in the conversation, e.g., a sales representative of an organization or a customer with who the sales representative is interacting. If any of the features indicate that the conversation had a laughter, then the moment identification system can record the moment metadata, which can include one or more of (a) one or more locations of the conversation at which there is a laughter (such as at minute 5 and minute 6 of the conversation), (b) who is the laughter from, or (c) how long the laughter is. The moment identification system can analyze recordings of a number of conversations and identify all conversations that contain the specified moment, and then generate a playlist of the conversations containing the specified moment.

The moment identification system can generate playlists for various moments and store them in a storage system. For example, a playlist can be generated for conversations having a moment in which the sales representatives responded to questions from the customer. In another example, a playlist can be generated for conversations having a moment in which laughter was recorded from the customer. In yet another example, a playlist can be generated for conversations having a moment in which the customer was angry. In still another example, a playlist can be generated for conversations having a moment in which the topic involved was “budget” or “pricing”. The playlists can be used for various purposes, e.g., coaching sales representatives for improving their sales performance, improving an outcome of the sales call. For example, the playlist for “laughter” moment can be used to identify what it is that the sales representative in the conversations said or did that made the customers laugh, and coach other sales representatives to converse on a similar line to make the customers laugh. In another example, the playlist for “anger” moment can be used to identify what is it that the sales representative said or did that made the customers angry, and coach other sales representatives to avoid or take control of such situations. In yet another example, a playlist for “high engagement by the prospect” can be used to coach other sales representative how to ask engaging questions. Another usage of such moment playlist could be to educate parts of the company outside the sales department. For example, a playlist of moments discussing a newly launched feature may be used to share the “voice of the customer” with the product and marketing teams.

Some moments can be referred to as “interesting” moments. A consumer user, e.g., a consumer of the playlists such as sales representatives or managers of sales representatives, can define what moments are interesting. For example, the consumer user can define moments such as “laughter,” discussions on topics such as “budget” or “pricing”, or prospects' reactions to newly launched product features, as interesting moments. In some embodiments, the moment identification system can automatically determine what moments may be interesting to the consumer users. The moment identification system can automatically identify certain moments as interesting moments in various ways. For example, the moment identification system can identify the interesting moments based on a usage pattern of a recording of a conversation, such as if a particular portion of the conversation has been tagged, or replayed more than a threshold number of times. In another example, the moment identification system can identify the interesting moments based on an anomaly in a conversation, such as a speaker suddenly starting to speak at a pace greater than a specified threshold or compared to that of in other portions of the conversation, a speaker exiting a meeting room while the meeting is still in progress, occurrence of a particular facial expression, a particular portion of the conversation being played in a speed or at a volume level different from the other portions. Various such moments can be identified as interesting moments by the moment identification system, and playlists can be generated for the identified interesting moments.

Turning now toFIG. 1,FIG. 1is a block diagram of a call-modeling system100in which the disclosed embodiments can be implemented. The call-modeling system100includes an offline analysis component110and a real-time analysis component130. The offline analysis component110can take as input historical call data105, which includes conversations between participants, e.g., audio recordings of calls between representatives and customers, and process the call data105using multiple components to generate features115of the conversations, and classifiers120.

The offline analysis component110includes a feature generation component111that generates features115by analyzing the call data105using various techniques, e.g., ASR, NLP, artificial intelligence (AI), machine learning (ML). The features115can include transcripts of audio recordings, vocabulary, semantic information of conversations, summarization of a call, summarizations of utterances and various natural language entailments, voice signal associated features (e.g., speech rate, speech volume, tone, and timber), emotions (e.g., fear, anger, happiness, timidity, fatigue), personality traits (e.g., trustworthiness, engagement, likeability, dominance, charisma, confidence, etc.), personal attributes (e.g., age, accent, and gender), and inter-speaker attributes that indicate a comparison between both the speakers (e.g., similarity of speech rate between the representative and the user, extrovert/introvert matching, or gender or age agreement).

The classifier component112analyzes the features115using various techniques, e.g., machine learning algorithms such as SVM, DNN, to generate the classifiers120. The classifiers120indicate conversation outcomes, e.g., “sales closed”, “sales failed,” “probability of recommending to a friend,” a measure of “customer satisfaction,” and Net Promoter Score (NPS). An outcome can have binary values, e.g., “yes/no”, “high/low”, or non-binary values, e.g., a probability score, enumerated values like “low, average, medium, high, very high,” values on a scale of 0-10, etc. For example, an outcome such as customer satisfaction can be measured using binary values such as “low/high”, or using non-binary values, such as a scale of 0-10, enumerated values. Each of the classifiers indicates a specific outcome, a probability of the specified outcome and can include a set of the features that contributed to the specific outcome. For example, in a sales call for renewing a magazine subscription, a classifier “C1” can indicate that when laughter by a customer and two open-ended questions from the representative are registered, there is a high chance, e.g., 83%, of renewal.

In some embodiments, the classifier component112generates different classifiers for different time windows of the conversations. For example, the classifier component112generates a classifier “C1” for the first two minutes of the conversations and a classifier “C2” for a third minute of the conversations. The classifier “C1” based on the first two minutes of the conversation can indicate that when laughter by a customer and two open-ended questions from the representative is registered, there is a high chance, e.g., 83%, of renewal. The classifier “C2” based on the third minute of the conversation can indicate that when a competitor magazine or the key-phrase “read online” is used, the renewal chances drop to 10%, all of which can occur if customer's speech rate drops below three words per second. Some of the classifiers include features for inter-speaker attributes that indicate a comparison between the speakers that contribute to a specified outcome (e.g., similarity of speech rate between the representative and the user, extrovert/introvert matching, or gender or age agreement).

The features, when extracted from the conversations, can include attributes and values. The classifier determines what values of the features influence a particular outcome of the call. The classifiers120can be generated in various formats and is not limited to the above illustrated example format. The classifier component112can generate multiple classifiers for the same outcome; however, the multiple classifiers can have distinct sets of features. Further, as described above, the classifier component112can generate different classifiers for different time windows of the conversation. The offline analysis component110can store the features115and the classifiers120in a storage system125.

The call-modeling system100includes a real-time analysis component130that uses the classifiers120to generate on-call guidance for both inbound and outbound calls that will help the representative optimize the call for a desired outcome, or optimize the call duration if the desired outcome is not predicted (i.e., very low chances of the desired outcome are predicted). The real-time analysis component130receives real-time call data150of an ongoing conversation between a customer and a representative and analyzes the real-time call data150to generate a set of features, e.g., call features135, for the ongoing conversation using a feature generation component113. In some embodiments, the feature generation component113is similar to or the same as the feature generation component111. The feature generation component113generates the call features135based on the real-time call data150, e.g., as described above with respect to the feature generation component111. The real-time call data150can be an early-stage or initial conversation between the customer and the representative.

After the call features135are generated, a classifier component114, which, in some embodiments, is the same as, or similar to the classifier component112, inputs the call features135to the classifiers120to determine a set of classifiers140that predict possible outcomes of the call based on the call features135. Each of the set of classifiers140indicates a specified outcome of the call and an associated probability of the corresponding outcome. In some embodiments, the classifier component114chooses classifiers that have the highest prediction power, which can be measured using an F-score, as the set of classifiers140. After the set of classifiers140are determined, a call-modeling component116generates an on-call guidance145that includes real-time probabilities of possible outcomes of the call as indicated by the set of classifiers140. The call-modeling component116can further analyze the set of classifiers140to determine features that have high prediction power, e.g., prediction power exceeding a specified threshold, for predicting a desired outcome, and include those features and values associated with those features in the on-call guidance145. The on-call guidance145notifies the representative to adopt, desist or persist with an on-call behavior consistent with those features to achieve the desired outcome, or to increase the probability of achieving the desired outcome. If the set of classifiers140predict that the desired outcome may not be achieved, the call-modeling component116may suggest, in the on-call guidance145, that the representative wrap up the call.

The call data105can be in various formats, e.g., audio recordings, transcripts of audio recordings, online chat conversations. Similarly, the real-time call data150can be in various formats, e.g., real-time audio stream of the call, a chat transcript of an ongoing conversation in an online chat application. Further, the real-time call data150, which can include an initial or early stage conversation, can be a conversation between the customer and an automated machine, e.g., an interactive voice response (IVR) system, or a representative for gathering preliminary information from the customer that can be useful for generating the on-call guidance.

In some embodiments, the call-modeling system100includes a search tool that empowers a consumer user to explore various aspects of a conversation. For example, the search tool allows the consumer user to search for anything that came up on the call, e.g., both linguistic and meta-linguistic. The search tool can be used to further analyze the conversation, extract appropriate features and use them to improve the classifiers in predicting the outcome of the calls. For example, the search tool can be used to find calls that registered a laughter from the customer, calls in which the customer spoke for the first time after a specified number of minutes, calls in which the customer sounded angry, calls in which customer mentioned competitors, calls in which the representatives engaged in rapport building, calls in which the representative modulated speech rates at various instances of the call, calls in which short or open-ended questions were asked at a high frequency, or any combination of the above.

FIG. 2is a block diagram of a feature generation component ofFIG. 1for extracting features from call data, consistent with various embodiments. In some embodiments, the feature generation component111includes an ASR component210, an NLP component225, an affect component215and a metadata component220. The ASR component210can analyze call data205, e.g., a voice recording, and produce a transcription, vocabulary, and a language model of the conversation. The NLP component225can extract semantic information, such as key objection handling responses, from the output of the ASR component210. The affect component215can analyze the call data205for emotional signals and personality traits (e.g., likeability, extroversion/introversion, charisma, confidence, and trustworthiness) as well as general personal attributes such as gender, age, and accent of the participants. The metadata component220can extract data regarding conversation flow (e.g., who spoke when, and how much silence and overlap occurred). In some embodiments, the above components can process the call data105in parallel. The output of the components can be generated as features115of the conversations, which can be analyzed to determine outcomes of the conversations.

The ASR component210may be tuned for specific applications, e.g., for sales calls. The features produced by the ASR component210may include full transcripts, vocabularies, statistical language models (e.g., transition probabilities), histograms of word occurrences (“bag of words”), weighted histograms (where words are weighted according to their contextual salience, using e.g., a Term Frequency-Inverse Document Frequency (TF-IDF) scheme), n-best results, or any other data available from the component's lattice, such as phoneme time-stamps, etc. The ASR component210may also be used to extract meta-linguistic features such as laughter, hesitation, gasping, background noise, etc. The ASR features can be extracted separately for the representative and the customer, and may be recorded separately for multiple speakers on each side of the conversation.

The NLP component225processes the text to produce various semantic features, e.g., identification of topics, identification of open-ended questions, identification of objections and their correlation with specific questions, named entity recognition (NER), identification of relations between entities, identification of competitors and/or products, identification of key phrases and keywords (either predetermined, or identified using salience heuristics such as TF-IDF), etc. Additional features that may be extracted by the NLP component225can be summarizations of utterances and various natural language entailments. The NLP features can be extracted separately for the representative and the customer, and may be recorded separately for multiple speakers on each side of the conversation.

The affect component215can extract low-level features and high-level features. The low-level features can refer to the voice signal itself and can include features such as speech rate, speech volume, tone, timber, range of pitch, as well as any statistical data over such features (e.g., maximal speech rate, mean volume, duration of speech over given pitch, standard deviation of pitch range, etc.). The high-level features can refer to learned abstractions and can include identified emotions (e.g., fear, anger, happiness, timidity, fatigue, etc.) as well as perceived personality traits (e.g., trustworthiness, engagement, likeability, dominance, charisma, confidence, etc.) and perceived or absolute personal attributes such as age, accent, and gender. Emotion identification, personality trait identification, and personal attributes, may be trained independently to produce models incorporated by the affect component, or trained using the human judgment tags optionally provided to the offline analysis component. In some embodiments, the affect component215can also extract features, such as a speaker engagement metric (“wow” metric), which measures how engaged a participant was in the conversation, e.g., based on the usage of vocabulary, rate of speech, pitch change. For example, the usage of phrase “Oh! cool” can indicate a higher degree of engagement than the phrase “cool!”. In another example, the same phrase but said in different pitches or pitch ranges can indicate different degrees of engagement. All features extracted by the affect component215may or may not include a corresponding confidence level, which can be used in modeling outcomes. The affect features can be extracted separately for the representative and the customer, and may be recorded separately for multiple speakers on each side of the conversation.

The metadata component220can measure conversation flow, including speaker diarisation (e.g., which speaker spoke when and for how long), silence times and duration, as well as overlap of two or more speakers in addition to other metadata such as time of day call was placed, geographical destination of call and known gender and age of participants. The data extracted with the metadata component220may be collected separately for multiple speakers on each side of the conversation, or pooled together for representative and customer sides, respectively.

All components may extract features for a group of representatives, a single representative and/or a customer, including multiple parties on either side, and may be customized to optimize feature extraction accordingly. In addition, the features115may be extracted on the representative's recording alone, on the customer's recording alone, or on both. The features115may also include comparisons between extracted attributes. For example, the affect component215may extract as a feature a mean difference between representative and customer's speech rates, or a maximum difference between representative and customer's speech pitches. Likewise, the ASR component210may extract transcriptions and keywords both as a combined transcript and as two separate transcripts, and may be tuned with an acoustic or language model specific to a group of representatives or an individual representative. Similarly, the NLP component225may extract features such as open-ended questions with or without the corresponding response.

In some embodiments, the feature generation component111can also generate a set of features that indicate a blueprint of a conversation. The blueprint can represent a skeleton of the conversation and indicate a presence or absence of a particular aspect in the conversation. For example, the blueprint can include various features that indicate whether the conversation included any agenda setting, rapport building, clarification questions, defining goals, setting expectations, mentioning of examples. The blueprint can also help in predictive analysis of the outcome of the calls, e.g., by the classifier component112. One or more components of the feature generation component111can use AL and/or ML techniques to extract one or more of the features115.

FIG. 3is a block diagram of the classifier component for generating classifiers, consistent with various embodiments. The example300illustrates the classifier component112using the features115extracted from the feature generation component111to generate a number of classifiers, “C1”-“CN”. In some embodiments, the classifier component112analyzes the features of a dedicated portion of the collected recordings, e.g., a training set, which is a subset of the entire recordings available for analysis, to generate the classifiers120. Each of the classifiers120can have a value, e.g., an F-score, that indicates a prediction power of the classifier for the specified outcome. The higher the prediction power, the higher the probability of achieving the specified outcome of the classifier based on the included features. In some embodiments, the prediction power is determined by running the classifiers120on, e.g., a portion of call recordings that is not yet analyzed, e.g., a test set, and computing the respective F-score.

The classifiers120may be further analyzed to determine what features carry the largest prediction power, e.g., speech rate early in the conversation, occurrence of first interrupt by customer, names of competitors mentioned, or number of open questions thoughtfully answered, and a subset of these classifiers that have features with the largest prediction power can be used to generate the on-call guidance.

The conversation outcome depicted by the classifiers120can be any configurable outcome, e.g., “sales closed”, “sales failed”, “demo scheduled”, “follow up requested,” NPS-like probability of recommending to a friend, etc. In some embodiments, the features115extracted from the feature generation component111can be fed into a machine learning algorithm (e.g., a linear classifier, such as a SVM, or a non-linear algorithm, such as a DNN or one of its variants) to produce the classifiers120. The classifiers may be further analyzed to determine what features carry the largest prediction powers (e.g., similarity of speech rate, occurrence of first interrupt by customer, extrovert/introvert matching, or gender or age agreement.)

The classifier component112can generate multiple classifiers for the same outcome. However, for a given outcome, different classifiers have different features. For example, the classifier component112can generate a first classifier305, “C1,” and a second classifier310, “C2,” for a specified outcome, “O1.” However, the first classifier “C1” has a first set of features, e.g., features “f1”-“f3,” and the second classifier “C2” has a second set of features, e.g., features “f5”-“f8.” The features in different classifiers can have different prediction powers and contribute to the specified outcome in different degrees.

Different classifiers may be built for a different number of participants, and may consider multiple participants as a single interlocutor, or as distinct entities. Further, as described above, the classifier component112can generate different classifiers for different time intervals of a conversation. The classifier component112analyzes the features115extracted from the feature generation component111at various time intervals, e.g., seconds 00:05-00:10, seconds 00:20-00:30, minutes 1:00-2:00, covering the entire conversation duration, and generates one or more classifiers for each of those time intervals. Each classifier can correspond to a specified time interval of the conversation. For example, if “100” conversations are being analyzed, then the classifier component112can analyze first 5-20 seconds each of the “100” conversations and generate one or more classifiers for all the conversations corresponding to the interval of 5-20 seconds. Similarly, it can generate one or more classifiers corresponding to the 10-25 seconds interval. If more than one classifier is generated for a specified time interval, in some embodiments, different classifiers can have different outcomes, and in some embodiments, can have the same outcome; however, different classifiers will have different sets of features that contribute to the corresponding outcome. In the example300, classifiers C1 and C5 correspond to a time window of seconds 00:05-00:20 of the conversations analyzed, and classifier C10 corresponds to minute 1:00-2:00 of the conversations.

The classifier315, “C3,” includes an example set of features extracted from analyzing various sales calls. The classifier315corresponds to the first two minutes of the conversations, and indicates that when laughter from the customer is registered and the representative greets the customer, indulges in rapport building and poses at least two open-ended questions, then there is a high chance, e.g., 83%, of renewal of a magazine subscription. The features and outcome of the classifier315“C3” can be “f1->customer laughter=yes” “f2->greeting customer=yes,” “f3->rapport building=yes,” (“f4->open ended questions asked=yes,” and “f5->number of open ended questions asked=2”), “outcome=renew subscription” “probability of outcome=83%.”

The classifiers120can be used by the real-time analysis component130, e.g., as described at least with reference toFIG. 1above andFIG. 4below, to generate an on-call guidance for representatives or both inbound and outbound calls.FIG. 4is a block diagram of the real-time analysis component ofFIG. 1for generating on-call guidance for a representative during a call between the representative and a customer, consistent with various embodiments. In some embodiments, the real-time analysis component130takes as input a live conversation stream, e.g., real-time call data150, between a representative410and a customer405, uses the feature generation component113to extract call features135, e.g., as described above at least with reference toFIGS. 1 and 3.

The classifier component114feeds the call features135into the classifiers120generated by the offline analysis component110and selects a subset of the classifiers120, e.g., a set of classifiers140, that includes features that match with the call features135extracted from the live conversation stream. In some embodiments, the set of classifiers140chosen by the call-modeling component116are also the classifiers that have high predictability power, e.g., as measured using an F-score and that have an F-score exceeding a specified threshold.

The call-modeling component116then generates an on-call guidance145, which includes information regarding real-time probabilities for specific outcomes to which the set of classifiers140correspond. The on-call guidance145may be used to notify the representative and/or their managers of the predicted outcome of the call. Additionally, the call-modeling component116can further analyze the set of classifiers140to determine classifiers that include features with the largest prediction powers, and present the values of those features in the on-call guidance145for suggesting the representative and/or their managers to modify or persist with an on-call behavior consistent with those features. For example, if one of the set of classifiers140predicts that conversations with rapport building and several open-ended questions being posed at the first few minutes of the conversation lead to favorable outcomes, the call-modeling component116may notify the representative and/or their managers as part of the on-call guidance145to engage in rapport building and pose questions at early stages of the conversation. Similarly, if one of the classifiers from the set of classifiers140indicates that matching speech rate to within 10% of customer's rate at a specified relative position of the call, e.g., during third minute of the call, leads to improved closing results, the call-modeling component116may notify the representative and/or their managers as part of the on-call guidance145to adjust their speech rate accordingly. On the other hand, if one of the classifiers from the set of classifiers140indicates that conversations beginning with over a specified number of objections, naming a specific competitor and mention of the phrase “read online” do not lead to good results, the call-modeling component116may notify the representative and/or their managers as part of the on-call guidance145to expedite wrap-up of conversations to avoid losing time on a call that is not likely to yield desired results.

In addition to live on-call guidance, the real-time analysis component130may be used to provide the representative and/or their managers with non-real time analysis as well, which provides insight into details of the conversations, e.g., what occurred in the conversations, when events occurred, and various such quantifiable analytics of the calls. For example, the classifiers can be used to find interesting calls that would interest the representatives to listen and learn from. The disclosed embodiments can be used to improve outcomes of the call not only during a real-time or a live call, but could also be used to inform representatives and/or managers for better training and coaching in retrospect.

The real-time analysis component130may also be used to auto-populate information fields in a customer relationship management (CRM) system or a similar system.

FIG. 5is a flow diagram of a process500for performing offline analysis of conversations between participants, consistent with various embodiments. In some embodiments, the process500can be implemented in the call-modeling system100ofFIG. 1. At block505, the offline analysis component110retrieves historical call data, e.g., call data105, regarding various conversations between participants, such as a customer and a representative. In some embodiments, the call data105can be audio recordings of calls between the participants, transcripts of audio recordings, chat transcripts, etc. The offline analysis component110can retrieve the call data105from the storage system125. Further, in some embodiments, the call data105can include data regarding only a subset of the conversations stored in the storage system125.

At block510, the feature generation component111analyzes the call data105to extract various features of the conversation, e.g., as described at least with reference toFIGS. 1 and 2. Some example features include transcripts of audio recordings, vocabulary, semantic information of conversations, summarizations of utterances and various natural language entailments, voice signal associated features (e.g., speech rate, speech volume, tone, and timber), emotions (e.g., fear, anger, happiness, timidity, fatigue), inter-speaker features (e.g., similarity of speech rate between speakers, occurrence of first interrupt by customer, extrovert/introvert matching, or gender or age agreement), personality traits (e.g., trustworthiness, engagement, likeability, dominance, charisma, confidence, etc.) and personal attributes (e.g., age, accent, and gender). The feature generation component111can also analyze the call data105to generate various tags as described above.

At block515, the classifier component112analyzes the features to generate classifiers, e.g., as described at least with reference toFIGS. 1 and 3. The classifier component112analyzes the features115using various techniques, e.g., machine learning algorithms such as SVM, DNN, to generate the classifiers120. The classifiers120indicate conversation outcomes, e.g., “sales closed”, “sales failed,” “probability of recommending to a friend,” NPS, or customer satisfaction. Each of the classifiers indicates a specific outcome and can include a set of the features that contributed to the specific outcome. For example, in a sales call for renewing a magazine subscription, a classifier “C1” can indicate that when laughter by a customer and two open-ended questions from the representative are registered, there is a high chance, e.g., 83%, of renewal. The classifier component112can generate multiple classifiers for the same outcome; however, they have distinct sets of features. Further, the classifier component112generates different classifiers for different time windows of the conversations. For example, the classifier component112generates a classifier “C1” for first two minutes of the conversations and a classifier “C2” for a third minute of the conversations. The offline analysis component110can store the features115and the classifiers120in a storage system125.

FIG. 6is a flow diagram of a process600for modeling calls between participants to generate on-call guidance, consistent with various embodiments. In some embodiments, the process600can be implemented in the call-modeling system100ofFIG. 1. At block605, the real-time analysis component130receives real-time call data150of an ongoing conversation, e.g., an audio stream of a voice call between a customer and a representative. At block610, the feature generation component113analyzes the real-time call data150to extract features, e.g., call features135, of the ongoing conversation, e.g., as described at least with reference toFIGS. 1 and 2. The feature generation component113can also analyze the real-time call data150to generate various tags as described above.

At block615, the classifier component114inputs the extracted features to classifiers in the storage system, e.g., classifiers120which are generated as described at least with reference to process500ofFIG. 5, to determine one or more classifiers that predict possible outcomes of the call based on the extracted features. For example, as described at least with reference toFIGS. 1 and 4, the classifier component114feeds the extracted features135into the classifiers120generated by the offline analysis component110, and selects a subset of the classifiers120, e.g., a set of classifiers140, that includes features that match with the call features135extracted from the live conversation stream. In some embodiments, the set of classifiers140include classifiers whose prediction power exceeds a specified threshold. The set of classifiers140corresponds to specific outcomes and include real-time probabilities for the specific outcomes.

At block620, the call-modeling component116generates on-call guidance, e.g., on-call guidance145, that presents the real-time probabilities of possible outcomes of the call as indicated by the set of classifiers140. The call-modeling component116can further analyze the set of classifiers140to determine features that have high prediction power, e.g., prediction power exceeding a specified threshold, for predicting a desired outcome, and then include those features and values associated with those features in the on-call guidance145. The on-call guidance145notifies the representative to adopt or persist with an on-call behavior consistent with those features to achieve the desired outcome, or at least to increase the probability of achieving the desired outcome. For example, the on-call guidance145can present instructions on a display screen of a user device associated with the representative recommending the representative to change the rate of speech, use specific key words, or pose more open-ended questions to the customer in order to increase the probability of achieving the desired outcome.

Example Usage of the Embodiments

The following is an example usage of the disclosed embodiments for modeling sales calls for renewal of a subscription for a magazine. At a first stage, e.g., before a call is received from a live customer or before a call is placed by a representative, a number of recordings of previous calls is processed by the offline analysis component110, e.g., using an ASR component210that is customized for the field of surgery institutions, an NLP component225, an affect component215and a metadata component220to generate various features. The classifier component112generates two classifiers, based on those features, that can be found to be highly predictive: (a) a first classifier based on the first two minutes of one or more of the analyzed conversations, which indicates that when a laughter by the customer is registered, the representative engages in rapport building, and at least two open-ended questions are posed by the representative, then there is a high chance, e.g., 83%, of subscription renewal; (b) a second classifier based on the third minute of one or more of the analyzed conversations, which indicates that when a competitor magazine or the key-phrase “read online” is used, and/or the speech rate of the customer is more than three words per second, the renewal chances drop to 10%.

The above two classifiers can then be used by the real-time analysis component130in a second stage, e.g., during a live call between the representative and the customer, for generating an on-call guidance to guide the sales representatives as follows. Based on the first classifier, the real-time analysis component130can indicate to the sales representative to ask questions within the first 2 minutes. Based on the second classifier, the real-time analysis component130can, at minute three of the conversation, urge the representative to reduce speech rate to get the customer to mirror their own speech rate if a competitor is mentioned or otherwise the phrase “read online” is used. If the speech rate is not reduced, the real time analysis component130can indicate to the representative and/or their managers to wrap up the call as soon as possible.

The embodiments disclosed above may be implemented as separate modules, e.g., as presented above, as a single module, or any combination thereof. Implementation details may vary, including core machine learning algorithms employed. The embodiments may be implemented using any software development environment or computer language. The embodiments may be provided as a packaged software product, a web-service, an API or any other means of software service. The embodiments may use expert taggers, crowdsourcing or a hybrid approach for tagging.

FIG. 7is a block diagram of a moment identification system, consistent with various embodiments. The moment identification system700can determine conversations having a specified moment710, and generate a playlist725of the conversations having the specified moment710. The moment identification system700includes a feature generation component, such as feature generation component111ofFIG. 1, that generates features115of the conversations by analyzing the recordings of the conversations, which are stored as call data105in the storage system125. The moment identification system700includes a moment determination component705that determines the conversations having the specified moment710and generates conversation moment metadata715having information regarding the specified moment710in a particular conversation. The moment identification system700includes a playlist generation component720that generates the playlist725based on the conversation moment metadata715.

The call data105can include recordings of a number of conversations between multiple participants. For example, the call data105can include (a) a first recording730, which is a recording of a telephone conversation between a first sales representative of an organization and a first customer of the organization, and (b) a second recording731, which is a recording of a telephone conversation between a second sales representative of the organization and a second customer of the organization. Note that the recordings can be of a conversation that is any of telephone based, VoIP based, video conference based, VR based, AR based, e-mail based, or in-person interaction based.

The feature generation component111can analyze the call data105to extract the features of the conversations. For example, the feature generation component111analyzes the first recording730to extract a first set of features735of the conversation in the first recording730, and analyzes the second recording731to extract a second set of features736of the conversation in the second recording731. In some embodiments, the first set of features735and the second set of features736are a subset of the features115. The features735can include transcripts of the conversations, vocabulary, semantic information of conversations, summarization of a call, summarizations of utterances and various natural language entailments, voice signal associated features (e.g., speech rate, speech volume, tone, and timber), detected emotions (e.g., fear, anger, happiness, timidity, fatigue, laughter), detected personality traits (e.g., trustworthiness, engagement, likeability, dominance, charisma, confidence, etc.), personal attributes (e.g., age, accent, and gender), and inter-speaker attributes that indicate a comparison between both the participants (e.g., similarity of speech rate between the representative and the customer, extrovert/introvert matching, or gender or age agreement). Further, the features735can include not only aural features, but also non-aural features, e.g., visual features such as body language of a participant, and facial expressions of the participant, or any combination of aural and non-aural features. The features735can also include information that indicates with which participant a particular feature is associated with. For example, if laughter is identified as one of the features735, then the corresponding feature can also include information which indicates whether the laughter is from a first participant (e.g., a sales representative) or a second participant (e.g., a customer) in the conversation. Further, the features could also be generated from the transcripts of any of emails, online messages, and online meetings. In some embodiments, the feature generation component111can determine that any of a word, a phrase, a text, emoji, symbols, or a combination thereof can convey a particular feature. For example, the feature generation component111can determine that a text such as “Ha Ha” or “rofl” in the transcript can indicate laughter.

The moment determination component705can analyze the features of the conversations to determine whether a particular conversation has the specified moment710. For example, the moment determination component705can analyze the features735of the conversation in the first recording730to determine whether the conversation has the specified moment710. The specified moment710can be any of a number of moments, such as “laughter,” “laughter by a customer,” “laughter by a sales representative,” “participant picking his nose,” “participant walking out of the meeting,” “participant having a specific facial expression,” “discussion on budget,” “discussion on pricing,” “discussion on renewal of subscription,” “prospect engaged by a sales representative's question”, etc. The moment determination component705can determine whether any of the features735of the conversation correspond to the specified moment710. In some embodiments, the moment determination component705can determine that the conversation has the specified moment710if one or more of the features735correspond to the specified moment710directly, or if the specified moment710can be derived from one or more of the features735. For example, if the specified moment710is “angry,” and one of the features735indicates that a participant is angry, then the moment determination component705can determine that the conversation has the specified moment710based on direct correspondence of the features with the specified moment710. However, if one or more of the features735indicate that a pace of the speech of a participant is “fast talking” and a tone of the speech as “high pitch,” or a facial expression indicates anger, or if the words used in a chat indicates anger, the moment determination component705can derive that the participant is “angry” based on the above features and thus, determine that the conversation has the specified moment710.

After identifying the features that correspond to the specified moment710, the moment determination component705generates the conversation moment metadata715, which includes information regarding the specified moment710in the conversation of the first recording730. The conversation moment metadata715can include a variety of information, such as an identification of the specified moment710, the features that correspond to the specified moment710, and a location at which each of the identified features is present in the first recording730. A moment can occur one or more times in a conversation. For example, if the specified moment710is “laughter,” and it occurs more than once in a conversation, and the moment determination component705can identify such multiple occurrences of “laughter” in the conversation. The conversation moment metadata715can include information regarding each of the multiple occurrences of the specified moment710. For example, if feature “f1” corresponds to a first occurrence of “laughter” at minute 5 of the first recording730, feature “f2” to corresponds to a second occurrence of “laughter” at minute 6 of the first recording730, and features “f3” and “f4” both combined correspond to a third occurrence of “laughter” at minute 7 of the first recording730, the conversation moment metadata715includes information regarding features “f1”-“f4” and their respective locations in the first recording730.

The moment determination component705can similarly analyze the features of conversations in other recordings, identify those of the conversations having one or more features that correspond to the specified moment710, and update or generate the conversation moment metadata715. The conversation moment metadata715can be generated as a single data object for each of the conversations having the specified moment710, or as one single data object having moment information for all the identified conversations.

The playlist generation component720can generate the playlist725by including conversations identified by the moment determination component705as having the specified moment710. The playlist725can either include the entire conversation which has the specified moment710or a snippet or portion of the conversation that has the specified moment710. In some embodiments, the playlist725includes a portion of the conversation that has the specified moment710. For example, if the specified moment710occurred at 1 minute 10 seconds of the first recording730, the playlist725can include a portion of the first recording730corresponding to 1 minute 10 seconds, such as from 40 seconds to 1 minute and 40 seconds of the first recording730. The playlist generation component720can include a portion of the conversation that happens immediately prior to and/or after the occurrence of the specified moment710in order to provide a context of the conversation around the specified moment710. In some embodiments, the duration of the conversation prior to and/or after the occurrence of the specified moment710to be included in the playlist725can be configured by a consumer user of the playlist725, such as a sales representative, a trainer, or a manager of the sales representative.

The playlist generation component720can generate playlists for various moments and store each of the playlists in the storage system125. A consumer user can input various criteria based on which the playlists are to be generated for the specified moment710. For example, the playlist generation component720can generate playlists that are specific for an entity. That is, the playlist generation component720can generate a playlist that includes only those of the conversations that have the specified moment710and are associated with a specific entity, e.g., an organization, a representative, a customer, a customer-representative pair, etc. For example, the playlist generation component720can generate a playlist of the specified moment710for conversations of a specific sales representative, a specific customer, a specific sales representative-customer pair, a specific group of users, or a specific organization. The playlist generation component720can also generate context-aware playlists, e.g., based on time or location, at which the conversations took place. For example, the playlist generation component720can generate a playlist of the specified moment710for conversations that happened during a night shift, e.g., from 9 pm to 6 am.

The playlist generation component720can facilitate tagging of playlists, e.g., for discovery, navigation and easy access. As described above, a tag can be used to specify a variety of information, such as, a specific moment, a specific entity, and/or a specific context for which the playlist is generated.

The specified moment710can be a user-specified moment or system-learned moment. For example, some moments can be referred to as “interesting” moments. A consumer user of the playlist725can define what moments are interesting, e.g., “laughter,” discussions on topics such as “budget” or “pricing,” and input one or more of them as the specified moment710. In some embodiments, the moment determination component705can automatically determine what moments may be interesting to the consumer users and use one or more of those moments to generate the playlist725. The moment determination component705can automatically determine certain moments as interesting moments in various ways. In some embodiments, the moment determination component705can identify the interesting moments based on a usage pattern of a recording of a conversation. For example, the moment determination component705can identify a particular portion of the conversation as associated with an interesting moment if the portion has been tagged, e.g., with comments indicating a topic discussed in the portion, if the particular portion is played or listened to more than a threshold number of times or more than the other portions of the conversation.

In some embodiments, the moment determination component705can identify the interesting moments based on an anomaly in a conversation. For example, the moment determination component705can identify a portion of the conversation in which a participant starts to speak at a pace greater than a specified threshold or compared to the pace in other portions of the conversation, as being associated with an interesting moment. In another example, the moment determination component705can identify a portion of the conversation in which a participant exits a meeting room while the meeting is still in progress as being associated with an interesting moment. In yet another example, the moment determination component705can identify a portion of the conversation in which a participant has a particular facial expression as being associated with an interesting moment. In still another example, the moment determination component705can identify a portion of the conversation which is played in a speed or at a volume level different from the other portions as being associated with an interesting moment. In still another example, the moment determination component705can identify a portion of the conversation as being associated with an interesting moment based on a number of clicks associated with the portion of the conversation. A consumer user can click on a portion of the recording to perform a number of actions, such as add a tag, add a comment, or indicate a player to play the recording from the clicked portion. The moment determination component705can identify various such moments as interesting moments, and the playlist generation component720can generate a playlist for the identified interesting moments. In some embodiments, the moment determination component705uses at least one of AI and ML techniques to automatically identify “interesting” moments. Usage of at least one of AI or ML by the moment determination component705may use a set of examples given for training to model and identify either a previously determined moment (e.g., “laughter”) or a novel, previously unspecified, type of moment (e.g., “sales reps touching his hair while discussing budget”).

FIG. 8is an example screenshot of a player tool to play a recording of the conversation, consistent with various embodiments. The player tool800can be used to, among others, play a recording of a conversation805. In some embodiments, the recording805is similar to the first recording730ofFIG. 7. A consumer user can click on the recording805to perform an action. For example, the consumer user can click on a first portion810of the recording805to add a tag815, e.g., to indicate a topic discussed in the first portion810of the conversation. As described above, a tag can indicate a variety of information, e.g., a trait of a participant, a call outcome, a moment of a conversation, or a summary of the conversation. The consumer user can click on the first portion810multiple times to perform multiple different actions. Similarly, the consumer user can click on any other portion of the recording805to perform a corresponding action. The number of clicks associated with a portion can be used in determining whether a portion of the conversation may have an interesting moment.

FIG. 9is a block diagram an example900for identifying conversations having a specified moment, consistent with various embodiments. In some embodiments, the example900can be implemented using the moment identification system700ofFIG. 7. The example900includes multiple recordings, such as a first recording905, a second recording910, a third recording915and a fourth recording920. The recordings can be part of the call data105, and can be similar to that of the first recording730.

The feature generation component110can extract features of the recordings to generate a first set of features906, f11-f13, of the first recording905, a second set of features911, f21-f23, of the second recording910, a third set of features916, f31-f35, of the third recording915, and a fourth set of features921, f41-f43, of the fourth recording920.

The example moment925for which a playlist of the conversations is to be generated can be similar to the specified moment710. Further, consider that the example moment925is “laughter.”

As described above, the moment determination component705can determine whether a conversation has a moment by determining whether any of the features of the conversation correspond to the moment. For example, the moment determination component705determines that the feature f11of a conversation of the first recording905, and features f33and f35of a conversation of the third recording915correspond to the example moment925of “laughter.”

The moment determination component705generates conversation moment metadata930for the first recording905and conversation moment metadata935for the third recording915. In some embodiments, the conversation moment metadata930and935are similar to the conversation moment metadata715. The conversation moment metadata930includes information such as the example moment925for which the conversation moment metadata930is generated, the feature f11of the first recording905that corresponds to the example moment925and the location of the feature f11in the first recording905, e.g., 1 minute and 52 seconds of the first recording905. Similarly, the conversation moment metadata935includes information such as the example moment925for which the conversation moment metadata935is generated, the features f33and f35of the third recording915that correspond to the example moment925and the location of the features in the third recording915, e.g., the features f33and f35occur at 2 minute and 15 seconds and 4 minute and 10 seconds of the third recording915, respectively.

FIG. 10is a block diagram of an example for generating a playlist of a moment, consistent with various embodiments. In some embodiments, the example900can be implemented using the moment identification system700ofFIG. 7. The playlist generation component720can generate the playlist1005for the example moment925based on the conversation moment metadata of various conversations, e.g., conversation moment metadata930and935. In some embodiments, the playlist1005is similar to the playlist725. The playlist generation component720analyzes the conversation moment metadata to identify the conversations having the example moment925and includes the identified conversations in the playlist1005. For example, the playlist generation component720determines, based on the conversation moment metadata930and935, that the first recording905and the third recording915include the example moment925. Accordingly, the playlist generation component720includes two conversations in the playlist1005—the first recording905and the third recording915.

In some embodiments, the playlist generation component720includes only a portion of the recording that includes the example moment925. For example, based on the conversation moment metadata930, the playlist generation component720determines that a first occurrence of the example moment925(m11) in the first recording905is at 1 minute and 52 seconds. Similarly, based on the conversation moment metadata935, the playlist generation component720determines that a first occurrence of the example moment925(m31) in the third recording915is at 2 minute and 15 seconds and a second occurrence (m32) is at 4 minute and 10 seconds. Accordingly, for the first recording905, the playlist generation component720includes a first portion1010of the first recording905corresponding to the 1 minute and 52 seconds, and for the third recording915, a second portion1015and a third portion1020corresponding to 2 minute and 15 seconds and 4 minute and 10 seconds, respectively.

As described above at least with reference toFIG. 7, the playlist generation component720when including a portion of the conversation in the playlist, it can include a portion of the conversation that happens immediately prior to and/or after the occurrence of the example moment925in order to provide a context of the conversation around the example moment925. For example, if the example moment925occurred in the first recording730at 1 minute 52 seconds, the playlist generation component720can include 30 seconds of recording prior to and/or after the 1 minute 52 seconds mark, such as from 1 minute and 22 seconds to 2 minute 22 seconds of the first recording905. An amount or duration of the conversation prior to and/or after a moment to be included in the playlist can be user-configurable.

The playlist1005can include an actual recording of the portion of the conversation or a link to a storage location from where the recording of the conversation is accessible. The link can also include a “start from” location and a “play until” location of the recording, both of which indicate a portion of the recording to be played by a player, such as a player tool800ofFIG. 8, for playing a portion of the conversation that has the example moment925. For example, for the first recording905, the “start from” location can be 1 minute and 22 seconds and “play until” location can be 2 minute 22 seconds.

FIG. 11is a flow diagram of a process for generating a playlist of conversations having a specified moment, consistent with various embodiments. In some embodiments, the process1100can be implemented using the moment identification system700ofFIG. 7. As described above, a moment can be a specific event or an occurrence of a specific characteristic in the conversation. Further, the moment can be any event or occurrence that is of specific interest for a specific application. For example, if the specific application for which the moment identification is implemented is for coaching sales representatives to improve the outcome of the sales call, the moment can be any of occurrence of laughter in a conversation, questions asked by a customer, responses provided by the sales representative, a discussion of topics like budget, pricing, or any other idiosyncratic behavior such as a salesperson touching his face while speaking, etc. or any combination of thereof.

At block1105, the feature generation component111retrieves call data, e.g., call data105, from a storage system, e.g., storage system125. The call data105can include recordings of a number of conversations between multiple participants. In some embodiments, a conversation has at least two participants, e.g., a sales representative of an organization and a customer of the organization.

At block1110, the feature generation component111extracts the features of the conversations, e.g., features115, and stores them in the storage system125. The features may be extracted on demand or automatically, e.g., upon a recording of a conversation being stored in the storage system125.

At block1115, the moment determination component705identifies conversations having features that are characteristic of the specified moment. For example, if the playlist is being generated for a moment such as “laughter,” the moment determination component705identifies conversations having features that are characteristic of “laughter.” The moment determination component705can determine that a conversation has the specified moment if one or more features of the conversation corresponds to the specified moment directly, or if the specified moment can be derived from one or more of the features. In some embodiments, the moment determination component705can use at least one of AI or ML techniques to identify a moment in the conversation. The moment determination component705can generate conversation moment metadata, which can include information regarding conversations that have the specified moment, the features which correspond to the specified moment and a location of the features in the corresponding conversation.

At block1120, the playlist generation component720generates a playlist of the conversations having the specified moment, such as the playlist725or1005. In some embodiments, the playlist generation component720analyzes the conversation moment metadata to identify the conversations and portions of the identified conversations having the specified moment, and includes the portions of the identified conversations in the playlist. As described above at least with reference toFIG. 10, the playlist can include a copy of the recording of the portion of the conversation having the specified moment or a link to the recording using which the recording can be accessed from a storage location of the recording, e.g., by a player tool when the playlist is played.

FIG. 12is a flow diagram of a process for generating a playlist of conversations having a specified moment, consistent with various embodiments. In some embodiments, the process1200can be implemented using the moment identification system700ofFIG. 7. At block1205, the feature generation component121retrieves call data, e.g., call data105, from a storage system, e.g., storage system125. The call data105can include recordings of a number of conversations between multiple participants. In some embodiments, a conversation has at least two participants, e.g., a sales representative of an organization and a customer of the organization.

At block1210, the feature generation component121extracts the features of the conversations, e.g., features125, and stores them in the storage system125. The features may be extracted on demand or automatically, e.g., upon a recording of a conversation being stored in the storage system125.

At block1215, the moment determination component705identifies multiple moments in each of the conversations. In some embodiments, the moment determination component705identifies a moment in the conversation as described at least with reference toFIGS. 7 and 11. The moment determination component705can generate conversation moment metadata for each of the identified moments. The conversation moment metadata can include information regarding a specified moment for which it is generated, the conversations that have the specified moment, the features which correspond to the specified moment and a location of the features in the corresponding conversation.

At block1220, the playlist generation component720generates a playlist of the conversations for each of the multiple moments. For example, the playlist generation component720generates a first playlist having those of the conversations having a first moment, e.g., laughter, and a second playlist having those of the conversations having a second moment, e.g., a participant having a specific facial expression. In some embodiments, the playlist generation component720generates a playlist of the conversations having a specified moment as described at least with respect toFIGS. 7, 10 and 11.

FIG. 13is a block diagram of a computer system as may be used to implement features of the disclosed embodiments. The computing system1300may be used to implement any of the entities, components or services depicted in the examples of the foregoing figures (and any other components described in this specification). The computing system1300may include one or more central processing units (“processors”)1305, memory1310, input/output devices1325(e.g., keyboard and pointing devices, display devices), storage devices1320(e.g., disk drives), and network adapters1330(e.g., network interfaces) that are connected to an interconnect1315. The interconnect1315is illustrated as an abstraction that represents any one or more separate physical buses, point to point connections, or both connected by appropriate bridges, adapters, or controllers. The interconnect1315, therefore, may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus, also called “Firewire”.

The instructions stored in memory1310can be implemented as software and/or firmware to program the processor(s)1305to carry out actions described above. In some embodiments, such software or firmware may be initially provided to the processing system1300by downloading it from a remote system through the computing system1300(e.g., via network adapter1330).

Remarks

Those skilled in the art will appreciate that the logic illustrated in each of the flow diagrams discussed above, may be altered in various ways. For example, the order of the logic may be rearranged, substeps may be performed in parallel, illustrated logic may be omitted; other logic may be included, etc.