Signal discovery using artificial intelligence models

Systems and methods for improving call topic models are described herein. In an embodiment a server computer receives call transcript data comprising an electronic digital representation of a verbal transcription of a call between a first person of a first person type and a second person of a second person type. The server computer splits the call transcript data into first person type data comprising words spoken by the first person in the call and second person type data comprising words spoken by the second person type in the call. The server computer uses a stored topic model to determine a topic of the call, the topic model simultaneously modeling the first person type data as a function of a first probability distribution of words used by the first person type for one or more topics and the second person type data as a function of a second probability distribution of words used by the second person type for the one or more topics, both the first probability distribution of words and the second probability distribution of words being modeled as a function of a third probability distribution of words for the one or more topics. The server computer then stores a data record identifying the topic of the call and/or stores data identifying the topic of the call with the call transcripts.

TECHNICAL FIELD

One technical field of the disclosure is computer-implemented artificial intelligence models that are programmed to derive semantics such as topics from a natural language dataset such as a transcript of a voice call communicated between a calling person and a called entity. Another technical field of the disclosure is improvements to Bayesian Belief Network models and model generation techniques.

BACKGROUND

Topic modeling in written and verbal communications can be extremely useful for grouping a large number of communications for review, analysis, or intervention. While there are many techniques for topic modeling of written communications, a lot of those techniques are insufficient for verbal communications which include a greater degree of variation between people discussing the same topic.

Neural networks and other structured machine learning algorithms are difficult to use for topic modeling as they require annotated training data and do not provide results that are easy to interpret. Thus, unsupervised learning models, such as the latent Dirichlet allocation model (LDA), are often used for topic modeling. The LDA model models topics in a plurality of calls based on the words spoken in the call. While the LDA model is often useful for written communications, the LDA model tends to underperform on oral communications for a few reasons.

A first problem with the LDA model is that it treats all words on the call as equal when performing topic modeling. This assumption causes words spoken by one party to be treated the same as words spoken by the other party. Given differences in speech patterns and word usage between a caller and an agent, the intermingling of language can lead to calls being misidentified due to the language used by the agent or caller skewing the model. Additionally, different calls may have a different proportion of the call where the agent speaks versus where the caller speaks, further causing the model to categorize calls incorrectly based on how much language of one party is used.

A second problem with the LDA model is that it assumes that people will generally use the same language to discuss the same topic. While this may be true with written conversations, speech patterns and word usage in verbal conversations often vary based on locality and person. Thus, the LDA model tends to generate different topics for different regions or different people based on differing language usage despite the same topic being discussed. For instance, the LDA model may generate a first topic that corresponds to purchasing a car in the Northwest United States and a second topic that corresponds to purchasing a car in Southern California.

A third problem with the LDA model is that it uses a flat prior distribution as the basis for both the distribution of topics and the distribution of words. A flat prior distribution gives equal probability for each topic to be pulled from the distribution. By using a flat prior distribution, the LDA model assumes an equal likelihood of any of a plurality of topics being discussed. This assumption ignores the realities of most businesses that may use a topic model: that some topics are discussed more often than others. For example, a client of a car dealership may only buy a car once every 5 years, but the client may bring the car in for repairs once a year. Thus, repair appointment conversations will be much more common than car purchasing appointments. Other topics may be rare but highly distinctive, such as calls associated with fraud. While these calls are important and need to be factored into a topic model, they may be much rarer than scheduling calls.

Thus, there is a need for improved artificial intelligence models for modeling topics from phone conversations.

DETAILED DESCRIPTION

1.0 GENERAL OVERVIEW

2.0 STRUCTURAL OVERVIEW

3.0 FUNCTIONAL OVERVIEW

4.0 TOPIC MODELS4.1 LATENT DIRICHLET ALLOCATION TOPIC MODEL4.2 IMPROVED TOPIC MODEL FOR CALLS4.3 IMPROVED TOPIC MODEL WITH PARTY SEGREGATION4.4 TOPIC MODEL GENERATOR

5.0 TOPIC DISPLAY

1.0 General Overview

Improvements to topic modeling are described herein for use in computer-implemented artificial intelligence models that are programmed to derive semantics such as topics from a natural language dataset such as a transcript of a voice call communicated between a calling person and a called entity. The disclosure also addresses improvements to Bayesian Belief Network models and model generation techniques. In an embodiment, a server computer stores a topic which improves on previous topic models by simultaneously modeling words spoken by a first party type as a function of words used by the first person type for various topics and modeling words spoken by a second party type as a function of words used by the second person type for the various topics. Both of the aforementioned models are then modeled, either directly or indirectly, on a probability distribution of words for the various topics. When call transcripts are received, the model is used to determine one or more topics for each of the calls.

In an embodiment, the model is further improved by modeling the probability distribution of words as a function of an inferred prior distribution which is modeled as a function of a flat prior distribution. A similar improvement may be performed on the topic side of the model where the probability distribution of topics is modeled as a function of an inferred prior distribution which is modeled as a function of a flat prior distribution.

In an embodiment, the model is further improved by modeling words as a function of call-specific topics which are modeled as a function of the probability distribution of words for various topics, thereby reducing the number of topics generated based on regional or personal variants in conversation.

In an embodiment, a method comprises receiving call transcript data comprising an electronic digital representation of a verbal transcription of a call between a first person of a first person type and a second person of a second person type; splitting the call transcript data into first person type data comprising words spoken by the first person in the call and second person type data comprising words spoken by the second person type in the call; storing a topic model, the topic model simultaneously modeling the first person type data as a function of a first probability distribution of words used by the first person type for one or more topics and the second person type data as a function of a second probability distribution of words used by the second person type for the one or more topics, both the first probability distribution of words and the second probability distribution of words being modeled as a function of a third probability distribution of words for the one or more topics; using the topic model, determining a topic of the call; storing the call transcript data with additional data indicating the topic of the call.

2.0 Structural Overview

FIG. 1depicts an example system for performing topic modeling on call transcripts.FIG. 1, and the other drawing figures and all of the description and claims in this disclosure, are intended to present, disclose and claim a wholly technical system with wholly technical elements that implement technical methods. In the disclosure, specially programmed computers, using a special-purpose distributed computer system design, execute functions that have not been available before in a new manner using instructions ordered in a new way, to provide a practical application of computing technology to the technical problem of automated, programmatic determination of topics in digitally stored natural language texts or transcripts. Every step or operation that is functionally described in the disclosure is intended for implementation using programmed instructions that are executed by computer. In this manner, the disclosure presents a technical solution to a technical problem, and any interpretation of the disclosure or claims to cover any judicial exception to patent eligibility, such as an abstract idea, mental process, method of organizing human activity or mathematical algorithm, has no support in this disclosure and is erroneous.

In an embodiment, a server computer110is communicatively coupled to client computing device120over network100. Network100broadly represents any combination of one or more data communication networks including local area networks, wide area networks, internetworks, or internets, using any of wireline or wireless links, including terrestrial or satellite links. The network(s) may be implemented by any medium or mechanism that provides for the exchange of data between the various elements ofFIG. 1. The various elements ofFIG. 1may also have direct (wired or wireless) communications links. The server computer110, client computing device120, and other elements of the system may each comprise an interface compatible with the network100and are programmed or configured to use standardized protocols for communication across the networks such as TCP/IP, Bluetooth, and higher-layer protocols such as HTTP, TLS, and the like.

The client computing device120is a computer that includes hardware capable of communicatively coupling the device to one or more server computers, such as server computer110, over one or more service providers. For example, client computing device120may include a network card that communicates with server computer110through a home or office wireless router (not illustrated inFIG. 1) coupled to an internet service provider. The client computing device120may be a smart phone, personal computer, tabled computing device, PDA, laptop, or any other computing device capable of transmitting and receiving information and performing the functions described herein.

The server computer110may be implemented using a server-class computer or other computer having one or more processor cores, co-processors, or other computers. The server computer110may be a physical server computer and/or virtual server instance stored in a data center, such as through cloud computing.

In an embodiment, server computer110receives call transcripts112over network100from client computing device120. The call transcripts may comprise an electronic digital representation of a verbal transcription of calls between two or more parties. For example, a call transcript for a call dealership may comprise written dialogue between an agent and a customer that has been transcribed from an audio conversation between the agent and the customer. The call transcripts may include data labeling portions of the dialogue with identifiers of the parties and/or party types. For example, when used for conversations between a customer and a goods or services provider, the portions of the dialogue may be labeled based on whether the portions were spoken by the customer or by an agent of the goods or services provider.

In an embodiment, server computer110stores a topic model114. The topic model114comprises computer readable instructions which, when executed by one or more processors, cause the server computer110to compute one or more output topics based on input call transcripts112. The topic model114may comprise a mathematical model that is trained at the server computer110or trained at an external computing device and provided to server computer110.

Call transcripts112are evaluated by the server computer110by using the call transcripts112as input into the topic model114. Using the topic model114, as described further herein, the server computer110identifies one or more topics for the call transcripts. The server computer then stores categorized call transcripts116including the call transcripts with data identifying the one or more topics. In an embodiment, further data is stored relating to the one or more topics. For example, the server computer110may store data identifying a length of a portion of a call corresponding to a particular topic, such as multiple topics are discussed during a single call. In some embodiments, the server computer removes the call transcripts from storage after a topic has been identified. The server computer may instead store the call topics and summary information from the call transcripts.

In an embodiment, the server computer generates topic data118from a plurality of categorized call transcripts116. The topic data118may comprise aggregated information from a plurality of categorized call transcripts116. For example, the topic data may identify each of a plurality of topics, average length of time spent on each topic per call, total amount of time spent on each topic, and/or other aggregated information regarding the call transcripts or modeled topics.

For purposes of illustrating a clear example,FIG. 1shows a limited number of instances of certain functional elements. However, in other embodiments, there may be any number of such elements. For example, embodiments with multiple client computing devices may include a first client computing device or first plurality of client computing devices which sends the call transcripts to the server computer and a second client computing device or second plurality of client computing devices which receives the topic data from the server computer. Further, the server computer110may be implemented using two or more processor cores, clusters, or instances of physical machines or virtual machines, configured in a discreet location or co-located with other elements in a datacenter, share computing facility, or cloud computing facility.

3.0 Functional Overview

FIG. 2depicts an example method of using a topic model to identify topics of audio dialogues based on call transcripts.

At step202, a topic model is stored which models words as a function of topics. For example, a topic model may model specific words spoken on a plurality of calls by identifying a latent set of one or more themes or topics which are shared across all calls. Examples of the topic model are described further herein. The server computer may store a model trained for a particular customer using previously received transcripts. The training of the topic model may be performed at the server computer and/or at an external computing device.

At step204, call transcripts for a call are received. The call transcripts may comprise electronic digital representations of verbal transcriptions of the call. For example, the call transcripts may include transcribed dialogue from a telephonic communication. The transcribed dialogue may uniquely identify the different parties to the conversation. In an embodiment, the different parties are identified as a person type, such as agent and customer. Tags may be placed in the transcriptions of the call which identify, for a block of dialogue, the party or party type which spoke the block of dialogue in the call. The call transcripts may additionally comprise metadata, such as timestamps for one or more blocks of text, total call length, or other call information. Receiving the call transcripts may comprise receiving the call transcripts from an external computing device and/or generating call transcripts from an audio file received from an external computing device and receiving the call transcripts from memory of the server computer.

At step206, the topic model is used to determine a topic of the call. For instance, the server computer may execute instructions to run the trained topic model using the call transcript as input to identify one or more topics discussed in the call. In an embodiment, the call transcript is augmented by the server computer prior to execution of the topic model to transform the call transcript into data which can be read by the topic model. The transformations may include editing the call transcription to change its form so it can be read by the topic model, such as by removing pieces of metadata, changing the file structure of the call transcripts, or splitting the call transcript based on person type, as described further herein.

In an embodiment, determining a topic of the call includes one or more post-processing steps. For example, the topic model may determine, for each word, probabilities of different topics given the word. The server computer may execute one or more post-processing steps to determine, from these probabilities, whether a topic was discussed during a call. The post-processing steps may include aggregating probabilities and/or evaluating one or more criteria. For example, the server computer may determine that a topic was discussed during a call if greater than a threshold number of words spoken during the call had greater than a threshold probability of being spoken given a particular topic. As a practical example, if more than fifteen words were spoken that had over 60% probability of being spoken given a particular topic, the server computer may determine that the particular topic was discussed during the call. The rules may vary based on implementation and may include other thresholds, such as percentage of words spoken in a particular time/word window, or other types of rules, such as rules which use aggregated values and/or maximum percentage values. The rules and thresholds may be configured in advance generally and/or for a specific implementation.

At step208, the call transcripts are stored with data identifying the topic of the call. For example, the server computer may store the call transcripts with metadata identifying one or more topics discussed during the call as identified by the topic model. The server computer may additionally store metadata identifying other attributes of the call, such as length of time spent on each topic. In an embodiment, the server computer separately stores the topic data. For example, the server computer may increment a call topic value by one for each call in which the topic was discussed. Additionally or alternatively, the server computer may store a data record for each call transcript which identifies at least one or more call topics of the call. The data record may additionally identify a date and/or time of the call, a length of the call, a length of time spent discussing each topic, an outcome of the call, or other data relating to the call and/or topic.

At step210, topic summary data is provided to a client computing device. For example, the server computer may cause display of a graphical user interface on the client computing device which displays aggregated topic summary data. Example displays are described further herein. The server computer may additionally or alternatively provide call transcripts with topic identifiers and/or data records for each of a plurality of call transcripts which identify at least one or more call topics of the call.

4.0 Topic Models

Topic modeling may be improved using one or more of the methods described herein. While improvements are sometimes described depicted together, a person of skill in the art would understand that the improvements may be independently applied to the topic model unless the improvements are specified to be dependent on previous improvements. For example, the party segregation improvements described in section 4.2 may be used on the word side of the topic model without the improvements to the topic side of the topic model described in section 4.2.

The topic models described herein comprise mathematical models described at a level that a person of skill in the art would be able to make and use the model without undue experimentation. Where improvements to the underlying mathematics of the models are described, sufficient equations are provided to allow one of skill in the art to make and use a model with the improvements.

Generally, the topics comprise probabilistic models for each of the words spoken on every call. These probabilities are modeled as functions of the topics relevant to the application, the vocabulary associated with each topic, and of how prevalent each topic is. In order to infer the topics from observed data, any standard technique, such as Markov-chain Monte Carlo, variational inference, maximum likelihood estimation, or other inference techniques, may be used to estimate the model parameters.

4.1 Latent Dirichlet Allocation Topic Model

FIG. 3depicts an example of a latent Dirichlet allocation (LDA) topic model. In the models depicted inFIG. 3-5, the bolded circles represent data while the remaining circles represent probability distributions over the elements below them. The squares represent repetition of the model across an attribute, such as words in the call or all topics. The lines represent the modeling of one set of data as a draw from the above distribution of data. Finally, the bolded circle represents known input data, such as the words spoken in a call.

In the LDA topic model, each word of words302is modeled as a sample from the topics304which represent one or more topics spoken on the call. Topics304represent one of several variables being calculated through the topic model. Each word of words302is thus modeled as a probability of that word occurring given a topic of topics304being spoken in the call and a probability of that topic occurring on the call. As denoted by the box around these two circles, this modeling is repeated for all words in the call.

On the topic side of the model, the topics304are modeled as being drawn from a distribution of topics306for each call. In the LDA model, topics304are modeled from the distribution of topics306using a categorical model, such as a generalized Bernoulli distribution. Thus, in each call, there is assumed to be a probability distribution of topics304for each word of words302. The probability distribution of topics304is assumed to be drawn from an overall call distribution of topics306, Each word of words302is thus modeled as being drawn from that word's probability distribution of topics304. This portion of the model is repeated across all calls which are used as input data into the model. The distribution of topics is modeled as being drawn from a prior distribution. In the LDA model, the prior distribution is a uniform prior distribution.

On the word side of the model, the words302are modeled as being drawn from a distribution of words308for each topic. In the LDA model, words302are modeled from the distribution of words308using a categorical distribution. The distribution of words308is replicated over topics, indicating that there exists a distribution of words for each of distinct topic of topics304. Thus, words302are modeled as being drawn from a distribution of words308given one or more topics. The distribution of words is also modeled as being drawn from a prior distribution. In the LDA model, the prior distribution is a uniform prior distribution.

The LDA model is trained using input data from previous conversations. The input data comprises data from a plurality of previous calls. The data for each call comprises the words spoken on the call and identified topics for the call. Using the input data, the parameters for the different distributions can be calculated. A generative process is then run to model the topics304as a function of the words302spoken on the call. This can be done through Bayesian updating, such as by using Gibbs sampling, or through any other type of Monte Carlo simulation.

When a new dataset is received, the new dataset comprising a transcription of a call, the system uses the model to compute one or more of a set of topics spoken on the call, a probability of each of the set of topics spoken on the call, a topic for each word spoken on the call, and/or a probability of each of the topic for each word spoken on the call.

4.2 Improved Topic Model for Calls

FIG. 4depicts an example of an improved topic model which captures the non-uniformity in probabilities of certain topics discussed. The LDA model assumes that each topic has an equal probability of being pulled from the prior distribution. In actuality, certain topics are more likely to occur. For example, repair calls may be more common than sales calls for a car dealership as a sale only occurs once per lifecycle of a car while repairs may be performed multiple times during the lifecycle of a car.

In the improved model ofFIG. 4, each of words402is modeled as a sample from per-word topic distributions404. Similar to the LDA model, the topics in each call are modeled as being pulled from a distribution of topics406. This process is repeated across all calls. Up to this point, the model has been similar to the LDA model. An improvement to the model ofFIG. 4is that while the LDA model models the distribution of topics as a sample from a Dirichlet distribution with a uniform prior, the model ofFIG. 4infers the prior distribution408which itself is modeled as a draw from a distribution with a uniform prior. In an embodiment, the model ofFIG. 4is further improved by using a Pitman-Yor process to model probabilities over each of the distributions. As the Pitman-Yor process is more flexible and better suited to language than the Dirichlet distribution used for the LDA model, the model's performance is improved through the use of the Pitman-Yor process. For instance, the Pitman-Yor process can model power-law distributions which better match distributions of words in language, thus providing more accurate and more efficient models.

An example method of modeling topics404as being drawn from a distribution of topics406which is drawn from an inferred prior distribution408draft from a flat prior distribution is described herein. Assuming topics (z) in a call are drawn from distribution of topics (θ) over a plurality of calls which are drawn from prior distribution (α) which is drawn from a flat prior distribution (α0), a probability of a particular topic being drawn may be computed as P(α, θ, z|α0) where only α0is a known variable. Given that the distributions α and θ are unknown, the distributions are described in terms of customer counts c, representing tallies of data within the distribution, which are partitioned into a set of latent counts called table counts t which represent the fraction of the customer counts which get passed up the hierarchy to inform the parent distribution, i.e. the number of customer counts that show up in the parent node or ckα≡tkθ. Using customer and table counts, the probability of a topic may be computed as:

As the customer counts in the above equation are a deterministic tally of data from x, the server computer may compute the probability above by sampling the table counts using a Gibbs sampler. Additionally or alternatively, a table indicator (u) may be defined as a Boolean variable indicating whether or not a data point created a new table count: tk=Σn=1ckun,k. The server computer may sample the table indicators instead of the table counts to reduce the computational cost of sampling table counts. Bayes theorem may then be used to compute the probability of a given data point using the above equations and table counts sampled from the Gibbs sampler. For example, the server computer may compute the probability of a particular data point being absent and divide the probability above by the probability of the absent data point to compute the joint probability for the latent variables associated with the data point. Samples can be drawn from the resulting equation and latent variables may be stored for the data point. The server computer may then continue this process for each additional latent variable.

In the improved model ofFIG. 4, a second improvement is displayed in the word side of the model. The LDA model assumes that each topic has a single distribution of words associated with it. While often true in text, language used in speech can vary from region to region or person to person based on differences in dialect and manner of discussing the same topic. Thus, the LDA model may identify a plurality of topics, one for each region or personal style of discussing a topic. In the improved model ofFIG. 4, a call-specific distribution of words410is modeled as being pulled from a corpus-wide probability distribution of words412for each topic of a plurality of topics. Words402in each call are thus modeled as being drawn from call-specific distributions of words410replicated across a plurality of topics.

As a further improvement, as with the topic side of the model, the probability distribution of words412is modeled as being drawn from an inferred prior distribution414which is drawn from a flat prior distribution. This process may be modeled using a Dirichlet distribution or Pitman-Yor Process. An example method of modeling words402as being drawn from a call-specific probability distribution of words410which is drawn from probability distribution of word412, drawn from an inferred prior distribution414, drawn from a flat prior distribution is described herein. Assuming words (w) in a call are drawn from a call-specific probability distribution for the call (ψ) which are drawn from a probability distribution of words (ϕ) for each of a plurality of topics which is drawn from an inferred prior distribution (β) which is drawn from a prior distribution (β0), a probability of a word being drawn from the model may be computed as:

P⁡(w,ψ,ϕ,β|β0)=[∏V⁢β0,vcvβ0]⁡[fβ][∏K⁢fϕ⁢k][∏K⁢∏D⁢fψd,k]
where v ranges over the dimension of the node V which represents the size of the vocabulary of words, k ranges over the dimension of the node K which represents the number of topics on the topic side of the model, d ranges over the dimension of the node D which represents the number of calls, and where:

f𝒩≡(b(𝒩)|a(𝒩))T(𝒩)(b(𝒩))C(𝒩)⁢∏J⁢Stj(𝒩)cj(𝒩)Htj(𝒩)cj(𝒩)
Where j indexes over the dimension of the distribution (i.e., J=K on the topic side of the model, and J=V on the word side of the model).

Improvements on either side of the model described above may be utilized independent of each other by using the depicted equations along with the equations of the LDA model. Additionally or alternatively, the two probabilities may be combined to compute the probability P(z, θ, α, w, ψ, ϕ, β|β0, α0) in the improved model ofFIG. 4using the equation below:

P=[∏V⁢β0,vcvβ0]⁡[fβ][∏K⁢fϕ⁢k][∏K⁢∏D⁢fψd,k]×[∏K⁢α0,kckα0]⁡[fα][∏D⁢fθd]
where the first part of the equation represents the word branch and the second part of the equation represents the topic branch.

As with the topic branch improvement described above, a Gibbs sampler may be defined which samples table counts from the dataset to compute a resulting probability from the above equation with Bayes theorem being used to compute the probability of a topic given the words spoken in a call. Since theterms are the only ones with table counts, a term may be defined as:

R(𝒩)≡f(𝒩)f⫬d,n(𝒩)
whereis the state of the model with the word wd,nremoved. The server computer may sample from the above equation and compute the product ofacross all nodes to produce the latent variables for each word wd,nin the dataset.

To obtain the state of the model with a word removed, the system may sample P(z, θ, α, w, ψ, ϕ, β|β0, α0) as computed above. Table indicators for the model with the word removed may be sampled from the following equation:

ud,n⁢∼⁢Bern⁡(tzd,nczd,n).
While sampling the state of the model with the word wd,nremoved, the server computer may check the following constraints: t≤c and t=0 if an only if c=0. If either constraint is violated, the server computer may restore the state of the model and continue the process with the next word.

The improvements of the model ofFIG. 4allow the server computer to more accurately model topics from a conversation by taking into account variances in topic likelihoods and variances in the way different people discuss different topics. Thus, the server computer is able to more accurately determine topic models with less input information, thereby decreasing the computational cost of providing accurate topic categorizations of phone calls. Additionally, the higher accuracy of the topic model ofFIG. 4decreases the need for post-processing steps to clean or otherwise alter results of the topic model, thereby reducing resources used in computing topics for individual conversations.

4.3 Improved Topic Model with Party Segregation

FIG. 5depicts an example of an improved topic model which segregates parties of the conversation. The LDA model and the model ofFIG. 4both treat all words in a phone conversation as coming from a singular source when sampling from distributions of words. Thus, agent dialogue, which is often scripted or based on specific language provided to the agent for use in different calls, is mixed with the customer's language, which often varies across individuals. The differences in how customers speak and how agents speak are not captured by the model which treats all words as coming from the same source.

The topic side of the model ofFIG. 5is similar to the topic side of the model ofFIG. 4. Topics504for words in a call are modeled as being drawn from a distribution of topics506in each call which is modeled as being drawn from an inferred prior distribution508which is drawn from a flat prior distribution. This process may be modeled using a Dirichlet distribution or Pitman-Yor Process.

On the word side of the model, prior to training the model the server computer may split words502into two sets of words, first person type words and second person type words. The first person type and second person type refer to types of people for a specific implementation of the model. For example, some businesses may split the calls into caller words502aand agent words502b. The model does not depend on the types of people being callers and agents and other implementations may be executed with the model ofFIG. 5provided that the calls comprise at least two topics of people. For example, campaigning calls may be split between campaign representative and voters.

While the model is described below with respect to person type distinctions, the segregation techniques described herein may segregate words in the model using any type of metadata. For example, instead of caller-specific and agent-specific distributions of words, there may be seasonal distributions of words, regional distributions of words, or any combination of segregated distributions. As is described below, a benefit of the topic model ofFIG. 5is its scalability to multiple topic segregations. Thus, there could be a distribution of words for each combination of caller/agent and season. This also allows for customization of a topic model to a specific type of business. For example, a topic model can be customized for a car dealership with a sale-time distribution, a new car release time distribution, and a distribution for non-sale and non-release times.

Each of the segregated sets of words are modeled simultaneously on person type-specific distributions. Thus, caller words502aare modeled as being drawn from a call-specific caller distribution of words510awhich is modeled as being drawn from a caller distribution of words512awhich is drawn from an overall distribution of words514for the topic. Similarly, agent words502bare modeled as being drawn from call-specific agent distribution of words510bwhich is drawn from an agent distribution of words for the topic512b, which is drawn from the general distribution of words514for each topic. Thus, while the caller words and agent words are separately modeled based on caller-specific and agent-specific distributions of words, both sets of distributions are modeled as being drawn from a general distribution of words514which is modeled as being drawn from an inferred prior distribution516which is modeled as being drawn from a flat prior distribution.

An example method of modeling words502by simultaneously modeling first person type data as a function of a first probability distribution of words used by the first person type for and the second person type data as a function of a second probability distribution of words used by the second person type, where both probability distributions are modeled as a function of a third probability distribution of words for one or more topics is described herein. As an example, the server computer may compute the probability P(z, θ, α, w, ψ, ϕ, β|β0, α0) for a plurality of speakers, S, using the equation below:

where the distribution η is the general distribution of words and ϕsis a distribution of words for an individual party to the call. Given that the addition of parties adds to the product of the terms with the s subscript, the model described above can be extrapolated to include any number of parties. As described above, sampling for the models described herein comprises sampling table counts for each node. Given that ckμ≡tkv, table counts are used to inform the customer counts of the parent nodes. When a node has more than one child node, such as the general distribution of words514, the table counts are summed across all children nodes. In order to increase the computational efficiency of summing the table counts, a hierarchical structure is defined where related sets of distributions are grouped together into nodes and related sets of nodes are grouped into layers. Where a parent node is drawing from a child node of the same size, each draw from a probability distribution of the child node may be passed to a corresponding distribution of the parent. Where a parent node with a single distribution draws from a child node with multiple distributions, the number table counts are summed across all children. If a number of distributions in a child node does not match a number of distributions in a parent node, the parent nodes may sum over a random or pseudo-random variable number of distributions in the child node while tracking which children nodes to sum over.

4.4 Topic Model Generator

In an embodiment, the server computer provides a topic model generator to the client computing device. The topic model generator, as used herein, refers to providing options for specifying nodes in a topic model which is then computed by the server computer. The topic model generator may comprise a graphical user interface with options for selecting and adding nodes to a graph and/or a text file with a designated structure such that adding nodes to the graph comprises editing data in the text file to identify nodes to be added to the graph in different locations.

FIG. 6depicts an example method for dynamically building a model based on user input. At step602, input is received specifying nodes for a model. For example, the server computer may receive a request to build a topic model with a plurality of nodes. The request may include call transcripts to be categorized through the topic model and/or specify a set of stored call transcripts to be categorized through the topic model. The input may specify nodes for both the topic side of the model and the word side of the model. The input may be received through a graphical user interface or through a text file specifying the building of a model based on particular nodes.

At step604, the server computer populates a matrix with terms for each node specified in the model. For example, a matrix may be defined with terms that rely on the table counts, using the variableas defined above. The columns of the matrix may correspond to topics (k), while the rows correspond to the nodes specified by the user input. The first row of the matrix may refer to the lowest child node aside from the final word or topic node. Thus, the first row is populated with terms for when u=0 on the lowest child node aside from the final word or topic node. The next row corresponds to the next lowest child node. Thus, u=0 for the next lowest child node and u=1 for the lowest child node. Anis only added to the matrix for a parent node when u=1 for its child node. While computing the probability over all states of the model could cause the sampling task to become exponentially more computationally complex for each node added to the model, the server computer may restrict analysis to only possible states of the model. The server computer may store the possible states in a two-dimensional matrix which is then used to compute the values in the matrix described above.

As an example, a matrix for both the topic branch (ptopic) and the word branch (pword) described inFIG. 4may be computed as:

ptopic≡[Rk=0,u=0θRk=1,u=0θ…Rk=0,u=1θ⁢Rk=0,u=0αRk=1,u=1θ⁢Rk=1,u=0α…Rk=0,u=1θ⁢Rk=0,u=1α⁢Rk=0,u=0α0Rk=1,u=1θ⁢Rk=1,u=1α⁢Rk=1,u=0α0…]pword≡[Rk=0,u=0ψRk=1,u=0ψ…Rk=0,u=1ψ⁢Rk=0,u=0ϕRk=1,u=1ψ⁢Rk=1,u=0ϕ…Rk=0,u=1ψ⁢Rk=0,u=1ϕ⁢Rk=0,u=0βRk=1,u=1ψ⁢Rk=1,u=1ϕ⁢Rk=1,u=0β…Rk=0,u=1ψ⁢Rk=0,u=1ϕ⁢Rk=0,u=1β⁢Rk=0,u=0β0Rk=1,u=1ψ⁢Rk=1,u=1ϕ⁢Rk=1,u=1β⁢Rk=1,u=0β0…⁢]
where each column corresponds to an increasing value of k and each row includes an additionalterm for a parent of the last node in the previous row. Each position in the matrix thus represents a state for that branch of the model with its value representing the probability for the branch to take on that state. Thus, the system can build matrices with any number of nodes by adding rows for each requested node.

The matrix may be initiated based on the user input. First, a matrix may be generated with a number of rows equal to the number of nodes specified in the user input. Thus, if the user input specifies three nodes to be added to the final topic node, the server computer may generate a matrix with three rows. All elements in the matrix may be initialized to 1. Then, for l∈[1, depth−1], the server computer may compute the vectors Rk,u=0land Rk,u=1l, compute the product of Pl,kand Rk,u=0l, and, for m∈[l+1, depth], multiply Pm,kand Rk,u=1l. The server computer may then compute Rk,u=0for the base node and compute the product of Pdepth,kby Rk,u=0.

At step606, a total marginal probability is computed using the matrix. For example, the server computer may initially compute the partial marginal probabilities by summing each matrix along its depth axis. The total marginal probability may then be computed by multiplying the partial marginal probabilities elementwise.

At step608, topics are sampled from the total marginal probability. For example, the server computer may evaluate the matrices defined in step604for each word within each dataset. The server computer may continue evaluating the matrix with different words until convergence is reached. The server computer may then use the matrices to compute the total marginal probabilities for each of a plurality of topics for each of the words in a dataset.

Using the method ofFIG. 6, the server computer can generate a model on demand with nodes specified by a client computing device. By automatically generating and training a model based on user input, the server computer provides configurable models without requiring in-depth knowledge or expertise in generating conversation models.

5.0 Topic Display

In an embodiment, the server computer provides topic information to the client computing device. The topic information may indicate, for each of a plurality of topics, a number or percentage of calls received for that topic over a particular period of time. For example, the server computer may send calls received for different topics on an hourly, daily, weekly, or monthly basis. The server computer may additionally provide options to customize the topic information. For example, the server computer may provide an interface where a client computing device specifies a start time/date and an end time/date. The server computer may provide the topic information for the specified period of time by identifying each call received during that period of time and incrementing a topic counter for each topic when a call was identified as corresponding to the topic.

The server computer may provide graphs that depict the topic information to the client computing device. For example, the server computer may generate a histogram with the x-axis corresponding to time intervals, such as hours, days, or weeks, and the y-axis corresponding to a number or percentage of calls that were received for a topic. Separate histograms may be provided for each topic and/or a joint histogram may be generated which includes a plurality of bars for each time interval, each of the plurality of bars corresponding to a different topic of a plurality of topics.

In an embodiment, the server computer further identifies the words that correspond to each of the topics, such as by computing the probabilities for words individually and identifying corresponding probabilities for different topics. As the topics may not be named in advance, specifying the words with the highest probabilities of being associated with a topic allow for easier identification of the topic. If the server computer receives input naming a particular topic, the server computer may update stored data to include the name of that topic for other data sent to the client computing device.

The server computer may use the identified words for each of the topics to generate a word bubble display for the client computing device. The word bubble display may include a plurality of bubbles, each corresponding to a different topic. The size of the bubble may correspond to the frequency with which the topic is discussed, with larger bubbles corresponding to topics that are discussed more frequently and smaller bubbles corresponding to topics that are discussed less frequently. The bubbles may include words inside them that correspond to the topic of the bubble. For example, a bubble for the topic of purchasing a vehicle may include the words “car”, “price”, “financing”, and “credit”.

The server computer may provide a graphical user interface to the client computing device with the topic information. The graphical user interface may provide charts and graphs for different and/or customizable time periods corresponding to call data provided by the client computing device. The graphical user interface may comprise insights to the call data, such as origins and destinations of the calls within different topics retrieved from metadata. The graphical user interface may additionally provide options to rename topics and/or merge topics.

In an embodiment, the topic information is provided to a real-time bidding platform where users bid on calls based on keywords of the call or other information. The topic information may additionally be used to intelligently route calls from a source to a destination.

Computer system700further includes a read only memory (ROM)708or other static storage device coupled to bus702for storing static information and instructions for processor704. A storage device710, such as a magnetic disk, optical disk, or solid-state drive is provided and coupled to bus702for storing information and instructions.