CUSTOMER SERVICE TICKET SIMILARITY DETERMINATION USING UPDATED ENCODING MODEL BASED ON SIMILARITY FEEDBACK FROM USER

Techniques are provided for customer service ticket similarity determination using an updated encoding model based on similarity feedback from a user. One method comprises obtaining encodings of customer service tickets in a vector space using an encoding model; determining pairwise similarities for the encodings of the customer service tickets; obtaining feedback from a user regarding the pairwise similarities for a subset of the encodings; updating pairwise similarities for the subset of the encodings using the feedback from the user, generating an updated encoding model by processing the updated pairwise similarities for the subset of the encodings of the customer service tickets using a supervised learning algorithm; and processing at least one customer service ticket based at least in part on the updated encoding model. The feedback from the user may indicate a similarity of two or more of the plurality of customer service tickets.

FIELD

The field relates generally to information processing systems, and more particularly to the processing of customer service tickets in such information processing systems.

BACKGROUND

Customer support requests are often documented using tickets. In the field of IT (Information Technology), for example, a ticketing system is often used to manage IT tickets. There is often a large number of duplicate and/or otherwise related tickets. For example, in the IT context, related tickets are often encountered when a server fails, or when a failure occurs with a single sign-on process. In these types of situations, a number of users will typically submit independent customer support requests, with many users often describing the nature of the request differently, although the root cause for the multiple requests is often the same or at least related.

SUMMARY

In one embodiment, a method comprises obtaining encodings of a plurality of customer service tickets in a vector space using an encoding model, wherein the encodings of the plurality of customer service tickets are generated using a self-supervised learning algorithm; determining pairwise similarities for at least a subset of the encodings of the plurality of customer service tickets; obtaining feedback from a user regarding at least some of the pairwise similarities for the subset of the encodings; updating one or more of the pairwise similarities for the subset of the encodings of the plurality of customer service tickets using at least some of the feedback from the user; generating an updated encoding model by processing the updated pairwise similarities for the subset of the encodings of the plurality of customer service tickets using a supervised learning algorithm; and processing at least one customer service ticket based at least in part on the updated encoding model.

In one or more embodiments, the obtaining the encodings of the plurality of customer service tickets in the vector space further comprises obtaining tokenized versions of the plurality of customer service tickets. The feedback from the user regarding the at least some pairwise similarities for the subset of the encodings may indicate a similarity of two or more of the plurality of customer service tickets.

In some embodiments, the generating the updated encoding model comprises, for a given training epoch of a plurality of training epochs, obtaining a batch of customer service tickets from the plurality of customer service tickets; transforming the batch of customer service tickets into encodings using a current encoding model; determining pairwise similarities for the encodings of the batch of customer service tickets; generating a first aggregate similarity value obtained from the pairwise similarities for the encodings of the batch of customer service tickets; generating a second aggregate similarity value obtained from the pairwise similarities for the encodings of the corresponding customer service tickets in the plurality of customer service tickets; and evaluating a loss function using the first aggregate similarity value and the second aggregate similarity value and applying a supervised learning algorithm to fit the encoding model with respect to the loss function.

DETAILED DESCRIPTION

Illustrative embodiments of the present disclosure will be described herein with reference to exemplary communication, storage and processing devices. It is to be appreciated, however, that the disclosure is not restricted to use with the particular illustrative configurations shown. One or more embodiments of the disclosure provide methods, apparatus and computer program products for customer service ticket similarity determination using an updated encoding model based on similarity feedback from a user.

As noted above, customer support requests are often documented using customer service tickets. Such customer service tickets are often the first interaction between a user requesting support and a team responsible for resolving the support issue. Thus, customer service tickets typically include information that the user believes is relevant for the support analysis.

Although there is often a large number of duplicate and/or otherwise related tickets, it may not be easy for different support ticket operators to identify such duplicate or similar tickets. This kind of situation can delay the pace of work of the support team since someone may be spending some effort resolving something that was previously resolved and the prior solution could be leveraged again.

One or more aspects of the disclosure recognize that the time consumed by such efforts unnecessarily impacts the metrics of a support team, and that knowing that a given customer service ticket is a duplicate (or similar) to a previously resolved customer service ticket would benefit the support team (e.g., by leveraging the prior solution and/or the prior service personnel). In this manner, such customer service tickets can be resolved in a shorter amount of time. Typically, customer service platforms require closing notes for a customer service ticket indicating the steps that were performed to resolve the issues. As used herein, the term “customer service tickets” shall be broadly construed to encompass service data structures, transaction data structures, interaction data structures and other data structures that record information obtained regarding interactions with or on behalf of a user, such as a customer.

In one or more embodiments, the disclosed ticket similarity determination techniques aim to find relationships within the support ticket data so that new incidents are solved more efficiently by leveraging previous solutions. In a typical environment for a support ticket system, the inputs of a user are stored as a ticket entry at a repository. This entry typically comprises multiple fields, such as a unique identifier (ID), a creation timestamp, and textual fields for details. The disclosed ticket similarity determination approach does not require any particular specific set of fields, and is adaptable to any number of fields, such as message and/or note fields that are typically present in such systems.

In at least some embodiments, the disclosed ticket similarity determination framework employs natural language processing techniques that identify support tickets in a database that relate to a newly opened ticket. Overall, the disclosed ticket similarity determination methodology identifies support tickets that have already been closed with a solution by encoding the ticket description and searching for similarities in the encoded space. The most similar previous tickets (e.g., a top N list) can be provided, for example, in a ranked manner, for review by a human operator. The review and analysis of the human operator can then be leveraged to fine-tune the weights of the encoding process for future support issues.

The disclosed ticket similarity determination framework is capable of suggesting previous tickets that are similar to a new ticket without the need for any initial supervised dataset. The framework then leverages available human revision (e.g., feedback from the user indicating whether prior results were accurate), resulting from normal operation guided by the disclosed approach, to adjust the weights of the features in the similarity metric.

FIG.1shows a computer network (also referred to herein as an information processing system)100configured in accordance with an illustrative embodiment. The computer network100comprises a plurality of user devices103-1through103-M, collectively referred to herein as user devices103. The user devices103are coupled to a network104, where the network104in this embodiment is assumed to represent a sub-network or other related portion of the larger computer network100. Accordingly, elements100and104are both referred to herein as examples of “networks” but the latter is assumed to be a component of the former in the context of theFIG.1embodiment. Also coupled to network104is one or more customer service ticket correlation servers105and one or more customer service ticket databases106, discussed below.

The user devices103may comprise, for example, host devices and/or devices such as mobile telephones, laptop computers, tablet computers, desktop computers or other types of computing devices. Such devices are examples of what are more generally referred to herein as “processing devices.” Some of these processing devices are also generally referred to herein as “computers.” The user devices103may comprise a network client that includes networking capabilities such as ethernet, Wi-Fi, etc. When the user devices103are implemented as host devices, the host devices may illustratively comprise servers or other types of computers of an enterprise computer system, cloud-based computer system or other arrangement of multiple compute nodes associated with respective users.

For example, the host devices in some embodiments illustratively provide compute services such as execution of one or more applications on behalf of each of one or more users associated with respective ones of the host devices.

As shown inFIG.1, an exemplary customer service ticket correlation server105may comprise a customer service ticket encoding module112, a customer service ticket candidate scoring module114, a customer service ticket re-encoding module116and a customer service ticket pruning module118. In some embodiments, the customer service ticket encoding module112assigns a numerical vectorial representation to each customer service ticket to encode a respective customer service ticket in an encoded space, as discussed further below in conjunction withFIGS.2and3, for example. The customer service ticket candidate scoring module114assigns a similarity score to each customer service ticket, as discussed further below in conjunction withFIGS.2and5, for example. The customer service ticket re-encoding module116updates pairwise similarity values for at least some of the customer service tickets using feedback from a user and generates an updated encoding model, as discussed further below in conjunction withFIGS.7through13, for example. The customer service ticket pruning module118removes tickets from the customer service ticket database106that satisfy one or more ticket pruning criteria.

It is to be appreciated that this particular arrangement of elements112,114,116,118illustrated in the customer service ticket correlation server105of theFIG.1embodiment is presented by way of example only, and alternative arrangements can be used in other embodiments. For example, the functionality associated with elements112,114,116,118in other embodiments can be combined into a single element, or separated across a larger number of elements. As another example, multiple distinct processors can be used to implement different ones of elements112,114,116,118or portions thereof.

At least portions of elements112,114,116,118may be implemented at least in part in the form of software that is stored in memory and executed by a processor. An exemplary process utilizing elements112,114,116,118of the customer service ticket correlation server105in computer network100will be described in more detail with reference toFIGS.2through13.

Other customer service ticket correlation servers105(not shown inFIG.1), if any, are assumed to be configured in a manner similar to that shown for customer service ticket correlation server105in the figure.

The customer service ticket correlation server105may be implemented, for example, on the cloud, such as a private cloud, or on the premises of an enterprise or another entity. In some embodiments, the customer service ticket correlation server105, or portions thereof, may be implemented as part of a host device.

Additionally, the customer service ticket correlation server105can have an associated customer service ticket database106configured to store, for example, information related to one or more customer service tickets, as discussed further below.

The customer service ticket database106in the present embodiment is implemented using one or more storage systems associated with the one or more customer service ticket correlation servers105. Such storage systems can comprise any of a variety of different types of storage such as, network-attached storage (NAS), storage area networks (SANs), direct-attached storage (DAS) and distributed DAS, as well as combinations of these and other storage types, including software-defined storage.

The one or more user devices103and/or customer service ticket correlation servers105may be implemented on a common processing platform, or on separate processing platforms. The one or more user devices103may be configured to interact over the network104in at least some embodiments with the one or more customer service ticket correlation servers105, for example.

Also associated with the one or more user devices103and/or customer service ticket correlation servers105can be one or more input-output devices (not shown), which illustratively comprise keyboards, displays or other types of input-output devices in any combination. Such input-output devices can be used, for example, to support one or more user interfaces to the customer service ticket correlation servers105, as well as to support communication between the customer service ticket correlation servers105and other related systems and devices not explicitly shown.

The one or more user devices103and/or customer service ticket correlation servers105in theFIG.1embodiment are assumed to be implemented using at least one processing device. Each such processing device generally comprises at least one processor and an associated memory, and implements one or more functional modules for controlling certain features of the respective device.

More particularly, the one or more user devices103and/or customer service ticket correlation servers105in this embodiment each can comprise a processor coupled to a memory and a network interface.

The network interface allows the one or more user devices103and/or customer service ticket correlation servers105to communicate in some embodiments over the network104with each other (as well as one or more other networked devices), and illustratively comprises one or more conventional transceivers.

It is to be understood that the particular set of elements shown inFIG.1for customer service ticket similarity determination using an updated encoding model based on similarity feedback from a user is presented by way of illustrative example only, and in other embodiments additional or alternative elements may be used. Thus, another embodiment includes additional or alternative systems, devices and other network entities, as well as different arrangements of modules and other components.

Text encoding (e.g., word encoding and sentence encoding) approaches are typical of natural language processing tasks in machine learning literature. These approaches allow numerical representations of textual tokens (e.g., words) in vector form such that similar words are closer in the vector space. A number of text encoding techniques are available. Doc2vec, for example, is based on the word2vec approach and requires no explicit labeling. Doc2vec relies on distributional semantics and obtains, from a corpus of texts, numeric representations of documents.

FIG.2illustrates an exemplary processing of customer service tickets to identify similar customer service tickets that have been previously resolved in accordance with an illustrative embodiment. A user typically enters the necessary information for a given customer service ticket which is stored as a ticket entry in a customer service ticket database260. Each support ticket item contains a textual description of the problem and other relevant identification fields. In the example ofFIG.2, a such customer service tickets may comprise a ticket identifier, a creation timestamp, and a system identifier. The ticket entry may track the status of the ticket in a ticket handling process255, for example, as being open, updated or closed, with a timestamp indicating an amount of time since changing to a current status. In addition, the disclosed ticket similarity determination techniques recognize that the operator(s) (of the ticket handling process255) often annotate the ticket entry with notes, typically relating to the resolution (e.g., closing) of the request.

The customer service ticket database260provides a source for the similarity searching framework and includes descriptions of previously closed support tickets and indications of the actions taken to resolve them. Since descriptions are provided in natural language, errors within and across support tickets may occur. The disclosed solution thus includes a pre-processing step to clean and standardize the textual representation of the database fields that will be involved in the similarity searches.

A given customer service ticket210is applied to a preprocessing module215that operates over information input by the user, and outputs the original ticket information in an encoded format (e.g., an embedding). The preprocessing module215formats the contents of each customer service ticket and comprises a sanitizer220, a tokenizer225and an encoder230. The sanitizer220may remove certain stop words. For example, the sanitizer220may format the data so that encodings, accent marks, whitespace and punctuation are changed/removed in a uniform fashion. Data normalization tasks may also be employed, such as: substituting contractions for the corresponding long form expression of each contraction; stemming and lemmatization of terms, by substituting them for the root or base word, such as in verbs changed into the radical/base form (e.g., “allowed” to “allow”) and modal adverbs changed into the base term (e.g., “happily” to “happy”, “quickly” to “quick”); the removal of specified stop words (e.g., “is”, “a”, “this”, “the”). The sanitizer220provides a reduced set of terms (with respect to variation) so that in the tokenization of the sanitized text the same token is assigned to semantically similar terms.

The tokenization process performed by the tokenizer225converts the text of each customer service ticket to textual tokens (e.g., obtaining a token for each resulting word in the sentence). In the case of the tickets in the customer service ticket database260that are closed and/or have additional text fields provided by operators, the context of such text may be appended to the description for the purposes of generating the resulting tokens.

The resulting set of tokens tiof each ticket is encoded by an encoding algorithm employed by the encoder230that assigns a numerical vectorial representation to each customer service ticket to encode a respective customer service ticket in an encoded space. This numerical representation determines features of the ticket, which determines the dimension of the encoded vector.FIG.3illustrates a learned encoding function330that generates encodings, E,350from the tokenized descriptions, t,310of each ticket, where e; represents the encoding of the i-th ticket in accordance with an illustrative embodiment. The encoder230may employ, for example, a doc2vec approach that trains a neural network to learn the learned encoding function330from a corpus of text, such that words that occur in similar contexts are codified into a similar vectorial representation, conditioned by unique document identifications.

A candidate scoring stage searches the historical data in the customer service ticket database260ofFIG.2for similar prior customer service tickets (in the encoded space), using similarity scoring235, discussed further below in conjunction withFIGS.4and5, and presents them to an operator on a user device250for candidate selection240, as discussed further below in conjunction withFIG.6. The disclosed ticket similarity determination approach further comprises an encoding revision stage, in which the annotations of correct/incorrect candidate suggestions from the operator are leveraged to fine-tune the candidate scoring step for future iterations, as discussed further below. A ticket handling process stores the processed ticket and the appropriate operator notes in the customer service ticket database260.

FIG.4illustrates a similarity matrix, M, comprising the pairwise similarity of the encodings, E,410from the tokenized descriptions of each customer service ticket, t, in accordance with an illustrative embodiment. The similarity matrix, M, comprises a row and a column for each customer service ticket and the pairwise similarity is obtained between each customer service ticket.

Each element i, j in the matrix, M, is computed as follows:

where s is a similarity function in the encoded space (typically, cosine similarity). InFIG.4, the similarity between encodings e0and e11is highlighted. The main diagonal reflects that all tickets are maximally similar to themselves (as expected from a coherent encoding scheme). The matrix, M, is symmetric, which can be exploited in some embodiments to reduce the computational complexity and the storage size of the resulting structure (e.g., the similarity s(ei, ej)≡s(ej, ei)).

FIG.5illustrates an encoding of a new customer service ticket520and a query540to identify similar previously processed customer service tickets in accordance with an illustrative embodiment. In the example ofFIG.5, the new customer service ticket520is preprocessed at stage525to generate a tokenized representation, tN+1, of the new customer service ticket520. The new customer service ticket520is then encoded at stage530to generate a corresponding encoding, eN+1, in a similar manner as discussed above in conjunction withFIG.2.

An operator submits a query540to identify, for example, a set550of the top-k most similar candidate customer service tickets (e.g., the k nearest neighbors (k-NN)) relative to the query element in the vectorial space of the description text encoding of the encoding, eN+1of the new customer service ticket520. In the example ofFIG.5, the three most similar tickets to the new customer service ticket520are tickets with index (in the customer service ticket database, and therefore, the similarity matrix, M)10,11, and7.

In at least some embodiments, the query540uses the same similarity metric s in the vectorial space as was used to determine the similarity matrix, M (again, typically, the cosine similarity), and determines which items of the database of customer service tickets are most similar to the query item.

It is noted that the encoding of the new customer service ticket520is performed over the description (as no closing notes and other textual fields are available). Thus, the computed similarity score considers similarity between description and closing notes interchangeably. This may result in false-positive similarities, but may also determine (correct) similarities between a new customer service ticket and an old customer service ticket when the solution to the old customer service ticket is similar to the description of the new customer service ticket.

FIG.6illustrates a presentation of a set610of the top-k most similar candidate customer service tickets to a user on a user interface650(e.g., a graphical user interface) as a set of candidate tickets660in accordance with an illustrative embodiment. The candidate tickets are presented to the operator review and analysis with respect to a new customer service ticket665. The user interface650may comprise functionality for the candidate tickets660to be compared to the new customer service ticket655in a coordinated (e.g., guided) manner. In addition, the user interface650may provide a variety of functionalities to streamline and facilitate the identification of similar concepts/contents between the new customer service ticket655and the candidate tickets660, as would be apparent to a person of ordinary skill in the art. The operator may optionally provide user annotations620(e.g., manual labels as ground truth), for example, in the form of positive or negative indications. The annotation v(ei, ej) by the operator informs whether e; and e; are similar. Thus, the operator may indicate which, if any, of the candidate tickets660are semantically related (typically, referring to a same issue, knowledge base article, solution or escalation process) to the new customer service ticket655.

The operator may then work on the new customer service ticket655, or resolve it, adding new textual data to the ticket entry (e.g., “closing notes”). The re-encoding of the new customer service ticket655is dealt with to account for the added textual information in the triggering or scheduling of the re-encoding process, discussed further below.

FIG.7illustrates a representative data structure, Kj, that holds the top-k similar candidates with any added labels provided by the annotation process ofFIG.6for a new customer service ticket, tj(j>N).

The structure holds tuples (e, s, v) indexed by i, in which:i is the index of the candidate ticket (a, b, c, in the example ofFIG.7) in the customer service ticket database (and therefore in the similarity matrix, M);e is the encoding ei;s is the similarity s(ej, e); andv is the “ground truth” provided annotation of similarity between customer service tickets.

The indexing notation Kjarepresents the first tuple in Kj; and Kja[e], Kja[s], and Kja[v] to represent fields e, s, and v the tuple, respectively.FIG.7highlights the notation v to refer to the operator annotation provided for the similarity between tickets tjand tb.

These annotations v provide a form of labelling for a weakly-supervised revision of the encoding, as described below.

In typical embodiments, the values of v may be constrained between −1 and 1 (with 1 meaning similarity, −1 meaning no similarity, and 0 representing absent annotation). Different configurations are possible (e.g., in which normalized values between 0 and 1 are possible). Below, this typical embodiment is considered for an adjustment of the similarity matrix, M.

FIG.8illustrates a batch processing of a set of new customer service tickets820and a query840against encodings810to identify similar previously processed customer service tickets in accordance with an illustrative embodiment. In the example ofFIG.8, each new customer service ticket820is preprocessed at stage825to generate a tokenized representation, tN+1, of the respective new customer service ticket820. Each new customer service ticket820is then encoded at stage830to generate a corresponding encoding, eN+1through eN+3, in a similar manner as discussed above in conjunction withFIG.2.

An operator submits a query840to identify, for example, a set K of the top-k most similar candidate customer service tickets (e.g., the k nearest neighbors (k-NN) relative to the query element in the vectorial space of the description text encoding of the encodings, eN+1through eN+3, of the new customer service tickets820. In the example ofFIG.8, the three most similar tickets to each new customer service ticket820are presented. For example, the three most similar tickets to the new customer service ticket associated with encoding, eN+3, are tickets with index (in the customer service ticket database, and therefore, the similarity matrix, M)94,23, and60.

In at least some embodiments, the query840uses the same similarity metric s in the vectorial space as was used to determine the similarity matrix, M (again, typically, the cosine similarity), and determines which items of the database of customer service tickets are most similar to the query item.

The disclosed ticket similarity determination approach recognizes that the initial encoding is learned just from contextual information (accounting for distributional similarity). An encoding revision process refines the initial encoding given the ground truth information provided by the operators when they assign similarity between tickets. Such a revision process may be computationally expensive. Thus, the batch processing approach ofFIG.8accumulates multiple such revisions/labels in some embodiments, where the update process is only triggered sparingly (for example, weekly or monthly).

In some embodiments, the set K is collected during a time period (e.g., a work week), and then the adjustment and re-encoding processes are triggered during a scheduled downtime of the ticket support system (e.g., the weekend). In this manner, the operation of the ticket support system is not significantly affected by the computational overhead imposed by that process.

The collected new tickets may be collected with additional closing notes. If that is the case, the encoding must be recomputed (with respect to the encoding initially computed for the new tickets during the candidate scoring). It is noted that with the new encodings of the most-similar candidates could be other tickets (e.g., than those originally thought to be similar). The similarity scores can be computed between the new customer service tickets and the originally selected candidates for those tickets, as those are the ones that were “supervised” by a human operator. Thus, the similarities may need to be recomputed to compose the structure K.

FIG.9illustrates a revised similarity matrix where the similarity values have been updated to reflect the annotation information provided an operator in accordance with an illustrative embodiment. In the example ofFIG.9, a similarity matrix (M)910comprises an initial set of pairwise similarity values (e.g., prior to the updating based on operator annotations). A revised similarity matrix (M′)920may be generated, for example, by copying the pairwise similarity values from the similarity matrix (M)910. In some embodiments, the revised similarity matrix (M′)920may comprise the same data structure as the initial similarity matrix (M)910in memory, with the adjustments performed in-place (i.e., overwriting the structure). The notation of M and M′ is adopted as different structures for ease of explanation.

FIG.10illustrates an adjustment of the pairwise similarity values in a similarity matrix (M′)1010to generate an updated similarity matrix (M′)1050based on positive feedback from an operator. In the example ofFIG.10, the pairwise similarity values are shown for a representative data structure, KN+1, holding added labels, v, provided by the annotation process ofFIG.6for customer service ticket, tN+1. The user annotation indicates that two candidate tickets1020with indices10and11were labeled similarly by the operator to be similar to the new ticket tN+1. Thus, the pairwise similarity values, s(e11, e10), for these two candidate tickets1020are incremented with a positive adjustment1030, relative to the values in the initial similarity matrix (M′)1010, as shown by the dark outline surrounding the cells in the updated similarity matrix (M′)1050associated with the two candidate tickets1020to reflect that the similarity s(e10, e11) should be greater.

FIG.11illustrates an adjustment of the pairwise similarity values in a similarity matrix (M′)1110to generate an updated similarity matrix (M′)1150based on negative feedback from an operator. In the example ofFIG.11, the pairwise similarity values are shown for the representative data structure, KN+1, holding added labels, v, provided by the annotation process ofFIG.6for customer service ticket, tN+1. The user annotation indicates that two candidate tickets1120with indices2and10were labeled differently by the operator with respect to the new ticket, tN+1(e.g., the candidate ticket with index10was labeled to be similar to customer service ticket, tN+1, and the candidate ticket with index2was labeled to be different than customer service ticket, tN+1). Thus, the pairwise similarity values, s(e2, e10), for these two candidate tickets1120are decremented with a negative adjustment1130, relative to the initial similarity matrix (M′)1110as shown by the dark outline surrounding the cells in the updated similarity matrix (M′)1150associated with the two candidate tickets1120to reflect that the similarity s(e10, e11) should be smaller.

The adjustments to the pairwise similarity values can be performed for each Kj∈K (and, thus, only once if a single K structure is collected). These steps are represented inFIGS.10and11with respect to KN+1, but recall that they would be repeated also for KN+2, KN+3, . . . in the running example.

The adjustment of the pairwise similarity values in updated similarity matrix (M′)1050,1150is performed to account for the differences/similarities across preexisting tickets (in a pairwise manner) in K. The intuitive reasoning is that for a pair of candidate tickets A and B:If both candidate tickets A and B are considered (by the human operator) to be similar to a new ticket, they are likely similar to each other as well (FIG.10) so their similarity scores are increased; andIf one candidate ticket is considered similar to a new ticket, but the other is not, then the adjustment reinforces that A and B are not similar to each other (FIG.11) so their similarity scores are decreased.

Formally, an algorithm for this adjustment may be expressed, as follows:For each Kj∈K:For each Kja∈Kj:For each Kjb∈Kj, a≠b:If Kja[v]=Kjb[v]Positive adjustment of M′a,band M′b,aElse if Kja[v]≠Kjb[v]:Negative adjustment of M′a,band M′b,a

Finally, the algorithm above may be adapted to account for cases in which either of the tickets have not been provided a label. In that case, typically no adjustments are applied to the corresponding similarity values in M′. In some embodiments, the similarity may not be changed in the case in which both tickets are negatively labeled—since both being dissimilar to a new ticket does not add enough information regarding their own inter-similarity. The concrete value for the adjustment may be defined in a domain-dependent manner.

FIG.12illustrates an expansion of a similarity matrix (M′) with additional rows and columns comprising the similarity values between preexisting customer service tickets and new customer service tickets in accordance with an illustrative embodiment for a new ticket tj(j>N). In the example ofFIG.12, the similarity matrix (M′)1210is expanded to add a row and a column for a new ticket tN+1, where the cells of the expanded similarity matrix (M′)1250are shown with a highlighted outline. The similarities between the new customer service ticket tjand a given preexisting customer service ticket ti, for i≤N, must be computed using the same similarity scoring function as used in the preprocessing stage and in the candidate scoring query (again, typically, the cosine similarity).

It is noted, however, that the similarities between a new ticket tjand its candidate tickets Kja, Kjb, . . . are precomputed as part of the structure Kj. Moreover, for some of those candidates, a label Kja[v], Kjb[v], . . . may be available. Thus, the following are determined:

It is also noted that this process can be performed for every Kj∈K in a batch embodiment in which operator revisions are accumulated before the encoding revision process takes place (in a similar manner as the adjustment of preexisting similarities in M′).

To avoid explosive growth of the matrix M′ over time (as the method is continually applied), several strategies may be adopted to curb the size of the matrix M′. An active pruning process may be performed in one or more embodiments to eliminate rows/columns corresponding to tickets that satisfy one or more of the following exemplary ticket pruning criteria:tickets that are too old (e.g., based on a timestamp of creation and last update);tickets that rarely appear in any candidate-selection query; or alternatively tickets whose encoding determines a very low level of similarity to all other tickets in M′; and/oran encoding of a ticket determines a very high similarity to another ticket, and close similarity scores to the other tickets as that ticket (i.e., two nearly identical rows/columns).

It is also noted that that the parts of the pruning and reducing process may also be performed constructively, as part of the process for adding new tickets described above.

Recall that the similarity checking process requires that the input tickets are transformed into a vector space (e.g., by an encoding process) where similarity metrics can be computed more easily and efficiently. At the beginning of this process, encodings are obtained solely from contextual relationships between words present in the tickets. As seen above, however, corrections (or other feedback) provided by the user can be leveraged in situations where the initially obtained encodings do not yield acceptable similarity results related to the problem domain (i.e., ticket resolution).

To leverage such user corrections, a new modelling approach is provided in which the objective is to obtain encodings that lead to better similarity results. In the initial modelling approach, a self-supervised learning method (e.g., a Doc2Vec approach) is used. In the disclosed encoding revision model, a supervised learning approach is used.

FIG.13is a flow chart illustrating an exemplary implementation of a process for updating an encoding model based on similarity feedback obtained from at least one user in accordance with an illustrative embodiment. In the example ofFIG.13, tokens, T,1310of each historical customer service ticket (together with any corresponding similarity feedback obtained from a user) are obtained and an iteration batch of the tokens1310is generated at step1320to create a token batch, T′,1330. The token batch, T′,1330is processed at step1340to generate an encoding, E,1350of each token, T, in the current token batch, T′,1330using the current encoding model. The pairwise similarity of the token, T, in the current token batch, T′,1330are generated at step1360, to provide a batch-based similarity matrix, {circumflex over (M)},1370.

The batch-based similarity matrix, {circumflex over (M)},1370is compared at step1380to the updated similarity matrix, M′,1150(e.g., the prior version) before being updated to reflect the new similarity feedback obtained from the at least one user. In this manner, the encoding model is updated by the back propagation of a loss computed using a loss function to address any differences between the similarity matrix, M′, and the similarity matrix, {circumflex over (M)}, in accordance with a supervised learning approach.

In some embodiments, the disclosed techniques train a new encoding model that generates encodings that yield similarity matrices, {circumflex over (M)}, that iteratively approximate M′ by means of the optimization process of the training. For each training epoch (e.g., each iteration), a batch of ticket tokens of size b, T′, is obtained from the original customer service ticket database and is transformed by an encoding model into corresponding encodings, E. The encoding model may be implemented, for example, as deep neural network, in a similar manner as the Doc2Vec model. From the corresponding encodings, E, pairwise similarities sim(,)) are computed into the similarity matrix, {circumflex over (M)}, and compared with the prior similarity matrix, M′.

The comparison between the two matrices, {circumflex over (M)} and M′, is defined via a loss function, loss(ŷ, y), where ŷ is an aggregate value obtained from {circumflex over (M)}, of size b×b, for all tickets in the batch of size b, and y is an aggregate value obtained from M′ for the same tickets found in the matrix, expressed as follows:

Generally,is the average similarity value between a given customer service ticket, ti, and all other customer service tickets in the similarity matrix, {circumflex over (M)}. Similarly, yiis the average similarity value between the same customer service ticket, ti, and all other customer service tickets in M′.

It is noted, however, that, the similarity matrix, M′, is a n N×N matrix, where N is, for example, the totality of customer service tickets, and the similarity matrix, the similarity matrix, {circumflex over (M)}, is a batch of customer service tickets.FIG.14illustrates an exemplary implementation of a process for finding equivalent customer service tickets in a batch-based similarity matrix, {circumflex over (M)},1410corresponding to customer service tickets processed by the process ofFIG.13and in the larger original data of the updated similarity matrix, M′,1450, in accordance with an illustrative embodiment. In the example ofFIG.14, the process identifies the customer service tickets in the larger similarity matrix, M′,1450corresponding to the customer service tickets in the similarity matrix, {circumflex over (M)},1410for the current training batch, so that {circumflex over (M)}(i, j) is matched with M′(k, l).

As users provide more revisions to matrix M′, with additional feedback regarding the accuracy of the generated similarity values, the process ofFIG.13is repeated so that new ticket encodings are generated by the model until the similarities are adjusted to satisfy the user revisions.

In one or more embodiments, the disclosed ticket similarity determination techniques address a cold-start effect by generating initial encodings to satisfy contextual similarities, while user input is not available. Once such input is available, the disclosed techniques transition to the training approach ofFIG.13.

FIG.15is a flow chart illustrating an exemplary implementation of a process for customer service ticket similarity determination using an updated encoding model based on similarity feedback from a user in accordance with an illustrative embodiment. In the example ofFIG.15, encodings of a plurality of customer service tickets are obtained in step1502in a vector space using an encoding model, wherein the encodings of the plurality of customer service tickets are generated using a self-supervised learning algorithm.

In step1504, pairwise similarities are determined for at least a subset of the encodings of the plurality of customer service tickets. Feedback is obtained from a user in step1506regarding at least some of the pairwise similarities for the subset of the encodings. In step1508, the process ofFIG.15updates one or more of the pairwise similarities for the subset of the encodings of the plurality of customer service tickets using at least some of the feedback from the user.

An updated encoding model is generated in step1510by processing the updated pairwise similarities for the subset of the encodings of the plurality of customer service tickets using a supervised learning algorithm, and in step1512, at least one customer service ticket is processed based at least in part on the updated encoding model.

In one or more embodiments, the obtaining the encodings of the plurality of customer service tickets in the vector space in step1502further comprises obtaining tokenized versions of the plurality of customer service tickets. The updated encoding model may employ a neural network.

In at least some embodiments, the feedback from the user regarding the at least some pairwise similarities for the subset of the encodings indicates a similarity of two or more of the plurality of customer service tickets. The generating the updated encoding model in step1510may comprise, for a given training epoch of a plurality of training epochs, obtaining a batch (e.g., a random batch) of customer service tickets from the plurality of customer service tickets and transforming the batch of customer service tickets into encodings using a current encoding model. The generating the updated encoding model in step1510may also comprise (i) determining pairwise similarities for the encodings of the batch of customer service tickets; (ii) generating a first aggregate similarity value obtained from the pairwise similarities for the encodings of the batch of customer service tickets; and (iii) generating a second aggregate similarity value obtained from the pairwise similarities for the encodings of the corresponding customer service tickets in the plurality of customer service tickets. In addition, the generating the updated encoding model in step1510may further comprise evaluating a loss function using the first aggregate similarity value and the second aggregate similarity value and applying a supervised learning algorithm to fit the encoding model with respect to the loss function.

In some embodiments, the processing the at least one customer service ticket based at least in part on the updated encoding model further comprises: generating an encoding of the at least one customer service ticket using the updated encoding model; determining pairwise similarities for the encoding of the at least one customer service ticket and one or more of the subset of the encodings of the plurality of customer service tickets; and identifying one or more customer service tickets of the plurality of customer service tickets that are similar to the at least one customer service ticket based at least in part on a similarity score.

One or more closing notes regarding a disposition of the at least one customer service ticket may be encoded into the encoding of the at least one customer service ticket using the updated encoding model. The plurality of customer service tickets may be pruned based at least in part on one or more of: an age of a given customer service ticket; a frequency with which the given customer service ticket appears in a candidate selection query; one or more rules regarding the pairwise similarity of the given customer service ticket to other customer service tickets in the plurality of customer service tickets.

The particular processing operations and other functionality described in conjunction withFIGS.2,13and15, for example, are presented by way of illustrative example only, and should not be construed as limiting the scope of the disclosure in any way. Alternative embodiments can use other types of processing operations for customer service ticket similarity determination using an updated encoding model based on similarity feedback from a user. For example, the ordering of the process steps may be varied in other embodiments, or certain steps may be performed concurrently with one another rather than serially. In one aspect, the process can skip one or more of the actions. In other aspects, one or more of the actions are performed simultaneously. In some aspects, additional actions can be performed.

In this manner, in one or more embodiments, the disclosed ticket similarity determination techniques find relationships within the support ticket data so that new incidents are solved more efficiently by leveraging previous solutions. The disclosed ticket similarity determination approach does not require any particular specific set of fields, and is adaptable to any number of fields, such as message and/or note fields that are typically present in such systems.

In at least some embodiments, the disclosed ticket similarity determination methodology identifies support tickets that have already been closed with a solution by encoding the ticket description and searching for similarities in the encoded space. The most similar previous tickets can be provided, for example, in a ranked manner, for review by a human operator. The review and analysis of the human operator is then be leveraged to fine-tune the weights of the encoding process for future support issues.

The disclosed ticket similarity determination framework suggests previous tickets that are similar to a new ticket without the need for any initial supervised dataset. The framework then leverages available human revision (e.g., feedback from the user indicating whether prior results were accurate), resulting from normal operation guided by the disclosed approach, to adjust the weights of the features in the similarity metric.

One or more embodiments of the disclosure provide improved methods, apparatus and computer program products for customer service ticket similarity determination using an updated encoding model based on similarity feedback from a user. The foregoing applications and associated embodiments should be considered as illustrative only, and numerous other embodiments can be configured using the techniques disclosed herein, in a wide variety of different applications.

The disclosed ticket similarity determination techniques may be implemented using one or more processing platforms. One or more of the processing modules or other components may therefore each run on a computer, storage device or other processing platform element. A given such element may be viewed as an example of what is more generally referred to herein as a “processing device.”

As noted above, illustrative embodiments disclosed herein can provide a number of significant advantages relative to conventional arrangements. It is to be appreciated that the particular advantages described above and elsewhere herein are associated with particular illustrative embodiments and need not be present in other embodiments. Also, the particular types of information processing system features and functionality as illustrated and described herein are exemplary only, and numerous other arrangements may be used in other embodiments.

In these and other embodiments, compute services can be offered to cloud infrastructure tenants or other system users as a PaaS offering, although numerous alternative arrangements are possible.

Cloud infrastructure as disclosed herein can include cloud-based systems such as AWS, GCP and Microsoft Azure. Virtual machines provided in such systems can be used to implement at least portions of a cloud-based ticket similarity determination platform in illustrative embodiments. The cloud-based systems can include object stores such as Amazon S3, GCP Cloud Storage, and Microsoft Azure Blob Storage.

Illustrative embodiments of processing platforms will now be described in greater detail with reference toFIGS.16and17. These platforms may also be used to implement at least portions of other information processing systems in other embodiments.

FIG.16shows an example processing platform comprising cloud infrastructure1600. The cloud infrastructure1600comprises a combination of physical and virtual processing resources that may be utilized to implement at least a portion of the information processing system100. The cloud infrastructure1600comprises multiple virtual machines (VMs) and/or container sets1602-1,1602-2, . . .1602-L implemented using virtualization infrastructure1604. The virtualization infrastructure1604runs on physical infrastructure1605, and illustratively comprises one or more hypervisors and/or operating system level virtualization infrastructure. The operating system level virtualization infrastructure illustratively comprises kernel control groups of a Linux operating system or other type of operating system.

The cloud infrastructure1600further comprises sets of applications1610-1,1610-2, . . .1610-L running on respective ones of the VMs/container sets1602-1,1602-2, . . .1602-L under the control of the virtualization infrastructure1604. The VMs/container sets1602may comprise respective VMs, respective sets of one or more containers, or respective sets of one or more containers running in VMs.

In some implementations of theFIG.16embodiment, the VMs/container sets1602comprise respective VMs implemented using virtualization infrastructure1604that comprises at least one hypervisor. Such implementations can provide ticket similarity determination functionality of the type described above for one or more processes running on a given one of the VMs. For example, each of the VMs can implement ticket similarity determination control logic and associated encoding model update functionality for one or more processes running on that particular VM.

In other implementations of theFIG.16embodiment, the VMs/container sets1602comprise respective containers implemented using virtualization infrastructure1604that provides operating system level virtualization functionality, such as support for Docker containers running on bare metal hosts, or Docker containers running on VMs. The containers are illustratively implemented using respective kernel control groups of the operating system. Such implementations can provide ticket similarity determination functionality of the type described above for one or more processes running on different ones of the containers. For example, a container host device supporting multiple containers of one or more container sets can implement one or more instances of ticket similarity determination control logic and associated encoding model update functionality.

The processing platform1700in this embodiment comprises at least a portion of the given system and includes a plurality of processing devices, denoted1702-1,1702-2,1702-3, . . .1702-K, which communicate with one another over a network1704. The network1704may comprise any type of network, such as a WAN, a LAN, a satellite network, a telephone or cable network, a cellular network, a wireless network such as WiFi or WiMAX, or various portions or combinations of these and other types of networks.

The processing device1702-1in the processing platform1700comprises a processor1710coupled to a memory1712. The processor1710may comprise a microprocessor, a microcontroller, an ASIC, an FPGA or other type of processing circuitry, as well as portions or combinations of such circuitry elements, and the memory1712, which may be viewed as an example of a “processor-readable storage media” storing executable program code of one or more software programs.

Also included in the processing device1702-1is network interface circuitry1714, which is used to interface the processing device with the network1704and other system components, and may comprise conventional transceivers.

The other processing devices1702of the processing platform1700are assumed to be configured in a manner similar to that shown for processing device1702-1in the figure.

Multiple elements of an information processing system may be collectively implemented on a common processing platform of the type shown inFIG.16or17, or each such element may be implemented on a separate processing platform.

As indicated previously, components of an information processing system as disclosed herein can be implemented at least in part in the form of one or more software programs stored in memory and executed by a processor of a processing device. For example, at least portions of the functionality shown in one or more of the figures are illustratively implemented in the form of software running on one or more processing devices.