Patent Description:
When a user desires help for a technical problem, such as requesting technical support in an installation and/or maintenance of components of an infrastructure for cellular phones, the user may desire help from a group of human experts. However, the user may be requesting help at time when human expertise is not available immediately.

Accordingly, automated software agents (e.g., automated bots and/or automated expert systems) have been developed to provide help when human expertise is not available. <CIT> discloses a method for determining a best routing of a customer contact to an agent based on analysis of one or more automatically generated answers. <CIT> discloses a method where individual measures of answer quality are used to assess the quality of an answer and to generate an overall quality measure for the answer using an automatic quality scoring (AQS) algorithm.

According to claim <NUM>, there is provided a method of controlling a federation of automated agents, the federation of automated agents including a plurality of in-service automated agents. The method comprises receiving a ticket from a terminal, routing the ticket to a number of in-service automated agents among the plurality of in-service automated agents, measuring a similarity of suggestions generated by the number of in-service automated agents among the plurality of in-service automated agents, the suggestions being suggestions for responding to the ticket, determining a subset of the suggestions based on a rating of the number of in-service automated agents and the similarity of suggestions generated by the number of in-service automated agents, the rating being based on a previous score indicative of a quality of a previous subset of suggestions provided by the number of in-service automated agents, and providing the subset of suggestions to the terminal.

The method may further comprise receiving, from the terminal, a score indicative of a quality of the subset of suggestions, and updating the rating of the number of in-service automated agents based on the score.

The ticket may include a number of fields. The method may further comprise measuring a complexity of the ticket based on at least one of the fields, and determining the number of in-service automated agents based on the complexity.

The measuring a complexity of the ticket may include measuring the complexity of the ticket based on a number of words in a description field of the ticket.

The method may further include soliciting the score indicative of the quality of the subset of suggestions provided to the terminal.

The method may further include adding the ticket and a suggestion generated by at least one of the number of in-service automated agents to a test set based on the similarity of suggestions.

The method may further include removing at least one in-service automated agent from the federation of automated agents based on the rating of the at least one in-service automated agent.

The method may further include generating, by the number of in-service automated agents in the federation of automated agents, at least one suggestion for responding to the ticket.

The generating, by the number of agents in the federation of automated agents, at least one suggestion for responding to the ticket may include reviewing a description field, comparing a description field to a training set, and generating the at least one suggestion based on the comparison of the description field to the training set.

The determining a subset of suggestions based on a rating of the number of in-service automated agents and the similarity of suggestions generated by the number of in-service automated agents may include consolidating the suggestions from the number of in-service automated agents, reviewing a rating associated with the number of in-service automated agents, and adding the suggestion to the subset of suggestions based on the review of the rating.

The method may further include adding the ticket and at least one of the suggestions to a test set in response to the score indicative of the quality of the subset of suggestions being greater than a quality rating threshold.

The method may further include adding the ticket and at least one of the suggestions to a test set in response to a level of similarity of the suggestions being greater than a similarity threshold.

According to an example, a non-transitory computer readable medium may comprise program instructions for causing an apparatus to perform any of the above methods.

According to another example, there is provided a method of adding a provisional automated agent to a federation of automated agents, the federation of automated agents including a plurality of in-service automated agents, at least some of the plurality of in-service automated agents including a list of computer-readable instructions that, when executed by a computer, cause the computer to generate suggestions for responding to tickets received from one or more terminals. The method may include receiving a ticket from a terminal, forwarding the ticket to a training manager, receiving, from the training manager, a provisional suggestion generated by the provisional automated agent, evaluating the provisional automated agent by comparing the provisional suggestion to suggestions generated by a number of in-service automated agents in the federation of automated agents, and adding the provisional automated agent to the plurality of in-service automated agents in response to the evaluating.

According to a further example, a non-transitory computer readable medium may comprise program instructions for causing an apparatus to perform the above method.

According to claim <NUM>, an apparatus is provided to control a federation of automated agents, the federation of automated agents including a plurality of in-service automated agents. The apparatus includes at least one processor, and at least one memory including computer program code. The at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to receive a ticket from a terminal, route the ticket to a number of in-service automated agents among the plurality of in-service automated agents, measure a similarity of suggestions generated by the number of in-service automated agents among the plurality of in-service automated agents, the suggestions being suggestions for responding to the ticket, determine a subset of the suggestions based on a rating of the number of in-service automated agents and the similarity of suggestions generated by the number of in-service automated agents, the rating being based on a previous score indicative of a quality of a previous subset of suggestions provided by the number of in-service automated agents, and provide the subset of suggestions to the terminal.

The at least one memory and the computer program may be configured to, with the at least one memory, cause the apparatus to, receive, from the terminal, a score indicative of a quality of the subset of suggestions, and update the rating of the number of in-service automated agents based on the score.

The at least one memory and the computer program may be configured to, with the at least one processor, cause the apparatus to, solicit a score indicative of the quality of the subset of suggestions provided to the terminal.

The at least one memory and the computer program configured to, with the at least one memory, cause the apparatus to, generate, by the number of in-service automated agents in the federation of automated agents, at least one suggestion for responding to the ticket.

The at least one memory and the computer program may be configured to, with the at least one processor, cause the apparatus to, add the ticket and at least one of the suggestions to a test set in response to the score indicative of the quality of the subset of suggestions being greater than a quality rating threshold.

According to an example, there is provided an apparatus for controlling a federation of automated agents, the federation of automated agents including a plurality of in-service automated agents, at least some of the plurality of in-service automated agents including computer-readable instructions that, when executed by a computer, cause the computer to generate suggestions for responding to tickets received from one or more terminals, the apparatus comprising means receiving a ticket from a terminal, means for routing the ticket to a number of in-service automated agents among the plurality of in-service automated agents, means for measuring a similarity of suggestions generated by the number of in-service automated agents among the plurality of in-service automated agents, the suggestions being suggestions for responding to the ticket, means for determining a subset of the suggestions based on a rating of the number of in-service automated agents and the similarity of suggestions generated by the number of in-service automated agents, the rating being based on a previous score indicative of a quality of a previous subset of suggestions, and means for providing the subset of suggestions to the terminal.

According to another example, there is provided an apparatus for adding a provisional automated agent to a federation of automated agents, the federation of automated agents including a plurality of in-service automated agents, at least some of the plurality of in-service automated agents including a list of computer-readable instructions that, when executed by a computer, cause the computer to generate suggestions for responding to tickets received from one or more terminals, the apparatus comprising means for receiving a ticket from a terminal, means for forwarding the ticket to a training manager, means for receiving, from the training manager, a provisional suggestion generated by the provisional automated agent, means for evaluating the provisional automated agent by comparing the provisional suggestion to suggestions generated by a number of in-service automated agents in the federation of automated agents, and means for adding the provisional automated agent to the plurality of in-service automated agents in response to the evaluating.

These and other aspects of example embodiments will become clear in the figures and the detailed description therein.

Some example embodiments will become more fully understood from the detailed description given herein below and the accompanying drawings, wherein like elements are represented by like reference numerals, which are given by way of illustration only and thus are not limiting of this disclosure.

Detailed illustrative embodiments are disclosed herein. However, specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments.

Accordingly, while example embodiments are capable of various modifications and alternative forms, the embodiments are shown by way of example in the drawings and will be described herein in detail. refer to like elements throughout the description of the figures.

<FIG> illustrates an environment in which example embodiments may be implemented.

Referring to <FIG>, an environment <NUM> may include a federation of software agents <NUM> and a device <NUM> having a user interface <NUM>. The user interface 150may be accessed by a user <NUM>.

The federation of software agents <NUM> may include hardware and/or software running on a single computer, or, alternatively, running on a collection of computers (not shown). The collection of computers may be distributed, for example distributed over a network of computers.

The collection of computers may include a plurality of automated agents <NUM>, <NUM>, <NUM>. At least some of the plurality of automated agents <NUM>, <NUM>, <NUM> may be implemented via computer-readable instructions that, when executed by the computer, cause the computer to provide suggestions for responding to a ticket. For example, at least some of the plurality of automated agents <NUM>, <NUM>, <NUM> may be implemented as software for executing a neural network and/or a machine-learning algorithm and/or an expert system. At least some of the automated agents <NUM>, <NUM>, <NUM> may have a corresponding rating indicative of a quality of suggestions offered by the agents.

At least some of the plurality of automated agents <NUM>, <NUM>, <NUM> may be trained, in a training phase, to generate responses to outstanding tickets. The training of the automated agents may be done differently for different automated agents. For example, training of automated agent <NUM> may include training of a neural network and/or a machine-learning algorithm using a first training set, containing first example tickets and corresponding first suggestions for responding to the first example tickets. Training of automated agent <NUM> may include training of a neural network and/or a machine-learning algorithm using a second training set, containing second example tickets and corresponding second suggestions for responding to the second example tickets.

The federation of software agents <NUM> may include a service manager <NUM>. The service manager <NUM> may be implemented via computer-readable instructions that, when executed by the computer, cause the computer to perform one or more methods according to example embodiments. For example, the service manager <NUM> may receive a ticket <NUM> through the device <NUM> displaying the user interface <NUM>, determine a complexity of the ticket <NUM>, and choose a subset of the plurality of automated agents <NUM>, <NUM>, <NUM> for providing suggestions through the user interface <NUM>. The service manager <NUM> may also request feedback from the user <NUM> through the user interface <NUM>, including a rating indicative of the quality of the suggestions.

The federation of software agents <NUM> may also include a training manager <NUM>. The training manager <NUM> may manage a number of provisional agents (not shown) that are being trained. The training manager <NUM> will be discussed further with reference to <FIG>.

The components of the federation of software agents <NUM> may communicate with each other. For example, each of, or at least some of, the plurality of software agents <NUM>, <NUM>, <NUM> may communicate with the service manager <NUM>. For example, the components of the federation of software agents <NUM> may communicate over a network with one another.

The user interface <NUM> may be or include a graphical user interface (GUI), for example a graphical user interface presented on a website and/or within an app, such as a mobile app. If the user <NUM> desires suggestions for addressing a technical problem, then the user <NUM> may engage with the user interface <NUM> to submit a ticket <NUM>. The user interface <NUM> will be described in more detail later with reference to <FIG>.

The user <NUM> may be a user of a system and/or a technology. As an example, the user <NUM> may be a technician responsible for the maintenance of equipment associated with a mobile network. As an example, the user <NUM> may be a user of a software, such as an operating system and/or word processing system and/or application used in an office environment. As another example, the user <NUM> may be a user of a computer game or computer gaming system. As yet another example, the user <NUM> may be a user of a system associated with the delivery of healthcare. Example embodiments are not limited to those above, and one of ordinary skill in the art may readily recognize other example embodiments. The system may be operated based on the suggestions provided by the software agents.

As an example, the user <NUM> may request support for a new customer fault, and request suggestions for responding to the new customer fault.

<FIG> illustrates a method of controlling a federation of software agents according to some example embodiments.

Referring to <FIG>, in step <NUM>, the user <NUM> may request a suggestion for support in resolving a technical issue by submitting a ticket <NUM> to the federation of automated agents <NUM> via the device <NUM>. The user <NUM> may use the user interface <NUM> to submit the ticket <NUM> to the service manager <NUM> in the federation of automated agents <NUM>.

In step <NUM>, the service manager <NUM> may determine a complexity of the ticket <NUM>. For example, the service manager <NUM> may count a number of characters and/or a number of words within the description field <NUM>, and may determine the complexity of the ticket based on the number of words.

The federation of automated agents <NUM> may determine that the ticket <NUM> is simple or complicated, depending on the fields. Example embodiments are not limited to a review of the description field <NUM>, and may include a review of other fields within the ticket <NUM>.

In step <NUM>, the service manager <NUM> may route the ticket <NUM> to a number of automated agents. The number of automated agents may be determined based on the complexity of the ticket <NUM>. Example embodiments of a method for routing the ticket <NUM> will be described in more detail below with reference to <FIG>.

In step <NUM>, the automated agents which received the ticket <NUM> may generate one or more suggestions for responding to the ticket <NUM>. Example methods for the generation of one or more suggestions will be described in more detail later with reference to <FIG>.

In step <NUM>, the service manager <NUM> may collate the suggestions from the automated agents, and may determine an amount of similarity by measuring an amount of overlap in the collated suggestions. For example, if the service manager <NUM> routes the ticket <NUM> to each of the automated agents <NUM>, <NUM>, <NUM>, and each of the automated agents <NUM>, <NUM>, <NUM> provides the same suggestion for responding to the ticket <NUM>, then the amount of overlap may be relatively high. However, if each of the automated agents <NUM>, <NUM>, <NUM> provides different suggestions for responding to the ticket <NUM>, then the amount of overlap may be relatively low.

In step <NUM>, the service manager <NUM> may compare the amount of similarity to an upper similarity threshold. The upper similarity threshold may be determined, e.g., determined by an operator (e.g., a human or a network), and may be based on empirical data.

For example, an amount of overlap may be equal to a ratio between a number of suggestions generated by each of the in-service automated agents <NUM>, <NUM>, <NUM> that are the same, and a total number of in-service automated agents <NUM>, <NUM>, <NUM> that generated a suggestion. For example, if each of the in-service automated agents <NUM>, <NUM>, <NUM> generated the same suggestion for responding to the ticket <NUM>, then the overlap may be <NUM>%. Alternatively, if each of the in-service automated agents <NUM>, <NUM>, <NUM> generated different suggestions for responding to the ticket <NUM>, then the overlap may be <NUM>%.

For example, the upper similarity threshold may correspond to an <NUM>% overlap.

If the amount of overlap exceeds the upper similarity threshold, both the ticket <NUM> and the suggestions generated by the automated agents may be added to a test set in step <NUM>. The test set may be used in the training of provisional agents. Example methods for the training of provisional agents will be described in more detail later with reference to <FIG>.

Returning to step <NUM>, if the amount of overlap does not exceed the upper similarity threshold, then in step <NUM>, the service manager <NUM> compares the overlap to a lower similarity threshold. The lower similarity threshold may be less than the upper similarity threshold. The lower similarity threshold may be defined by an operator (e.g., human or network) based on data.

For example, the lower similarity threshold may correspond to a <NUM>% overlap in the same suggestions generated by the agents within the federation of automated agents <NUM>.

If the overlap is less than the lower similarity threshold, then the ticket <NUM> may be routed to a human expert for further review in step <NUM>.

Returning to step <NUM>, if the overlap is not less than the lower similarity threshold, then the method proceeds to step <NUM>.

In step <NUM>, the service manager <NUM> may select a subset of suggestions to present to the user <NUM> through the user interface <NUM>. For example, the service manager <NUM> may select the subset of suggestions based on a rating of the automated agents, and/or the amount of overlap among the suggestions. Example methods for the determination of the subset of suggestions will be described in more detail later with respect <FIG>.

In step <NUM>, the service manager <NUM> may provide the subset of suggestions to the user <NUM> (e.g., through the user interface <NUM>). Example methods for the providing of the subset of suggestions will be described in more detail later with respect to <FIG>.

In step <NUM>, the service manager <NUM> may solicit feedback from the user <NUM> through the user interface <NUM>. For example, the service manager <NUM> may send a message to the user <NUM> through the user interface <NUM> and/or through some other method of communication. The message may be or include a request for feedback on the quality of suggestions offered to the user <NUM> described above with reference to step <NUM>.

In step <NUM>, the service manager <NUM> may receive feedback from the user <NUM> through the user interface <NUM>. The feedback may indicate a level of satisfaction of each of, or some of, the suggestions provided in step <NUM>. The user <NUM> may indicate that he or she was very or extremely satisfied, somewhat satisfied, neutral, somewhat dissatisfied, or not satisfied at all, to each of, or at least some of, the suggestions provided in step <NUM>.

In step <NUM>, the service manager <NUM> may determine whether the response was helpful, based on the feedback. For example, the user <NUM> may provide a numerical value indicative of a level of satisfaction with the suggestions provided in step <NUM>. For example, the user <NUM> may indicate that he or she is satisfied at a certain level on a ten-point scale. The numerical value may be compared with a feedback threshold. The feedback threshold may be defined by an operator (e.g., human or network) based on empirical data.

For example, the feedback threshold may correspond to a numerical value of a level of satisfaction being <NUM> out of <NUM> or higher.

If the numerical value associated with the feedback exceeds the feedback threshold, then in step <NUM>, the ticket <NUM> and the suggestions generated by the automated agents that were rated useful by the user <NUM> may be added to a test set.

In step <NUM>, the service manager <NUM> may update the rating of the in-service automated agents that provided a suggestion, based on the feedback received from the user <NUM> in step <NUM>. Furthermore, the membership of in-service automated agents in the federation of automated agents <NUM> may be adjusted based on the rating. For example, an automated agent that has been underperforming may be retired from the federation of automated agents <NUM>.

Further, a provisional automated agent that has provided provisional suggestions may be included in the federation of automated agents <NUM>. Example methods for adjusting the federation of automated agents <NUM> will be described later with reference to <FIG>.

Returning to step <NUM>, if the numerical value associated with the feedback is less than or equal to the feedback threshold, then the process proceeds to step <NUM>, and continues as discussed above.

Thus, based on the method outlined in <FIG>, more complicated tickets may have more useful suggestions provided. Further, by measuring the level of similarity in the suggestions, new automated software agents may be added to the federation of software agents, without extensive training (e.g., training with human input). Further, a decision may be made to retire a software agent that provides outlier and/or inconsistent suggestions. This may help simplify the federation of automated agents <NUM>. Furthermore, by measuring a level of similarity, a determination may be made to forward the ticket to request help of a human expert, thus improving the quality of suggestions provided to the user.

<FIG> illustrates an example of a user interface according to some example embodiments.

Referring to <FIG>, the user interface <NUM> may be presented as a graphical user interface (GUI). The GUI may include a number of fields that may be filled in by a user, for example the user <NUM>. For example, the user interface <NUM> may include a title field <NUM>, a button field <NUM>, a drop-down list field <NUM>, and/or a description field <NUM>.

The user <NUM> may identify a title to the ticket <NUM> to be submitted, and enter this into the title field <NUM>. Further, the user <NUM> may check a box in the button field <NUM> and/or choose an item from a drop-down list field <NUM>. Still further, the user <NUM> may provide a more detailed description of the problem in the description field <NUM>. The description field <NUM> may allow for a user to draft, in free form, a description of the problem for which he or she requests support. The description field <NUM> may include natural language.

Example embodiments are not limited thereto. The design of the user interface <NUM> illustrated in <FIG> is not limited thereto, and more or fewer fields may be presented to a user for request.

The user interface <NUM> may be presented as a website on the device <NUM>, and/or may be presented on a mobile app of a mobile device. Example embodiments are not limited to these examples.

<FIG> illustrates a method of routing a ticket to a number of automated agents, according to some example embodiments. The method may be performed by the service manager <NUM>.

Referring to <FIG>, in step <NUM>, the service manager <NUM> may decide if the complexity of the ticket <NUM> is high, for example higher than a ticket complexity threshold. For example, if there are a relatively large number of words, e.g. between five and fifty words, in the description field <NUM>, then the service manager <NUM> may determine that the ticket <NUM> is complicated. Alternatively, if there are a relatively small number of words (e.g., less than five words) in the description field <NUM>, then the service manager <NUM> may determine that the ticket <NUM> is simple.

If the complexity of the ticket <NUM> exceeds the ticket complexity threshold, then in step <NUM> the service manager <NUM> may route (e.g., forward) the ticket <NUM> to a large number of in-service automated agents. For example, if the ticket <NUM> has a complexity exceeding the ticket complexity threshold, the federation of automated agents <NUM> may route the ticket <NUM> to three, four, or five in-service automated agents. For example, the service manager <NUM> may route the ticket to three, four, or five random in-service automated agents.

Returning to step <NUM>, if the complexity of the ticket <NUM> does not exceed the ticket complexity threshold, then in step <NUM> the service manager <NUM> may route (e.g., forward) the ticket <NUM> to a smaller number of in-service automated agents. For example, if the ticket <NUM> has a complexity not exceeding the ticket complexity threshold, the service manager <NUM> may route the ticket to one or two in-service automated agents. For example, the service manager <NUM> may route the ticket to one or two random in-service automated agents.

<FIG> illustrates a method of generating a suggestion for responding to a ticket, according to some example embodiments.

For example purposes, the method of <FIG> will be discussed with respect to automated agent <NUM>. However, each automated agent may perform the method to generate one or more suggestions for responding to a ticket.

Referring to <FIG>, at step <NUM> the automated agent <NUM> may receive the ticket <NUM>.

In step <NUM>, the automated agent <NUM> may review at least some of the fields included in the ticket <NUM>. For example, the automated agent <NUM> may parse the words associated with the description field <NUM> to determine which issue is to be addressed. For example, the automated agent <NUM> may parse the words associated with the description field <NUM> to determine key words associated with the ticket <NUM>.

In step <NUM>, the automated agent <NUM> may compare the fields included in the ticket <NUM> to a training set used to train the automated agent <NUM>. The training set may have been generated from historical data, and further may be augmented during a process described in more detail later with reference to <FIG>.

Based on a comparison, the automated agent <NUM> may generate one or more suggestions for responding to the ticket <NUM>. For example, if the automated agent <NUM> is embodied as a neural network trained to review the free form content of the description field <NUM>, then the automated agent <NUM> may generate a suggestion based on the features of the trained neural network.

For example, the automated agent may generate a suggestion based on a k-nearest neighbor algorithm and/or an algorithm to determine the joint complexity between the ticket <NUM> and the training set. The automated agent <NUM> may generate the suggestion corresponding to suggestions having the nearest neighbor and/or the highest joint complexity between the ticket <NUM> and the training set.

The suggestion may be a suggestion chosen from a list of suggestions. For example, the list of suggestions may have been previously generated. For example, the list of suggestions may include discussions in a reference manual. The list of suggestions may be stored in a database and/or a data structure.

In step <NUM>, the automated agent <NUM> may provide (e.g., transmit) the one or more suggestions to the service manager <NUM>.

<FIG> illustrates a method of generating a subset of suggestions for responding to a ticket, according to some example embodiments.

Referring to <FIG>, in step <NUM>, the service manager <NUM> may review a rating (e.g., a quality rating) associated with the automated agents that generated a response.

For each automated agent, if the quality rating of the automated agent exceeds a quality rating threshold, then in step <NUM> the service manager <NUM> may add the suggestion to a subset of responses to be presented to the user through the user interface. The quality rating threshold may be defined by an operator (e.g., human or network) based on empirical data.

For example, the quality rating threshold may have a numerical value corresponding to an average level of satisfaction being <NUM> on a ten-point scale, based on user feedback.

<FIG> illustrates a method of providing a subset of suggestions to respond to a ticket, according to some example embodiments.

The service manager <NUM> may prepare the suggestions to present to the user <NUM> through the user interface <NUM>.

Referring to <FIG>, in step <NUM> the service manager <NUM> may collate and sort the suggestions based on a likelihood of usefulness to format the suggestions to send to the user <NUM>. The suggestions may be sorted based on an amount of overlap. For example, if a first suggestion has been generated by three of five automated agents (e.g., automated agents <NUM>, <NUM>, <NUM>), then the first suggestion may be ordered first. If a second suggestion has been generated by only two of the five automated agents (e.g., automated agents <NUM>, <NUM>), then the second suggestion may be ordered second.

In step <NUM>, the service manager <NUM> may provide the subset of responses to the user <NUM>. The subset of responses may be provided through the user interface <NUM>, and/or through an e-mail, and/or through a text message, and/or through some other method of communicating with the user <NUM>. Example embodiments are not limited to these examples.

<FIG> illustrates a method of adjusting a rating of an in-service automated agent, according to some example embodiments.

At least some of the automated agents <NUM>, <NUM>, <NUM> may have a corresponding rating. In one example, the corresponding rating may be a corresponding quality rating (e.g., a numerical value associated with the quality of suggestions provided by the automated agents <NUM>, <NUM>, <NUM>). The rating may be adjusted based on feedback from the user <NUM>.

For example, in step <NUM> the service manager <NUM> may receive feedback from the user <NUM>.

In step <NUM>, the service manager <NUM> may determine whether the user <NUM> was satisfied with the suggestion.

If the user <NUM> indicated that he or she was not satisfied with the suggestion, for example by giving a low numerical value to the suggestion, then at step <NUM> the rating of automated agents that provided the suggestion for which the user <NUM> was not satisfied may be reduced.

In step <NUM>, the service manager <NUM> may compare the numerical value of the rating of each automated agents <NUM>, <NUM>, <NUM> to a delisting threshold. The delisting threshold may be defined by an operator (e.g., human or network) based on empirical data.

For example, the delisting threshold may correspond to a rating threshold of two out of ten, on a ten point scale.

Any of the automated agents <NUM>, <NUM>, <NUM> with a rating less than the delisting threshold may be removed from and/or retired from the federation of automated agents <NUM> in step <NUM>. Thus, the federation of automated agents <NUM> may be reduced and/or simplified.

Returning to step <NUM>, if the user <NUM> was satisfied with the suggestion, then in step <NUM> the service manager <NUM> may increase the rating of the in-service automated agents that provided the suggestions with which the user <NUM> was satisfied. Still further, the suggestion may be added to the test set, as described with reference to step <NUM>.

<FIG> illustrates a method of introducing a new software agent into the federation of software agents, according to some example embodiments.

The federation of automated agents <NUM> may be populated with in-service automated agents. An in-service automated agent may be trained in the generation of suggestions for responding to tickets such as ticket <NUM>. For example, the in-service automated agent may be embodied as a neural network and/or a machine learning algorithm and/or an expert system, and may be trained in the generation of suggestions based on a test set.

However, before a new automated agent is added to an established federation of automated agents <NUM>, the new automated agent may be treated as a provisional agent.

Referring to <FIG>, there may be the training manager <NUM> that manages a number of provisional agents (not shown) that are being trained.

Initially, in step <NUM>, the user <NUM> may submit the ticket <NUM> through the user interface <NUM>. The federation of automated agents <NUM> may receive the ticket.

In step <NUM>, the federation of automated agents <NUM> may produce one or more suggestions for responding to the ticket <NUM>. The federation of automated agents <NUM> may produce one or more suggestions according to the methods outlined with reference to <FIG>.

In step <NUM>, the service manager <NUM> may provide the suggestions through the user interface <NUM> to the user <NUM>. Furthermore, in step <NUM>, the federation of automated agents <NUM> may forward the ticket <NUM> to the training manager <NUM>.

In step <NUM>, the training manager <NUM> may provide the ticket <NUM> to a number of provisional automated agents. The training manager <NUM> may forward a response from at least one of the provisional automated agents to the service manager <NUM> in the federation of automated agents <NUM> in step <NUM>. The ticket <NUM> provided to the provisional agents may correspond to a ticket <NUM> marked as training data in step <NUM>.

In step <NUM>, the service manager <NUM> may review and evaluate the response generated by the provisional agent(s). For example, the service manager <NUM> may compare and determine a model precision and recall score of the response(s) generated by the provisional agent.

For example, the service manager <NUM> may determine a precision score corresponding to a fraction of the suggestions provided by the provisional agent(s) that are the same as the suggestions generated by the plurality of in-service automated agents <NUM>, <NUM>, <NUM>. The service manager <NUM> may determine a recall score corresponding to a fraction of the suggestions generated by the plurality of in-service automated agents <NUM>, <NUM>, <NUM> that are the same as the suggestions generated by the provisional agent(s).

The service manager <NUM> may determine the model precision and recall score, which relates the precision score to the recall score, to determine a measurement of the quality of the suggestions from the provisional agent(s). The model precision and recall score may be twice the ratio of the product of the precision score and recall score divided by the sum of the precision and recall score. Twice the ratio of the product of the precision score and recall score divided by the sum of the precision and recall score may also be called the F1 score. The model precision and recall score may be expressed as a ratio, and/or a fraction, and/or a percentage.

If the model precision and recall score is greater than or equal to an entrance threshold, then the provisional agent is accepted and deployed within the federation of automated agents in step <NUM>.

The entrance threshold may be defined by an operator (e.g., human or network) based on empirical data. For example, the entrance threshold may correspond to an F1 score of <NUM>%.

If the model precision and recall score is less than the entrance threshold, the provisional agent is not accepted and is not deployed within the federation of automated agents in step <NUM>.

<FIG> illustrates a device for implementing one or more of the service manager <NUM>, one or more of the automated agents <NUM>, <NUM>, <NUM>, one or more of the training manager <NUM>, etc., according to some example embodiments.

As shown, a device <NUM> may include a memory <NUM>; a processor <NUM> connected to the memory <NUM>; various interfaces <NUM> connected to the processor <NUM>; and one or more connections <NUM> connected to the various interfaces <NUM>.

The memory <NUM> may be a computer readable storage medium that generally includes a random access memory (RAM), read only memory (ROM), and/or a permanent mass storage device, such as a disk drive. The memory <NUM> also stores an operating system and any other routines/modules/applications for providing the functionalities of the device <NUM> to be executed by the processor <NUM>. These software components may also be loaded from a separate computer readable storage medium into the memory <NUM> using a drive mechanism (not shown). Such separate computer readable storage medium may include a disc, tape, DVD/CD-ROM drive, memory card, or other like computer readable storage medium (not shown). In some example embodiments, software components may be loaded into the memory <NUM> via one of the various interfaces <NUM>, rather than via a computer readable storage medium.

The processor <NUM> may be configured to carry out instructions of a computer program by performing the arithmetical, logical, and input/output operations of the system. Instructions may be provided to the processor <NUM> by the memory <NUM>.

For example, the processor <NUM> may be configured to carry out instructions corresponding to any of the methods described above with reference to <FIG> and/or <NUM>-<NUM>.

The various interfaces <NUM> may include components that interface the processor <NUM> with other input/output components. As will be understood, the various interfaces <NUM> and programs stored in the memory <NUM> to set forth the special purpose functionalities of the device <NUM> will vary depending on the implementation of the device <NUM>.

The interfaces <NUM> may also include one or more user input devices (e.g., a keyboard, a keypad, a mouse, a touch-screen, and/or the like) and user output devices (e.g., a display, a speaker, a touch-screen, and/or the like).

One or more example embodiments provide mechanisms for determining when to induct (e.g. introduce) newly trained automated software agents into the federation of software agents. One or more example embodiments provide mechanisms for developing example test sets of tickets and corresponding suggestions. Furthermore, one or more example embodiments may provide mechanisms for determining when to involve more than one software agent in the generation of suggestions. Furthermore, one or more example embodiments provide mechanisms for determining when to deregister/retire underperforming software agents from the federation may be desirable. Still further, one or more example embodiments provide mechanisms for determining when a ticket should be routed to an expert human for review.

As discussed herein, illustrative embodiments are described with reference to acts and symbolic representations of operations (e.g., in the form of flow charts, flow diagrams, data flow diagrams, structure diagrams, block diagrams, etc.) that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types and may be implemented using existing hardware at, for example, existing network management devices, network management entities, clients, gateways, nodes, agents, controllers, computers, cloud based servers, web servers, proxies or proxy servers, application servers, load balancers or load balancing servers, device management servers, or the like. As discussed later, such existing hardware may include, inter alia, one or more Central Processing Units (CPUs), system-on-chip (SOC) devices, digital signal processors (DSPs), application-specific-integrated-circuits, field programmable gate arrays (FPGAs) computers or the like.

As disclosed herein, the term "storage medium", "computer readable storage medium" or "non-transitory computer readable storage medium" may represent one or more devices for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and/or other tangible machine readable mediums for storing information. The term "computer-readable medium" may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other mediums capable of storing, containing or carrying instruction(s) and/or data.

Furthermore, example embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine or computer readable medium such as a computer readable storage medium. When implemented in software, a processor or processors will perform the necessary tasks.

A code segment may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc..

The terms "including" and/or "having", as used herein, are defined as comprising (i.e., open language). The term "coupled", as used herein, is defined as connected, although not necessarily directly, and not necessarily mechanically. Terminology derived from the word "indicating" (e.g., "indicates" and "indication") is intended to encompass all the various techniques available for communicating or referencing the object/information being indicated. Some, but not all, examples of techniques available for communicating or referencing the object/information being indicated include the conveyance of the object/information being indicated, the conveyance of an identifier of the object/information being indicated, the conveyance of information used to generate the object/information being indicated, the conveyance of some part or portion of the object/information being indicated, the conveyance of some derivation of the object/information being indicated, and the conveyance of some symbol representing the object/information being indicated.

According to example embodiments, network management devices, network management entities, clients, gateways, nodes, agents controllers, computers, cloud based servers, web servers, application servers, proxies or proxy servers, load balancers or load balancing servers, device management servers, or the like, may be (or include) hardware, firmware, hardware executing software or any combination thereof. Such hardware may include one or more Central Processing Units (CPUs), system-on-chip (SOC) devices, digital signal processors (DSPs), application-specific-integrated-circuits (ASICs), field programmable gate arrays (FPGAs) computers or the like configured as special purpose machines to perform the functions described herein as well as any other well-known functions of these elements. In at least some cases, CPUs, SOCs, DSPs, ASICs and FPGAs may generally be referred to as processing circuits, processors and/or microprocessors.

Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments of the invention. However, the benefits, advantages, solutions to problems, and any element(s) that may cause or result in such benefits, advantages, or solutions, or cause such benefits, advantages, or solutions to become more pronounced are not to be construed as a critical, required, or essential feature or element of any or all the claims.

Claim 1:
A method of controlling a federation of automated agents, the federation of automated agents including a plurality of in-service automated agents, the method comprising:
receiving a ticket from a terminal;
routing the ticket to a number of in-service automated agents among the plurality of in-service automated agents;
measuring a similarity of suggestions generated by the number of in-service automated agents among the plurality of in-service automated agents, the suggestions being suggestions for responding to the ticket;
determining a subset of the suggestions based on a rating of the number of in-service automated agents and the similarity of suggestions generated by the number of in-service automated agents, the rating being based on a previous score indicative of a quality of a previous subset of suggestions provided by the number of in-service automated agents; and
providing the subset of suggestions to the terminal.