Training artificial intelligence to respond to user utterances

A method improves a functionality of a conversational agent that is generated by an artificial intelligence (AI) system. A conversational agent receives a first utterance from a first entity. The AI system identifies an unverified response to the first utterance; sends the unverified response to the first entity; and receives a positive feedback indication about the unverified response from the first entity. The AI system searches a data store in order to identify an entry for a second utterance by a second entity, where the second entity has sent a positive feedback for the unverified response. The AI system sends the second utterance and the unverified response to the first entity, and receives a positive feedback for the unverified response to the second utterance from the first entity in order to mark the unverified response as a verified response, which responds to future receipts of the first utterance.

BACKGROUND

The present invention relates to the field of artificial intelligence. Still more specifically, the present invention relates to the field of training and/or utilizing artificial intelligence based on user feedback to responses to user utterances.

SUMMARY

In an embodiment of the present invention, a method improves a functionality of a conversational agent that is generated by an artificial intelligence (AI) system. A conversational agent receives a first utterance from a first entity. The AI system identifies an unverified response to the first utterance; sends the unverified response to the first entity; and receives a positive feedback indication about the unverified response from the first entity. The AI system searches a data store in order to identify an entry for a second utterance by a second entity, where the first utterance and the second utterance are textually different from one another, where the unverified response has been sent to the second entity in response to the second utterance, and where the second entity has sent a positive feedback for the unverified response. The AI system sends the second utterance and the unverified response to the first entity, and receives a positive feedback for the unverified response to the second utterance from the first entity. In response to receiving the positive feedback for the unverified response to the second utterance from the first entity, the AI system marks the unverified response as a verified response. The AI system then receives the first utterance from a third entity, and responds to the first utterance from the third entity with the verified response.

In an embodiment of the present invention, the processor(s) activate, modify, and/or otherwise affect a hardware device based on the event that is identified by the AI system.

In one or more embodiments, the method(s) described herein are performed by an execution of a computer program product and/or a computer system.

DETAILED DESCRIPTION

In one or more embodiments, the present invention is a system, a method, and/or a computer program product at any possible technical detail level of integration. In one or more embodiments, the computer program product includes a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams represents a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block occur out of the order noted in the figures. For example, two blocks shown in succession are, in fact, executed substantially concurrently, or the blocks are sometimes executed in the reverse order, depending upon the functionality involved. It will also be noted that, in one or more embodiments of the present invention, each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, are implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

With reference now to the figures, and in particular toFIG. 1, there is depicted a block diagram of an exemplary system and network that may be utilized by and/or in the implementation of the present invention. Note that some or all of the exemplary architecture, including both depicted hardware and software, shown for and within computer102may be utilized by software deploying server150and/or user computer(s)152and/or remote device154and/or neural network124.

Exemplary computer102includes a processor104that is coupled to a system bus106. Processor104may utilize one or more processors, each of which has one or more processor cores. A video adapter108, which drives/supports a display110, is also coupled to system bus106. System bus106is coupled via a bus bridge112to an input/output (I/O) bus114. An I/O interface116is coupled to I/O bus114. I/O interface116affords communication with various I/O devices, including a keyboard118, a mouse120, a media tray122(which may include storage devices such as CD-ROM drives, multi-media interfaces, etc.), a neural network124, and external USB port(s)126. While the format of the ports connected to I/O interface116may be any known to those skilled in the art of computer architecture, in one embodiment some or all of these ports are universal serial bus (USB) ports.

As depicted, computer102is able to communicate with a software deploying server150and/or the user computer(s)152and/or the remote device154and/or the neural network124using a network interface130to a network128. Network interface130is a hardware network interface, such as a network interface card (NIC), etc. Network128may be an external network such as the Internet, or an internal network such as an Ethernet or a virtual private network (VPN).

Application programs144in computer102's system memory (as well as software deploying server150's system memory) also include an Artificial Intelligence Conversational Training and Utilization Logic (AICTUL)148. AICTUL148includes code for implementing the processes described below, including those described inFIGS. 2-5. In one embodiment, computer102is able to download AICTUL148from software deploying server150, including in an on-demand basis, wherein the code in AICTUL148is not downloaded until needed for execution. Note further that, in one embodiment of the present invention, software deploying server150performs all of the functions associated with the present invention (including execution of AICTUL148), thus freeing computer102from having to use its own internal computing resources to execute AICTUL148.

Also connected to (or alternatively, as part of) computer102is a neural network124. In exemplary embodiments of the present invention, neural network124is a non-convolutional neural network (seeFIG. 4), a convolutional neural network, or another type of heuristic artificial intelligence.

Also connected to computer102are user computer(s)152, which are used by one or more users who are in an electronic communication session with an artificial intelligence (AI) based conversational agent, as described herein.

Also commented to computer102is a remote device154. One or more examples of remote device154are presented inFIG. 3.

An artificial intelligence (AI) conversation is a conversation between a person and a computer-generated responder. For example, a chat bot is an AI logic that engages in a text conversation with a person, such as a customer. Thus, if the customer texts a question to a service such as “What time is it in Paris?”, the chat bot would respond with “It is currently 14:37 in Paris.”

In order to answer this question, in one or more embodiments of the present invention the AI identifies what the question (“utterance”) is asking by identifying key words/phrases and/or context in the question.

Context is derived explicitly from the utterance and/or answer, implicitly from the utterance and/or answer, and/or from another source.

For example, assume that the utterance is “How do I get to Store A today?” The term “today” gives a temporal context to the question of “How do I get to Store A?” by giving the time context (“today”) of when the person needs to get to the location. Thus, the (temporal) context is derived explicitly from the term “today” in the utterance “How do I get to Store A today?”

Assume now that the utterance is “How do I get to Store A before it closes?”. The term “before it closes” implicitly gives the context that the utterer is asking for a means (e.g., walking route, driving route, public transportation, etc.) for getting to the location of Store A before 5:00 pm, which is implicitly derived from the phrase “before it closes”. That is, the system correlates the term “before it closes” with information known by a conversational agent (e.g., using a lookup table, a browser, etc.) that Store A closes at 5:00 pm. Using this implicit context, the conversational agent provides information that will enable the utterer to arrive at Store A before 5:00 pm. Thus, the (temporal) context is derived implicitly from the term “before it closes” in the utterance “How do I get to Store A before it closes?”

Assume now that the utterance is “How do I get there?”. In this example, nothing in the utterance “How do I get there?” explicitly or implicitly give the context of what “there” is. In this example, the conversational agent receives information from another source to identify what “there” is (e.g., the location of Store A). In an embodiment of the present invention, the conversational agent detects that the utterer is using a chat portal from a website for Store A. As such, the conversational agent determines from this other information source that the utterer is looking for directions to Store A.

In accordance with one or more embodiments of the present invention, “intent” is defined as the purpose of the communication from the user, and preferably is identified by the system and then tagged with a hash tag (#). Examples of tagged intent include, but are not limited to, # location (asking for directions to a facility), # order_food (ordering food for delivery or take-out), # turn_on_device (directing the system to turn on a light, an appliance, etc.), etc.

In order to determine the intent of the communication, the system identifies one or more “entities” in the communication. In a preferred embodiment, an “entity” is a key word and/or phrase in the communication that leads the system to understanding what the intent of the communication is, and preferably are identified and tagged by the system with an “at” symbol (@). Examples of tagged entities include, but are not limited to, @city (which can lead to the identification of the intent # location), @pizza (which can lead to the identification of the intent # order_food), @dark (which can lead to the identification of the intent # turn_on_device), etc.

The communication being evaluated can be a question or a comment. Therefore, such a communication is referred to as an “utterance”. For example, the utterance can be a question (e.g., “How do I get to your location?”) that expressly describes the need of the utterer. Alternatively, the utterance can be a statement (e.g., “I wish my computer ran faster”) that implicitly describes the need of the utterer for help in making his/her computer run faster.

Conversational solutions are solutions that improve the responses to utterances. That is, a conversational solution can modify AI responses to questions in order to improve the relevance of answers/responses to utterances. One approach used by conversational solutions is to ask the utterer to provide feedback about the answer(s) provided. For example, the answer/response can include a “Thumbs up” and “Thumbs down” image that, when clicked by the utterer, lets the chat bot know if the answer was useful/accurate/appropriate (“Thumbs up”) or not (“Thumbs down”).

In the prior art, conversational solutions are labor intensive when evaluating user feedback, since a subject matter expert (SME) must look at each feedback, particularly if the feedback is text written by the utterer. For example, assume that a customer (the “utterer”) asks the chat bot “How do I change the font in a formula in my document?” Assume further that the chat bot sends an answer “Highlight the formula and then click the appropriate font icon in the tool bar”. Assume then that the customer sends back the feedback “I want something faster”. Such a response requires an SME to determine that the answer was not helpful, and thus, if enough negative comments/feedback such as this are received (e.g., 10% of the feedback responses), change the answer to the question/utterance “How do I change the font in a formula in my document?”

Prior art methods do not currently provide a way to reduce the amount of work required for a SME or other user of the tooling to look through user utterances or feedback to improve the system. For example, if 10,000 user utterances are processed and 1,000 of those received negative feedback, the process of determining the correct intent is still a lengthy and tedious task for a human expert. Being able to quickly iterate on this feedback, as provided by the present invention, leads to an improvement of the performance of the system by speeding up the feedback review process.

Furthermore, automation of such feedback analysis is not obvious since some negative feedback is based not on technical accuracy, but rather the utterer's own views. In the example above, the answer for changing the font is correct, but the user simply doesn't like it, since he/she feels that it is too slow/cumbersome. As such, eliminating correct answers (“Highlight the formula and then click the appropriate font icon in the tool bar”) from the system is detrimental to performance, since accurate information is taken out of the system.

Furthermore, in the prior art all feedback, not just negative feedback, needs to be validated by an SME to ensure its correctness. That is, in the prior art all user feedback must be validated in some manner so as to reduce the amount of invalid feedback, which significantly hinders the ability to automate this process.

As such, one or more embodiments of the present invention provide a method for training an artificial intelligence (AI) system to validate a single user feedback before putting it into ground truth. In order to do this the AI system uses other users as a verification method. When a user sends feedback about an answer to a first question, the AI system looks for similar feedback about the answer for a different utterance. That is, after the user provides positive feedback stating that the answer is useful to the question posed by that user, the user is then asked to provide positive feedback that this same answer is also useful to another question posed by another user (who has already provided feedback that this answer is useful to this other question). As such, only when two (or more) users agree that the utterance-to-answer mapping is correct will the AI system incorporate it into ground truth (e.g., trust and/or validate the answer).

As such, one or more embodiments of the present invention provide the benefit of validating user feedback in-context to another highly related piece feedback. Prior art systems do not validate feedback in this optimal “in the moment” setting.

With reference now toFIG. 2, assume that a user of one of the user computer(s)152shown inFIG. 1is communicating with computer102shown inFIG. 1via display110, which displays graphical user interface (GUI)200and GUI210.

As shown in GUI200, the user sends the utterance shown in box202of “Can I activate my new credit card at my ATM?” to computer102, which responds (via a chat bot using artificial intelligence) with the answer shown in box204. The user then provides feedback regarding whether or not the answer shown in box204was useful by clicking positive icon206(if the user found the answer helpful) or by clicking negative icon208(if the user found the answer to be not helpful).

In order to achieve these actions, the AI system (and more specifically, a communication agent associated with the AI system) extracts the intent I1(e.g., asking for help in activating a credit card) and entities E1(e.g., “activate”, “credit card”, “ATM”) from the utterance U1shown in box202in order to generate and return the answer A1shown in box204. The user then clicks either positive icon206or negative icon208to provide feedback about answer A1.

If the feedback is positive (e.g., the user has clicked positive icon206, which results in a positive message being sent to computer102indicating that the user found the answer A1useful in answering the utterance U1), then that user has now 1) validated the dialog response that is answer A1, and 2) implicitly validated the intent/entities I1/E1extracted by the system for utterance U1. However, at this time, the feedback is not deemed to be “golden”. That is, at this point the positive feedback provided by the user may or may not be accurate, since it is just that single user's opinion, which might not be valid. Thus, the system does not promote this feedback as “golden”, but rather holds it in a feedback database for future verification as a “singly-verified” feedback entry that includes the utterance U1, the intent I1from utterance U1, entities E1from the utterance, answer A1, and the verifier (i.e., a “positive” feedback) for the utterance U1.

Thereafter, the user is asked to verify other utterances (e.g., U2, U3, etc.) from the feedback database with matching intent/entities (I1/E1) and answer (A1). That is, the user is given answer A1and asked if it answers utterance U2. For example, as shown in GUI210, which is also displayed to the same user (a “first entity”) that responded using GUI200, a new utterance U2(“I have a new credit card”) is displayed in a new box212.

Assume now that a second entity has previously provided this new utterance U2(“I have a new credit card”) to the conversational agent, which responded with answer A1. Assume further that the second entity stored this pairing U2:A1in the feedback database, indicating that answer A1is a valid answer for utterance U2. This U2:A1pairing and storing is without further validation from another entity in one embodiment, while in yet another embodiment of the present invention the U2:A1storage in the feedback database occurs only after a third entity has validated that answer A1is a valid answer to utterance U2.

Since the pairing of utterance U2and answer A1is found in the feedback database, then the AI system knows that the “second entity” has previously affirmed that answer A1is a useful/correct answer for utterance U2. The first entity who originally affirmed that answer A1is a useful/correct answer for utterance U1is then asked whether the answer A1shown in box204(which is the same answer A1shown in the box204in GUI200) is also a useful answer to the new utterance U2shown in box212. As shown inFIG. 2, the utterance U1(from box202) and the utterance U2(from box212) have different text, but have a same intent of learning how to activate a new credit card (asked explicitly in utterance U1and implicitly in utterance U2) meaning they share at least one intent (“new credit card”).

If the first entity validates (approves of) answer A1as being a proper answer for utterance U2(by clicking the positive icon214instead of the negative icon216), then the system determines that answer A1is confirmed to be a valid answer for utterance U1. That is, since the first entity stated that answer A1is a valid/useful answer to both utterance U1and utterance U2, then the first entity's validation of utterance U1is now trusted, since the first entity agreed with the second entity regarding the usefulness of answer A1for utterance U2.

Thus, in an embodiment of the present invention, the validation of answer A1for utterance U1is dependent upon the utterer of utterance U1agreeing with the utterer of utterance U2that utterance U2is properly answered by answer A1. In a further embodiment of the present invention, the context of utterance U1and utterance U2are also required to match. That is, not only does the answer A1need to be appropriate for utterance U1and utterance U2, and that the utterer of utterance U1and the utterer of utterance U2both agree that answer A1is a valid answer for both utterance U1and utterance U2, but the context of utterance U1and the context of utterance U2must also agree. For example, if the context of utterance U1is “today” when asking “Can I activate my new credit card at my ATM?”, then the context of utterance U2must also be “today” when uttering “I have a new credit card” before the utterer of utterance U1is presented with utterance U2. That is, in this further embodiment only utterances U2that have a same context (e.g., for a particular time, a particular product, a particular type of uttering entity, etc.) are used to cross-validate the answer for the first utterance U1.

While this example describes the first entity agreeing with the second entity in order to establish the credibility of the first entity (i.e., two users agreeing), in one or more embodiments the number of users agreeing on the validity of a given answer for a given question is greater than 2.

If the first entity (who originally validated answer A1for utterance U1) rejects answer A1for utterance U2(by clicking the negative icon216), then various actions take place in various embodiments of the present invention.

In an embodiment of the present invention, if the first entity and the second entity disagree about the usefulness of answer A1for utterance U2, then the first entity's opinion about answer A1with regard to utterance U1is disregarded (i.e., the “approval” vote from the first entity regarding A1:U1is deleted from the system).

In an embodiment of the present invention, if the first entity and the second entity disagree about the usefulness of answer A1for utterance U2, but other users (e.g., more than some predefined quantity) approve of answer A1for utterance U1, then answer A1is deemed useful for utterance U1, regardless of the fact that the first and second entity disagree about the usefulness of answer A1for utterance U2.

In an embodiment of the present invention, “reputational voting” gives different weights to user votes based on the quality of their feedback or some other profile/reputational score. For example, assume that the first entity has no expertise in the topic addressed in utterance U1. As such, this first entity is assigned a low weighting (e.g., 0.1). Assume now that a third entity has a high level of expertise in the topic addressed in utterance U1. As such, this third entity is assigned a high weighting (e.g., 0.9). Thus, the opinion of the third entity is so much greater than the opinion of the first entity that, even if several other users (with similar credentials as the first entity) who are evaluating utterance U1disagree with the second entity regarding utterance U2, the opinion of the third entity is so valued that it “overrides” the opinions of the first and other users.

In an embodiment of the present invention, the AI system does not simply give instructions on how to interact with a device, but rather directly controls the device. For example, consider box204shown inFIG. 2. Assume that rather than giving the first entity directions on how to activate the new credit card, box204includes a message that an ATM (e.g., that the first entity is standing in front of) has now had a slot opened to receive the new credit card for activation. As such, the system directly controls the ATM device simply based on the utterance U1shown in box202.

As such, one or more embodiments of the present invention directly control a physical device.

With reference now toFIG. 3, several embodiments of the present invention are represented in which a computer302(analogous to computer102shown inFIG. 1) directly controls various devices in accordance with the verified response to the first utterance U1.

FIG. 3depicts a remote computing device354, a remote storage device355, a communications controller device356, a traffic controller device357, and a manufacturing equipment controller device358, all of which are exemplary embodiments of the remote device154shown inFIG. 1.

With regard to remote computing device354, assume that an utterance (question, statement, etc.) from a user indicates that the user needs his/her computer to be upgraded (e.g., have a security patch installed, open up a new network port, add new cloud-based virtual machine (VM) processors, etc.). As such, once the AI system determines that such an action is appropriate (based on the user validating both utterance U1and utterance U2as being properly addressed by the verified response that is answer A1), then computer302will automatically upgrade the user's computer accordingly.

With regard to remote storage device355, assume that an utterance (question, statement, etc.) from a user indicates that the user needs additional data storage (e.g., turning on additional memory devices on the user's network, configuring new virtual storage devices on a cloud, etc.). As such, once the AI system determines that such an action is appropriate (based on the user validating both utterance U1and utterance U2as being properly addressed by the verified response that is answer A1), then computer302will automatically upgrade the user's storage capability accordingly.

With regard to communications controller device356, assume that an utterance (question, statement, etc.) from a user indicates that the user needs cellular coverage in an area in which cellular coverage is usually limited (e.g., for security reasons). As such, once the AI system determines that such an action is appropriate (based on the user validating both utterance U1and utterance U2as being properly addressed by the verified response that is answer A1), then computer302will automatically turn on the appropriate cell tower for the user.

With regard to traffic controller device357, assume that an utterance (question, statement, etc.) from a user indicates that the user is a first responder's emergency vehicle that needs traffic lights along its route to be turned green (in order to arrive at the scene in need of emergency assistance quickly). As such, once the AI system determines that such an action is appropriate (based on the user validating both utterance U1and utterance U2as being properly addressed by the verified response that is answer A1), then computer302will automatically clear the route of the emergency vehicle by controlling the traffic lights along that route.

With regard to manufacturing equipment controller device358, assume that an utterance (question, statement, etc.) from a user indicates that the user needs for a certain unit of manufacturing equipment (e.g., a pump in a refinery, a roller in a factory, etc.) to be turned on. As such, once the AI system determines that such an action is appropriate (based on the user validating both utterance U1and utterance U2as being properly addressed by the verified response that is answer A1), then computer302will automatically activate and/or modify the appropriate equipment accordingly.

As indicated by the dashed line to neural network324(analogous to neural network124shown inFIG. 1), in one or more embodiments the actions of computer302are under the direction of neural network324. For example, assume that a verified response (associated with the output of neuron404shown inFIG. 4) is to activate, modify, etc. one of the devices shown inFIG. 3. As such, the computer302takes this information from the neural network324.

Thus, in one or more embodiments, the present invention uses an electronic neural network, such as the neural network124shown inFIG. 1, to respond to user utterances and to receive feedback to responses to the user utterances. In various embodiments of the present invention, the neural network124shown inFIG. 1is a Non-Convolutional Neural Network, a Convolutional Neural Network (CNN), and/or another type of machine learning system. In a preferred embodiment, a NN is used to evaluate text/numeric data in a user utterance, while a CNN is used to evaluate an image provided by a user in a comment, question, etc. that is the user's utterance.

A neural network, as the name implies, is roughly modeled after a biological neural network (e.g., a human brain). A biological neural network is made up of a series of interconnected neurons, which affect one another. For example, a first neuron can be electrically connected by a synapse to a second neuron through the release of neurotransmitters (from the first neuron) which are received by the second neuron. These neurotransmitters can cause the second neuron to become excited or inhibited. A pattern of excited/inhibited interconnected neurons eventually lead to a biological result, including thoughts, muscle movement, memory retrieval, etc. While this description of a biological neural network is highly simplified, the high-level overview is that one or more biological neurons affect the operation of one or more other bio-electrically connected biological neurons.

An electronic neural network similarly is made up of electronic neurons. However, unlike biological neurons, electronic neurons are never technically “inhibitory”, but are only “excitatory” to varying degrees.

In a NN, neurons are arranged in layers, known as an input layer, hidden layer(s), and an output layer. The input layer includes neurons/nodes that take input data, and send it to a series of hidden layers of neurons, in which all neurons from one layer in the hidden layers are interconnected with all neurons in a next layer in the hidden layers. The final layer in the hidden layers then outputs a computational result to the output layer, which is often a single node for holding vector information.

With reference now toFIG. 4, a Neural Network (NN)324used to evaluate textual data in one or more embodiments of the present invention is presented. For example, assume, for illustrative purposes, that first utterance, response, and feedback data shown in box400400are text and/or data that describes the first utterance U1, the response that is answer A1, and the user's feedback for U1:A1as described above (seeFIG. 2). Assume further, for illustrative purposes, that the second utterance, response, and feedback data shown in box401are text and/or data that describes the second utterance U2, and the response that is answer A1, and the user's feedback for U2:A1as described inFIG. 2.

The electronic neurons in NN324are arranged in layers, known as an input layer403, hidden layers405, and an output layer407. The input layer403includes neurons/nodes that take input data, and send it to a series of hidden layers of neurons (e.g., hidden layers405), in which neurons from one layer in the hidden layers are interconnected with all neurons in a next layer in the hidden layers405. The final layer in the hidden layers405then outputs a computational result to the output layer407, which is often a single node for holding vector information. In an embodiment of the present invention, each neuron in the output layer407is associated with a particular response label from response labels402, as shown inFIG. 4.

As just mentioned, each node in the depicted NN324represents an electronic neuron, such as the depicted neuron409. As shown in block411, each neuron (including neuron409) includes at least four features: a mathematical function, an output value, a weight, and a bias value.

The mathematical function is a mathematic formula for processing data from one or more upstream neurons. For example, assume that one or more of the neurons depicted in the middle hidden layers405send data values to neuron409. Neuron409then processes these data values by executing the mathematical function shown in block411, in order to create one or more output values, which are then sent to another neuron, such as another neuron within the hidden layers405or a neuron in the output layer407. Each neuron also has a weight that is specific for that neuron and/or for other connected neurons. Furthermore, the output value(s) are added to bias value(s), which increase or decrease the output value, allowing the NN324to be further “fine tuned”.

For example, assume that neuron413is sending the results of its analysis of a piece of data to neuron409. Neuron409has a first weight that defines how important data coming specifically from neuron413is. If the data is important, then data coming from neuron413is weighted heavily, and/or increased by the bias value, thus causing the mathematical function (s) within neuron409to generate a higher output, which will have a heavier impact on neurons in the output layer407. Similarly, if neuron413has been determined to be significant to the operations of neuron409, then the weight in neuron413will be increased, such that neuron409receives a higher value for the output of the mathematical function in the neuron413. Alternatively, the output of neuron409can be minimized by decreasing the weight and/or bias used to affect the output of neuron409. These weights/biases are adjustable for one, some, or all of the neurons in the NN324, such that a reliable output will result from output layer407. Such adjustments are alternatively performed manually or automatically.

When manually adjusted, the weights and/or biases are adjusted by the user in a repeated manner until the output from output layer407matches expectations. For example, assume that input layer403receives inputs from box400that describes how the user approved of answer A1for utterance U1. Assume further that input layer403receives inputs from box401that describes the same user approving of answer A1for utterance U2. If NN324has been properly trained (by manually adjusting the mathematical function(s), output value(s), weight(s), and biases in one or more of the electronic neurons within NN324) to output a correct output vector (e.g., a 2-tuple output vector of 0.9, 0.2) to the output layer407, then the neuron404for the verified response has the highest value (0.9). Furthermore, the NN324, when properly trained, gives a value of 0.2 to neuron406, indicating that the user's approval of answer A1for utterance U1is not valid, since the user did not agree with another user that answer A1is valid for utterance U2.

Thus, a properly trained NN324will output a value from neuron404that is higher than the value from other neurons in the output layer407based on the interactions between the neurons in input layer403and hidden layers405and output layer407and the data that is input into the input layer403. That is, the neurons in NN324are manually adjusted such that when the user validates answer A1for both utterance U1and utterance U2, then neuron404has the highest output value (indicating that the user's feedback about A1:U1is valid), but neuron406has the highest output value when the input data suggests that the user's feedback about A1:U1is invalid.

When automatically adjusted, the weights (and/or mathematical functions) are adjusted using “back propagation”, in which weight values of the neurons are adjusted by using a “gradient descent” method that determines which direction each weight value should be adjusted to. For example, if the output from neuron404is just 0.5 and the output from neuron406is also 0.5, but the output from the neuron404(which is associated with Event A, such as a trade meeting) should be higher than any other neuron from the output layer407, then the output from neuron404is manually changed to a high value (e.g., 0.9) and the output of neuron406is changed to 0.2 or smaller. The back-propagation gradient descent process moves the weight and/or bias in each neuron in a certain direction until the output from output layer407improves (e.g., gets closer to outputting a highest value to neuron404or a highest value to neuron406, depending on whether the user's opinion of the answer A1for utterance U1is valid or not).

In an embodiment of the present invention, the adjustment of the NN automatically incorporates “goldenized” feedback into the chat system by retraining the underlying algorithm/system/NN that does intent classification and other activities associated with the chat system. For example, assume that once the NN determines that a particular answer A1is a verified response (see node404inFIG. 4), then this verified response is incorporated into the chat system, such that the chat system (chat bot) uses answer A1when responding to utterance U1as described above.

A CNN is similar to a NN in that both utilize interconnected electronic neurons. However, a CNN is different from a NN in that 1) a CNN has neural layers whose sizes are based on filter sizes, stride values, padding values, etc., and 2) a CNN utilizes a convolution scheme to analyze image data. A CNN gets its “convolutional” name based on a convolution (i.e., a mathematical operation on two functions to obtain a result) of filtering and pooling pixel data (a mathematical operation on two functions) in order to generate a predicted output (obtain a result).

With reference now toFIG. 5, a high-level flow chart of one or more embodiments of the present invention is presented.

After initiator block501, a conversational agent (e.g., a text bot, an AI system, etc.) receives an utterance from a first entity (e.g., a user, an organization, a computer system, etc.), as depicted in block503. In one or more embodiments of the present invention, the conversational agent is generated by an artificial intelligence system. For example, the NN324shown inFIG. 4not only determines whether a response (e.g., a feedback) from the first entity is verified or unverified as described above, but the NN324also is trained to act as the conversational agent (e.g., text bot) by taking in utterances into input layer403, and to output an answer/response in output layer407. For example, if an utterance “Can I activate my new credit card at my ATM?” is input into input layer403, then the information shown as answer A1in box204inFIG. 2is output as a string of text in the output layer407once the NN324is properly trained (as described above). That is, the intent/entities from the utterance cause the neurons in the NN324to produce an output that is the answer A1.

As described in block505, one or more processors (e.g., processor104shown inFIG. 1) identify an unverified response to the utterance by the conversational agent. That is, at this point the answer (response) is unverified, since it might or might not be a useful answer/response to the utterance (e.g., question).

As described in block507, the processor(s) then send the unverified response (e.g., answer A1) to the first entity.

As described in block509, the processor(s) send a request for feedback about the unverified response to the first entity. For example, a request is sent to the first entity in the form of a text question about whether answer A1is useful to addressing the intent of the utterance U1, or in the form of a “thumbs up” or “thumbs down” icon, etc.

As described in block511, the processor(s) receive a positive feedback indication about the unverified response from the first entity. For example, the user clicks the “thumbs up” icon, indicating that he/she thinks that answer A1is a good response to utterance U1.

As described in block513, the processor(s) search a data store in order to identify at least one other entry meeting a similarity criteria to the utterance, where the similarity criteria is less than an exact match between the utterance and the at least one other entry. That is, the data store includes other utterance:response (e.g., “question:answer”) pairs other than the utterance:response pair of U1:A1. As described herein, the text in utterance U1is different from the text in utterance U2, even though they are related closely enough that they are both properly answered by answer A1.

As described in block515, responsive to identifying at least one other entry in the data store meeting the similarity criteria for the utterance, the processor(s) send the at least one other entry and a request to provide feedback about the at least one other entry to the first entity. That is, the processor(s) find an utterance U2that 1) is closely related by its intent and/or its entity to utterance U1, and 2) has been answered by the same answer A1that has been presented by the system for utterance U1.

As described in block517, responsive to receiving a positive feedback for the at least one other entry from the first entity, the processor(s) mark the unverified response as a verified response to the utterance. That is, the response (e.g., answer A1) is not verified as being an appropriate response to utterance U1since the first entity has not been verified by providing the same feedback for answer A1responsive to utterance U2. The system now has answer A1(the response) that is deemed to be properly vetted, approved, etc., and will be used to respond to future instances of utterance U1.

Thus, and as described in block519, the processor(s) subsequently receive the utterance U1from a third entity (i.e., another user, customer, enterprise, etc.). Since answer A1has now been verified as being a valid/useful answer to utterance U1, the processors respond to the utterance from the third entity with the verified response (answer A1), as described in block521.

The flow-chart ends at terminator block523.

In an embodiment of the present invention, responsive to receiving the positive feedback indication about the unverified response from the first entity, the processor(s) store a representation of the utterance in the data store. This representation comprises the utterance (e.g., for utterance U1), the unverified response (e.g., answer A1), one or more entities (E1) from the utterance, one or more intents (I1) derived from the utterance, and an indication (approval or disapproval) of a single verification of the unverified response by the first entity, as described herein.

In an embodiment of the present invention, the artificial intelligence system is a neural network, and the method further comprises training the neural network to verify the unverified response in order to create the verified response based on identifying the at least one other entry in the data store that meets the similarity criteria for the utterance (seeFIG. 4).

In an embodiment of the present invention, the verified response activates a physical device (seeFIG. 3). For example, the verified response not only confirms that the answer to the utterance is valid, but also causes a device (e.g., a pump, a computer, etc.) to turn on.

In an embodiment of the present invention, the verified response modifies a physical device. For example, the verified response not only confirms that the answer to the utterance is valid, but also causes a device (e.g., a computer, a storage device, etc.) to be modified (e.g., by adding virtual processors, cloud-based storage devices, etc.).

In an embodiment of the present invention, the verified response improves a functionality of a physical device. For example, the verified response not only confirms that the answer to the utterance is valid, but also causes a device (e.g., a computer, a storage device, etc.) to be modified (e.g., by adding virtual processors, cloud-based storage devices, etc.) in order to make the device run more efficiently, have greater processing power, etc.

Characteristics are as follows:

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for an organization. In one or more embodiments, it is managed by the organization or a third party and/or exists on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by several organizations and supports a specific community that has shared concerns (e.g., mission, security requirements, policy, and compliance considerations). In one or more embodiments, it is managed by the organizations or a third party and/or exists on-premises or off-premises.

Virtualization layer70provides an abstraction layer from which the following examples of virtual entities that are provided in one or more embodiments: virtual servers71; virtual storage72; virtual networks73, including virtual private networks; virtual applications and operating systems74; and virtual clients75.

Workloads layer90provides examples of functionality for which the cloud computing environment are utilized in one or more embodiments. Examples of workloads and functions which are provided from this layer include: mapping and navigation91; software development and lifecycle management92; virtual classroom education delivery93; data analytics processing94; transaction processing95; and neural network training processing96, which performs one or more of the features of the present invention described herein.

In one or more embodiments of the present invention, any methods described in the present disclosure are implemented through the use of a VHDL (VHSIC Hardware Description Language) program and a VHDL chip. VHDL is an exemplary design-entry language for Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), and other similar electronic devices. Thus, in one or more embodiments of the present invention any software-implemented method described herein is emulated by a hardware-based VHDL program, which is then applied to a VHDL chip, such as a FPGA.