Patent ID: 12248755

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Natural-language understanding (NLU) is a growing field of language processing focused on interpreting received language inputs. Recently, neural networks for machine learning have been found to perform well as a base for NLU systems and models. Using machine learning techniques, NLU models may be trained on large sets of training data to produce a robust model for language understanding. Once trained, a NLU model can interpret received language inputs. In some examples, the NLU model outputs an “intent” characterizing the received language input. The intent can be used by the NLU and/or other models to determine an appropriate action in response to the language input.

NLU models have a wide variety of uses, from machine translation and transcription of speech to question answering. Question answering can be seen in common applications, such as a chatbot (or “bot”), which is a text-based interface for automated machine generated responses to customer inputs. In question answering, an NLU model can determine the intent of a language input which can be used to generate a response. In some examples, the NLU can feed the intent to another model, such as a logic model, and other model then determines a response (in this context, a text-based response) based on the intent.

There are a number of challenges in generating an effective question answering bot. For one, NLU models are typically large, as the more extensively the model is trained, the better it performs. However, it can be difficult to acquire large data sets to train NLUs. While language inputs are readily available (in the form of transcripts and audio samples of spoken language) to be used as training data, the language inputs must be labelled. Typically, language inputs are labelled manually, which is a time consuming and labor intensive process. Further, logic models are typically manually generated which causes the chatbot to be inflexible (i.e., it can be difficult/time consuming to implement changes to a logic model). Also, there are currently no effective automated systems for evaluating such logic models.

Implementations herein are directed toward automatically generating training data for NLU models. In particular, a logic model may be used to automatically generate labels for transcripts such that the transcripts can be used to train the NLU model. Additionally or alternatively, implementations herein are directed toward dynamic bot building by automatically populating logic models. For example, when the logic model does not have a satisfactory response to a given language input, a new response can be added to the logic model. Further, implementations herein are directed toward automatically evaluating a logic model by determining how the logic model performs against various inputs.

Referring now toFIG.1, in some implementations, an example system100for dynamically generating training data220for a model210includes a remote system140(e.g., a cloud computing environment140) in communication with one or more customer devices10via a network112. The remote system140may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable/elastic resources142including computing resources144(e.g., data processing hardware) and/or storage resources146(e.g., memory hardware). Additionally, an agent device15may be communicatively coupled to the remote system140, via the network112(or a different network), such that an agent11can communicate with a customer12. The customer12may transmit customer inputs20the agent11(e.g., “What discounts are available?”) and the agent11may response with one or more agent inputs30(e.g., “No discounts are currently available.”). A data store150(i.e., a remote storage device) may be overlain on the storage resources146to allow scalable use of the storage resources146by one or more of the clients (e.g., the customer device10), the agents11(e.g., the agent device15), and/or the computing resources144. The data store150is configured to store a plurality of transcripts152,152a-n. Each transcript includes one or more of the customer inputs20and one or more of the agent inputs30. The data store150may store any number of transcripts152at any point in time.

The remote system140is configured to receive the one or more customer inputs20from the customer device10associated with the respective customer12via, for example, the network112and the one or more agent inputs30from the agent device15associated with the respective agent11via, for example, the network112. The customer device10and/or agent device15may correspond to any computing device, such as a desktop workstation, a laptop workstation, or a mobile device (i.e., a smart phone). The customer device10(and similarly agent device15, though not illustrated) includes computing resources18(e.g., data processing hardware) and/or storage resources16(e.g., memory hardware).

The remote system140executes a bot module160for dynamically generating training data220. In some implementations, the bot module160generates one or more responses310in response to the customer inputs20. Additionally or alternatively, the bot module160is configured to dynamically build or generate a chat bot by improving and/or evaluating a logic model300. The bot module160is configured to receive the one or more transcripts152. Each transcript152includes at least one customer input20and one agent input30. The bot module160may use the logic model300to label the transcripts152to be used as training data220for an NLU model210, as described in greater detail below (FIG.2A).

In other implementations the bot module160provides the transcript152to the NLU model210to generate an intent305. The bot module may160may then provide the generated intent305to the logic model300. The logic model300iterates through one or more responses310,310a-nto select a response310,310S based on the intent305. Here, each response310may be a potential reply to the customer input20(i.e., a possible response to the question/comment represented by the customer input20). The bot module160optionally provides the selected response310S to a comparison module250. The comparison module250, in some examples, generates a similarity score253for the selected response310S. In some implementations, the similarity score253is based on a comparison between the selected response310S and the agent input30that corresponds to the customer input20of the transcript152. That is, the similarity score253represents how similar the selected response310S (using natural language) to the agent input30that represents the “ground truth” response to the customer input20. The comparison may use embedding techniques to generate the similarity score253. The comparison module250, in some implementations, compares the similarity score253to a similarity threshold255. Based on the comparison between the similarity score and to the similarity threshold255, the bot module160performs one or more actions, as described in greater detail below (FIG.2B).

The system ofFIG.1is presented for illustrative purposes only and is not intended to be limiting. For example, although only a single example of each component is illustrated, the system100may include any number of components10,15,112,140,150, and160. Further, although some components are described as being located in a cloud computing environment140, in some implementations, some or all of the components may be hosted locally on the customer device10and/or agent device15. Further, in various implementations, some or all of the components150and160, are hosted locally on customer device10and/or agent device15, remotely (such as in the cloud computing environment140), or some combination thereof.

FIG.2Aincludes a schematic view200A of an example bot module160dynamically generating training data220for a model210. The training data220is based on transcripts152which include at least one customer inputs20and at least one agent input30. Here, the bot module160feeds the transcript152directly to the logic model300. In some implementations, the bot module160parses through the transcript152by analyzing one customer input20/agent input30pair at a time. For example, the bot module160analyzes a first customer input20/agent input30pair to attempt to generate training data220from the first pair. When successful, the bot module160, in some examples, moves to the next customer input20/agent input30pair. When the bot module160determines that a customer input20/agent input30pair cannot be used for training data (i.e., when not successful), the bot module160optionally discards the rest of the transcript (e.g., the conversation between the customer12and the agent11has moved off-topic and/or beyond the scope of the logic model300).

The bot module160may generate training data for a customer input20/agent input30pair of a transcript by feeding the pair to the logic model300. In some implementations, the bot module160selects a response310from the logic model300based on the customer input20(i.e., analyze the customer input20and try to determine the response310that is most appropriate). In other implementations, the bot module160selects the response310that is most similar to the agent response30corresponding to the customer input20. For example, the bot module160iteratively provides each respective response310to the comparison module250to determine a similarity score253for each response310. In this example, the bot module160selects the response310with the highest similarity score253(i.e., the response310that is most similar, based on the similarity score253, to the agent input30). Next, the bot module160compares the similarity score253of the response310to a similarity threshold255. When the similarity score253of the response310satisfies the similarity threshold255(e.g., when the similarity score253is equal to or larger than the similarity threshold255), the bot module160generates training data220based on the transcript152. That is, in some examples, when the similarity score253of the response310satisfies the similarity threshold255, the bot module160determines that the transcript152is suitable for training data220. Put another way, in this scenario, the bot module160determines that the transcript152includes inputs20,30relevant to training the model210. The training data220may include the customer input20, the agent input30(not illustrated), the response310, and any combination thereof. In some implementations, the response310also corresponds to one or more intents305, as discussed in greater detail below (FIG.3).

As an example, a first customer input20of a transcript152is “I want a new plan.” and the corresponding agent input30(i.e., the response of the agent11to the customer12) is “What plan do you want?” The bot module160searches for the response310of the logic model300that is closest to the agent input30(i.e., most similar to “What plan do you want?”). For example, the most similar response310of the logic model is “What kind of plan would you like?” which satisfies the similarity threshold255. In this example, the bot module160, in response, stores the customer input20along with the response310, and any other data from the training model300corresponding to the response310(i.e., one or more intents305), as the training data220.

Continuing with the above example, a second customer input20of the transcript152is “Oh no, I spilled my coffee” and the corresponding agent input30is “I'm sorry to hear that, I hope you are okay.” Here, the bot module160is unable to find a response310of the logic model300that is similar to the second agent input30. Accordingly, the bot module160discards the rest of the transcript152(i.e., does not include this input pair or any subsequent input pairs as training data220).

The above examples are for illustrative purposes and are not intended to be limiting. The bot module160may receive any number of transcripts152including any number of customer inputs20and agent inputs30to dynamically generate training data220.

FIG.2Bis a schematic view200B of an example bot module160dynamically building a logic model300for a chat bot. Here, the bot module160includes a natural-language understanding (NLU) model210. The NLU model210may be trained using training data220generated using a logic model300, as described herein (FIG.2AandFIG.4). The bot module160may provide a customer input20of a transcript152to the NLU model210. In turn, the NLU model210outputs an intent305based on the customer input20. The bot module160provides the intent305to a logic model300. The bot module160, in some examples, parses through the logic model300to determine a response310that corresponds to the intent305. When the bot module160determines that no response310corresponds to the intent305, the bot module160optionally decreases a metric265of the logic model300. Alternatively, when the bot module160determines that a response310corresponds to the intent305, the bot module160increases the metric265. Here, the metric265corresponds to a rating or quality or comprehensiveness of the logic model300. A higher metric265may indicate that the logic model300has broad coverage. Conversely, a low metric265may indicate that the logic model300has poor coverage or has insufficient coverage. Further, the metric265, in some implementations, compares various logic models300.

When the bot module160does select a response310S that corresponds to the intent305, the bot module160provides the response310S to a comparison module250, along with the agent input30that corresponds to the customer input20of the transcript152. Optionally, the comparison module250generates a similarity score253based on the response310S and the agent input30. In some implementations, the comparison module250employs embedding techniques to generate the similarity score253. The bot module160also compares the similarity score253of the response310to the similarity threshold255. When the similarity score253of the response310satisfies the similarity threshold255(e.g., when the similarity score253is equal to or larger than the similarity threshold255), the bot module160may increase the metric265of the logic model300. Alternatively, when the similarity score253of the response310does not satisfy the similarity threshold255(e.g., when the similarity score253is smaller than the similarity threshold255), the bot module160decreases the metric265. In some examples, the bot module160adds a new response310,310X to the logic model300. For example, the bot module160adds the agent input30as a new response310X to the logic model300, based on the corresponding intent305.

In some implementations, the bot module160receives “golden” transcripts152when dynamically building a logic model300for a chat bot, as illustrated inFIG.2B. Such golden transcripts152are pre-screened transcripts152(e.g., manually by a human) that represent conversations that the logic model300should ideally be able to replicate. In other words, the golden transcripts152are intended as ideal training examples to which the logic model300is expected to have a satisfactory response310for each customer input20. In these implementations, the metric265is increased/decreased more harshly based on the success of the logic model300. For example, when the bot module160cannot find a response310in the logic model300that is satisfactorily similar to an agent input30of the golden transcript152, the bot module160may decrease the metric265by a greater factor than compared to a similar occurrence when using a regular (i.e., not “golden”) transcript152.

FIG.3is a schematic view of an example logic model300. In some implementations, the logic model300is a logic tree, as illustrated. However, the illustration is not intended to be limiting and the logic model300can be in any appropriate form. The logic model300can include one or more nodes connected by one or more branches. The nodes may correspond to an intent305and/or a response310. For example, an intent305,305acorresponds to billing. That is, this portion of the logic model300corresponds to responses310related to customer inputs20that are related to billing. Here, the intent305ais connected to two child nodes representing more specific intents305related to billing intent305a. For example, an intent305,305bcorresponds to billing plans. The intent305bis connected to two child nodes representing responses310,310a-b. Here, each of the response310a(i.e., “Do you want to cancel your plan?”) and the response310b(i.e., “What plan do you want?”) correspond to both intent305aand intent305b. For example, when an NLU model210, in response to a transcript152, outputs an intent305a(i.e., billing), the bot module160selects either response310aand/or310bof logic model300as an appropriate response310to the customer input20.

With continued reference toFIG.3, an exemplary intent305,305ccorresponds to discounts related to a billing intent305a. Here, the intent305conly has a single related response310c(i.e., “We do not have any available discounts.”). In some implementations, the bot module160is configured to add additional responses310to the logic model300. As described above (FIG.2B), when the logic model300does not include a response310that satisfies a similarity threshold255, the bot module160adds one or more responses310to the logic model300. For example, when a transcript152includes a customer input20saying “I would like discounts on my bill,” the NLU model210determines intents305a,305care related to billing and discounts based on the customer input20. Here, the logic model300only includes one response310crelated to both intents305a,305c. When the response310cis not sufficiently similar to an agent input30corresponding to the customer input20(i.e., a similarity score253determined based on the response310cand the agent input30does not satisfy the similarity threshold255), then the bot module160adds a new response310X to the logic model. In this example, when the agent input30is “I will try to find a discount for you,” the response310cof “We do not have any available discounts” is not sufficiently similar. Thus, the bot module160adds the agent input30as the new response310X (i.e., “I will try and find a discount for you” to the logic model300under the nodes related to the intents305a,305c(i.e., discounts and billing). In some implementations, the bot module alters a metric265corresponding to the performance of the logic model300.

Referring now toFIG.4, a training process400illustrates training a model210(i.e., the NLU model210) using the training data220(generated, for example, by the bot module160ofFIG.1). Though a single model210is illustrated, the training process400may generate and/or train multiple models210of different types or with different parameters. For example, the model210is a differentiable learning model and includes any of a deep neural network, a recurrent neural network, a temporal convolutional network, a long short term memory network, etc. Further, the training process400may include using mini-batch stochastic gradient descent techniques for training the model210.

In some implementations, the process400employs a two-step training technique that includes pre-training and training. Pre-training is a technique used for initializing a model210which can then be further fine-tuned based on additional training data220. For the model210, pre-training may include initiating the model210with pre-training data405including transcripts that have been previously labeled. In some implementations, a base or a generic NLU model210is configured in pre-training and then fine-tuned for a specific entity using training data220generated at the entity. For example, an entity may have thousands of transcripts (e.g., transcripts152ofFIG.1) of conversations between customers and agents of the entity. The transcripts may be labeled to be used as training data220as described above (FIG.2A). Accordingly, the NLU model210trained on such training data220will be specifically tailored based on transcripts corresponding to the entity.

The process400, in some examples, includes fine-tuning parameters of the pre-trained model210. In these examples, the process400includes providing the training data220to the model210. The training data220can include any data that can be used to train an NLU model210to interpret language inputs. For example, the training data220can include multiple transcripts of conversations. In some implementations, each sample of the training data220includes a customer input20and a response310. Here, the response310acts as a label or a target output of the NLU model210based on the customer input20. Each response310may correspond to (and/or include) one or more intents (e.g., the intents305ofFIG.3).

Upon receiving the training data220, the model210, in some implementations, generates an output305(e.g., the intent305). The intent305may be represented as a vector, an embedding, a probability distribution, or any other appropriate representation. In some implementations, the output305is used by a loss function440to generate a loss450. That is, the loss function440compares the output305and the response310to generate the loss450, where the loss450indicates a discrepancy between the response310(i.e., the target output) and the output305. The loss function440may implement any suitable technique to determine a loss such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-class loss, etc. In some examples, the loss450is provided directly to the model210. In these examples, the model210processes the loss450and adjusts one or more parameters of the model210(e.g., weights) to account for the loss450. In some implementations, the model210is continually trained (or retrained) as additional training data220is received.

Once the NLU model210is sufficiently trained, the NLU model210may be implemented to dynamically build a chatbot. In other words, the NLU model210may be implemented in a process to automatically generate a logic model300, as discussed above (FIG.2B).

FIG.5is a flowchart of an exemplary arrangement of operations for a method500of dynamically generating training data for a model. The method500may be performed, for example, by various elements of the dynamic bot building system100ofFIG.1and/or the computing device600ofFIG.6. At operation502, the method500includes receiving a transcript152corresponding to a conversation between a customer12and an agent11, the transcript152including a customer input20and an agent input30. At operation504, the method500includes receiving a logic model300including a plurality of responses310, each response310of the plurality of responses310a potential reply to the customer input20. At operation506, the method500further includes selecting, based on the agent input30, a response310from the plurality of responses310of the logic model300. At operation508, the method500includes determining that a similarity score253between the selected response310and the agent input30satisfies a similarity threshold255. At operation510, the method500includes, based on determining that the similarity score253between the selected response310and the agent input30satisfies the similarity threshold255, training a machine learning model210using the customer input20and the selected response310.

FIG.6is a schematic view of an example computing device600that may be used to implement the systems and methods described in this document. The computing device600is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

The computing device600includes a processor610, memory620, a storage device630, a high-speed interface/controller640connecting to the memory620and high-speed expansion ports650, and a low speed interface/controller660connecting to a low speed bus670and a storage device630. Each of the components610,620,630,640,650, and660, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor610can process instructions for execution within the computing device600, including instructions stored in the memory620or on the storage device630to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display680coupled to high speed interface640. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices600may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory620stores information non-transitorily within the computing device600. The memory620may be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). The non-transitory memory620may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.

The storage device630is capable of providing mass storage for the computing device600. In some implementations, the storage device630is a computer-readable medium. In various different implementations, the storage device630may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory620, the storage device630, or memory on processor610.

The high speed controller640manages bandwidth-intensive operations for the computing device600, while the low speed controller660manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller640is coupled to the memory620, the display680(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports650, which may accept various expansion cards (not shown). In some implementations, the low-speed controller660is coupled to the storage device630and a low-speed expansion port690. The low-speed expansion port690, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device600may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server600aor multiple times in a group of such servers600a, as a laptop computer600b, or as part of a rack server system600c.

Various implementations of the systems and techniques described herein can be realized in digital electronic and/or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an “application,” an “app,” or a “program.” Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer readable medium, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

The processes and logic flows described in this specification can be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user and optionally a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.