Patent ID: 12197872

DETAILED DESCRIPTION

The present technology will now be described with respect to the following exemplary systems and methods.

Example Systems

A high-level system diagram100of an exemplary processing system for performing the methods described herein is shown inFIG.1. The processing system102may include one or more processors104and memory106storing instructions and data. The instructions and data may include the virtual assistant system and the NLG model described herein. Processing system102may be resident on a single computing device. For example, processing system102may be the device with which the user is interacting (e.g., a mobile device or a PC), and the virtual assistant and the NLG model may thus be local to that device.

However, in some aspects of the technology, processing system102may be distributed, such that the virtual assistant system and the NLG model (or portions thereof) reside on different physical computing devices. For example, processing system102may be distributed between a user device and one or more remote servers (e.g., a cloud-computing system). In that regard, the virtual assistant system may be distributed between a user device and a remote server such that the user-side portion is configured only to accept the user's request and convey the final response, and the server-side portion is configured to receive the request (from the user-side portion), parse it, make all necessary API calls, generate the natural language response, and pass the final response back to the user-side portion. Likewise, in some aspects of the technology, a user-side portion of the virtual assistant may be configured to pass the user's request to a server-side portion solely for NLG processing tasks, such that the client-side portion handles all API calls, creates all dialogue acts or templated responses, and then passes those dialogue acts or templated responses back to the NLG model for generating the final natural language response.

Processing system102may be implemented on any type of computing device(s), such as any type of general computing device, server, or set thereof, and may further include other components typically present in general purpose computing devices or servers. Memory106stores information accessible by the one or more processors104, including instructions108and data110that may be executed or otherwise used by the processor(s)104. Memory106may be of any non-transitory type capable of storing information accessible by the processor(s)104. For instance, memory106may include a non-transitory medium such as a hard-drive, memory card, optical disk, solid-state, tape memory, or the like. Computing devices suitable for the roles described herein may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.

In all cases, the computing devices described herein may further include any other components normally used in connection with a computing device such as a user interface subsystem. The user interface subsystem may include one or more user inputs (e.g., a mouse, keyboard, touch screen and/or microphone) and one or more electronic displays (e.g., a monitor having a screen or any other electrical device that is operable to display information). Output devices besides an electronic display, such as speakers, lights, and vibrating, pulsing, or haptic elements, may also be included in the computing devices described herein.

The one or more processors included in each computing device may be any conventional processors, such as commercially available central processing units (“CPUs”), graphics processing units (“GPUs”), tensor processing units (“TPUs”), etc. Alternatively, the one or more processors may be a dedicated device such as an ASIC or other hardware-based processor. Each processor may have multiple cores that are able to operate in parallel. The processor(s), memory, and other elements of a single computing device may be stored within a single physical housing, or may be distributed between two or more housings. Similarly, the memory of a computing device may include a hard drive or other storage media located in a housing different from that of the processor(s), such as in an external database or networked storage device. Accordingly, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel, as well as one or more servers of a load-balanced server farm or cloud-based system.

The computing devices described herein may store instructions capable of being executed directly (such as machine code) or indirectly (such as scripts) by the processor(s). The computing devices may also store data, which may be retrieved, stored, or modified by one or more processors in accordance with the instructions. Instructions may be stored as computing device code on a computing device-readable medium. In that regard, the terms “instructions” and “programs” may be used interchangeably herein. Instructions may also be stored in object code format for direct processing by the processor(s), or in any other computing device language including scripts or collections of independent source code modules that are interpreted on demand or compiled in advance. By way of example, the programming language may be C#, C++, JAVA or another computer programming language. Similarly, any components of the instructions or programs may be implemented in a computer scripting language, such as JavaScript, PHP, ASP, or any other computer scripting language. Furthermore, any one of these components may be implemented using a combination of computer programming languages and computer scripting languages.

Example Methods

Schema-Guided Text Generation

FIG.2depicts an exemplary method200showing how a virtual assistant may receive and respond to an input from a user using a schema-guided approach in accordance with aspects of the disclosure. In step202, the virtual assistant receives an input from a user. The input may be a verbal input from the user, a typed input from the user, an input generated by the user pressing a button (e.g., a physical button or a soft button) or clicking or selecting something on a user interface, or any other form of input that the virtual assistant is configured to accept. Likewise, the content of the input may be a command (e.g., find nearby restaurants), a question (e.g., what is the highest rated restaurant within 2 miles), or any other type of input that the virtual assistant is configured to handle. For the purposes of illustrating method200, we will assume going forward that the user verbally asks the virtual assistant “What is the best place for a sandwich nearby, and how far is it from public transportation?”

In step204, the virtual assistant processes the input from the user to identify one or more APIs to be called, and the data to be passed to each API. Thus, continuing with the same example, the virtual assistant will be configured to process the question “What is the best place for a sandwich nearby, and how far is it from public transportation?” to determine what API or APIs would need to be called, and what data would need to be provided to each such API to obtain the information the user is requesting. Where this processing entails parsing natural language (as it does in the present example), the virtual assistant will utilize a natural language understanding (NLU) module to interpret the meaning of the question. In addition, where the input is conveyed verbally, the virtual assistant (or its NLU module) may be configured to first process the audio data using voice recognition software in order to create a textual version of the input. In that regard, the user device may be configured to perform this voice recognition step, or the user device may be configured to pass the audio file to a remote processing system configured to perform the voice recognition. Of course, in cases where the input is not verbal (e.g., typed, or generated by a user clicking or selecting something), voice recognition is not necessary. In the present example, the virtual assistant may identify an API configured to locate nearby restaurants by type, which requires three pieces of information: current location; search radius; and type of restaurant. In addition, the virtual assistant may also identify an API configured to find the closest public transportation hub to a given location which only requires a reference location to be entered.

In step206, the virtual assistant will check whether any additional information is needed from the user. For example, if the virtual assistant was unable to understand the input originally, it may ask the user to repeat or restate the original question or command. Likewise, if the virtual assistant was able to understand the original input, but was not provided enough information to identify the API(s) to be invoked, it may ask for additional information. In addition, if the virtual assistant was able to identify one or more APIs to be invoked, but requires additional information in order to make one or more of the API calls, it may ask for that additional information. Thus, in the present example, the virtual assistant's NLG model may be able to match “place for a sandwich” to a relevant “deli/sandwich” subcategory of restaurants defined in that API corresponding to restaurants that mainly serve sandwiches. Likewise, the virtual assistant may be configured to automatically access the user's current location so that a follow-up question in that regard is not required.

However, as the user's original request did not specify a particular search radius, the virtual assistant may be configured to ask the user how far they are willing to travel. In such a case, as indicated by the “yes” arrow extending from step206, the method200would loop back through steps202-206in order to receive and process that additional input and determine if any further input is still needed. These additional user prompts may be generating by the NLG model based on a dialogue act such as “request” or “req_more” as described further below. In all cases, as indicated by the “no” arrow extending from step206, once the virtual assistant has collected all the information it needs to make each indicated API call, the method200will proceed to step208.

In step208, for each API identified in step204, the virtual assistant issues an API call based on the user's input(s). Continuing with the present example, the virtual assistant will issue a call to first identified API that is configured to locate nearby restaurants of a certain type, and will pass the user's current location, indicated search radius (received from a follow-up question to the user), and the identified subcategory of “deli/sandwich” to that first API. In addition, the virtual assistant may be configured to issue a call to the second identified API that is configured to locate all nearby public transportation hubs, and that call would include user's current location.

In step210, for each API called in step208, the virtual assistant will receive a response and generate one or more dialogue acts based on that response. Thus, in the present example, the virtual assistant may receive a first list of “deli/sandwich” restaurants within the indicated search radius from the first API, and may receive a second list of nearby public transportation hubs from the second API. As will be appreciated, the virtual assistant may be configured to identify the highest rated “deli/sandwich” restaurant from the first list and use its address to identify the nearest public transportation hub to that address from the second list. However, as indicated by the dashed arrow connecting steps208and210, in some aspects of the technology, steps208and210may also be repeated as necessary to make all indicated API calls. For example, the virtual assistant may determine to only call the first API in the initial pass through steps208and210, and then to return to step208so that the second API can be called using the address of the highest rated “deli/sandwich” restaurant in order obtain a more targeted list of public transportation hubs proximate to that restaurant.

In any event, as indicated in step210, once responses have been received for all APIs identified in step204, the virtual assistant will generate a dialogue act for each response that was received. In that regard, a dialogue act may take any suitable form, but will generally consist of a basic action (e.g., inform, request, “req_more” (request more), and then may include one or more “slots” indicating a type or category of information, and/or one or more “values” indicating a specific piece of information in a given “slot.” Thus, based on the responses received from the first and second APIs in the present example, the virtual assistant may generate a first dialogue act of inform(restaurant=“Al's Deli”) based on the first API having indicated that “Al's Deli” is the highest rated “deli/sandwich” restaurant in the indicated search radius. Likewise, the virtual assistant may generate a second dialogue act of inform(type=“bus stop”, address=“North Avenue and Center Street”, distance=“203 ft”) based on the second API having indicated that the closest public transportation hub to “Al's Deli” is a bus stop located 203 feet away from Al's Deli at the intersection of North Avenue and Center Street.

In some cases, the virtual assistant may determine from the API responses that further information is needed. In such cases, the virtual assistant may issue a dialogue act based on one or more of the API calls that asks for more information for a given slot. For example, if the second API returns both a bus stop and a train station that are determined to be equally close to “Al's deli,” the virtual assistant may be configured to generate a dialogue act of request(type) in order to prompt the user to indicate which form of transportation they prefer. Likewise, if the first API cannot find any “deli/sandwich” restaurant in the search radius, the virtual assistant may be configured to generate dialogue acts of inform(restaurant=null) and req_more( ) in order to inform the user that it was unable to find the requested type of restaurant, and to ask if the user would like to initiate another search. If the user does choose to initiate another search, this may result in the method being restarted at step202.

The virtual assistant may be further configured to include in each dialogue act more information than was received in each API response. For example, in some aspects of the technology, the virtual assistant may be configured to include in each dialogue act the relevant information input by the user, so that the context of the user's question will be reflected in the eventual natural language response. For example, the virtual assistant may be configured to include the type of restaurant, and search radius in the first dialogue act so that it becomes inform(restaurant=“Al's Deli”, type=“deli/sandwich”, radius=“5 miles”). Likewise, the virtual assistant may be configured to include the reference address in the second dialogue act so that it becomes inform(type=“bus stop”, address=“North Avenue and Center Street”, distance=“203 ft”, ref address=“345 North Avenue”).

In step212, for each dialogue act that includes a slot, the virtual assistant will replace the slot with a natural language description of the type of information represented by the slot. Thus, assuming that the virtual assistant has generated a first dialogue act of inform(restaurant=“Al's Deli”) and a second dialogue act of inform(type=“bus stop”, address=“North Avenue and Center Street”, distance=“203 ft”), the virtual assistant will replace the slot names of “restaurant,” “type,” “address,” and “distance” with natural language descriptions thereof. For example, the virtual assistant may convert the first dialogue act into a modified dialogue act of inform(“name of restaurant”=“Al's Deli”), and may convert the second dialogue act into inform(“type of public transportation hub”=“bus stop”, “address of transportation hub”=“North Avenue and Center Street”, “distance to final location”=203 ft). Although the intention is to replace each slot name, in some cases the slot name may be deemed descriptive enough that a further natural language description is not necessary or helpful. In addition, although step212describes modifying the dialogue acts created in step210with natural language descriptions of each slot, in some aspects of the technology, step210may be omitted for any dialogue act that would include a slot, and the virtual assistant may instead be configured to generate each dialogue act in the first instance using the natural language descriptions of each slot.

The natural language descriptions used in step212may be obtained directly from the API developer, or may come from a database or other data structure maintained by the developer of the virtual assistant. For example, in some aspects of the technology, the response from the API may include the natural language description of each slot. In some aspects of the technology, the API may be configured to provide the natural language descriptions in response to separate follow-up calls from the virtual assistant. In some aspects of the technology, the API developer may provide to the virtual assistant a full database of the natural language descriptions for each slot used by the API, so that the virtual assistant can query the database locally. In some aspects of the technology, the virtual assistant developer may create its own database of natural language descriptions for the slots of a given API.

In step214, the virtual assistant concatenates all dialogue acts, using the modified dialogue acts generated in step212wherever possible. In that regard, as step212will not result in modification of any dialogue act that lacks a slot name, any such slot-less dialogue acts (e.g., the “req_more( )” act described previously) will thus be included unchanged in the concatenated dialogue act created in step214. Thus, assuming that the virtual assistant generated the first and second dialogue acts mentioned in the prior examples, it would then combine them to create the following concatenated dialogue act: (inform(“name of restaurant”=“Al's Deli”), inform(“type of public transportation hub”=“bus stop”, “address of transportation hub”=“North Avenue and Center Street”, “distance to final location”=203 ft)).

Finally, in step216, the virtual assistant provides the concatenated dialogue act generated in step214to the NLG model for generation of a natural language response based on the concatenated dialogue act. The resulting natural language response may then be provided back to the user either in a textual or audio format. Where the virtual assistant is configured to provide the natural language response in audio format, it may do so by processing the natural language response of step216using text-to-speech software. Here as well, any such text-to-speech processing may be performed by the user device, or by one or more remote computing devices to which the user device has access.

The NLG model of step216may be any model that has been trained to transform dialogue acts into corresponding natural language utterances. For example, the NLG model may be a deep neural network pretrained as a “T5” Text-to-Text Transfer Transformer model, and fine-tuned on a representative set of dialogue acts so that it can reliably translate a given dialogue act into an accurate, complete, and fluent natural language utterance. In addition, any suitable architecture may be used for the NLG model. For example, in some aspects of the technology, the NLG model may be a T5-small model having 6 layers each in the encoder and decoder, and roughly 60 million parameters in total. Further, any suitable training regimen may be used. For example, the NLG model may be fine-tuned using a constant learning rate of 0.001 and a batch size of 256 for 5000 steps, and one or more automatic metrics (e.g., BLEU, SER (Slot Error Rate)) may be employed in fine-tuning to assess how well each generated natural language response matches the reference response of each training example.

Template-Guided Text Generation

FIG.3depicts an exemplary method300showing how a virtual assistant may receive and respond to an input from a user using a template-guided approach in accordance with aspects of the disclosure. In step302, the virtual assistant receives an input from a user. Here as well, any suitable type and content of input may be provided, as described above with respect to step202ofFIG.2. Likewise, we will again assume that the user verbally asks the virtual assistant “What is the best place for a sandwich nearby, and how far is it from public transportation?”

In step304, the virtual assistant processes the input from the user to identify one or more APIs to be called, and the data to be passed to each API, in the same way as has been described above with respect to step204ofFIG.2. Thus, continuing with the same example, the virtual assistant will be configured to process the question “What is the best place for a sandwich nearby, and how far is it from public transportation?” to determine what API or APIs would need to be called, and what data would need to be provided to each such API to obtain the information the user is requesting. Where this processing entails parsing natural language (as it does in the present example), the virtual assistant will utilize its NLU module to interpret the meaning of the question. In addition, where the input is conveyed verbally, the virtual assistant (or its NLU module) may be configured to first process the audio data using voice recognition software in order to create a textual version of the input as has been described above. In the present example, the virtual assistant may identify an API configured to locate nearby restaurants by type, which requires three pieces of information: current location, search radius, type of restaurant. In addition, the virtual assistant may also identify an API configured to find the closest public transportation hub to a given location which only requires a reference location to be entered.

In step306, the virtual assistant will check whether any additional information is needed from the user, in the same way as has been described above with respect to step206ofFIG.2. Again, as indicated by the “yes” arrow extending from step306, if the virtual assistant does require more information, it will prompt the user for that information, and the method300will loop back through steps302-306in order to receive and process that additional input and determine if any further input is still needed. Likewise, as indicated by the “no” arrow extending from step306, once the virtual assistant has collected all the information it needs to make the each indicated API call, the method300will proceed to step308.

In step308, for each API identified in step304, the virtual assistant issues an API based on the user's input(s). Continuing with the present example, the virtual assistant will issue a call to the first identified API that is configured to locate nearby restaurants of a certain type, and will pass the user's current location, indicated search radius (e.g., as received from a follow-up question to the user), and an identified type of restaurant (e.g., “deli/sandwich”) to that first API. In addition, the virtual assistant may be configured to issue a call to the second identified API that is configured to locate all nearby public transportation hubs, and that call will include the user's current location.

In step310, for each API called in step308, the virtual assistant will receive a response and generate one or more templated responses based on that API response. The templates may be generated by the API developer or the developer of the virtual assistant system. Thus, in some aspects of the technology, the API response may be one or more complete templated responses. In some aspects of the technology, the API response may include the data and the one or more templates, and the virtual assistant may be responsible for inserting the data into the template(s) in order to generate the one or more complete templated responses. In some aspects of the technology, the API response may include only selected data, and the virtual assistant may be responsible for generating one or more templated responses based on that data and its own templates. This may include the virtual assistant selecting one or more appropriate templates from a database or other data structure maintained by the developer of the virtual assistant.

Thus, in the present example, the first API may provide a first list of “deli/sandwich” restaurants within the indicated search radius from the first API, and the second API may provide a second list of nearby public transportation hubs from the second API. Here again, the virtual assistant may be configured to identify the highest rated “deli/sandwich” restaurant from the first list and use its address to identify the nearest public transportation hub to that address from the second list. Likewise, as indicated by the dashed arrow connecting steps308and310, in some aspects of the technology, steps308and310may also be repeated as necessary to make all indicated API calls. Thus, for example, the virtual assistant may determine to only call the first API in the initial pass through steps308and310, and then to return to step308so that the second API can be called using the address of the highest rated “deli/sandwich” restaurant in order obtain a more targeted list of public transportation hubs proximate to that restaurant.

In any event, once the highest rated “deli/sandwich” restaurant and the corresponding nearest public transportation hub have been identified, the virtual assistant will insert that information into the basic templates for each API. For example, based on the first API having indicated that “Al's Deli” is the highest rated “deli/sandwich” restaurant in the indicated search radius of 5 miles, and based on the first API having two relevant templates of “[restaurant] is top rated [type] restaurant” and “It is within [radius] of your present location,” the virtual assistant would generate a first templated response of “Al's Deli is top rated deli/sandwich restaurant,” and a second templated response of “It is within 5 miles of your present location.” As can be seen from this example, the individual templates need not be perfect, as the first templated response is not grammatically correct, and uses the somewhat awkward term “deli/sandwich restaurant,” and the second templated response is uninformative by itself due to using “it” in place of the name of the restaurant.

Likewise, based on the second API having indicated that the closest public transportation hub to “Al's Deli” is a bus stop located 203 feet away at the intersection of North Avenue and Center Street, and based on the second API having three relevant templates of “The closest public transportation is [type],” “The address is [address],” and “It is [distance] away,” the virtual assistant would generate a third templated response of “The closest public transportation is bus stop,” a fourth templated response of “The address is North Avenue and Center Street,” and a fifth templated response of “It is 203 feet away.” Here as well, the third templated response is ungrammatical, and the fourth and fifth templated responses are uninformative by themselves due to not repeating the subject (“bus stop”).

As shown in these examples, the templates may be configured to incorporate both data that has been received from each API call (e.g., best restaurant, type of closest public transportation, address of bus stop, distance from final location), as well as other data such as the information that was passed to the API in each API call (e.g., search radius, restaurant type). However, in some aspects of the technology, the APIs may instead be configured to return all information necessary to create each templated response.

Here as well, in some cases the virtual assistant may determine from the API responses that further information is needed. In such cases, the virtual assistant may prompt the user for any such further information, as already explained above with respect to step210ofFIG.2.

In step312, the virtual assistant concatenates all templated responses. Thus, assuming that the virtual assistant generated the five simple templated responses mentioned in the prior example, the virtual assistant would then combine them to create the following concatenated sequence: “Al's Deli is top rated deli/sandwich restaurant. It is within 5 miles of your present location. The closest public transportation is bus stop. The address is North Avenue and Center Street. It is 203 feet away.” As can be seen from this example, a collection of simple templated responses can become even less fluent once combined, and can also become more unclear where individual templates fail to expressly state a subject, e.g., it is not clear in this example whether “The address” pertains to Al's deli or the bus stop.

Finally, in step314, the virtual assistant provides the concatenated sequence generated in step312to the NLG model, which generates a natural language response based on that concatenated sequence. For example, using the illustrative concatenated sequence of step312, the NLG model may return a final natural language response of: “Al's Deli is the best place for a sandwich within 5 miles of your location, and the closest bus stop is 203 feet away at North Avenue and Center Street.” As can be seen from this example, through the smoothing introduced by the learned sequence-to-sequence transformer, an otherwise unnatural and unclear concatenated sequence can be transformed into an utterance that is both fluent and comprehensible. Here again, the resulting natural language response may be provided back to the user either in a textual or audio format, as described above with respect to step216ofFIG.2.

The NLG model of step314may be any model trained in sequence-to-sequence text transformation. For example, the NLG model may be a deep neural network pretrained as a “T5” Text-to-Text Transfer Transformer model so that it can reliably translate a given sequence of text into a different sequence of text that accurately and fluently conveys the same meaning. Here as well, any suitable architecture may be used for the NLG model. Again, for example, in some aspects of the technology, the NLG model may be a T5-small model having 6 layers each in the encoder and decoder, and roughly 60 million parameters in total. Further, any suitable training regimen may be used to train the NLG model to perform sequence-to-sequence text transformation. For example, the NLG model may be pre-trained according to the training regimen described in Raffel et al.,Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,arXiv preprint arXiv:1910.10683, 2019.

Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. As these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of exemplary systems and methods should be taken by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the provision of the examples described herein, as well as clauses phrased as “such as,” “including,” “comprising,” and the like, should not be interpreted as limiting the subject matter of the claims to the specific examples; rather, the examples are intended to illustrate only some of the many possible embodiments. Further, the same reference numbers in different drawings can identify the same or similar elements.