Patent ID: 12223951

DETAILED DESCRIPTION

FIG.1illustrates an overview of a system100for responding to a natural language query. In particular, the example shown inFIG.1illustrates a user101interacting with a user device102, such as a smart speaker, home assistant, a smart TV or a smartphone. However, the user device102may be any appropriate type of user device configured to receive a natural language query, e.g., a spoken query103, from user101. System100may also include network108such as the Internet, configured to communicatively couple user device102to one or more servers104and/or one or more databases106from which data, information, content, such as TV shows, movies and/or advertisement content, may be obtained. In some examples, server104may be configured to process the natural language query, e.g., using control circuitry, and access database106to obtain content for use in providing a response to the natural language query. In the example shown inFIG.1, system100is configured to allow user device102to deliver the response, e.g., an audio response105, to the natural language query. Additionally or alternatively, system100may be configured to allow another user device, e.g., a smartphone110, to deliver the response, e.g., a graphical response109, to the natural language query. User device102(and/or user device110) and the one or more servers104may be communicatively coupled to one another by way of network108, and the one or more servers104may be communicatively coupled to content database106by way of one or more communication paths, such as a proprietary communication path and/or network108. In some examples, server104may be a server of a service provider who provides media content for display on user device102.

The present disclosure, as discussed in detail below, provides improved systems and methods for responding to a natural language query of a user, e.g., by training a first natural language understanding model to deliver a response to a user in a faster and/or more accurate manner. In some examples, the first natural language understanding model is trained by reprocessing a natural language query of a user in response to determining that the first natural language understanding model is unable to deliver a meaningful response to the user query. In the context of the present disclosure, a query that the first natural language understanding model is unable to deliver a meaningful or unsatisfactory response to is referred to as a failed query, e.g., a query having a response below a confidence level threshold. In some example, the reprocessing of a failed query occurs offline, such that the failed query can be run through one or more alternative processing module to generate one or more other interpretations of the query, without further impact on the interaction between the user101and the user device102. Additionally or alternatively, the present disclosure, as discussed in detail below, provides improved systems and methods for responding to a natural language query of a user, e.g., by reprocessing a failed query, e.g., in real-time, as the user101is interacting with the user device102. In this manner, the response to the natural language query may be augmented/supplemented by a response generated by one or more of the reprocessing modules. In some examples, the augmented/supplemented response can be used to train the first natural language understanding model, e.g., as described in the below examples.

In some examples, system100may comprise an application that provides guidance through an interface, e.g., a graphical user interface, that allows users to efficiently navigate media content selections, navigate an interactive media content item, and easily identify media content that they may desire, such as content provided on a database on one or more live streams. Such guidance is referred to herein as an interactive content guidance application or, sometimes, a content guidance application, a media guidance application, or a guidance application. In some examples, the application may be configured to provide a recommendation for a content item, e.g., based on a user profile and/or an endorsement profile of the content item. For example, the application may provide a user with a recommendation based for a content item based on one or more endorsements present, e.g., visibly and/or audibly present, in the content item. In some examples, the application provides users with access to a group watching session and/or group communication functionality. For example, the application may provide a user with an option to join a group watching session and participate in group communication with one or more other users participating in the group watching session.

Interactive media guidance applications may take various forms, depending on the content for which they provide guidance. One typical type of media guidance application is an interactive television program guide. Interactive television program guides (sometimes referred to as electronic program guides) are well-known guidance applications that, among other things, allow users to navigate among and locate many types of content or media assets. Interactive media guidance applications may generate graphical user interface screens that enable a user to navigate among, locate and select content. As referred to herein, the terms “media asset”, “content items” and “content” should each be understood to mean an electronically consumable user asset, such as television programming, as well as pay-per-view programs, on-demand programs (as in video-on-demand (VOD) systems), Internet content (e.g., streaming content, downloadable content, Webcasts, etc.), video clips, audio, content information, pictures, rotating images, documents, playlists, websites, articles, books, electronic books, blogs, chat sessions, social media, applications, games, and/or any other media or multimedia and/or combination of the same. Guidance applications also allow users to navigate amid and locate content. As referred to herein, the term “multimedia” should be understood to mean content that utilizes at least two different content forms described above, for example, text, audio, images, video, or interactivity content forms. Content may be recorded, played, displayed or accessed by user equipment devices, but can also be part of a live performance.

The media guidance application and/or any instructions for performing any of the examples discussed herein may be encoded on computer-readable media. Computer-readable media includes any media capable of storing data. The computer-readable media may be transitory, including, but not limited to, propagating electrical or electromagnetic signals, or may be non-transitory, including, but not limited to, volatile and non-volatile computer memory or storage devices such as a hard disk, floppy disk, USB drive, DVD, CD, media card, register memory, processor cache, random access memory (RAM), etc.

With the ever-improving capabilities of the Internet, mobile computing, and high-speed wireless networks, users are accessing media on user equipment devices on which they traditionally did not. As referred to herein, the phrases “user equipment device,” “user equipment,” “user device,” “computing device,” “electronic device,” “electronic equipment,” “media equipment device,” or “media device” should be understood to mean any device for accessing the content described above, such as a television, a Smart TV, a set-top box, an integrated receiver decoder (IRD) for handling satellite television, a digital storage device, a digital media receiver (DMR), a digital media adapter (DMA), a streaming media device, a DVD player, a DVD recorder, a connected DVD, a local media server, a BLU-RAY player, a BLU-RAY recorder, a personal computer (PC), a laptop computer, a tablet computer, a WebTV box, a personal computer television (PC/TV), a PC media server, a PC media center, a hand-held computer, a stationary telephone, a personal digital assistant (PDA), a mobile telephone, a portable video player, a portable music player, a portable gaming machine, a smartphone, or any other television equipment, computing equipment, or wireless device, and/or combination of the same. In some examples, the user equipment device may have a front-facing screen and a rear-facing screen, multiple front screens, or multiple angled screens. In some examples, the user equipment device may have a front-facing camera and/or a rear-facing camera. On these user equipment devices, users may be able to navigate among and locate the same content available through a television. Consequently, media guidance may be available on these devices, as well. The guidance provided may be for content available only through a television, for content available only through one or more of other types of user equipment devices, or for content available through both a television and one or more of the other types of user equipment devices. The media guidance applications may be provided as online applications (i.e., provided on a website), or as stand-alone applications or clients on user equipment devices. Various devices and platforms that may implement media guidance applications are described in more detail below.

One of the functions of the media guidance application is to provide media guidance data to users. As referred to herein, the phrase “media guidance data” or “guidance data” should be understood to mean any data related to content or data used in operating the guidance application. For example, the guidance data may include program information, subtitle data, guidance application settings, user preferences, user profile information, media listings, media-related information (e.g., broadcast times, broadcast channels, titles, descriptions, ratings information (e.g., parental control ratings, critics' ratings, etc.), genre or category information, actor information, logo data for broadcasters' or providers' logos, etc.), media format (e.g., standard definition, high definition, 3D, etc.), on-demand information, blogs, websites, and any other type of guidance data that is helpful for a user to navigate among and locate desired content selections.

FIG.2is an illustrative block diagram showing exemplary system200configured to respond to a natural language query, e.g., by generating an audio response and/or by displaying content on a user device or user devices. AlthoughFIG.2shows system200as including a number and configuration of individual components, in some examples, any number of the components of system200may be combined and/or integrated as one device, e.g., as user device102. System200includes computing device202, server204(e.g., server106and/or server108), and content database206, each of which is communicatively coupled to communication network208, which may be the Internet or any other suitable network or group of networks. In some examples, system200excludes server204, and functionality that would otherwise be implemented by server204is instead implemented by other components of system200, such as computing device202. In still other examples, server204works in conjunction with computing device202to implement certain functionality described herein in a distributed or cooperative manner.

Server204includes control circuitry210and input/output (hereinafter “I/O”) path212, and control circuitry210includes storage214and processing circuitry216. Computing device202, which may be a personal computer, a laptop computer, a tablet computer, a smartphone, a smart television, a smart speaker, or any other type of computing device, includes control circuitry218, I/O path220, speaker222, display224, and user input interface226, which in some examples provides a user selectable option for enabling and disabling the display of modified subtitles. Control circuitry218includes storage228and processing circuitry220. Control circuitry210and/or218may be based on any suitable processing circuitry such as processing circuitry216and/or220. As referred to herein, processing circuitry should be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores). In some examples, processing circuitry may be distributed across multiple separate processors, for example, multiple of the same type of processors (e.g., two Intel Core i9 processors) or multiple different processors (e.g., an Intel Core i7 processor and an Intel Core i9 processor).

Each of storage214, storage228, and/or storages of other components of system200(e.g., storages of content database206, and/or the like) may be an electronic storage device. As referred to herein, the phrase “electronic storage device” or “storage device” should be understood to mean any device for storing electronic data, computer software, or firmware, such as random-access memory, read-only memory, hard drives, optical drives, digital video disc (DVD) recorders, compact disc (CD) recorders, BLU-RAY disc (BD) recorders, BLU-RAY 2D disc recorders, digital video recorders (DVRs, sometimes called personal video recorders, or PVRs), solid state devices, quantum storage devices, gaming consoles, gaming media, or any other suitable fixed or removable storage devices, and/or any combination of the same. Each of storage214, storage228, and/or storages of other components of system200may be used to store various types of content, metadata, and or other types of data. Non-volatile memory may also be used (e.g., to launch a boot-up routine and other instructions). Cloud-based storage may be used to supplement storages214,228or instead of storages214,228. In some examples, control circuitry210and/or218executes instructions for an application stored in memory (e.g., storage214and/or228). Specifically, control circuitry214and/or228may be instructed by the application to perform the functions discussed herein. In some implementations, any action performed by control circuitry214and/or228may be based on instructions received from the application. For example, the application may be implemented as software or a set of executable instructions that may be stored in storage214and/or228and executed by control circuitry214and/or228. In some examples, the application may be a client/server application where only a client application resides on computing device202, and a server application resides on server204.

The application may be implemented using any suitable architecture. For example, it may be a stand-alone application wholly implemented on computing device202. In such an approach, instructions for the application are stored locally (e.g., in storage228), and data for use by the application is downloaded on a periodic basis (e.g., from an out-of-band feed, from an Internet resource, or using another suitable approach). Control circuitry218may retrieve instructions for the application from storage228and process the instructions to perform the functionality described herein. Based on the processed instructions, control circuitry218may determine what action to perform when input is received from user input interface226.

In client/server-based examples, control circuitry218may include communication circuitry suitable for communicating with an application server (e.g., server204) or other networks or servers. The instructions for carrying out the functionality described herein may be stored on the application server. Communication circuitry may include a cable modem, an Ethernet card, or a wireless modem for communication with other equipment, or any other suitable communication circuitry. Such communication may involve the Internet or any other suitable communication networks or paths (e.g., communication network208). In another example of a client/server-based application, control circuitry218runs a web browser that interprets web pages provided by a remote server (e.g., server204). For example, the remote server may store the instructions for the application in a storage device. The remote server may process the stored instructions using circuitry (e.g., control circuitry210) and/or generate displays. Computing device202may receive the displays generated by the remote server and may display the content of the displays locally via display224. This way, the processing of the instructions is performed remotely (e.g., by server204) while the resulting displays, such as the display windows described elsewhere herein, are provided locally on computing device202. Computing device202may receive inputs from the user via input interface226and transmit those inputs to the remote server for processing and generating the corresponding displays.

A user may issue a query, e.g., a request to view an interactive media content item and/or select one or more programming options of the interactive media content item, to control circuitry210and/or218using user input interface226. User input interface226may be any suitable user interface, such as a remote control, trackball, keypad, keyboard, touchscreen, touchpad, stylus input, joystick, voice recognition interface, gaming controller, or other user input interfaces. User input interface226may be integrated with or combined with display224, which may be a monitor, a television, a liquid crystal display (LCD), an electronic ink display, or any other equipment suitable for displaying visual images.

Server204and computing device202may transmit and receive content and data via I/O path212and220, respectively. For instance, I/O path212and/or I/O path220may include a communication port(s) configured to transmit and/or receive (for instance to and/or from content database206), via communication network208, content item identifiers, content metadata, natural language queries, and/or other data. Control circuitry210,218may be used to send and receive commands, requests, and other suitable data using I/O paths212,220.

FIG.3is a flowchart representing an illustrative process300for training an NLU model to respond to a natural language query, in accordance with some examples of the disclosure.FIG.5is a block diagram showing control circuitry of an exemplary system for reprocessing a natural language query. While the below example shown inFIG.3refers to the use of system100, as shown inFIG.1, and the reprocessing module, as shown inFIG.5, it will be appreciated that the illustrative process shown inFIG.3, and any of the other following illustrative processes, may be implemented on system100, either alone or in combination with any other appropriately configured system architecture, such as system200shown inFIG.2.

At step302, user device102receives a natural language query, e.g., vocal query103, from user101. In the example shown inFIG.1, user device102is configured to transmit the query to server104via network108.

At step304, server104receives the query, e.g., using control circuitry, and processes the query using a first NLU model. For example, the first NLU model may be configured to break down the user's query into one or more entities, intents and/or utterances, and map the breakdown of the query against one or more templates in order to understand the query. For example, user101may issue query A—“Play Tears of the Sun”. In response, the first NLU model may breakdown the query into intent=play, and entity=Tears of the Sun. In such an example, the first NLU model may be able to map the query onto a template intent×entity(movie), and return the movie Tears of the Sun. In some cases, the query may not be quite so clear. For example, the user101may issue query B—“I'd like to watch Willis in that Navy SEAL movie”. In response, the first NLU model may breakdown the query into utterance=I'd like to watch, intent=watch, entity=Willis, and entity=Navy SEAL movie. In such an example, the first NLU model may be able to map the query onto a template utterance(intent)×intent×entity(movie), and return the movie Tears of the Sun. In other cases, the query may be even less clear. For example, the user101may issue query C—“Show me that Bruce Willis war movie”. In response, the first NLU model may breakdown the query into utterance=Show me, intent=show, entity=Bruce Willis, and entity=war movie. In such an example, the first NLU model may be able to map the query onto a template utterance(intent)×intent×entity(person)×entity(movie), and return the movie Hart's War. Although not shown inFIG.3, in response to processing the user's query, control circuitry may then generate and provide to the user a response, e.g., a response comprising one or more search results corresponding to the query, e.g., as described below in more detail with reference to process600shown inFIG.6.

At step306, control circuitry determines a confidence level of the understanding of the user's natural language query. For example, the confidence level may be based on one or more factors, including, but not limited to: one or more actions of the user101after receiving a response to the query, e.g., a user input into a user device, such as a selection of an entity provided in the response and/or navigation though a media guidance application; metadata of one or more entities, e.g., metadata of one or more content items provided in the response; metadata of one or more content items not provided in the response and selected following navigation though a media guidance application. Further detail of how the confidence level is determined is provided below with reference to process600shown inFIG.6.

At step308, control circuitry reprocesses the natural language query using a reprocessing module502(shown inFIG.5), e.g., in response to the determined confidence level being below a confidence level threshold. In some examples, it is beneficial to perform the reprocessing of the failed query offline, e.g., such that the reprocessing does not affect the latency of providing the user with a response to the query. Further detail of how the confidence level is compared to the confidence level threshold is provided below with reference to process600shown inFIG.6.

In the example shown inFIG.5, the reprocessing module502is provided as part of control circuitry, e.g., control circuitry210and/or218, and comprises one or more modules configured to reprocess the natural language query in a manner different from how the first NLU model processes the natural language query. For example, the reprocessing module502may comprise a rule-based module504, a parameter adjustment module506and a natural language understanding module508.

The rule-based module504may be configured to parse the natural language query using i) linguistic rules and patterns, e.g., to identify the structure of a sentence (e.g., subject-verb-object), ii) synonym extraction, e.g., using a thesaurus and/or a list of similar words, and/or iii) sentiment analysis, which analyses linguistic terms to deterministically classify them as positive or negative terms. One advantage of using rule-based processing is that precise results to a query can be returned. However, rule-based processing can be limited by the predetermined rules that are programed into the rule-based module504and may take a longer time to process the query than the first NLU model.

The parameter adjustment module506may be configured to adjust the manner in which the first NLU model maps the query onto a template. For example, the parameter adjustment module506may be configured to adjust a dataset (e.g., decrease/increase the size of a dataset and/or use one or more alternative datasets) used in the processing of the natural language query by the first NLU model. In some cases, it may be inappropriate to use a larger dataset for processing the query at step304, since processing a larger dataset may increase the computational requirements at a server, and thus increase the time taken to generate a response to the user's query. As such, it may be preferable to perform the reprocessing of the query offline. Additionally or alternatively, the parameter adjustment module506may be configured to adjust a time limit (e.g., increase a time limit) for reprocessing the natural language query using the first NLU model. For example, the time limit for the processing of the query at step304may be set according to a service level agreement between a service provider and a client, and increasing the time limit for processing the query at step304may not be possible, in some cases. As such, it may be preferable to perform the reprocessing of the query offline, where the time limit for the processing of the query may be increased without affecting the latency for the query. In this manner, the first NLU may be able to generate one or more different interpretations of the query when reprocessing the query with adjusted parameters.

The natural language understanding module508may be configured to issue an application programming interface (API) call to one or more other systems configured to run a different NLU model (e.g., a second NLU model). For example, the second NLU model may have access to richer data sources, which can improve the granularity at which the model can determine an entity and/or an intent in a query. Additionally or alternatively, the second NLU model may have access to a domain that the first NLU model does not have access to. In this manner, the second NLU model may be able to generate a different interpretation of the query, e.g., by mapping the query onto a template not present in the first NLU model. In some examples, querying a second NLU model may be based on the original query, such that the second NLU model can target a specific part of the original query. For example, a first NLU model may issue query a second NLU model, e.g., in real-time to confirm an intent (intent recognition), an entity (entity recognition) and/or to confirm a sentiment (sentiment analysis), etc.

At step310, control circuitry is configured to update the first natural language model based on the reprocessing of the natural language query by the reprocessing module502. For example, where the reprocessing module502has generated one or more alternative interpretations of the query, a new template may be generated by the reprocessing module502. The new template may be used to update the templates of the first NLU model, such that a subsequent query received by user device102may be processed in a manner not previously possible. In this manner, the first NLU model is trained, using the reprocessing module, in response to a failed query.

The actions or descriptions ofFIG.3may be used with any other example of this disclosure, e.g., the example described below in relation toFIGS.4and6. In addition, the actions and descriptions described in relation toFIG.3may be done in any suitable alternative orders or in parallel to further the purposes of this disclosure.

FIG.4is a flowchart representing an illustrative process400for responding to a natural language query, in accordance with some examples of the disclosure.FIG.5is a block diagram showing control circuitry of an exemplary system for reprocessing a natural language query. While the below example shown inFIG.4refers to the use of system100, as shown inFIG.1, and the reprocessing module, as shown inFIG.5, it will be appreciated that the illustrative process shown inFIG.4, and any of the other above or following illustrative processes, may be implemented on system100, either alone or in combination with any other appropriately configured system architecture, such as system200shown inFIG.2.

At step402, user device102receives a natural language query, e.g., vocal query103, from user101, e.g., in a similar manner to that described for step302of process300.

At step404, server104receives the query, e.g., using control circuitry, and processes the query using a first NLU model, e.g., in a similar manner to that described for step302of process300.

At step406, control circuitry determines a confidence level of the understanding of the user's natural language query, e.g., in a similar manner to that described for step302of process300.

At step408, control circuitry reprocesses the natural language query using a reprocessing module502(shown inFIG.5), e.g., in response to the determined confidence level being below a confidence level threshold, e.g., in a different manner to that described for step302of process300. One important difference between step408and step308is that the reprocessing of the query occurs in real-time (not off-line), e.g., so that the output from the reprocessing of the query can be provided to the user along with, e.g., at substantially the same time as, the output from the processing at step404.

At step410, control circuitry is configured to generate a response to the query, e.g., by providing to the user one or more search results relating to the query. For example, where the reprocessing module502has generated one or more alternative interpretations of the query, the response may be generated based on the processing and the reprocessing of the query. While it is acknowledged that the reprocessing of the query (step408) may take longer than the processing of the query (step404), one benefit of providing the output of the reprocessing to the user is that any search results generated as a result of the processing at step404may be updated, supplemented or augmented based on the reprocessing at step408, which can help reduce network traffic resulting from the user performing further user inputs to user device102, e.g., to perform one or more additional searches. In some examples, a client application (e.g., an application that the user utilizes to issue the query) supports a single page application (SPA) architecture, which would allow the client to receive and automatically provide content in addition to that provided as a result of the processing by the first NLU model without reloading the page. Similarly, the reprocessing module502may be configured to send the additional content directly to the client application if the first NLU model and the reprocessing module502(and/or the second NLU model) supports the exchange of existing query sessions, e.g., by exchanging device address or other destination data. In some examples, the client application might display an indication to the user that additional information relating to the query is coming. This is useful when the user receives a first set of content to scroll through (e.g., as a result of processing at step404), and then a second set of content is displayed when available from the reprocessing module502(e.g., as a result of reprocessing at step408).

The actions or descriptions ofFIG.4may be used with any other example of this disclosure, e.g., the example described below in relation toFIGS.3and6. In addition, the actions and descriptions described in relation toFIG.4may be done in any suitable alternative orders or in parallel to further the purposes of this disclosure.

FIG.6is a flowchart representing an illustrative process600for responding to a natural language query, in accordance with some examples of the disclosure.FIG.5is a block diagram showing control circuitry of an exemplary system for reprocessing a natural language query. While the below example shown inFIG.6refers to the use of system100, as shown inFIG.1, and the reprocessing module, as shown inFIG.5, it will be appreciated that the illustrative process shown inFIG.6, and any of the other above illustrative processes, may be implemented on system100, either alone or in combination with any other appropriately configured system architecture, such as system200shown inFIG.2.

At step602, user device102receives a natural language query, e.g., vocal query103, from user101, e.g., in a similar manner to that described for step302of process300and/or step402of process400. For example, the user101may issue a query “Show me Bruce's war movie”.

At step604, control circuitry, e.g., control circuitry210and/or218, determines whether the query matches one or more stored query templates, e.g., one or more templates that are stored on storage214and/or228. For example, the first NLU model may breakdown the query into syntactic components, e.g., utterance=Show me, intent=show, entity=Bruce, and entity=war movie, and control circuitry may access a storage module to compare the query breakdown against one or more stored templates, such as Template A—intent×entity(movie), Template B—utterance(intent)×intent×entity(movie) and/or Template C—utterance(intent)×intent×entity(person)×entity(movie). When control circuitry is able to match the query to a template, process600moves to step606, since, in this example, the syntactic components of the query match the components (and/or the order of the components) of Template C. When control circuitry is not able to match the query to one or more templates, process600moves to step633(see arrow A).

At step606, the first NLU model processes the natural language query using the matching template. For example, control circuitry may access content database106and/or206and compare the syntactic components of the query, e.g., the entities of the query, to metadata of one or more content items. For example, control circuitry may compare entity=Bruce and entity=war movie to metadata of content items, and return a match for the movie Robert the Bruce, which is a war movie concerning the historical king of the same name.

At step608, control circuitry generates a response to the query. For example, the response may be generated as an audio response105, e.g., “Would you like to watch Robert the Bruce?”, and/or as a search result displayed on a screen of user device102, e.g., as a link to playback the movie Robert the Bruce.

At step610, control circuitry determines a confidence level of the understanding of the natural language query, e.g., in a similar manner to that described for step306of process300and/or step406of process400. In the example shown inFIG.6, step610comprises steps612,614,616,618and620. For the avoidance of doubt, step610may comprise any appropriate combination of steps612,614,616,618and620. Step610is important, since is desirable to determine a confidence level of the understanding of the natural language query to help ensure that the first NLU model has correctly understood the user's intent, e.g., the first NLU model might incorrectly determine a user's intent/return unexpected results from a given query due to factors such as mispronunciation of an important keyword in the query, and/or the first NLU model predicting the wrong user's intent based on the then-available training dataset for the system, e.g., the then-available templates and/or metadata for the content items. As such, step610is implemented to help determine a level of accuracy of the generated response, since users may perform many different types of operations following generating a response. For example, even where the first NLU model has generated accurate results, a user may simply wish to not progress further with playback of a content item returned as part of the response, whereas in other situations, the generated response may be completely inaccurate and not at all what the user expects. In some examples, a user may issue a voice query for content items but then does not interact with the content items that were retrieved by the search system (e.g., does not select them for viewing), rather the user resorts to manual navigation instead. In such an example, NLU system100may determine that the query was a failure and may tag the query as a failed query, so that it can be reinterpreted offline and used to retrain the NLU system100. In some examples, one or more subsequent navigation steps (e.g., using a remote control) within a predetermined period are captured and correlated to the original voice query for retraining purposes. This allows the mapping of the various entities (actor, movie, broadcast time, etc.) present in the original voice query during retraining. For example, a user might issue the voice query “Show me Bruce's war movie”. Such a query might result in presenting search results that do not match the user's intent, e.g., by generating a response comprising the movie Robert the Bruce, which is a war movie concerning the historical king of the same name, instead of Tears of the Sun, which is a movie about the Nigerian civil war starring Bruce Willis. For example, the user's intent here was to see the movie Tears of the Sun which features the actor, Bruce Willis. The user might take an action after such a failed query. For example, the user might browse to the on-demand section of their TV service, and type in the name of the movie or a portion of it, or browse by actor, genre, etc. to find the movie and play it. All such data can be used to retrain the NLU system100. Additionally or alternatively, the accuracy of the mapping of various entities onto the metadata of a content item may be checked against the metadata of an item (including EPG metadata such as broadcast time or availability) that was eventually selected for viewing by the user (e.g., the first selected content item after issuing the original query) to determine whether the words in the query exist in the metadata of the selected content item (e.g., description, list of actors, etc.). This can act as a confirmation step. The confirmation step is important since the user might have decided to browse for some other content items after ignoring the initial results from the first NLU model (results that were returned in response to the user's original voice query). Therefore, the query might not necessarily be classified as a failure and does not need to be used in retraining the NLU system100. In some examples, the user may select multiple items before deciding to play one. In such a scenario, the metadata associated with the multiple items can be used to attempt mapping the various entities in the voice query (e.g., actor, movie name, broadcast time, etc.). Therefore, multiple selections can be used during the retraining process to perform accurate mapping.

In view of the above, at step612, control circuitry monitors user input (e.g., a verbal instruction and/or physical operation) to one or more user devices, e.g., user device102and/or user device110. For example, control circuitry may monitor a user's operation of/navigation through a media guidance application, e.g., following the generation of the response to the query at step608. In some examples, control circuitry monitors a number of and/or type of inputs to the user device. For example, control circuitry may monitor and determine user input to a user device and determine the number of and/or the type of inputs to the user device, e.g., for a predetermined period, e.g., 5 second, or 30 seconds, following generation of the response at step608. In some cases, the user may perform a series of operations, e.g., scrolling through a list of content items returned as part of the response and/or navigation away from the list of content items returned as part of the response and to other content.

At step614, control circuitry compares the determined user input, e.g., the number of and/or the type of inputs, to a user input threshold. For example, a user input threshold may be defined by a maximum number of and/or type of inputs to a user device, such as a maximum number of button presses, screens navigated through, operations selected and/or verbal instructions issued. For example, in response to generating an inaccurate response to the query, a user may operate a media guidance application to scroll through multiple screens, perform manual searches, access content item data (such as a synopsis, cast details, etc.). In some examples, the user input threshold may be set at a maximum number of button presses (e.g., 10 scrolling operations through a list of search results) in combination with the number of times (e.g., 3 times) content item information is accessed for various content items listed in the search results. In this manner, the user input threshold may be set to indicate when a number of actions performed by the user exceeds a likely number of inputs that a user may take when presented with accurate search results. When the user input is above the user input threshold, process600moves to step622(see arrow B). In some cases, in response to generating an accurate response to the query, a user may operate a media guidance application to select a content item returned as part of the response to the query. For example, a user may select for playback the first of multiple content items presented to the user as part of the response to the query. As such, the number and/or type of user inputs following an accurate query response may be less/fewer than the number and/or type of user inputs following an inaccurate query response. In other words, the manner in which a user responds to the query response generated at step608may give an indication of how accurate the query response is. For example, the accuracy of the response generated at step608may correlate to the number and/or type of user inputs following the query response being generated at step608. When the user input is below the user input threshold, process600moves to step615, where control circuitry generates and stores a high confidence level for the query, since the response generated at step608is likely to be accurate and the first NLU model does not require training. In some examples, the confidence level generated at step615may be used in step determining an average confidence score when processing a subsequent query, e.g., at step628.

Returning to step612, the user input to the user device is monitored. At step616, control circuitry receives a user selection of an entity, e.g., a content item, provided as part of the generated response, and then determines metadata for the user-selected entity. For example, control circuitry may monitor user input into a user device, determine that the user has scrolled through multiple screens, performs a manual search and then receives a user selection of a content item that was not presented as part of the generated response. Using the above example of a response comprising the content item Robert the Bruce, upon receiving the response, the user may navigate away from the response, access a search function of a media guidance application, search for movies starring Bruce Willis, and scroll through a list of search results to find and select the movie Tears of the Sun. In some examples, step614is carried out at the same time as, or in response to, step616. Once control circuitry receives a user selection of the movie Tears of the

Sun, control circuitry then accesses metadata for Tears of the Sun for use in determining a confidence level of the generated response.

Additionally or alternatively, at step618, control circuitry accesses metadata for an entity provided in the response. For example, where the generated response comprises the content item Robert the Bruce, control circuitry then accesses metadata for Robert the Bruce for use in determining a confidence level of the generated response. In some examples, step618is carried out at the same time as, or in response to, step616.

At step620, control circuitry compares the metadata of the entity provided in the response to the metadata of the selected entity, e.g., to determine a level of accuracy of the generated response. For example, control circuitry may compare metadata (e.g., tags for Bruce Willis, US Navy SEAL, Nigerian civil war, etc.) for the user-selected content item (Tears of the Sun) to metadata (e.g., tags for Robert the Bruce, historical war movie, etc.) for the content item provided as part of the response (Robert the Bruce). In this manner, control circuitry can determine whether there is a correlation between the content item provided as part of the response (Robert the Bruce) and the user-selected content item (Tears of the Sun) to determine a level of accuracy of the generated response. In another example, the generated response may comprise the content item Tears of the Sun, and the user may still choose to navigate away from the generated response, e.g., when they might have changed their mind about wanting to watch Tears of the Sun. As such, step610(and sub-steps612-620) is important in determining whether the generated response is accurate, so that a determination can be made as to whether the generated query response is an accurate response or an inaccurate response (e.g., a failed query). In some examples, the decision on whether to reprocess the query depends on the level of accuracy of the response, e.g., on a correlation between metadata for the user-selected content item and metadata for the content item provided as part of the response. In the above example, the correlation between the metadata for the movie Robert the Bruce and the movie Tears of the Sun may be relatively low, since the two movies share few attributes, whereas the correlation between the metadata for another movie, e.g., Hart's War, and the movie Tears of the Sun may be higher, since they both star Bruce Willis. Thus, at step620, when correlation between metadata for the user-selected content item and metadata for the content item provided as part of the response meets a minimum predetermined level, process600moves to step615, where control circuitry generates and stores a high confidence level for the query, since the response generated at step608is likely to be accurate and the first NLU model does not require training (e.g., the response has been determined to be with an allowable accuracy range), whereas when correlation between metadata for the user-selected content item and metadata for the content item provided as part of the response does not meet a minimum predetermined level, process600moves to step622(see arrow B).

At step622, control circuitry generates a confidence level, e.g., a value assigned to the likelihood that the generated response is accurate, based on one or more of steps612-620. The confidence level for the generated query is stored in a database for later access.

Steps624,626,628,630and632define a process that can be used to determine if system100should send a failed query for reprocessing, e.g., by determining a reinforcement score for the query. The reinforcement score may be used as an indicator for whether to reprocess a failed query, since it may not be desirable to reprocess every failed query, e.g., owing to a computational demand of the reprocessing. As such, process600comprises steps, e.g.,624,626,628,630and632, that work to optimize the allocation of resource (e.g., to reprocess or to not reprocess) by determining a probability that the reprocessing might provide a different interpretation from the generated response, which can be used to supplement the generated response and/or to retrain the first NLU model.

At step624, control circuitry determines the identity of the user who generated the natural language query. For example, control circuitry may access a user profile, e.g., at step626, to determine a user who is signed-in to system100. In some examples, control circuitry may analyse a vocal signature of a user and compare the vocal signature to one or more stored and labelled vocal signatures to determine the identity of the user who issued a vocal query. Additionally or alternatively, control circuitry may utilize one or more image processing techniques to determine the identity of the user who generated the natural language query, e.g., by analyzing one or more images/videos, which may be captured by a user device, such as user device102and/or104as the user issues the query.

At step628, control circuitry determines an average confidence score for previous natural language queries of the identified user. For example, control circuitry may access stored confidence levels, such as those generated at step615and/or step622, for respective previous queries of the user, and determine an average confidence score for the stored confidence levels. In some examples, a high confidence level may be assigned a value=1, and a low confidence level may be assigned a value=0. Alternatively, a confidence level range may be set as appropriate, e.g., a range having appropriate upper and lower limits, e.g., based on the user input threshold and/or a correlation coefficient between the metadata of the entity provided in the response and the metadata of the selected entity. In some examples, the average confidence score (determined as an average of the sum of the confidence levels of respective generated responses) represents the success rate of the previous queries (e.g., queries in a specific domain such as a media domain) that have been processed using the first NLU model (e.g., a primary NLU model). A high average confidence score associated with queries issued by the identified user could indicate that the identified user's queries are normally clear, articulate and concise. Thus, a new query from the identified user that generates a low confidence level compared to the average confidence score might be a query that is suitable for reprocessing (e.g., using at least a secondary NLU model). For example, when a generated confidence level for a query is above (or within a predetermined range of) a threshold confidence score, system100may determine to not reprocess the query. Additionally or alternatively, the user profile may record when a user issues a joke query that lack seriousness. For example, sometimes a user may mess around with a digital assistant, e.g., by asking goofy questions, such as “Are you married?”, “What's your nationality?”, etc. As such, where a new query from the identified user is tagged as a joke query, system100may determine to not reprocess the query.

At step630, control circuitry determines whether the confidence level (generated at622) is less than the average confidence score (determined at628). In response to determining that the confidence level is greater than the average confidence score, process600moves to step632, which is described below in more detail. In response to determining that the confidence level is less than the average confidence score, process600moves to step633, where the query is reprocessed.

At step633, control circuitry reprocesses the query, e.g., in a similar manner to that described for step308of process300and/or step408of process400. In the example shown inFIG.6B, step633comprises steps634,636,638,640,642and644, while other examples may comprise any appropriate combination of these steps.

At step634, reprocessing module502reprocesses the query using one or more rule-based modules, e.g., module504as described above.

At step636, reprocessing module502determines a domain of the query received at step602. For example, reprocessing module502may determine that the query “Show me the Bruce war movie” relates to a domain comprising actors called Bruce and war movies, e.g., by fitting the query to a template, such as Template D—utterance(intent)×intent×entity(actor)×entity(movie genre).

At step638, reprocessing module502selects a second NLU model based on the determined domain. For example, the reprocessing module502may have access to multiple NLU models, some of which may be better suited to processing the query than the first NLU model (and/or others of the multiple NLU models). As such, step638acts to increase the probability that the reprocessing of the query returns a more accurate response (e.g., a response having a higher confidence level).

At step640, the reprocessing module502reprocesses the query using the second NLU model, e.g., using NLU module508as described above.

At step642, parameter adjustment module506increases a time limit for the processing the natural language query and/or increases a size of a dataset used for the processing the natural language query, e.g., in a similar manner as described above in relation to processes300and400.

At step644, the query is reprocessed using the first NLU model having one or more adjusted parameters, e.g., in a similar manner as described above in relation to processes300and400.

At step645, a new confidence level is generated and stored for the reprocessed query, e.g., using methodology similar to that outlined for step610above. For example, the low confidence level generated at622may have been improved upon by one or more of the reprocessing steps634-644. Step645moves back to step630, where the new confidence level is compared to the average confidence score.

Returning to step630, control circuitry determines whether the newly generated confidence level of the reprocessed query is less than the average confidence score. Where the newly generated confidence level of the reprocessed query is still less than the average confidence score, process600repeats step633and step645, e.g., until the newly generated confidence level of the reprocessed query is greater than the average confidence score, or until a predetermined number of reprocessing cycles have occurred. Where the newly generated confidence level of the reprocessed query is has increased to be greater than the average confidence score, process600moves to step632.

At step632, control circuitry determines whether the query has already undergone reprocessing, e.g., as at least part of step633. For example, where a new query, e.g., the query received at step602, generates a low confidence level that is not less than the average confidence score, step632acts to optimise operational efficiency by determining to not reprocess that query, and process600moves to step608, where the response to the query is generated (see arrow D). However, where the query has been reprocessed to increase the confidence level, e.g., to above the average confidence level, process600moves to step646, or optionally, directly to step652, (see arrow C).

At step646, control circuitry updates the response generated at step608, e.g., with one or more search results having a higher confidence level, e.g., higher accuracy, than search results presented as part of the original response. For example, search results presented as part of the original response may display a user-selectable option to view the movie Robert the Bruce, whereas search results presented as part of the updated response may display user-selectable options to view the movies Robert the Bruce, Tears of the Sun and Hart's War. In some examples, a client application (e.g., an application that the user utilizes to issue the query) supports a single page application (SPA) architecture, which allows the client to receive and automatically display updated/additional response content, e.g., additional content received from the reprocessing module502, without reloading a page, e.g., a page on which the response generated at step608is displayed. In some examples, the reprocessing module502may be configured to transmit updated/additional response content directly to the client application if control circuitry for the first NLU model and control circuitry for the reprocessing module502support the exchange of existing query sessions (e.g., by exchanging device address or other destination data). In one example, the client application might display an indication that additional information might be coming, e.g., within a certain time frame, which is useful when the user receives a first set of content to scroll through (i.e., the generated response at step608), and then a second set of content is displayed when available from the reprocessing module502(i.e., the generated updated response at step646).

At step648, control circuitry monitors user input (e.g., in a similar manner to that described for step612) and determines a user selection of an entity provided in the updated response. For example, control circuitry may monitor user inputs as the user scrolls through a list of search results comprising one or more content item provided by the processing of the query by the first NLU model (e.g., Robert the Bruce), and one or more other content items provided by the processing of the query by the reprocessing module502(e.g., Tears of the Sun and Hart's War).

At step650, upon user-selection of one of the content items provided as part of the updated response, control circuitry determines whether the selected content item was provided by the processing of the query by the first NLU model or by the processing of the query by the reprocessing module502. In response to determining that the selected content item was provided by the processing of the query by the first NLU model, process600ends. In some examples, control circuitry may store the decision of the user to not select one of the content items provided by the processing of the query by the reprocessing module502, e.g., so that reprocessing module502is able to perform step633in a different manner, such as by using a different second NLU model, using one or more different and/or additional rule-based processing methods, and/or by adjusting the parameters used by the first NLU model in a different manner. In this way, a different set of content items may be presented by reprocessing of a similar, subsequent query. In some examples, process600may move from step650again to step633to reprocess the query in a different manner, e.g., to provide a further updated response. On the other hand, in response to determining that the selected content item was provided by the reprocessing of the query by the reprocessing module, process600moves to step652. In some examples, control circuitry may cause display of the user-selected content item following step648or step650.

At step652, control circuitry generates a new template, e.g., a first new template, for the first natural language understanding model, e.g., based on the user-selection being a content item being provided by the reprocessing of the query by the reprocessing module502. For example, control circuitry may cause a new template to be generated that corresponds to a template used by the reprocessing module502when reprocessing the query. For example, where the reprocessing module502used a template that was not available to the first NLU model when originally processing the query (e.g., Template D), control circuitry may generate a new template for the first NLU model that corresponds to Template D. In some examples, process600may comprise generating one or more additional new templates based on the first new template. For example, control circuitry may generate one or more additional new templates in a language different from the language of the first new template, which acts to automatically diversify the number and type of templates available to the first NLU model, e.g., without performing any reprocessing steps. In this manner, process600acts to predict a future intent of a larger population that has various accents, lives in different geographical areas, demographics, etc. In some examples, control circuitry may group queries in specific domains, by locations and/or by certain demographics since users within the same demographics might search for content in the same manner, style, etc. In some examples, control circuitry defines one or more subsets of users based on historic queries issued in specific domains, by certain locations and/or by certain demographics. In such cases, the average confidence score (used in step628) may be associated with one or more subsets corresponding to the user's query, such that the confidence level of the generated response can be compared to a confidence score of historic queries from a similar domain, location and/or demographic.

At step654, control circuitry updates the templates available to the first NLU model with the new template(s), so that the new template(s) can be used by the first NLU model when processing one or more subsequent queries. In this manner, process600provides a method of training the first NLU to handle new types of queries, e.g., queries in languages and/or in domains not previously known to the first NLU model.

At step656, control circuitry receives a subsequent natural language query, where process600moves back to step604, where control circuitry attempts to match the subsequent query to the set of updated templates. In this manner, a continuous process is enabled whereby the first NLU model continues to be trained, such that the accuracy of generated responses may improve upon receiving each subsequent query.

The actions or descriptions ofFIG.6may be used with any other example of this disclosure, e.g., the example described above in relation toFIGS.3and4. In addition, the actions and descriptions described in relation toFIG.6may be done in any suitable alternative orders or in parallel to further the purposes of this disclosure.

The processes described above are intended to be illustrative and not limiting. One skilled in the art would appreciate that the steps of the processes discussed herein may be omitted, modified, combined, and/or rearranged, and any additional steps may be performed without departing from the scope of the invention. More generally, the above disclosure is meant to be exemplary and not limiting. Only the claims that follow are meant to set bounds as to what the present invention includes. Furthermore, it should be noted that the features and limitations described in any one example may be applied to any other example herein, and flowcharts or examples relating to one example may be combined with any other example in a suitable manner, done in different orders, or done in parallel. In addition, the systems and methods described herein may be performed in real time. It should also be noted that the systems and/or methods described above may be applied to, or used in accordance with, other systems and/or methods.