PROCESSING VOICE INPUT IN INTEGRATED ENVIRONMENT

Systems and methods are described for causing a device to perform an action based on a voice command. Devices connected to a localized network and capable of performing one or more actions based on one or more voice inputs may be identified, and device state information for each of the devices may be determined. The systems and methods may determine, based at least in part on the device state information, a predicted voice command, and a particular device of the plurality of devices for which the predicted voice command is intended. A voice input may be received, and based on receiving the voice input, the particular device may be caused to perform an action related to the predicted voice command.

BACKGROUND

This disclosure is directed to systems and methods for causing a particular device to perform an action related to a predicted voice command. In particular, based at least in part on device state information, the predicted voice command, and the particular device, may be determined.

SUMMARY

Many users have become accustomed to interacting with digital assistants (e.g., voice-based, text-based, a combination thereof, etc.). For example, a user may request a digital assistant to play music, find local restaurants in his or her area, or provide a weather report. Digital assistants are also becoming an integral part of connected home and home automation solutions. For example, a digital assistant may receive analog voice signals, convert the analog signals into a digital signal, perform speech recognition, perform a web search, infer the intent of the user, and take action or generate a response. The action may be to send a specific command to another connected device or generate an audio or visual response.

However, as more and more devices are able to support voice commands and voice interactions, there are growing concerns among a large segment of users about privacy, particularly when captured voice data is sent by an always-on digital assistant to a cloud or remote server for processing. Moreover, there are costs (e.g., usage of computing resources of service providers and users, financial costs for service providers, and time required to perform processing) associated with accessing cloud services of a service provider. One approach attempts to address privacy concerns by performing processing locally without sending sensitive user data away from the user’s home. However, the capacity of such local processing may be limited, e.g., generally only a limited number of voice commands can be processed locally.

Many digital assistants require the user to utter a wake word (e.g., “Hey Siri”) in order to activate and interact with the digital assistant. However, the user may forget to utter such wake word when attempting to use the digital assistant, or may be frustrated with having to use such wake word, which detracts from the user experience, at least in part because the usage of a wake word may be unnatural for the user in what is intended to be a simulated conversation. In one approach, a digital assistant on a designated device may provide a limited set of commands that can be processed locally without requiring the usage of a wake word in association with the command. However, in such approach, the set of commands is limited to a specific device and requires a user to manually specify the set of commands for such device.

To overcome these problems, systems and methods are provided herein for identifying a plurality of devices connected to a localized network and capable of performing one or more actions based on one or more voice inputs. The systems and methods may comprise control circuitry configured to determine device state information for each of the plurality of devices. The control circuitry may determine, based at least in part on the device state information, a predicted voice command, and a particular device of the plurality of devices for which the predicted voice command is intended. The control circuitry may receive a voice input, and based on receiving the voice input, cause the particular device to perform an action related to the predicted voice command.

Such aspects may provide a dynamic, flexible system in which device state information is analyzed to determine a predicted voice command for a particular device, even if voice input is not yet received, or is received and does not specify a device or specific command. The provided systems and methods may leverage collective voice processing capabilities of connected devices within an environment (e.g., a home network) and minimize the cost associated with processing voice input (e.g., automatic speech recognition and intent identification). The system may leverage the interconnectivity of disparate voice assistant devices or other devices, and monitor dynamic criteria, such as, for example, device state information of such devices, in a networked environment, to disambiguate or predict voice inputs and/or commands and/or intended devices.

In some embodiments, the system may maintain temporal and local knowledge graphs locally for faster processing. Such knowledge graphs may be built based on compilation and parsing of voice shortcuts associated with connected devices identified in the localized network, and based on received node and connection data, inferred in the cloud based on the device state information (e.g., active device states and/or metadata) related to the identified devices. The knowledge graph may have limited nodes and connections representing recognized words, commands, recognized entities, recognized intent. In some embodiments, the number of nodes of the local knowledge graph may be limited, and the system may continuously update the knowledge graph based on the device state information to minimize the user of computing resources. For example, instead of copying all nodes and connections related to a particular context from the cloud, the system may selectively add specific nodes and connections and otherwise perform updating of the knowledge graph, based on inferences (e.g., prediction of potential commands, and/or comments, and/or instructions and/or intents) made by the cloud server. As another example, the provided systems and methods, in predicting voice commands, may leverage the observation that many commands originating from a particular location (e.g., a home) may have similar phonetics and repeated voice characteristics. The system may improve accuracy and processing speed in recognizing the commands/queries as the system may be trained for a specific set of users having similar phonetics and speech patterns, and/or based on analyzing and performing matching with respect to a more limited set of alternatives.

In some embodiments, voice shortcuts may be dynamically updated based on the present state of the connected devices, e.g., for local and/or remote processing. In some embodiments, the system may perform parsing of voice input and/or voice commands, to decouple device names from the parsed input or command and associate multiple devices to a particular command. In some embodiments, the system may collectively utilize microphones of the connected devices and perform routing of voice data between different devices and/or services. In some embodiments, the system may route voice data to a particular voice assistant device or service in the absence of another voice assistant device or service being available, e.g., based on analyzing profiles of the user and/or devices and/or services. In some embodiments, the system may enable users to avoid wake words, and avoid the requirement that users be descriptive or specific with respect to their voice input, since the system may maintain an updated list of voice shortcuts pertinent to the current device state information. In some embodiments, if a conflict in recognizing an entity emerges with respect to a current voice input, the system can take advantage of a last resolved, or recently resolved, phonetically similar word(s) as reference.

In some embodiments, each of determining the predicted voice command, and determining the particular device, is performed prior to receiving the voice input, and causing the particular device to perform the action related to the predicted voice command is performed in response to determining the received voice input matches or is related to the predicted voice command. At least a portion of the determining of the predicted voice command, and the determining of the particular device, may be performed locally on one or more of the plurality of devices connected to the localized network.

In some aspects of this disclosure, each of determining the predicted voice command, and determining the particular device, is performed in response to receiving the voice input and based at least in part on processing the voice input. At least a portion of the determining of the predicted voice command, the determining of the particular device and the processing of the voice input, may be performed locally on one or more of the plurality of devices connected to the localized network.

In some embodiments, determining the particular device further comprises determining a first candidate device of the plurality of identified devices and a second candidate device of the plurality of identified devices, for which the predicted voice command is intended. Determining the particular device may further comprise determining that a user associated with the voice input is located closer to the first candidate device than the second candidate device, and therefore identifying the first candidate device as the particular device.

In some aspects of this disclosure, determining that the user associated with the voice input is located closer to the first candidate device than the second candidate device is performed based at least in part on wireless signals. Wireless signals may be received over the localized network by networking equipment from the first candidate device, and wireless signals may be received over the localized network by the networking equipment from the second candidate device

In some embodiments, each of determining the predicted voice command, and determining the particular device, comprises transmitting the device state information to a server, and receiving from the server an indication of the predicted voice command and an indication of the particular device.

In some aspects of this disclosure, the systems and methods provided herein further comprise generating a knowledge graph comprising a respective node for each of at least a subset of the plurality of identified devices, and updating the knowledge graph to comprise a relationship between a node representing the particular device and a node representing the predicted voice command. The updated knowledge graph may be used to determine the predicted voice command and to determine the particular device. In some embodiments, the knowledge graph may be updated to include a relationship between a node associated with the determined device state information for the particular device and a node representing the predicted voice command. At least a portion of the knowledge graph may be stored locally at one or more of the plurality of devices connected to the localized network.

In some embodiments, the device state information comprises, for each respective device of the plurality of devices, one or more of an indication of whether the device is turned on; an indication of current settings of the device; an indication of voice processing capabilities of the device; an indication of one or more characteristics of the device; an indication of an action previously performed, currently being performed or to be performed by the device; or metadata related to a media asset being played via the device.

In some aspects of this disclosure, the systems and methods provided herein further comprise generating a list comprising a first signature word that is associated with voice inputs for a first device of the plurality of devices, and a second signature word that is associated with voice inputs for a first device of the plurality of devices. The systems and methods may determine, at the first device, that the voice input comprises the second signature word, and in response to determining that the voice input comprises the second signature word, perform processing of the voice input at least in part at the first device.

DETAILED DESCRIPTION

FIG.1shows an illustrative environment100in which a predicted voice command, and a particular device for which the predicted voice command is intended, may be determined, in accordance with some embodiments of this disclosure. Environment100may be a particular physical location (e.g., a household of user102and/or user104, a place of business, an office, a school, an other organization, or any other suitable location, or any combination thereof). Environment100may comprise any suitable number and types of computing devices, e.g., smart television106,108; digital assistant110,112; mobile device114(e.g., smartphone, tablet, smart watch, and/or any other suitable mobile device); networking equipment116; Internet of Things (IoT) devices (e.g., smart refrigerator118, smart lamp120, security cameras, and/or any other suitable IOT device); speakers123; a biometric device; a desktop computer; laptop computer; virtual reality (VR) device; augmented reality (AR) device; and/or any other suitable device(s). In some embodiments, at least some of such devices may be configured to be connected to a localized network (e.g., a home network, a business network, etc., facilitated at least in part by networking equipment116) and may be capable of receiving and processing voice inputs and/or voice commands and/or voice queries.

The computing devices of environment100may be configured to be connected to a network over which voice inputs and/or voice commands and/or voice queries may be received and subsequently processed. For example, such devices may be equipped with microphones and suitable circuitry to receive and process speech or voice input received from user102and/or user104and/or any other suitable user and/or transmitted via any suitable device. In some embodiments, the computing devices in environment100may be equipped with antennas for transmitting and receiving electromagnetic signals at frequencies within the electromagnetic spectrum, e.g., radio frequencies, to communicate over a network in environment100. The network may correspond to, e.g., a Wi-Fi network, such as, for example, 802.11n, 802.11ac, 802.11ax, or Wi-Gig/802.11ad. The devices of environment100may communicate wirelessly over a wireless local area network (WLAN) and with the Internet, and may be present within an effective coverage area of the localized network. The Internet may include a global system of interconnected computer networks and devices employing common communication protocols, e.g., the transmission control protocol (TCP), user datagram protocol (UDP) and the Internet protocol (IP) in the TCP/IP Internet protocol suite. In some embodiments, the devices of environment100may additionally or alternatively be configured to communicate via a short-range wired or wireless communication technique (e.g., Bluetooth, RFID, NFC, or any other suitable technique, or any combination thereof).

Networking equipment116may comprise a router, configured to forward data packets from the Internet connection, received by way of a modem, to devices within the localized network of environment100and receive data packets from such devices. In some embodiments, networking equipment116may include a built-in modem to provide access to the Internet for the household (e.g., received by way of cable or fiber connections of a telecommunications network) In some embodiments, networking equipment116may include built-in switches or hubs to deliver data packets to the appropriate devices within the Wi-Fi network, and/or built-in access points to enable devices to wirelessly connect to the Wi-Fi network, and/or environment100may include one or more stand-alone modems, switches, routers, access points and/or mesh access points. In some embodiments, media asset(s) may be provided to user102and/or user104via any suitable device, by way of wireless signals transmitted through the localized network, and/or responses to voice inputs or any other data related to voice inputs may be provided via the network. As referred to herein, the term “media asset” should be understood to refer to an electronically consumable user asset, e.g., television programming, as well as pay-per-view programs, on-demand programs (as in video-on-demand (VOD) systems), Internet content (e.g., streaming content, downloadable content, webcasts, etc.), augmented reality content, virtual reality content, video clips, audio, playlists, websites, articles, electronic books, blogs, social media, applications, games, and/or any other media or multimedia, and/or combination of the above.

FIG.2shows an illustrative block diagram of a system200for determining a predicted voice command, and a particular device for which the predicted voice command is intended, in accordance with some embodiments of this disclosure. In some embodiments, a voice processing application (e.g., executing at least in part on one or more of the network-connected computing devices of environment100and/or one or more remote servers126ofFIG.1) may be configured to perform the functionalities described herein. In some embodiments, the voice processing system may be offered as an add-on application, e.g.., installed on any home gateway, mobile device, or central device of the home, and/or certain functionalities provided by the voice processing application may be provided via an Application Programming Interface (API).

In some embodiments, the voice processing application may be executable to implement home networked smart assistance system202and home networked smart assistance system204and each of the modules included therein. System202may be implemented locally within environment100(and/or at one or more remote servers126), and system204may be implemented at one or more servers126located at a remote location from environment100(and/or locally within environment100). System202may comprise smart assistance discovery module206, wake word compilation module208, voice shortcuts compilation module210, active state tracking module212, dynamic voice shortcuts configuration module214, dynamic wake word configuration module216, voice input routing module218, voice feedback/affirmative response generation module220, and/or any other suitable modules, and/or may include or otherwise be in communication with common database222which may store one or more knowledge graphs and/or any other suitable data. System202may comprise prediction module224, probable voice command prediction module226and/or any other suitable modules, and/or may include or otherwise be in communication with common database222.

Smart assistance discovery module206may be configured to discover devices in environment100(e.g., connected to a home network) using simple service discovery protocol (SSDP) techniques and/or any other suitable technique. In some embodiments, smart assistance discovery module206may discover all connected devices in environment100having a voice assistant feature. As an example, the voice processing application, e.g., running at least in part on networking equipment116and/or any other suitable device in environment100, may request and/or scan for data packets to discover other devices (e.g., smart televisions106,108and/or voice assistants110,112, etc.) on the localized network of environment100. In some embodiments, the voice processing application may be configured to discover the presence of devices on the wireless network with specific capabilities (e.g., voice command processing capabilities and/or the capability of performing an action in response to another device’s processing of voice input). In some embodiments, the voice command capability of the device, and/or other device capabilities or device characteristics, can be broadcast for discovery as part of the service description of the device, e.g., over any suitable wireless network or short-range communication path or wired communication path. In some embodiments, the voice processing application may continuously or periodically (e.g., at predefined intervals) perform scanning for connected devices and their associated voice command and/or voice input and/or voice query processing capabilities, and/or may do so in response to a triggering event (e.g., receiving user input).

Wake word compilation module208may be configured to compile a list of signature words or wake words configured for particular connected devices in environment100. For example, module208may query each device in environment100or otherwise receive data (e.g., from each device or other information source) indicating whether a particular device is associated with a wake term or signature term and if so, an indication of such wake term. As an example, wake terms or signature terms for a Google voice assistant device may be “Hey Google” and a wake terms or signature terms for an Amazon voice assistant device may be “Alexa.” In some embodiments, the list of signature words or wake words may be stored at one or more devices local to environment100, and/or at remote server126and/or any other suitable database (e.g., common database222).

Connected device active state tracking module212may be configured to collect and monitor active device state information of the connected devices in environment100, e.g., discovered by way of module206. In some embodiments, module206may receive such device state information, e.g., as part of the discovery process. In some embodiments, the device state information may comprise an indication of whether a particular device is turned on or off; an indication of voice processing capabilities of a particular device; an indication of device settings of a particular device; an indication of one or more characteristics of a particular device; an indication of an action previously performed, currently being performed or to be performed by a particular device; metadata related to a media asset being played via the device, and/or any other suitable device state information and/or metadata associated with discovered devices of environment100. The voice processing application may be configured to generate data structure124, e.g., in a tabular format, and/or any other suitable format, based on data requested and/or received from the devices of environment100and/or data received from any suitable information source. For example, the video processing application may cause devices in environment100to transmit, to a local centralized location and/or a remote server, an indication of a type of the device (e.g., including a device identifier, which may be a descriptive attribute such as, for example, at least one of the device name, device type, model number, serial number, manufacturer name, battery life, etc.), and device capabilities and/or commands that the devices are capable of executing, as shown in data structure124, may be determined from such information.

Data structure124may comprise an identifier for each detected device in environment100or a subset of the devices, and an indication of associated device state information for each device or a subset of the devices. For example, data structure124may indicate that TV106is capable of executing actions based on various voice commands or other types of commands (e.g., Turn on; Turn off; Increase volume, Decrease volume). Data structure124may specify that TV106has audio and display capabilities, the ability to process voice input, the ability to skip segments of a media asset, and/or any other suitable capabilities. Data structure124may indicate that a media asset currently being provided to user102is a James Bond VOD movie being played at a particular volume level, the particular scene being played is an action scene, and/or any other suitable settings of TV106(e.g., brightness, battery life, any suitable audio or display settings, trick-play settings, etc.).

FIG.3shows an illustrative knowledge graph, in accordance with some embodiments of this disclosure. In some embodiments, the voice processing application may be configured to generate one or more knowledge graphs, such as, for example, knowledge graph300, based at least in part on the device state information of data structure124. Knowledge graph300may comprise nodes for each device, or a subset of the devices, of environment100, e.g., discovered by way of smart assistance discovery module206and confirmed as having voice processing capabilities. In some embodiments, at least a portion of knowledge graph300may be stored at one or more devices local to environment100, and/or at remote server126and/or any other suitable database (e.g., common database222). In some embodiments, knowledge graph300, and/or any of the knowledge graphs disclosed herein, may be used in conjunction with one or more machine models and/or any other suitable computer-implemented applications or techniques.

In some embodiments, one or more of the knowledge graphs disclosed herein may be a knowledge graph locally stored at environment100and may comprise a limited number of nodes based on parsing a compilation of voice shortcuts of connected devices of environment100. Such voice shortcuts, and/or other nodes and relationships between nodes, may be determined based upon receiving an indication of nodes and connections inferred (e.g., at remote server126in the cloud), based at least in part on the device state information (e.g., active device state and/or metadata). For example, if a particular device is determined to be turned on and/or user102is determined to be proximate to such device, the knowledge graph may be caused to include a node representing the particular device and one or more nodes for respective voice commands such device is capable of processing or otherwise performing an action based on. In some embodiments, one or more of the knowledge graphs disclosed herein may selectively include nodes and connections representing recognized words, commands, recognized entities, recognized intent, etc. In some embodiments, the voice processing system may be configured to continuously update (e.g., add or remove nodes and relationships between nodes) the one or more knowledge graphs based on the active states of devices indicated in the device state information, such as shown at data structure124.

Knowledge graph300may comprise node302representing smart TV106; node304representing mobile device114; node306representing speaker123; node308representing smart refrigerator118and/or a node for each identified device of environment100capable of processing, and/or otherwise performing actions based on, voice inputs. Knowledge graph300may further comprise nodes representing predicted or received voice commands that the devices of environment100are determined to be capable of. For example, the voice processing application may implement voice shortcuts compilation module210, which may be configured to compile a list of predicted or received voice commands, such as, for example, including quick tasks and/or voice shortcuts supported by the connected devices of environment100. In some embodiments, voice shortcuts compilation module210may pull, request or otherwise receive a list of recognized voice shortcuts and/or commands and/or queries that each of the discovered devices of environment100can process. In some embodiments, specific permissions may be granted by connected devices to enable the voice processing application to access such voice commands, and optionally audio samples of such voice commands and/or keywords associated with such voice commands may be provided, e.g., to enable performing keyword spotting locally. Such accessed voice commands, such as, for example, including quick tasks and/or voice shortcuts, may be parsed and used to build knowledge graph300, which may be used for processing voice inputs, queries or commands (locally and/or remotely) to identify intents and target devices for voice inputs.

Module210may parse audio of previously received or otherwise accessed voice inputs (and optionally text associated with the audio), to cause knowledge graph300to include nodes for such voice inputs and/or nodes for voice commands corresponding to such inputs and/or nodes for particular devices and/or device state information corresponding to such inputs. In some embodiments, nodes for particular voice commands from among the accessed voice commands may be selectively added to knowledge graph300in connection with nodes for one or more devices in environment100, based on the current device state information. This may be performed in response to a determination that the current device state information suggests that voice input matching or related to such voice command(s), and requesting the identified one or more devices to perform an action, is likely to be received or has been received. In some embodiments, for voice-to-text, i.e., ASR, conversion, the voice processing application can use common voice samples for keyword spotting with respect to a received voice input122and use the knowledge graph for inferring a context of a predicted or received voice input. In some embodiments, the compiled list of voice commands and/or quick tasks and/or voice shortcuts may be stored at one or more devices local to environment100, and/or at remote server126and/or any other suitable database (e.g., common database222).

In some embodiments, based on voice command information obtained by voice shortcuts compilation module210, knowledge graph300may be generated to include node310representing a “Turn on” predicted or received voice command associated with smart TV106. In such example, an edge between node302representing smart TV106and node310may indicate a relationship between node302and node310. In some embodiments, relatedness between nodes may be a function of connections between nodes, a distance between the nodes, and/or a weight assigned to a connection between nodes. Knowledge graph300may further include node312representing a “Turn off” predicted or received voice command associated with smart TV106, where node302for smart TV106shares an edge with node312. Knowledge graph300may further include nodes314and317, representing an “Increase volume” predicted or received voice command and a “Skip a segment” predicted or received voice command, respectively, each sharing an edge with node302for smart TV106. Knowledge graph300may further include node328for representing a predicted or received “Decrease volume” voice command sharing an edge with node302representing smart TV106.

Knowledge graph300may further indicate a relationship between each of node306representing speakers123of environment100, node310, node312, node314, node318of “Switch to ABC mode,” and node328. Knowledge graph300may further indicate a relationship between node308representing smart refrigerator118of environment100and each of node310and node312. Knowledge graph300may further indicate a relationship between node304representing mobile device114and each of node310; node312; node314; node316of the predicted or received voice command to “Capture an image”; node320of the predicted or received voice command “Increase brightness”; node322of the predicted or received voice command “Decrease brightness”; node324representing a predicted or received voice command to “Answer the call”; node326of the predicted or received voice command to “Make a call”; node328; and node330for the predicted or received “Switch to XYZ app” voice command. In some embodiments, the absence of an edge between two nodes of knowledge graph300may denote that no association between such nodes exists. In some embodiments, an edge between two entities in knowledge graph300may be associated with a weight (e.g., a real number, which may be normalized to a predefined interval) that reflects how likely the nodes connected by the edge are to be associated in a given context. For example, a relatively high weight may serve as an indication that there is a strong link between the nodes connected by the edge. Conversely, a relatively low weight may indicate that there is a weak association between the nodes connected by the edge.

In some embodiments, next state prediction module224of system204may be configured to predict an N number of states (e.g., finite states) for each of the connected devices of environment100. In some embodiments, probable voice command prediction module226of system204may be configured to predict probable commands, instructions and/or comments that a user can issue based on a last active state of the device. Module224and/or module226can be a cloud-based service (e.g., implemented at one or more servers126), and/or may be implemented locally, to receive device state information and/or metadata from connected devices in environment100. In some embodiments, system204may determine, and transmit to one or more devices of environment100, frequently used text and/or a voice sample and/or associated text and/or intents for the predicted voice commands and/or predicted instructions and/or predicted comments, e.g., determined based on device state information. Such information may be stored at, e.g., database222. Module224may implement any suitable computer-implemented technique, e.g., a finite state machine (FSM), heuristic-based techniques, and/or machine learning techniques, to efficiently predict next states that the device(s) can take. In some embodiments, the FSM may be built based on previous inferences that the voice processing application may have made. In some embodiments, for devices and/or states that may not have a finite set of outcomes, the system may employ advanced automation logic, e.g., a machine learning engine, to predict the next states.

In some embodiments, for each state, probable command or instruction prediction module226can predict a limited set of commands, instructions and/or comments that can be used by dynamic voice shortcuts configuration module214to dynamically update the list of voice shortcuts. For example, system202(e.g., module214) may receive, from the cloud service which may be implemented by way of system204, audio samples associated with the predicted voice commands, to enable the local performance of voice-to-text conversion of the audio samples, and keyword spotting. In some embodiments, system202may receive entity and connection details, e.g., from cloud server126, and update knowledge graph300(e.g., locally and/or remotely stored) based on such entity and connection details, e.g., an indication of a specified relationship between “It is loud” and “Decrease volume.” A locally stored knowledge graph (e.g., hosted at devices of environment100and/or common database222ofFIG.2) may be updated with a final predicted inference128made by the voice processing application, with or without including intermediate nodes used to infer such inference in the update. The updated knowledge graph may include new connections and nodes (e.g., an edge connection between nodes “Decrease volume”328and “It is loud”402ofFIG.4) to be added, thereby reflecting the logic and mapping carried out in association with analyzing voice input122. In some embodiments, the voice processing application may infer potential user voice commands and/or comments and/or intents based on a history of enunciation associated with the predicted states of a device. In some embodiments, there may be multiple expressions of intent that map to a same predicted state of a device. In some embodiments, the voice processing application may be configured to perform the prediction of a voice command at128, and the collection and/or transmission of device state information, prior to receiving voice input122, to facilitate processing of voice input122upon its receipt. In some embodiments, the voice processing application may be configured to perform the prediction of a voice command at128, and the collection and/or transmission of device state information, in response to receiving voice input122. In such an instance voice, input122and/or the device state information may be provided to server126together with the device state information to facilitate the prediction, or otherwise processed locally to facilitate the prediction. In some embodiments, the intent of the conversational session may be stored temporally, and on additional speech input, the system may refer to attempt to resolve queries locally and/or remotely. The voice processing application may be configured to automatically update routing logic of the received or predicted voice input or voice command based on the identification of the appropriate device.

Upon receiving the last active state and/or predicted states of the one or more discovered devices of environment100, module226can use any suitable computer-implemented technique, e.g., a machine learning model (e.g., an artificial neural network and/or any other suitable machine learning model) to predict probable commands, instructions and/or comments. In some embodiments, server126may determine and provide audio samples, e.g., associated with the predicted probable voice command, that can be stored locally in a home environment100, and used for keyword spotting. Module226may be configured to predict user comments, e.g., the voice command at128, based at least in part on past observations, and/or based at least in part on received device state information (e.g., active device status and metadata) from environment100. For example, module226may receive metadata indicating that a current scene of a media asset, or the media asset itself, played via TV106has certain characteristics (e.g., an action or fight scene or genre) and/or that TV106is set at a particular volume (e.g., above a threshold), and/or that user102is proximate to TV106and/or any other suitable information. As another example, the voice processing application may perform audiovisual processing of a current scene to identify certain characteristics, e.g., the occurrence of fast-paced movement of objects or actors and/or audio above a certain threshold, to determine the scene is likely loud. Based on such factors, the voice processing application may determine a predicted voice comment or input of “It is loud” and/or a predicted voice command of “Decrease volume” and that such predicted voice input and/or command relates to TV106. Server126may transmit such information to one or more devices of environment100and/or common database222. Based on such information, a knowledge graph400ofFIG.4may be caused to add a new node402for the predicted “It is loud” voice input with an edge connection (e.g., direct or indirect) to node328for the predicted “Decrease volume” voice command. In some embodiments, dynamic voice shortcuts configuration module214may be configured to update a list of voice shortcuts based on the device state information and/or information or instructions received from system204. For example, in response to receiving the predicted voice commands and/or other predicted voice input, module214may be configured to delete a least used and/or oldest voice shortcut, and/or another shortcut that is determined to be unrelated to the received predictions. In some embodiments, an instruction to update a weight of specific connections of a local knowledge graph can be sent from the cloud service based on context resolved in the cloud.

FIG.4shows an illustrative knowledge graph400, in accordance with some embodiments of this disclosure. The voice processing application may receive voice input122and parse such input in real time, e.g., using the updated locally stored knowledge graph400, as an alternative to communicating voice input122to cloud servers for processing. Such features may be beneficial in terms of minimizing the use of computing resources, e.g., by performing keyword spotting locally as opposed to by natural language techniques at a remote server, and may be desirable from a privacy perspective of a user. For example, when voice input122is received, the voice processing application may refer to the local database for keyword spotting and refer to the local knowledge graph for recognized words and entities, and reconstruct the intent by correlating terms and entities.

Knowledge graph400may be populated with nodes and relationships between received device state information and/or received predicted voice inputs. Knowledge graph400may be used to determine that received voice input or voice command122matches or is otherwise related to predicted voice inputs and/or commands, e.g., as predicted by module226ofFIG.2and as shown at node402and node328ofFIG.4. Knowledge graph400may comprise node402representing a predicted voice input, e.g., predicted by module226ofFIG.2. Node402may be linked to node328based at least in part on the aforementioned analysis of device state information and/or other contextual information of environment100, such as, for example, a location of user102. Alternatively, node402may represent received voice input122itself.

For example, the location of user102with respect to two or more candidate devices may be taken into account, and may be used to determine which candidate device should be ranked as a more likely device intended by voice input122. The voice processing application may determine (e.g., based on network traffic data of networking equipment116) that multiple devices are streaming content, and that the predicted or received voice input does not specify a particular device to which action should be taken. The voice processing application can identify based on the last state and predicted states of the connected device, to which device the voice command should apply. If the voice processing application determines that user102is within a same room or closer to smart TV106than smart TV108, such determination may weigh towards a finding that smart TV106is the more likely subject of a predicted or received voice command or voice input than smart TV108. As another example, the voice processing application may collect device state information from connected devices, correlate such information with resolved intent, and perform an action and/or generate a voice command prediction, based on such information, even if a device name is not specified in the input. For example, for a predicted or receive voice command of “Alexa turn off the notification,” the voice processing application may determine that a notification has been generated for display on a particular device, and thus a received or predicted voice command should relate to such device. The system may additionally or alternatively consider parameters, e.g., user proximity to a device, correlation score, etc., to identify which device the predicted query or command or instruction relates to. For example, for a received or predicted command of “Turn off the notification,” the system can identify which device is close to the user, check if the instruction is suitable for that device and take action on the closest device only when the action is suitable for the closest device or pass it on the command for performance at a next closest suitable device. If, for a particular command, two or more devices are determined to be associated with the command, the voice processing application may cause such command to be executed on the device having a predicted state determined to be most closely associated with the command.

Locations of users and/or connected devices within environment100may be determined using any suitable technique. In some embodiments, a location of user102may be ascertained based on a location of a user device (e.g., mobile device114, a smartwatch, etc.) associated with user102, based on location data, e.g., using any suitable technique, such as, for example, GPS techniques, VPS techniques, analyzing wireless signal characteristics, determining that a voice command was received from the user at a voice assistant located in a particular room, or any other suitable technique or any combination thereof. In some embodiments, the voice processing application may identify locations of users and/or devices in environment100based on determined wireless signal characteristics, e.g., channel state information (CSI), received signal strength indicator (RSSI) and/or received channel power indicator (RCPI), as discussed in more detail in Doken et al., Application No. 17/481,931, the contents of which are hereby incorporated by reference herein in their entirety. In some embodiments, the device state information may specify a location of a connected device, and/or user input may be received specifying a particular device.

In some embodiments, the voice processing system may determine a location of user102and/or the connected devices within environment100, and/or build a map of such devices and/or users, based on sensor data (e.g., by performing image processing techniques on images captured by one or more of the connected devices and/or one or more cameras positioned at various locations in environment100; by processing audio signals captured by a microphone of a user device; processing data from IoT devices and indicating a location of a user or device; by using ultrasonic sensors, radar sensors, LED sensors, or LIDAR sensors to detect locations of users and/or devices, or using any other suitable sensor data or any combination thereof), or using any other suitable technique or any combination thereof. The voice processing application may determine that a user is proximate (e.g., within a threshold distance) to a connected device based on comparing the current location of user102to a stored or determined location of each respective connected device in environment100. In some embodiments, a Cartesian coordinate plane may be used to identify a position of a device or user in environment100, with the position recorded as (X, Y) coordinates on the plane. The coordinates may include a coordinate in the Z-axis, to identify the position of each identified object in 3D space, based on images captured using 3D sensors and any other suitable depth-sensing technology. In some embodiments, coordinates may be normalized to allow for comparison to coordinates stored at the database in association with corresponding objects. As an example, the voice processing application may specify that an origin of the coordinate system is considered to be a corner of a room within or corresponding to environment100, and the position of a connected device or user may correspond to the coordinates of the center of the object or one or more other portions of the object.

As shown by intermediate nodes406and408, the voice processing application may determine that user102is proximate to a portion of environment100associated with loud sounds (e.g., above a threshold decibel level, or above decibel levels of other portions of environment100), as shown by edge404, and/or that a source of the loud sounds is a device coupled to assistant110. The voice processing application may further determine, based on the device state information for the plurality of devices in environment100, one or more candidate devices that may pertain to voice input122and the inferred command “Decrease volume” represented by node328. For example, the voice processing application may rule out (e.g., remove from the knowledge graph) devices in environment100for which the device state information indicates the device is off, lacks the ability to play audio, has a current volume level below a threshold, and/or is associated with a location in environment100that is not proximate to user102.

Node410may represent the inference that the device must be on in order to be loud, and edge412and node414may indicate that a loud sound is coming from a particular smart TV106represented by node302. Thus, node402may be mapped to inferred command “Decrease volume” represented by node328as well as node302representing smart TV106for which voice input122and/or predicted voice input is determined as likely to be intended. In some embodiments, certain attributes of the media asset playing at smart TV106and indicated by the device state information, e.g., a current scene is an action scene that is likely to be loud, may be taken into account in identifying smart TV106as a top-ranked candidate device for a particular determined voice command, as shown at128ofFIG.1.

At130, the voice processing application may cause an action associated with the predicted voice command determined at128to be performed. For example, voice assistant110may transmit an instruction to smart TV106to decrease a volume of a currently playing media asset, based on voice command128, e.g., determined locally or at server126. In some embodiments, smart television106may receive such instruction from server126(e.g., via networking equipment116) or from any other suitable device within or external to environment100. In some embodiments, the voice processing application, in processing the determined intended voice command at128and causing the volume of TV106to be decreased at130, may convert text associated with the voice command to speech, e.g., to generate for output an audio notification indicating that the volume of TV106has been decreased. For example, voice feedback/affirmative response generation module220may be configured to generate an affirmative response indicating on which device action will be taken, e.g., for a voice command “Turn off the alarm output audio,” the response of “Turning off the mobile device [114] alert”. Additionally or alternatively, the voice processing application may provide a textual indication to notify user102that the volume of TV106has been decreased.

In some embodiments, if voice input122is determined not to match and not to be related to the predicted voice inputs, and/or is determined to be ambiguous as to which device it is referring to, voice input122may be forwarded to server126for further processing, or it may be further processed locally. In some embodiments, voice input122may be provided, or may not be provided, to server126. In some embodiments, at least a portion of the processing of voice input122may be performed locally. For example, digital assistant device110may receive and digitize voice input122received via a microphone of digital assistant device110in analog form, and/or may perform parsing of voice input122. For example, the voice processing application running at digital assistant device110and/or any other suitable local device and/or server126may be configured to perform automatic speech recognition (ASR) on voice input122to convert “It is loud” from audio format to textual format and/or any other suitable processing of voice input122.

The voice processing application may be configured to transcribe voice input122into a string of text using any suitable ASR technique. For example, to interpret received voice input122, one or more machine learning models may be employed, e.g., recurrent neural networks, bidirectional recurrent neural networks, LSTM-RNN models, encoder-decoder models, transformers, conditional random fields (CRF) models, and/or any other suitable model(s). Such one or more models may be trained to take as input labeled audio files or utterances, and output one or more candidate transcriptions of the audio file or utterance. In some embodiments, the voice processing application may pre-process the received audio input for input into the neural network, e.g., to filter out background noise and/or normalize the signal, or such processing may be performed by the neural network. In some embodiments, in generating the candidate transcriptions, the voice processing application may analyze the received audio signal to identify phonemes (i.e., distinguishing units of sound within a term) within the signal, and utilize statistical probability techniques to determine most likely next phonemes in the received query. For example, the neural network may be trained on a large vocabulary of words, to enable the model to recognize common language patterns and aid in the ability to identify candidate transcriptions of voice input. Additionally or alternatively, transcription of the audio signal may be achieved by external transcription services (e.g., Amazon Transcribe by Amazon, Inc. of Seattle, WA and Google Speech-to-Text by Google, Inc. of Mountain View, CA). The transcription of audio is discussed in more detail in U.S. Pat. Application No. 16/397,004, filed Apr. 29, 2019, which is hereby incorporated by reference herein in its entirety.

The voice processing application may further employ natural language processing (NLP) including natural language understanding (NLU), e.g., tokenization of the string of voice input122, stemming and lemmatization techniques, parts of speech tagging, domain classification, intent classification and named entity recognition with respect to voice input122. In some embodiments, rule-based NLP techniques or algorithms may be employed to parse text included in voice input122. For example, NLP circuitry or other linguistic analysis circuitry may apply linguistic, sentiment, and grammar rules to tokenize words from a text string, and may perform chunking of the query, which may employ different techniques, e.g., N-gram extraction, skip gram, and/or edge gram; identify parts of speech (i.e., noun, verb, pronoun, preposition, adverb, adjective, conjunction, participle, article); perform named entity recognition; and identify phrases, sentences, proper nouns, or other linguistic features of the text string. In some embodiments, statistical natural language processing techniques may be employed. In some embodiments, a knowledge graph may be employed to discern relationships among entities. In some embodiments, one or more machine learning models may be utilized to categorize one or more intents of voice input122. In some embodiments, the NLP system may employ a slot-based filling pipeline technique and templates to discern an intent of a query. For example, the voice processing application may reference a collection of predetermined template queries having empty slots to be filled. In some embodiments, the predetermined templates may be utilized in association with a knowledge graph to determine relationships between terms of a query.

FIG.5shows an illustrative knowledge graph500, in accordance with some embodiments of this disclosure. In the example ofFIG.5, node330may represent a received or predicted voice command of “Switch to XYZ App.” Based at least in part on the device state information, the voice processing application may determine that a smart TV, e.g., smart TV106represented by node302, is intended by the received or predicted voice command of “Switch to XYZ App” represented by node330. For example, node330may be connected, via edge502representing the current state of TV106, to intermediate node504representing an indication that TV106is currently on. Knowledge graph500may further comprise intermediate node508, indicating a current app running on TV106is not the XYZ app specified at node330, via edge506representing a state of TV106with respect to an application running on TV106. The device state information may indicate that mobile device114is capable of executing the XYZ app, but that mobile device114is currently off, as indicated by intermediate node512and edge510of knowledge graph500, and thus node302representing TV106may be connected to node330by way of intermediate nodes512,508,504. For example, the voice processing application may infer that the voice command represented by node330is not intended for mobile device114because mobile device114is off and thus cannot switch to a particular app.

In some embodiments, as shown inFIG.5, the voice processing application may cause a connection between node302and node330in updated knowledge graph500, even if such relationship did not previously exist in knowledge graph300. In some embodiments, the voice processing application may prioritize, based on the device state information, which nodes/connections to maintain (e.g., remotely and/or locally) and which nodes to delete when certain entities/connections are not relevant. For example, node304representing mobile device114may be temporarily removed if mobile device114is off or is being used for another task such as a video conference and is not available to switch to another application. In some embodiments, certain nodes and/or edges (e.g.,502,504,506,508,510) may be temporal nodes or edges. For example, the voice processing application may delete such as soon as the command is processed or after a fixed time duration after processing is performed and/or the voice processing application may be configured to dynamically create multiple voice shortcuts and process them locally. In some embodiments, when the voice command represented by node330is received or predicted, the voice processing application may refer to a local database for keyword spotting and refer to the local knowledge graph for recognized words, intents and/or entities. The voice processing application may reconstruct the intent by correlating the words and entities of the predicted or received command (e.g., including separating recognized words, entities, and intents from other words, entities, and intents present in a query).

In some embodiments, the voice processing application, in storing the predicted or received voice input or commands in association with device state information and/or any other suitable data, may separate device characteristics from the predicted or received command. The voice processing system may decouple device-specific words from the predicted or received voice input or voice command by parsing such input to separate any reference to a device from other portions of the input. For example, if TV106is configured to process the voice command “Switch off the TV,” the voice processing application system can separate “Switch off” from “the TV” using any suitable NLP techniques dsicusssed herein or any other suitable techniques. Such aspects may allow the command “Switch off” to be used for another device that may not originally have support for such command, thereby enabling interoperability among the voice assistant devices and/or service of environment100to facilitate an integrated home environment.

In some embodiments, recognized commands may be stored in common database222and/or other suitable devices or databases, and may be stored in association with one or more connected devices capable of processing and/or performing an action based upon such commands. Such devices may, but need not, store these common commands locally. Such features may enable connected devices to process more commands, and mapping of intent to device-specific command format may help in constructing commands for specific devices. For example, as shown at data structure124, voice commands of “Increase volume, “Decrease volume,” “Mute volume” and/or “Provide remote notification” may be mapped to TV106and108, voice assistant devices110and112, mobile device114, speakers123, a smart doorbell, a microphone, a security alarm, and/or any other suitable device. As another example, the voice command of “Skip present segment” or “Skip segment” may be mapped to TV106, TV108, mobile device114, speakers123, voice assistant devices110and112, and/or any other suitable device. For example, instead of storing “Increase the volume of the TV” and “Increase the volume of the phone” as two different templates, the system may use a single instance of common part “Increase volume” and associate devices therewith.

In some embodiments, the voice processing application may be configured to maintain and store registered user accounts and/or profiles. For example, user102may be associated with a particular user account or profile with the voice processing application, accessible via any number of user devices at which the user provides his or her credentials, and/or from any number of different locations. The voice processing application may monitor and store any suitable type of user information associated with user102, and may reference the particular user profile or account to determine an identity of a human (e.g., user102) in environment100. The user profile or account may include user information input by the user, e.g., characteristics of the user, such as, interests, or any other suitable user information, or any combination thereof, and/or user information gleaned from monitoring of the user or other activities of the user, e.g., current and/or historical biometric data of the user, facial or voice characteristics of the user, historical actions or behaviors of the user, user interactions with websites or applications (e.g., social media, or any other suitable website or application, or any combination thereof) or purchase history, or any other suitable user information, or any combination thereof. In some embodiments, certain devices may be associated with a particular user device or user account, e.g., device identifiers for one or more of user devices106,110,114may be stored in association with a user profile of user102. In some embodiments, such profiles may be used to tailor predicted voice inputs and/or commands to specific user profiles.

FIG.6shows an illustrative knowledge graph600, in accordance with some embodiments of this disclosure. As shown inFIG.6, node602may represent a received or predicted voice command of “I like XYZ app,” which can be mapped to node330representing the received or predicted voice command of “Switch to XYZ App.” For example, the voice processing application may infer that the comment represented by node602indicates a desire to switch to the XYZ app. Based at least in part on the device state information, the voice processing application may determine that the XYZ app is installed on one or more devices (e.g., TV106represented by node302and mobile device114represented by node330, in communication with voice assistant110), as shown by way of edge604. The voice processing application may further determine that one of such devices, mobile device114represented by node330, is not proximal to user102associated with the uttered or predicted voice command represented by node602(as shown by edge606and node608). Thus, mobile device114may be ruled out as a candidate device for which action should be taken based on the voice command, as shown by the absence of an edge connection between node330and knowledge graph600. On the other hand, as shown by edge610and node612and edge614and node616, the voice processing application may determine, based on the device state information and/or any suitable location or contextual information, that TV106is on, and that user102is proximal to TV106. The voice processing application may further determine, as shown by edge618and node620, that TV106is not currently running or executing the XYZ app, weighing towards an inference that “I like XYZ app” represented by node602likely constitutes a command for the particular device to be switched to the XYZ app. The voice processing application may further determine, as shown by edge622and node624, that mobile device114is off. Accordingly, based on any suitable portion of the above-described logical steps, the voice processing application may cause smart TV106to perform an action, e.g., switch a currently executing app to the XYZ app, related to predicted or received voice input represented by node602.

FIG.7shows an illustrative knowledge graph700, in accordance with some embodiments of this disclosure. As shown inFIG.7, node320represents a predicted or received voice command (e.g., a voice shortcut) of “Increase brightness.” As shown inFIG.3, such node320, at least initially, may not share a connection with node308representing refrigerator118. However, as shown inFIG.7, the voice processing application may, based at least in part on device state information and contextual information, determine that a particular user, e.g., user104currently providing voice input or predicted to provide voice input, is proximate to refrigerator118, as shown by way of edge702and node704. Further, the voice processing application may determine that such refrigerator118is a smart appliance that comprises a graphical user interface (GUI) that is currently on, as shown by edge706and/or node708. In some embodiments, certain devices that user104may be proximate to, e.g., voice assistant112, may be determined not to have a display, and thus the voice input represented by node320may be determined as inapplicable to such a device, which may be removed from the knowledge graph at least temporarily. The voice processing application may further determine, based on the device state information, that the brightness of such GUI of refrigerator118is not at a maximum level or maximum value (as shown by edge710and node712), and thus the voice input represented by node320is relevant. Accordingly, the voice processing application may identify refrigerator118as a target device, by creating the temporal nodes and connections shown inFIG.7, and the voice processing application may cause refrigerator118to execute a command to increase the brightness of its GUI.

FIG.8shows an illustrative knowledge graph800, in accordance with some embodiments of this disclosure. The example ofFIG.8is similar to the example ofFIG.7, except the predicted or received voice command may be “Decrease brightness” represented by node322ofFIG.8, rather than the voice command “Increase brightness” represented by node320ofFIG.7. Accordingly, the voice processing application may perform similar processing as described inFIG.7, except the voice processing application may check, as shown at node814, whether the brightness of refrigerator118represented by node308is at a minimum value or level. Upon determining that such brightness is not at a minimum level or minimum value, as shown by edge710and node814, the voice processing application may identify refrigerator118as a target device, by creating the temporal nodes and connections, and may cause refrigerator118to execute a command to decrease the brightness of its GUI.

FIG.9shows an illustrative knowledge graph 900, in accordance with some embodiments of this disclosure. Node317of knowledge graph 900 represents a voice command, e.g., received from user102and/or predicted as a potential voice command, of “Skip a segment.” As shown by edge902and node904of knowledge graph 900, the voice processing application may determine, based at least in part on device state information and a location of mobile device114and user102, that user102providing or predicted to provide voice input is proximal to mobile device114. The voice processing application may further determine, as shown by edge906and node908, that a current state of mobile device114indicates that such device is currently streaming content, e.g., a media asset received from a content provider (e.g., associated with media content source1202ofFIG.12). Such information weighs in favor of a prediction that the received or predicted voice input corresponding to node317is intended for mobile device114, and taken together with edge910and node912indicating that TV106is off (and thus cannot skip segments), may lead to a determination that “Skip a segment” is intended for mobile device114. Accordingly, the voice processing application may cause mobile device114to skip a current segment of a media asset being played at or via mobile device114.

FIG.10shows an illustrative knowledge graph1000, in accordance with some embodiments of this disclosure. As shown inFIG.10, knowledge graph300ofFIG.3may be updated to correspond to knowledge graph1000. The voice processing application may cause knowledge graph1000to include new node308representing the predicted or received voice input of “It is loud” based on the processing ofFIG.4, and to include new node602representing the predicted or received voice input of “I like XYZ app” based on the processing ofFIG.6. The voice processing application may further cause knowledge graph1000to include new edge connection1002establishing a relationship between node602and node330(based on the processing ofFIG.6). The voice processing application may further cause knowledge graph1000to include new edge connection1004establishing a relationship between node302and node330(based on the processing ofFIG.5). The voice processing application may further cause knowledge graph1000to include new edge connection1006establishing a relationship between node308and node328(based on the processing ofFIG.4). The voice processing application may further cause knowledge graph1000to include new edge connection1008establishing a relationship between node308and node320(based on the processing ofFIG.7). The voice processing application may further cause knowledge graph1000to include new edge connection1010establishing a relationship between node308and node322(based on the processing ofFIG.8). The voice processing application may further cause knowledge graph1000to include new edge connection1012establishing a relationship between node304and node317(based on the processing ofFIG.9). As shown, in at least some embodiments, the intermediate nodes shown inFIGS.4-9may be omitted from the updated knowledge graph700. Alternatively, the voice processing application may add the intermediate nodes and edges, associated with the inferences made based on the device state information, predicted states of the device, predicted commands and inferred connections, to the one or more knowledge graphs stored locally and/or remotely. As another example, an updated knowledge graph1000stored at cloud server126may include all the predefined nodes, predefined connections, temporal nodes, and temporal connections along with its inferences. In some embodiments, knowledge graph1000stored locally in environment100may include a subset thereof, and may be updated with new nodes and connections based on the predicted voice commands and/or instruction and/or comments.

In some embodiments, environment100may have a limited number of users, most of whose voices may have similar phonetics, and thus the voice processing application may store audio signals to word mappings, and audio to entity mappings, for specific users, e.g., in connection with user profiles for users of environment100, locally and/or remotely. This may enable certain commands to be processed locally that would otherwise be transmitted to server126for processing, and/or may improve tailoring of voice command predictions for specific users. In some embodiments, dynamic wake word configuration module216may be configured to automatically add and/or edit and/or delete a wake word or wake term based on predicted voice commands and/or predicted voice instructions. In some embodiments, the voice processing application may ensure that a list of certain wake words (e.g., “Hey Siri”; “Hey Google”; “Alexa,” etc.) are not deleted, unless a software update or other indication from a manufacturer indicates such wake words or wake terms have changed.

In some embodiments, voice input routing module218may be configured to receive, at a microphone associated with a first device, voice input having a wake word or term associated with a second device, and route the voice input to be processed by the second device or another device having a similar capability as the second device. Such routing may leverage the common list of wake words, e.g., compiled and maintained by modules208and216. For example, the voice processing application may support cross usage of wake words using networked microphone capability of various devices in environment100capable of processing voice input and/or voice commands. As an example, the voice processing application may receive a voice command (e.g., “Hey Siri, switch on the living room light”) by way of a first device (e.g., voice assistant110), which can be activated by any of the wake words or wake terms compiled by the voice processing application. User102may not be near a second device, such as, for example, mobile device114(e.g., an iPhone, associated with the wake word “Hey Siri”), and voice assistant110may be an Amazon device typically responsive to the wake word “Alexa.” Nonetheless, voice assistant110may recognize the “Hey Siri” wake word based on the compiled list of wake words, capture subsequent voice input, and route such data to mobile device114for processing, via the home network or gateway of environment100. Such second device (e.g., an iPhone) may perform the action associated with the received voice input, e.g., transmit an instruction to turn on the living room light. In some embodiments, the voice processing application may cause the first device (e.g., voice assistant110) to process the voice command, or a third device having a similar capability to process the voice command and/or transmit the data to a remote server for further processing, in the absence of the second device (e.g., the iPhone cannot be found). For example, it may not matter to the user which device processes the request to turn the living room light on, just that the action is performed.

In some embodiments, such feature enabling processing of the request by a particular voice assistant device, even when another voice assistant device was intended by a user, can be enabled or disabled by the user. Such features may create an open framework, where all the smart assistance devices and/or services may reference the compiled list of recognized wake-words to detect voice commands. Such open architecture may enable creating an ad-hoc wireless network of trusted devices where any device can work as a gateway, and in which the voice processing capabilities of connected devices can be exploited. In some embodiments, such features may allow the first device (e.g., an Amazon voice assistant) to capture audio and internally route the raw audio signal to Siri, e.g., if the voice query/command referencing Siri can be recognized locally. In some embodiments, the voice processing application may enable reduction of the number of microphones embedded in some voice assistance devices, e.g., by utilizing microphones of other user-connected devices (e.g., smart TV106) which might be located closer to user102.

FIGS.11-12describe illustrative devices, systems, servers, and related hardware for determining a predicted voice command, and a particular device for which the predicted voice command is intended, in accordance with some embodiments of this disclosure.FIG.11shows generalized embodiments of illustrative devices1100and1101, which may correspond to, e.g., devices106,108,110,112,114,116,118,120and/or123ofFIG.1and/or other suitable devices. For example, device1100may be a smartphone device, a tablet or any other suitable device capable of processing and/or performing an action based on voice input or otherwise interfacing with the voice processing application described herein. In another example, device1101may be a user television equipment system or device. User television equipment device1101may include set-top box1115. Set-top box1115may be communicatively connected to microphone1116, audio output equipment (e.g., speaker or headphones1114), and display1112. In some embodiments, microphone1116may receive audio corresponding to a voice of a user, e.g., a voice input or a voice command. In some embodiments, display1112may be a television display or a computer display. In some embodiments, set-top box1115may be communicatively connected to user input interface1110. In some embodiments, user input interface1110may be a remote control device. Set-top box1115may include one or more circuit boards. In some embodiments, the circuit boards may include control circuitry, processing circuitry, and storage (e.g., RAM, ROM, hard disk, removable disk, etc.). In some embodiments, the circuit boards may include an input/output path. More specific implementations of devices are discussed below in connection withFIG.12. In some embodiments, device1100may comprise any suitable number of sensors (e.g., gyroscope or gyrometer, or accelerometer, etc.), as well as a GPS module (e.g., in communication with one or more servers and/or cell towers and/or satellites) or any other suitable localization technique, to ascertain a location of device1100. In some embodiments, device1100comprises a rechargeable battery that is configured to provide power to the components of the device.

Each one of device1100and device1101may receive content and data via input/output (I/O) path1102. I/O path1102may provide content (e.g., broadcast programming, on-demand programming, Internet content, content available over a local area network (LAN) or wide area network (WAN), and/or other content) and data to control circuitry1104, which may comprise processing circuitry1106and storage1108. Control circuitry1104may be used to send and receive commands, requests, and other suitable data using I/O path102, which may comprise I/O circuitry. I/O path1102may connect control circuitry1104(and specifically processing circuitry1106) to one or more communications paths (described below). I/O functions may be provided by one or more of these communications paths, but are shown as a single path inFIG.11to avoid overcomplicating the drawing. While set-top box1115is shown inFIG.11for illustration, any suitable computing device having processing circuitry, control circuitry, and storage may be used in accordance with the present disclosure. For example, set-top box1115may be replaced by, or complemented by, a personal computer (e.g., a notebook, a laptop, a desktop), a smartphone (e.g., device1100), a tablet, a network-based server hosting a user-accessible client device, a non-user-owned device, any other suitable device, or any combination thereof.

Control circuitry1104may be based on any suitable control circuitry such as processing circuitry1106. As referred to herein, control circuitry should be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or supercomputer. In some embodiments, control circuitry may be distributed across multiple separate processors or processing units, for example, multiple of the same type of processing units (e.g., two Intel Core i7 processors) or multiple different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor). In some embodiments, control circuitry1104executes instructions for the voice processing application stored in memory (e.g., storage1108). Specifically, control circuitry1104may be instructed by the voice processing application to perform the functions discussed above and below. In some implementations, processing or actions performed by control circuitry1104may be based on instructions received from the voice processing application.

In client/server-based embodiments, control circuitry1104may include communications circuitry suitable for communicating with a server or other networks or servers. The voice processing application may be a stand-alone application implemented on a device or a server. The voice processing application may be implemented as software or a set of executable instructions. The instructions for performing any of the embodiments discussed herein of the voice processing application may be encoded on non-transitory computer-readable media (e.g., a hard drive, random-access memory on a DRAM integrated circuit, read-only memory on a BLU-RAY disk etc.), For example, inFIG.11, the instructions may be stored in storage1108, and executed by control circuitry1104of a device1100.

In some embodiments, the voice processing application may be a client/server application where only the client application resides on device1100, and a server application resides on an external server (e.g., server1204). For example, the voice processing application may be implemented partially as a client application on control circuitry1104of device1100and partially on server1204(which may correspond to server126ofFIG.1) as a server application running on control circuitry1211. Server1204may be a part of a local area network with one or more of devices1100or may be part of a cloud computing environment accessed via the internet. In a cloud computing environment, various types of computing services for performing searches on the internet or informational databases, providing storage (e.g., for a database) or parsing data are provided by a collection of network-accessible computing and storage resources (e.g., server1204), referred to as “the cloud.” Device1100may be a cloud client that relies on the cloud computing capabilities from server1204to determine whether processing should be offloaded and facilitate such offloading. When executed by control circuitry1104or1211, the voice processing application may instruct control circuitry1104or1211circuitry to perform processing tasks for determining a predicted voice command, and a particular device for which the predicted voice command is intended. The client application may instruct control circuitry1104to perform processing tasks for determining a predicted voice command, and a particular device for which the predicted voice command is intended.

Control circuitry1104may include video generating circuitry and tuning circuitry, such as one or more analog tuners, one or more MPEG-2 decoders or other digital decoding circuitry, high-definition tuners, or any other suitable tuning or video circuits or combinations of such circuits. Encoding circuitry (e.g., for converting over-the-air, analog, or digital signals to MPEG signals for storage) may also be provided. Control circuitry1104may also include scaler circuitry for upconverting and downconverting content into the preferred output format of user equipment1100. Control circuitry1104may also include digital-to-analog converter circuitry and analog-to-digital converter circuitry for converting between digital and analog signals. The tuning and encoding circuitry may be used by device1100,1101to receive and to display, to play, or to record content. The tuning and encoding circuitry may also be used to receive media consumption data. The circuitry described herein, including for example, the tuning, video generating, encoding, decoding, encrypting, decrypting, scaler, and analog/digital circuitry, may be implemented using software running on one or more general purpose or specialized processors. Multiple tuners may be provided to handle simultaneous tuning functions (e.g., watch and record functions, picture-in-picture (PIP) functions, multiple-tuner recording, etc.). If storage1108is provided as a separate device from device1100, the tuning and encoding circuitry (including multiple tuners) may be associated with storage1108.

Control circuitry1104may receive instruction from a user by way of user input interface1110. User input interface1110may be any suitable user interface, such as a remote control, mouse, trackball, keypad, keyboard, touch screen, touchpad, stylus input, joystick, voice recognition interface, or other user input interfaces. Display1112may be provided as a stand-alone device or integrated with other elements of each one of device1100and device1101. For example, display1112may be a touchscreen or touch-sensitive display. In such circumstances, user input interface1110may be integrated with or combined with display1112. In some embodiments, user input interface1110includes a remote-control device having one or more microphones, buttons, keypads, any other components configured to receive user input or combinations thereof. For example, user input interface1110may include a handheld remote-control device having an alphanumeric keypad and option buttons. In a further example, user input interface1110may include a handheld remote-control device having a microphone and control circuitry configured to receive and identify voice commands and transmit information to set-top box1115.

Audio output equipment1114may be integrated with or combined with display1112. Display1112may be one or more of a monitor, a television, a liquid crystal display (LCD) for a mobile device, amorphous silicon display, low-temperature polysilicon display, electronic ink display, electrophoretic display, active matrix display, electro-wetting display, electro-fluidic display, cathode ray tube display, light-emitting diode display, electroluminescent display, plasma display panel, high-performance addressing display, thin-film transistor display, organic light-emitting diode display, surface-conduction electron-emitter display (SED), laser television, carbon nanotubes, quantum dot display, interferometric modulator display, or any other suitable equipment for displaying visual images. A video card or graphics card may generate the output to the display1112. Audio output equipment1114may be provided as integrated with other elements of each one of device1100and equipment1101or may be stand-alone units. An audio component of videos and other content displayed on display1112may be played through speakers (or headphones) of audio output equipment1114. In some embodiments, audio may be distributed to a receiver (not shown), which processes and outputs the audio via speakers of audio output equipment1114. In some embodiments, for example, control circuitry1104is configured to provide audio cues to a user, or other audio feedback to a user, using speakers of audio output equipment1114. There may be a separate microphone1116or audio output equipment1114may include a microphone configured to receive audio input such as voice commands or speech. For example, a user may speak letters, terms, phrases, alphanumeric characters, words, etc. that are received by the microphone and converted to text by control circuitry1104. In a further example, a user may voice commands that are received by a microphone and recognized by control circuitry1104. Camera1118may be any suitable camera integrated with the equipment or externally connected and capable of capturing still and moving images. In some embodiments, camera1118may be a digital camera comprising a charge-coupled device (CCD) and/or a complementary metal-oxide semiconductor (CMOS) image sensor. In some embodiments, camera1118may be an analog camera that converts to digital images via a video card.

The voice processing application may be implemented using any suitable architecture. For example, it may be a stand-alone application wholly-implemented on each one of device1100and device1101. In such an approach, instructions of the application may be stored locally (e.g., in storage1108), and data for use by the application is downloaded on a periodic basis (e.g., from an out-of-band feed, from an Internet resource, or using another suitable approach). Control circuitry1104may retrieve instructions of the application from storage1108and process the instructions to provide the functionality of the voice processing application discussed herein. Based on the processed instructions, control circuitry1104may determine what action to perform when input is received from user input interface1110. For example, movement of a cursor on a display up/down may be indicated by the processed instructions when user input interface1110indicates that an up/down button was selected. An application and/or any instructions for performing any of the embodiments discussed herein may be encoded on computer-readable media. Computer-readable media includes any media capable of storing data. The computer-readable media may be non-transitory including, but not limited to, volatile and non-volatile computer memory or storage devices such as a hard disk, floppy disk, USB drive, DVD, CD, media card, register memory, processor cache, Random Access Memory (RAM), etc.

Control circuitry1104may allow a user to provide user profile information or may automatically compile user profile information. For example, control circuitry1104may access and monitor network data, video data, audio data, processing data, participation data from a voice processing application and social network profile. Control circuitry1104may obtain all or part of other user profiles that are related to a particular user (e.g., via social media networks), and/or obtain information about the user from other sources that control circuitry1104may access. As a result, a user can be provided with a unified experience across the user’s different devices.

In some embodiments, the voice processing application is a client/server-based application. Data for use by a thick or thin client implemented on each one of device1100and device1101may be retrieved on-demand by issuing requests to a server remote to each one of device1100and device1101. For example, the remote server may store the instructions for the application in a storage device. The remote server may process the stored instructions using circuitry (e.g., control circuitry1104) and generate the displays discussed above and below. The client device may receive the displays generated by the remote server and may display the content of the displays locally on device1100. This way, the processing of the instructions is performed remotely by the server while the resulting displays (e.g., that may include text, a keyboard, or other visuals) are provided locally on device1100. Device1100may receive inputs from the user via input interface1110and transmit those inputs to the remote server for processing and generating the corresponding displays. For example, device1100may transmit a communication to the remote server indicating that an up/down button was selected via input interface1110. The remote server may process instructions in accordance with that input and generate a display of the application corresponding to the input (e.g., a display that moves a cursor up/down). The generated display may then be transmitted to device1100for presentation to the user.

FIG.12is a diagram of an illustrative system1200, in accordance with some embodiments of this disclosure. Devices1207,1208,1210,1212,1214,1216,1218and1220, or any other suitable devices, or any combination thereof, may be coupled to communication network1206. Such devices may be present in a particular environment, such as, for example, environment100ofFIG.1. In some embodiments, at least a portion of such devices may correspond to device1100or1101ofFIG.11, or may include any suitable portion of the same or similar components as described in connection withFIG.11. Communication network1206may be one or more networks including the Internet, a mobile phone network, mobile voice or data network (e.g., a 5G, 4G, or LTE network, or any other suitable network or any combination thereof), cable network, public switched telephone network, or other types of communication network or combinations of communication networks. Paths (e.g., depicted as arrows connecting the respective devices to the communication network1206) may separately or together include one or more communications paths, such as a satellite path, a fiber-optic path, a cable path, a path that supports Internet communications (e.g., IPTV), free-space connections (e.g., for broadcast or other wireless signals), or any other suitable wired or wireless communications path or combination of such paths. Communications with the client devices may be provided by one or more of these communications paths but are shown as a single path inFIG.12to avoid overcomplicating the drawing.

Although communications paths are not drawn between devices, these devices may communicate directly with each other via communications paths as well as other short-range, point-to-point communications paths, such as USB cables, IEEE 1394 cables, wireless paths (e.g., Bluetooth, infrared, IEEE 702-11x, etc.), or other short-range communication via wired or wireless paths. The devices may also communicate with each other directly through an indirect path via communication network1206.

System1200may comprise media content source1202and one or more servers1204. Communications with media content source1202and server1204may be exchanged over one or more communications paths but are shown as a single path inFIG.12to avoid overcomplicating the drawing. In addition, there may be more than one of each of media content source1202and server1204, but only one of each is shown inFIG.12to avoid overcomplicating the drawing. If desired, media content source1202and server1204may be integrated as one source device. In some embodiments, the voice processing application may be executed at one or more of control circuitry1211of server1204(and/or control circuitry of devices1207,1208,1210,1212,1214,1216,1218and/or1220, or any other suitable devices, or any combination thereof). In some embodiments, data structure124ofFIG.1, or any other suitable data structure or any combination thereof, may be stored at database1205maintained at or otherwise associated with server1204, and/or at storage of one or more of devices1207,1208,1210,1212,1214,1216,1218and/or1220, at least one of which may be configured to host or be in communication with database222ofFIG.2. In some embodiments, systems202and204, and the modules thereof, may be implemented by the voice processing application across any suitable combination of the devices or servers or databases ofFIG.12. The knowledge graphs described in connection withFIGS.2-10may be stored at, e.g., any of devices1207,1208,1210,1212,1214,1216,1218and/or1220, and/or server1204, and/or database1205and/or database222or any other suitable device described herein.

In some embodiments, server1204may include control circuitry1211and storage1214(e.g., RAM, ROM, Hard Disk, Removable Disk, etc.). Storage1214may store one or more databases. Server1204may also include an input/output path1212. I/O path1212may provide media consumption data, social networking data, device information, or other data, over a local area network (LAN) or wide area network (WAN), and/or other content and data to control circuitry1211, which may include processing circuitry, and storage1214. Control circuitry1211may be used to send and receive commands, requests, and other suitable data using I/O path1212, which may comprise I/O circuitry. I/O path1212may connect control circuitry1211(and specifically control circuitry) to one or more communications paths. I/O path1212may comprise I/O circuitry.

Control circuitry1211may be based on any suitable control circuitry such as one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores) or supercomputer. In some embodiments, control circuitry1211may be distributed across multiple separate processors or processing units, for example, multiple of the same type of processing units (e.g., two Intel Core i7 processors) or multiple different processors (e.g., an Intel Core i5 processor and an Intel Core i7 processor). In some embodiments, control circuitry1211executes instructions for a voice processing application stored in memory (e.g., the storage1214). Memory may be an electronic storage device provided as storage 1014 that is part of control circuitry 1011.

Device1210may be a smart television, e.g., corresponding to smart TV106,108, device1207may be user computer equipment, and device1208may be a wireless user communication device, each of which may be configured to include some or all of the features of the voice processing application described herein. The voice processing application may be tailored to the capabilities of the particular device.. For example, on user computer equipment1207, the voice processing application may be provided in a visual layout where the voice processing application may recite audio prompts of the voice processing application. In another example, the voice processing application may be scaled down for wireless user communications device1208. In another example, the assistant application may not provide a graphical user interface (GUI) and may listen to and dictate audio to a user such as for voice assistant device1212, which in some instances, may not comprise a display. Various network-connected devices or IoT devices may be connected via a home network and may be capable of being controlled using the voice processing application and/or IOT applications and/or using voice assistant device1212.

Voice assistant device1212(which may correspond to voice assistant devices110and112ofFIG.1) may include a smart speaker, a stand-alone voice assistant, smarthome hub, etc. Voice assistant device1212may be configured to interface with various devices, such as, for example, devices1207,1208and/or1210, and/or autonomous cleaning device1213, smart doorbell1216, smart lamp1218(which may correspond to smart lamp120ofFIG.1), security camera1220, smart refrigerator118, networking equipment116, and/or any other suitable device. In some embodiments, voice assistant device1212may be configured to process a voice command and transmit such processed voice command or voice input to any of such devices with an instruction for such device to perform an action or otherwise forward voice input to a particular device, determined to be intended by the user, for processing. In some embodiments, devices, e.g., devices1207,1208,1210and/or any other suitable device, may be configured to

FIG.13is a flowchart of a detailed illustrative process for determining a predicted voice command, and a particular device for which the predicted voice command is intended, in accordance with some embodiments of this disclosure. In various embodiments, the individual steps of process 1300 may be implemented by one or more components of the devices and systems ofFIGS.1-12. Although the present disclosure may describe certain steps of process 1300 (and of other processes described herein) as being implemented by certain components of the devices and systems ofFIGS.1-12, this is for purposes of illustration only, and it should be understood that other components of the devices and systems ofFIGS.1-12may implement those steps instead.

At1302, control circuitry (e.g., control circuitry1104, and/or control circuitry of any suitable device in environment100and/or control circuitry1211of server1204) may identify a plurality of devices connected to a localized network and capable of processing, or performing one or more actions based on, one or more voice inputs. For example, the control circuitry may identify smart television106,108; digital assistant110,112; mobile device114, smart refrigerator118, smart lamp120and/or speakers123and/or any other suitable devices within environment100ofFIG.2. For example, the control circuitry may query each device in environment100or otherwise receive data indicating whether a particular device is capable of processing a voice input and/or performing an action based on voice input. Such processing may comprise, for example, detecting an utterance and/or performing at least some processing to analyze or parse the input, and performing an action based on voice input may comprise a device capable of receiving an instruction modifying its state, performing a task and/or providing an output.

At1304, the control circuitry may determine state information for each of the plurality of identified devices. In some embodiments, such state information may include the voice processing capabilities described as being identified in connection with1302. In some embodiments, the device state information may comprise an indication of whether a particular device is turned on or off; an indication of voice processing capabilities of a particular device; an indication of device settings of a particular device; an indication of one or more characteristics of a particular device; an indication of an action previously performed, currently being performed or to be performed by a particular device; metadata related to a media asset being played via the device; and/or any other suitable device state information and/or metadata associated with discovered devices of environment100. The voice processing application may be configured to generate data structure124ofFIG.1, e.g., in a tabular format, and/or any other suitable format, based on data requested and/or received from the devices of environment100ofFIG.1and/or data requested and/or received from any suitable information source.

At1306, the control circuitry may determine, based at least in part on the device state information obtained at1304and/or1302, a predicted voice command, and a particular device for which the predicted voice command is intended. In some embodiments, the control circuitry may further determine a location of user102and/or104in environment100using any suitable technique, e.g., by analyzing wireless signal information in environment100, determining a location of user device114and comparing such location to a known location of TV106, etc.) For example, based on such information, if the control circuitry determines that the device state information indicates that TV106is currently turned on and/or that user102is located proximate to TV106, the control circuitry may predict probable voice commands, instructions and/or comments that a user might utter with respect to a TV that is currently providing content. Such predicted voice commands may be based on, for example, a compilation of voice shortcuts of connected devices of environment100previously received or otherwise obtained by the control circuitry with respect to particular devices. In some embodiments, determining a particular device for which the predicted voice command might be intended may be based on the contextual information such as, for example, a location of a user, considered in conjunction with the state information, e.g., whether TV106is currently on and/or providing content. In some embodiments, if a most recent user interaction with TV106is within a threshold time period, this may weigh towards a user actively using TV106, rather than just passing by TV106, which may indicate that user102is more likely to issue a voice command associated with TV106. In some embodiments, a knowledge graph (e.g., knowledge graph300) may be updated (e.g., such as knowledge graph400ofFIG.4) based on the detected state information and contextual information such as, for example, a location of a user, e.g., to include various nodes and edges mapping out inferences based on the received information, with respect to the particular device.

In some embodiments, candidate devices may be ranked to determine which device a predicted voice input likely is intended for. For example, if each of TV108and TV106is determined to be on, based on device state information, it may be unclear which TV is more likely to be the subject of a voice query. However, if the location of user102is considered, and user102is determined to be located closer to TV106than TV108, TV106may be ranked higher as a candidate device for voice input than TV108. In some embodiments, the ranking may be based on past voice commands issued by user102in connection with a user profile. For example, if a user historically has issued certain voice commands at certain times and/or under certain circumstances (e.g., when a user is in a particular room, and/or when a particular device is on or in a particular state), such voice commands may be ranked high as probable commands the user might issue.

At1308, the control circuitry may receive voice input, e.g., voice input122of “It is loud.” In some embodiments, such voice input may be received prior to control circuitry performing steps1302,1304, and1306. For example, the control circuitry may determine a predicted voice command and device to which the predicted voice command is intended, in response to receiving the voice input. In such an instance, the control circuitry may take into account the voice input itself, e.g., “It is loud,” when determining a predicted voice command potentially intended by such voice input, and the particular device for which the predicted voice command may be intended. For example, as shown inFIG.4, the control circuitry may map voice input122of “It is loud,” which may be represented by node402, to voice command “Decrease volume” indicated at node328, and TV106indicated at node302.

In some embodiments, the voice input, e.g., voice input122of “It is loud,” may be received subsequent to the control circuitry performing steps1302,1304and1306. For example, the control circuitry may anticipate that a voice input or voice command is likely to be received based on the device state information and/or other contextual information, and/or historical tendencies of a user profile of user102. In some embodiments, the control circuitry may determine predicted voice commands at all times, based on user settings specifying certain times to determine predicted voice commands, based on certain times of day user102is likely to be awake, and/or based on any other suitable criteria.

At1310, when voice input e.g., voice input122of “It is loud,” is received from the user, e.g., user102, the control circuitry may utilize knowledge graphs (e.g., local to environment100and/or at remote server126) or any other suitable technique to determine if voice input122is related to the voice command of node328ofFIG.4. For example, the control circuitry may, in real time, analyze an intent of the voice input and determine whether any actions should be performed. Control circuitry1211of server1204may, based on device state information and/or contextual information received from one or more of the connected devices of environment100, update knowledge graph300ofFIG.3in the manner shown inFIG.4and/or any of the other examples described herein, based on voice input122. In some embodiments, updating the knowledge graph may comprise adding or deleting certain nodes (e.g., deleting a node representing a device that is off or unplugged) on a temporary basis. Control circuitry1211of server1204may transmit at least a portion of such updated knowledge graph400, and/or any other suitable data, to one or more of the connected devices of environment100and/or database222, and/or any other suitable data. Based on such received data, the control circuitry may enable the mapping of voice input122(represented by node402) to node302(e.g., the particular device to which predicted voice command “Decrease volume” is intended) and the voice command of node328. For example, the control circuitry may determine that received input122matches predicted input or node402, and thus TV106should be instructed to perform the action of node328, and/or that received input122is related to the predicted voice command represented by node328, as evidenced by the mapping of knowledge graph400. In some embodiments, such aspects may enable the control circuitry to process locally a large number of frequently used or predicted commonly used commands, instructions and/or comments.

Processing may proceed to1312upon determining that voice input122is related to (e.g., matches or is otherwise linked to) the predicted voice command of “Decrease volume” represented by node328. At1312, the control circuitry may cause an action associated with the predicted voice command determined to be performed. For example, voice assistant110may transmit an instruction to smart TV106to decrease a volume of a currently playing media asset, based on voice command128, e.g., determined locally or at server126. In some embodiments, smart television106may receive such instruction from server126(e.g., via networking equipment116) or from any other suitable device within or external to environment100. In some embodiments, the control circuitry, in processing the determined intended voice command and causing the volume of TV106to be decreased at1312, may convert text associated with the voice command to speech, e.g., to generate for output an audio notification indicating that the volume of TV106has been decreased. For example, the control circuitry may be configured to generate an affirmative response indicating on which device action will be taken, e.g., for a voice command “Turn off the alarm output audio” the affirmative response of “Turning off the mobile device [114] alert”. Additionally or alternatively, the control circuitry may provide a textual indication to notify user102that the volume of TV106has been decreased. In some embodiments, the control circuitry may prompt the user for feedback to indicate whether the inferred voice command aligns with the user’s intent by uttering voice input122, and the weights and/or nodes and/or edges of the knowledge graph may be refined based on such input. As another example, the control circuitry may determine based on subsequent user action or inaction whether an inferred voice command was accurate, e.g., if the user continues watching the content on TV106at the adjusted volume, and/or does not immediately utter another voice input or otherwise interact with TV106.

Processing may proceed to1314upon determining at1310that voice input122is unrelated to the predicted voice command of “Decrease volume” represented by node328ofFIG.4. For example, such circumstance may be inferred if the control circuitry is unable to resolve the voice input locally (e.g., using the updated knowledge graph, via one or more of the connected devices of environment100), and such voice input may be transmitted to server126for further processing. For example, complex queries and/or unpredicted voice inputs, comments or queries may be transmitted to a server, optionally only if a wake word or signature word is detected. In some embodiments, processing may proceed to1314if, at1310, the control circuitry determines that a particular device to which the received voice input is directed is ambiguous, and/or the user may be prompted to clarify what he or she intends by the voice input.