Patent Description:
<CIT> discloses an information provision method using voice recognition function and control method for a device.

While the invention is defined in the independent claims, further aspect of the invention are set forth in the dependent claims, the drawings and the following description. According to an aspect of the disclosure, a system to process packetized audio signals in a voice activated computer network. The system includes a digital assistant application executed by a processor of a client device to transmit data packets via a client device network interface over a network, the data packets including a first input audio signal, detected by a sensor of the client device, and first client device configuration data. The system also includes a natural language processor component executed by a data processing system to receive, over the network, via a network interface of the data processing system, the data packets, the natural language processor component to parse the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request. The system further includes a client device configuration appliance executed by the data processing system to determine from the first client device configuration data that a display associated with the client device is in an OFF state. The configuration appliance executed by the data processing system to generate, based in part on the first trigger keyword and the OFF state of the display, a fist response data structure, the first response data structure including audio data and not including video data responsive to the first request, and transmit, over the network, via the network interface of the data processing system, the first response data structure to the digital assistant application at the client device.

According to an aspect of the disclosure, a method to process packetized audio signals in a voice activated computer network. The method includes transmitting data packets by a digital assistant application executed by a processor of a client device, via a client device network interface, over a network, the data packets including a first input audio signal, detected by a sensor of the client device, and a first client device configuration data. The method further includes receiving, at a natural language processor component executed by a data processing system, over the network, via a network interface of the data processing system, the data packets, the natural language processor component to parse the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request. The method also includes determining, at a client device configuration appliance executed by the data processing system, from the first client device configuration data that a display associated with the client device is in an OFF state. The method further includes generating, at the client device configuration appliance, based in part on the first trigger keyword and the OFF state of the display, a first response data structure, the first response data structure including audio data and not including video data responsive to the first request. The method additionally includes transmitting, over the network, via the network interface of the data processing system, the first response data structure to the digital assistant application at the client device.

The foregoing information and the following detailed description include illustrative examples of various aspects and implementations and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations.

For purposes of clarity, not every component may be labelled in every drawing. In the drawings:.

Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems for multi-modal transmission of packetized data in a voice activated data packet (or other protocol) based computer network environment. The various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways.

The present disclosure is generally directed to improving the efficiency and effectiveness of information transmission and processing over disparate computing resources. It is challenging for disparate computing resource to efficiently process responses to audio-based instructions in a voice-based computing environment. For example, the disparate computing resources may receive audio-based instructions for client devices that include both a display and a speaker. Generating and transmitting to the client device responses that include a visual component in addition to an audio component can consume computing resources, and can impact the response time to the instructions at the client device.

The present solution can reduce resource consumption, processor utilization, battery consumption, or bandwidth utilization, by generating responses to voice-based instructions based in part on configuration data received from the client device. In particular, the data processing system can reduce forego generating visual responses if the display of the client device is switched OFF.

Systems and methods of the present disclosure are generally directed to a data processing system that routes packetized actions via a computer network. The data processing system can process the voice-based instructions based in part on a configuration of the client device from which the voice-based instructions are received. For example, a digital assistant executing on a client device, such as a television (TV) can acquire voice-based instructions from a user and transmit the voice-based instructions in addition to configuration data associated with the TV, such as whether a display of the TV is in an ON or an OFF state. The data processing system can process the voice-based instruction and generate a response based in part on the state of the display. For example, if the state of the display is OFF, the data processing system may generate a response that includes an audio component but does not include a video component.

<FIG> illustrates an example system <NUM> to process packetized audio signals in a voice activated computer network. The system <NUM> can include at least one data processing system <NUM>, one or more client computing devices <NUM> ("client device <NUM>"). The data processing system <NUM> can include an interface <NUM>. The data processing system <NUM> can include a natural language processor (NLP) component <NUM> to parse audio-based inputs. The data processing system <NUM> can include an interface management component <NUM> to detect and manage the interfaces of other devices in the system <NUM>. The data processing system <NUM> can include an audio signal generator component <NUM> to generate audio-based signals. The data processing system <NUM> can include a video signal generator component <NUM> to generate video-based signals. The data processing system <NUM> can include a direction action application programming interface (API) <NUM>. The data processing system <NUM> can include a response selector component <NUM> to select responses to audio-based input signals. The data processing system <NUM> can include a client device configuration appliance <NUM>. The data processing system <NUM> can include a data repository <NUM> in which the data processing system <NUM> can store parameters <NUM>, policies <NUM>, response data <NUM>, and templates <NUM>. The client device <NUM> can include and execute instances of the components of the data processing system <NUM>.

The client devices <NUM> can include sensors <NUM>, speakers <NUM>, and displays <NUM>. The client device <NUM> can also execute an instance of the NLP component <NUM>. The system <NUM> can also include one or more data provider computing devices <NUM>. The components of the system <NUM> can communicate over a network <NUM>. The network <NUM> can include computer networks such as the internet, local, wide, metro, or other area networks, intranets, satellite networks, other computer networks such as voice or data mobile phone communication networks, and combinations thereof. The network <NUM> can be used by the data processing system <NUM> and client devices <NUM> to access information resources such as web pages, web sites, domain names, uniform resource locators, or data providers <NUM>. For example, the data processing system <NUM> can, via the network <NUM>, access a data provider <NUM> that provides weather data for specific locations, such as a location associated with a the client devices <NUM>.

The network <NUM> can include, for example, a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, a SDH (Synchronous Digital Hierarchy) network, a wireless network or a wireline network, and combinations thereof. The network <NUM> can include a wireless link, such as an infrared channel or satellite band. The topology of the network <NUM> may include a bus, star, or ring network topology. The network <NUM> can include mobile telephone networks using any protocol or protocols used to communicate among mobile devices, including advanced mobile phone protocol (AMPS), time division multiple access (TDMA), code-division multiple access (CDMA), global system for mobile communication (GSM), general packet radio services (GPRS), or universal mobile telecommunications system (UMTS). Different types of data may be transmitted via different protocols, or the same types of data may be transmitted via different protocols.

The client devices <NUM> can each include at least one logic device such as a computing device having a processor to communicate with each other with the data processing system <NUM> via the network <NUM>. The client devices <NUM> can include an instance of the any of the components described in relation to the data processing system <NUM>. The client devices <NUM> can include a television, a desktop computer, a laptop, a tablet computer, a personal digital assistant, a smartphone, a mobile device, a portable computer, a thin client computer, a virtual server, a speaker-based digital assistant, or other computing device. The client devices <NUM> can include a computing device that is capable of reproducing audio as well as visual data received from the data processing system <NUM> or the data provider <NUM>.

The client device <NUM> can include at least one sensor <NUM>, at least one speaker <NUM>, and at least one display <NUM>. The sensor <NUM> can include a microphone or audio input sensor. The sensor <NUM> can also include at least one of a GPS sensor, proximity sensor, ambient light sensor, temperature sensor, motion sensor, accelerometer, or gyroscope. The sensor <NUM> can include an occupancy or weight sensor. The speaker <NUM> can render audio signals by converting electrical signals into audible waves. The display <NUM> can include an light emitting diode (LED) display, an organic LED (OLED) display, a plasma display, a projector display, a holographic display, and the like. The display <NUM> can render images and videos received by the client device <NUM>.

The client device <NUM> can be associated with an end user that enters voice queries as input audio signal into the client device <NUM> (via the sensor <NUM>) and receives audio or video output in the form of a computer generated voice or images that can be provided from the data processing system <NUM>. In response to the input audio signals, the client device <NUM> can also receive action data structures to perform predetermined functions or actions. The client device <NUM> can receive or provide data messages to the direct action API <NUM> of the data processing system <NUM> and enable communication between the components of the system <NUM>. The client device <NUM> can also include a user interface that enables a user to interact with the components of the system <NUM>.

The system's <NUM> data processing system <NUM> can include at least one server having at least one processor. For example, the data processing system <NUM> can include a plurality of servers located in at least one data center or server farm. The data processing system <NUM> can determine from an audio input signal a request and a trigger keyword associated with the request. Based on the request and trigger keyword, the data processing system <NUM> can generate or select response data. The response data can include audio-based, video based, or text-based data. For example, the response data can include one or more audio files that when rendered provide an audio output or acoustic wave. The data within the response data can also be referred to as content items.

The data processing system <NUM> can include multiple, logically grouped servers and facilitate distributed computing techniques. The logical group of servers may be referred to as a data center, server farm or a machine farm. The servers can be geographically dispersed. A data center or machine farm may be administered as a single entity, or the machine farm can include a plurality of machine farms. The servers within each machine farm can be heterogeneous-one or more of the servers or machines can operate according to one or more type of operating system platform. The data processing system <NUM> can include servers in a data center that are stored in one or more high-density rack systems, along with associated storage systems, located for example in an enterprise data center. The data processing system <NUM> with consolidated servers in this way can improve system manageability, data security, the physical security of the system, and system performance by locating servers and high performance storage systems on localized high performance networks. Centralization of all or some of the data processing system <NUM> components, including servers and storage systems, and coupling them with advanced system management tools allows more efficient use of server resources, which saves power and processing requirements and reduces bandwidth usage. Each of the components of the data processing system <NUM> can include at least one processing unit, server, virtual server, circuit, engine, agent, appliance, or other logic device such as programmable logic arrays configured to communicate with the data repository <NUM> and with other computing devices.

The data processing system <NUM> can include the data repository <NUM>. The data repository <NUM> can include one or more local or distributed databases and can include a database management system. The data repository <NUM> can include computer data storage or memory and can store one or more parameters <NUM>, one or more policies <NUM>, response data <NUM>, and templates <NUM>, among other data. The parameters <NUM>, policies <NUM>, and templates <NUM> can include information such as rules about a voice based session between the client device <NUM> and the data processing system <NUM>. The response data <NUM> can include content items for audio output, image/video output, or associated metadata, as well as input audio messages that can be part of one or more communication sessions with the client device <NUM>.

An application, script, program, or other components that are associated with the data processing system <NUM> can be installed at the client device <NUM>. The application can enable the client device <NUM> to communicate input audio signals (and other data) to the interface <NUM> of the data processing system <NUM>. The application can enable the client device <NUM> to drive components of the client device <NUM> to render the output audio, image, or video signals.

The data processing system's NLP component <NUM> can receive input audio signals. The data processing system <NUM> can receive the input audio signal from the client device <NUM>. The NLP component <NUM> can convert input audio signals into recognized text by comparing the input audio signal against a stored, representative set of audio waveforms and choosing the closest matches. The representative waveforms can be generated across a large set of input audio signals. Once the input audio signal is converted into recognized text, the NLP component <NUM> can match the text to words that are associated, for example, via a learning phase, with actions or output audio signals. From the input audio signal, the NLP component <NUM> can identify at least one request or at least one trigger or hot keyword corresponding to the request. The request can indicate intent or subject matter of the input audio signal. The trigger keyword can indicate a type of action likely to be taken.

The response selector component <NUM> can obtain information from the data repository <NUM>, where it can be stored as part of the response data <NUM>. The response selector component <NUM> can query the data repository <NUM> to select or otherwise identify response phrases or content item, e.g., from the response data <NUM>.

The audio signal generator component <NUM> can generate or otherwise obtain an output signal that includes the content item. The data processing system <NUM> can execute the audio signal generator component <NUM> to generate or create an output signal corresponding to the content item or request. For example, once a request is fulfilled, the audio signal generator component <NUM> can generate an audio output signal that includes the phrase "The action was completed.

The video signal generator component <NUM> can generate or otherwise obtain an output signal that includes a content item. The data processing system <NUM> can execute the video signal generator component <NUM> to generate or create an output signal corresponding to the content item or request. For example, once a request is fulfilled, the video signal generator component <NUM> can generate an image or video output signal, which when displayed on the display <NUM> of the client device, can show the phrase "The action was completed.

The data processing system <NUM> can execute both the audio signal generator component <NUM> and the video signal generator component <NUM> to generate or create an output signal including an audio-visual output.

The interface <NUM> can be a data interface or a network interface that enable the components of the system <NUM> to communicate with one another. The interface <NUM> of the data processing system <NUM> can provide or transmit one or more data packets that include the response data structure, audio, image/video signals, or other data via the network <NUM> to the client device <NUM>. For example, the data processing system <NUM> can provide the output signal from the data repository <NUM> or from the audio signal generator <NUM> to the client device <NUM>. The data processing system <NUM> can also instruct, via data packet (or other protocol) based data transmissions, the client device <NUM> to perform the functions indicated in the response data structure. The output signal can be obtained, generated, transformed to or transmitted as one or more data packets (or other communications protocol) from the data processing system <NUM> (or other computing device) to the client device <NUM>.

The direct action API <NUM> of the data processing system <NUM> can generate, based on, for example, the request, response data structures. The direct action API <NUM> can generate response data structures based also on a configuration of the client device <NUM>. The response data structure can include data or instructions for the execution of a specified action to satisfy the request. The response data structure can include a JSON formatted data structure or an XML formatted data structure.

The client device configuration appliance <NUM> can determine a current configuration of the client device <NUM>. The data packets received from the client device <NUM> can include configuration data of the client device <NUM>. The configuration data can include statuses of one or more components of the client device <NUM>. For example, the configuration data can include information about the ON or OFF state of the display <NUM> of the client device <NUM>. The configuration data can also include the current volume setting ("mute" or "volume level") of the speakers of the client device <NUM>. Depending on the action specified in the request, and the configuration data, the client device configuration appliance <NUM> can execute code or a dialog script that identifies the parameters required to fulfill the request. The client device configuration appliance <NUM> can generate response data structures responsive to the request and the configuration data. The response data structure can be included in messages that are transmitted to or received by the client device <NUM>. The operation of the client device configuration appliance <NUM> in conjunction with that of the client device <NUM> is discussed further below after the discussion of an example client device.

<FIG> illustrates a block diagram of an example client device <NUM>. In particular, the client device <NUM> can include a TV or other display based devices, and can be used to implement the client device <NUM> shown in <FIG>. The client device <NUM> includes a display screen <NUM> surrounded by a frame or bezel <NUM>. The display screen <NUM> can include light emitting elements, such as light emitting diodes (LEDs), organic LEDs (OLEDs), plasma, and the like. The display screen <NUM> may also include a touch interface. The client device <NUM> can include a projector (such as, for example, a digital light projector (DLP)) instead of a display screen. The projector can project an image or video on a projection surface remote from the client device <NUM>. The client device <NUM> can further include a control module <NUM> for controlling the operations of the client device <NUM>. The control module <NUM> can include a processor <NUM>, a memory <NUM>, speakers <NUM>, microphones <NUM>, a display controller <NUM>, storage <NUM>, and a network interface <NUM>. The components of the control module <NUM> can be connected to a data bus <NUM>, which allows communication between the data modules. However, various components of the control module <NUM> can communicate directly with other components over communication channels other than the data bus <NUM>. While not shown in <FIG>, the client device can further include or be connectable to an radio frequency antenna or a radio frequency connector such as a coaxial cable. The control module <NUM> can further include a radio-frequency tuner and radio-frequency receiver circuitry for and tuning to, and receiving information on, radio frequency channels. For example, the client device can be a television that includes the radio-frequency tuner and the radio-frequency for receiving various channels of programing, which can be displayed on the display <NUM>. The client device <NUM> also includes light emitting status indicators <NUM> disposed on the bezel <NUM>. The light emitting status indicators <NUM> can include one or more light emitting elements, such as, LEDs, OLEDs, incandescent bulbs, or other light emitting elements. While the light emitting status indicators <NUM> are shown in <FIG> as being positioned on the bezel <NUM>, the light emitting status indictors can also be positioned anywhere on the client device <NUM> as long as they are viewable by a user, such as for example, on a stand or along a side of the client device <NUM>.

The processor <NUM> can include one or more microprocessors that can execute instructions associated with one or more programs, software modules, or applications stored in memory <NUM>. The memory <NUM> can include several software modules such as a NLP <NUM>, a digital assistant application <NUM>, TV applications <NUM>, a TV operating system <NUM>, and TV configuration data <NUM>. The NLP <NUM> can be similar to the NLP <NUM> discussed above in relation to the data processing system <NUM> shown in <FIG>. The NLP <NUM> can process voice commands received by the digital assistant application <NUM> and determine requests and trigger keywords, which can be used by the digital assistant application <NUM> to process the voice command. The digital assistant application <NUM> can perform certain tasks or provide information to the user based on the voice commands. The digital assistant application <NUM> can communicate with the data processing system <NUM> (<FIG>) for processing and responding to the voice commands. For example, the digital assistant can process an audio signal associated with a voice command into data packets, and send the data packets to the data processing system <NUM>. The digital assistant application <NUM> also can receive audio or video signal responses from the data processing system <NUM>, and reproduce the audio or video signals on the client device <NUM>. The digital assistant application <NUM> can process and respond to the user commands without communicating with the data processing system <NUM>, e.g., in a conversational manner. For example, if the voice commands include requests that can be fulfilled locally, the digital assistant application <NUM> can process the request locally at the client device <NUM> instead of sending the request to the data processing system <NUM>. Examples of requests that can be fulfilled locally can include "turn off the lights," "switch OFF the TV," "mute the speakers," and the like.

TV applications <NUM> can include various applications that can be executed on the client device <NUM>. The TV applications can include utility, entertainment, video, banking, settings, and other such applications. The TV operating system <NUM> can include smart TV operating systems such as webOS Smart TV, Android TV, etc. The TV operating system <NUM> can provide a user interface to receive user commands via a remote controller, switches/buttons on the client device, a touch interface on the display <NUM>, or other devices such as mobile phones wirelessly communicating with the client device <NUM>. The TV operating system also can provide processor, peripheral, and processing resources to launch and run the TV applications <NUM>, the digital assistant application <NUM>, and the TV applications <NUM>. For example, the TV operating system <NUM> can allow the digital assistant application <NUM> access to the speakers <NUM>, microphones <NUM>, the digital controller <NUM>, storage <NUM>, and the network interface <NUM>.

The memory <NUM> also can store TV configuration data <NUM>, which can include information regarding the state of the client device <NUM>. The TV configuration data <NUM> can include a data structure including identities of various aspects or components of the client device <NUM> and their corresponding state. For example, the TV configuration data <NUM> can include an identity of the display (such as, for example, "display") and the current state of the display (such as "ON" or "OFF"). The configuration data <NUM> may include identities and corresponding states of other components. For example, the configuration data <NUM> can include the currently set volume of the speakers <NUM> in addition to storing the current state, such as "ON" or "OFF" of the speakers. The configuration data <NUM> is adapted to also store a minimum volume level of the speaker <NUM> with which the digital assistant application <NUM> can communicate with the user. In some instances, the digital assistant application <NUM> can allow the user to set the preferred minimum volume level of the speakers <NUM> that the digital assistant application <NUM> can use to provide audible responses, reminders, or alarms to the user. The digital assistant application <NUM> can allow the user to grant the digital assistant application <NUM> permission to override a mute state of the speakers and set the speakers <NUM> volume to the minimum volume level when providing audible responses to the user. The digital assistant application <NUM> can send the configuration data <NUM> to the data processing system <NUM> separately from, or together with, voice command requests.

The speakers <NUM> can include one or more transducers that transform audio signals into corresponding audible sound. The speakers <NUM> can receive audio signals from an audio controller, which can include digital-to-analog converters, amplifiers, filers, and signal processing circuitry. The TV operating system <NUM> can provide applications running on the client device <NUM> application programmable interfaces (APIs) to interface with the audio controller. For example, applications such as the digital assistant application <NUM> can use the APIs to send audio signals to the audio controller, which, in turn, can send corresponding analog signals to the speakers <NUM> to generate sound. The digital assistant application <NUM> also can send control signals such as "mute" to mute the volume of the speakers <NUM>, or send volume levels to set the volume of the speakers <NUM>. The microphone <NUM> can include one or more transducers for converting sound energy into input audio signals. At least one microphone <NUM> can be located on the client device <NUM>. At least one microphone <NUM> can also be located remotely from the client device, such as, for example, on a remote controller, a smart phone, or another device. The audio signals generated by the remotely located microphone can be transmitted to the client device <NUM> over a wireless link, such as through the network interface <NUM>. The TV operating system <NUM> can also provide APIs and audio controllers to control the microphone <NUM>. For example, the digital assistant application <NUM> can send configuration parameters to the microphone <NUM> and receive input audio signals from the microphone <NUM> via the APIs.

The display controller <NUM> can include hardware and software for controlling the display <NUM>. In particular, the display controller <NUM> can receive video or image data and convert the video or image data into images on the display <NUM>. The TV operating system <NUM> can provide APIs that can be used by application programs running on the client device <NUM> to send and receive data from the display controller <NUM>. For example, the digital assistant application <NUM> can send video or image signals received from the data processing system <NUM> to the display controller <NUM> for rendering on the display <NUM>. The digital assistant <NUM> also can send control signals or data to the digital controller <NUM> to control the operation of the display <NUM>. For example, the digital assistant <NUM> can send display ON or display OFF commands to switch ON or switch OFF the display <NUM>. In addition, the digital assistant <NUM> can request the status of the display <NUM>, where the requested status can include the current state, for example, an ON state or an OFF state of the display <NUM>. The display controller <NUM> can return the requested status to the digital assistant application <NUM>, which can store the received status of the display in the TV configuration data <NUM>. The display controller <NUM> may also control the operation of the light emitting status indicators <NUM>. The client device <NUM> may include a separate controller for controlling the light emitting status indicators <NUM>. The digital assistant application <NUM> can control the status and the operation of the light emitting status indicators <NUM> via the display controller <NUM> or any other appropriate controller. The digital assistant application <NUM> can receive from the data processing system <NUM> instructions to activate the light emitting status indicators <NUM> to display a particular pattern indicating a corresponding status. For example, one pattern can indicate that the data processing system <NUM> is processing a request. Another pattern can indicate that the request has been completed. The digital assistant application <NUM> can store a sequence of instructions and data corresponding to each of these patterns, and send the appropriate instructions and data to the display controller <NUM> or any other appropriate controller to actuate the light emitting status indicators <NUM> accordingly.

The storage <NUM> can include non-volatile memory such as hard-disk drives, solid-state drives, flash memory, and the like. The storage <NUM> can store application programs such as the operating system, and the various applications that can be run on the client device <NUM>. The storage <NUM> also can store data such as entertainment data, music data, video data, and the like that can be accessed and manipulated by one or more applications running on the client device <NUM>. The network interface <NUM> can include wired and wireless network interfaces which can provide network connectivity to the client device <NUM>. For example, the network interface <NUM> can include WIFI, near field communication, wireless communication, and other wireless network interfaces, and can include Ethernet, DOCSIS, and other hardware network interfaces. The TV operating system <NUM> can provide APIs for applications running on the client system to utilize the network interface <NUM> to communicate data over a network. For example, the digital assistant <NUM> can use the APIs for the network interface <NUM> to communicate with the data processing system <NUM> over the network <NUM> (<FIG>).

The client device <NUM> can be configured such that the display <NUM> can be switched OFF or ON independently of the control module <NUM> and the light emitting status indicators <NUM>. That is, the display <NUM> can be switched OFF while maintaining the control module <NUM> operational. Thus, while the display <NUM> is switched OFF, the digital assistant application <NUM> can be running on the processor <NUM> of the client device <NUM>. The digital assistant application can send and receive data packets from a data processing system, such as the data processing system <NUM> shown in <FIG>, while the display is switched OFF. Further, the digital assistant application <NUM> can affect a change in the state of the display <NUM>, for example from ON state to OFF state, or from OFF state to ON state.

Referring again to <FIG>, the client device configuration appliance <NUM> can communicate with the digital assistant application <NUM> on the client device <NUM> to receive audio signals associated with voice commands and configuration data of the client device <NUM>. As mentioned above, the digital assistant application <NUM> can send configuration data <NUM>, such as, for example, the current state ("ON" or "OFF") of the display device, the current volume settings ("mute" "volume level") of the speakers <NUM>. The configuration data may also include the minimum volume level at which the digital assistant application <NUM> reproduces sound at the client device <NUM>. The NLP <NUM> can parse the audio signals to generate a request and a trigger keyword associated with the voice command. The client device configuration appliance <NUM> can generate an response data structure based on the trigger keyword and the configuration data.

<FIG> illustrates a flow diagram of a method <NUM> to process packetized audio signals in a voice activated computer network. In particular, the method <NUM> can be executed by the data processing system <NUM> shown in <FIG>. At least a portion of the method <NUM> can be executed by the client device configuration appliance <NUM> of the data processing system <NUM>. The method <NUM> can include receiving data packets from a client device where the data packets include audio signals and configuration data (block <NUM>). As discussed above in reference to <FIG> and <FIG>, a user at the client device <NUM> can speak voice commands, which are captured by the microphone <NUM> and converted into input audio signals. These audio signals are provided to the digital assistant application <NUM>. The digital assistant application <NUM> also accesses the configuration data <NUM> to determine the current configuration of the client device, and in particular, the ON/OFF state of the display <NUM>, the current volume setting ("mute" or the volume level), and a minimum response volume level indicating the minimum volume level at which the digital assistant application <NUM> can reproduce sound from the speakers <NUM>. The digital assistant application <NUM> can send the audio signals corresponding to the voice command and the configuration data to the data processing system <NUM> via the network interface <NUM> and the network <NUM>.

The method <NUM> can include parsing the audio signals (block <NUM>). The NLP <NUM> can use language processing to parse the audio signals to determine a request and a trigger keyword. For example, if the audio signal corresponds to a voice command "what's the weather?" then the NLP <NUM> may determine that the phrase "the weather" corresponds to a request while the phrase "what" corresponds to the trigger keyword. The method <NUM> can include determining from the configuration data whether the display of the client device is in an OFF state (block <NUM>). As mentioned above, the configuration data received from the client device <NUM> can include an identifier identifying a display, and the corresponding state of the display. The client device configuration appliance <NUM> can look-up the identity of the display and determine the current state. The state of the display can be either ON of OFF. First assuming that the client device configuration appliance <NUM> determines that the state of the display is OFF, the method <NUM> can include determining whether the response to the voice command received from the user can be generated using a non-video response (block <NUM>). For example, the client device configuration appliance <NUM> can determine that the response to the user command "what's the weather?" can be generated without a video or image output. That is, the response can include an audio response providing the current weather at the location of the client device <NUM>.

The method <NUM> can include generating the audio signals associated with the non-video response (block <NUM>). The client device configuration appliance <NUM> can use the trigger keyword and the request to communicate with the data provider <NUM>, which can be a weather data service provider, and request the weather data associated with the location of the client device <NUM>. Upon receiving the weather data, the client device configuration appliance <NUM> can instruct the audio signal generation component <NUM> to generate audio signals corresponding to the weather data received from the data provider <NUM>. For example, if the data received from the data provider is "<NUM> degrees and sunny" the audio signal generator component <NUM> can utilize the weather data, and the current location information of the client device <NUM> and generate an audio signal for a response such as "Currently in New York it is <NUM> degrees and sunny.

The method <NUM> can include setting the volume levels of the speaker at the client device (block <NUM>). As discussed above, the client device configuration appliance <NUM> is adapted to receive, in the configuration data, the current volume levels of the speakers <NUM> at the client device <NUM>. The volume levels can either be "mute" or a particular volume level within a range of volume levels (say for example between <NUM> and <NUM>). The configuration data is adapted to also include the minimum response volume level indicating the minimum volume level at which the digital assistant application <NUM> can reproduce sound from the speakers <NUM>. The client device configuration appliance <NUM> can select higher of the current volume level and minimum response volume level to be the volume level at which the audio signal corresponding to the generated response is to be delivered to the user at the client device <NUM>. The client device configuration appliance <NUM> is adapted to generate client configuration settings based on the selected volume level. For example, the client device configuration appliance <NUM> can include in the client configuration settings an identifier associated with the speaker volume and an associated volume level.

The method <NUM> can include generating an response data structure corresponding to the response to the voice command received from the client device <NUM> (block <NUM>). The client device configuration appliance <NUM> is adapted to generate a data structure that can include the audio signal and configuration settings. The audio signal can correspond to the audio signal generated by the audio signal generation component <NUM>, and the configuration settings can include the selected volume level (in block <NUM>). The client device configuration appliance <NUM> can also include an identity of the voice command in response to which the audio signal has been generated. The method <NUM> can include transmitting the response data structure to the client device (block <NUM>). The client device configuration appliance <NUM> can transmit the data structure including the audio signal and the configuration settings to the client device <NUM> via the interface <NUM>. The interface <NUM> can convert the data structure into data packets and transmit the data packets to the client device <NUM> over the network <NUM>. At the client device <NUM>, the digital assistant application <NUM> receives the data packets via the network interface <NUM>. The digital assistant application <NUM> regenerates the audio signal and the configuration settings from the data packets. The digital assistant application <NUM> can identify from the identity of the voice command, that the received audio signal is in response to an previously received voice command. The digital assistant application <NUM> can control the volume level of the speakers <NUM> to be set at the volume level included in the received configuration settings. The digital assistant <NUM> can output the audio signal from the speaker <NUM>, thereby providing the user with an audio response to the voice command.

As mentioned above, in block <NUM> the client device configuration appliance <NUM> determines whether the response to the voice command received from the user can be generated using a non-video response. In some instances, the response to the voice command may need the client device configuration appliance <NUM> to generate a video based response. For example, if the voice command of the user is "Show me the weather for this week. " The NLP <NUM> can determine that the phrase "weather for this week" corresponds to the request, while the phrase "Show me" corresponds to the trigger keyword. Based on the trigger keyword being "show" the client device configuration appliance <NUM> may determine that the user is requesting at least a visual response. Assuming that the client device configuration appliance <NUM> determines that a non-video response is not to be generated, the method <NUM> can include generating audio or video signals corresponding to the response (block <NUM>). The client device configuration appliance <NUM> can determine obtain the weather data, as discussed above, from the data provider <NUM>. The client device configuration appliance can then instruct the video signal generator component <NUM> to generate an image or a video signal corresponding to the weather data provided by the data provider <NUM>. Optionally, the client device configuration appliance <NUM> may also instruct the audio signal generator component <NUM> to generate audio signals corresponding to the weather data.

The method <NUM> can include setting the configuration setting for the display to be ON (block <NUM>). The client device configuration appliance <NUM> can generate configuration settings for the display <NUM> of the client device <NUM> to be switched ON, as the current state of the display is OFF and the response includes visual content. The method <NUM> can include setting the volume levels of the speaker at the client device (block <NUM>). The client device configuration appliance <NUM> can set the volume levels of the speakers <NUM> of the client device <NUM> in a manner similar to that discussed above in relation to block <NUM>. That is, the client device configuration appliance <NUM> selects the greater of the current volume level and the minimum response volume level as the volume level of the speaker <NUM> to generate the audio signal. In some instances, where the response does not include an audio signal and only includes the video signal, the client device configuration appliance <NUM> can skip executing block <NUM>.

The method <NUM> can include generating a response data structure corresponding to the response to the voice command received from the client device (block <NUM>). The client device configuration appliance <NUM> can generate the response data structure to include the video signal generated by the video signal generator component <NUM>, and any audio signal generated by the audio signal generator component <NUM>, in response to the voice command. The client device configuration appliance <NUM> can also include configuration settings, which can include instructions to switch the display <NUM> ON, and can include the volume level of the speakers <NUM>. The method <NUM> can include transmitting the response data structure to the client device (block <NUM>). The client device configuration appliance <NUM> can send the generated response data structure to the client device <NUM> in a manner similar to that discussed above in relation to the block <NUM>. The digital assistant application <NUM> can receive the configuration data, and determine that the display <NUM> is to be switched ON. Responsive to this determination, the digital assistant application <NUM> can instruct the display controller <NUM> to switch ON the display <NUM>. The digital assistant application <NUM> may also determine the volume levels from the configuration settings, and accordingly set the volume level of the speaker <NUM> to that indicated in the configuration settings. Thereafter, the digital assistant application <NUM> can send the video or image signal to the display controller <NUM> for rendering on the display <NUM> can send any audio signals to the speaker <NUM> for sound reproduction.

As discussed above in relation to the block <NUM>, the client device configuration appliance <NUM> determines whether the display <NUM> of the display device <NUM> is in the OFF state. Assuming that the client device configuration appliance <NUM> determines that the display device <NUM> is not in the OFF state, and is instead in the ON state, the method <NUM> can include generating audio and video signals corresponding to the voice command (block <NUM>). As the display <NUM> of the client device <NUM> is already in the ON state, the client device configuration appliance <NUM> can responds to the voice command with a visual response. Thus, the client device configuration appliance <NUM> can instruct the video signal generation command to generate a video signal corresponding to a visual response to the voice command. The client device configuration appliance <NUM> may also instruct the audio signal generation component <NUM> to generate an audio signal corresponding to a response to the voice command. The client device configuration appliance <NUM> may forego generating the audio signal, and may only generate the video signal including the visual response to the voice command. The method <NUM> can include setting the volume level of the speakers (block <NUM>), generating a response data structure (block <NUM>) and transmitting the response data structure to the client device (block <NUM>). The client device configuration appliance <NUM> can execute this portion of the method <NUM> (including blocks <NUM>, <NUM>, and <NUM>) in a manner similar to that discussed above in relation to blocks <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>.

The client device configuration appliance <NUM> can instruct the client device to activate light emitting status indicators (block <NUM>). In particular, the client device configuration appliance <NUM> may send these instructions to the client device <NUM> as soon as it receives or parses the data packets including the voice commands. The client device configuration appliance <NUM> can send configuration settings to the client device <NUM> including an identity of the light emitting status indicators <NUM> and the corresponding "ON" setting. The digital assistant application <NUM> can parse the configuration settings and determine that the light emitting status indicators <NUM> are to be switched ON. Responsive to this determination, the digital assistant application <NUM> can instruct the display controller <NUM>, or any other appropriate controller controlling the light emitting status indicators <NUM>, to switch ON the light emitting status indicators <NUM>. The switching ON of the light emitting status indicators <NUM> can indicate to the user that the data processing system <NUM> is processing the voice command provided by the user. The client device configuration appliance <NUM> may send client configuration settings including instructions to switch ON the light emitting status indicators <NUM> only if the configuration data received from the client device <NUM> indicates that the display <NUM> is in the OFF state. The client device configuration appliance <NUM> may refrain from sending instructions to switch ON the light emitting status indicators <NUM> if the configuration data indicates that the display is ON. In some such instances, the client device configuration appliance <NUM> can instead send a status video data or status image data to the client device <NUM> for display on the ON display <NUM> while the data processing system processes the voice command. The status video data or status image data indicating the status of the processing of the voice command can be overlaid on whatever video or image is currently being displayed on the display <NUM>. The video signal generation component <NUM> can generate the status video or status image signal. The video signal generation component <NUM> can generate the status video or status image data or signal that is configured to be positioned on a corner of the display <NUM> and occupies only a small portion (say less than <NUM>%) of the entire area of the display <NUM>.

As discussed above, the client device configuration appliance <NUM> determines the response to the voice command received from a user at the client device <NUM> based in part on the state of the display <NUM>. In instances where responses are generated regardless of the state of the display <NUM>, such responses may include both video signals and audio signals. By generating non-video responses when the display is determined to be in the OFF state, processing resources, which would have otherwise been allocated to generating a video based response, can be saved. Moreover, processing time for generation of video signals can be considerable greater than the processing time for generation of audio signals. By determining the OFF state of the display, and refraining from generating video signals, the overall processing time for generating a response to the voice command can be advantageously reduced. Thus, the user can receive relatively quicker responses to voice commands. In addition, because no video signals are transmitted to the client device over the network when the display <NUM> is determined to be in the OFF state, bandwidth resources of the network <NUM> can be more efficiently utilized.

The natural language processing of the voice commands received from the user can be carried out at the client device, instead of at the data processing system <NUM> in an unclaimed embodiment. For example, referring to <FIG>, the digital assistant application <NUM> can instruct the NLP <NUM> running on the client device <NUM> to process the voice commands received from the user. The digital assistant application <NUM> may then send the identified request and the trigger keywords to the data processing system <NUM>. By processing the voice commands at the client device, audio signals, transmission of which can consume relatively larger bandwidth compared to the transmission of the request and trigger keyword phrases, are not sent over the network, thereby reducing congestion in the network <NUM>.

<FIG> shows a flow diagram of a method <NUM> to process packetized audio signals in a voice activated computer network. The method <NUM> includes transmitting data packets (block <NUM>). For example, the client device <NUM> is adapted to transmit data packets to the data processing system <NUM>. The data packets include an input audio signal and client device configuration data The audio signal can be detected by a sensor or a microphone <NUM> at the client device <NUM>. A digital assistant application <NUM> executed by a processor <NUM> on the client device <NUM> can transmit the data packets via a client device network interface <NUM>, over a network <NUM>.

The method <NUM> includes receiving data packets (block <NUM>). For example, a natural language processor component <NUM> can receive the data packets including the input audio signal and client device configuration data. The natural language processor component <NUM> can be executed by a data processing system <NUM>, and can receive the data packets via a network interface <NUM> of the data processing system <NUM> over the network <NUM>. The natural language processor component <NUM> is adapted to parse the audio signal to identify a request and a trigger keyword corresponding to the request.

The method <NUM> can include determining state of display (block <NUM>). For example, a client device configuration appliance <NUM> executed by the data processing system <NUM> can determine the state of the display <NUM> from the client device configuration data. The state of the display can be an ON state or an OFF state. The method <NUM> can include generating a response data structure (block <NUM>). For example, the client device configuration appliance <NUM> can generate, based in part on the trigger keyword and the OFF state of the display <NUM>, a response data structure including audio data and not including video data responsive to the request. As another example, the client device configuration appliance <NUM> can generate, based in part on the trigger keyword and the OFF state of the display <NUM>, a response data structure including audio data, video data, and client device configuration settings including instructions to switch ON the display <NUM> of the client device <NUM>.

The method <NUM> includes transmitting response data structure to client device (block <NUM>). For example, the client device configuration appliance <NUM> is adapted to transmit over the network <NUM> via a network interface <NUM> of the data processing system <NUM> the response data structure to the digital assistant application <NUM> executed on the client device <NUM>. The response data structure can include audio data and does not include video data. As another example, the response data structure can include audio data, video data, and configuration settings data that instruct the client device <NUM> to change the state of one or more components of the client device <NUM>.

The computer system or computing device <NUM> can include or be used to implement the system <NUM> or its components such as the data processing system <NUM>. The computing system <NUM> includes a bus <NUM> or other communication component for communicating information and a processor <NUM> or processing circuit coupled to the bus <NUM> for processing information. The computing system <NUM> can also include one or more processors <NUM> or processing circuits coupled to the bus for processing information. The computing system <NUM> also includes main memory <NUM>, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus <NUM> for storing information, and instructions to be executed by the processor <NUM>. The main memory <NUM> can be or include the data repository <NUM>. The main memory <NUM> can also be used for storing position information, temporary variables, or other intermediate information during execution of instructions by the processor <NUM>. The computing system <NUM> may further include a read-only memory (ROM) <NUM> or other static storage device coupled to the bus <NUM> for storing static information and instructions for the processor <NUM>. A storage device <NUM>, such as a solid state device, magnetic disk or optical disk, can be coupled to the bus <NUM> to persistently store information and instructions. The storage device <NUM> can include or be part of the data repository <NUM>.

The computing system <NUM> may be coupled via the bus <NUM> to a display <NUM>, such as a liquid crystal display or active matrix display, for displaying information to a user. An input device <NUM>, such as a keyboard including alphanumeric and other keys, may be coupled to the bus <NUM> for communicating information and command selections to the processor <NUM>. The input device <NUM> can include a touch screen display <NUM>. The input device <NUM> can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor <NUM> and for controlling cursor movement on the display <NUM>. The display <NUM> can be part of the data processing system <NUM>, the client computing device <NUM> or other component of <FIG>, for example.

For situations in which the systems discussed herein collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features that may collect personal information (e.g., information about a user's social network, social actions, or activities; a user's preferences; or a user's location), or to control whether or how to receive content from a content server or other data processing system that may be more relevant to the user. In addition, certain data may be anonymized in one or more ways before it is stored or used, so that personally identifiable information is removed when generating parameters. For example, a user's identity may be anonymized so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, postal code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about him or her and used by the content server.

The subject matter and the operations described in this specification can be implemented in digital electronic circuitry or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, data processing apparatuses. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial-access memory array or device, or a combination of one or more of them. While a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices).

The terms "data processing system," "computing device," "component," or "data processing apparatus" encompass various apparatuses, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing. The apparatus can include special-purpose logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The components of system <NUM> can include or share one or more data processing apparatuses, systems, computing devices, or processors.

A computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code).

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs (e.g., components of the data processing system <NUM>) to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatuses can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

The subject matter described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such back end, middleware, or front end components.

The computing system such as system <NUM> or system <NUM> can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network (e.g., the network <NUM>). In some implementations, a server transmits data (e.g., data packets representing a content item) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server (e.g., received by the data processing system <NUM> from the client computing device).

The separation of various system components does not require separation in all implementations, and the described program components can be included in a single hardware or software product. For example, the NLP component <NUM> and the client device configuration appliance <NUM> can be a single component, app, or program, or a logic device having one or more processing circuits, or part of one or more servers of the data processing system <NUM>.

Having now described some illustrative implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. Acts, elements, and features discussed in connection with one implementation are not intended to be excluded from a similar role in other implementations or implementations.

References to "or" may be construed as inclusive so that any terms described using "or" may indicate any of a single, more than one, and all of the described terms. References to "at least one of 'A' and 'B'" can include only 'A', only 'B', as well as both 'A' and 'B'. Such references used in conjunction with "comprising" or other open terminology can include additional items.

Where technical features in the drawings, detailed description, or any claim are followed by reference signs, the reference signs have been included to increase the intelligibility of the drawings, detailed description, and claims.

Claim 1:
A system to process packetized audio signals in a voice activated computer network, comprising:
a digital assistant application executed by a processor of a client device (<NUM>) to transmit data packets via a client device network interface over a network, the client device comprising at least one speaker (<NUM>), the data packets including a first input audio signal, detected by a sensor of the client device, and first client device configuration data comprising a currently set volume of the one or more speakers, a current state, and a minimum response volume level of the speakers;
a natural language processor component (<NUM>) executed by a data processing system to receive, over the network, via a network interface of the data processing system, the data packets, the natural language processor component to parse the first input audio signal to identify a first request and a first trigger keyword corresponding to the first request; and
a client device configuration appliance (<NUM>) executed by the data processing system to, in response to receiving the data packets:
determine, from the first client device configuration data, a current volume level, a current state and a minimum response volume level of at least one speaker at the client device,
generate first client device configuration settings including a first volume level for the at least one speaker at the client device, the generated first volume level being the greater of the current volume level and the minimum response volume level,
generate, based in part on the first trigger keyword, a fist response data structure, the first response data structure including audio data, and
transmit, over the network, via the network interface of the data processing system and to the digital assistant application at the client device:
the first client device configuration settings, and
the first response data structure,
wherein the digital assistant application executed by the processor of the client device is further configured to:
receive, via the network interface of the client device, over the network, the first response data structure responsive to the first request,
determine the audio data of the first response data structure, and
cause the audio data to be played, at the first volume level, on at least one speaker of the client device.