Patent Description:
The present invention is defined by the claims and include a method according to claim <NUM> and a system according to claim <NUM>.

As it is useful for understanding the present invention, there is also described herein a computer-implemented method (not encompassed by the claims) for using voice commands for one or more computing devices. The method includes receiving first voice input data from a first computing device associated with a user account, where the first voice input data comprise a first voice command captured at the first computing device. The method further includes receiving second voice input data from a second computing device associated with the user account, where the second voice input data comprise a second voice command captured at the second computing device. The method further includes determining an intended voice command based on the obtained first and second voice input data. The method further includes determining a first target computing device based on the intended voice command. The method further includes providing first instructions associated with the intended voice command to the first target computing device for execution.

The first voice input data may further comprise a first timestamp associated with the first voice command. The second voice input data may further comprise a second timestamp associated with the second voice command. The method may further comprise determining that the first voice command comprises a first part of the intended voice command and the second voice command comprises a second part of the intended voice command based on a comparison of the first timestamp and the second timestamp, and a comparison of a first location of the first computing device and a second location of the second computing device. Determining the intended voice command may further comprise merging the first part of the intended voice command and the second part of the intended voice command based on the first timestamp and the second timestamp. The method may further comprise determining that each of the first and second voice commands comprises the intended voice command based on a first comparison of the first timestamp and the second timestamp and a second comparison of a first location of the first computing device and a second location of the second computing device. The first voice input data may further comprise a first recognized word segment and a first confidence value. The second voice input data may further comprise a second recognized word segment and a second confidence value. Determining the intended voice command may comprise determining that the first recognized word segment is different from the second recognized word segment; and selecting one of the first recognized word segment or the second recognized word segment based on a comparison of the first confidence value and the second confidence value. The intended voice command may comprise the selected one of the first recognized word segment or the second recognized word segment. The first voice input data may further comprise a first plurality of recognized word segments and their respective first confidence values and the second voice input data may further comprises a second plurality recognized word segments and their respective second confidence values. Determining the intended voice command may further comprise merging the first plurality of recognized word segments and the second plurality of recognized word segments based on the respective first confidence values and the respective second confidence values, wherein the intended voice command comprises the merged first plurality of recognized word segments and second plurality of recognized word segments.

The intended voice command may comprise a device identifier associated with the first target computing device. The first voice input data may further comprise a first sound level associated with the first voice command. The second voice input data may further comprise a second sound level associated with the second voice command. Determining the first target computing device may further be based on a comparison of the first sound level and the second sound level. Determining the first target computing device may further be based on a first comparison of commands available to the user on the first computing device and the intended voice command and a second comparison of commands available to the user on the second computing device and the intended voice command.

The method may further comprise determining a second target computing device based on the intended voice command; and providing second instructions associated with the intended voice command to the second target computing device for execution.

Determining the first target computing device may further be based on a first location of the first computing device and a second location of the second computing device.

Aspects of the subject also relates to a system. The system includes one or more processors and a non-transitory computer-readable medium including instructions stored therein, which, when processed by the one or more processors, cause the one or more processors to perform operations. The operations (not encompassed by the claims) include receiving first voice input data from a first computing device associated with a user account, where the first voice input data comprise a first voice command captured at the first computing device and a first timestamp associated with the first voice command. The operations also include receiving second voice input data from a second computing device associated with the user account, where the second voice input data comprise a second voice command captured at the second computing device and a second timestamp associated with the second voice command. The operations also include determining an intended voice command based on the obtained first and second voice input data. The operations also include determining a first target computing device based on the intended voice command. The operations also include providing first instructions associated with the intended voice command to the first target computing device for execution.

Aspects of the subject technology also relates to a non-transitory machine-readable medium including instructions stored therein (not encompassed by the claims), which when executed by a machine, cause the machine to perform operations. The operations include receiving first voice input data from a first computing device associated with multiple user accounts The operations also include determining, using voice recognition, a first intended voice command associated with a first user account of the plurality of user accounts and a second intended voice command associated with a second user account of the plurality of user accounts based on the first voice input data. The operations also include determining a first target computing device based on the first intended voice command. The operations also include determining a second target computing device based on the second intended voice command. The operations also include providing first instructions associated with the first intended voice command to the first target computing device for execution. The operations also include providing second instructions associated with the second intended voice command to the second target computing device for execution.

It is understood that other configurations of the subject technology will become readily apparent to those skilled in the art from the following detailed description, where various configurations of the subject technology are shown and described by way of illustration. As will be realized, the subject technology is capable of other and different configurations and its several details are capable of modification in various other respects, all without departing from the scope of the appended claims.

The accompanying drawings, which are included to provide further understanding and are incorporated in and constitute a part of this specification, illustrate disclosed aspects and together with the description serve to explain the principles of the disclosed aspects.

The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology may be practiced. However, the subject technology is not limited to the specific details set forth herein and may be practiced without these specific details. In some instances, structures and components are shown in block diagram form in order to avoid obscuring the concepts of the subject technology.

The subject technology enables a user to utilize voice commands to interact with one or more computing devices. According to various aspects, the user may register multiple computing devices with a user account associated with an online or cloud-based service. A user registers a computing device in association with the user account through authentication of user account credentials. User authentication may be initiated by signing into the user account through, for example, a web portal, a web application, an application log-in page, etc. The user registers a computing device in association with the user account by registering a corresponding network or device identifier in association with the user account. Voice commands are captured at least one of any of the multiple computing devices registered with the user account. In some aspects, only voice commands captured at computing devices in which the user is currently logged into the user account may be processed according to the subject technology.

First voice input data from a first computing device associated with a user account may be received. In some aspects, second voice input data from a second computing device associated with the user account may be received. An intended voice command may be determined based on the first voice input data and the second voice input data. A target computing device may be determined based on the intended voice command, and first instructions associated with the intended voice command may be provided to the first target computing device for execution.

In one or more embodiments, the subject technology enables a user to use voice commands to interact with a computing device lacking the capabilities of capturing voice commands. For example, the user may wish to interact with a smart thermostat, which does not have a microphone. A first computing device (e.g., smartphone) may capture a voice command and transmit first voice input data to the server. The server may receive the first voice input data and determine that an intended voice command is for a second computing device different from the first computing device (e.g., smart thermostat). The server may provide instructions associated with the intended voice command to the second computing device.

<FIG> illustrates an example network environment <NUM> in which voice commands may be utilized to interact with multiple computing devices. The network environment <NUM> includes one or more computing devices <NUM>, <NUM> and <NUM>, network <NUM>, and server <NUM>. Server <NUM> can include one or more computing devices <NUM> and one or more data stores <NUM>.

Computing devices <NUM>, <NUM> and <NUM> can represent various forms of processing devices. By way of example and not of limitation, processing devices can include a desktop computer, a laptop computer, a handheld computer, a personal digital assistant (PDA), a cellular telephone, a network appliance, a camera, a smart phone, an enhanced general packet radio service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, smart appliances or a combination of any of these data processing devices or other data processing devices. Some computing devices, such as computing devices <NUM> and <NUM>, may have the capabilities to capture user voice commands. For example, computing devices <NUM> and <NUM> may include microphones, and may have instructions stored in memory, which when executed by their respective processors, allow computing devices <NUM> and <NUM> to record the user voice commands. Other computing devices, such as computing device <NUM>, may not be able to capture user voice commands because, for example, the devices lack a microphone. In addition, computing devices <NUM>, <NUM> and <NUM> may include processing circuitry and/or instructions for speech recognition and voice recognition.

According to various implementations, computing devices <NUM>, <NUM> and <NUM> are associated with an online or cloud-based user account. In some cases, computing devices <NUM>, <NUM> and/or <NUM> may be associated with multiple different cloud-based user accounts. Even when a computing device is associated with multiple different cloud-based user accounts, the computing device may be associated with one current, active user account. For example, multiple users may have previously authenticated user account credentials on a computing device, but there may be one user who is actively signed into the user account on the computing device. Information stored in connection with user accounts may be located in the data store <NUM> associated with the server <NUM>. In some aspects, information stored in connection with users account may be located on a separate server (not pictured).

In some aspects, the server <NUM> is configured to execute computer instructions to process voice commands from one or more computing devices. When a user makes a voice command near a computing device associated with the user's account, such as computing device <NUM> or computing device <NUM>, the voice command is captured and voice input data is transmitted to the server <NUM>. Based on the voice input data received from the one or more computing devices associated with the user's account, server <NUM> determines an intended voice command and provide instructions associated with the intended voice command to a target computing device.

The server <NUM> can be a single computing device (e.g., computing device <NUM>). In other implementations, the server <NUM> can represent more than one computing device working together to perform the actions of a computer server (e.g., server farm). Further, the server <NUM> can represent various forms of servers including, but not limited to, a web server, an application server, a proxy server, a network server, or a server farm.

In some aspects, the computing devices, including computing devices <NUM>, <NUM> and <NUM>, and server <NUM>, may communicate wirelessly through a communication interface (not shown), which may include digital signal processing circuitry where necessary. The communication interface may provide for communications under various modes or protocols, for example, Global System for Mobile communication (GSM) voice calls, Short Message Service (SMS), Enhanced Messaging Service (EMS) or Multimedia Messaging Service (MMS) messaging, Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Personal Digital Cellular (PDC), Wideband Code Division Multiple Access (WCDMA), CDMA2000, or General Packet Radio System (GPRS), etc. For example, the communication may occur through a radio-frequency transceiver (not shown). In addition, short-range communication may occur, for example, using a Bluetooth, WiFi, or other such transceiver.

In some aspects, network environment <NUM> can be a distributed client/server system that spans one or more networks such as, for example, network <NUM>. Network <NUM> can be a large computer network such as, for example, a local area network (LAN), wide area network (WAN), the Internet, a cellular network, or a combination thereof connecting any number of mobile clients, fixed clients, and servers. Further, the network <NUM> can include, but is not limited to, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star-bus network, tree or hierarchical network, and the like. In some aspects, communication between each client (e.g., computing devices <NUM>, <NUM> and <NUM>) and server (e.g., server <NUM>) can occur via a virtual private network (VPN), Secure Shell (SSH) tunnel, or other secure network connection. In some aspects, network <NUM> may further include a corporate network (e.g., intranet) and one or more wireless access points.

<FIG> shows a flowchart illustrating an example process <NUM> for processing voice commands, in accordance with various aspects of the subject technology. Process <NUM> does not need to be performed in the order shown. It is understood that the depicted order is an illustration of one or more example approaches, and the subject technology is not meant to be limited to the specific order or hierarchy presented. Steps can be rearranged, and/or two or more of the steps can be performed simultaneously. While the steps of process <NUM> have been described with respect to two computing devices (e.g., computing device <NUM>, and <NUM>), the subject technology is understood to allow users to process voice commands in association with more than two computing devices.

In block <NUM> of <FIG>, first voice input data is received from a first computing device (e.g., computing device <NUM>) associated with a user account, where the first voice input data comprises a first voice command captured at the first computing device <NUM>. The first voice input data may include, for example, a raw audio file captured at the first computing device <NUM>, processed word segments based on the raw audio file, a location of the first computing device <NUM>, a timestamp, a sound level of the audio file, etc. The server <NUM> may receive the first voice input data comprising the raw audio file from first computing device <NUM>. In some aspects, the server <NUM> may receive the processed word segments from the first computing device <NUM>. The first computing device <NUM> may capture a raw audio file of the first voice command and may process the raw audio file to determine the word segments through, for example, the use of speech recognition. The first computing device <NUM> may send the first voice input data comprising the determined word segments to the server <NUM>.

In block <NUM> of <FIG>, second voice input data is received from a second computing device (e.g., computing device <NUM>) associated with a user account, where the second voice input data comprises a second voice command captured at the second computing device <NUM>. The second voice input data may include, for example, a raw audio file captured at the second computing device <NUM>, processed word segments based on the raw audio file, a location of the second computing device <NUM>, a timestamp, a sound level of the audio, etc. The server <NUM> may receive the second voice input data comprising the raw audio file from the second computing device <NUM>. In some aspects, the server <NUM> may receive the processed word segments from the second computing device <NUM>. The second computing device <NUM> may capture a raw audio file of the first voice command and may process the raw audio file to determine the word segments through, for example, the use of speech recognition. The second computing device <NUM> may send the second voice input data comprising the determined word segments to the server <NUM>.

In one or more implementations, the server <NUM> may determine whether a first voice command captured at the first computing device <NUM> and a second voice command captured at the second computing device <NUM> are related. The server <NUM> may receive voice input data from both the first computing device <NUM> and the second computing device <NUM>. The received voice input data may be associated with the same command. For example, a voice command may be made in proximity of the first computing device <NUM> and the second computing device <NUM>. Each of the computing devices may capture the voice command and send its respective voice input data to the server <NUM>. However, some of the received voice input data may be associated with different commands. For example, a first voice command may be made in the morning and a second voice command may be made in the afternoon. In another example, a first voice command associated with a first user may be captured at the first computing device <NUM> and a second voice command associated with a second user may be captured at the second computing device <NUM>. Accordingly, the subject technology may determine whether the first voice command and the second voice command are related before executing the rest of the process <NUM>.

Various information and techniques may be used to determine whether a first voice input data and a second voice input data are related. In some aspects, the server <NUM> may compare a timestamp associated with the first voice input data with a timestamp associated with the second voice input data. Each of these timestamps may be associated with an internal time in respective computing devices when a voice command is captured. If the timestamp associated with the first voice input data and the timestamp associated with the second voice input data are within a predetermined time threshold, then the server <NUM> may determine that the first voice input data and the second voice input data are related. As described above, the server <NUM> may receive multiple voice input data, and it may be more likely that the first voice input data and the second voice input data are related when the timestamp associated with the first voice input data is temporally proximate to the timestamp associated with the second voice input data.

In certain circumstances, the first computing device <NUM> may capture a first part of the command and the second computing device <NUM> may capture a second part of the command. For example, the user may be moving around from a location near the first computing device <NUM> to a location near the second computing device <NUM> while speaking a voice command. The first computing device <NUM> may have captured only a first part of the voice command and the second computing device <NUM> may have captured only a second part of the voice command. In this case, the first voice input data and the second voice input data are related even though the timestamps associated with the first voice input data and the second voice input data are not identical. As such, the predetermined time threshold may be selected to allow some variability between the timestamp associated with the first voice input data and the timestamp associated with the second voice input data. The first computing device <NUM> and the second computing device <NUM> may synchronize their internal time with the server <NUM> periodically to make sure that a standard time is being used to generate timestamps.

In one or more implementations, the locations of each of the computing devices may also be considered. The server <NUM> may determine that the first voice input data and the second voice input data are related when the first computing device <NUM> and the second computing device <NUM> are located within a predetermined distance threshold. It may be more likely that the first voice input data and second voice input data are related when the location associated with the first computing device <NUM> is proximate to the location associated with the second computing device <NUM>. However, when a user is moving around while issuing a command, the first voice input data and the second voice input data may be related even though the locations associated with the first computing device <NUM> and the second computing device <NUM> are not identical. As such, the predetermined distance threshold may be selected to allow some variability between the location associated with the first computing device <NUM> and the location associated with the second computing device <NUM>. The location of each of the computing devices may be received by the server <NUM> as part of the respective voice input data or may be accessed by the server <NUM>.

In some cases, voice recognition may also be used determine whether a first voice input data and a second voice input data are related. For example, the server <NUM> may access a sample voice recording of the user associated with the user account, and compare the sample voice recording with the first and second voice input data to determine that the first and second voice input data are associated with the user associated with the user account. In another example, the server <NUM> may compare a voice profile associated with the user account with the first and second voice input data to determine that the first and second voice input data are associated with the user associated with the user account.

In block <NUM> of <FIG>, an intended voice command is determined based on the obtained first and second voice input data. In one or more implementations, the server <NUM> may determine that the first voice command associated with the first voice input data comprises a first part of the intended voice command and the second voice command associated with the second voice input data comprises a second part of the intended voice command. For example, the first computing device <NUM> and the second computing device <NUM> may be in different locations. A user may be moving as the user is issuing a voice command (e.g., "raise the temperature by <NUM> degrees"). The first computing device <NUM> may capture the first part of the intended voice command, for example, the phrase "raise the. " The second computing device <NUM> may capture the second part of the intended voice command, for example, the phrase "temperature by <NUM> degrees. " The server <NUM> may merge the first part of the intended voice command and the second part of the intended voice command based on the timestamps of the respective voice input data.

The server <NUM> may compare the timestamp associated with the first voice input data and the timestamp associated with the second voice input data. If the timestamp associated with the first voice input data and the timestamp associated with the second voice input data are within a predetermined time threshold, then the server <NUM> may determine that the first voice input data and the second voice input data are related. In addition, the server <NUM> may use speech recognition to determine the first part of the intended voice command based on the first voice input data and the second part of the intended voice command based on the second voice input data. The server <NUM> may determine that the determined first part of the intended voice command is not associated with commands available on any computing devices associated with the user account.

In response to determining that the first part of the intended voice command is not associated with commands available to the user, the server <NUM> may combine the first part of the intended voice command (e.g., "raise the") and the second part of the voice command (e.g., "temperature by <NUM> degrees") to determine the intended voice command (e.g., "raise the temperature by <NUM> degrees"). The order in which the first part of the intended voice command and the second part of the voice command is combined may be determined based on the timestamps. For example, if the timestamp associated with the first voice input data is earlier than the timestamp associated with the second voice input data, then the second part of the intended voice command may be added to the end of the first part of the intended voice command.

In some aspects, the server <NUM> may receive a first set of recognized word segments from the first computing device <NUM> and a second set of recognized word segments from the second computing device <NUM>. Based on methods known in the art, the computing devices may capture the voice command and process the captured voice command such that each syllable of the captured voice command is parsed and translated into a recognized word segment. The first computing device <NUM> may send the first set of recognized word segments to the server <NUM> and the second computing device <NUM> may send the second set of recognized word segments to the server <NUM>. The server <NUM> may determine that there is an overlap of recognized word segments between the first set of recognized word segments and the second set of recognized word segments. The intended voice command may be determined based on merging the first set of recognized word segments and the second set of recognized word segments.

In one or more implementations, the server <NUM> may determine that the first and second voice commands each comprise the intended voice command based on a first comparison of a first timestamp and a second timestamp and a second comparison of a first location of the first computing device <NUM> and a second location of the second computing device <NUM>. For example, the received first voice input data may comprise, for example, the first location of the first computing device <NUM> and the first timestamp when the first voice command is captured. The received second voice input data may comprise, for example, the second location of the second computing device <NUM> and the second timestamp when the second voice command is captured. The server <NUM> may compare the first timestamp and the second timestamp to determine whether the two timestamps are within a predetermined time threshold. The server <NUM> may compare the first location and the second location to determine whether the two locations are within a predetermined location threshold. Based on the two comparisons, the server <NUM> may determine that the first and second voice commands each comprise the intended voice command.

In some cases, the quality of voice commands captured at different devices may be different. For example, the user may be moving as the user is issuing a command, or the microphone associated with a computing device may be facing away from the user. In some cases, each of the computing devices that detect the voice command may capture a raw audio file of the voice command. Each of the computing devices may process the raw audio file to determine recognized word segments and respective confidence values. Based on methods known in the art, each syllable of the captured voice command is parsed and translated into a recognized word segment. The confidence value may also be calculated and may indicate the probability that the recognized word segment accurately represents the corresponding syllable of the voice command. In some cases, server <NUM> may receive a raw audio file associated with a voice command and may process the raw audio file to determine recognized word segments and respective confidence values.

In one or more implementations, the first voice input data may comprise a first recognized word segment and a first confidence value and the second voice input data further comprises a second recognized word segment and a second confidence value. Similar to the determination that the first voice input data is related to the second voice input data, the server <NUM> may determine that the first recognized word segment is related to the second recognized word segment. For example, the server <NUM> may determine that the difference between the timestamp associated with the first recognized word segment and the timestamp associated with the second recognized word segment is within a predetermined time threshold. The server <NUM> may determine the intended voice command by determining that the first recognized word segment is different from the second recognized word segment. As mentioned above, the quality of voice commands captured at different devices may be different, and the difference may be reflected as the difference in the first confidence value and the second confidence value. The server <NUM> may select one of the first recognized word segment or the second recognized word segment based on a comparison of the first confidence value and the second confidence value. For example, a higher confidence value may indicate that there is a higher probability that a recognized word segment accurately represents the intended voice command. In this case, the server <NUM> may select a word segment that has a higher confidence value.

For example, the user may speak a one syllable voice command, such as "off. " The first computing device <NUM> may process the voice command as captured on the first computing device <NUM> and determine a first recognized word segment, for example, a text indicating "off" and a first confidence value, for example,. The second computing device <NUM> may process the voice command as captured on the second computing device <NUM> and determine a second recognized word segment, for example, a text indicating "of" and a second confidence value, for example,. Each of the computing devices may send the respective recognized word segment to the server <NUM>. The server <NUM> may determine that the first recognized word segment and the second recognized word segment are related because the difference between the timestamp associated with the first recognized word segment and the timestamp associated with the second recognized word segment is below the predetermined time threshold. The server <NUM> may compare the first confidence value and the second confidence value. Based on the comparison of the first confidence value (. <NUM>) and the second confidence value (. <NUM>), the server <NUM> may select "off" as the intended voice command.

In one or more implementations, the first voice input data may comprise a first set of recognized word segments and their respective first confidence values and the second voice input data may comprise a second set of recognized word segments and their respective second confidence values. The server <NUM> may receive the first voice input data and the second voice input data and compare each of first set of word segments with its respective second word segment. The server <NUM> may determine that the first set of recognized word segments is related to the second set of recognized word segments. For example, the server <NUM> may determine that the difference between the timestamp associated with the first set of recognized word segments and the timestamp associated with the second set of recognized word segments is within a predetermined time threshold.

The server <NUM> may merge the first set of recognized word segments and the second set of recognized word segments based on the respective first confidence values, and the respective second confidence values. In some aspects, the server <NUM> may, for each of the first set of recognized word segments, combine a first word segment with the respective second word segment when the first and the second word segments are determined to be the same, and select between the first word segment and the respective second word segment based on their respective confidence values when the first and the second word segments are determined to be different.

For example, the user may speak a voice command, such as "print document one. " The first computing device <NUM> may process the voice command and determine a first set of word segments corresponding to "print document one," where each word segment corresponds to a syllable of the voice command. The second computing device <NUM> may process the voice command and determine a second set of word segments corresponding to "tint document one," where each word segment corresponds to a syllable of the voice command. The server <NUM> may determine that the first set of recognized word segments (for example, "print document one") is related to the second set of recognized word segments (for example, "tint document one") based on timestamps. For example, the timestamp associated with the first set of word segments and the second set of word segments may be the same, which may indicate that the first set of recognized word segments may be related to the second set of recognized word segments. The server <NUM> may determine that a first recognized word segment (for example, "print") among the first set of recognized word segments is different from its respective second recognized word segment (e.g., "tint") among the second set of recognized word segments. As described above, the server <NUM> may select a recognized word segment between the first recognized word segment (e.g., "print") and the second recognized word segment (e.g., "tint") based on their respective confidence values. In this example, the first recognized word segment may have a higher confidence value. According to the subject technology, server <NUM> may select the first word segment (e.g., "print") and combine the remaining word segments of the first and second word segments (e.g., document one) after determining that each of the remaining first word segments and the second word segments are the same. Based on this process, the server <NUM> may determine that the intended voice command is "print document one.

In block <NUM> of <FIG>, a first target computing device is determined based on the intended voice command. Since multiple devices may capture a voice command, conflicts or redundancies in execution of the voice command may occur without implementations of proper method for resolving conflicts.

In one or more implementations, the intended voice command includes a device identifier associated with the first target computing device. For example, the user may state the name of the device that the user wishes to interact with. The device identifier may be stored in the data store <NUM> when the user registered the device with the user account. In some aspects, the server <NUM> receives the device identifier as part of the voice input data. The server <NUM> may compare the stored device identifiers and received device identifiers with the intended voice command, and determine the first target computing device based on the comparison.

In one or more implementations, the received first and second voice input data further comprise a sound level associated with their respective voice command. The server <NUM> may compare a first sound level associated with the first voice command with a second sound level associated with the second voice command. The server <NUM> may determine, for example, that the computing device associated with a louder voice command is the first target computing device. The user may be more likely to interact with the computing device closer to the user and a computing device closer to the user may be associated with the louder voice command. In some cases, the sound level captured at the first computing device <NUM> and the second computing device <NUM> may be different even when the user is the same distance away from the first computing device <NUM> and the second computing device <NUM> due to, for example, the quality of microphones. The server <NUM> may receive data associated with microphones of the first computing device <NUM> and second computing device <NUM>, and may standardize the first and the second sound levels based on their respective data associated with the microphone before comparing the first and second sound levels.

In one or more implementations, the received first and second voice input data comprise data associated with commands available to the user on the respective computing devices. The server <NUM> may compare the first data associated with commands available to the first computing device <NUM> to the intended voice command. The server <NUM> may also compare the second data associated with commands available to the second computing device <NUM> to the intended voice command. If the intended voice command is available to both the first computing device <NUM> and the second computing device <NUM>, then other methods, such as the ones mentioned above, may be used to determine the first target computing device. If the intended voice command is available to the first computing device <NUM>, but not available to the second computing device <NUM>, then server <NUM> may determine that the first computing device <NUM> is the first target computing device. If the intended voice command is available to the second computing device <NUM>, but not available to the first computing device <NUM>, then server <NUM> may determine that the second computing device <NUM> is the first target computing device.

In some cases, the server <NUM> may have access to previously received voice input data. In particular, if the first and second voice input data were received within a predetermined time threshold after the previously received voice input data, then the server <NUM> may determine the first target computing device based on a previously determined target computing device.

For example, the server <NUM> may determine that the intended task is to "raise the volume by <NUM>. " The server <NUM> may not be able to identify the first target computing device because multiple computing devices associated with the user account (e.g., a TV, radio, other music player) may perform the intended task of "rais[ing] the volume by <NUM>. " However, if the user had previously spoken a voice command, such as "turn on the radio," then the server <NUM> may determine the first target computing device based on previously determined target computing device associated with the previous voice command (e.g., turn on the radio). The server <NUM> may have previously determined that a radio, for example, the first computing device <NUM>, was the target computing device. If the time difference between receiving the first voice input data and data associated with the previously spoken voice command is less than a predetermined time threshold, then the server <NUM> may determine that the first target computing device is the first computing device <NUM>.

In one or more implementations, the server <NUM> may not be able to identify a first target computing device even after performing the methods described above. In this case, the server <NUM> may have determined that both the first computing device <NUM> and the second computing device <NUM> may execute the user command. The server <NUM> may choose between the first computing device <NUM> and the second computing device <NUM> based on additional background data. In some aspects, the server <NUM> may have access to background data associated with each of the computing devices associated with the user account. The background data may include, for example, frequency and duration of user use of a computing device, current battery level, screen size (if applicable), etc. In some aspects, the server <NUM> may determine the first target computing device based on a comparison of background data associated with the first computing device <NUM> and background data associated with the second computing device <NUM>. For example, if the frequency of user use of the first computing device <NUM> is higher than the frequency of user use of the second computing device <NUM>, then the server <NUM> may determine that the first computing device <NUM> is the target computing device. In another example, if the current battery level of the first computing device <NUM> is higher than the current battery level of the second computing device <NUM>, then the server <NUM> may determine that the first computing device <NUM> is the target computing device.

In block <NUM> of <FIG>, instructions associated with the intended voice command are provided to the first target computing device for execution. In some aspects, the intended voice command may be associated with a first target computing device and a second target computing device. The server may receive first voice input data from a first computing device <NUM> and a second voice input data from a second computing device <NUM>. The server <NUM> may determine an intended voice command and determine the first target computing device and the second target computing device. The server <NUM> may provide first instructions associated with the intended voice command to the first target computing device and second instructions associated with the intended voice command to the second target computing device. For example, the user may wish to transfer photos from the first computing device <NUM> to second computing device <NUM>. The first target computing device may be the first computing device <NUM> and the second target computing device may be the second computing device <NUM>. The first instructions may be associated with initiating a photo transfer application on the first computing device <NUM>. The second instructions may be associated with accepting the photo transfers from the first computing device <NUM> on the second computing device <NUM>. In some aspects, the first instructions associated with the intended voice command and the second instructions associated with the intended voice command may be the same. For example, the user may wish to "turn off" multiple computing devices. The server <NUM> may send the same instructions to the first target computing device and the second target computing device.

In one or more implementations, the server <NUM> may receive user feedback data associated with the provided instructions. In some aspects, after the first instructions are provided to the first target computing device, the server <NUM> may determine that receiving an indication of user interaction from a computing device that is not the first target computing device within a predetermined time threshold may indicate that the determination of the intended voice command was incorrect. The server <NUM> may store an entry of the first voice input data, the second voice input data and the indication of user interaction for future reference. Next time the server <NUM> receives voice input data, the server <NUM> may compare the voice input data with previously stored entries. Future determination of an intended voice command and a target computing device may be further based on the previously stored entries.

Although the first computing device <NUM> and the second computing device <NUM> are described as being associated with a single user account, it is understood that the first computing device <NUM> and the second computing device <NUM> may be associated with different user accounts. For example, the first computing device <NUM> may receive a first voice command from a first user and the second computing device <NUM> may receive a second voice command from a second user. The first and the second voice commands may be associated with the same target computing device. The server <NUM> may receive first voice input data comprising the first voice command from the first computing device <NUM> and second voice input data comprising the second voice command from the second computing device <NUM>. The server <NUM> may determine that the first voice input data are associated with the first user of the first user account and that the second voice input data are associated with the second user of the second user account. The server <NUM> may determine a first intended voice command based on the first voice input data and a second intended voice command based on the second voice input data. The server <NUM> may further determine that target computing device is associated with the first user account and the second user account and that the first intended voice command and the second intended voice command are conflicting. In some cases, the server <NUM> may send instructions to computing devices <NUM> and <NUM>. In response, the computing devices <NUM> and <NUM> may provide for display graphical user elements to receive further instructions or confirmations from the first and the second users. The users may select which instruction has priority. In other cases, certain user accounts associated with the target computing device may have higher priority or privileges. In this case, the server <NUM> may transmit instructions associated with a user account with the highest priority to the target computing device.

shows a flowchart illustrating an example process <NUM> for providing first instructions associated with an intended voice command to a target computing device, in accordance with various aspects of the subject technology. The steps of the process <NUM> do not need to be performed in the order shown. It is understood that the depicted order is an illustration of one or more example approaches, and the subject technology is not meant to be limited to the specific order or hierarchy presented. The steps can be rearranged, and/or two or more of the steps can be performed simultaneously.

In block <NUM> of <FIG>, first voice input data is received from a first computing device associated with multiple user accounts, where the first voice input data comprise a first voice command associated with a first user account of the multiple user accounts and a second voice command associated with a second user account of the multiple user accounts. A first user and a second user may speak voice commands within a predetermined time threshold near the first computing device <NUM> and the first computing device <NUM> may capture both the first voice command and the second voice command as, for example, a single audio file and send the audio file to the server <NUM>.

In block <NUM> of <FIG>, a first intended voice command and a second intended voice command are determined based on the first voice input data. The server <NUM> may use voice recognition techniques to identify users associated with the obtained first voice input data. For example, the server <NUM> may receive an audio file comprising multiple commands from multiple users. The server <NUM> may separate the audio file into multiple portions, where the portions of the raw audio file may be associated with different users.

For example, the first computing device <NUM> may be associated with a first user account and a second user account. The first user may speak a voice command, such as, "raise the volume of the TV by <NUM>" near the first computing device <NUM> and the second user may also issue a voice command, such as, "raise the temperature to <NUM>" near the first computing device <NUM>. The two voice commands may be detected in close temporal proximity of each other and may overlap. For instance, the phrase "raise the volume of the TV by" may be detected at t<NUM>, the phrase "raise the temperature to" may be detected at t<NUM>, the phrase "<NUM>" may be detected at t<NUM>, and the phrase "<NUM>" may be detected at t<NUM>. The first computing device <NUM> may determine the speaker associated with each of the phrases that was detected through use of voice recognition techniques. The first computing device <NUM> may determine that the phrase "raise the volume of the TV by" and "<NUM>" are associated with the first user based on, for example, comparison of a voice profile associated with the first user account and the detected phrases. The first computing device <NUM> may further determine that the phrase "raise the temperature to" and "<NUM>" are associated with the second user based on, for example, comparison of a voice profile associated with the second user account and the detected phrases. Based on these determinations, the first computing device <NUM> may create a first portion of the raw audio file associated with the first user and a second portion of the raw audio file associated with the second user. These portions of the raw audio file may be sent to the server <NUM>. In some implementations, the server <NUM> may receive a raw audio file associated with the first user and the second user, and may distinguish the command from the first user and the command from the second user based on the above process.

In block <NUM> of <FIG>, first target computing device is determined based on the first intended voice command. In block <NUM> of <FIG>, second target computing device is determined based on the second intended voice command. The first and the second target computing device may not have a microphone and/or may not have captured the voice command. For example, the user may wish to change the temperature of the house, and may speak a voice command, such as "raise temperature by three degrees," near the user's smartphone. Computing device <NUM> (e.g., a smartphone) may capture the voice command and send the first voice input data to the server <NUM>. The server <NUM> may determine that a computing device <NUM> (e.g., a smart thermostat) is the first target computing device. In some implementations, the voice command may include a device identifier. The server <NUM> may access a list of computing devices associated with the user account and determine that the computing device <NUM> is the first target computing device. In some implementations, the computing device <NUM> may send data, which may comprise for example, location of the computing device <NUM>, commands available to the user on computing device <NUM>, etc. The server <NUM> may compare the intended voice command to the commands available to the user on computing devices associated with the user account. If the intended voice command is a command available to the user on computing device <NUM>, then server <NUM> may determine that the computing device <NUM> is the first target computing device.

In block <NUM> of <FIG>, first instructions associated with the first intended voice command are provided to the first target computing device for execution. In block <NUM> of <FIG>, second instructions associated with the second intended voice command to the second target computing device is provided for execution.

<FIG> conceptually illustrates an example electronic system with which some implementations of the subject technology can be implemented. Electronic system <NUM> can be a computer, phone, PDA, or any other sort of electronic device. Such an electronic system includes various types of computer readable media and interfaces for various other types of computer readable media. Electronic system <NUM> includes a bus <NUM>, processing unit(s) <NUM>, a system memory <NUM>, a read-only memory (ROM) <NUM>, a permanent storage device <NUM>, an input device interface <NUM>, an output device interface <NUM>, and a network interface <NUM>.

Bus <NUM> collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of electronic system <NUM>. For instance, bus <NUM> communicatively connects processing unit(s) <NUM> with ROM <NUM>, system memory <NUM>, and permanent storage device <NUM>.

From these various memory units, processing unit(s) <NUM> retrieves instructions to execute and data to process in order to execute the processes of the subject disclosure. The processing unit(s) can be a single processor or a multi-core processor in different implementations.

ROM <NUM> stores static data and instructions that are needed by processing unit(s) <NUM> and other modules of the electronic system. Permanent storage device <NUM>, on the other hand, is a read-and-write memory device. This device is a non-volatile memory unit that stores instructions and data even when electronic system <NUM> is off. Some implementations of the subject disclosure use a mass-storage device (for example, a magnetic or optical disk and its corresponding disk drive) as permanent storage device <NUM>.

Other implementations use a removable storage device (for example, a floppy disk, flash drive, and its corresponding disk drive) as permanent storage device <NUM>. Like permanent storage device <NUM>, system memory <NUM> is a read-and-write memory device. However, unlike storage device <NUM>, system memory <NUM> is a volatile read-and-write memory, such as a random access memory. System memory <NUM> stores some of the instructions and data that the processor needs at runtime. In some implementations, the processes of the subject disclosure are stored in system memory <NUM>, permanent storage device <NUM>, or ROM <NUM>. For example, the various memory units include instructions for displaying web pages, processing user entries to the web pages, and generating URLs, in accordance with some implementations. From these various memory units, processing unit(s) <NUM> retrieves instructions to execute and data to process in order to execute the processes of some implementations.

Bus <NUM> also connects to input and output device interfaces <NUM> and <NUM>. Input device interface <NUM> enables the user to communicate information and select commands to the electronic system. Input devices used with input device interface <NUM> include, for example, alphanumeric keyboards and pointing devices (also called "cursor control devices"). Output device interfaces <NUM> enables, for example, the display of images generated by the electronic system <NUM>. Output devices used with output device interface <NUM> include, for example, printers and display devices, for example, cathode ray tubes (CRT) or liquid crystal displays (LCD). Some implementations include devices, for example, a touchscreen that functions as both input and output devices.

Finally, as shown in <FIG>, bus <NUM> also couples electronic system <NUM> to a network (not shown) through a network interface <NUM>. In this manner, the computer can be a part of a network of computers (for example, a local area network (LAN), a wide area network (WAN), or an Intranet, or a network of networks, for example, the Internet. Any or all components of electronic system <NUM> can be used in conjunction with the subject disclosure.

Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer readable storage medium (also referred to as computer readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. The computer readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.

In this specification, the term "software" is meant to include firmware residing in read-only memory or applications stored in magnetic storage, which can be read into memory for processing by a processor. Also, in some implementations, multiple software aspects of the subject disclosure can be implemented as sub-parts of a larger program while remaining distinct software aspects of the subject disclosure. In some implementations, multiple software aspects can also be implemented as separate programs. Finally, any combination of separate programs that together implement a software aspect described here is within the scope of the subject disclosure. In some implementations, the software programs, when installed to operate on one or more electronic systems, define one or more specific machine implementations that execute and perform the operations of the software programs.

A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment.

These functions described above can be implemented in digital electronic circuitry, in computer software, firmware, or hardware. The techniques can be implemented using one or more computer program products. Programmable processors and computers can be included in or packaged as mobile devices. The processes and logic flows can be performed by one or more programmable processors and by one or more programmable logic circuitry. General and special purpose computing devices and storage devices can be interconnected through communication networks.

Some implementations include electronic components, for example, microprocessors, storage, and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, read-only compact discs (CD-ROM), recordable compact discs (CD-R), rewritable compact discs (CD-RW), read-only digital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a variety of recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), magnetic or solid state hard drives, read-only and recordable Blu-Ray® discs, ultra density optical discs, any other optical or magnetic media, and floppy disks. The computer-readable media can store a computer program that is executable by at least one processing unit and includes sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, for example, is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.

While the above discussion primarily refers to microprocessor or multi-core processors that execute software, some implementations are performed by one or more integrated circuits, for example, application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions that are stored on the circuit itself.

To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

It is understood that any specific order or hierarchy of steps in the processes disclosed is an illustration of example approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged, or that all illustrated steps be performed. Some of the steps may be performed simultaneously. For example, in certain circumstances, multitasking and parallel processing may be advantageous.

Thus, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean "one and only one" unless specifically so stated, but rather "one or more". Unless specifically stated otherwise, the term "some" refers to one or more. Headings and subheadings, if any, are used for convenience only and do not limit the subject disclosure.

Claim 1:
A computer-implemented method comprising:
registering, at a server, a first computing device in association with a cloud-based user account for a user, the registering comprising:
authenticating user account credentials for the cloud-based user account; and
associating with the user account a device identifier associated with the first computing device;
receiving, at the server, from a second computing device associated with the cloud-based user account for the user, a voice command spoken by the user, wherein:
the second computing device captured the voice command spoken by the user; and
the voice command includes the device identifier associated with the first computing device; determining, by the server, an intended voice command based on the captured voice command,
identifying, by the server, based on the intended voice command, the first computing device as a target computing device for providing instructions associated with the intended voice command; and
transmitting, by the server, to the first computing device, instructions associated with the intended voice command for execution by the first computing device;
wherein the server, the first computing device and the second computing device are in communication with one another via a network.