Source: http://patents.com/us-9972320.html
Timestamp: 2018-11-12 22:10:23
Document Index: 497726629

Matched Legal Cases: ['Application No. 10', 'Application No. 16195834', 'Application No. 16207288', 'Application No. 10', 'Application No. 2016', 'Application No. 10', 'Application No. 15781811', 'Application No. 10', 'Application No. 2016', 'Application No. 10', 'Application No. 10', 'Application No. 62']

US Patent # 9,972,320. Hotword detection on multiple devices - Patents.com
United States Patent 9,972,320
Melendo Casado , et al. May 15, 2018
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for hotword detection on multiple devices are disclosed. In one aspect, a method includes the actions of receiving audio data that corresponds to an utterance. The actions further include determining that the utterance likely includes a particular, predefined hotword. The actions further include transmitting (i) data indicating that the computing device likely received the particular, predefined hotword, (ii) data identifying the computing device, and (iii) data identifying a group of nearby computing devices that includes the computing device. The actions further include receiving an instruction to commence speech recognition processing on the audio data. The actions further include in response to receiving the instruction to commence speech recognition processing on the audio data, processing at least a portion of the audio data using an automated speech recognizer on the computing device.
Melendo Casado; Diego (San Francisco, CA), Gruenstein; Alexander H. (Mountain View, CA), Foerster; Jakob Nicolaus (Oxford, GB)
Family ID: 1000003293521
15/278,269
US 20180061419 A1 Mar 1, 2018
62378869 Aug 24, 2016
Current CPC Class: G10L 15/30 (20130101); G10L 25/78 (20130101); H04L 67/10 (20130101); G10L 2015/088 (20130101); G10L 15/16 (20130101)
Current International Class: G10L 21/00 (20130101); H04L 29/08 (20060101); G10L 25/78 (20130101); G10L 15/30 (20130101); G10L 15/08 (20060101); G10L 15/16 (20060101)
Field of Search: ;704/243,9,275,233,236,232,270,270.1,251,255,257,1-10 ;709/203 ;713/323
5659665 August 1997 Whelpley, Jr.
5895448 April 1999 Vysotsky et al.
6023676 February 2000 Erell
6671672 December 2003 Heck
6744860 June 2004 Schrage
6973426 December 2005 Schier
7016833 March 2006 Gable
7904297 March 2011 Mirkovic et al.
8209184 June 2012 Dragosh et al.
8543402 September 2013 Ma
8595007 November 2013 Kane
8645137 February 2014 Bellegarda et al.
8709018 May 2014 Dinerstein
8719018 May 2014 Dinerstein
8768687 July 2014 Quasthoff et al.
2002/0049596 April 2002 Burchard et al.
2002/0072905 June 2002 White
2002/0123890 September 2002 Kopp
2002/0193991 December 2002 Bennett et al.
2003/0200090 October 2003 Kawazoe
2003/0220797 November 2003 Ito
2004/0158457 August 2004 Veprek et al.
2005/0135583 June 2005 Kardos et al.
2005/0165607 July 2005 Di Fabbrizio et al.
2006/0184370 August 2006 Kwak et al.
2007/0100620 May 2007 Tavares
2007/0198849 August 2007 Mozer
2009/0240488 September 2009 White
2009/0258333 October 2009 Yu
2009/0292541 November 2009 Daya
2010/0070276 March 2010 Wasserblat
2010/0110834 May 2010 Kim
2011/0026722 February 2011 Jing
2011/0054892 March 2011 Jung
2011/0066429 March 2011 Shperling
2011/0282661 November 2011 Dobry et al.
2011/0304648 December 2011 Kim
2012/0136923 May 2012 Grube
2012/0232896 September 2012 Taleb
2013/0006634 January 2013 Grokop et al.
2013/0024196 January 2013 Ganong et al.
2013/0060571 March 2013 Soemo et al.
2013/0132086 May 2013 Xu et al.
2013/0183944 July 2013 Mozer
2013/0262873 October 2013 Read et al.
2013/0289996 October 2013 Fry
2013/0325484 December 2013 Chakladar et al.
2014/0012573 January 2014 Hung
2014/0074471 March 2014 Sankar et al.
2014/0163978 June 2014 Basye
2014/0222430 August 2014 Rao
2014/0257821 September 2014 Adams
2014/0278383 September 2014 Fan
2014/0278435 September 2014 Ganong, III
2014/0337032 November 2014 Aleksic et al.
2015/0081295 March 2015 Yun
2015/0154953 June 2015 Bapat
2015/0228274 August 2015 Leppanen
2015/0262577 September 2015 Nomura
2015/0379989 December 2015 Balasubramanian
2016/0155443 June 2016 Khan
2016/0189715 June 2016 Nishikawa
2016/0217790 July 2016 Sharifi
2016/0260431 September 2016 Newendorp
1511277 Mar 2005 EP
S59-180599 Oct 1984 JP
H11-52976 Feb 1999 JP
H11-231896 Aug 1999 JP
2006-227634 Aug 2006 JP
10-2014-0031391 Mar 2014 KR
WO 1998/040875 Sep 1998 WO
2014/008194 Jan 2014 WO
2015/025330 Feb 2015 WO
International Preliminary Report on Patentability in International application No. PCT/US2015/030569, dated Jan. 24, 2017, 8 pages. cited by applicant .
Office Action in Korean Application No. 10-2016-7021778, dated Dec. 13, 2016, 4 pages. cited by applicant .
Extended European Search Report issued in Application No. 16195834.3-1910, dated Nov. 23, 2016, 9 pages. cited by applicant .
Extended European Search Report in European Application No. 16207288.8-1914, dated Apr. 4, 2017, 8 pages. cited by applicant .
Office Action issued in Korean Application No. 10-2016-7027999, dated Apr. 12, 2017, 9 pages (English Translation). cited by applicant .
Jae-Seung, Choi, "Text-dependent Speaker Recognition using Characteristic Vectors in Speech Signal and Normalized Recognition Method," Journal of Korean Institute of Information Technology, 10(5), May 2012, 7 pages (English Abstract). cited by applicant .
Auckenthaler et al. "Score Normalization for Text-independent Speaker Verification System," Digital Signal Processing, vol. 10, 2000, 13 pages. cited by applicant .
International Preliminary Report on Patentability in International Application No. PCT/US2015/052860, dated Apr. 11, 2017, 9 pages. cited by applicant .
"Speaker Recognition", Wikipedia, the free encyclopedia, 5 pages, last modified on Jan. 30, 2014; downloaded from the internet on May 15, 2014, http://en.wikipedia.org/wiki/Speaker_recognition. cited by applicant .
"Voice Biometrics Evaluation Studio.TM. 8.2 :: Product Description", Customer Care Solutions from Nuance, 2012, 6 pages. cited by applicant .
Germain, Francois G. et al., "Speaker and Noise Independent Voice Activity Detection", Mar. 26, 2013, 9 pages. cited by applicant .
Hurmalainen, Antti et al., "Detection, Separation and Recognition of Speech from Continuous Signals Using Spectral Factorisation", 20.sup.th European Signal Processing Conference (EUSIPCO 2012), Bucharest Romania, Aug. 27-31, 2012, 5 pages. cited by applicant .
Kim, Youngmoo E. et al., "Comparison of a Joint Iterative Method for Multiple Speaker Identification with Sequential Blind Source Separation and Speaker Identification", Proceedings of the 2008 IEEE Odyssey Workshop on Speaker and Language Recognition, Stellenbosch, South Africa, 8 pages. cited by applicant .
Kotti, Margarita et al., "Speaker Segmentation and Clustering", Preprint submitted to Elsevier Preprint Oct. 31, 2007, 54 pages. cited by applicant .
Kwon, Soonil et al., "Unsupervised Speaker Indexing Using Generic Models", IEEE Transaction on Speech and Audio Processing, vol. 13, No. 5, Sep. 2005, 10 pages. cited by applicant .
Maes, Stephanie H. et al., "Conversational Speech Biometrics," J. Liu and Y. Ye (Eds.): E-Commerce Agents, LNAI 2033, Springer-Verlag, Berlin Heidelberg 2001, 14 pages. cited by applicant .
Navratil, U.V. et al., "A Speech Biometrics System with Multi-Grained Speaker Modeling", 2000, 5 pages. cited by applicant .
Ramirez, Javier et al., "Statistical Voice Activity Detection Using a Multiple Observation Likelihood Ratio Test", IEEE Signal Processing Letters, vol. 12, No. 10, Oct. 2005, 4 pages. cited by applicant .
Schabus, Dietmar et al., "Objective and Subjective Feature Evaluation for Speaker-Adaptive visual Speech Synthesis", International Conference on Auditory-Visual Processing, 2013, 6 pages. cited by applicant .
Walsh, John MacLaren et al., Joint Iterative Multi-Speaker Identification and Source Separation Using Expectation Propagation:, Proceedings of the 2007 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, New Paltz, NY, 4 pages. cited by applicant .
Office Action issued in U.S. Appl. No. 15/346,914, dated Jun. 20, 2017, 25 pages. cited by applicant .
Office Action in Japanese Application No. 2016-551250, dated Apr. 10, 2017, 6 pages (English Translation). cited by applicant .
Notice of Allowance in Korean Application No. 10-2016-7021778, dated Apr. 25, 2017, (English Translation) 3 pages. cited by applicant .
Notice of Allowance in U.S. Appl. No. 15/201,972, dated Jun. 1, 2017, 10 pages. cited by applicant .
European Office Action issued in Application No. 15781811.3, dated Aug. 23, 2017, 6 pages. cited by applicant .
International Search Report and Written Opinion issued in International Application No. PCT/US2017/045123, dated Nov. 8, 2017, 15 pages. cited by applicant .
Korean Office Action issued in Application No. 10-2016-7033161, dated Dec. 19, 2017, 10 pages (with English Translation). cited by applicant .
Notice of Allowance in Japanese Application No. 2016-551250, dated Aug. 7, 2017, 3 pages (English Translation). cited by applicant .
Office Action issued in Korean Application No. 10-2016-7022701, dated Aug. 17, 2017, 4 pages (English Translation). cited by applicant .
Korean Office Action issued in Application No. 10-2016-7027999, dated Aug. 28, 2017, 9 pages (with English Translation. cited by applicant.
This application claims the benefit of U.S. Provisional Application No. 62/378,869, filed Aug. 24, 2016, the contents of which are incorporated by reference herein.
1. A computer-implemented method comprising: receiving, by a first computing device that is configured to respond to a particular, predefined hotword and from a second computing device that is in a vicinity of the first computing device, data indicating that the second computing device is configured to respond to the particular, predefined hotword; transmitting, to the second computing device and by the first computing device, data indicating that the first computing device is configured to respond to the particular, predefined hotword; determining a group identifier that identifies the first computing device and the second computing device; receiving, by the first computing device, audio data that corresponds to an utterance; determining that the utterance likely includes the particular, predefined hotword; in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to a server, (i) data indicating that the first computing device likely received the particular, predefined hotword, (ii) data identifying the first computing device, and (iii) the group identifier that identifies the first computing device and the second computing device; receiving, from the server, an instruction to commence speech recognition processing on the audio data; and in response to receiving the instruction to commence speech recognition processing on the audio data, processing at least a portion of the audio data using an automated speech recognizer on the first computing device.
2. The method of claim 1, comprising: receiving, by the first computing device, additional audio data that corresponds to an additional utterance; determining that the additional utterance likely includes the particular, predefined hotword; in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, (i) the data indicating that the first computing device likely received the particular, predefined hotword, (ii) the data identifying the first computing device, and (iii) the data identifying the group of nearby computing devices that includes the first computing device; receiving, from the server, an instruction to not commence speech recognition processing on the audio data; and in response to receiving the instruction to not commence speech recognition processing on the audio data, ceasing processing of the audio data using the automated speech recognizer on the first computing device.
3. The method of claim 1, comprising: determining a loudness of the audio data associated with the particular, predefined hotword; and in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, the loudness of the audio data associated with the particular, predefined hotword.
4. The method of claim 3, wherein determining a loudness of the audio data associated with the particular, predefined hotword comprises: determining a power of the audio data associated with the particular, predefined hotword; and determining a power of audio data that is not associated with the particular, predefined hotword and that the first computing device received before the audio data associated with the particular, predefined hotword, wherein the loudness of the audio data associated with the particular, predefined hotword is based on the power of the audio data associated with the particular, predefined hotword and the power of the audio data that is not associated with the particular, predefined hotword and that the first computing device received before the audio data associated with the particular, predefined hotword.
5. The method of claim 1, comprising: determining a confidence score that reflects a likelihood that the audio data associated with the particular, predefined hotword corresponds to the particular, predefined hotword; and in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, confidence score.
6. The method of claim 5, wherein determining a confidence score that reflects a likelihood that the audio data associated with the particular, predefined hotword corresponds to the particular, predefined hotword comprises: determining audio features from the audio data associated with the particular, predefined hotword; and based on the audio features, determining, using a neural network, the confidence score.
7. The method of claim 1, comprising: in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, data indicating a location of the first computing device.
8. The method of claim 1, comprising: in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, data indicating an elapsed time since a previous use of the first computing device.
9. The method of claim 1, comprising: in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, data indicating a previous action performed by the first computing device.
10. A computer-implemented method comprising: receiving, by a server and from a first computing device, data indicating that the first computing device is configured to respond to a particular, predefined hotword; receiving, by the server and from a second computing device that is in a vicinity of the first computing device, data indicating that the second computing device is configured to respond to the particular, predefined hotword; determining a group identifier that identifies the first computing device and the second computing device; receiving, by a server and from the first device, (i) data indicating that the first computing device likely received the particular, predefined hotword, (ii) data identifying the first computing device, and (iii) the group identifier that identifies the first computing device and the second computing device; accessing context data that indicates a context of the first computing device; based on the context data of the first computing device, determining that the first computing device commence speech recognition processing on audio data associated with the particular, predefined hotword; and transmitting, to the first computing device, an instruction to commence speech recognition processing on the audio data associated with the particular, predefined hotword.
11. The method of claim 10, comprising: receiving, by the server and from the second computing device, (i) data indicating that the second computing device likely received the particular, predefined hotword, (ii) data identifying the second computing device, and (iii) the group identifier that identifies the first computing device and the second computing device; accessing context data that indicates a context of the second computing device; based on the context data of the first computing device and the context data of the second computing device, determining that the second computing device not commence speech recognition processing on the audio data associated with the particular, predefined hotword; and transmitting, to the second computing device, an instruction to not commence speech recognition processing on the audio data associated with the particular, predefined hotword.
12. The method of claim 10, wherein the audio data associated with the particular, predefined hotword is audio data that corresponds to an utterance.
13. The method of claim 10, wherein the context data that indicates a context of the first computing device comprises data indicating one of more capabilities of the first computing device, wherein determining that the first computing device commence speech recognition processing on audio data associated with the particular, predefined hotword is based on the one of more capabilities of the first computing device.
14. The method of claim 10, wherein the context data that indicates a context of the first computing device comprises data indicating an elapsed time since a previous use of the first computing device, wherein determining that the first computing device commence speech recognition processing on audio data associated with the particular, predefined hotword is based on the elapsed time since the previous use of the first computing device.
15. The method of claim 10, wherein context data that indicates a context of the first computing device comprises data indicating a previous action performed by the first computing device, wherein determining that the first computing device commence speech recognition processing on audio data associated with the particular, predefined hotword is based on the previous action performed by the first computing device.
16. The method of claim 10, comprising: receiving, from the first computing device data indicating a loudness of the audio data associated with the particular, predefined hotword, wherein determining that the first computing device commence speech recognition processing on the audio data associated with the particular, predefined hotword is further based on the loudness of the audio data associated with the particular, predefined hotword.
17. The method of claim 10, comprising: receiving, by the server and from the first computing device, data indicating a location of the first computing device, wherein determining that the first computing device commence speech recognition processing on audio data associated with the particular, predefined hotword is further based on the location of the first computing device.
18. The method of claim 10, comprising: receiving, from the first computing device, data indicating an action associated with the audio data and data indicating actions performed by the first computing device within a particular period of time after receiving the instruction to commence speech recognition processing on the audio data associated with the particular, predefined hotword; and updating the context data of the first computing device based on the data indicating the action associated with the audio data and the data indicating the actions performed by the first computing device within the particular period of time after receiving the instruction to commence speech recognition processing on the audio data associated with the particular, predefined hotword.
This specification generally relates to automated speech recognition.
For a speech-enabled system, the users' manner of interacting with the system is designed to be primarily, if not exclusively, by means of voice input. Consequently, the system, which potentially picks up all utterances made in the surrounding environment including those not directed to the system, must have some way of discerning when any given utterance is directed at the system as opposed, e.g., to being directed at an individual present in the environment. One way to accomplish this is to use a hotword, which by agreement among the users in the environment, is reserved as a predetermined word that is spoken to invoke the attention of the system. In an example environment, the hotword used to invoke the system's attention are the words "OK computer." Consequently, each time the words "OK computer" are spoken, it is picked up by a microphone, conveyed to the system, which may perform speech recognition techniques or use audio features and neural networks to determine whether the hotword was spoken and, if so, awaits an ensuing command or query. Accordingly, utterances directed at the system take the general form [HOTWORD] [QUERY], where "HOTWORD" in this example is "OK computer" and "QUERY" can be any question, command, declaration, or other request that can be speech recognized, parsed and acted on by the system, either alone or in conjunction with the server via the network.
As systems that are configured to respond to hotwords become more prevalent, the likelihood of more than one being near a user when the user speaks the hotword increases. For example, a user may be in the living room watching television. A tablet may be on the table, and a phone may be on the couch next to the user. Upon speaking, "OK computer, play music," the user likely does not want the phone, tablet, and television to begin playing music. Instead, the user only wants one of the devices to respond. When the phone, tablet, and television detect a hotword, they each communicate with a server and notify the server that the device detected a hotword. With each notification sent to the server, each device transmits a group identifier and the type of device. The group identifier is data that identifies a device group from which only one device should respond to a hotword. For example, devices in the same groom will share a group identifier.
When the server receives data from each of the phone, tablet, and television, the server compares the context data of each device to determine which one should respond to the hotword. The context data may include the capabilities of the devices, the time since each device was previously used, the previous action preformed, the location of the device, and the loudness of the detected hotword. Based on this context data, the server selects one of the devices to respond to the hotword. For example, the server may select the tablet to respond to the hotword based on the devices being located at the user's home and the fact that the user most commonly uses the tablet while at home. The server then provides an instruction to the tablet to process the user's speech. The tablet processes the "play music" command and begins to play music. The server also provides instructions to the television and phone not to process the user's speech. The phone returns to its previous state, and the television continues to play the television show without interruption.
An innovative aspect of the subject matter described in this specification may be embodied in methods that include the actions of receiving, by a computing device, audio data that corresponds to an utterance; determining that the utterance likely includes a particular, predefined hotword; in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to a server, (i) data indicating that the computing device likely received the particular, predefined hotword, (ii) data identifying the computing device, and (iii) data identifying a group of nearby computing devices that includes the computing device; receiving, from the server, an instruction to commence speech recognition processing on the audio data; and in response to receiving the instruction to commence speech recognition processing on the audio data, processing at least a portion of the audio data using an automated speech recognizer on the computing device.
These and other embodiments can each optionally include one or more of the following features. The actions further include receiving, from an additional computing device, data indicating that the additional computing device is configured to respond to a particular, predefined hotword; transmitting, to the additional computing device, data indicating that the computing device is configured to respond to a particular, predefined hotword; determining a group identifier for a group of computing devices that includes the computing device and the additional computing device; and transmitting, to the additional computing device, the group identifier. The additional computing device is a computing device that is nearby the computing device. The data identifying the group of nearby computing devices that includes the computing device is the group identifier. The actions further include receiving, by the computing device, additional audio data that corresponds to an additional utterance; determining that the additional utterance likely includes the particular, predefined hotword; in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, (i) the data indicating that the computing device likely received the particular, predefined hotword, (ii) the data identifying the computing device, and (iii) the data identifying the group of nearby computing devices that includes the computing device; receiving, from the server, an instruction to not commence speech recognition processing on the audio data; and in response to receiving the instruction to not commence speech recognition processing on the audio data, ceasing processing of the audio data using the automated speech recognizer on the computing device.
The actions further include determining a loudness of the audio data associated with the particular, predefined hotword; and in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, the loudness of the audio data associated with the particular, predefined hotword. The action of determining a loudness of the audio data associated with the particular, predefined hotword includes determining a power of the audio data associated with the particular, predefined hotword; and determining a power of audio data that is not associated with the particular, predefined hotword and that the computing device received before the audio data associated with the particular, predefined hotword. The loudness of the audio data associated with the particular, predefined hotword is based on the power of the audio data associated with the particular, predefined hotword and the power of the audio data that is not associated with the particular, predefined hotword and that the computing device received before the audio data associated with the particular, predefined hotword. The actions further include determining a confidence score that reflects a likelihood that the audio data associated with the particular, predefined hotword corresponds to the particular, predefined hotword; and in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, confidence score.
The action of determining a confidence score that reflects a likelihood that the audio data associated with the particular, predefined hotword corresponds to the particular, predefined hotword includes determining audio features from the audio data associated with the particular, predefined hotword; and based on the audio features, determining, using a neural network, the confidence score. The actions further include, in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, data indicating a location of the computing device. The actions further include, in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, data indicating an elapsed time since a previous use of the computing device. The actions further include, in response to determining that the utterance likely includes the particular, predefined hotword, transmitting, to the server, data indicating a previous action performed by the computing device.
Another innovative aspect of the subject matter described in this specification may be embodied in methods that include the actions of receiving, by a server and from a computing device, (i) data indicating that the computing device likely received a particular, predefined hotword, (ii) data identifying the computing device, and (iii) data identifying a group of computing devices that are near the computing device and that includes the computing device; accessing context data that indicates a context of the computing device; based on the context data of the computing device, determining that the computing device commence speech recognition processing on audio data associated with the particular, predefined hotword; and transmitting, to the computing device, an instruction to commence speech recognition processing on the audio data associated with the particular, predefined hotword.
These and other embodiments can each optionally include one or more of the following features. The actions further include receiving, by a server and from an additional computing device, (i) data indicating that the additional computing device likely received the particular, predefined hotword, (ii) data identifying the additional computing device, and (iii) data identifying the group of computing devices that are near the additional computing device and that includes the computing device and the additional computing device; accessing context data that indicates a context of the additional computing device; based on the context data of the computing device and the context data of the additional computing device, determining that the additional computing device not commence speech recognition processing on the audio data associated with the particular, predefined hotword; and transmitting, to the additional computing device, an instruction to not commence speech recognition processing on the audio data associated with the particular, predefined hotword.
The actions further include receiving, by a server and from a first computing device, data indicating that the first computing device is configured to respond to a particular, predefined hotword; receiving, by a server and from a second computing device, data indicating that the second computing device is configured to respond to the particular, predefined hotword; determining, by the server, a group identifier for a group of computing devices that includes the first computing device and the second computing device; and transmitting, to the first computing device and the second computing device, the group identifier. The data identifying the group of computing devices that are near the computing device and that includes the computing device is the group identifier. The audio data associated with the particular, predefined hotword is audio data that corresponds to an utterance. The context data that indicates a context of the computing device includes data indicating one of more capabilities of the computing device.
The action of determining that the computing device commence speech recognition processing on audio data associated with the particular, predefined hotword is based on the one of more capabilities of the computing device. The context data that indicates a context of the computing device includes data indicating an elapsed time since a previous use of the computing device. The action of determining that the computing device commence speech recognition processing on audio data associated with the particular, predefined hotword is based on the elapsed time since the previous use of the computing device. The context data that indicates a context of the computing device includes data indicating a previous action performed by the computing device. The action of determining that the computing device commence speech recognition processing on audio data associated with the particular, predefined hotword is based on the previous action performed by the computing device. The actions further include receiving, from the computing device data indicating a loudness of the audio data associated with the particular, predefined hotword. The action of determining that the computing device commence speech recognition processing on the audio data associated with the particular, predefined hotword is further based on the loudness of the audio data associated with the particular, predefined hotword.
The actions further include receiving, by the server and from the computing device, data indicating a location of the computing device. The action of determining that the computing device commence speech recognition processing on audio data associated with the particular, predefined hotword is further based on the location of the computing device. The actions further include receiving, from the computing device, data indicating an action associated with the audio data and data indicating actions performed by the computing device within a particular period of time after receiving the instruction to commence speech recognition processing on the audio data associated with the particular, predefined hotword; and updating the context data of the computing device based on the data indicating the action associated with the audio data and the data indicating the actions performed by the computing device within the particular period of time after receiving the instruction to commence speech recognition processing on the audio data associated with the particular, predefined hotword.
Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. A user can speak a hotword in a room with multiple devices that are capable of responding to hotwords and only one device will respond. The user can speak a hotword and only the nearest device will respond to the hotword even if other devices hear the hotword. The system that selects the device to respond to the hotword can select an appropriate device based on the context of the device and other nearby devices.
FIG. 2 is a flowchart of an example process for hotword detection.
FIG. 3 is a flowchart of an example process for hotword detection.
FIG. 4 is an example of a computing device and a mobile computing device.
FIG. 1 illustrates an example system 100 for hotword detection. Briefly, and as described in more detail below, a user 102 speaks an utterance 104 that is detected by microphones of computing devices 106 and 108. The computing devices 106 and 108 process the audio data of the utterance 104 and determine whether the utterance 104 includes a hotword. Before performing speech recognition on the utterance 104, the computing devices 106 and 108 communicate with the server 110. The server 110 uses context data for both the computing device 106 and the computing device 108 to select one of the computing devices 106 or 108 to commence speech recognition on the audio data that includes the hotword. Based on the context data, the server 110 selects the computing device 106 and sends instructions to the computing device 106 to commence speech recognition on the audio data. The server 110 also sends instructions to the computing device 108 not to commence speech recognition on the audio data.
In the example shown in FIG. 1, the user 102 is in a room with both computing device 106, for example, a mobile phone, and computing device 108, for example, a television. When the user 102 speaks a voice command, both the phone 106 and the television 108 receive through their respective microphones the audio data that corresponds to the user's voice command. Both the phone 106 and the television 108 are configured to respond to voice commands, but it is unlikely that when the user 102 speaks "OK computer, play music" that the user 102 wishes for both the phone 106 and the television 108 to begin playing music. To avoid the situation of both devices responding to a voice command, the computing devices 106 and 108 may begin by identifying other devices in the nearby vicinity that are configured to respond to voice commands that include a hotword. The nearby vicinity may include devices that are on the same local network, devices that can communicate through a short range radio signals, or devices that are in a same room.
In some implementations, the phone 106 and the television 108 are on the same local network, for example, a home wireless network. The phone 106, or the television 108, communicates with the other devices on the network and identifies the devices that are configured to respond to voice commands that include a hotword. Once the phone 106 identifies the television 108 and confirms that the television 108 is also configured to respond to a hotword, the phone 106 and television 108, in stage A, negotiate a group identifier for use when notifying the server 110 that either the phone 106 or the television 108 has detected a hotword. For example, the group identifier may be the email address, user@example.com, of the user logged into one or both of the devices. The phone 106 stores user@example.com as the group identifier 112, and the television 108 stores user@example.com as the group identifier 114. In some implementations, this negotiation happens periodically to ensure that a device identifies nearby devices that are configured to respond to a hotword. For example, the negotiation can happen once an hour. Alternatively, a device can search for nearby devices when the device moves at least some threshold distance, for example, one hundred meters, as based on location data received by the device.
In some implementations, the phone 106 and the television 108 are on different networks. For example, the phone 106 is on a cellular network and the television 108 is on a local wireless network. In this instance, the phone 106 and the television 108 may be able to communicate through local short range radio and negotiate a group identifier. If the phone 106 and the television 108 are unable to communicate, then each device may select a group identifier that corresponds to the user who is logged into the device. For example, if the user 102 is logged into both the phone 106 and the television 108, then both devices may select the group identifier user@example.com.
At stage B, the user 102 speaks the utterance 104, "OK computer, play music." The utterance 104 includes both a hotword 116, "OK computer," and a voice command 118, "play music." Because the user 102 is close enough to the phone 106 and the television 108, both the phone 106 and the television 108 receive, through their respective microphones, the audio data that corresponds to the utterance 102.
At stage B1, the phone 106 receives and processes the audio data that corresponds to the utterance 102. The phone 106 includes a buffer 120. The buffer 120 stores the current audio received through the microphone of the phone 106. For example, the buffer 120 stores the previous five seconds of audio data. The hotworder 122 continuously analyzes the audio data stored in the buffer 120. The hotworder 122 computes a hotword confidence score that reflects the likelihood that current audio data in the buffer 120 includes a hotword. To compute the hotword confidence score, the hotworder 122 may extract audio features from the audio data such as filterbank energies or mel-frequency cepstral coefficients. The hotworder 122 may use classifying windows to process these audio features such as by using a support vector machine or a neural network. In some implementations, the hotworder 122 does not perform speech recognition to determine a hotword confidence score.
At approximately the same time that the phone 106 processes the utterance 104 and at stage B2, the television 108 processes audio data received through its microphone. The television 108 also includes a buffer 124 that stores the current audio data received through the microphone. For example, the buffer 124 may store the previous seven seconds of audio data. The hotworder 126 continuously analyzes the audio data stored in the buffer 124. The hotworder 126 continuously analyzes the audio data stored in the buffer 124. The hotworder 126 computes a hotword confidence score that reflects the likelihood that current audio data in the buffer 124 includes a hotword. To compute the hotword confidence score, the hotworder 126 may extract audio features from the audio data such as filterbank energies or mel-frequency cepstral coefficients. The hotworder 126 may use classifying windows to process these audio features such as by using a support vector machine or a neural network. In some implementations, the hotworder 126 does not perform speech recognition to determine a hotword confidence score.
As the phone 106 calculates a hotword confidence score for the audio data in the buffer 120, the phone 106 compares the hotword confidence score to a hotword confidence score threshold. If the hotword confidence score satisfies the hotword confidence score threshold, then the phone 106 proceeds to stage C1. For example, if the hotword confidence score is 0.8 and the threshold is 0.7, then the phone 106 proceeds to stage C1. The hotword confidence score threshold represents a hotword confidence score that likely corresponds to detecting a hotword. In instances where a user speaks, "OK computer" clearly and is near the receiving device, the hotword confidence score may be 0.9. Hotword confidence scores may decrease if the speaker is farther from the devices or speaks less clearly. In instances where a user speaks, "Call Alice," the hotword confidence score may be 0.1.
In stage C1, the phone 106 generates and transmits a data packet to the server 110. The data packet can include data that indicates that the phone 106 detected a hotword, a data identifier 128 for the phone 106, and the group identifier 112. The data that indicates that the phone 106 detected a hotword can include a flag that indicates the phone 106 detected a hotword or can include the hotword confidence score or both. The data identifier 128 for the phone 106 may be any unique identifier for the phone 106 such as the MAC address, the IMEI number, a user given name, phone number, or the serial number and model. In some implementations, the data identifier 128 at least includes the model of the phone 106.
During about the same time that the phone 106 computes the hotword confidence score and sends the server 110 the data packet that includes data that indicates that the phone 106 detected a hotword 116, a data identifier 128 for the phone 106, and the group identifier 112, the television 108 also calculates a hotword confidence score for the audio data in the buffer 124, the phone 106 compares the hotword confidence score to a hotword confidence score threshold. If the hotword confidence score satisfies the hotword confidence score threshold, then the television 108 proceeds to stage C2. For example, if the hotword confidence score is 0.75 and the hotword confidence score threshold is 0.7, then the television 108 proceeds to stage C2. The hotword confidence score for the television 108 may be lower than the hotword confidence score for the phone 106 because the user 102 is farther from the television 108 or because the television is currently displaying a television show with sounds that interferes with the microphone detecting the user's speech.
In stage C2 and similar to stage C1, the television 108 generates and transmits a data packet to the server 110 a data packet that can include data that indicates that the television 108 detected a hotword 116, a data identifier 130 for the television 108, and the group identifier 114. The data that indicates that the television 108 detected a hotword can include a flag that indicates the television 108 detected a hotword or can include the hotword confidence score or both. The data identifier 130 for the television 108 may be any unique identifier for the television 108 such as the MAC address, a user given name, or the serial number and model. In some implementations, the data identifier 130 at least includes the model of the television 108.
In some implementations and as illustrated in stage C1', the phone 106 sends additional data to the server. The additional data may include data that indicates a time elapsed since the phone 106 was previously used and how the phone 106 was used. For example, the user 102 may have previously used the phone 106 to send an SMS message two minutes prior to the phone 106 detecting the hotword 116. The phone 106 may store a history of the prior usage of the phone 106 in the usage data 132. Each entry in the usage data 132 may include any combination of an application used, an action performed by the phone 106, an action performed by the user 102, and a timestamp. In some implementations, the phone 106 may send additional data related to more than one previous action, for example, the previous three actions.
In some implementations and in stage C1', the phone 106 sends data that indicates a current state of the phone 106 when the phone 106 detected the hotword 116. For example, the current state may be that the phone 106 is asleep, the phone 106 is awake with the screen off or dimmed, or the phone 106 is awake and the user 102 is using the email application of the phone 106. In the example shown in FIG. 1, the phone 106 is currently in a sleep state when it detects the hotword 116.
In some implementations and as illustrated in stage C1', the phone 106 sends data to the server 110 that indicates the loudness of the detected hotword 116. The loudness scorer 134 accesses the buffer 120 and measures the loudness of the portion of audio data that corresponds to the hotword 116. In some implementations, the loudness scorer 134 calculates the power or the root mean square of the audio data that corresponds to the hotword 116. For example, the loudness scorer 134 may calculate the loudness to be fifty decibels. In some implementations, the loudness scorer 134 normalizes the power or the root mean square of the audio data that corresponds to the hotword 116 by subtracting the power or the root mean square of the audio data in the buffer 120 that does not correspond to the hotword 116. The audio data in the buffer 120 that does not correspond to the hotword 116 may correspond to the ambient noise around the phone 108. For example, the loudness scorer 134 calculates the power or the root mean square of about one second of audio data before the hotword 116 to be thirty decibels. Therefore, the loudness difference is ten decibels. The phone 106 may then send the loudness data to the server 110. In some implementations, the phone 106 may calculate the loudness of the entire utterance 104 or of only the voice command 116. The phone 106 may then transmit data indicating the loudness of the entire utterance 104 or of only the voice command 116 to the server 110.
In some implementations and as illustrated in stage C1', the phone 106 sends the hotword confidence score to the server 110. For example, the phone 106 calculates a hotword confidence score of 0.8 for the hotword 116 and transmits the score to the server 110. In some implementations, the phone 106 sends location data to the server 110. For example, the GPS unit on the phone 106 may detect a any combination of geographic location, geographic speed, and geographic movement, and the phone 106 may transmit that geographic location, geographic speed, and geographic movement to the server 110. As another example, the phone 106 may use triangulation to determine a geographic location. In some implementations, the speech recognizer 136 of the phone 106 may perform speech recognition on the audio data that is stored in the buffer 120 including the audio data that correspond to the hotword 116 or the voice command 118 or both. The speech recognizer 136 may calculate a speech recognition confidence score and transmit the speech recognition confidence score to the server 110. The speech recognition confidence score may reflect the likelihood that the transcription generated by the speech recognizer 136 is accurate. For example, the speech recognizer 136 may calculate a speech recognition confidence score of 0.7 and transmit the score of 0.7 to the server 110. In some implementations, the phone 106 transmits, to the server, 110 the transcription of the utterance 104 or only a portion of the utterance 104, for example, the hotword 116 or the voice command 118. For example, the speech recognizer 136 transcribes the voice command 118 and transmits "play music" to the server 110. In some implementations, the phone 106 also sends data that indicates the applications installed on the phone 106. For example, if the phone 106 has the application "Instant Movies" and "Love Music" installed, then the phone 106 may send data identifying "Instant Movies" and "Love Music." In some implementations, the phone 106 may have many applications installed. In this case, the phone 106 may only send data identifying the most used applications, for example, the ten most frequently used applications.
In some implementations and as illustrated in stage C2', the television 108 sends additional data to the server. The television 108 sends additional data that is similar to the additional data described above with respect to the phone 106 in stage C1'. The television 108 may store usage data 138 that is related to previous uses of the television 108 and transmit the usage data to the server 110. For example, the television 108 may transmit data indicating that the user 102 previously used the "Instant Movies" application to watch "Love Story" two hours ago. The television 108 may transmit data that indicates it current state. For example, the television 108 may transmit data indicating that the television 108 is currently showing the World News Channel and has been for the past fifteen minutes.
The television 108 may compute the loudness of the utterance 104, the hotword 116, or voice command 118. The loudness scorer 134 may compute the loudness based on the power or the root mean square of the corresponding audio data. The loudness scorer 134 may also normalize the power or the root mean square of the corresponding audio data by subtracting the power or the root mean square of the audio data stored in the buffer 124 that does not correspond to the utterance 104. For example, the loudness scorer 134 may calculate the loudness of the hotword 116 to be thirty decibels and the loudness of the audio data received in the two seconds before the hotword 116 to be thirty-two decibels. Therefore, the loudness value sent to the server 110 is two decibels.
In some implementations, the television 108 sends a hotword confidence score. For example, the hotworder 126 may calculate a hotword confidence score of 0.75, and the television 108 transmits the score of 0.75 to the server 110. The television 108 may send its geographic location to the server 110. The speech recognizer 142 may perform speech recognition on the audio data in the buffer 124. The speech recognizer 142 may generate a transcription as well as a speech recognition confidence score. The television 108 may send the transcription or the speech recognition confidence score or both to the server 110. The television 108 may send data that indicates the applications installed on the television 108 or the most frequently installed applications. For example, the television 108 may send data that identifies the installed application of "Jungle Movies" and "Social Pictures."
The server 110 initially receives either the phone 106 data packet from stage C1 or the television data packet from stage C2, depending on which device processes and generates the data packet sooner and possible depending any network latency. In this example, the server 110 receives the phone 106 data packet first. Upon receiving the phone 106 data packet, the server 110 accesses the received hotword detections 144 and compares the group identifier of the phone 106 data packet to the group identifier data and timestamp data in the received hotword detections 144. The server 110 stores previously received data packets that are received in response to hotword detections in the received hotword detections 144. By comparing incoming data packets to stored data packets, the server 110 can use the device selector 146 to select the device to respond to the hotword.
In this example, the server 110 compares the data received from the phone 106. The server 110 searches the received hotword detections 144 and does not find other hotword detection data packets that include the group identifier 112, user@example.com. With no other user@example.com data packets, the server 110 stores the data packet along with a timestamp indicating the time of receipt in the received hotword detections 144. The server 110 compares subsequently received hotword detection data packets to the data packets stored in the received hotword detections 144. If a threshold period of time passes, for example, two seconds, and the server 110 does not receive any additional data packets with the group identifier user@example.com, then the server 110 will transmit a command to the phone 106 to commence speech recognition on the audio data and to respond to the command. After transmitting the command to the phone 106 to commence speech recognition, then the server 110 will remove the data packet received from the phone 106 from the received hotword detections 144. In some implementations, the phone 106 may not identify any nearby devices, and, in this case, may not communicate with the server 110. Instead, the phone 106 may commence speech recognition on the audio data without communicating with the server 110.
Within the threshold period of time, the server 110 receives a hotword detection data packet from the television 108. The data packet from the television includes the same group identifier, user@example.com. The server 110 may then use the device selector 146 to select a device to commence speech recognition on the audio data, or the server 110 may wait for a threshold period of time in case other hotword detection data packet arrive at the server 110 with the same group identifier. In the example of FIG. 1, the server 110 receives hotword detection data packets from the television 108 and from the phone 106 and uses the device selector 146 to select a device.
In stage D, the server 110 accesses the context data 148, and the device selector 146 uses the phone context data 150 and the television context data 152 to select a device to commence speech recognition on the audio data. The server 110 may gather context data while the user 102 uses the phone 106 and the television 108, especially in instances where the phone 106 or the television 108 access the server 110 or a server that shares data with the server 110. For example, the user 102 may play music on the phone 106 by accessing a streaming music service on a server in the cloud 154. The streaming music server may communicate with the server 110 that is also in the cloud 154 and share the data that the phone 106 accessed the streaming music service. The server 110 may then store that information in the phone context data 150 possibly along with a timestamp. As another example, the user 102 may use the phone 106 to send an email using an email application that interacts with a server in the cloud 154. The email server may communicate with the server 110 and the server may store phone context data 150 that indicates the user 102 accessed the email application on the phone 106 along with a timestamp.
Similarly, the server 110 may receive television context data 152 from other servers in the cloud 154 that communicate with the television 108. For example, the user 102 may use the television to perform an internet search using a search engine that is in the cloud 154. The search engine may communicate with the server 110, and the server 110 may store television context data 152 that indicates the user 102 accessed the search engine on the television 108 along with a timestamp. As another example, the user 102 may use the television 108 to watch a video using a streaming video service from a server in the cloud 154. The streaming video server may communicate with the server 110, and the server 110 may store television context data 152 that indicates that the user 102 watched a video using the streaming video server, along with a timestamp. In some implementations, the phone context data 150 and the television context data 152 may include location data. For example, the phone context data may include data indicating the geographic location of the phone 106 when the user accessed the streaming music server.
In some implementations, the context data 148 may include or have access to data that describes capabilities of various devices. The capabilities may include whether the device has a camera, cellular radio, short range radio, GPS, or other components. The capabilities may also include specifications of the device such as screen size, screen resolution, processor type and speed, memory, etc. The context data 148 may not actually store this capability data. Instead the context server 110 may be able to query another server to retrieve device capability data. In some instances, the context server 110 may store some capability data within the context data 148 for some more commonly used devices such as popular phone models.
To select a device to commence speech recognition on the utterance 104, the device selector 146, at stage E, may consider a number of factors, and, in some instances, the device selector 146 may not have access to all the factors each time that the device selector 146 selects a device. In some implementations, the device selector 146 may select the device that detected the highest loudness for the hotword 116. For example, the phone 106 detected a hotword loudness that was normalized of ten decibels. The television 108 detected a hotword loudness that was normalized of two decibels. The device selector 146 may select the phone 106 because the hotword loudness was the greatest. In some implementations, the device selector 146 may have access to hotword loudness levels and the previous use of each device. In this instance, the device selector 146 may disregard the hotword loudness levels and instead select the device that was used most recently. For example, if the phone 106 was more active recently according to the phone context data 150 and the television context data 152, then the device selector 146 may select the phone 106. In some implementations, the device selector 146 may have access to context data that indicates that the television 108 is currently in use by displaying a video and the phone 106 has not been used in the last ten minutes. The device selector 152 may select the phone 106 to respond to commence speech recognition on the utterance 104 because the phone 106 is not being used.
In some implementations, the sever 110 may receive a hotword confidence score. In this instance, the device selector 146 may select the device with the highest hotword confidence score. In some implementations, the server 110 may receive location data and select a device based the user 102 being in a location where one of the devices is used more commonly than the other. For example, the devices may be a tablet and a phone. When the user 102 is away from the geographic locations that correspond to the user's home or work, then the context data 148 may reflect that the user 102 is more likely to use the phone instead of the tablet. Accordingly, when the user 102 speaks the hotword 116 and the phone and tablet send hotword detection packets that include location information to the server 110, then the server 110 may select the phone if the device selector 146 determines that the user 102 is not at home or work.
In some implementations, the server 110 may receive a transcription of the voice command 118. In this instance, the devices have already performed speech recognition on the voice command 118 and the device selector 146 selects a device to act on the voice command 118. With the transcription of the voice command, the device selector 146 can access the context data 148 to select an appropriate device to respond to the voice command. For example, if the voice command is "Call Alice," then the device selector 146 would select the phone 106 to process the voice command because the user 102 typically uses the phone 106 to call even in the case where the television 108 is configured to process voice calls or because the phone 106 is the only device configured to process voice calls.
In some implementations, the device selector 146 may use the current or previous use of the device to select a device to respond to the voice command. The current or previous use may be related to the voice query. For example, the voice command may be, "stop playing music." If the device selector 146 has access to context data 148 that describes one of the devices as playing music, then the device selector 146 selects the device that is playing music to respond to the voice command. As another example, the voice command may be, "show the music video again." If the device selector 146 has access to context data 148 that indicates that one of the devices played a music video for "Rock Group" in the last ten minutes, then the device selector 146 will select that device to respond to that voice command.
In some implementations, the server 110 may receive a speech recognition confidence score from each device. The device selector 146 may select the device with the highest speech recognition confidence score. For example, the phone 106 may calculate a speech recognition confidence score of 0.7 and the television 108 may calculate a speech recognition confidence score of 0.65. The device selector 146 selects the phone 106 to respond to the search query as a consequence of the higher speech recognition confidence score.
In some implementations, the server 110 may receive context data related to the applications installed on the phone 106 and television 108. The device selector 146 may use the context data related to the installed applications to select a device to respond to the voice query. For example, the voice command may be "play Super Ninja." The device selector 146 may have context information that the phone 106 has Super Ninja installed. Therefore, the device selector 146 selects the phone 106 to respond to the "play Super Ninja" command. In some implementations, the server 110 may receive data indicated installed applications if the user downloads the applications from an application server in the cloud 154 and the application server communicates with the server 110 regarding the installed applications.
In stage E, the server 110 selects a device to perform speech recognition on the voice command and then perform the voice command. In the case where the server 110 receives a transcription of the voice command, the server 110 selects a device to perform the voice command. In the example show in FIG. 1, the server 110 selects the phone 106 based on the television context data 152 that indicates that the television is currently being used, so the phone 106 may be better to respond to a new voice command. In instances where the server 110 receives a transcription of the voice command 118, the server may use context data that indicates the television 108 is showing a video while the phone 106 is idle to determine that the phone should respond to the "play music" the voice command. The device selector 146 may the use any of the criteria described above in any combination and weight each factor differently to select a device.
In stage F1 and F2, the server 110 transmits the instructions to the phone 106 and television 108. The instructions for the phone 106 instruct the phone 106 to process the query and commencing speech recognition on the utterance 104, if necessary. The instructions for the television 108 instruct the television 108 to cease processing the query.
In stages G1 and G2, the phone 106 and television 108 execute the instructions received by the server 110. The phone 106 processes the utterance 104, possibly by performing speech recognition, and then playing music 156. The television does not process the utterance 104 and continues to show the news video 158.
In some implementations, the phone 106 and television 108 may transmit to the server the actions performed by the user 102 after the phone 106 and television 108 process the utterance 104. For example, if the phone 106 begins to play music 156 and the user 102 quickly turns off the music 156, then changes the television 108 from showing the news video 158 to playing music. The phone 106 and television 108 then report these actions back to the server 110. The server 110 may update the selection algorithm of the device selector 146 using machine learning. With machine learning, the server 110 may change the weights given to each of the items of context data to select the most likely device to process the utterance 104. In some implementations, the machine learning may be user specific. For example, the device selector 146 may adjust the selection algorithm based on user 102's tendency to use the phone 106 during traditional business hours and the television outside of business hours or based on the user's tendency to use particular devices when in particular locations.
In some implementations, two devices, possibly of the same type, may end up in the same room and both are associated with a different group identifier. In this instance, when each device sends a hotword detection data packet to the server 110, the server 110 receives and compares the group identifier from the received data packets to the group identifiers stored in the received hotword detections 144 that are from other recently received data packets. With different group identifiers, the server 110 may not determine that only one device should respond to the hotword. To remedy this potential issue, the devices may be configured to send one or more of geographic location data, hotword loudness data, speaker identification data, device context data, speech recognition confidence scores, hotword confidence scores, and a transcription of the utterance 104. The server 110 may identify that two hotword detection data packets include geographic location data that are within a threshold distance of each other, for example, ten meters, and that the server 110 received within less than a threshold time of each other, for example, one second. In this case, the server 110 may determine that even though the devices were associated with different group identifiers, only one should respond to the hotword. To select the device to respond to the hotword, the device selector 146 may, for example, select the loudness data and confidence scores that are the highest and instruct the corresponding device to respond to the hotword while instructing the other devices not to respond to the hotword. In instances where the server 110 receives the transcription of the utterance 104, the server 110 may determine that the hotword detection data packets correspond to the same utterance because the text of the utterance 104 was the same and the data packets were received at about the same time. The device selector 146 may the select the device to respond to the utterance 104 based on the transcription, device capabilities and context, geographic location data, hotword loudness data, speaker identification data, speech recognition confidence scores, and hotword confidence scores using any of the techniques described above.
In some implementations, the voice command 118 associated with a single instance of a hotword may include more than one command. In certain instances, it may be appropriate for more than one device to process the commands, either serially or in parallel. For example, a user 102 may speak, "OK computer, play Love Story and call mom." In this instance, it may be better for the phone 106 to handle the call and the television 108 to handle playing the movie "Love Story". For the server 110 to provide instructions to both the phone 106 and the television 108, the server 110 may need the transcription of the utterance 104. With the transcription of the utterance 104, the device selector 146 may identify the two commands in the utterance 104 and the nearby devices that are best equipped to handle the commands based on the context data of each device and the capabilities data of each device. In this example, the device selector 146 may determine that the television 108 is better equipped to show "Love Story" because the television 108 has a larger screen and more powerful speakers than the phone 106. The device selector 146 may determine that the phone 106 is better equipped to handle the telephone call because the phone 106 has a cellular radio. The server 110 may then send instructions to the phone 106 to call mom and to the television 108 to play the movie "Love Story."
In some implementations, the phone 106 and television 108 may not transmit hotword detection packets to the server 110 and instead negotiate between each other to determine which device should respond to the hotword 116. The devices may use the context of the devices, for example, geographic location and motion data, to determine which one should respond to the hotword. For example, if the user 102 is moving at, on average or periodically, greater than fifteen miles per hour, then there is a high likelihood that the user 102 is traveling in a car. If the car is configured to respond to a hotword, then the car may communicate with any other nearby device and notify the other devices that the car will respond to the hotword. The car may still utilize communication features of other device, such as the car completing a call through a mobile phone, but the car will originate the telephone call. In instances where the user 102 is in the car, but not moving for a threshold period of time, then the car may not notify the other devices that it will respond to any hotwords or notify the other devices to communicate with the server 110 upon detecting a hotword. The car and the other devices may communicate with the server 110 and receive instructions from the server 110 when responding to hotwords.
In some implementations, the phone 106 or the television 108 or any other device may present an interface that permits the user 102 to enter a preference for which device responds to a hotword. For example, the user 102 may enter into the interface that when the user 102 is with the phone 106 and the television 108, then the user 102 prefers for the phone 106 to respond to a hotword. The user 102 may also be able to select an option that allows the server 110 to override the user's preference in instances where a device may be incapable of performing the voice command. In instances where the server 110 does not receive the transcription of the utterance, then the server 110 may select the device according to the user preference. As an example of a server override, the server 110 may receive hotword detection data packet from both the phone 106 and the television 108. The hotword detection data packets include the transcription of the voice command, which may be "OK computer, turn on "World News Now." The user 102 has previously selected that when the user 102 is near the television 108 and the phone 106, that the phone 106 should respond to a voice command. However, in this case, the phone 106 is not configured to show the news program "World News Now" because the phone 106 does not have access to the service that provides that news program. Accordingly, the server 110 instructs the television 108 to respond to the hotword and voice command and the phone 106 not to respond to the hotword and voice command.
FIG. 2 illustrates an example process 200 for hotword detection. In general, the process 300 processes audio data, and based on the audio data likely including a hotword, the process commences speech recognition on the audio data if instructed by a server. The process 200 will be described as being performed by a computer system comprising one or more computers, for example, the mobile device 106 or the television 108 as shown in FIG. 1.
The system receives audio data that corresponds to an utterance (210). For example, the user may receive audio data that corresponds to the user speaking, "OK computer, give me directions home." In some implementations, the system may receive from another system, data indicating that the other system is configured to respond to hotwords. The system and the other system may negotiate a group identifier that each can include in data packets to send to the server when either the system or the other system detects a hotword. For example, the group identifier may be an email address of the user logged into one or both of the devices. In some implementations, the system receives audio data when it is in a sleep state and with the screen off. In this case, the system may remain in a sleep state and still receive and process the audio data. In some implementations, the system may be in an active state, such as playing a video. In this case, the system receives and processes the audio data and continues to play the video.
The system determines that the utterance likely includes a particular, predefined hotword (220). For example, the system may determine that the utterance likely includes "OK computer." In some implementations, the system may be in a sleep state and upon receipt of identifying the hotword, the screen of the system activates, possibly on a dimmed low power state, and indicates the system is processing audio data.
The system, in response to determining that the utterance likely includes the particular, predefined hotword, transmits, to a server, (i) data indicating that the computing device likely received the particular, predefined hotword, (ii) data identifying the computing device, and (iii) data identifying a group of nearby computing devices that includes the computing device (230). In some implementations, the data identifying the computing device is the model and the serial number. In some implementations, the data identifying a group of nearby computing devices is the group identifier. In some implementations, the system determines a loudness of the audio data that corresponds to the hotword and transmits the loudness to the server in response to determining that the utterance likely includes the hotword. For example, the system may determine the loudness to be thirty decibels and transmit that value to the server. In some implementations, the system determines the power of the audio data that corresponds to the hotword and the power of the audio data received just before the hotword. The system then computes the difference between the two values and transmits the difference to the server in response to determining that the utterance likely includes the hotword.
In some implementations, the system determines a hotword confidence score that reflects the likelihood that the system detected a hotword. The system then transmits the hotword confidence score to the server in response to determining that the utterance likely includes the hotword. The system may use a neural network to determine the hotword confidence score. In some implementations, the system may transmit its geographic location to the server in response to determining that the utterance likely includes the hotword. In some implementations, the system may transmit data indicating an elapsed time since a previous use of the computing device and possibly including the previous action. For example, the user may have used the system two minutes ago to send a text message. The system may transmit this information to the server.
The system receives, from the server, an instruction to commence speech recognition processing on the audio data (240). For example, the system performs speech recognition on the audio data that does not correspond to the hotword and determines the transcription, "give me directions home." In some implementations, the system may receive an instruction not to commence speech recognition on the audio data. In some implementations, the system may have already have performed speech recognition on the audio data and transmitted the transcription to the server. In this instance the system may receive an instruction to process the voice command in the transcription or not to process the voice command. If the system receives an instruction not to process the voice command, then the system may return to its previous state such as a sleep state or displaying a video. The system, in response to receiving the instruction to commence speech recognition processing on the audio data, processes at least a portion of the audio data using an automated speech recognizer on the computing device (250). The system may then execute any voice command spoken by the user. For example, the system may provide the user directions to the user's home.
FIG. 3 illustrates an example process 300 for hotword detection. In general, the process 300 selects a computing device to commence speech recognition on audio data that likely includes a hotword. The process 300 will be described as being performed by a computer system comprising one or more computers, for example, the server 110 as shown in FIG. 1.
The system receives, from a computing device, (i) data indicating that the computing device likely received a particular, predefined hotword, (ii) data identifying the computing device, and (iii) data identifying a group of computing devices that are near the computing device and that includes the computing device (310). In some implementations, prior to receiving this data, the server receives data indicating that the computing device and maybe another computing device are configured to respond to a hotword. If the devices are nearby each other, then server selects a group identifier to identify the group when notifying that the computing device detected a hotword.
The system accesses context data that indicates a context of the computing device (320). In some implementations, the context data corresponds to the capabilities of the computing device. The system may access the capabilities data based on the model number provided by the computing device. For example, a phone may have a camera, short range radio, five inch screen, memory card slot, etc. In some implementations, the context data corresponds to an elapsed time since a previous use or a previous action or both of the computing device. For example, the previous use may be watching a news video and that may be occurring at the same time the computing device detected the hotword. The computing device may provide the time since the previous use and previous action. In some implementations, the system may access the data or receive the data from other servers. In some implementations, the context data includes a loudness of the audio data corresponding to the hotword. In some implementations, the system receives geographic location data from the computing device.
The system, based on the context data of the computing device, determines that the computing device commence speech recognition processing on audio data associated with the particular, predefined hotword (330). In some implementations, the audio data corresponds to an utterance, such as, a user speaking, "OK computer, give me directions home." In some implementations, the system selects the computing device based on any combination of the capabilities of the device, the location of the device, the time since the device was last used, the last action performed by the device, or the loudness of the audio data. In some implementations, the system may not have enough information to select a computing device. In this instance, the system may prompt the computing device to preform speech recognition on the utterance and provide a transcription of the utterance. The system may then use the context data and the transcription to select the computing device. In some implementations, the system may prompt the computing device for additional context information instead of or in addition to prompting for the transcription. For example, the system may prompt the computing device for confidence scores of the hotword and the speech recognition.
The system transmits, to the computing device, an instruction to commence speech recognition processing on the audio data associated with the particular, predefined hotword (340). In some implementations, the system may transmit an instruction not to commence speech recognition processing to other computing devices not selected by the system.
In some implementations, the system may receive data from the computing device identifying the actions taken by the user after the computing device processed the audio data. The computing device may execute the voice command of the audio data and have a user stop the execution of the voice command. The user may then manually execute the action of the voice command on the computing device not selected by the system. In this instance, the system will receive this data and determine that the system selected the wrong device to respond to the hotword. The system may then use machine learning to adjust the selection algorithm. In some implementations, the selection algorithm may be user specific.
Previous Patent US 9,972,319 | Next Patent US 9,972,321