Avoiding false trigger of wake word from remote device during call

A method for wake word detection during a communication session includes receiving a local audio signal from a microphone of a local communication device, the microphone capturing sound from a local environment of a local communication device, and receiving a far-end audio signal from a far-end communication device on a far-end of the communication session, the far-end communication device connected to the local communication device during the communication session. A first occurrence of a wake sound in the local audio signal is detected, and responsive to detecting the first occurrence of the wake sound in the local audio signal and determining that the far-end audio signal does not contain a second occurrence of the wake sound within a threshold period of time from the first occurrence, activation signal is transmitted to a controller. The activation signal causes a processor associated with the communication device to perform an operation.

BACKGROUND

Field of Art

The present disclosure relates to a communication device capable of performing wake word detection for the activation of a task during live two-way audio or audio-video communication.

Description of the Related Art

In smart home devices and audio assistant devices, a wake word is often used by a local user to prompt the activation of a task. In many cases, the task may be to activate audio recording of the local user. During live two-way audio or audio-video communication performed on such a smart home device or audio assistant device, audio communication with a user on a far-end may cause undesired activation of the task. For example, the user on the far-end may say the wake word during the live two-way communication. This may cause the smart home device or audio assistant device to mistakenly detect the local user as having said the wake word, if a microphone or audio input on the device captures the speech of the user on the far-end that includes the wake word.

Some smart home devices and audio assistant devices solve this problem by deactivating the function of wake word detection during live two-way audio or audio-video communication. A smart home device or audio assistant device may also use acoustic echo cancellation to try to remove an audio signal provided by a user on the far-end from an audio signal captured locally by the microphone or input, during live two-way communication. However, in many cases this is not sufficient for preventing erroneous wake word detection from occurring. For example, acoustic echo cancellation techniques may not be successful in completely removing the audio signal provided by the user on the far-end from the audio signal captured locally by the microphone or input. In this case, the wake word detector may erroneously detect the wake word and active the task due to the user on the far-end.

SUMMARY

Embodiments of the present disclosure relate to a communication device capable of performing wake sound detection for the activation of a task during live two-way audio or audio-video communication with another communication device on a far-end of the communication session over a communication network. The communication device allows a wake sound captured by a microphone in the local environment of the communication device to be detected and then activates the corresponding task, while preventing detection of a wake sound that is received from a far-end device over the communication network. Beneficially, the communication device accomplishes this without turning off the local wake sound detector (also referred to herein as a “wake sound detection module”), enabling the user to activate the corresponding tasks even during live two-way audio or audio-video communication sessions. In some embodiments, the wake sound is a wake word or phrase verbally expressed by a user.

The communication device prevents the communication device from erroneously detecting a wake word received from the far-end device by using an audio circuit including a dual-input wake sound detector. In one embodiment, the dual-input wake sound detector accepts an audio signal at each of a first input and a second input. A first input audio signal provided at the first input is a local audio signal provided from a microphone or audio input device local to the communication device. A second input audio signal provided at the second input is a far-end audio signal provided by the communication device on a far-end of the audio or audio-video communication session. The dual-input wake sound detector sends an activation signal to a controller to activate the task only when an occurrence of the wake sound is detected at the first input and determines that the second input audio signal does not contain the wake sound within a threshold period of time from occurrence of the wake word detected at the first input. If the wake word is also detected at the second input within the time threshold, the dual-input wake word detector determines not to send the activation signal to activate the task.

Embodiments of the present disclosure thus relate to a communication device that prevents the erroneous detection of a wake word for activating a task during live two-way audio or audio-video communication. By using a dual-input wake sound detector that only activates the task when the wake sound is detected at a first input and not a second input, the communication device enables a smart home device or an audio assistant device to perform accurate wake sound detection during live two-way audio or audio-video communication. This allows the smart home device or the audio assistant device to perform features that require wake sound detection simultaneously with the live two-way audio or audio-video communication.

DETAILED DESCRIPTION

I. System Architecture

FIG. 1is a block diagram of a system environment100for an online system140and one or more client devices110, according to some embodiments. The system environment100shown byFIG. 1comprises one or more client devices110, a network120, one or more third-party systems130, and the online system140. Additionally, in the system environment100shown byFIG. 1, a controller is coupled to a client device110. In alternative configurations, different and/or additional components may be included in the system environment100. For example, the online system140is a social networking system, a content sharing network, or another system providing content to users.

The client devices110(also referred to herein as a “communication device”) are one or more computing devices capable of receiving user input as well as transmitting and/or receiving data via the network120. In one embodiment, a client device110is a conventional computer system, such as a desktop or a laptop computer. Alternatively, a client device110may be a device having computer functionality, such as a personal digital assistant (PDA), a mobile telephone, a smartphone, or another suitable device. A client device110is configured to communicate via the network120. In one embodiment, a client device110executes an application allowing a user of the client device110to interact with the online system140. For example, a client device110executes a browser application to enable interaction between the client device110and the online system140via the network120. In another embodiment, a client device110interacts with the online system140through an application programming interface (API) running on a native operating system of the client device110, such as IOS® or ANDROID™. As further described below in conjunction withFIG. 3, a client device110includes an audio interface112configured to receive audio signals, receive audio communications, transmit audio communications, present audio content, and provide instructions to the controller. A client device110may engage in live two-way audio or audio-video communication with another communication device on a far-end of the communication session over the network120, according to some embodiments. For example, the communication device on the far-end may be an embodiment of the client device110.

As discussed below, with respect toFIG. 3, the audio interface112may be connected to a local audio input of the client device110(also referred to herein, as a “microphone”) which captures audio from a local environment of the client device110. The audio interface112may also be connected to a local audio output of the client device110(also referred to herein, as a “speaker”) which outputs audio content, including audio communications, into the local environment. The microphone and/or the speaker may not be physically coupled to the client device110. For example, the microphone and/or the speaker may be a part of a headset associated with the client device110. The audio interface112may be connected to a communication interface of the client device110. The communication interface may connect the client device110to one or more other client devices110, for audiovisual communication, according to some embodiments. In some embodiments, the communication interface connects the client device110to one or more other client devices110for audio communication without video. The audio interface112is connected to a controller114, according to some embodiments.

One or more third party systems130may be coupled to the network120for communicating with the online system140. In one embodiment, a third party system130is an application provider communicating information describing applications for execution by a client device110or communicating data to client devices110for use by an application executing on the client device. In other embodiments, a third party system130provides content or other information for presentation via a client device110. A third party system130may also communicate information to the online system140, such as advertisements, content, or information about an application provided by the third party system130.

The controller114is connected to the audio interface112and comprises a storage device coupled to a processor. In various embodiments, the controller114is connected to other components of the client device110. In some embodiments, the controller114provides instructions to a processor of the client device110to perform an operation, in response to receiving an activation signal from the audio interface112. For example, a wake word detector of the audio interface112may transmit the activation signal to the controller114, in response to a local user saying a pre-determined phrase, also referred to herein as a “wake word.”

FIG. 2is an interaction diagram for the audio interface112of a client device110, illustrating interactions between components of the audio interface112and other components of the client device110, according to some embodiments. According to some embodiments, the client device110has multiple functions including, but not limited to acting as a smart home device or an audio assistant device. The client device110also functions to provide live two-way audio or audio-visual communication between users that are remote from each other.

The client device110includes a local audio input210(also referred to herein as a “microphone”210) configured to capture and record audio on a near-end, where a local user201communicates to the device verbally, and a local audio output220(also referred to herein as a “speaker”220) configured to output sound in to the local environment of the client device110. The microphone210and the speaker220are connected to the audio interface112. In some embodiments, the speaker220is in a vicinity of the microphone210.

The audio interface112includes an acoustic echo cancellation (AEC) module230and a dual-input wake sound detection module240(also referred to as a “wake word detector”240). The output of the microphone210is connected to a first input of the AEC module230, and the output of the AEC module230is connected to a first input of the wake word detector240and, although not shown inFIG. 2, a communication interface250of the client device110. In some embodiments, the output of the microphone210is connected directly to an input of the wake word detector240. For example, the output of the microphone210may be connected to the first input of the wake word detector240, instead of the output of the AEC module230. A communication device on a far-end260provides an audio signal to the communication interface250, which provides the audio signal to a speaker220of the client device110, a second input of the AEC module230, and a second input of the wake word detector240. The communication device on the far-end260may send the audio signal to the communication interface250via a network connection, for example by voice-over IP using an internet connection.

The wake word detector240is connected to a controller114, and the controller114is configured to perform an operation, in response to receiving an activation signal from the wake word detector240. When the desired conditions are met, for example the user saying a wake word in the vicinity of the microphone210, the wake word detector240will send an activation signal to the processor to begin a task. In some embodiments, the wake word detector240detects an occurrence of a wake sound in the audio signals received at the first input and the second input of the wake word detector240, and based on the detected occurrences of the wake sound, transmits or withholds the activation signal to the controller114. For example, the task may involve capturing audio through the microphone210, to receive a command verbally expressed by the local user201. A function of the wake word detector is to prevent activation of the task by audio received from the communication device on the far-end260and to ensure that only audio from the local user201or the local user's environment will activate the initiation of the operation (also referred to herein as a “task”). By using the dual input wake word detector of the present disclosure, the communication device can perform wake word detection and live two-way audio communication simultaneously with accurate wake word detection. Although in the examples provided below, the wake sound may be a word or a phrase uttered by a user, in other embodiments, the wake sound may be a different sound.

During live two-way audio or audio-video communication, the local user201may use the client device110for audio and/or audio-video communication to the communication device on the far-end260. The live two-way audio or audio-video communication may involve a user on the far-end using the communication device on the far-end260. The local user201may simultaneously speak in order to both command the audio interface112to activate a task and to verbally communicate to the user on the far-end.

The output from the AEC module230is connected to the communication interface. The AEC module230modifies the output signal from the microphone210, removing an audio signal provided by the communication device on the far-end260. The modified output signal from the microphone210is provided by the AEC module230to the input of the communication interface250. The communication interface250then sends the modified output signal to the communication device on the far-end260, via a network connection. The modified output signal aids in preventing audio provided by the speaker220from being communicated back to the communication device on the far-end260which can cause an echo effect or feedback on the far-end. Thus, the client device110and the communication device on the far-end260may each experience a reduced echo effect during live two-way audio communication.

In some embodiments, the output of the AEC module230is also connected in parallel with the output of the microphone210to the first input of the wake word detector240. In this case, the AEC module230also modifies the output of the microphone210before it reaches the first input of the wake word detector240. In other embodiments, the microphone210is connected to the first input of the wake word detector240, without the AEC module230intervening. The audio interface112may include only the wake word detector240and not include the AEC module230, according to some embodiments. In this case, the output of microphone210is connected to the first input of the wake word detector240and the communication interface250.

II. Operation of the Audio Device

FIG. 3illustrates the logic of the wake word detector240in determining whether to send an activation signal to the controller114to begin a task, according to some embodiments. In this example, a desired condition for sending the activation signal is the local user201saying the word “wake.” A function of the wake word detector240is to prevent the erroneous activation of the task, during live two-way communication. If a far-end user says the word “wake” which is captured by the communication device on the far-end260, the microphone210may capture audio of the far-end user saying “wake” from the speaker220. The wake word detector240prevents this from activating the task.

FIG. 3shows a table showing the logic for the dual-input wake word detection module. Following the example above, the activation condition for the wake word detector240is the local user201saying a word “wake.” In other embodiments, the activation condition for the wake word detector is a different sound being captured by the microphone210. The wake word detector240will check both a first input audio signal at the first input of the wake word detector240and a second input audio signal at the second input audio signal of the wake word detector240for an audio signal corresponding to an utterance of the word “wake.”

If the first input audio signal contains the word “wake” and the second signal contains the word “wake” within a threshold time of the occurrence of the word “wake” in the first input audio signal, the wake word detector240will not transmit the activation signal to the controller114. If the first input audio signal contains the word “wake” and the second signal does not contain the word “wake” within the threshold time from the occurrence of the word “wake” in the first input audio signal, the wake word detector240transmits the activation signal to the controller114. If the first input audio signal does not contain the word “wake,” the wake word detector240will not send the activation signal, regardless of the second input audio signal.

In one embodiment, the wake word detector240is always listening at the second input (i.e., from the far-end communication). In another embodiment, the wake word detector240is triggered only when the wake word is detected at the first input (i.e., from the microphone210). This may help to save power consumption by activating a reduced amount of the circuitry of the wake word detector240.

In some embodiments, the wake word detector240performs wake sound detection on a single input audio signal from one source at a time. In further embodiments, the wake word detector240may have a single audio input. For example, the wake word detector240may receive the local audio signal from the microphone210(or the modified local audio signal from the AEC module230) and perform wake sound detection on the local audio signal. Upon detecting an occurrence of the wake sound in the local audio signal from the microphone210, the wake word detector240stops the wake sound detection on the local audio signal. The wake word detector240then receives far-end audio signal data from the from the communication device on the far-end260that is within a threshold period of time from the occurrence of the wake sound in the local audio signal. If the far-end audio signal data does not include the wake sound, the wake word detector240transmits the activation signal to the controller114. The wake word detector240may then resume receiving the local audio signal from the microphone210(or the modified local audio signal from the AEC module230) and performing wake sound detection on the local audio signal. According to some embodiments, this may reduce the required computing resources needed to perform the wake sound detection during a live two-way audio communication session.

III. Additional Considerations

The figures and descriptions of the present disclosure have been simplified to illustrate elements that are relevant for a clear understanding of the present disclosure, while eliminating, for the purpose of clarity, many other elements found in a typical system. Those of ordinary skill in the art may recognize that other elements and/or steps are desirable and/or required in implementing the present disclosure.