Patent ID: 12236939

DETAILED DESCRIPTION

The following detailed description of example embodiments refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements.

The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

As used herein, the term “component” is intended to be broadly construed as hardware, firmware, or a combination of hardware and software.

It will be apparent that systems and/or methods, described herein, may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and/or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and/or methods were described herein without reference to specific software code—it being understood that software and hardware may be designed to implement the systems and/or methods based on the description herein.

Even though particular combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.

No element, act, or instruction used herein should be construed as critical or essential unless explicitly described as such. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and may be used interchangeably with “one or more.” Furthermore, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.), and may be used interchangeably with “one or more.” Where only one item is intended, the term “one” or similar language is used. Also, as used herein, the terms “has,” “have,” “having,” or the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless explicitly stated otherwise.

FIGS.1A and1Bare diagrams showing a general overview of a method for generating a trained trigger word detection model according to an embodiment.

FIG.1Ashows a first step100aof the training method which includes training an auxiliary model based on an auxiliary task according to an embodiment. As shown inFIG.1A, the training method100may obtain a feature vector by preprocessing and featurizing audio data. The auxiliary model may then be trained based on an auxiliary task using the feature vector.

The auxiliary task may be selected to train the model to learn context from an audio recording and/or concentrate on utterances in an audio recording. For example, as discussed below with regard to the non-limiting embodiment ofFIG.3, the auxiliary task may be predicting a distance between two utterances utilizing contrastive loss. According to another embodiment, as discussed below with regard to the non-limiting embodiment ofFIG.4, the auxiliary task may predict whether three audio segments are in sequence or are jumbled. Accordingly, by training the auxiliary model based on these auxiliary tasks which are concerned with the utterances and context, the auxiliary model will be trained to learn context from an audio file and concentrate on utterances in an audio file.

FIG.1Bshows a second step of the training method100bwhich includes retraining one or more of the final layers (such as the final dense layers) of the auxiliary model to obtain a trigger word detection model according to an embodiment. As shown inFIG.1B, a feature vector may be obtained by preprocessing and featurizing audio data. The one or more final layers of the auxiliary model, which have initial weights based on the auxiliary task training, may then be retrained based on a main task (trigger word detection) using the feature vector. Accordingly, the one or more final layers of the trigger word detection model may be weighted based on the trigger word detection task while the initial layers may be weighted based on the auxiliary task. According to another embodiment, all the model layers of the auxiliary model may be retrained based on the main task with a reduced learning rate.

By pretraining the trigger word detection model to learn context from an audio recording and/or concentrate on utterances in an audio recording based on the auxiliary task, the trigger word detection model may provide more robust trigger word detection.

Further, the trigger word detection model can be more efficiently trained to detect a specific trigger word utterance of a speaker (i.e., the trigger word spoken in the user's accent). That is, retraining the one or more final layers of the pretrained auxiliary model based on the specific trigger word utterance of the speaker can be accomplished with less training than training a full, untrained model based on the specific trigger word utterance of the speaker.

Additionally, by pretraining the trigger word detection model based on the auxiliary task, the trigger word detection model is less likely to overfit to certain voice textures, accents, and noises.

FIG.2is a diagram of a method200of operating the trained trigger word detection model according to an embodiment. As shown inFIG.2, a feature vector may be obtained by preprocessing and featurizing audio data, such as real time audio data obtained from a microphone of an electronic device. The feature vector may then be input into the trained trigger word detection model which detects whether the audio data includes the trigger word by analyzing the feature vector.

FIG.3is a diagram of a process300for training an auxiliary model using self-supervised contrastive learning according to an embodiment.

As shown inFIG.3, two vectors corresponding to respective audio files (Audio A and Audio B) are input into an auxiliary model (Neural Network N) which obtains two respective representations of Audio A and Audio B (Final Dense Layer A) and (Final Dense Layer B). A contrastive loss based on distance is then determined between the two respective representations of Audio A and Audio B. Using process300, the auxiliary model is trained to move audio files having the same utterance closer together and audio files with difference utterances further away in a virtual space. According to an embodiment, the Neural Network N may be a Siamese network, but the disclosure is not limited thereto.

According to an embodiment, Audio A may correspond to clean audio of a trigger word such as “hi Bixby.” Audio B may correspond to a positive or a negative audio. The positive audio may include the trigger word augmented with noise, the trigger word augmented with noise and reverberations, or the trigger word in a different accent. According to an embodiment, the positive audio may include an augmented version of the clean audio. The negative audio may be only the noise used for augmentation, only noise and reverberations used for augmentation, or a different word other than the trigger word which may or may not be augmented with noise and/or reverberations. The following Table 1 is an example of a non-limiting batch of Audio A and Audio B that may be used to train an auxiliary model.

TABLE 1Audio AAudio BPositive“Hi Bixby”“Hi Bixby” clean audio from speaker 1 +clean fromnoise 1speaker 1Hi Bixby” clean audio from speaker 1 +noise 1 + reverberation 1“Hi Bixby” clean audio from speaker 2 +noise 1“Hi Bixby” clean audio from speaker 2 +noise 1 + reverberation 1“Hi Bixby” in a different accent from the cleanaudioNegative“Hi Bixby”Noise 1clean fromNoise 1 + Reverberation 1speaker 1Different word clean audio from speaker 1 or 2Different word clean audio from speaker 1 or2 + noise 1 + reverberation 1

Other training batches may include pairings of trigger words from other speakers and other noise/reverberation types. According to an embodiment, the auxiliary tasks can be generic (not include the trigger word). According to an embodiment, the training batches may include multiple trigger words and/or trigger phrases (e.g. “hi bixby”).

Using process300, the auxiliary model may learn useful representations of the trigger word irrespective of the accent/speaker and noise environment.

FIG.4is a diagram of a process400for training an auxiliary model using jumbled utterance detection according to an embodiment. According to an embodiment, the neural network being trained by process400may be a triplet Siamese network, but the disclosure is not limited thereto.

As shown inFIG.4, audio of a trigger word may be split into multiple segments. In the non-limiting embodiment shown inFIG.4, the trigger word is split into three segments (Split 1, Split 2, Split 3), however, in other embodiment the trigger word may be split into more or less than 3 segments. The split audio may be augmented with noise or reverberations. Augmentation may be done either after splitting or before splitting depending on model performance.

A portion of the segments are then jumbled before being input into an auxiliary model (i.e. triplet Siamese network) so the model receives a mix of jumbled and in sequence audios. The auxiliary model may determine if the segments are in sequence or are jumbled using a triplet loss function. According to an embodiment, the auxiliary model may also or alternatively be trained with audio of utterance other than the trigger word based on the same conditions.

Training process400may teach the auxiliary model to understand the arrangement of phonemes in words irrespective of noise conditions. By making the auxiliary model predict jumbled sequences, the model learns to concentrate more on features of the trigger words. Once the auxiliary model is trained on this auxiliary task, the feature layers of the model may be able to model part of the audio that has the trigger word. By using the weights based on this auxiliary task to initialize the trigger word detection model, the trigger model may be more robust to noise and other perturbations.

FIG.5is a diagram of a cloud based process500for training the trigger word detection model, according to an embodiment.

As shown inFIG.5, the auxiliary model may be trained on a server or in the cloud with generic data based on the auxiliary task. The generic data may be data from a generic, open-source data set that includes utterances other than the trigger word. One or more of the final layers or all of the model layers of the auxiliary model, that is weighted based on the auxiliary task, may then be retrained for trigger word classification and/or detection on the server or in the cloud. In this embodiment, the fully trained trigger word detection model may be sent to an edge device such as a smartphone or smart earbuds that may use the model.

For example, as shown inFIG.5, a user may say play, pause, or next when listening to music playlist. The trigger word detection model may detect that the trigger word was uttered by the user and classify the trigger word to generate a capsule for an action associated with the detected trigger word (i.e. the capsule may direct the edge device to pause music when the trigger word “pause” is detected).

FIG.6is a diagram of a hybrid cloud and an on device-based process600for training the trigger word detection model, according to an embodiment.

As shown inFIG.6, the auxiliary model may be trained on a server or in the cloud with generic data based on the auxiliary task. One or more of the final layers or, all of the model layers of the auxiliary model that is weighted based on the auxiliary task may then be retrained for trigger word classification and/or detection on an edge device such as a smartphone or earbuds. In this embodiment, the more resource intensive training is performed on the server which is has greater resources than an edge device. Then, the less resource intensive retraining of the one or more final layers may be performed on the resource restricted device. This embodiment may also allow a user to perform the final training based on their own voice without having to upload private data to a server.

The method of generating a trained trigger word detection model or the method of detecting a trigger word may be performed by electronic device701in a network environment700as shown inFIG.7, according to an embodiment.FIG.7is for illustration only, and other embodiments of the electronic device and network could be used without departing from the scope of this disclosure.

Referring toFIG.7, according to an embodiment, an electronic device701may be included in a network environment700. The electronic device701may include at least one of a bus710, a processor720, a memory730, an input/output interface750, a display760, a communication interface770, or an event processing module780. In some embodiments, the electronic device701may exclude at least one of the components or may add another component.

The bus710may include a circuit for connecting the components720through780with one another and transferring communications (e.g., control messages and/or data) between the components.

The processing module720may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor720may perform control on at, least one of the other components of the electronic device701, and/or perform an operation or data processing relating to communication.

The memory730may include a volatile and/or non-volatile memory. For example, the memory730may store commands or data related to at least one other component of the electronic device701. According to an embodiment of the present disclosure, the memory730may store software and/or a program740. The program740may include, e.g., a kernel741, middleware743, an application programming interface (API)745, and/or an application program (or “application”)747. At least a portion of the kernel741, middleware743, or API745may be denoted an operating system (OS).

For example, the kernel741may control or manage system resources (e.g., the bus710, processor720, or a memory730) used to perform operations or functions implemented in other programs (e.g., the middleware743, API745, or application program747). The kernel741may provide an interface that allows the middleware743, the API745, or the application747to access the individual components of the electronic device701to control or manage the system resources.

The middleware743may function as a relay to allow the API745car the application747to communicate data with the kernel741, for example. A plurality of applications747may be provided. The middleware743may control work requests received from the applications747, e.g., by allocation the priority of using the system resources of the electronic device701(e.g., the bus710, the processor720, or the memory730) to at least one of the plurality of applications734.

The API745may be an interface allowing the application747to control functions provided from the kernel741or the middleware743. For example, the API733may include at least one interface or function (e.g., a command) for filing control, window control, image processing or text control.

The input/output interface750may serve as an interface that may, e.g., transfer commands or data input from a user or other external devices to other component(s) of the electronic device701. Further, the input/output interface750may output commands or data received from other component(s) of the electronic device701to the user or the other external device.

The display760may include, e.g., a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a microelectromechanical systems (MEMS) display, or an electronic paper display. The display760may display, e.g., various contents (e.g., text, images, videos, icons, or symbols) to the user. The display760may include a touchscreen and may receive, e.g., a touch, gesture, proximity or hovering input using an electronic pen or a body portion of the user.

For example, the communication interface770may set up communication between the electronic device701and an external electronic device (e.g., a first electronic device702, a second electronic device704, or a server706). For example, the communication interface770may be connected with the network762or764through wireless or wired communication to communicate with the external electronic device.

The first external electronic device702or the second external electronic device704may be a wearable device or an electronic device similar to electronic device701which may be a mountable wearable device (e.g., a head mounted display (HMD)). When the electronic device701is mounted in a HMD (e.g., the first electronic device702), the electronic device701may detect the mounting in the HMD and operate in a virtual reality mode. When the electronic device701is mounted in the electronic device702(e.g., the HMD), the electronic device701may communicate with the electronic device702through the communication interface770. The electronic device101may be directly connected with the electronic device702to communicate with the electronic device702without involving with a separate network.

The wireless communication may use at least one of, e.g., long term evolution (LTE), long term evolution-advanced (LTE-A), code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a cellular communication protocol. The wired connection may include at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS).

The network762may include at least one of communication networks, e.g., a computer network (e.g., local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.

The first and second external electronic devices702and704each may be a device of the same or a different type from the electronic device701. According to an embodiment, the server706may include a group of one or more servers. According to an embodiment, all or some of operations executed on the electronic device701may be executed on another or multiple other electronic devices (e.g., the electronic devices702and704or server706). According to an embodiment of the present disclosure, when the electronic device701should perform some function or service automatically or at a request, the electronic device701, instead of executing the function or service on its own or additionally, may request another device (e.g., electronic devices702and704or server706) to perform at least some functions associated therewith. The other electronic device (e.g., electronic devices702and704or server706) may execute the requested functions or additional functions and transfer a result of the execution to the electronic device701. The electronic device701may provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique may be used, for example.

AlthoughFIG.7shows that the electronic device701includes the communication interface770to communicate with the external electronic device704or706via the network762, the electronic device701may be independently operated without a separate communication function, according to an embodiment of the present invention.

The server706may support to drive the electronic device701by performing at least one of operations (or functions) implemented on the electronic device701. For example, the server706may include an event processing server module that may support the event processing module780implemented in the electronic device701.

For example, the event processing server module may include at least one of the components of the event processing module780and perform (or instead perform) at least one of the operations (or functions) conducted by the event processing module780.

The event processing module780may process at least part of information obtained from other elements (e.g., the processor720, the memory730, the input/output interface750, or the communication interface770) and may provide the same to the user in various manners.

Although inFIG.7the event processing module780is shown to be a module separate from the processor720, at least a portion of the event processing module780may be included or implemented in the processor720or at least one other module, or the overall function of the event processing module780may be included or implemented in the processor720shown or another processor. The event processing module780may perform operations according to embodiments of the present invention in interoperation with at least one program740stored in the memory730.

Exemplary embodiments described herein are not meant to be limiting and merely illustrative of various aspects of the invention. While exemplary embodiments may be indicated as applicable to a particular device category (e.g., mobile phone, etc.) the processes and examples provided are not intended to be solely limited to the device category and can be broadly applicable to various device categories (e.g., appliances, computers, automobiles, etc.). According to a non-limiting embodiment, an electronic device may include a smartphone, an augmented reality device, a smart speaker, a personal computer, a tablet computer; or any device connected to the internet of things.

AlthoughFIG.7shows one example of a network configuration700including an electronic device701, two external electronic devices702and704, and a server706, various changes may be made toFIG.7. For example, the network configuration700could include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, andFIG.7does not limit the scope of this disclosure to any particular configuration. Also, whileFIG.7shows one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

The method of generating a trained trigger word detection model or the method of detecting a trigger word may be written as computer-executable programs or instructions that may be stored in a medium.

The medium may continuously store the computer-executable programs or instructions, or temporarily store the computer-executable programs or instructions for execution or downloading. Also, the medium may be any one of various recording media or storage media in which a single piece or plurality of pieces of hardware are combined, and the medium is not limited to a medium directly connected to electronic device701, but may be distributed on a network. Examples of the medium include magnetic media, such as a hard disk, a floppy disk, and a magnetic tape, optical recording media, such as CD-ROM and DVD, magneto-optical media such as a floptical disk, and ROM, RAM, and a flash memory, which are configured to store program instructions. Other examples of the medium include recording media and storage media managed by application stores distributing applications or by websites, servers, and the like supplying or distributing other various types of software.

The method of generating a trained trigger word detection model or the method of detecting a trigger word may be provided in a form of downloadable software. A computer program product may include a product (for example, a downloadable application) in a form of a software program electronically distributed through a manufacturer or an electronic market. For electronic distribution, at least a part of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server or a storage medium of server106.

FIG.8is a flowchart of a method800of generating a trained trigger word detection model according to an embodiment.

At operation810, an auxiliary model may be trained, based on an auxiliary task, to concentrate on one or more utterances and/or learn content of the one or more utterances. According to an embodiment, the training may be self-supervised. The training data may be unlabeled generic single word and/or phrase data that may include utterances of the one or more trigger words or may include utterances that are different than the one or more trigger words.

According to an embodiment, the auxiliary model may be trained based on contrastive learning such that audio files with the same utterance are drawn closer together and audio filed with different utterances are drawn away from each other. For example, training the auxiliary model based on the auxiliary task may include repeatedly inputting a first type of audio file and a second type of audio file. The first type of audio file may include clean audio of a comparison utterance and the second type of audio file may include positive audio files and negative audio files. The positive audio files may include the comparison utterance and one or ore of noise or reverberation. The negative audio files may include one or more of noise, reverberations, or an utterance other than the comparison utterance. According to an embodiment, the auxiliary model may be a neural network such as a Siamese network.

According to an embodiment, the one or more noise or reverberations may include environment induced reverberations or noise of different noise types. The positive audio files may include spectral augmentation of the comparison utterance audio input.

According to another embodiment, the auxiliary model may be trained to predict whether an input audio file of an utterance is in sequence or jumbled. For example, training the auxiliary model based on the auxiliary task may include obtaining a plurality of audio files, where each filed include same or different utterances. The audio files may then be augmented with noise or reverberations and split into multiple segments to provide a plurality of segmented audio files. A portion of the segmented audio files may be jumbled to provide a plurality of partially jumbled, segmented audio files. The partially jumbled segmented audio files may be repeatedly input into the auxiliary model which is tasked to determine whether the files are jumbled.

At operation820, one or more final layers of the auxiliary model, which is weighted based on the auxiliary task, may be retrained based on a trigger word detection task that detects and one or more trigger words.

Accordingly, to an embodiment, one or ore of the final dense layers, all of the dense layers, or all of the model layers of the auxiliary model may be retrained based on the trigger word detection task. When the layers of the auxiliary model are retrained based on the trigger word detection task, they may be trained with less training data per class and/or using training data with less or minimal augmentation. That is, the layers may be retrained with a reduced learning rate.

According to an embodiment, the one or more final layers may be retrained on an edge device of a user using training data that is specific to a voice of the user. According to an embodiment, the one or more final layers may be retrained with a low learning rate. According to an embodiment, the trigger word detection task may detect on and a classify multiple trigger words such as pause, play, and next in an audio environment, however, the disclosure is not limited to the example trigger words.

The foregoing disclosure provides illustration and description but is not intended to be exhaustive or to limit the implementations to the precise form disclosed. Modifications and variations are possible in light of the above disclosure or may be acquired from practice of the implementations.

As used herein, the term “component” is intended to be broadly construed as hardware, firmware, or a combination of hardware and software.

It will be apparent that systems and/or methods, described herein, may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and/or methods is not limiting of the implementations. Thus, the operation and behavior of the systems and/or methods were described herein without reference to specific software code—it being understood that software and hardware may be designed to implement the systems and/or methods based on the description herein.

Even though particular combinations of features are recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification. Although each dependent claim listed below may directly depend on only one claim, the disclosure of possible implementations includes each dependent claim in combination with every other claim in the claim set.