Patent Description:
Advances in technology have resulted in smaller and more powerful computing devices. For example, there currently exist a variety of portable personal computing devices, including wireless telephones such as mobile and smart phones, tablets and laptop computers that are small, lightweight, and easily carried by users. These devices can communicate voice and data packets over wireless networks. Further, many such devices incorporate additional functionality such as a digital still camera, a digital video camera, a digital recorder, and an audio file player. Also, such devices can process executable instructions, including software applications, such as a web browser application, that can be used to access the Internet. As such, these devices can include significant computing capabilities, including, for example a Sound Event Classification (SEC) system that attempts to recognize sound events (e.g., slamming doors, car horns, etc.) in an audio signal.

An SEC system is generally trained using a supervised machine learning technique to recognize a specific set of sounds that are identified in labeled training data. As a result, each SEC system tends to be domain specific (e.g., capable of classifying a predetermined set of sounds). After the SEC system is trained, it is difficult to update the SEC system to recognize new sound classes that were not identified in the labeled training data. Additionally, some sound classes that the SEC system is trained to detect may represent sound events that have more variants than are represented in the labeled training data. To illustrate, the labeled training data may include audio data samples for many different doorbells but is unlikely to include all of the existing variants of doorbell sounds. Retraining the SEC system to recognize a new sound that was not represented in the training data used to train the SEC system may involve completely retraining the SEC system using a new set of labeled training data that includes examples for the new sound in addition to the original training data. As a result, training an SEC system to recognize a new sound (whether for a new sound class or a variant of an existing sound class) requires approximately the same computing resources (e.g., processor cycles, memory, etc.) as generating a brand-new SEC system. Further, over time, as even more sounds are added to be recognized, the number of audio data samples that must be maintained and used to train the SEC system can become unwieldy.

In <CIT>, a neural network-based classifier system can receive a query including a media signal and, in response, provide an indication that the query corresponds to a specified media type or media class. The neural network-based classifier system can select and apply various models to facilitate media classification. In an example embodiment, a query can be analyzed for various characteristics, such as a noise profile, before it is input to the network-based classifier. If the query has greater than a specified threshold noise characteristic, then a successful classification can be unlikely and a classification process based on the query can be terminated before computational resources are expended. Query signals that meet or exceed a threshold condition can be provided to the network-based classifier for media classification. In an example embodiment, a remote device or a central media classifier circuit can determine a noise profile for a query.

In the article titled "<NPL>) there is presented a learning strategy for a Sound Event Detection (SED) system to tackle the issues of i) knowledge migration from a pre-trained model to a new target model and ii) learning new sound events without forgetting the previously learned ones without re-training from scratch. In order to migrate the previously learned knowledge from the source model to the target one, a neural adapter is employed on the top of the source model. The source model and the target model are merged via this neural adapter layer. The neural adapter layer facilitates the target model to learn new sound events with minimal training data and maintaining the performance of the previously learned sound events similar to the source model.

In the article titled "<NPL>) there is presented an open-set evolving audio classification technique, which can recognize and learn unknown classes continuously in an unsupervised manner.

Preferred embodiments are recited in dependent claims.

Sound event classification models can be trained using machine-learning techniques. For example, a neural network can be trained as a sound event classifier using backpropagation or other machine-learning training techniques. A neural network trained in this manner is referred to herein as a "sound event classification model. " A sound event classification model trained in this manner can be small enough (in terms of storage space occupied) and simple enough (in terms of computing resources used during operation) for a portable computing device to store and use the sound event classification model. The process of training a sound event classification model uses significantly more processing resources than are used to perform sound event classification using a sound event classification model. Additionally, the training process uses a large set of labeled training data including many audio data samples for each sound class that the sound event classification model is being trained to detect. It may be prohibitive, in terms of memory utilization or other computing resources, to train a sound event classification model from scratch on a portable computing device or another resource limited computing device. As a result, a user who desires to use a sound event classification model on a portable computing device may be limited to downloading pre-trained sound event classification models onto the portable computing device from a less resource constrained computing device or from a library of pre-trained sound event classification models. Thus, the user has limited customization options.

The disclosed systems and methods use transfer learning techniques to update sound event classification models in a manner that uses significantly fewer computing resources than training sound event classification models from scratch. According to a particular aspect, the transfer learning techniques can be used to update a sound event classification model to account for drift within a sound class or to recognize a new sound class. In this context, "drift" refers to variation within a sound class. For example, a sound event classification model may be able to recognize some examples of the sound class but may not be able to recognize other examples of the sound class. To illustrate, a sound event classification model trained to recognize a car horn sound class may be able to recognize many different types of car horns but may not be able to recognize some examples of car horns. Drift can also occur due to variations in an acoustic environment. To illustrate, the sound event classification model may be trained to recognize the sound of a bass drum played in a concert hall but may not recognize the bass drum if it is played by a marching band in an outdoor parade. The transfer learning techniques disclosed herein facilitate updating a sound event classification model to account for such drift, which enables the sound event classification model to detect a broader range of sounds within a sound class. Because the drift may correspond to sounds that were encountered by a user device but that were unrecognized by the user device, updating the sound event classification model for the user device to accommodate these encountered variations of sound classes enables the user device to more accurately identify specific variations of sound classes that are commonly encountered by that particular user device.

According to a particular aspect, when the sound event classification model is determined to have not recognized a sound class of a sound (based on audio data samples of the sound), a determination is made whether the sound was not recognized due to drift or because the sound event classification model does not recognize sound classes of a type associated with the sound. For example, information distinct from the audio data samples, such as a timestamp, location data, image data, video data, user input data, settings data, other sensor data, etc., is used to determine scene data indicating a sound environment (or audio scene) associated with the audio data samples. The scene data is used to determine whether the sound event classification model corresponds to (according to the claimed invention: is trained to recognize sound events in) the audio scene. If the sound event classification model corresponds to the audio scene, the audio data samples are saved as model update data and indicated to be drift data. In some aspects, if the sound event classification model does not correspond to the audio scene, the audio data samples are discarded as unknown or saved as model update data and indicated to be associated with an unknown sound class (e.g., unknown data).

Periodically or occasionally (e.g., when initiated by a user or when an update condition is satisfied), the sound event classification model is updated using the model update data. For example, to account for drift data, the sound event classifier can be trained (e.g., further trained, by starting with the already trained sound event classifier) using backpropagation or other similar machine-learning techniques. In this example, the drift data is associated with a label of a sound class already recognized by the sound event classification model, and the drift data and corresponding label are used as labeled training data. Updating the sound event classification model using the drift data can be augmented by adding other examples of the sound class to the labeled training data, such as examples taken from training data originally used to train the sound event classification model. In some aspects, a device automatically (e.g., without user input) updates one or more sound event classification models when drift data is available. Thus, a sound event classification system can automatically adapt to account for drift within a sound class using significantly fewer computing resources that would be used to train the sound event classification model from scratch.

To account for unknown data, the sound event classification model can be trained using more complex transfer learning techniques. For example, when unknown data is available, a user may be queried to indicate whether the user desires to update the sound event classification model. The audio representing the unknown data can be played out to the user, and the user can indicate that the unknown data is to be discarded without updating the sound event classification model, can indicate that the unknown data corresponds to a known sound class (e.g., to reclassify the unknown data as drift data), or can assign a new sound class label to the unknown data. If the user reclassifies the unknown data as drift data, the machine-learning technique(s) used to update the sound event classification model to account for drift data are initiated, as described above.

If the user assigns a new sound class label to the unknown data, the label and the unknown data are used as labeled training data to generate an updated sound event classification model. According to a particular aspect, a transfer learning technique used to update the sound event classification model includes generating a copy of the sound event classifier model that includes an output node associated with the new sound class. The copy of the sound event classifier model is referred to as an incremental model. The transfer learning technique also includes connecting the sound event classification model and the incremental model to one or more adapter networks. The adapter network(s) facilitate generation of a merged output that is based on both the output of the sound event classification model and the output of the incremental model. Audio data samples including the unknown data and one or more audio data samples corresponding to known sound classes (e.g., sound classes that the sound event classifier was previously trained to recognize) are provided to the sound event classification model and the incremental model to generate the merged output. The merged output indicates a sound class assigned to the audio data samples based on analysis by the sound event classification model, the incremental model, and the one or more adapter networks. During training, the merged output is used to update link weights of the incremental model and the adapter network(s). When training is complete, the sound event classifier may be discarded if the incremental model is sufficiently accurate. If the incremental model alone is not sufficiently accurate, the sound event classification model, the incremental model, and the adapter network(s) are retained together and used as a single updated sound event classification model. Thus, the techniques disclosed herein enable customization and updating of sound event classification models in a manner that is less resource intensive (in terms of memory resources, processor time, and power) than training a neural network from scratch. Additional, in some aspects, the techniques disclosed enable automatic updating of sound event classification models to account for drift.

The disclosed systems and methods provide a context-aware system that can detect dataset drift, associate drift data with a corresponding class (e.g., by utilizing available multi-modal inputs), and refine/fine-tune an SEC model utilizing the drift data with little or no supervision and without training a new SEC model from scratch. In some aspects, before refining/fine-tuning the SEC model, the SEC model is trained to recognize multiple variants of a particular sound class, and refining/fine-tuning the SEC model modifies the SEC model to enable the SEC model to recognize an additional variant of the particular sound class.

In some aspects, the disclosed systems and methods may be used for applications that suffer from dataset drift during test. For example, the systems and methods can detect the dataset drift and refine the SEC model without retraining previously learned sound classes from scratch. In some aspects, the disclosed systems and methods may be used to add new sound classes to an existing SEC model (e.g., an SEC model that is already trained for certain sound classes) without re-training the SEC model from scratch, without the need to access all the training data used to train the SEC model initially, and without introducing any performance degradation with respect to the sound classes the SEC model was initially trained to recognize.

In some aspects, the disclosed systems and methods may be used in applications where a continuous learning capability at a low footprint constraint is desirable. In some implementations, the disclosed systems and methods may have access to a database of various detection models (e.g., SEC models) for a diverse range of applications (e.g., various sound environments). In such implementations, an SEC model may be selected, during operation, based on a sound environment, and the SEC model may be loaded and utilized as the source model.

Particular aspects of the present disclosure are described below with reference to the drawings. In the description, common features are designated by common reference numbers. As used herein, various terminology is used for the purpose of describing particular implementations only and is not intended to be limiting of implementations. For example, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, some features described herein are singular in some implementations and plural in other implementations. To illustrate, <FIG> depicts a device <NUM> including one or more sensors ("sensor(s) <NUM> in <FIG>), which indicates that in some implementations the device <NUM> includes a single sensor <NUM> and in other implementations the device <NUM> includes multiple sensors <NUM>. For ease of reference herein, such features are generally introduced as "one or more" features and are subsequently referred to in the singular or optional plural (generally indicated by terms ending in "(s)") unless aspects related to multiple of the features are being described.

The terms "comprise," "comprises," and "comprising" are used herein interchangeably with "include," "includes," or "including. " Additionally, the term "wherein" is used interchangeably with "where. " As used herein, "exemplary" indicates an example, an implementation, and/or an aspect, and should not be construed as limiting or as indicating a preference or a preferred implementation. As used herein, an ordinal term (e.g., "first," "second," "third," etc.) used to modify an element, such as a structure, a component, an operation, etc., does not by itself indicate any priority or order of the element with respect to another element, but rather merely distinguishes the element from another element having a same name (but for use of the ordinal term). As used herein, the term "set" refers to one or more of a particular element, and the term "plurality" refers to multiple (e.g., two or more) of a particular element.

As used herein, "coupled" may include "communicatively coupled," "electrically coupled," or "physically coupled," and may also (or alternatively) include any combinations thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., a wired network, a wireless network, or a combination thereof), etc. Two devices (or components) that are electrically coupled may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may send and receive electrical signals (digital signals or analog signals) directly or indirectly, such as via one or more wires, buses, networks, etc. As used herein, "directly coupled" refers to two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.

In the present disclosure, terms such as "determining," "calculating," "estimating," "shifting," "adjusting," etc. may be used to describe how one or more operations are performed. It should be noted that such terms are not to be construed as limiting and other techniques may be utilized to perform similar operations. Additionally, as referred to herein, "generating," "calculating," "estimating," "using," "selecting," "accessing," and "determining" may be used interchangeably. For example, "generating," "calculating," "estimating," or "determining" a parameter (or a signal) may refer to actively generating, estimating, calculating, or determining the parameter (or the signal) or may refer to using, selecting, or accessing the parameter (or signal) that is already generated, such as by another component or device.

<FIG> is a block diagram of an example of a device <NUM> that is configured to generate sound identification data responsive to audio data samples <NUM> and configured to update a sound event classification model. The device <NUM> of <FIG> includes one or more microphones <NUM> configured to generate audio signals <NUM> based on sound <NUM> detected within an acoustic environment. The microphone(s) <NUM> are coupled to a feature extractor <NUM> that generates audio data samples <NUM> based on the audio signals <NUM>. For example, the audio data samples <NUM> may include an array or matrix of data elements, with each data element corresponding to a feature detected in the audio signals <NUM>. As a specific example, the audio data samples <NUM> can correspond to Mel spectrum features extracted from one second of the audio signals <NUM>. In this example, the audio data samples <NUM> can include a 128x128 element matrix of feature values. In other examples, other audio data sample configurations or sizes can be used.

The audio data samples <NUM> are provided to a sound event classification (SEC) engine <NUM>. The SEC engine <NUM> is configured to perform inference operations based on one or more SEC models, such as an SEC model <NUM>. "Inference operations" refer to assigning the audio data samples <NUM> to a sound class, if the sound class of the audio data samples <NUM> is recognized by the SEC model <NUM>. For example, the SEC engine <NUM> may include or correspond to software that implements a machine-learning runtime environment, such as the Qualcomm Neural Processing SDK, which is available from Qualcomm Technologies, Inc. of San Diego, California, USA. In a particular aspect, the SEC model <NUM> is one of a plurality of SEC models (e.g., available SEC models <NUM>) that are available to the SEC engine <NUM>.

In a particular example, each of the available SEC models <NUM> includes or corresponds to a neural network that is trained as a sound event classifier. To illustrate, the SEC model <NUM> (as well as each of the other available SEC models <NUM>) may include an input layer, one or more hidden layers, and an output layer. In this example, the input layer is configured to correspond to the array or matrix of values of the audio data samples <NUM> generated by the feature extractor <NUM>. To illustrate, if the audio data samples <NUM> include <NUM> data elements, the input layer may include <NUM> nodes (e.g., one per data element). The output layer is configured to correspond to the sound classes that the SEC model <NUM> is trained to recognize. The specific arrangement of the output layer can vary depending on information to be provided as output. As one example, the SEC model <NUM> may be trained to output an array that includes one bit per sound class, where the output layer performs "one hot encoding" such that all but one of the bits of the output array have a value of zero, and the bit corresponding to a detected sound class has a value of one. Other output schemes can be used to indicate, for example, a value of a confidence metric for each sound class, where the value of the confidence metric indicates a probability estimate that the audio data samples <NUM> correspond to the respective sound class. To illustrate, if the SEC model <NUM> is trained to recognize four sound classes, the SEC model <NUM> may generate output data that includes four values (one per sound class), and each value may indicate a probability estimate that the audio data samples <NUM> correspond to the respective sound class.

Each of the hidden layers includes a plurality of nodes, and each node is interconnected (via a link) with other nodes in the same layer or in a different layer. Each input link of a node is associated with a link weight. During operation, a node receives input values from other nodes that it is linked to, weights the input values based on corresponding link weights to determine a combined value, and subjects the combined value to an activation function to generate an output value of the node. The output value is provided to one or more other nodes via output links of the node. The nodes may also include bias values that are used to generate the combined value. The nodes can be linked in various arrangements and can include various other features (e.g., memory of prior values) to facilitate processing of particular data. In the case of audio data samples, convolutional neural networks (CNNs) may be used. To illustrate, one or more of the SEC models <NUM> may include three linked CNNs, and each CNN may include a two-dimensional (2D) convolution layer, a maxpooling layer, and a batch normalization layer. In other implementations, the hidden layers include a different number of CNNs or other layers. Training the neural network includes modifying the link weights to reduce an output error of the neural network.

During operation, the SEC engine <NUM> may provide the audio data samples <NUM> as input to a single SEC model (e.g., the SEC model <NUM>), to multiple selected SEC models (e.g., the SEC model <NUM> and a Kth SEC model <NUM> of the available SEC models <NUM>), or to each of the SEC models (e.g., to the SEC model <NUM>, a first SEC model <NUM>, the Kth SEC model <NUM>, and any other SEC models of the available SEC models <NUM>). For example, the SEC engine <NUM> (or another component of the device <NUM>) may select the SEC model <NUM> from among the available SEC models <NUM> based on, for example, user input, device settings associated with the device <NUM>, sensor data, a time when the audio data samples <NUM> are received, or other factors. In this example, the SEC engine <NUM> may select to use only the SEC model <NUM> or may select to use two or more of the available SEC models <NUM>. To illustrate, the device settings may indicate that the SEC model <NUM> and the first SEC model <NUM> are to be used during a particular time frame. In another example, the SEC engine <NUM> may provide the audio data samples <NUM> to each of the available SEC models <NUM> (e.g., sequentially or in parallel) to generate output from each. In a particular aspect, the SEC models are trained to recognize different sound classes, to recognize the same sound classes in different acoustic environments, or both. For example, the SEC model <NUM> may be configured to recognize a first set of sound classes and the first SEC model <NUM> may be configured to recognize a second set of sound classes, where the first set of sound classes is different from the second set of sound classes.

In a particular aspect, the SEC engine <NUM> determines, based on output of the SEC model <NUM>, whether the SEC model <NUM> recognized the sound class of the audio data samples <NUM>. If the SEC engine <NUM> provides the audio data samples <NUM> to multiple SEC models, the SEC engine <NUM> may determine, based on output of each of the SEC models, whether any of the SEC models recognized the sound class of the audio data samples <NUM>. If the SEC model <NUM> (or another of the available SEC models <NUM>) recognized the sound class of the audio data samples <NUM>, the SEC engine <NUM> generates an output <NUM> that indicates the sound class <NUM> of the audio data samples <NUM>. For example, the output <NUM> may be sent to a display to notify a user of detection of the sound class <NUM> associated with the sound <NUM> or may be sent to another device or another component of the device <NUM> and used to trigger an action (e.g., to send a command to activate lights in response to recognizing the sound of a door shutting).

If the SEC engine <NUM> determines that the SEC model <NUM> (and others of the available SEC models <NUM> that were provided the audio data samples <NUM>) did not recognize the sound class of the audio data samples <NUM>, the SEC engine <NUM> provides a trigger signal <NUM> to a drift detector <NUM>. For example, the SEC engine <NUM> may set a trigger flag in a memory of the device <NUM>. In some implementations, the SEC engine <NUM> may also provide other data to the drift detector <NUM>. To illustrate, if the SEC model <NUM> generates a value of a confidence metric for each sound class that the SEC model <NUM> is trained to recognize, one or more of the values of the confidence metric may be provided to the drift detector <NUM>. For example, if the SEC model <NUM> is trained to recognize three sound classes, the SEC engine <NUM> may provide a highest confidence value among three confidence values (one for each of the three sound classes) output by the SEC model <NUM> to the drift detector <NUM>.

In a particular aspect, the SEC engine <NUM> determines whether the SEC model <NUM> recognized the sound class of the audio data samples <NUM> based on a value of a confidence metric. In this particular aspect, a value of the confidence metric for a particular sound class indicates the probability that the audio data samples <NUM> are associated with the particular sound class. To illustrate, if the SEC model <NUM> is trained to recognize four sound classes, the SEC model <NUM> may generate as output an array that includes four values of the confidence metric, one for each sound class. In some implementations, the SEC engine <NUM> determines that the SEC model <NUM> recognized the sound class <NUM> of the audio data samples <NUM> if the value of the confidence metric for the sound class <NUM> is greater than a detection threshold. For example, the SEC engine <NUM> determines that the SEC model <NUM> recognized the sound class <NUM> of the audio data samples <NUM> if the value of the confidence metric for the sound class <NUM> is greater than <NUM> (e.g., <NUM>% confidence), <NUM> (e.g., <NUM>% confidence), or some other value of the detection threshold. In some implementations, the SEC engine <NUM> determines that the SEC model <NUM> did not recognize a sound class of the audio data samples <NUM> if the value of the confidence metric for each sound class that the SEC model <NUM> is trained to recognize is less than the detection threshold. For example, the SEC engine <NUM> determines that the SEC model <NUM> did not recognize the sound class <NUM> of the audio data samples <NUM> if each value of the confidence metric is less than <NUM> (e.g., <NUM>% confidence), <NUM> (e.g., <NUM>% confidence), or some other value of the detection threshold.

The drift detector <NUM> is configured to determine whether the SEC model <NUM> that was not able to recognize the sound class of the audio data samples <NUM> corresponds to an audio scene <NUM> associated with the audio data samples <NUM>. In the example illustrated in <FIG>, a scene detector <NUM> is configured to receive scene data <NUM> and to use the scene data <NUM> to determine the audio scene <NUM> associated with the audio data samples <NUM>. In a particular aspect, the scene data <NUM> is generated based on settings data <NUM> indicating one or more device settings associated with the device <NUM>, output of a clock <NUM>, sensor data from one or more sensors <NUM>, input received via an input device <NUM>, or a combination thereof. In some aspects, the scene detector <NUM> uses different information to determine the audio scene <NUM> than the SEC engine <NUM> uses to select the SEC model <NUM>. To illustrate, if the SEC engine <NUM> selects the SEC model <NUM> based on time of day, the scene detector <NUM> may use position sensor data from a position sensor of the sensor(s) <NUM> to determine the audio scene <NUM>. In some aspects, the scene detector <NUM> uses at least some of the same information that the SEC engine <NUM> uses to select the SEC model <NUM> and uses additional information. To illustrate, if the SEC engine <NUM> selects the SEC model <NUM> based on time of day and the settings data <NUM>, the scene detector <NUM> may use the position sensor data and the settings data <NUM> to determine the audio scene <NUM>. Thus, the scene detector <NUM> uses a different audio scene detection mode than is used by the SEC engine <NUM> to select the SEC model <NUM>.

In a particular implementation, the scene detector <NUM> is a neural network that is trained to determine the audio scene <NUM> based on the scene data <NUM>. In other implementations, the scene detector <NUM> is a classifier that trained using a different machine-learning technique. For example, the scene detector <NUM> may include or correspond to a decision tree, a random forest, a support vector machine, or another classifier that is trained to generate output indicating the audio scene <NUM> based on the scene data <NUM>. In still other implementations, the scene detector <NUM> uses heuristics to determine the audio scene <NUM> based on the scene data <NUM>. In yet other implementations, the scene detector <NUM> uses a combination of artificial intelligence and heuristics to determine the audio scene <NUM> based on the scene data <NUM>. For example, the scene data <NUM> may include image data, video data, or both, and the scene detector <NUM> may include an image recognition model that is trained using a machine-learning technique to detect particular objects, motions, backgrounds, or other image or video information. In this example, output of the image recognition model may be evaluated via one or more heuristics to determine the audio scene <NUM>.

The drift detector <NUM> compares the audio scene <NUM> indicated by the scene detector <NUM> to information descriptive of the SEC model <NUM> to determine whether the SEC model <NUM> is associated with the audio scene <NUM> of the audio data samples <NUM>. If the drift detector <NUM> determines that the SEC model <NUM> is associated with the audio scene <NUM> of the audio data samples <NUM>, the drift detector <NUM> causes drift data <NUM> to be stored as model update data <NUM>. In a particular implementation, the drift data <NUM> includes the audio data samples <NUM> and a label, where the label identifies the SEC model <NUM>, indicates a sound class associated with the audio data samples <NUM>, or both. If the drift data <NUM> indicates a sound class associated with the audio data samples <NUM>, the sound class may be selected based on a highest value of the confidence metric generated by the SEC model <NUM>. As an illustrative example, if the SEC engine <NUM> uses a detection threshold of <NUM>, and the highest value of the confidence metric output by the SEC model <NUM> is <NUM> for a particular sound class, the SEC engine <NUM> determines that the sound class of the audio data samples <NUM> was not recognized and sends the trigger signal <NUM> to the drift detector <NUM>. In this example, if the drift detector <NUM> determines that the SEC model <NUM> corresponds to the audio scene <NUM> of the audio data samples <NUM>, the drift detector <NUM> stores that the audio data samples <NUM> as drift data <NUM> associated with the particular sound class. In a particular aspect, metadata associated with the SEC models <NUM> includes information specifying an audio scene or audio scenes associated with each SEC model <NUM>. For example, the SEC model <NUM> may be configured to detect sound events in a user's home, in which case the metadata associated with the SEC model <NUM> may indicate that the SEC model <NUM> is associate with a "home" audio scene. In this example, if the audio scene <NUM> indicates that the device <NUM> is at a home location (e.g., based on position information, user input, detection of a home wireless network signal, image or video data representing home locations, etc.), the drift detector <NUM> determines that the SEC model <NUM> corresponds to the audio scene <NUM>.

In some implementations, the drift detector <NUM> also causes some audio data samples <NUM> to be stored as model update data <NUM> and designated as unknown data <NUM>. As a first example, the drift detector <NUM> may store the unknown data <NUM> if the drift detector <NUM> determines that the SEC model <NUM> does not correspond to the audio scene <NUM> of the audio data samples <NUM>. As a second example, the drift detector <NUM> may store the unknown data <NUM> if the value of the confidence metric output by the SEC model <NUM> fails to satisfy a drift threshold. In this example, the drift threshold is less than the detection threshold used by the SEC engine <NUM>. For example, if the SEC engine <NUM> uses a detection threshold of <NUM>, the drift threshold may have a value of <NUM>, of <NUM>, or some other value less than <NUM>. In this example, if the highest value of the confidence metric for the audio data samples <NUM> is less than the drift threshold, the drift detector <NUM> determines that the audio data samples <NUM> belong to a sound class that the SEC model <NUM> is not trained to recognize, and designates the audio data samples <NUM> as unknown data <NUM>. In a particular aspect, the drift detector <NUM> only stores the unknown data <NUM> if the drift detector <NUM> determines that the SEC model <NUM> corresponds to the audio scene <NUM> of the audio data samples <NUM>. In another particular aspect, the drift detector <NUM> stores the unknown data <NUM> independently of whether the drift detector <NUM> determines that the SEC model <NUM> corresponds to the audio scene <NUM> of the audio data samples <NUM>.

After the model update data <NUM> is stored, a model updater <NUM> can access the model update data <NUM> and use the model update data <NUM> to update one of the available SEC models <NUM> (e.g., the SEC model <NUM>). For example, each entry of the model update data <NUM> indicates an SEC model with which the entry is associated, and the model updater <NUM> uses the entry as training data to update the corresponding SEC model. In a particular aspect, the model updater <NUM> updates an SEC model when an update criterion is satisfied or when a model update is initiated by a user or another party (e.g., a vendor of the device <NUM>, the SEC engine <NUM>, the SEC models <NUM>, etc.). The update criterion may be satisfied when a particular number of entries are available in the model update data <NUM>, when a particular number of entries for a particular SEC model are available in the model update data <NUM>, when a particular number of entries for a particular sound class are available in the model update data <NUM>, when a particular amount of time has passed since a prior update, when other updates occur (e.g., when a software update associated with the device <NUM> occurs), or based on occurrence of another event.

The model updater <NUM> uses the drift data <NUM> as labeled training data to update training of the SEC model <NUM> using backpropagation or a similar machine-learning optimization process. For example, the model updater <NUM> provides audio data samples from the drift data <NUM> of the model update data <NUM> as input to the SEC model <NUM>, determines a value of an error function (also referred to as a loss function) based on output of the SEC model <NUM> and a label associate with the audio data samples (as indicated in the drift data <NUM> stored by the drift detector <NUM>), and determines updated link weights for the SEC model <NUM> using a gradient descent operation (or some variant thereof) or another machine-learning optimization process.

The model updater <NUM> may also provide other audio data samples (in addition to audio data samples of the drift data <NUM>) to the SEC model <NUM> during the update training. For example, the model update data <NUM> may include one or more known audio data samples (such as a subset of the audio data samples originally used to train the SEC model <NUM>), which may reduce the chances of the update training causing the SEC model <NUM> to forget previous training (where "forgetting" here refers to losing reliability for detecting sound classes that the SEC model <NUM> was previously trained to recognize). Since the sound class associated with the audio data samples of the drift data <NUM> is indicated by the drift detector <NUM>, update training to account for drift can be accomplished automatically (e.g., without user input). As a result, functionality of the device <NUM> (e.g., accuracy in recognizing sound classes) can improve over time without user intervention and using fewer computing resources than would be used to generate a new SEC model from scratch. A particular example of a transfer learning process that the model updater <NUM> can use to update the SEC model <NUM> based on the drift data <NUM> is described with reference to <FIG>.

In some aspects, the model updater <NUM> can also use the unknown data <NUM> of the model update data <NUM> to update training of the SEC model <NUM>. For example, periodically or occasionally, such as when the update criterion is satisfied, the model updater <NUM> may prompt a user to ask the user to label the sound class of the entries of the unknown data <NUM> in the model update data <NUM>. If the user choses to label the sound class of an entry of unknown data <NUM>, the device <NUM> (or another device) may playout sound corresponding to audio data samples of the unknown data <NUM>. The user can provide one or more labels <NUM> (e.g., via the input device <NUM>) identifying a sound class of the audio data samples. If the sound class indicated by the user is a sound class that the SEC model <NUM> is trained to recognize, then the unknown data <NUM> is reclassified as drift data <NUM> associated with the user-specified sound class and the SEC model <NUM>. Depending on the configuration of the model updater <NUM>, if the sound class indicated by the user is a sound class that the SEC model <NUM> is not trained to recognize (e.g., is a new sound class), the model updater <NUM> may discard the unknown data <NUM>, send the unknown data <NUM> and the user-specified sound class to another device for use to generate a new or updated SEC model, or may use the unknown data <NUM> and the user-specified sound class to update the SEC model <NUM>. A particular example of a transfer learning process that the model updater <NUM> can use to update the SEC model <NUM> based on the unknown data <NUM> and the user-specified sound class is described with reference to <FIG>.

An updated SEC model <NUM> generated by the model updater <NUM> is added to the available SEC models <NUM> to make the updated SEC model <NUM> available to evaluate audio data samples <NUM> received after the updated SEC model <NUM> is generated. Thus, the set of available SEC models <NUM> that can be used to evaluate sounds is dynamic. For example, one or more of the available SEC models <NUM> can be automatically updated to account for drift data <NUM>. Additionally, one or more of the available SEC models <NUM> can be updated to account for unknown sound classes using transfer learning operations that use fewer computing resources (e.g., memory, processing time, and power) than training a new SEC model from scratch.

<FIG> is a diagram that illustrates aspects of updating an SEC model <NUM> to account for drift according to a particular example. The SEC model <NUM> of <FIG> includes or corresponds to a particular one of the available SEC models <NUM> of <FIG> that is associated with the drift data <NUM>. For example, if the SEC engine <NUM> generated the trigger signal <NUM> in response to output of the SEC model <NUM>, the drift data <NUM> is associated with the SEC model <NUM>, and the SEC model <NUM> corresponds to or includes the SEC model <NUM>. As another example, if the SEC engine <NUM> generated the trigger signal <NUM> in response to output of the Kth SEC model <NUM>, the drift data <NUM> is associated with the Kth SEC model <NUM>, and the SEC model <NUM> corresponds to or includes the Kth SEC model <NUM>.

In the example illustrated in <FIG>, training data <NUM> is used to update the SEC model <NUM>. The training data <NUM> includes the drift data <NUM> and one or more labels <NUM>. Each entry of the drift data <NUM> includes audio data samples (e.g., audio data samples <NUM>) and is associated with a corresponding label of the label(s) <NUM>. The audio data samples of an entry of the drift data <NUM> include a set of values representing features extracted from or determined based on a sound that was not recognized by the SEC model <NUM>. The label <NUM> corresponding to an entry of the drift data <NUM> identifies a sound class to which the sound is expected to belong. As an example, the label <NUM> corresponding to an entry of the drift data <NUM> may be assigned by the drift detector <NUM> of <FIG> in response to determining that the SEC model <NUM> corresponds to the audio scene in which the audio data samples were generated. In this example, the drift detector <NUM> may assign the audio data samples to the sound class that was associated, in the output of the SEC model <NUM>, with a highest confidence metric value.

In <FIG>, audio data samples <NUM> corresponding to a sound are provided to the SEC model <NUM>, and the SEC model <NUM> generates output <NUM> that indicates a sound class to which the audio data samples <NUM> are assigned, one or more values of a confidence metric, or both. The model updater <NUM> uses the output <NUM> and the label <NUM> corresponding to the audio data samples <NUM> to determine updated link weights <NUM> for the SEC model <NUM>. The SEC model <NUM> is updated based on the updated link weights <NUM>, and the training process is repeated iteratively until a training termination condition is satisfied. During training, each of the entries of the drift data <NUM> may be provided to the SEC model <NUM> (e.g., one entry per iteration). Additionally, in some implementations, other audio data samples (e.g., audio data samples previously used to train the SEC model <NUM>) may also be provided to the SEC model <NUM> to reduce the chance of the SEC model <NUM> forgetting prior training.

The training termination condition may be satisfied when all of the drift data <NUM> has been provided to the SEC model <NUM> at least once, after a particular number of training iterations have been performed, when a convergence metric satisfies a convergence threshold, or when some other condition indicative of the end of training is met. When the training termination condition is satisfied, the model updater <NUM> stores the updated SEC model <NUM>, where the updated SEC model <NUM> corresponds to the SEC model <NUM> with link weights based on the updated link weights <NUM> applied during training.

<FIG> is a diagram that illustrates aspects of updating an SEC model <NUM> to account for unknown data according to a particular example. The SEC model <NUM> of <FIG> includes or corresponds to a particular one of the available SEC models <NUM> of <FIG> that is associated with the unknown data <NUM>. For example, if the SEC engine <NUM> generated the trigger signal <NUM> in response to output of the SEC model <NUM>, the unknown data <NUM> is associated with the SEC model <NUM>, and the SEC model <NUM> corresponds to or includes the SEC model <NUM>. As another example, if the SEC engine <NUM> generated the trigger signal <NUM> in response to output of the Kth SEC model <NUM>, the unknown data <NUM> is associated with the Kth SEC model <NUM>, and the SEC model <NUM> corresponds to or includes the Kth SEC model <NUM>.

In the example of <FIG>, the model updater <NUM> generates an update model <NUM>. The update model <NUM> includes the SEC model <NUM> that is to be updated, an incremental model <NUM>, and one or more adapter networks <NUM>. The incremental model <NUM> is a copy of the SEC model <NUM> with a different output layer than the SEC model <NUM>. In particular, the output layer of the incremental model <NUM> includes more output nodes than the output layer of the SEC model <NUM>. For example, the output layer of the SEC model <NUM> includes a first count of nodes (e.g., N nodes, where N is a positive integer corresponding to the number of sound classes that the SEC model <NUM> is trained to recognize), and the output layer of the incremental model <NUM> includes a second count of nodes (e.g., N+M nodes, where M is a positive integer corresponding to a number of new sound classes that an updated SEC model <NUM> is to be trained to recognized that the SEC model <NUM> is not trained to recognize). The first count of nodes corresponds to the count of sound classes of a first set of sound classes that the SEC model <NUM> is trained to recognize (e.g., the first set of sound classes includes N distinct sound classes that the SEC model <NUM> can recognize), and the second count of nodes corresponds to the count of sound classes of a second set of sound classes that the updated SEC model <NUM> is to be trained to recognize (e.g., the second set of sound classes includes N+M distinct sound classes that the updated SEC model <NUM> is to be trained to recognize). The second set of sound classes includes the first set of sound classes (e.g., N classes) plus one or more additional sound classes (e.g., M classes). Model parameters (e.g., link weights) of the incremental model <NUM> are initialized to be equal to model parameters of the SEC model <NUM>.

The adapter network(s) <NUM> include a neural adapter and a merger adapter. The neural adapter includes one or more adapter layers configured to receive input from the SEC model <NUM> and to generate output that can be merged with the output of the incremental model <NUM>. For example, the SEC model <NUM> generates a first output corresponding to the first count of classes of the first set of sound classes. In a particular aspect, the first output includes one data element for each node of the output layer of the SEC model <NUM> (e.g., N data elements). In contrast, the incremental model <NUM> generates a second output corresponding to the second count of classes of the second set of sound classes. For example, the second output includes one data element for each node of the output layer of the incremental model <NUM> (e.g., N+M data elements). In this example, the adapter layer(s) of the adapter network(s) <NUM> receive the output of the SEC model <NUM> as input and generate an output having the second count of data elements (e.g., N+M). In a particular example, the adapter layer(s) of the adapter network(s) <NUM> include two fully connected layers (e.g., an input layer including N nodes and an output layer including N+M nodes, with each node of the input layer connected to every node of the output layer).

The merger adapter of the adapter network(s) <NUM> is configured to generate output <NUM> of the update model <NUM> by merging the output of the adapter layer(s) and the output of the incremental model <NUM>. For example, the merger adapter combines the output of the adapter layer(s) and the output of the incremental model <NUM> in an element-by-element manner to generate a combined output and applies an activation function (such as a sigmoid function) to the combined output to generate the output <NUM>. The output <NUM> indicates a sound class to which the audio data samples <NUM> are assigned by the update model <NUM>, one or more confidence metric values determined by the update model <NUM>, or both.

The model updater <NUM> uses the output <NUM> and a label <NUM> corresponding to the audio data samples <NUM> to determine updated link weights <NUM> for the incremental model <NUM>, the adapter network(s) <NUM>, or both. Link weights of the SEC model <NUM> are unchanged during training. The training process is repeated iteratively until a training termination condition is satisfied. During training, each of the entries of the unknown data <NUM> may be provided to the update model <NUM> (e.g., one entry per iteration). Additionally, in some implementations, other audio data samples (e.g., audio data samples previously used to train the SEC model <NUM>) may also be provided to the update model <NUM> to reduce the chance of the incremental model <NUM> forgetting prior training of the SEC model <NUM>.

The training termination condition may be satisfied when all of the unknown data <NUM> has been provided to the update model <NUM> at least once, after a particular number of training iterations have been performed, when a convergence metric satisfies a convergence threshold, or when some other condition indicative of the end of training is met. When the training termination condition is satisfied, a model checker <NUM> selects the updated SEC model <NUM> from between the incremental model <NUM> and the update model <NUM> (e.g., the combination of the SEC model <NUM>, the incremental model <NUM>, and the adapter network(s) <NUM>).

In a particular aspect, the model checker <NUM> selects the updated SEC model <NUM> based on an accuracy of sound classes <NUM> assigned by the incremental model <NUM> and an accuracy of the sound classes <NUM> assigned by the SEC model <NUM>. For example, the model checker <NUM> may determine an F1-score for the incremental model <NUM> (based on the sound classes <NUM> assigned by the incremental model <NUM>) and an F1-score of the SEC model <NUM> ( based on the sound classes <NUM> assigned by the SEC model <NUM>). In this example, if the value of the F1-score of incremental model <NUM> is greater than or equal to the value of the F1-score of the SEC model <NUM>, the model checker <NUM> selects the incremental model <NUM> as the updated SEC model <NUM>. In some implementations, the model checker <NUM> selects the incremental model <NUM> as the updated SEC model <NUM> if the value of the F1-score of the incremental model <NUM> is greater than or equal to the value of the F1-score of the SEC model <NUM> (or is less than the value of the F1-score of the SEC model <NUM> by less than a threshold amount). If the value of the F1-score of incremental model <NUM> is less than the value of the F1-score for the SEC model <NUM> (or is less than the value of the F1-score for the SEC model <NUM> by more than the threshold amount), the model checker <NUM> selects the update model <NUM> as the updated SEC model <NUM>. If the incremental model <NUM> is selected as the updated SEC model <NUM>, the SEC model <NUM>, the adapter network(s) <NUM>, or both may be discarded.

In some implementations, the model checker <NUM> is omitted or integrated with the model updater <NUM>. For example, after training the update model <NUM>, the update model <NUM> can be stored as the updated SEC model <NUM> (e.g., with no selection between the update model <NUM> and the incremental model <NUM>). As example, while training the update model <NUM>, the model updater <NUM> can determine an accuracy metric for the incremental model <NUM>. In this example, the training termination condition may be based on the accuracy metric for the incremental model <NUM> such that after training, the incremental model <NUM> is stored as the updated SEC model <NUM> (e.g., with no selection between the update model <NUM> and the incremental model <NUM>).

Utilizing the transfer learning techniques described with reference to <FIG>, the model checker <NUM> enables the device <NUM> of <FIG> to update an SEC model to recognize a previously unknown sound class. Additionally, the transfer learning techniques described use significantly less computer resources (e.g., memory, processing time, and power) than would be used to train an SEC model from scratch to recognize the previously unknown sound class.

In some implementations, the operations described with reference to <FIG> (e.g., generating the updated SEC model <NUM> based on the drift data <NUM>) are performed at the device <NUM> of <FIG>, and the operations described with reference to <FIG> (e.g., generating the updated SEC model <NUM> based on the unknown data <NUM>) are performed at different device (such as a remote computing device <NUM> of <FIG>). To illustrate, the unknown data <NUM> and label(s) <NUM> can be captured at the device <NUM> and transmitted to a second device that has more available computing resources. In this example, the second device generates the updated SEC model <NUM> and the device <NUM> downloads or receives a transmission or data representing the updated SEC model <NUM> from the second device. Generating the updated SEC model <NUM> based on the unknown data <NUM> is a more resource intensive process (e.g., uses more memory, power, and processor time) than generating the updated SEC model <NUM> based on the drift data <NUM>. Thus, dividing operations described with reference to <FIG> and the operations described with reference to <FIG> among different devices can conserve resources of the device <NUM>.

<FIG> is a diagram illustrating a particular example of operation of the device of <FIG>. <FIG> illustrates an implementation in which a determination of whether an active SEC model (e.g., the SEC model <NUM>) corresponds to an audio scene in which audio data samples <NUM> are captured is based on comparing a current audio scene to a prior audio scene.

In <FIG>, audio data captured by the microphone(s) <NUM> is used to generate the audio data samples <NUM>. The audio data samples <NUM> are used to perform audio classification <NUM>. For example, one or more of the available SEC models <NUM> is used as an active SEC model by the SEC engine <NUM> of <FIG>. In a particular aspect, the active SEC model is selected from among the available SEC models <NUM> based on an audio scene indicated by the scene detector <NUM> during a prior sampling period, which is also referred to as a prior audio scene <NUM>.

Audio classification <NUM> generates a result <NUM> based on analysis of the audio data samples <NUM> using the active SEC model. The result <NUM> may indicate a sound class associated with the audio data samples <NUM>, a probability that the audio data samples <NUM> correspond to a particular sound class, or that a sound class of the audio data samples <NUM> is unknown. If the result <NUM> indicates that the audio data samples <NUM> correspond to a known sound class, a decision is made, at block <NUM>, to generate an output <NUM> indicating the sound class <NUM> associated with the audio data samples <NUM>. For example, the SEC engine <NUM> of <FIG> may generate the output <NUM>.

If the result <NUM> indicates that the audio data samples <NUM> do not correspond to a known sound class, a decision is made, at block <NUM>, to generate the trigger <NUM>. The trigger <NUM> activates a drift detection scheme, which in <FIG> includes causing the scene detector <NUM> to identify the current audio scene <NUM> based on data from the sensor(s) <NUM>.

The current audio scene <NUM> is compared, at block <NUM>, to the prior audio scene <NUM> to determine whether an audio scene change has occurred since the active SEC model was selected. At block <NUM>, a determination is made whether the sound class of the audio data samples <NUM> was not recognized due to drift. For example, if the current audio scene <NUM> does not correspond to the prior audio scene <NUM>, the determination at block <NUM> is that drift was not the cause of the sound class of the audio data samples <NUM> not being recognized. In this circumstance, the audio data samples <NUM> may be discarded or, at block <NUM>, stored as unknown data.

If the current audio scene <NUM> corresponds to the prior audio scene <NUM>, the determination at block <NUM> is that the sound class of the audio data samples <NUM> was not recognized due to drift because the active SEC model corresponds to the current audio scene <NUM>. In this circumstance, the sound class that has drifted is identified, at block <NUM>, and the audio data samples <NUM> and an identifier of the sound class are stored as drift data, at block <NUM>.

When sufficient drift data is stored, the SEC model is updated, at block <NUM>, to generate the updated SEC model <NUM>. The updated SEC model <NUM> is added to the available SEC models <NUM>. In some implementations, the updated SEC model <NUM> replaces the active SEC model that generated the result <NUM>.

<FIG> is a diagram illustrating another particular example of operation of the device of <FIG>. <FIG> illustrates an implementation in which a determination of whether an active SEC model (e.g., SEC model <NUM>) corresponds to an audio scene in which audio data samples <NUM> are captured is based on comparing a current audio scene to information descriptive of the active SEC model.

In <FIG>, audio data captured by the microphone(s) <NUM> is used to generate the audio data samples <NUM>. The audio data samples <NUM> are used to perform audio classification <NUM>. For example, one or more of the available SEC models <NUM> is used as an active SEC model by the SEC engine <NUM> of <FIG>. In a particular aspect, the active SEC model is selected from among the available SEC models <NUM>. In some implementations, an ensemble of the available SEC models <NUM> is used rather than selecting one or more of the available SEC models <NUM> as an active SEC model.

The audio classification <NUM> generates a result <NUM> based on analysis of the audio data samples <NUM> using one or more of the available SEC model <NUM>. The result <NUM> may indicate a sound class associated with the audio data samples <NUM>, a probability that the audio data samples <NUM> correspond to a particular sound class, or that a sound class of the audio data samples <NUM> is unknown. If the result <NUM> indicates that the audio data samples <NUM> correspond to a known sound class, a decision is made, at block <NUM>, to generate an output <NUM> indicating the sound class <NUM> associated with the audio data samples <NUM>. For example, the SEC engine <NUM> of <FIG> may generate the output <NUM>.

If the result <NUM> indicates that the audio data samples <NUM> do not correspond to a known sound class, a decision is made, at block <NUM>, to generate the trigger <NUM>. The trigger <NUM> activates a drift detection scheme, which in <FIG> includes causing the scene detector <NUM> to identify the current audio scene based on data from the sensor(s) <NUM> and to determine whether the current audio scene corresponds to the SEC model that generated the result <NUM> that caused the trigger <NUM> to be sent.

At block <NUM>, a determination is made whether the sound class of the audio data samples <NUM> was not recognized due to drift. For example, if the current audio scene does not correspond to the SEC model that generated the result <NUM>, the determination at block <NUM> is that drift was not the cause of the sound class of the audio data samples <NUM> not being recognized. In this circumstance, the audio data samples <NUM> may be discarded or, at block <NUM>, stored as unknown data.

If the current audio scene corresponds to the SEC model that generated the result <NUM>, the determination at block <NUM> is that the sound class of the audio data samples <NUM> was not recognized due to drift. In this circumstance, the sound class that has drifted is identified, at block <NUM>, and the audio data samples <NUM> and an identifier of the sound class are stored as drift data, at block <NUM>.

<FIG> a block diagram illustrating a particular example of the device <NUM> of <FIG>. In <FIG>, the device <NUM> is configured to use the SEC models <NUM> to generate sound identification data (e.g., the output <NUM> of <FIG>) responsive to input of audio data samples (e.g., the audio data samples <NUM> of <FIG>). Additionally, the device <NUM> of <FIG> is configured to update the one or more of the SEC models <NUM> based on the model update data <NUM>. For example, the device <NUM> is configured to update the SEC models <NUM> using the drift data <NUM> as described with reference to <FIG>, is configured to update the SEC models <NUM> using the unknown data <NUM> as described with reference to <FIG>, or both. In some implementations, a remote computing device <NUM> updates the SEC models <NUM> in some circumstances. To illustrate, the device <NUM> may update the SEC models <NUM> using the drift data <NUM>, and the remote computing device <NUM> may update the SEC models <NUM> using the unknown data <NUM>. In various implementations, the device <NUM> may have more or fewer components than illustrated in <FIG>.

In a particular implementation, the device <NUM> includes a processor <NUM> (e.g., a central processing unit (CPU)). The device <NUM> may include one or more additional processor(s) <NUM> (e.g., one or more digital signal processors (DSPs)). The processor <NUM>, the processor(s) <NUM>, or both, may be configured to generate sound identification data, to update the SEC model <NUM>, or both. For example, in <FIG>, the processor(s) <NUM> include the SEC engine <NUM>. The SEC engine <NUM> is configured to analyze audio data samples using one or more of the SEC models <NUM>.

In <FIG>, the device <NUM> also includes a memory <NUM> and a CODEC <NUM>. The memory <NUM> stores instructions <NUM> that are executable by the processor <NUM>, or the processor(s) <NUM>, to implement one or more operations described with reference to <FIG>. In an example, the instructions <NUM> include or correspond to the feature extractor <NUM>, the SEC engine <NUM>, the scene detector <NUM>, the drift detector <NUM>, the model updater <NUM>, the model checker <NUM>, or a combination thereof. The memory <NUM> may also store the settings data <NUM>, the SEC models <NUM>, and the model update data <NUM>.

In <FIG>, speaker(s) <NUM> and the microphone(s) <NUM> may be coupled to the CODEC <NUM>. In a particular aspect, the microphone(s) <NUM> are configured to receive audio representing an acoustic environment associated with the device <NUM> and to generate audio signals that the feature extractor uses to generate audio data samples. In the example illustrated in <FIG>, the CODEC <NUM> includes a digital-to-analog converter (DAC <NUM>) and an analog-to-digital converter (ADC <NUM>). In a particular implementation, the CODEC <NUM> receives analog signals from the microphone(s) <NUM>, converts the analog signals to digital signals using the ADC <NUM>, and provides the digital signals to the processor(s) <NUM>. In a particular implementation, the processor(s) <NUM> provide digital signals to the CODEC <NUM>, and the CODEC <NUM> converts the digital signals to analog signals using the DAC <NUM> and provides the analog signals to the speaker(s) <NUM>.

In <FIG>, the device <NUM> also includes an input device <NUM>. The device <NUM> may also include a display <NUM> coupled to a display controller <NUM>. In a particular aspect, the input device <NUM> includes a sensor, a keyboard, a pointing device, etc. In some implementations, the input device <NUM> and the display <NUM> are combined in a touchscreen or similar touch or motion sensitive display. The input device <NUM> can be used to provide a label associated with the unknown data <NUM> to generate the training data <NUM>. The input device <NUM> can also be used to initiate a model update operation, such as to start the model update process described with reference to <FIG>, or the model update process described with reference to <FIG>. In some implementations, the input device <NUM> can also, or alternatively, be used to select a particular SEC model of the available SEC models <NUM> to be used by the SEC engine <NUM>. In a particular aspect, the input device <NUM> can be used to configure the settings data <NUM>, which may be used to select an SEC model to be used by the SEC engine <NUM>, to determine an audio scene <NUM>, or both. The display <NUM> can be used to display results of analysis by one of the SEC models (e.g., the output <NUM> of <FIG>), to display a prompt to the user to provide a label associated with unknown data <NUM>, or both.

In some implementations, the device <NUM> also includes a modem <NUM> coupled to a transceiver <NUM>. In <FIG>, the transceiver <NUM> is coupled to an antenna <NUM> to enable wireless communication with other devices, such as the remote computing device <NUM>. In other examples, the transceiver <NUM> is also, or alternatively, coupled to a communication port (e.g., an ethernet port) to enable wired communication with other devices, such as the remote computing device <NUM>.

In <FIG>, the device <NUM> includes the clock <NUM> and the sensors <NUM>. As specific examples, the sensors <NUM> include one or more cameras <NUM>, one or more position sensors <NUM>, the microphone(s) <NUM>, other sensor(s) <NUM>, or a combination thereof.

In a particular aspect, the clock <NUM> generates a clock signal that can be used to assign a timestamp to particular audio data samples to indicate when particular audio data samples were received. In this aspect, the SEC engine <NUM> can use the timestamp to select a particular SEC model <NUM> to use to analyze the particular audio data samples. Additionally or alternatively, the timestamp can be used by the scene detector <NUM> to determine the audio scene <NUM> associated with the particular audio data samples.

In a particular aspect, the camera(s) <NUM> generate image data, video data, or both. The SEC engine <NUM> can use the image data, the video data, or both, to select a particular SEC model <NUM> to use to analyze audio data samples. Additionally or alternatively, the image data, the video data, or both, can be used by the scene detector <NUM> to determine the audio scene <NUM> associated with the particular audio data samples. For example, the particular SEC model <NUM> can be designated for outdoor use, and the image data, the video data, or both, may be used to confirm that the device <NUM> is located in an outdoor environment.

In a particular aspect, the position sensor(s) <NUM> generate position data, such as global position data indicating a location of the device <NUM>. The SEC engine <NUM> can use the position data to select a particular SEC model <NUM> to use to analyze audio data samples. Additionally or alternatively, the position data can be used by the scene detector <NUM> to determine the audio scene <NUM> associated with the particular audio data samples. For example, the particular SEC model <NUM> can be designated for use at home, and the position data may be used to confirm that the device <NUM> is located at a home location. The position sensor(s) <NUM> may include a receiver for a satellite-based positioning system, a receiver for a local positioning system receiver, an inertial navigation system, a landmark-based positioning system, or a combination thereof.

The other sensor(s) <NUM> can include, for example, an orientation sensor, a magnetometer, a light sensor, a contact sensor, a temperature sensor, or any other sensor that is coupled to or included within the device <NUM> and that can be used to generate scene data <NUM> useful for determining the audio scene <NUM> associated with the device <NUM> at a particular time.

In a particular implementation, the device <NUM> is included in a system-in-package or system-on-chip device <NUM>. In a particular implementation, the memory <NUM>, the processor <NUM>, the processor(s) <NUM>, the display controller <NUM>, the CODEC <NUM>, the modem <NUM>, and the transceiver <NUM> are included in the system-in-package or system-on-chip device <NUM>. In a particular implementation, the input device <NUM> and a power supply <NUM> are coupled to the system-on-chip device <NUM>. Moreover, in a particular implementation, as illustrated in <FIG>, the display <NUM>, the input device <NUM>, the speaker(s) <NUM>, the sensors <NUM>, the clock <NUM>, the antenna <NUM>, and the power supply <NUM> are external to the system-on-chip device <NUM>. In a particular implementation, each of the display <NUM>, the input device <NUM>, the speaker(s) <NUM>, the sensors <NUM>, the clock <NUM>, the antenna <NUM>, and the power supply <NUM> may be coupled to a component of the system-on-chip device <NUM>, such as an interface or a controller.

The device <NUM> may include, correspond to, or be included within a voice activated device, an audio device, a wireless speaker and voice activated device, a portable electronic device, a car, a vehicle, a computing device, a communication device, an internet-of-things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, a smart speaker, a mobile computing device, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a gaming console, an appliance, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, or any combination thereof. In a particular aspect, the processor <NUM>, the processor(s) <NUM>, or a combination thereof, are included in an integrated circuit.

<FIG> is an illustrative example of a vehicle <NUM> that incorporates aspects of the device <NUM> of <FIG>. According to one implementation, the vehicle <NUM> is a self-driving car. According to other implementations, the vehicle <NUM> is a car, a truck, a motorcycle, an aircraft, a water vehicle, etc. In <FIG>, the vehicle <NUM> includes the display <NUM>, one or more of the sensors <NUM>, the device <NUM>, or a combination thereof. The sensors <NUM> and the device <NUM> are shown using a dotted line to indicate that these components might not be visible to passengers of the vehicle <NUM>. The device <NUM> can be integrated into the vehicle <NUM> or coupled to the vehicle <NUM>.

In a particular aspect, the device <NUM> is coupled to the display <NUM> and provides an output to the display <NUM> responsive to one of the SEC models <NUM> detecting or recognizing various events (e.g., sound events) described herein. For example, the device <NUM> provides the output <NUM> of <FIG> to the display <NUM> indicating a sound class of a sound <NUM> (such as a car horn) that was recognized by one of the SEC models <NUM> in audio data samples <NUM> received from the microphone(s) <NUM>. In some implementations, the device <NUM> can perform an action responsive to recognizing a sound event, such as alerting an operator of the vehicle or activating one of the sensors <NUM>. In a particular example, the device <NUM> provides an output that indicates whether an action is being performed responsive to the recognized sound event. In a particular aspect, a user can select an option displayed on the display <NUM> to enable or disable a performance of actions responsive to recognized sound events.

In a particular implementations, the sensors <NUM> include the microphone(s) <NUM> of <FIG>, vehicle occupancy sensors, eye tracking sensor, position sensor(s) <NUM>, or external environment sensors (e.g., lidar sensors or cameras). In a particular aspect, sensor input of the sensors <NUM> indicates a location of the user. For example, the sensors <NUM> are associated with various locations within the vehicle <NUM>.

The device <NUM> in <FIG> includes the SEC models <NUM>, the SEC engine <NUM>, the drift detector <NUM>, the scene detector <NUM>, and the model updater <NUM>. In other implementations, the device <NUM>, when installed in or used in the vehicle <NUM>, omits the model updater <NUM>. To illustrate, the model update data <NUM> of <FIG> may be sent to the remote computing device <NUM>, and the remote computing device <NUM> may update one of the SEC models <NUM> based on the model update data <NUM>. In such implementations, the updated SEC model <NUM> can be downloaded to the vehicle <NUM> for used by the SEC engine <NUM>. In some implementations, the device <NUM>, when installed in or used in the vehicle <NUM>, further includes the model checker <NUM> of <FIG>.

Thus, the techniques described with respect to <FIG> enable a user of the vehicle <NUM> to generate an updated SEC model to account for drift that is specific (perhaps unique) to the acoustic environment in which the device <NUM> operates. In some implementations, the device <NUM> can generate the updated SEC model without user intervention. Further, the techniques described with respect to <FIG> enable a user of the vehicle <NUM> to generate an updated SEC model to detect one or more new sound classes. In addition, the SEC model can be updated without excessive use of computing resources onboard the vehicle <NUM>. For example, the vehicle <NUM> does not have to store all of the training data used train an SEC model from scratch in a local memory.

<FIG> depicts an example of the device <NUM> coupled to or integrated within a headset <NUM>, such as a virtual reality headset, an augmented reality headset, a mixed reality headset, an extended reality headset, a head-mounted display, or a combination thereof. A visual interface device, such as the display <NUM>, is positioned in front of the user's eyes to enable display of augmented reality, mixed reality, or virtual reality images or scenes to the user while the headset <NUM> is worn. In a particular example, the display <NUM> is configured to display output of the device <NUM>, such as an indication of a recognized sound event (e.g., the output <NUM> of <FIG>). The headset <NUM> includes the sensors <NUM>, such as the microphone(s) <NUM> of <FIG>, the camera(s) <NUM> of <FIG>, the position sensor(s) <NUM> of <FIG>, the other sensors <NUM> of <FIG>, or a combination thereof. Although illustrated in a single location, in other implementations the sensors <NUM> can be positioned at other locations of the headset <NUM>, such as an array of one or more microphones and one or more cameras distributed around the headset <NUM> to detect multi-modal inputs.

The sensors <NUM> enable detection of audio data, which the device <NUM> uses to detect sound events or to update the SEC models <NUM>. For example, the SEC engine <NUM> uses one or more of the SEC models <NUM> to generate the sound event classification data which may be provided to the display <NUM> to indicate that a recognized sound event, such as a car horn, is detected in audio data samples received from the sensors <NUM>. In some implementations, the device <NUM> can perform an action responsive to recognizing a sound event, such as activating a camera or another one of the sensors <NUM> or providing haptic feedback to the user.

In the example illustrated in <FIG>, the device <NUM> includes the SEC models <NUM>, the SEC engine <NUM>, the drift detector <NUM>, the scene detector <NUM>, and the model updater <NUM>. In other implementations, the device <NUM>, when installed in or used in the headset <NUM>, omits the model updater <NUM>. To illustrate, the model update data <NUM> of <FIG> may be sent to the remote computing device <NUM>, and the remote computing device <NUM> may update one of the SEC models <NUM> based on the model update data <NUM>. In such implementations, the updated SEC model <NUM> can be downloaded to the headset <NUM> for use by the SEC engine <NUM>. In some implementations, the device <NUM>, when installed in or used in the headset <NUM>, further includes the model checker <NUM> of <FIG>.

<FIG> depicts an example of the device <NUM> integrated into a wearable electronic device <NUM>, illustrated as a "smart watch," that includes the display <NUM> and the sensors <NUM>. The sensors <NUM> enable detection, for example, of user input and audio scenes based on modalities such as location, video, speech, and gesture. The sensors <NUM> also enable detection of sounds in an acoustic environment around the wearable electronic device <NUM>, which the device <NUM> uses to detect sound events or to update the SEC models <NUM>. For example, the device <NUM> provides the output <NUM> of <FIG> to the display <NUM> indicating that a recognized sound event is detected in audio data samples received from the sensors <NUM>. In some implementations, the device <NUM> can perform an action responsive to recognizing a sound event, such as activating a camera or another one of the sensors <NUM> or providing haptic feedback to the user.

In the example illustrated in <FIG>, the device <NUM> includes the SEC models <NUM>, the SEC engine <NUM>, the drift detector <NUM>, the scene detector <NUM>, and the model updater <NUM>. In other implementations, the device <NUM>, when installed in or used in the wearable electronic device <NUM>, omits the model updater <NUM>. To illustrate, the model update data <NUM> of <FIG> may be sent to the remote computing device <NUM>, and the remote computing device <NUM> may update one of the SEC models <NUM> based on the model update data <NUM>. In such implementations, the updated SEC model <NUM> can be downloaded to the wearable electronic device <NUM> for use by the SEC engine <NUM>. In some implementations, the device <NUM>, when installed in or used in the wearable electronic device <NUM>, further includes the model checker <NUM> of <FIG>.

<FIG> is an illustrative example of a voice-controlled speaker system <NUM>. The voice-controlled speaker system <NUM> can have wireless network connectivity and is configured to execute an assistant operation. In <FIG>, the device <NUM> is included in the voice-controlled speaker system <NUM>. The voice-controlled speaker system <NUM> also includes a speaker <NUM> and sensors <NUM>. The sensors <NUM> include microphone(s) <NUM> of <FIG> to receive voice input or other audio input.

During operation, in response to receiving a verbal command or a recognized sound event, the voice-controlled speaker system <NUM> can execute assistant operations. The assistant operations can include adjusting a temperature, playing music, turning on lights, etc. The sensors <NUM> enable detection of audio data samples, which the device <NUM> uses to detect sound events or to update one or more of the SEC models <NUM>. Additionally, the voice-controlled speaker system <NUM> can execute some operations based on sound events recognized by the device <NUM>. For example, if the device <NUM> recognizes the sound of a door closing, the voice-controlled speaker system <NUM> can turn on one or more lights.

In the example illustrated in <FIG>, the device <NUM> includes the SEC models <NUM>, the SEC engine <NUM>, the drift detector <NUM>, the scene detector <NUM>, and the model updater <NUM>. In other implementations, the device <NUM>, when installed in or used in the voice-controlled speaker system <NUM> omits the model updater <NUM>. To illustrate, the model update data <NUM> of <FIG> may be sent to the remote computing device <NUM>, and the remote computing device <NUM> may update one of the SEC models <NUM> based on the model update data <NUM>. In such implementations, the updated SEC model <NUM> can be downloaded to the voice-controlled speaker system <NUM> for use by the SEC engine <NUM>. In some implementations, the device <NUM>, when installed in or used in the voice-controlled speaker system <NUM>, further includes the model checker <NUM> of <FIG>.

<FIG> illustrates a camera <NUM> that incorporates aspects of the device <NUM> of <FIG>. In <FIG>, the device <NUM> is incorporated in or coupled to the camera <NUM>. The camera <NUM> includes an image sensor <NUM> and one or more other sensors (e.g., the sensors <NUM>), such as the microphone(s) <NUM> of <FIG>. Additionally, the camera <NUM> includes the device <NUM>, which is configured to identify sound events based on audio data samples and to update one or more of the SEC models <NUM>. In a particular aspect, the camera <NUM> is configured to perform one or more actions in response to a recognized sound event. For example, the camera <NUM> may cause the image sensor <NUM> to capture an image in response to the device <NUM> detecting a particular sound event in the audio data samples from the sensors <NUM>.

In the example illustrated in <FIG>, the device <NUM> includes the SEC models <NUM>, the SEC engine <NUM>, the drift detector <NUM>, the scene detector <NUM>, and the model updater <NUM>. In other implementations, the device <NUM>, when installed in or used in the camera <NUM>, omits the model updater <NUM>. To illustrate, the model update data <NUM> of <FIG> may be sent to the remote computing device <NUM>, and the remote computing device <NUM> may update one of the SEC models <NUM> based on the model update data <NUM>. In such implementations, the updated SEC model <NUM> can be downloaded to the camera <NUM> for use by the SEC engine <NUM>. In some implementations, the device <NUM>, when installed in or used in the camera <NUM>, further includes the model checker <NUM> of <FIG>.

<FIG> illustrates a mobile device <NUM> that incorporates aspects of the device <NUM> of <FIG>. In <FIG>, the mobile device <NUM> includes or is coupled to the device <NUM> of <FIG>. The mobile device <NUM> includes a phone or tablet, as illustrative, non-limiting examples. The mobile device <NUM> includes a display <NUM> and the sensors <NUM>, such as the microphone(s) <NUM> of <FIG>, the camera(s) <NUM> of <FIG>, the position sensor(s) <NUM> of <FIG>, or the other sensor(s) <NUM> of <FIG>. During operation, the mobile device <NUM> may perform particular actions in response to the device <NUM> recognizing a particular sound event. For example, the actions can include sending commands to other devices, such as a thermostat, a home automation system, another mobile device, etc..

In the example illustrated in <FIG>, the device <NUM> includes the SEC models <NUM>, the SEC engine <NUM>, the drift detector <NUM>, the scene detector <NUM>, and the model updater <NUM>. In other implementations, the device <NUM>, when installed in or used in the mobile device <NUM>, omits the model updater <NUM>. To illustrate, the model update data <NUM> of <FIG> may be sent to the remote computing device <NUM>, and the remote computing device <NUM> may update one of the SEC models <NUM> based on the model update data <NUM>. In such implementations, the updated SEC model <NUM> can be downloaded to the mobile device <NUM> for use by the SEC engine <NUM>. In some implementations, the device <NUM>, when installed in or used in the mobile device <NUM>, further includes the model checker <NUM> of <FIG>.

<FIG> illustrates an aerial device <NUM> that incorporates aspects of the device <NUM> of <FIG>. In <FIG>, the aerial device <NUM> includes or is coupled to the device <NUM> of <FIG>. The aerial device <NUM> is a manned, unmanned, or remotely piloted aerial device (e.g., a package delivery drone). The aerial device <NUM> includes a control system <NUM> and the sensors <NUM>, such as the microphone(s) <NUM> of <FIG>, the camera(s) <NUM> of <FIG>, the position sensor(s) <NUM> of <FIG>, or the other sensor(s) <NUM> of <FIG>. The control system <NUM> controls various operations of the aerial device <NUM>, such as cargo release, sensor activation, take-off, navigation, landing, or combinations thereof. For example, the control system <NUM> may control flight of the aerial device <NUM> between specified points and deployment of cargo at a particular location. In a particular aspect, the control system <NUM> performs one or more action responsive to detection of a particular sound event by the device <NUM>. To illustrate, the control system <NUM> may initiate a safe landing protocol in response to the device <NUM> detecting an aircraft engine.

In the example illustrated in <FIG>, the device <NUM> includes the SEC models <NUM>, the SEC engine <NUM>, the drift detector <NUM>, the scene detector <NUM>, and the model updater <NUM>. In other implementations, the device <NUM>, when installed in or used in the aerial device <NUM>, omits the model updater <NUM>. To illustrate, the model update data <NUM> of <FIG> may be sent to the remote computing device <NUM>, and the remote computing device <NUM> may update one of the SEC models <NUM> based on the model update data <NUM>. In such implementations, the updated SEC model <NUM> can be downloaded to the aerial device <NUM> for use by the SEC engine <NUM>. In some implementations, the device <NUM>, when installed in or used in the aerial device <NUM>, further includes the model checker <NUM> of <FIG>.

<FIG> illustrates a headset <NUM> that incorporates aspects of the device <NUM> of <FIG>. In <FIG>, the headset <NUM> includes or is coupled to the device <NUM> of <FIG>. The headset <NUM> includes the microphone(s) <NUM> of <FIG> positioned to primarily capture speech of a user. The headset <NUM> may also include one or more additional microphone positioned to primarily capture environmental sounds (e.g., for noise canceling operations) and one or more of the sensors <NUM>, such as the camera(s) <NUM>, the position sensor(s) <NUM>, or the other sensor(s) <NUM> of <FIG>. In a particular aspect, the headset <NUM> performs one or more actions responsive to detection of a particular sound event by the device <NUM>. To illustrate, the headset <NUM> may activate a noise cancellation feature in response to the device <NUM> detecting a gunshot. The headset <NUM> may also update one or more of the SEC models <NUM>.

<FIG> illustrates an appliance <NUM> that incorporates aspects of the device <NUM> of <FIG>. In <FIG>, the appliance <NUM> is a lamp; however, in other implementations, the appliance <NUM> includes another Internet-of-Things appliance, such as a refrigerator, a coffee maker, an oven, another household appliance, etc. The appliance <NUM> includes or is coupled to the device <NUM> of <FIG>. The appliance <NUM> includes the sensors <NUM>, such as the microphone(s) <NUM> of <FIG>, the camera(s) <NUM> of <FIG>, the position sensor(s) <NUM> of <FIG>, or the other sensor(s) <NUM> of <FIG>. In a particular aspect, the appliance <NUM> performs one or more actions responsive to detection of a particular sound event by the device <NUM>. To illustrate, the appliance <NUM> may activate a light in response to the device <NUM> detecting a door closing. The appliance <NUM> may also update one or more of the SEC models <NUM>.

In the example illustrated in <FIG>, the device <NUM> includes the SEC models <NUM>, the SEC engine <NUM>, the drift detector <NUM>, the scene detector <NUM>, and the model updater <NUM>. In other implementations, the device <NUM>, when installed in or used in the appliance <NUM>, omits the model updater <NUM>. To illustrate, the model update data <NUM> of <FIG> may be sent to the remote computing device <NUM>, and the remote computing device <NUM> may update one of the SEC models <NUM> based on the model update data <NUM>. In such implementations, the updated SEC model <NUM> can be downloaded to the appliance <NUM> for use by the SEC engine <NUM>. In some implementations, the device <NUM>, when installed in or used in the appliance <NUM>, further includes the model checker <NUM> of <FIG>.

<FIG> is a flow chart illustrating an example of a method <NUM> of operation of the device <NUM> of <FIG>. The method <NUM> can be initiated, controlled, or performed by the device <NUM>. For example, the processor(s) <NUM> or <NUM> of <FIG> can execute instructions <NUM> from the memory <NUM> to cause the drift detector <NUM> to generate the model update data <NUM>.

In block <NUM>, the method <NUM> includes providing audio data samples as input to a sound event classification model. For example, the SEC engine <NUM> of <FIG> (or the processor(s) <NUM>, <NUM> of <FIG> executing instructions <NUM> corresponding to the SEC engine <NUM>) may provide the audio data samples <NUM> as input to the SEC model <NUM>. In some implementations, the method <NUM> also includes capturing audio data corresponding to the audio data samples. For example, the microphone(s) <NUM> of <FIG> may generate the audio signals <NUM> based on the sound <NUM> detected by the microphone(s) <NUM>. Further, in some implementations, the method <NUM> includes selecting the sound event classification model from among a plurality of sound event classification models stored at a memory. For example, the SEC engine <NUM> of <FIG> (or the processor(s) <NUM>, <NUM> of <FIG> executing instructions <NUM> corresponding to the SEC engine <NUM>) may select the SEC model <NUM> from among the available SEC models <NUM> based on sensor data associated with the audio data samples, based on input identifying an audio scene or the SEC model <NUM>, based on when the audio data samples are received, based on settings data, or based on a combination thereof.

In block <NUM>, the method <NUM> includes determining, based on an output of the sound event classification model responsive to the audio data samples, whether a sound class of the audio data samples was recognized by the sound event classification model. For example, the SEC engine <NUM> of <FIG> (or the processor(s) <NUM>, <NUM> of <FIG> executing instructions <NUM> corresponding to the SEC engine <NUM>) may determine whether a sound class <NUM> of the audio data samples <NUM> was recognized by the SEC model <NUM>. To illustrate, the SEC model <NUM> may generate a confidence metric associated with each sound class that the SEC model <NUM> is trained to recognize, and a determination of whether the sound class was recognized by the SEC model may be based on the values of the confidence metric(s). In a particular aspect, based on a determination that the sound class <NUM> was recognized by the SEC model <NUM>, the SEC engine <NUM> generates the output <NUM> indicating the sound class <NUM> associated with the audio data samples <NUM>.

In block <NUM>, the method <NUM> includes based on a determination that the sound class was not recognized, determining whether the sound event classification model corresponds to an audio scene associated with the audio data samples. For example, the drift detector <NUM> of <FIG> (or the processor(s) <NUM>, <NUM> of <FIG> executing instructions <NUM> corresponding to the drift detector <NUM>) may determining whether the SEC model <NUM> corresponds to the audio scene <NUM> associated with the audio data samples <NUM>. To illustrate, the scene detector <NUM> may determine the audio scene <NUM> based on input received via that input device <NUM>, based on sensor data from the sensor(s) <NUM>, based on a timestamp associated with the audio data samples <NUM> as indicated by the clock <NUM>, based on the settings data <NUM>, or based on a combination thereof. In implementations in which the SEC model <NUM> is selected from among the available SEC models, the determination of the audio scene <NUM> may be based on different information than information used to select the SEC model <NUM>.

In block <NUM>, the method <NUM> includes, based on a determination that the sound event classification model corresponds to the audio scene associated with the audio data samples, storing model update data based on the audio data samples. For example, the drift detector <NUM> of <FIG> (or the processor(s) <NUM>, <NUM> of <FIG> executing instructions <NUM> corresponding to the drift detector <NUM>) may store the model update data <NUM> based on the audio data samples <NUM>. In a particular aspect, if the drift detector <NUM> determines that the SEC model <NUM> corresponds to the audio scene <NUM> associated with the audio data samples <NUM>, the drift detector <NUM> stores the drift data <NUM> as the model update data <NUM>, and if the drift detector <NUM> determines that the SEC model <NUM> does not corresponds to the audio scene <NUM> associated with the audio data samples <NUM>, the drift detector <NUM> stores the unknown data <NUM> as the model update data <NUM>.

The method <NUM> may also include updating the SEC model based on the model update data. For example, the model updater <NUM> of <FIG> (or the processor(s) <NUM>, <NUM> of <FIG> executing instructions <NUM> corresponding to the model updater <NUM>) may update the SEC model <NUM> based on the model update data <NUM> as described with reference to <FIG>, as described with reference to <FIG>, or both.

In a particular aspect, the method <NUM> includes after storing the model update data, determining whether a threshold quantity of model update data has been accumulated. For example, the model updater <NUM> may determine when the model update data <NUM> of <FIG> includes sufficient data (e.g., a threshold quantity of model update data <NUM> associated with a particular SEC model and a particular sound class) to initiate update training of the SEC model <NUM>. The method <NUM> may also include, based on a determination that the threshold quantity of model update data has been accumulated, initiating an automatic update of the sound event classification model using accumulated model update data. For example, the model updater <NUM> may initiate the model update without input from the user. In a particular implementation, the automatic update fine-tunes the SEC model <NUM> to generate the updated SEC model <NUM>. For example, before the automatic update, the SEC model <NUM> was trained to recognize multiple variants of a particular sound class, and the automatic update modifies the SEC model <NUM> to enable the SEC model <NUM> to recognize an additional variant of the particular sound class as corresponding to the particular sound class.

Due to the manner in which training occurs, an SEC model is generally a closed-set. That is, the number and type of sound classes that the SEC model can recognize is fixed and limited during the training. After training, an SEC model typically has a static relationship between the input and output. This static relationship between input and output means that the mapping learned during training is valid in the future (e.g., when evaluating new data), and that the relationships between input and output data do not change. However, it is difficult to collect an exhaustive set of training samples for each sound class, and it is difficult to properly annotate all of the available training data to train a comprehensive and sophisticated SEC model.

In contrast, during use SEC models face an open-set problem. For example, during use, the SEC model may be provided data samples associated with both known and unknown sound events. Additionally, the distribution of sounds or sound features in each sound class that the SEC model is trained to recognize can change over time or may not be comprehensively represented in the available training data. For example, for traffic sounds, differences in sounds based on locations, times, busy or non-busy intersection, etc. may not be explicitly captured in the training data for a traffic sound class. For these and other reasons, there can be discrepancies between the training data used to train an SEC model and the dataset that the SEC model is provided during use. Such discrepancies (e.g., dataset shift or drift) depends on various factors, such as location, time, device that is capturing the sound signal, etc. Dataset shift can lead to poor prediction results from the SEC model. The disclosed systems and methods overcome these and other problems by adapting an SED model to detect such shift data with little or no supervision. Additionally, in some aspects, an SEC model can be updated to recognize new sound classes without forgetting the previous trained sound classes.

In a particular aspect, no training of the SEC models is performed while the system is operating in an inference mode. Rather, during operation in the inference mode, existing knowledge, in the form of one or more previously trained SEC models, is used to analyze detected sounds. More than one SEC model can be used to analyze the sound. For example, an ensemble of SEC models can be used during operation in the inference mode. A particular SEC can be selected from a set of available SEC models based on detection of a trigger condition. To illustrate, a particular SEC model will be used, as the active SEC model, which may also be referred to as the "source SEC model", whenever a certain trigger (or triggers) is activated. The trigger(s) may be based on locations, sounds, camera information, other sensor data, user input, etc. For example, a particular SEC model may be trained to recognized sound events related to crowded areas, such as theme parks, outdoor shopping malls, public squares, etc. In this example, the particular SEC model may be used as the active SEC model when global positioning data indicates that a device capturing sound is at any of these locations. In this example, the trigger is based on the location of the device capturing sound, and the active SEC model is selected and loaded (e.g., in addition to or in place of a previous active SEC model) when the device is detected to be in the location.

In conjunction with the described implementations, an apparatus includes means for providing audio data samples to a sound event classification model. For example, the means providing audio data samples to a sound event classification model include the device <NUM>, the instructions <NUM>, the processor <NUM>, the processor(s) <NUM>, the SEC engine <NUM>, the feature extractor <NUM>, the microphone(s) <NUM>, the CODEC <NUM>, one or more other circuits or components configured to provide audio data samples to a sound event classification model, or any combination thereof.

The apparatus also includes means for determining, based on an output of the sound classification model, whether a sound class of the audio data samples was recognized by the sound event classification model. For example, the means for determining whether the sound class of the audio data samples was recognized by the sound event classification model includes the device <NUM>, the instructions <NUM>, the processor <NUM>, the processor(s) <NUM>, the SEC engine <NUM>, one or more other circuits or components configured to determine whether the sound class of the audio data samples was recognized by the sound event classification model, or any combination thereof.

The apparatus also includes means for determining, responsive to a determination that the sound class was not recognized, whether the sound event classification model corresponds to an audio scene associated with the audio data samples. For example, the means for determining whether the sound event classification model corresponds to the audio scene associated with the audio data samples includes the device <NUM>, the instructions <NUM>, the processor <NUM>, the processor(s) <NUM>, the drift detector <NUM>, the scene detector <NUM>, one or more other circuits or components configured to determine whether the sound event classification model corresponds to the audio scene associated with the audio data samples, or any combination thereof.

The apparatus also includes means for storing, responsive to a determination that the sound event classification model corresponds to the audio scene associated with the audio data samples, model update data based on the audio data samples. For example, the means for storing the model update data includes the remote computing device <NUM>, the device <NUM>, the instructions <NUM>, the processor <NUM>, the processor(s) <NUM>, the drift detector <NUM>, the memory <NUM>, one or more other circuits or components configured to store model update data, or any combination thereof.

In some implementations, the apparatus includes means for selecting the sound event classification model from among a plurality of sound event classification models based on a selection criterion. For example, the means for selecting the sound event classification model includes the device <NUM>, the instructions <NUM>, the processor <NUM>, the processor(s) <NUM>, the SEC engine <NUM>, one or more other circuits or components configured to select the sound event classification model, or any combination thereof.

In some implementations, the apparatus includes means for updating the sound event classification model based on the model update data. For example, the means for updating the sound event classification model based on the model update data includes the remote computing device <NUM>, the device <NUM>, the instructions <NUM>, the processor <NUM>, the processor(s) <NUM>, the model updater <NUM>, the model checker <NUM>, one or more other circuits or components configured to update the sound event classification model, or any combination thereof.

Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or combinations of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, such implementation decisions are not to be interpreted as causing a departure from the scope of the present disclosure.

Claim 1:
A device comprising:
one or more processors configured to:
provide (<NUM>) audio data samples (<NUM>) to a sound event classification model (<NUM>);
determine (<NUM>), based on an output of the sound event classification model responsive to the audio data samples, whether a sound class (<NUM>) of the audio data samples was recognized by the sound event classification model (<NUM>);
based on a determination that the sound class (<NUM>) was not recognized, determine (<NUM>) whether the sound event classification model (<NUM>) is trained to recognize sound events in an audio scene associated with the audio data samples; and
based on a determination that the sound event classification model (<NUM>) is trained to recognize sound events in the audio scene associated with the audio data samples and that the sound class (<NUM>) was not recognized, store (<NUM>) model update data (<NUM>) based on the audio data samples for updating the sound event classification model (<NUM>).