Patent ID: 12198057

V. DETAILED DESCRIPTION

Systems to perform operations such as SEC, noise reduction, ASR, NLP, etc., can use models that are trained to provide broad applicability but that are difficult or expensive to update. Although a user's experience can be improved by updating the models to improve accuracy for environments that the user, or user's device, is typically exposed to, re-training such models can be time consuming and require large amounts of data. In addition, as the user encounters an increasing number of environments and situations in everyday life, updating the models to adapt to the new environments and situations can cause the models to consume an ever-increasing amount of memory.

The disclosed systems and methods use context-aware model selection to select from among multiple models based on the context (e.g., the acoustic environment) in which the selected model is to be used, according to some aspects. For example, a proper acoustic model can be selected for a particular acoustic environment to deliver a better user experience in most voice user interface applications. As an illustration, in order to provide a user of a hearing aid device with a pleasant listening experience, the surrounding noise should be properly estimated. An appropriate model can be selected based on the particular location of the user, such as a particular street corner, building, or restaurant, or based on a type of environment of the user, such as in a vehicle, in a park, or in a metro station, as illustrative, non-limiting examples. A particular context can be identified based on one or more of a variety of techniques, such as via location data (e.g., from a global positioning system (GPS) system), activity detection data, camera recognition, audio classification, user input, one or more other techniques, or any combination thereof.

According to some aspects, the acoustic characteristics of noisy areas such as shopping malls, restaurants, stadiums, etc., may be known, and their models are made publicly available for users. As a user walks into any of these locations, the access permission to the model of that location could be granted to the user. After the user leaves, the model may be removed or “pruned” from the user's device. In another aspect, as a user travels from place to place (e.g., driving, walking, or via public transit), the user's device may swap models so that the most appropriate model can be used for each location or setting that the user encounters. In some aspects, a library of available models is provided to enable searching and download of an appropriate model. Some of the models may be uploaded by other users, such as updated models that have been trained based on exposure of the other users' devices to various environments, and may be publicly available (or available with specific access permissions) as part of a crowdsourced, context-aware model library.

According to a particular aspect, models can be combined or “generalized” by grouping relevant categories of classes into one model and creating an ensemble of various source models. For example, in SEC applications, relevant sound classes can be grouped based on location, sound type, or one or more other characteristics. To illustrate, one model can include a group of sound classes to be generally representative of crowed areas, such as public squares, shopping malls, subways, etc., while another model can include a group of sound classes related to home activities. These generalized models enable generally improved performance based on the broad category of the generalized model, while using reduced memory as compared to the amount of memory used for multiple specific models to accommodate each specific environment or activity. In addition, if specific models are unavailable due to privacy issues or other accessibility limitations, the more general models may be used instead. For example if a user arrives a busy restaurant that does not have a specific public model for that restaurant available for use, a general purpose model for crowded areas, or a general model for crowded restaurants, can be downloaded to the user's device and used instead.

By changing models based on the context of a device, systems that use such models can perform with higher accuracy as compared to using a single model for all contexts. Further, changing models enables such systems to perform with increased accuracy without incurring the power consumption, memory requirements, and processing resource usage associated with re-training an existing model from scratch at the device for a particular context. Use of generalized context-based models enables improved performance of systems as compared to using default models and also enables reduced bandwidth, memory, and processing resource usage as compared to downloading and switching between multiple high-accuracy, context-specific models. In addition, operation of such systems using context-based models enables improved operation of the device itself, such as by enabling faster convergence when performing an iterative or dynamic process (e.g., in a noise cancellation technique) due to using a higher-accuracy model that is specific to the particular context.

Particular aspects of the present disclosure are described below with reference to the drawings. In the description, common features are designated by common reference numbers. As used herein, various terminology is used for the purpose of describing particular implementations only and is not intended to be limiting of implementations. For example, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, some features described herein are singular in some implementations and plural in other implementations. To illustrate,FIG.1depicts a device101that includes one or more processor (“processor(s)110inFIG.1), which indicates that in some implementations the device101includes a single processor110and in other implementations the device101includes multiple processors110. For ease of reference herein, such features are generally introduced as “one or more” features and are subsequently referred to in the singular or optional plural (generally indicated by terms ending in “(s)”) unless aspects related to multiple of the features are being described.

The terms “comprise,” “comprises,” and “comprising” are used herein interchangeably with “include,” “includes,” or “including.” Additionally, the term “wherein” is used interchangeably with “where.” As used herein, “exemplary” indicates an example, an implementation, and/or an aspect, and should not be construed as limiting or as indicating a preference or a preferred implementation. As used herein, an ordinal term (e.g., “first,” “second,” “third,” etc.) used to modify an element, such as a structure, a component, an operation, etc., does not by itself indicate any priority or order of the element with respect to another element, but rather merely distinguishes the element from another element having a same name (but for use of the ordinal term). As used herein, the term “set” refers to one or more of a particular element, and the term “plurality” refers to multiple (e.g., two or more) of a particular element.

As used herein, “coupled” may include “communicatively coupled,” “electrically coupled,” or “physically coupled,” and may also (or alternatively) include any combinations thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., a wired network, a wireless network, or a combination thereof), etc. Two devices (or components) that are electrically coupled may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may send and receive electrical signals (digital signals or analog signals) directly or indirectly, such as via one or more wires, buses, networks, etc. As used herein, “directly coupled” refers to two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.

In the present disclosure, terms such as “determining,” “calculating,” “estimating,” “shifting,” “adjusting,” etc. may be used to describe how one or more operations are performed. It should be noted that such terms are not to be construed as limiting and other techniques may be utilized to perform similar operations. Additionally, as referred to herein, “generating,” “calculating,” “estimating,” “using,” “selecting,” “accessing,” and “determining” may be used interchangeably. For example, “generating,” “calculating,” “estimating,” or “determining” a parameter (or a signal) may refer to actively generating, estimating, calculating, or determining the parameter (or the signal) or may refer to using, selecting, or accessing the parameter (or signal) that is already generated, such as by another component or device.

FIG.1is a block diagram of an example of a system that includes a device100that is configured to perform context-based model selection. The device100includes one or more processors110coupled to a memory108. The memory108includes L available models114(L is an integer greater than 1) that may be selected by the one or more processors110, illustrated as a first model116and one or more additional models including a Lth model118.

The one or more processors110are configured to receive sensor data138from one or more sensor devices134and to determine a context142of the device100based on the sensor data138. Although the sensor devices134are illustrated as coupled to the device100, in other implementations one or more (or all) of the sensor devices134are integrated with or included in the device100.

The one or more sensor devices134include one or more microphones104coupled to the one or more processors110, and the sensor data138includes audio data105from the one or more microphones104. In an example, the audio data105corresponds to an audio scene, and the context142is at least partially based on audio scene. To illustrate, based on the amount and type of noise detected in the audio data, as well as acoustic characteristics such as echoes and absorption, the audio scene can indicate that the device100is in a confined noisy space, a large enclosed space, a large outdoor space, a traveling vehicle, etc.

The one or more sensor devices134include a location sensor152coupled to the one or more processors110, and the sensor data138includes location data153from the location sensor152, such as global positioning sensor that provides global position data for the device100. In an example, the location data153is indicative of a location of the device100, and the context142is at least partially based on the location.

The one or more sensor devices134include a camera150coupled to the one or more processors110, and the sensor data138includes image data151(e.g., still image data, video data, or both) from the camera150. In an example, the image data151corresponds to a visual scene, and the context142is at least partially based on the visual scene.

The one or more sensor devices134includes an activity detector154coupled to the one or more processors110, and the sensor data138includes motion data, such as the activity data155from the activity detector154. In an example, the motion data corresponds to motion of the device100, and the context142is at least partially based on the motion of the device100.

The one or more sensor devices134may also include one or more other sensors156that provide additional sensor data157to the one or more processors110for use in determining the context142. The other sensor(s)156can include, for example, an orientation sensor, a magnetometer, a light sensor, a contact sensor, a temperature sensor, or any other sensor that is coupled to or included within the device100and that can be used to generate sensor data157useful for determining the context142associated with the device100at a particular time. As another example, the other sensor(s)156can include a wireless network detector that can be used to determine the context142, such as by detecting when the device100is in the vicinity of recognized wireless network locations (e.g., by detecting a home or business WiFi network, or a Bluetooth network associated with a friend of family member of a user of the device100).

The one or more processors110include a context detector140, a model selector190, and a model-based application192. In a particular implementation, the context detector140is a neural network that is trained to determine the context142based on the sensor data138. In other implementations, the context detector140is a classifier that trained using a different machine-learning technique. For example, the context detector140may include or correspond to a decision tree, a random forest, a support vector machine, or another classifier that is trained to generate output indicating the context142based on the sensor data138. In still other implementations, the context detector140uses heuristics to determine the context142based on the sensor data138. In yet other implementations, the context detector140uses a combination of artificial intelligence and heuristics to determine the context142based on the sensor data138. For example, the sensor data138may include image data, video data, or both, and the context detector140may include an image recognition model that is trained using a machine-learning technique to detect particular objects, motions, backgrounds, or other image or video information. In this example, output of the image recognition model may be evaluated via one or more heuristics to determine the context142.

The model selector190is configured to select a model112based on the context142. In some implementations, the model112is selected from among the multiple available models114stored at the memory108. In some implementations, the model112is selected from a model library162that is accessible via a network160, such as a cloud-based library of models available for searching and download. An example of the model library162is described in further detail with respect toFIG.2.

As used herein, “downloading” and “uploading” a model includes transferring of data (e.g., compressed data) corresponding to the model over a wired link, over a wireless link, or a combination thereof. For example, wireless local area networks (“WLANs”) may be used in place of, or in addition to, wired networks. Wireless technologies, such as Bluetooth® (“Bluetooth”) and Wireless Fidelity “Wi-Fi” or variants of Wi-Fi (e.g. Wi-Fi Direct), enable high speed communications between mobile electronic devices (e.g., cellular phones, watches, headphones, remote controls, etc.) that are within relatively short distances of one another (e.g., 100 to 200 meters or less depending on the specific wireless technology). Wi-Fi is often used to connect and exchange information between a device with an access point, (e.g. a router) and devices that are Wi-Fi enabled. Examples of such devices are smart televisions, laptops, thermostats, personal assistant devices, home automation devices, wireless speakers and other similar devices. Similarly, Bluetooth is also used to couple devices together. Example of such are mobile phones, computers, digital cameras, wireless headsets, keyboards, mice or other input peripherals, and similar devices.

Devices (e.g., those previously mentioned) may have both Bluetooth and Wi-Fi capabilities, or other wireless means to communicate with each other. Inter-networked devices may have wireless means to communicate with each other and may also be connected based on different cellular communication systems, such as, a Long Term Evolution (LTE) system, a Code Division Multiple Access (CDMA) system, a Global System for Mobile Communications (GSM) system, a wireless local area network (WLAN) system, or some other wireless system. A CDMA system may implement Wideband CDMA (WCDMA), CDMA 1X, Evolution-Data Optimized (EVDO), Time Division Synchronous CDMA (TD-SCDMA), or some other version of CDMA. As used herein, “wireless” refers to one or more of the above-listed technologies, one or more other technologies that enable transfer of information other than via wires, or a combination thereof.

The model-based application192is configured to process an input signal106using the selected model112to generate a context-specific output122. Although a single model-based application192is illustrated, in other implementations the one or more processors110can execute multiple model-based applications192using various models112selected based on the context142to perform various operations.

For example, in some implementations, the model112includes a sound event detection model, and the input signal106includes an audio signal, such as the audio data105from the microphone104, audio data retrieved from an audio file at the memory108, an audio signal received via wireless transmission, such as a phone call or streaming audio session, or any combination thereof. The model-based application192is configured to process the input signal106using the sound event detection model to generate the context-specific output122that includes a classification of a sound event in the audio signal.

In some implementations, the model112includes a noise reduction model, and the input signal106includes an audio signal such as the audio data105from the microphone104, audio data retrieved from an audio file at the memory108, an audio signal received via wireless transmission, such as a phone call or streaming audio session, or any combination thereof. The model-based application192is configured to process the input signal106using the noise reduction model to generate the context-specific output122that includes a noise reduced audio signal based on the audio signal.

In some implementations, the model112includes an automatic speech recognition (ASR) model, and the input signal106includes an audio signal such as the audio data105from the microphone104, audio data retrieved from an audio file at the memory108, an audio signal received via wireless transmission, such as a phone call or streaming audio session, or any combination thereof. The model-based application192is configured to process the input signal106using the ASR model to generate the context-specific output122that includes text data representative of speech in the audio signal.

In some implementations, the model112includes a natural language processing (NLP) model, and the input signal106includes text data, such as text data generated by an NLP model, user keyboard input, text message, etc. The model-based application192is configured to process the input signal106using the NLP model to generate the context-specific output122that includes NLP output data based on the text data.

In some implementations, the model112is associated with an automatic adjustment of a device operating mode. For example, the model112can map or otherwise associate user inputs (e.g., voice commands, gestures, touchscreen selections, etc.) to operating mode adjustments based on the context142. To illustrate, when the model112is selected based on the context142corresponding to a public area, the model112can cause the model-based application192to map a user command “play music” to a playback operation at the user's earphones, and the context-specific output122can include a signal to adjust the device operating mode to initiate an audio playback operation and to route the output audio signal to the user's earphones. However, when the model112is selected based on the context142corresponding to the user's house, the model112can cause the model-based application192to map the “play music” command to a playback operation at the user's home entertainment system, and the context-specific output122can include a signal to adjust the device operating mode to initiate the audio playback and to route the output audio signal to loudspeakers of the home entertainment system.

In some implementations, the model selector190is configured to prune the model112in response to detection of a change in the context142, such as when the change in the context142results in the model112no longer being appropriate for the changed context142or less appropriate for the changed context142than another available model. As used herein, “pruning” a model includes removing the model from use, such as by replacing the model with another model at the model-based application192, permanently deleting the model from the memory108, deleting the model from the model-based application192, marking the model as unused, otherwise rendering the model inaccessible (e.g., by deleting access permission for the model as described further with reference toFIG.2), or a combination thereof. In some implementations, pruning a model can reduce memory usage, processor cycles, one or more other processing or system resources, or a combination thereof, and may therefore improve functioning (e.g., increased speed, reduced power consumption) of the one or more processors110or of the device100overall. In implementations in which a model has been updated at the device100, pruning the model can include preserving updates to the model. For example, if a sound model has been created or modified during use at the device100to identify new sound classes, such as described with reference toFIGS.3-5, the new sound classes may be preserved, such as in the memory108or uploaded to the model library162, as illustrative, non-limiting examples.

During operation, the context detector140monitors the sensor data138and may update the context142based on detecting changes in the sensor data138. In response to a change of the context142, the model selector190accesses the available models114in the memory108, in the model library162, or both, to select one or more models that may be more appropriate for the updated context142than the current model112. For example, the model selector190may send a query to the model library162that identifies one or more aspects of the context142, such as a geographic location, a venue name, an acoustic scene or visual scene description, one or more other aspects, or a combination thereof. The model selector190may receive results of the query, select a particular model based on the query results, and cause the device100to download the particular model from the model library162and store the downloaded model in the memory108. In some implementations, in response to detecting that the context142has changed, the currently selected model112is pruned (e.g., the model selector190removes the model112from use by the model-based application192) and replaced with the downloaded model from the memory108.

By changing models based on the context142, the device100may execute the model-based application192with higher accuracy as compared to using a single default model. As a result, a user experience with the device100may be improved.

In some implementations, the device100is also configured to modify or customize the selected model112to further improve accuracy, such as in response to detection of a new sound event or variations of an existing classification that may not be accurately identified by the model112. Examples of modifying the model112based on data acquired at the device100(e.g., the sensor data138) are described further with regard toFIGS.3-5. After modifying the model112, the device100may upload the modified model112to the model library162to be available to other devices. In some implementations, the model112includes a trained model uploaded to the library162from another user device. Thus, the device100may utilize and contribute to a crowdsourced library of models in a distributed context-aware system.

The device100may include, correspond to, or be included within a voice activated device, an audio device, a wireless speaker and voice activated device, a portable electronic device, a car, a vehicle, a computing device, a communication device, an internet-of-things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, a hearing aid device, a smart speaker, a mobile computing device, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a gaming console, an appliance, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, or any combination thereof. In a particular aspect, the one or more processors110, the memory108, or a combination thereof, are included in an integrated circuit. Various implementations that include aspects of the device100are described further with reference toFIGS.7-16.

Although the device100is described as storing the available models114in the memory108and accessing the model library162, in other implementations the device100may not store models at the memory108and may instead retrieve models from the model library162in response to detecting changes in the context142. In other implementations, the device100operates without accessing the model library162and instead selects from among the available models114at the memory108. In some implementations, the available models114at the memory108represent a locally stored portion of a distributed context-aware system and may be accessible to other devices as part of the model library162, such as in a peer-to-peer model sharing configuration.

Although the sensors134are illustrated as including the microphone(s)104, the camera(s)150, the location sensor(s)152, the activity detector154, and the other sensor(s)156, in other implementations one or more of the microphone(s)104, the camera(s)150, the location sensor(s)152, the activity detector154, or the other sensor(s)156are omitted. In an illustrative example, the context detector140operates using only the audio data105, the image data151, the location data153, the activity data155, or the other sensor data157. In another illustrative example, the context detector140operates using two of the audio data105, the image data151, the location data153, the activity data155, and the other sensor data157, using three of the audio data105, the image data151, the location data153, the activity data155, and the other sensor data157, or using four of the audio data105, the image data151, the location data153, the activity data155, and the other sensor data157.

Although the context detector140, the model selector190, and the model-based application192are described as separate components, in other implementations, the context detector140and the model selector190are combined into a single component, the model selector190and the model-based application192are combined into a single component, or the context detector140, the model selector190, and the model-based application192are combined into a single component. In some implementations, each of the context detector140, the model selector190, and the model-based application192may be implemented via processor-executed instructions, dedicated hardware or circuitry, or a combination of both.

FIG.2illustrates a particular example of aspects of operation of the device100and the model library162. The device100is illustrated within a building202that includes a room204, a room206, and an elevator208, such as an office building. The device100is in wireless communication with the model library162.

The model library162includes various types of models, such as a representative sound event detection model220, a representative noise reduction model222, a representative ASR model224, a representative NLP model226, a representative operating mode adjustment model228, and various acoustic models250. Although a single one of each of the sound event detection model220, the noise reduction model222, the ASR model224, the NLP model226, and the operating mode adjustment model228is illustrated for clarity of illustration, it should be understood that the model library162can include multiple versions of each of the different types of models, such as models that have been trained for different contexts, different personalization, and different levels of generality, in a similar manner as described below for the acoustic models250.

The acoustic models250are illustrated in an arrangement in which models for general categories (e.g., generalized models) are depicted as roots of tree structures, and models for more specific contexts are depicted as branches or as leaves of the tree structures. For example, a “crowded areas” model252is a general category model with branches including “highway” model(s), “metro center” model(s), “suburbs” model(s), “theme parks” model(s), “shopping malls” model(s), and “public squares” model(s). Although not illustrated, each of the branch models may serve as a general category model for various more specific models. For example, the model library162may include acoustic models for multiple specific theme parks. If a user's device (e.g., the device100) is at a particular theme park, the device may request an acoustic model for that particular theme park. If no acoustic model is available for that particular theme park, the user's device may request a “theme parks” model that is applicable to theme parks generally but not specifically for any particular theme park. If a “theme parks” model is not available, the user's device may request a “crowded areas” model that is applicable to crowded areas generally but not specially to theme parks. Thus, the user's device can search for and select a most specific model that is available in the model library162for that device's particular context.

The acoustic models250also include a “confined spaces” model254that is a general category model with branches corresponding to an “office building” model262, a “house” model264, and a “vehicle” model266. The “office building” model262is a general category model for various locations within office buildings. More specific models for various locations within office buildings include a “lobby” model270, an “elevator” model272, and an “office” model274. The “house” model264is a general category model for various locations within houses. More specific models for locations within houses include a “kitchen” model, a “room” model, and a “garage” model. The “vehicle” model266is a general category model for various locations within a vehicle. More specific models for locations within vehicles include a “driver seat” model, a “passenger seat” model, and a “back seat” model.

It should be understood that the illustrated models are depicted for purposes of illustration and clarity of explanation. In other implementations, the model library162may have any number of models (e.g., hundreds, thousands, millions, etc.) arranged at any number of levels of generality, for any number of different contexts, for any number of different applications. It should also be understood that although the acoustic models250are organized according to a tree structure, in other implementations the model library162utilizes one or more other data structures or categorization techniques in place of, or in addition to, a tree structure.

As illustrated, the device100determines, based on the sensors134, that the context142of the device100is within the room204. The device100may determine whether the available acoustic models250of the model library162include an acoustic model that is specific to the particular acoustic environment associated with the context142and that is available to the device100(e.g., the model selector190). For example, the device100transmits data indicative of the acoustic environment210of the device100, such as location data, a name of the building202, a general description of the building202(e.g., “office building”), a general description of the room204(e.g., “office”), or any combination thereof. In a particular implementation, in response to the model library162having no acoustic model that is specific to the particular acoustic environment210and available to the one or more processors110, the device100(e.g., the model selector190) determines whether an acoustic model for a general category of the particular acoustic environment210is available.

As an illustrative example, the device100may send data indicative of the acoustic environment210by transmitting the location coordinates of the device100. If the model library162has an acoustic model that is specific to (e.g., matches) the location of the device100(e.g., an acoustic model that corresponds to a geo-fence of a region that contains the location of the device100), the device100downloads an acoustic model212corresponding to the location. Otherwise, in response to the model library216not having a model specific to the location coordinates (e.g., not specific to the building202), the device100may transmit additional data indicative of the acoustic environment210, such as an “office” descriptor of the room204. In response to a determination that the model library162includes the “office” model274as available to the device100, the device100downloads the “office” model274as the acoustic model212for use within the room204. In the event the “office” model274is not available, the device100may request the more general “office building” model262, or the even more general “confined spaces” model254. In some implementations, the model library162is configured to automatically locate and transmit to the device100the most specific model corresponding to the acoustic environment210, instead of the device100sending a series of requests for increasingly more generalized models until an appropriate model is located.

In some implementations, the device100may also receive one or more access permissions214that authorize the device100to access the acoustic model212. The access permissions214may enable embargoing of models that the device100is predicted to use. For example, embargoed models may be downloaded from the model library162to the memory108of the device100in advance of the predicted use. Downloading such models in advance may be scheduled based on available bandwidth (e.g., during periods of reduced network traffic) or to reduce the latency of accessing the models when the context142of the device100is detected to have changed. Each of the embargoed models remains inaccessible (e.g., encrypted) at the memory108until a corresponding permission214(e.g., an encryption key) for the model is received from the model library162or another permissions management system. For example, the device100may receive an access permission214for the model112at least partially based on the location of the device100matching the particular location associated with the model112(e.g., a particular location).

In a particular example, the device100transmits data indicating the acoustic environment210of the room204, receives the “office” model274(and any associated access permissions214) from the model library162, stores a copy of the “office” model274at the memory108, and uses the “office” model274at the model-based application197, such as to perform noise reduction. When the device100is moved from the room204to the elevator208within the building202, the device100detects the new context142and requests an acoustic model for the acoustic environment210corresponding to the elevator208. In response, the “elevator” model272is transmitted to the device100as the acoustic model212, and access permissions214for the “elevator” model272may also be transmitted. The device100replaces the “office” model274with the “elevator” model272at the model-based application192. In some implementations, the device100removes the “office” model274from the memory108, such as when available storage capacity at the memory108is restricted.

Upon exiting the elevator208and entering the room206, the device100may search the available models114at the memory108for the “office” model274. If the “office” model is not available at the memory108, the device100transmits data indicating the acoustic environment210of the room206, receives the “office” model274(and any associated access permissions214) from the model library162, stores a copy of the “office” model274at the memory108, and uses the “office” model274at the model-based application197. Thus, as the device100moves from one location to the next, models are switched out for more appropriate models for the changing context142of the device100. In some implementations, when the device100exits the building202, any stored models in the memory108that are specific to the building202may be deleted or archived to conserve storage space in the memory108or may be rendered inaccessible in response to the access permissions214imposing a geographical or other restriction on the use of the models.

In conjunction with various aspects described with reference toFIG.1andFIG.2, the device100includes the one or more processors110configured to select an acoustic model corresponding to a particular room, of a building, in which the device100is located, such as the acoustic model212corresponding to the room204of the building202, and process an input audio signal using the acoustic model212. For example, the input signal106can include the audio data105that is generated by the microphone(s)104and that is processed at the model-based application192to preform noise reduction.

In some implementations, the one or more processors110are configured to download the acoustic model212from the library162of acoustic models in response to a determination that the device100has entered the particular room204. In some implementations, the one or more processors110are further configured to remove the acoustic model212in response to the device100leaving the particular room204. As non-limiting examples, removing the acoustic model212can include replacing the acoustic model212with another model at the model-based application192, deleting the acoustic model212from the memory108, deleting the acoustic model212from the model-based application192, marking the acoustic model212as unused, or rendering the acoustic model212inaccessible (e.g., deleting the access permission214for the acoustic model212).

The one or more sensor devices134(also referred to as “sensors134”) coupled to the one or more processors110are configured to generate the sensor data138indicative of a location of the device100, and the one or more processors110are configured to select the acoustic model212based on the sensor data138. For example, the device100can include a modem, as described with reference toFIG.6, that is coupled to the one or more processors110and configured to receive the location data153indicative of a location of the device100, and the one or more processors110are configured to select the acoustic model212based on the location data153.

In some implementations, selection of models can be performed predictively based on the context142. For example, based on the sensor data138(e.g., activity detection, GPS analysis, camera recognition, audio classification, or a combination thereof), the one or more processors110may determine that a user of the device100is traveling to a new location (e.g., New York City) where assistive Internet-of-Things (IoT) devices associated with the user can exhibit improved performance with updated settings for the new location. As a result, appropriate source models (e.g., an acoustic model for traffic) can be retrieved from the memory108or from one or more model libraries (e.g., the model library162) and used. Upon leaving the new location, the source models for the new location are removed and the prior source models may be restored.

In conjunction with various aspects described with reference toFIG.1andFIG.2, in response to the device100entering a vehicle, such as described further with reference toFIG.7, the one or more processors110are configured to select a personalized acoustic model for a user of the device100from among multiple personalized acoustic models corresponding to the vehicle and process an input audio signal using the personalized acoustic model. For example, the device100may train or otherwise generate models specific to particular users of the device100, as described in further detail with reference toFIGS.3-5, and may access the personalized acoustic models from the model library162, the memory108, or both. To illustrate, the one or more processors110may be configured to download the personalized acoustic model from a library of acoustic models (e.g., the acoustic models250in the model library162) in response to a determination that the device100has entered the vehicle. In some implementations, the one or more processors110are further configured to remove the personalized acoustic model in response to the device100leaving the vehicle.

For example, the one or more processors110may be configured to determine the device100has entered the vehicle based on the sensor data138. To illustrate, the one or more processors110may to determine the device100has entered the vehicle (or exited the vehicle) based on the location data153.

In conjunction with various aspects described with reference toFIG.1andFIG.2, the one or more processors110of the device100are configured to download an acoustic model corresponding to a particular location in which the device100is located, process an input audio signal using the acoustic model, and remove the acoustic model in response to the device100exiting the location. In an illustrative example, the location corresponds to a particular restaurant, and the acoustic model is downloaded from a library of acoustic models (e.g., the acoustic models250in the model library162) in response to a determination that the device100has entered the particular restaurant.

In conjunction with various aspects described with reference toFIG.1andFIG.2, the one or more processors110of the device100are configured to select an acoustic model corresponding to a particular location, receive an access permission for the acoustic model at least partially based on a location of the device100matching the particular location, and process an input audio signal using the acoustic model.

In some implementations, the device100ofFIG.1andFIG.2is further configured to update one or more models, such as to personalize the models for particular users or to improve the accuracy of the models for environments the device100frequently encounters, as illustrative, non-limiting examples.FIGS.3-5depict illustrative examples in which the device100is configured to update models. AlthoughFIGS.3-5describe updating sound event classification models as a particular example, the described techniques are generally applicable to updating any type of model that may be used by the device100.

FIG.3is a block diagram of an example of components of the device100configured to generate sound identification data responsive to audio data samples310and configured to update a sound event classification model. The device100ofFIG.3includes one or more microphones304(e.g., the microphone(s)104) configured to generate audio signals306(e.g., the audio data105) based on sound302detected within an acoustic environment. The microphone(s)304are coupled to a feature extractor308that generates audio data samples310based on the audio signals306. For example, the audio data samples310may include an array or matrix of data elements, with each data element corresponding to a feature detected in the audio signals306. As a specific example, the audio data samples310can correspond to Mel spectrum features extracted from one second of the audio signals306. In this example, the audio data samples310can include a 128×128 element matrix of feature values. In other examples, other audio data sample configurations or sizes can be used.

The audio data samples310are provided to a sound event classification (SEC) engine320(e.g., the model-based application192). The SEC engine320is configured to perform inference operations based on one or more SEC models, such as an SEC model312. “Inference operations” refer to assigning the audio data samples310to a sound class, if the sound class of the audio data samples310is recognized by the SEC model312. For example, the SEC engine320may include or correspond to software that implements a machine-learning runtime environment, such as the Qualcomm Neural Processing SDK, which is available from Qualcomm Technologies, Inc. of San Diego, California, USA. In a particular aspect, the SEC model312is one of a plurality of SEC models (e.g., available SEC models314) that are available to the SEC engine320.

In a particular example, each of the available SEC models314(e.g., stored at the memory108or at the model library162) includes or corresponds to a neural network that is trained as a sound event classifier. To illustrate, the SEC model312(as well as each of the other available SEC models314) may include an input layer, one or more hidden layers, and an output layer. In this example, the input layer is configured to correspond to the array or matrix of values of the audio data samples310generated by the feature extractor308. To illustrate, if the audio data samples310include 15 data elements, the input layer may include 15 nodes (e.g., one per data element). The output layer is configured to correspond to the sound classes that the SEC model312is trained to recognize. The specific arrangement of the output layer can vary depending on information to be provided as output. As one example, the SEC model312may be trained to output an array that includes one bit per sound class, where the output layer performs “one hot encoding” such that all but one of the bits of the output array have a value of zero, and the bit corresponding to a detected sound class has a value of one. Other output schemes can be used to indicate, for example, a value of a confidence metric for each sound class, where the value of the confidence metric indicates a probability estimate that the audio data samples310correspond to the respective sound class. To illustrate, if the SEC model312is trained to recognize four sound classes, the SEC model312may generate output data that includes four values (one per sound class), and each value may indicate a probability estimate that the audio data samples310correspond to the respective sound class.

Each of the hidden layers includes a plurality of nodes, and each node is interconnected (via a link) with other nodes in the same layer or in a different layer. Each input link of a node is associated with a link weight. During operation, a node receives input values from other nodes that it is linked to, weights the input values based on corresponding link weights to determine a combined value, and subjects the combined value to an activation function to generate an output value of the node. The output value is provided to one or more other nodes via output links of the node. The nodes may also include bias values that are used to generate the combined value. The nodes can be linked in various arrangements and can include various other features (e.g., memory of prior values) to facilitate processing of particular data. In the case of audio data samples, convolutional neural networks (CNNs) may be used. To illustrate, one or more of the SEC models312may include three linked CNNs, and each CNN may include a two-dimensional (2D) convolution layer, a maxpooling layer, and a batch normalization layer. In other implementations, the hidden layers include a different number of CNNs or other layers. Training the neural network includes modifying the link weights to reduce an output error of the neural network.

During operation, the SEC engine320may provide the audio data samples310as input to a single SEC model (e.g., the SEC model312), to multiple selected SEC models (e.g., the SEC model312and a Kth SEC model318of the available SEC models314), or to each of the SEC models (e.g., to the SEC model312, a first SEC model316, the Kth SEC model318, and any other SEC models of the available SEC models314). For example, the SEC engine320(or another component of the device100) may select the SEC model312from among the available SEC models314based on, for example, user input, device settings associated with the device100, sensor data, a time when the audio data samples310are received, or other factors. In this example, the SEC engine320may select to use only the SEC model312or may select to use two or more of the available SEC models314. To illustrate, the device settings may indicate that the SEC model312and the first SEC model316are to be used during a particular time frame. In another example, the SEC engine320may provide the audio data samples310to each of the available SEC models314(e.g., sequentially or in parallel) to generate output from each. In a particular aspect, the SEC models are trained to recognize different sound classes, to recognize the same sound classes in different acoustic environments, or both. For example, the SEC model312may be configured to recognize a first set of sound classes and the first SEC model316may be configured to recognize a second set of sound classes, where the first set of sound classes is different from the second set of sound classes.

In a particular aspect, the SEC engine320determines, based on output of the SEC model312, whether the SEC model312recognized the sound class of the audio data samples310. If the SEC engine320provides the audio data samples310to multiple SEC models, the SEC engine320may determine, based on output of each of the SEC models, whether any of the SEC models recognized the sound class of the audio data samples310. If the SEC model312(or another of the available SEC models314) recognized the sound class of the audio data samples310, the SEC engine320generates an output324that indicates the sound class322of the audio data samples310. For example, the output324may be sent to a display to notify a user of detection of the sound class322associated with the sound302or may be sent to another device or another component of the device100and used to trigger an action (e.g., to send a command to activate lights in response to recognizing the sound of a door shutting).

If the SEC engine320determines that the SEC model312(and others of the available SEC models314that were provided the audio data samples310) did not recognize the sound class of the audio data samples310, the SEC engine320provides a trigger signal326to a drift detector328. For example, the SEC engine320may set a trigger flag in a memory of the device100. In some implementations, the SEC engine320may also provide other data to the drift detector328. To illustrate, if the SEC model312generates a value of a confidence metric for each sound class that the SEC model312is trained to recognize, one or more of the values of the confidence metric may be provided to the drift detector328. For example, if the SEC model312is trained to recognize three sound classes, the SEC engine320may provide a highest confidence value among three confidence values (one for each of the three sound classes) output by the SEC model312to the drift detector328.

In a particular aspect, the SEC engine320determines whether the SEC model312recognized the sound class of the audio data samples310based on a value of a confidence metric. In this particular aspect, a value of the confidence metric for a particular sound class indicates the probability that the audio data samples310are associated with the particular sound class. To illustrate, if the SEC model312is trained to recognize four sound classes, the SEC model312may generate as output an array that includes four values of the confidence metric, one for each sound class. In some implementations, the SEC engine320determines that the SEC model312recognized the sound class322of the audio data samples310if the value of the confidence metric for the sound class322is greater than a detection threshold. For example, the SEC engine320determines that the SEC model312recognized the sound class322of the audio data samples310if the value of the confidence metric for the sound class322is greater than 0.90 (e.g., 90% confidence), 0.95 (e.g., 95% confidence), or some other value of the detection threshold. In some implementations, the SEC engine320determines that the SEC model312did not recognize a sound class of the audio data samples310if the value of the confidence metric for each sound class that the SEC model312is trained to recognize is less than the detection threshold. For example, the SEC engine320determines that the SEC model312did not recognize the sound class322of the audio data samples310if each value of the confidence metric is less than 0.90 (e.g., 90% confidence), 0.95 (e.g., 95% confidence), or some other value of the detection threshold.

The drift detector328is configured to determine whether the SEC model312that was not able to recognize the sound class of the audio data samples310corresponds to an audio scene342associated with the audio data samples310. In the example illustrated inFIG.1, a scene detector340(e.g., the context detector140) is configured to receive scene data338(e.g., including a portion of the sensor data138) and to use the scene data338to determine the audio scene342(e.g., the context142) associated with the audio data samples310. In a particular aspect, the scene data338is generated based on settings data330indicating one or more device settings associated with the device100, output of a clock332, sensor data from one or more sensors334(e.g., the sensors134), input received via an input device336, or a combination thereof. In some aspects, the scene detector340uses different information to determine the audio scene342than the SEC engine320uses to select the SEC model312. To illustrate, if the SEC engine320selects the SEC model312based on time of day, the scene detector340may use position sensor data from a position sensor of the sensor(s)334to determine the audio scene342. In some aspects, the scene detector340uses at least some of the same information that the SEC engine320uses to select the SEC model312and uses additional information. To illustrate, if the SEC engine320selects the SEC model312based on time of day and the settings data330, the scene detector340may use the position sensor data and the settings data330to determine the audio scene342. Thus, the scene detector340uses a different audio scene detection mode than is used by the SEC engine320to select the SEC model312.

In a particular implementation, the scene detector340is a neural network that is trained to determine the audio scene342based on the scene data338. In other implementations, the scene detector340is a classifier that trained using a different machine-learning technique. For example, the scene detector340may include or correspond to a decision tree, a random forest, a support vector machine, or another classifier that is trained to generate output indicating the audio scene342based on the scene data338. In still other implementations, the scene detector340uses heuristics to determine the audio scene342based on the scene data338. In yet other implementations, the scene detector340uses a combination of artificial intelligence and heuristics to determine the audio scene342based on the scene data338. For example, the scene data338may include image data, video data, or both, and the scene detector340may include an image recognition model that is trained using a machine-learning technique to detect particular objects, motions, backgrounds, or other image or video information. In this example, output of the image recognition model may be evaluated via one or more heuristics to determine the audio scene342.

The drift detector328compares the audio scene342indicated by the scene detector340to information descriptive of the SEC model312to determine whether the SEC model312is associated with the audio scene342of the audio data samples310. If the drift detector328determines that the SEC model312is associated with the audio scene342of the audio data samples310, the drift detector328causes drift data344to be stored as model update data348. In a particular implementation, the drift data344includes the audio data samples310and a label, where the label identifies the SEC model312, indicates a sound class associated with the audio data samples310, or both. If the drift data344indicates a sound class associated with the audio data samples310, the sound class may be selected based on a highest value of the confidence metric generated by the SEC model312. As an illustrative example, if the SEC engine320uses a detection threshold of 0.90, and the highest value of the confidence metric output by the SEC model312is 0.85 for a particular sound class, the SEC engine320determines that the sound class of the audio data samples310was not recognized and sends the trigger signal326to the drift detector328. In this example, if the drift detector328determines that the SEC model312corresponds to the audio scene342of the audio data samples310, the drift detector328stores that the audio data samples310as drift data344associated with the particular sound class. In a particular aspect, metadata associated with the SEC models314includes information specifying an audio scene or audio scenes associated with each SEC model314. For example, the SEC model312may be configured to detect sound events in a user's home, in which case the metadata associated with the SEC model312may indicate that the SEC model312is associate with a “home” audio scene. In this example, if the audio scene342indicates that the device100is at a home location (e.g., based on position information, user input, detection of a home wireless network signal, image or video data representing home locations, etc.), the drift detector328determines that the SEC model312corresponds to the audio scene342.

In some implementations, the drift detector328also causes some audio data samples310to be stored as model update data348and designated as unknown data346. As a first example, the drift detector328may store the unknown data346if the drift detector328determines that the SEC model312does not correspond to the audio scene342of the audio data samples310. As a second example, the drift detector328may store the unknown data346if the value of the confidence metric output by the SEC model312fails to satisfy a drift threshold. In this example, the drift threshold is less than the detection threshold used by the SEC engine320. For example, if the SEC engine320uses a detection threshold of 0.95, the drift threshold may have a value of 0.80, of 0.75, or some other value less than 0.95. In this example, if the highest value of the confidence metric for the audio data samples310is less than the drift threshold, the drift detector328determines that the audio data samples310belong to a sound class that the SEC model312is not trained to recognize, and designates the audio data samples310as unknown data346. In a particular aspect, the drift detector328only stores the unknown data346if the drift detector328determines that the SEC model312corresponds to the audio scene342of the audio data samples310. In another particular aspect, the drift detector328stores the unknown data346independently of whether the drift detector328determines that the SEC model312corresponds to the audio scene342of the audio data samples310.

After the model update data348is stored, a model updater352can access the model update data348and use the model update data348to update one of the available SEC models314(e.g., the SEC model312). For example, each entry of the model update data348indicates an SEC model with which the entry is associated, and the model updater352uses the entry as training data to update the corresponding SEC model. In a particular aspect, the model updater352updates an SEC model when an update criterion is satisfied or when a model update is initiated by a user or another party (e.g., a vendor of the device100, the SEC engine320, the SEC models314, etc.). The update criterion may be satisfied when a particular number of entries are available in the model update data348, when a particular number of entries for a particular SEC model are available in the model update data348, when a particular number of entries for a particular sound class are available in the model update data348, when a particular amount of time has passed since a prior update, when other updates occur (e.g., when a software update associated with the device100occurs), or based on occurrence of another event.

The model updater352uses the drift data344as labeled training data to update training of the SEC model312using backpropagation or a similar machine-learning optimization process. For example, the model updater352provides audio data samples from the drift data344of the model update data348as input to the SEC model312, determines a value of an error function (also referred to as a loss function) based on output of the SEC model312and a label associate with the audio data samples (as indicated in the drift data344stored by the drift detector328), and determines updated link weights for the SEC model312using a gradient descent operation (or some variant thereof) or another machine-learning optimization process.

The model updater352may also provide other audio data samples (in addition to audio data samples of the drift data344) to the SEC model312during the update training. For example, the model update data348may include one or more known audio data samples (such as a subset of the audio data samples originally used to train the SEC model312), which may reduce the chances of the update training causing the SEC model312to forget previous training (where “forgetting” here refers to losing reliability for detecting sound classes that the SEC model312was previously trained to recognize). Since the sound class associated with the audio data samples of the drift data344is indicated by the drift detector328, update training to account for drift can be accomplished automatically (e.g., without user input). As a result, functionality of the device100(e.g., accuracy in recognizing sound classes) can improve over time without user intervention and using fewer computing resources than would be used to generate a new SEC model from scratch. A particular example of a transfer learning process that the model updater352can use to update the SEC model312based on the drift data344is described with reference toFIG.4.

In some aspects, the model updater352can also use the unknown data346of the model update data348to update training of the SEC model312. For example, periodically or occasionally, such as when the update criterion is satisfied, the model updater352may prompt a user to ask the user to label the sound class of the entries of the unknown data346in the model update data348. If the user choses to label the sound class of an entry of unknown data346, the device100(or another device) may playout sound corresponding to audio data samples of the unknown data346. The user can provide one or more labels350(e.g., via the input device336) identifying a sound class of the audio data samples. If the sound class indicated by the user is a sound class that the SEC model312is trained to recognize, then the unknown data346is reclassified as drift data344associated with the user-specified sound class and the SEC model312. Depending on the configuration of the model updater352, if the sound class indicated by the user is a sound class that the SEC model312is not trained to recognize (e.g., is a new sound class), the model updater352may discard the unknown data346, send the unknown data346and the user-specified sound class to another device for use to generate a new or updated SEC model, or may use the unknown data346and the user-specified sound class to update the SEC model312. A particular example of a transfer learning process that the model updater352can use to update the SEC model312based on the unknown data346and the user-specified sound class is described with reference toFIG.5.

An updated SEC model354generated by the model updater352is added to the available SEC models314to make the updated SEC model354available to evaluate audio data samples310received after the updated SEC model354is generated. Thus, the set of available SEC models314that can be used to evaluate sounds is dynamic. For example, one or more of the available SEC models314can be automatically updated to account for drift data344. Additionally, one or more of the available SEC models314can be updated to account for unknown sound classes using transfer learning operations that use fewer computing resources (e.g., memory, processing time, and power) than training a new SEC model from scratch.

FIG.4is a diagram that illustrates aspects of updating an SEC model408to account for drift according to a particular example. The SEC model408ofFIG.4includes or corresponds to a particular one of the available SEC models314ofFIG.3that is associated with the drift data344. For example, if the SEC engine320generated the trigger signal326in response to output of the SEC model312, the drift data344is associated with the SEC model312, and the SEC model408corresponds to or includes the SEC model312. As another example, if the SEC engine320generated the trigger signal326in response to output of the Kth SEC model318, the drift data344is associated with the Kth SEC model318, and the SEC model408corresponds to or includes the Kth SEC model318.

In the example illustrated inFIG.4, training data402is used to update the SEC model408. The training data402includes the drift data344and one or more labels404. Each entry of the drift data344includes audio data samples (e.g., audio data samples406) and is associated with a corresponding label of the label(s)404. The audio data samples of an entry of the drift data344include a set of values representing features extracted from or determined based on a sound that was not recognized by the SEC model408. The label404corresponding to an entry of the drift data344identifies a sound class to which the sound is expected to belong. As an example, the label404corresponding to an entry of the drift data344may be assigned by the drift detector328ofFIG.3in response to determining that the SEC model408corresponds to the audio scene in which the audio data samples were generated. In this example, the drift detector328may assign the audio data samples to the sound class that was associated, in the output of the SEC model408, with a highest confidence metric value.

InFIG.4, audio data samples406corresponding to a sound are provided to the SEC model408, and the SEC model408generates output410that indicates a sound class to which the audio data samples406are assigned, one or more values of a confidence metric, or both. The model updater352uses the output410and the label404corresponding to the audio data samples406to determine updated link weights412for the SEC model408. The SEC model408is updated based on the updated link weights412, and the training process is repeated iteratively until a training termination condition is satisfied. During training, each of the entries of the drift data344may be provided to the SEC model408(e.g., one entry per iteration). Additionally, in some implementations, other audio data samples (e.g., audio data samples previously used to train the SEC model408) may also be provided to the SEC model408to reduce the chance of the SEC model408forgetting prior training.

The training termination condition may be satisfied when all of the drift data344has been provided to the SEC model408at least once, after a particular number of training iterations have been performed, when a convergence metric satisfies a convergence threshold, or when some other condition indicative of the end of training is met. When the training termination condition is satisfied, the model updater352stores the updated SEC model414, where the updated SEC model414corresponds to the SEC model408with link weights based on the updated link weights412applied during training.

FIG.5is a diagram that illustrates aspects of updating an SEC model510based on training data502to account for unknown data according to a particular example. The SEC model510ofFIG.5includes or corresponds to a particular one of the available SEC models314ofFIG.3that is associated with the unknown data346. For example, if the SEC engine320generated the trigger signal326in response to output of the SEC model312, the unknown data346is associated with the SEC model312, and the SEC model510corresponds to or includes the SEC model312. As another example, if the SEC engine320generated the trigger signal326in response to output of the Kth SEC model318, the unknown data346is associated with the Kth SEC model318, and the SEC model510corresponds to or includes the Kth SEC model318.

In the example ofFIG.5, the model updater352generates an update model506. The update model506includes the SEC model510that is to be updated, an incremental model508, and one or more adapter networks512. The incremental model508is a copy of the SEC model510with a different output layer than the SEC model510. In particular, the output layer of the incremental model508includes more output nodes than the output layer of the SEC model510. For example, the output layer of the SEC model510includes a first count of nodes (e.g., N nodes, where N is a positive integer corresponding to the number of sound classes that the SEC model510is trained to recognize), and the output layer of the incremental model508includes a second count of nodes (e.g., N+M nodes, where M is a positive integer corresponding to a number of new sound classes that an updated SEC model524is to be trained to recognized that the SEC model510is not trained to recognize). The first count of nodes corresponds to the count of sound classes of a first set of sound classes that the SEC model510is trained to recognize (e.g., the first set of sound classes includes N distinct sound classes that the SEC model510can recognize), and the second count of nodes corresponds to the count of sound classes of a second set of sound classes that the updated SEC model524is to be trained to recognize (e.g., the second set of sound classes includes N+M distinct sound classes that the updated SEC model524is to be trained to recognize). The second set of sound classes includes the first set of sound classes (e.g., N classes) plus one or more additional sound classes (e.g., M classes). Model parameters (e.g., link weights) of the incremental model508are initialized to be equal to model parameters of the SEC model510.

The adapter network(s)512include a neural adapter and a merger adapter. The neural adapter includes one or more adapter layers configured to receive input from the SEC model510and to generate output that can be merged with the output of the incremental model508. For example, the SEC model510generates a first output corresponding to the first count of classes of the first set of sound classes. In a particular aspect, the first output includes one data element for each node of the output layer of the SEC model510(e.g., N data elements). In contrast, the incremental model508generates a second output corresponding to the second count of classes of the second set of sound classes. For example, the second output includes one data element for each node of the output layer of the incremental model508(e.g., N+M data elements). In this example, the adapter layer(s) of the adapter network(s)512receive the output of the SEC model510as input and generate an output having the second count of data elements (e.g., N+M). In a particular example, the adapter layer(s) of the adapter network(s)512include two fully connected layers (e.g., an input layer including N nodes and an output layer including N+M nodes, with each node of the input layer connected to every node of the output layer).

The merger adapter of the adapter network(s)512is configured to generate output514of the update model506by merging the output of the adapter layer(s) and the output of the incremental model508. For example, the merger adapter combines the output of the adapter layer(s) and the output of the incremental model508in an element-by-element manner to generate a combined output and applies an activation function (such as a sigmoid function) to the combined output to generate the output514. The output514indicates a sound class to which the audio data samples504are assigned by the update model506, one or more confidence metric values determined by the update model506, or both.

The model updater352uses the output514and a label350corresponding to the audio data samples504to determine updated link weights516for the incremental model508, the adapter network(s)512, or both. Link weights of the SEC model510are unchanged during training. The training process is repeated iteratively until a training termination condition is satisfied. During training, each of the entries of the unknown data346may be provided to the update model506(e.g., one entry per iteration). Additionally, in some implementations, other audio data samples (e.g., audio data samples previously used to train the SEC model510) may also be provided to the update model506to reduce the chance of the incremental model508forgetting prior training of the SEC model510.

The training termination condition may be satisfied when all of the unknown data346has been provided to the update model506at least once, after a particular number of training iterations have been performed, when a convergence metric satisfies a convergence threshold, or when some other condition indicative of the end of training is met. When the training termination condition is satisfied, a model checker520selects the updated SEC model524from between the incremental model508and the update model506(e.g., the combination of the SEC model510, the incremental model508, and the adapter network(s)512).

In a particular aspect, the model checker520selects the updated SEC model524based on an accuracy of sound classes522assigned by the incremental model508and an accuracy of the sound classes522assigned by the SEC model510. For example, the model checker520may determine an F1-score for the incremental model508(based on the sound classes522assigned by the incremental model508) and an F1-score of the SEC model510(based on the sound classes522assigned by the SEC model510). In this example, if the value of the F1-score of incremental model508is greater than or equal to the value of the F1-score of the SEC model510, the model checker520selects the incremental model508as the updated SEC model524. In some implementations, the model checker520selects the incremental model508as the updated SEC model524if the value of the F1-score of the incremental model508is greater than or equal to the value of the F1-score of the SEC model510(or is less than the value of the F1-score of the SEC model510by less than a threshold amount). If the value of the F1-score of incremental model508is less than the value of the F1-score for the SEC model510(or is less than the value of the F1-score for the SEC model510by more than the threshold amount), the model checker520selects the update model506as the updated SEC model524. If the incremental model508is selected as the updated SEC model524, the SEC model510, the adapter network(s)512, or both may be discarded.

In some implementations, the model checker520is omitted or integrated with the model updater352. For example, after training the update model506, the update model506can be stored as the updated SEC model524(e.g., with no selection between the update model506and the incremental model508). As example, while training the update model506, the model updater352can determine an accuracy metric for the incremental model508. In this example, the training termination condition may be based on the accuracy metric for the incremental model508such that after training, the incremental model508is stored as the updated SEC model524(e.g., with no selection between the update model506and the incremental model508).

Utilizing the transfer learning techniques described with reference toFIG.5, the model checker520enables the device100ofFIG.3to update an SEC model to recognize a previously unknown sound class. Additionally, the transfer learning techniques described use significantly less computer resources (e.g., memory, processing time, and power) than would be used to train an SEC model from scratch to recognize the previously unknown sound class.

In some implementations, the operations described with reference toFIG.4(e.g., generating the updated SEC model414based on the drift data344) are performed at the device100ofFIG.3(e.g., at the one or more processors110), and the operations described with reference toFIG.5(e.g., generating the updated SEC model524based on the unknown data346) are performed at different device (such as a remote computing device818ofFIG.8). To illustrate, the unknown data346and label(s)350can be captured at the device100and transmitted to a second device that has more available computing resources. In this example, the second device generates the updated SEC model524and the device100downloads or receives a transmission or data representing the updated SEC model524from the second device. Generating the updated SEC model524based on the unknown data346is a more resource intensive process (e.g., uses more memory, power, and processor time) than generating the updated SEC model414based on the drift data344. Thus, dividing operations described with reference toFIG.4and the operations described with reference toFIG.5among different devices can conserve resources of the device100.

FIG.6is a diagram illustrating a particular example of operation of the device100ofFIG.1in which a determination of whether an active SEC model (e.g., the SEC model312) corresponds to an audio scene in which audio data samples310are captured is based on comparing a current audio scene to a prior audio scene.

InFIG.6, audio data captured by the microphone(s)104is used to generate the audio data samples310. The audio data samples310are used to perform audio classification602. For example, one or more of the available SEC models314is used as an active SEC model by the SEC engine320ofFIG.3. In a particular aspect, the active SEC model is selected from among the available SEC models314based on an audio scene indicated by the scene detector340during a prior sampling period, which is also referred to as a prior audio scene608.

Audio classification602generates a result604based on analysis of the audio data samples310using the active SEC model. The result604may indicate a sound class associated with the audio data samples310, a probability that the audio data samples310correspond to a particular sound class, or that a sound class of the audio data samples310is unknown. If the result604indicates that the audio data samples310correspond to a known sound class, a decision is made, at block606, to generate an output324indicating the sound class322associated with the audio data samples310. For example, the SEC engine320ofFIG.1may generate the output324.

If the result604indicates that the audio data samples310do not correspond to a known sound class, a decision is made, at block606, to generate the trigger326. The trigger326activates a drift detection scheme, which inFIG.6includes causing the scene detector340to identify the current audio scene607based on data from the sensor(s)134.

The current audio scene607is compared, at block610, to the prior audio scene608to determine whether an audio scene change has occurred since the active SEC model was selected. At block612, a determination is made whether the sound class of the audio data samples310was not recognized due to drift. For example, if the current audio scene607does not correspond to the prior audio scene608, the determination at block612is that drift was not the cause of the sound class of the audio data samples310not being recognized. In this circumstance, the audio data samples310may be discarded or, at block614, stored as unknown data.

If the current audio scene607corresponds to the prior audio scene608, the determination at block612is that the sound class of the audio data samples310was not recognized due to drift because the active SEC model corresponds to the current audio scene607. In this circumstance, the sound class that has drifted is identified, at block616, and the audio data samples310and an identifier of the sound class are stored as drift data, at block618.

When sufficient drift data is stored, the SEC model is updated, at block620, to generate the updated SEC model354. The updated SEC model354is added to the available SEC models314. In some implementations, the updated SEC model354replaces the active SEC model that generated the result604.

FIG.7is a diagram illustrating another particular example of operation of the device100ofFIG.1in which a determination of whether an active SEC model (e.g., SEC model312) corresponds to an audio scene in which audio data samples310are captured is based on comparing a current audio scene to information descriptive of the active SEC model.

InFIG.7, audio data captured by the microphone(s)104is used to generate the audio data samples310. The audio data samples310are used to perform audio classification602. For example, one or more of the available SEC models314is used as an active SEC model by the SEC engine320ofFIG.3. In a particular aspect, the active SEC model is selected from among the available SEC models314. In some implementations, an ensemble of the available SEC models314is used rather than selecting one or more of the available SEC models314as an active SEC model.

The audio classification602generates a result604based on analysis of the audio data samples310using one or more of the available SEC model314. The result604may indicate a sound class associated with the audio data samples310, a probability that the audio data samples310correspond to a particular sound class, or that a sound class of the audio data samples310is unknown. If the result604indicates that the audio data samples310correspond to a known sound class, a decision is made, at block606, to generate an output324indicating the sound class322associated with the audio data samples310. For example, the SEC engine320ofFIG.3may generate the output324.

If the result604indicates that the audio data samples310do not correspond to a known sound class, a decision is made, at block606, to generate the trigger326. The trigger326activates a drift detection scheme, which inFIG.7includes causing the scene detector340to identify the current audio scene based on data from the sensor(s)134and to determine whether the current audio scene corresponds to the SEC model that generated the result604that caused the trigger326to be sent.

At block612, a determination is made whether the sound class of the audio data samples310was not recognized due to drift. For example, if the current audio scene does not correspond to the SEC model that generated the result604, the determination at block612is that drift was not the cause of the sound class of the audio data samples310not being recognized. In this circumstance, the audio data samples310may be discarded or, at block614, stored as unknown data.

If the current audio scene corresponds to the SEC model that generated the result604, the determination at block612is that the sound class of the audio data samples310was not recognized due to drift. In this circumstance, the sound class that has drifted is identified, at block616, and the audio data samples310and an identifier of the sound class are stored as drift data, at block618.

When sufficient drift data is stored, the SEC model is updated, at block620, to generate the updated SEC model354. The updated SEC model354is added to the available SEC models314. In some implementations, the updated SEC model354replaces the active SEC model that generated the result604.

Operations to update models as described with reference toFIGS.3-7may be used in conjunction with context-based model selection as described with reference toFIGS.1-2. For example, a first person and a second person that live together in a first residence (e.g., a rural house) may have devices that use common models associated with the first residence, and such models may be updated by training the models to accommodate new sound events and drift associated with the first residence. After the first person moves to a second residence (e.g., a college dormitory), the device of the first person may update one or more models for improved accuracy at the second residence, and therefore the models used by the devices of the first person and the second person may diverge significantly. Upon the first person returning to the first residence, the device of the first person may select and download models used by the device of the second person to achieve higher accuracy during use at the first residence. For example, the second person may provide access permissions to share one or more models with the first person's device, such as via peer-to-peer transfer between devices or via a local wireless home network. Upon exiting the first residence, the first person's device may remove the shared models associated with the first residence and revert to the models associated with the second residence.

FIG.8is a block diagram illustrating a particular example of the device100ofFIG.1. In various implementations, the device100may have more or fewer components than illustrated inFIG.8.

In a particular implementation, the device100includes a processor804(e.g., a central processing unit (CPU)). The device100may include one or more additional processor(s)806(e.g., one or more digital signal processors (DSPs)). The processor804, the processor(s)806, or both, may correspond to the one or more processors110. For example, inFIG.8, the processor(s)806include the context detector140, the model selector190, and the model-based application192.

InFIG.8, the device100also includes the memory108and a CODEC824. The memory108stores instructions860that are executable by the processor804, or the processor(s)806, to implement one or more operations described with reference toFIGS.1-7. In an example, the memory108corresponds to a non-transitory computer-readable medium that stores the instructions860executable by the one or more processors110, and the instructions860include or correspond to (e.g., are executable by a processor to perform operations attributed to) the context detector140, the model selector190, the model-based application192, or a combination thereof. The memory108may also store the available models114.

InFIG.8, speaker(s)822and the microphone(s)104may be coupled to the CODEC824. In the example illustrated inFIG.8, the CODEC824includes a digital-to-analog converter (DAC826) and an analog-to-digital converter (ADC828). In a particular implementation, the CODEC824receives analog signals from the microphone(s)104, converts the analog signals to digital signals using the ADC828, and provides the digital signals to the processor(s)806. In a particular implementation, the processor(s)806provide digital signals to the CODEC824, and the CODEC824converts the digital signals to analog signals using the DAC826and provides the analog signals to the speaker(s)822.

InFIG.8, the device100also includes an input device336. The device100may also include a display820coupled to a display controller810. In a particular aspect, the input device336includes a sensor, a keyboard, a pointing device, etc. In some implementations, the input device336and the display820are combined in a touchscreen or similar touch or motion sensitive display.

In some implementations, the device100also includes a modem812coupled to a transceiver814. InFIG.8, the transceiver814is coupled to an antenna816to enable wireless communication with other devices, such as the remote computing device818(e.g., a server or network memory storing at least a portion of the model library162). For example, the modem812may be configured to receive the model112, an access permission for the model112, or both, at least partially based on the location of the device100matching the particular location, from the remote computing device818via wireless transmission. In other examples, the transceiver814is also, or alternatively, coupled to a communication port (e.g., an ethernet port) to enable wired communication with other devices, such as the remote computing device818.

InFIG.8, the device100includes the clock332and the sensors134. As specific examples, the sensors134include one or more cameras150, one or more location sensors152, the microphone(s)104, the activity detector154, other sensor(s)156, or a combination thereof.

In a particular aspect, the clock332generates a clock signal that can be used to assign a timestamp to particular sensor data samples to indicate when particular sensor data samples were received. In this aspect, the model selector190can use the timestamp to select a model to use to process input data. Additionally or alternatively, the timestamp can be used by the context detector140to determine the context142associated with the particular sensor data samples.

In a particular aspect, the camera(s)150generate image data, video data, or both. The model selector190can use the image data, the video data, or both, to select a particular model to use to analyze input data. Additionally or alternatively, the image data, the video data, or both, can be used by the context detector140to determine the context142associated with the particular sensor data samples. For example, the particular model112can be designated for outdoor use, and the image data, the video data, or both, may be used to confirm that the device100is located in an outdoor environment.

In a particular aspect, the location sensor(s)152generate location data, such as global position data indicating a location of the device100. The model selector190can use the location data to select a model to use to analyze input data. Additionally or alternatively, the position data can be used by the context detector140to determine the context142associated with the particular sensor data samples. For example, the particular model112can be designated for use at home, and the location data may be used to confirm that the device100is located at a home location. The location sensor(s)852may include a receiver for a satellite-based positioning system, a receiver for a local positioning system receiver, an inertial navigation system, a landmark-based positioning system, or a combination thereof.

The other sensor(s)156can include, for example, an orientation sensor, a magnetometer, a light sensor, a contact sensor, a temperature sensor, or any other sensor that is coupled to or included within the device100and that can be used to generate sensor data useful for determining the context142associated with the device100at a particular time.

In a particular implementation, the device100is included in a system-in-package or system-on-chip device802. In a particular implementation, the memory108, the processor804, the processor(s)806, the display controller810, the CODEC824, the modem812, and the transceiver814are included in the system-in-package or system-on-chip device802. In a particular implementation, the input device336and a power supply830are coupled to the system-on-chip device802. Moreover, in a particular implementation, as illustrated inFIG.8, the display820, the input device336, the speaker(s)822, the sensors134, the clock332, the antenna816, and the power supply830are external to the system-on-chip device802. In a particular implementation, each of the display820, the input device336, the speaker(s)822, the sensors134, the clock332, the antenna816, and the power supply830may be coupled to a component of the system-on-chip device802, such as an interface or a controller.

The device100may include, correspond to, or be included within a voice activated device, an audio device, a wireless speaker and voice activated device, a portable electronic device, a car, a vehicle, a computing device, a communication device, an internet-of-things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, a smart speaker, a mobile computing device, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a gaming console, an appliance, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, or any combination thereof. In a particular aspect, the processor804, the processor(s)806, or a combination thereof, are included in an integrated circuit.

FIG.9is an illustrative example of a vehicle900that incorporates aspects of the device100ofFIG.1. According to one implementation, the vehicle900is a self-driving car. According to other implementations, the vehicle900is a car, a truck, a motorcycle, an aircraft, a water vehicle, etc. InFIG.9, the vehicle900includes the display820, one or more of the sensors134, the device100including the context detector140, the model selector190, the model-based application192, or a combination thereof. The sensors134, the context detector140, the model selector190, and the model-based application192are shown using a dotted line to indicate that these components might not be visible to passengers of the vehicle900. The device100can be integrated into the vehicle900or coupled to the vehicle900.

In a particular aspect, the device100is coupled to the display820and provides an output to the display820responsive to the model-based application192, such in response to detecting or recognizing various events (e.g., sound events) described herein. For example, the device100provides the output324ofFIG.3to the display820indicating a sound class of a sound302(such as a car horn) in audio data105received from the microphone(s)104. In some implementations, the device100can perform an action responsive to recognizing a sound event, such as alerting an operator of the vehicle or activating one of the sensors134. In a particular example, the device100provides an output that indicates whether an action is being performed responsive to the recognized sound event. In a particular aspect, a user can select an option displayed on the display820to enable or disable a performance of actions responsive to recognized sound events.

In a particular implementations, the sensors134include the microphone(s)104ofFIG.1, vehicle occupancy sensors, eye tracking sensor, location sensor(s)152, or external environment sensors (e.g., lidar sensors or cameras). In a particular aspect, sensor input of the sensors134indicates a location of the user. For example, the sensors134are associated with various locations within the vehicle900.

Thus, the techniques described with respect toFIGS.1-8enable a user of the vehicle900to select a model to be used based on the specific context in which the device100operates.

FIG.10depicts an example of the device100coupled to or integrated within a headset1002, such as a virtual reality headset, an augmented reality headset, a mixed reality headset, an extended reality headset, a head-mounted display, or a combination thereof. A visual interface device, such as the display820, is positioned in front of the user's eyes to enable display of augmented reality, mixed reality, or virtual reality images or scenes to the user while the headset1002is worn. In a particular example, the display820is configured to display output of the device100. The headset1002includes the sensors134, such as the microphone(s)104, the camera(s)150, the location sensor(s)152, the other sensors156, or a combination thereof. Although illustrated in a single location, in other implementations the sensors134can be positioned at other locations of the headset1002, such as an array of one or more microphones and one or more cameras distributed around the headset1002to detect multi-modal inputs.

The sensors134enable detection of sensor data, which the device100uses to detect a context of the headset1002and to update models based on the detected context. For example, the model-based application192(e.g., the SEC engine320) may use one or more models to generate the sound event classification data which may be provided to the display820to indicate that a recognized sound event, such as a car horn, is detected in audio data samples received from the sensors134. In some implementations, the device100can perform an action responsive to recognizing a sound event, such as activating a camera or another one of the sensors134or providing haptic feedback to the user.

FIG.11depicts an example of the device100integrated into a wearable electronic device1102, illustrated as a “smart watch,” that includes the display820and the sensors134. The sensors134enable context detection, for example, based on modalities such as location, video, speech, and gesture, which the device100may use to update one or models used by the model-based application192. The sensors134also enable detection of sounds and other events in an environment around the wearable electronic device1102, which the device100may detect or interpret using the model-based application192. For example, the device100provides the output324ofFIG.3to the display820indicating that a recognized sound event is detected in audio data samples received from the sensors134. In some implementations, the device100can perform an action responsive to recognizing a sound event, such as activating a camera or another one of the sensors134or providing haptic feedback to the user.

FIG.12is an illustrative example of a voice-controlled speaker system1200. The voice-controlled speaker system1200can have wireless network connectivity and is configured to execute an assistant operation. InFIG.12, the device100is included in the voice-controlled speaker system1200. The voice-controlled speaker system1200also includes a speaker1202and sensors134. The sensors134include microphone(s)104ofFIG.1to receive voice input or other audio input.

During operation, in response to receiving a verbal command or a recognized sound event, the voice-controlled speaker system1200can execute assistant operations. The assistant operations can include adjusting a temperature, playing music, turning on lights, etc. The sensors134enable detection of data samples, which the device100may use to update a context of the voice-controlled speaker system1200and to update one or more models based on the context. Additionally, the voice-controlled speaker system1200can execute some operations based on events recognized by the device100. For example, if the device100recognizes the sound of a door closing, the voice-controlled speaker system1200can turn on one or more lights.

FIG.13illustrates a camera1300that incorporates aspects of the device100ofFIG.1. InFIG.13, the device100is incorporated in or coupled to the camera1300. The camera1300includes an image sensor1302and one or more other sensors (e.g., the sensors134), such as the microphone(s)104ofFIG.1. Additionally, the camera1300includes the device100, which is configured to determine a context of camera1300and to update one or more models based on the context. In a particular aspect, the camera1300is configured to perform one or more actions in response to a recognized sound event. For example, the camera1300may cause the image sensor1302to capture an image in response to the device100detecting a particular sound event in audio data samples from the sensors134.

FIG.14illustrates a mobile device1400that incorporates aspects of the device100ofFIG.1. InFIG.14, the mobile device1400includes or is coupled to the device100ofFIG.1. The mobile device1400includes a phone or tablet, as illustrative, non-limiting examples. The mobile device1400includes a display820and the sensors134, such as the microphone(s)104, the camera(s)150, the location sensor(s)152, or the other sensor(s)156. During operation, the mobile device1400may perform particular actions in response to the device100recognizing a particular sound event. For example, the actions can include sending commands to other devices, such as a thermostat, a home automation system, another mobile device, etc.

FIG.15illustrates a hearing aid device1500that incorporates aspects of the device100ofFIG.1. InFIG.15, the hearing aid device1500includes or is coupled to the device100ofFIG.1. The hearing aid device1500includes the sensors134, such as the microphone(s)104, the camera(s)150, the location sensor(s)152, or the other sensor(s)156. During operation, the hearing aid device1500may update one or more models in response to the device100recognizing a context of the hearing aid device1500, such as an acoustic environment of the hearing aid device1500, for use by the model-based application192for processing audio data, such as for location-specific noise reduction.

FIG.16illustrates an aerial device1600that incorporates aspects of the device100ofFIG.1. InFIG.16, the aerial device1600includes or is coupled to the device100ofFIG.1. The aerial device1600is a manned, unmanned, or remotely piloted aerial device (e.g., a package delivery drone). The aerial device1600includes a control system1602and the sensors134, such as the microphone(s)104, the camera(s)150, the location sensor(s)152, or the other sensor(s)156. The control system1602controls various operations of the aerial device1600, such as cargo release, sensor activation, take-off, navigation, landing, or combinations thereof. For example, the control system1602may control flight of the aerial device1600between specified points and deployment of cargo at a particular location. During operation, the aerial device1600may update one or more models in response to the device100recognizing a context of the aerial device1600, such as a location or acoustic environment of the aerial device1600, for use by the model-based application192for detecting events. To illustrate, the control system1602may initiate a safe landing protocol in response to the device100detecting an aircraft engine.

FIG.17illustrates a headset1700that incorporates aspects of the device100ofFIG.1. InFIG.17, the headset1700includes or is coupled to the device100ofFIG.1. The headset1700includes one or more of the microphone(s)104ofFIG.1positioned to primarily capture speech of a user. The headset1700may also include one or more additional microphone positioned to primarily capture environmental sounds (e.g., for noise canceling operations) and one or more of the sensors134, such as the camera(s)150, the location sensor(s)152, or the other sensor(s)156. In a particular aspect, the headset1700may update one or more models in response to the device100recognizing a change in context of the headset1700, such as a location or acoustic environment of the headset1700, for use by the model-based application192for performing operations such as a noise cancellation feature.

FIG.18illustrates an appliance1800that incorporates aspects of the device100ofFIG.1. InFIG.18, the appliance1800is a lamp; however, in other implementations, the appliance1800includes another Internet-of-Things appliance, such as a refrigerator, a coffee maker, an oven, another household appliance, etc. The appliance1800includes or is coupled to the device100ofFIG.1. The appliance1800includes the sensors134, such as the microphone(s)104, the camera(s)150, the location sensor(s)152, the activity detector154, or the other sensor(s)156. In a particular aspect, the appliance1800may update one or more models in response to the device100recognizing a change in context of the appliance1800, for use by the model-based application192for performing operations such as to activate a light in response to the device100detecting a door closing.

FIG.19is a flow chart illustrating an example of a method1900of operation of the device100ofFIG.1. The method1900can be initiated, controlled, or performed by the device100. For example, the processor(s)110can execute instructions, such as the instructions860ofFIG.8, from the memory108to perform context-based model selection.

The method1900includes, in block1902, receiving, at one or more processors of a device, sensor data from one or more sensor devices. For example, the context detector140in the one or more processors110receives the sensor data138from the one or more sensor devices134. In some implementations, the sensor data includes location data of a location of the device, such as the location data153, and the context is at least partially based on the location. In some implementations, the sensor data includes image data corresponding to a visual scene, such as the image data151, and the context is at least partially based on the visual scene. In some implementations, the sensor data includes audio corresponding to an audio scene, such as the audio data105, and the context is at least partially based on audio scene. In some implementations, the sensor data includes motion data corresponding to motion of the device, such as the activity data155, and the context is at least partially based on the motion of the device.

In block1904, the method1900includes determining, at the one or more processors, a context of the device based on the sensor data. For example, the context detector140in the one or more processors110receives the sensor data138from the one or more sensor devices134and determines the context142based on the sensor data138.

In block1906, the method1900includes selecting, at the one or more processors, a model based on the context. For example, the model selector190selects the model112based on the context142. In a particular implementation, the model is selected from among multiple models stored at a memory of the device, such as the available models114. According to some implementations, the model is downloaded from a library, such as the model library162, that corresponds to a library of shared models. The model may include a trained model uploaded to the library from another user device. In an example, the library corresponds to a crowdsourced library of models. The library may be included in a distributed context-aware system.

In block1908, the method1900includes processing, at the one or more processors, an input signal using the model to generate a context-specific output. For example, the model-based application192processes the input signal106using the selected model112to generate the context-specific output122.

In some implementations, the method1900includes pruning the model in response to determining that the context has changed. For example, the model selector190may permanently delete the model in response to detecting that the context142has changed (e.g., the device100is moved to a different location) and that the current model is no longer appropriate for the new context, or that another model is more appropriate for the new context.

In some implementations, the model includes a sound event detection model, the input signal includes an audio signal, and the context-specific output includes a classification of a sound event in the audio signal. In some implementations, the model includes an automatic speech recognition model, the input signal includes an audio signal, and the context-specific output includes text data representative of speech in the audio signal. In some implementations, the model includes a natural language processing (NLP) model, the input signal includes text data, and the context-specific output includes NLP output data based on the text data. In some implementations, the model includes a noise reduction model, the input signal includes an audio signal, and the context-specific output includes a noise reduced audio signal based on the audio signal. In some implementations, the model is associated with an automatic adjustment of a device operating mode, and wherein the context-specific output includes a signal to adjust the device operating mode.

In some implementations, the method1900includes receiving the model from a second device via wireless transmission. For example, the context may correspond to a location of the device, and the model includes an acoustic model corresponding to a particular location. In some implementations, the method1900includes receiving an access permission for the model at least partially based on the location of the device matching the particular location.

In some implementations, the context includes a particular acoustic environment, and the method1900includes determining whether a library of available acoustic models includes an acoustic model that is specific to the particular acoustic environment and available to the device and, in response to no acoustic model that is specific to the particular acoustic environment being available to the device, determining whether an acoustic model for a general category of the particular acoustic environment is available to the device. For example, in response to the model library162not having the “office” model274available for the device100when the device100is located in the room204, the device100may request the “office building” model262for a general model of the acoustic environment for the room204.

By selecting a model based on the context of the device, the method1900enables the device to perform with higher accuracy as compared to using a single model for all contexts. Further, changing models enables the device to perform with increased accuracy without incurring the power consumption, memory requirements, and processing resource usage associated with re-training an existing model from scratch at the device for a particular context. In addition, operation of the device using context-based models enables improved operation of the device itself, such as by enabling faster convergence when performing an iterative or dynamic process (e.g., in a noise cancellation technique) due to using a higher-accuracy model that is specific to the particular context.

FIG.18is a flow chart illustrating an example of a method1800of operation of the device100ofFIG.1. The method1800can be initiated, controlled, or performed by the device100. For example, the processor(s)110can execute instructions, such as the instructions660ofFIG.6, from the memory108to perform context-based model selection.

The method1800includes, in block1802, selecting, at one or more processors of a device, an acoustic model corresponding to a particular room, of a building, in which the device is located. For example, the model selector190selects the acoustic model212ofFIG.2corresponding to the room204in which the device100is located.

The method2000includes, in block2004, processing, at the one or more processors, an input audio signal using the acoustic model. As an illustrative example, the model-based application192uses the acoustic model212to perform noise reduction on the input signal106(e.g., the audio data105from the microphone(s)104) to generate a noise-reduced audio signal as the context-specific output122.

In some implementations, the method2000includes downloading the acoustic model from a library of acoustic models in response to a determination that the device has entered the particular room. In some implementations, the method2000includes pruning (e.g., removing) the acoustic model in response to the device leaving the particular room. In some implementations, the method2000includes selecting the acoustic model based on sensor data indicative of a location of the device, such as detecting a location of the device100by analyzing the image data151. In some implementations, the method2000includes selecting the acoustic model based on location data indicative of a location of the device, such as the location data153.

By selecting an acoustic model based on the context of the device, the method2000enables the device to perform with higher accuracy as compared to using a single acoustic model for all contexts. Further, changing acoustic models enables the device to perform with increased accuracy without incurring the power consumption, memory requirements, and processing resource usage associated with re-training an existing acoustic model from scratch at the device for a particular context. In addition, operation of the device using context-based acoustic models enables improved operation of the device itself, such as by enabling faster convergence when performing an iterative or dynamic process (e.g., in a noise cancellation technique) due to using a higher-accuracy acoustic model that is specific to the particular context.

FIG.21is a flow chart illustrating an example of a method2100of operation of the device100ofFIG.1within a vehicle, such as integrated in the vehicle900ofFIG.9. The method2100can be initiated, controlled, or performed by the device100. For example, the processor(s)110can execute instructions, such as the instructions860ofFIG.8, from the memory108to perform context-based model selection.

The method2100includes, in block2102, selecting, at one or more processors of a device and in response to detecting a user entering a vehicle, a personalized acoustic model for the user from among multiple personalized acoustic models corresponding to the vehicle. For example, the device100in the vehicle900ofFIG.9may store multiple acoustic models personalized for each user of the vehicle900and may select a personalized model or set of models for the particular user entering the vehicle900. In some implementations, the method2100includes determining that the user has entered the vehicle based on sensor data indicative of a location of the user. To illustrate, the sensor(s)134ofFIG.9may determine the user within the vehicle900via facial recognition, voice recognition, entry of an identification via gesture, voice, or interaction with an input device (e.g., a touchscreen in the vehicle900), or one or more other techniques to identify the user.

The method2100includes, in block2104, processing, at the one or more processors, an input audio signal using the personalized acoustic model. For example, in some implementations the personalized acoustic model corresponds to an ASR model trained for the particular user, and the personalized acoustic model is used by the model-based application192in the vehicle900to perform speech recognition for speech of the user that is captured via one or more microphones in the vehicle900. To illustrate, the ASR model is used to enhance accuracy of a voice interface to control one or more operations of the vehicle900(e.g., a navigation system, an entertainment system, climate control, driver assistance or self-driving settings, etc.). As another example, the personalized acoustic model can include a SEC model that is personalized for the particular user. To illustrate, if the particular user frequently brings the user's dog on driving trips, the user's personalized SEC model for use in the vehicle900may be trained to recognize the additional sound class of “dog barking inside vehicle.”

In some implementations, the method2100includes downloading the personalized acoustic model from a library of acoustic models in response to a determination that the user has entered the vehicle. In some implementations, the method2100includes pruning (e.g., removing) the personalized acoustic model in response to the device leaving the vehicle.

By selecting a personalized acoustic model in response to detecting a user entering the vehicle, the method2100enables the device to perform with higher accuracy as compared to using a single acoustic model for all contexts. Further, changing acoustic models enables the device to perform with increased accuracy without incurring the power consumption, memory requirements, and processing resource usage associated with re-training an existing acoustic model from scratch at the device for a particular context. In addition, operation of the device using context-based acoustic models enables improved operation of the device itself, such as by enabling faster convergence when performing an iterative or dynamic process (e.g., in a noise cancellation technique) due to using a higher-accuracy acoustic model that is specific to the particular context.

FIG.22is a flow chart illustrating an example of a method2200of operation of the device100ofFIG.1. The method2200can be initiated, controlled, or performed by the device100. For example, the processor(s)110can execute instructions, such as the instructions860ofFIG.8, from the memory108to perform context-based model selection.

The method2200includes, in block2202, downloading, at one or more processors of a device, an acoustic model corresponding to a particular location in which the device is located. In some implementations, the method2200includes determining that the device has entered the particular location based on sensor data indicative of a location of the device. In an example, the device100downloads the acoustic model212(e.g., the “office” model274corresponding to the room204) in response to the device100entering the room204or determining that the location of the device is within the room204(e.g., based on the location data153).

The method2200includes, in block2204, processing, at the one or more processors, an input audio signal using the acoustic model. In an example, the device100corresponds to the hearing aid device1500ofFIG.15and uses the acoustic model212(e.g., the “office” model274corresponding to the room204) to perform noise reduction at the model-based application192.

The method2200includes, in block2206, removing, at the one or more processors, the acoustic model in response to the device exiting the location. In some implementations, the method2200includes determining that the device has entered the particular location based on location data indicative of a location of the device. In an example, the device100prunes the acoustic model212(e.g., the “office” model274corresponding to the room204) in response to the device100exiting the room204or determining that the location of the device is no longer within the room204(e.g., based on the location data153).

Although the example provided above illustrate the method2200performed in the building202ofFIG.2, it should be understood that the method2200is not limited to any particular location or location type. For example, in some implementations, the location corresponds to a particular restaurant, and the acoustic model is downloaded from a library of acoustic models in response to a determination that the device has entered the particular restaurant. In other implementations, the location can correspond to a public park, a subway station, a car, a train, a plane, a particular room in the user's house, a particular room in a museum, an auditorium or concert hall, etc.

By selecting an acoustic model corresponding to the particular location of the device, the method2200enables the device to perform with higher accuracy as compared to using a single acoustic model for all locations. Further, changing acoustic models enables the device to perform with increased accuracy without incurring the power consumption, memory requirements, and processing resource usage associated with re-training an existing acoustic model from scratch at the device for a particular context, and removing the acoustic model in response to the device exiting the location improves operation of the device by freeing memory and resources associated with continuing to store the acoustic model when no longer in use, which enables reduce power consumption associated with model storage and latency associated with subsequent searches of acoustic models that are stored at the device. In addition, operation of the device using location-based acoustic models enables improved operation of the device itself, such as by enabling faster convergence when performing an iterative or dynamic process (e.g., in a noise cancellation technique) due to using a higher-accuracy acoustic model that is specific to the particular location.

FIG.23is a flow chart illustrating an example of a method2300of operation of the device100ofFIG.1. The method2300can be initiated, controlled, or performed by the device100. For example, the processor(s)110can execute instructions, such as the instructions860ofFIG.8, from the memory108to perform context-based model selection.

The method2300includes, in block2302, selecting, at the one or more processors of a device, an acoustic model corresponding to a particular location.

The method2300includes, in block2304, receiving, at the one or more processors, an access permission for the acoustic model at least partially based on a location of the device matching the particular location. In some implementations, the method2300includes receiving the access permission in response to detection of the device within the particular location. For example, the device100may receive the access permissions214to use the “office” model274ofFIG.2in response to detection of the device100in the room204. The device100may transmit the data indicating the acoustic environment210, such as location data, and in response a permissions management system may send the access permissions to the device100, such as described with reference toFIG.2.

The method2300includes, in block2306, processing, at the one or more processors, an input audio signal using the acoustic model.

By selecting an acoustic model corresponding to the particular location of the device, the method2300enables the device to perform with higher accuracy as compared to using a single acoustic model for all locations. Further, receiving access permission for the acoustic model based on location matching enables security of the acoustic model to be maintained by preventing use of the acoustic model other than when the device is in the particular location. Such security enables the acoustic model to be downloaded to the device as embargoed data, such as to reduce peak network bandwidth usage and to reduce the latency associated with using the acoustic model because the acoustic model may already be stored at the device and need not be downloaded upon entering the particular location. In addition, operation of the device using location-based acoustic models enables improved operation of the device itself, such as by enabling faster convergence when performing an iterative or dynamic process (e.g., in a noise cancellation technique) due to using a higher-accuracy acoustic model that is specific to the particular location.

FIG.24is a flow chart illustrating an example of a method2400of operation of the device100ofFIG.1. The method2400can be initiated, controlled, or performed by the device100. For example, the processor(s)110can execute instructions, such as the instructions860ofFIG.8, from the memory108to perform context-based model selection.

The method2400includes, in block2402, detecting, at one or more processors of a device, a context of the device. To illustrate, in some implementations, the one or more processors110are configured to detect the context, such as the context142detected by the context detector142based on the sensor data138. In illustrative, non-limiting examples, the context corresponds to a location or activity, such as driving in a car,

The method2400includes, in block2404, sending a request that indicates the context to a remote device. To illustrate, in some implementations, the one or more processors110are configured to initiate sending the request that indicates the context (e.g., the acoustic environment210) to a remote device, such as the remote computing device818(e.g., a server (which may be part of a service) or network memory storing at least a portion of the model library162).

The method2400includes, in block2406, receiving a model corresponding to the context. To illustrate, in some implementations, the one or more processors110are configured to receive the model112from the remote computing device818or another server or network memory storing at least a portion of the model library162in response to sending the request indicating the context. In some implementations, the model112is received as a compressed source model and is decompressed by the one or more processors110for use at the device100. In some implementations, the model is received based on private access. In an illustrative example, the acoustic model212is received in conjunction with the access permissions214that authorize the device100to access the acoustic model212(e.g., in conjunction with a license to use the model212). In an illustrative example, the model is received based on access granted by family or friends of a user of the device100.

The method2400includes, in block2408, using, at the one or more processors, the model while the context remains detected. To illustrate, in some implementations, the one or more processors110are configured to receive the model112in response to sending the request indicating the the context142and to continue using the model112at the model-based application192while the context142remains unchanged (e.g., use the received model112temporarily as long as the context142is detected).

The method2400includes, in block2410, pruning, at the one or more processors, the model in response to detecting a change of the context. To illustrate, in some implementations, the one or more processors110are configured to prune the model112in response to detecting a change of the context142(e.g., prune the model112once context changes). In some implementations, pruning the model includes permanently deleting the model.

In some implementations, the method2400includes generating at least one new sound class while the context remains detected, and pruning the model includes preserving the at least one new sound class. To illustrate, in some implementations, the one or more processors110are configured to generate new sound classes, such as by generating updated models or new models as described with reference toFIGS.3-5(e.g., the update model506), which are preserved by storing the updated or new models, such as at the memory108or via upload to the model library162.

In conjunction with the described implementations, an apparatus includes means for receiving sensor data. For example, the means receiving sensor data includes the device100, the instructions860, the processor804, the processor(s)806, context detector140, the microphone(s)104, the camera(s)150, the location sensor(s)152, the activity detector154, the other sensor(s)156, the CODEC824, one or more other circuits or components configured to receive sensor data, or any combination thereof.

The apparatus also includes means for determining a context based on the sensor data. For example, the means for determining the context based on the sensor data includes the device100, the instructions860, the processor804, the processor(s)806, the context detector140, one or more other circuits or components configured to determine a context based on the sensor data, or any combination thereof.

The apparatus also includes means for selecting a model based on the context. For example, the means for selecting a model based on the context includes the device100, the instructions860, the processor804, the processor(s)806, the model selector190, one or more other circuits or components configured to select a model based on the context, or any combination thereof.

The apparatus also includes means for processing an input signal using the model to generate a context-specific output. For example, the means for processing an input signal using the model to generate a context-specific output includes the device100, the instructions860, the processor804, the processor(s)806, the model-based application192, one or more other circuits or components configured to process an input signal using the model to generate a context-specific output, or any combination thereof.

Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or combinations of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, such implementation decisions are not to be interpreted as causing a departure from the scope of the present disclosure.

The steps of a method or algorithm described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor may read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a computing device or user terminal.

Particular aspects of the disclosure are described below in a first set of interrelated clauses:

According to Clause 1, a device includes one or more processors configured to: receive sensor data from one or more sensor devices; determine a context of the device based on the sensor data; select a model based on the context; and process an input signal using the model to generate a context-specific output.

Clause 2 includes the device of Clause 1, and further includes a location sensor coupled to the one or more processors, wherein the sensor data includes location data from the location sensor, the location data indicative of a location of the device, and wherein the context is at least partially based on the location.

Clause 3 includes the device of Clause 1 or Clause 2, and further includes a camera coupled to the one or more processors, wherein the sensor data includes image data from the camera, the image data corresponding to a visual scene, and wherein the context is at least partially based on the visual scene.

Clause 4 includes the device of any of Clauses 1 to 3, and further includes a microphone coupled to the one or more processors, wherein the sensor data includes audio data from the microphone, the audio data corresponding to an audio scene, and wherein the context is at least partially based on audio scene.

Clause 5 includes the device of any of Clauses 1 to 4, and further includes an activity detector coupled to the one or more processors, wherein the sensor data includes motion data from the activity detector, the motion data corresponding to motion of the device, and wherein the context is at least partially based on the motion of the device.

Clause 6 includes the device of any of Clauses 1 to 5, and further includes a memory coupled to the one or more processors, wherein the model is selected from among multiple models stored at the memory.

Clause 7 includes the device of any of Clauses 1 to 6, wherein the model includes a sound event detection model, the input signal includes an audio signal, and the context-specific output includes a classification of a sound event in the audio signal.

Clause 8 includes the device of any of Clauses 1 to 7, wherein the model includes an automatic speech recognition model, the input signal includes an audio signal, and the context-specific output includes text data representative of speech in the audio signal.

Clause 9 includes the device of any of Clauses 1 to 8, wherein the model includes a natural language processing (NLP) model, the input signal includes text data, and the context-specific output includes NLP output data based on the text data.

Clause 10 includes the device of any of Clauses 1 to 9, wherein the model includes a noise reduction model, the input signal includes an audio signal, and the context-specific output includes a noise reduced audio signal based on the audio signal.

Clause 11 includes the device of any of Clauses 1 to 10, wherein the model is associated with an automatic adjustment of a device operating mode, and wherein the context-specific output includes a signal to adjust the device operating mode.

Clause 12 includes the device of any of Clauses 1 to 11, and further includes a modem coupled to the one or more processors and configured to receive the model from a second device via wireless transmission.

Clause 13 includes the device of Clause 12, wherein the context corresponds to a location of the device, and wherein the model includes an acoustic model corresponding to a particular location.

Clause 14 includes the device of Clause 13, wherein the one or more processors are further configured to receive, via the modem, an access permission for the model at least partially based on the location of the device matching the particular location.

Clause 15 includes the device of any of Clauses 1 to 14, wherein the one or more processors are further configured to prune the model in response to determining that the context has changed.

Clause 16 includes the device of any of Clauses 1 to 15, wherein the model is downloaded from a library of shared models.

Clause 17 includes the device of Clause 16, wherein the model includes a trained model uploaded to the library from another user device.

Clause 18 includes the device of Clause 16 or Clause 17, wherein the library corresponds to a crowdsourced library of models.

Clause 19 includes the device of any of Clauses 16 to Clause 18, wherein the library is included in a distributed context-aware system.

Clause 20 includes the device of any of Clauses 1 to 19, wherein the context includes a particular acoustic environment, and wherein the one or more processors are configured to: determine whether a library of available acoustic models includes an acoustic model that is specific to the particular acoustic environment and available to the one or more processors; and in response to no acoustic model that is specific to the particular acoustic environment being available to the one or more processors, determine whether an acoustic model for a general category of the particular acoustic environment is available to the one or more processors.

Clause 21 includes the device of any of Clauses 1 to 20, wherein the one or more processors are integrated in integrated circuit.

Clause 22 includes the device of any of Clauses 1 to 20, wherein the one or more processors are integrated in a vehicle.

Clause 23 includes the device of any of Clauses 1 to 20, wherein the one or more processors are integrated in at least one of a mobile phone, a tablet computer device, a virtual reality headset, an augmented reality headset, a mixed reality headset, a wireless speaker device, a wearable device, a camera device, or a hearing aid device.

Particular aspects of the disclosure are described below in a second set of interrelated clauses:

According to Clause 24, a method includes receiving, at one or more processors of a device, sensor data from one or more sensor devices; determining, at the one or more processors, a context of the device based on the sensor data; selecting, at the one or more processors, a model based on the context; and processing, at the one or more processors, an input signal using the model to generate a context-specific output.

Clause 25 includes the method of Clause 24, wherein the sensor data includes location data of a location of the device, and wherein the context is at least partially based on the location.

Clause 26 includes the method of Clause 24 or Clause 25, wherein the sensor data includes image data corresponding to a visual scene, and wherein the context is at least partially based on the visual scene.

Clause 27 includes the method of any of Clauses 24 to 26, wherein the sensor data includes audio corresponding to an audio scene, and wherein the context is at least partially based on audio scene.

Clause 28 includes the method of any of Clauses 24 to 27, wherein the sensor data includes motion data corresponding to motion of the device, and wherein the context is at least partially based on the motion of the device.

Clause 29 includes the method of any of Clauses 24 to 28, wherein the model is selected from among multiple models stored at a memory of the device.

Clause 30 includes the method of any of Clauses 24 to 29, wherein the model includes a sound event detection model, the input signal includes an audio signal, and the context-specific output includes a classification of a sound event in the audio signal.

Clause 31 includes the method of any of Clauses 24 to 30, wherein the model includes an automatic speech recognition model, the input signal includes an audio signal, and the context-specific output includes text data representative of speech in the audio signal.

Clause 32 includes the method of any of Clauses 24 to 31, wherein the model includes a natural language processing (NLP) model, the input signal includes text data, and the context-specific output includes NLP output data based on the text data.

Clause 33 includes the method of any of Clauses 24 to 32, wherein the model includes a noise reduction model, the input signal includes an audio signal, and the context-specific output includes a noise reduced audio signal based on the audio signal.

Clause 34 includes the method of any of Clauses 24 to 33, wherein the model is associated with an automatic adjustment of a device operating mode, and wherein the context-specific output includes a signal to adjust the device operating mode.

Clause 35 includes the method of any of Clauses 24 to 34, and further includes receiving the model from a second device via wireless transmission.

Clause 36 includes the method of any of Clauses 24 to 35, wherein the context corresponds to a location of the device, and wherein the model includes an acoustic model corresponding to a particular location.

Clause 37 includes the method of Clause 36, and further includes receiving an access permission for the model at least partially based on the location of the device matching the particular location.

Clause 38 includes the method of any of Clauses 24 to 37, and further includes pruning the model in response to determining that the context has changed.

Clause 39 includes the method of any of Clauses 24 to 38, wherein the model is downloaded from a library of shared models.

Clause 40 includes the method of Clause 39, wherein the model includes a trained model uploaded to the library from another user device.

Clause 41 includes the method of Clause 39 or Clause 40, wherein the library corresponds to a crowdsourced library of models.

Clause 42 includes the method of any of Clauses 39 to 41, wherein the library is included in a distributed context-aware system.

Clause 43 includes the method of any of Clauses 24 to 42, wherein the context includes a particular acoustic environment, and further includes: determining whether a library of available acoustic models includes an acoustic model that is specific to the particular acoustic environment and available to the device; and in response to no acoustic model that is specific to the particular acoustic environment being available to the device, determining whether an acoustic model for a general category of the particular acoustic environment is available to the device.

Particular aspects of the disclosure are described below in a third set of interrelated clauses:

According to Clause 44, a device includes means for receiving sensor data; means for determining a context based on the sensor data; means for selecting a model based on the context; and means for processing an input signal using the model to generate a context-specific output.

Clause 45 includes the device of Clause 44, wherein the sensor data includes location data of a location of the device, and wherein the context is at least partially based on the location.

Clause 46 includes the device of Clause 44 or Clause 45, wherein the sensor data includes image data corresponding to a visual scene, and wherein the context is at least partially based on the visual scene.

Clause 47 includes the device of any of Clauses 44 to 46, wherein the sensor data includes audio corresponding to an audio scene, and wherein the context is at least partially based on audio scene.

Clause 48 includes the device of any of Clauses 44 to 47, wherein the sensor data includes motion data corresponding to motion of the device, and wherein the context is at least partially based on the motion of the device.

Clause 49 includes the device of any of Clauses 44 to 48, and further includes means for storing models, wherein the model is selected from among multiple models stored at the means for storing models.

Clause 50 includes the device of any of Clauses 44 to 49, wherein the model includes a sound event detection model, the input signal includes an audio signal, and the context-specific output includes a classification of a sound event in the audio signal.

Clause 51 includes the device of any of Clauses 44 to 50, wherein the model includes an automatic speech recognition model, the input signal includes an audio signal, and the context-specific output includes text data representative of speech in the audio signal.

Clause 52 includes the device of any of Clauses 44 to 51, wherein the model includes a natural language processing (NLP) model, the input signal includes text data, and the context-specific output includes NLP output data based on the text data.

Clause 53 includes the device of any of Clauses 44 to 52, wherein the model includes a noise reduction model, the input signal includes an audio signal, and the context-specific output includes a noise reduced audio signal based on the audio signal.

Clause 54 includes the device of any of Clauses 44 to 53, wherein the model is associated with an automatic adjustment of a device operating mode, and wherein the context-specific output includes a signal to adjust the device operating mode.

Clause 55 includes the device of any of Clauses 44 to 54, wherein the model is received from a second device via wireless transmission.

Clause 56 includes the device of any of Clauses 44 to 55, wherein the context corresponds to a location of the device, and wherein the model includes an acoustic model corresponding to a particular location.

Clause 57 includes the device of Clause 56, wherein an access permission for the model is received at least partially based on the location of the device matching the particular location.

Clause 58 includes the device of Clause 56, and further includes means for removing the model in response to the device leaving the particular location.

Clause 59 includes the device of any of Clauses 44 to 58, wherein the model is downloaded from a library of shared models.

Clause 60 includes the device of Clause 59, wherein the model includes a trained model uploaded to the library from another user device.

Clause 61 includes the device of Clause 59 or Clause 60, wherein the library corresponds to a crowdsourced library of models.

Clause 62 includes the device of any of Clauses 59 to 61, wherein the library is included in a distributed context-aware system.

Particular aspects of the disclosure are described below in a fourth set of interrelated clauses:

According to Clause 63, a non-transitory computer-readable storage medium comprising instructions that, when executed by a processor of a device, cause the processor to: receive sensor data from one or more sensor devices; determine a context on the sensor data; select a model based on the context; and process an input signal using the model to generate a context-specific output.

Clause 64 includes the non-transitory computer-readable storage medium of Clause 63, wherein the sensor data includes location data of a location of the device, and wherein the context is at least partially based on the location.

Clause 65 includes the non-transitory computer-readable storage medium of Clause 63 or Clause 64, wherein the sensor data includes image data corresponding to a visual scene, and wherein the context is at least partially based on the visual scene.

Clause 66 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 65, wherein the sensor data includes audio corresponding to an audio scene, and wherein the context is at least partially based on audio scene.

Clause 67 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 66, wherein the sensor data includes motion data corresponding to motion of the device, and wherein the context is at least partially based on the motion of the device.

Clause 68 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 67, wherein the model is selected from among multiple models stored at a memory of the device.

Clause 69 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 68, wherein the model includes a sound event detection model, the input signal includes an audio signal, and the context-specific output includes a classification of a sound event in the audio signal.

Clause 70 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 69, wherein the model includes an automatic speech recognition model, the input signal includes an audio signal, and the context-specific output includes text data representative of speech in the audio signal.

Clause 71 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 70, wherein the model includes a natural language processing (NLP) model, the input signal includes text data, and the context-specific output includes NLP output data based on the text data.

Clause 72 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 71, wherein the model includes a noise reduction model, the input signal includes an audio signal, and the context-specific output includes a noise reduced audio signal based on the audio signal.

Clause 73 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 72, wherein the model is associated with an automatic adjustment of a device operating mode, and wherein the context-specific output includes a signal to adjust the device operating mode.

Clause 74 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 73, wherein the instructions further cause the processor to receive the model from a second device via wireless transmission.

Clause 75 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 74, wherein the context corresponds to a location of the device, and wherein the model includes an acoustic model corresponding to a particular location.

Clause 76 includes the non-transitory computer-readable storage medium of Clause 75, wherein the instructions further cause the processor to receive an access permission for the model at least partially based on the location of the device matching the particular location.

Clause 77 includes the non-transitory computer-readable storage medium of Clause 75, wherein the instructions further cause the processor to remove the model in response to the device leaving the particular location.

Clause 78 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 77, wherein the model is downloaded from a library of shared models.

Clause 79 includes the non-transitory computer-readable storage medium of Clause 78, wherein the model includes a trained model uploaded to the library from another user device.

Clause 80 includes the non-transitory computer-readable storage medium of Clause 78 or Clause 79, wherein the library corresponds to a crowdsourced library of models.

Clause 81 includes the non-transitory computer-readable storage medium of any of Clauses 78 to 80, wherein the library is included in a distributed context-aware system.

Clause 82 includes the non-transitory computer-readable storage medium of any of Clauses 63 to 81, wherein the context includes a particular acoustic environment, and wherein the instructions further cause the processor to: determine whether a library of available acoustic models includes an acoustic model that is specific to the particular acoustic environment and available to the device; and in response to no acoustic model that is specific to the particular acoustic environment being available to the device, determine whether an acoustic model for a general category of the particular acoustic environment is available to the device.

Particular aspects of the disclosure are described below in a fifth set of interrelated clauses:

According to Clause 83, a device includes one or more processors configured to: select an acoustic model corresponding to a particular room, of a building, in which the device is located; and process an input audio signal using the acoustic model.

Clause 84 includes the device of Clause 83, wherein the one or more processors are configured to download the acoustic model from a library of acoustic models in response to a determination that the device has entered the particular room.

Clause 85 includes the device of Clause 83 or 84, wherein the one or more processors are further configured to remove the acoustic model in response to the device leaving the particular room.

Clause 86 includes the device of any of Clauses 83 to 85, and further includes one or more microphones configured to generate the input audio signal.

Clause 87 includes the device of any of Clauses 83 to 86, and further includes one or more sensor devices coupled to the one or more processors and configured to generate sensor data indicative of a location of the device, and wherein the one or more processors are configured to select the acoustic model based on the sensor data.

Clause 88 includes the device of any of Clause 83 to 87, and further includes a modem coupled to the one or more processors and configured to receive location data indicative of a location of the device, and wherein the one or more processors are configured to select the acoustic model based on the location data.

Particular aspects of the disclosure are described below in a sixth set of interrelated clauses:

According to Clause 89, a method includes: selecting, at one or more processors of a device, an acoustic model corresponding to a particular room, of a building, in which the device is located; and processing, at the one or more processors, an input audio signal using the acoustic model.

Clause 90 includes the method of Clause 89, and further includes downloading the acoustic model from a library of acoustic models in response to a determination that the device has entered the particular room.

Clause 91 includes the method of Clause 89 or Clause 90, and further includes removing the acoustic model in response to the device leaving the particular room.

Clause 92 includes the method of any of Clauses 89 to 91, and further includes selecting the acoustic model based on sensor data indicative of a location of the device.

Clause 93 includes the method of any of Clauses 89 to 91, and further includes selecting the acoustic model based on location data indicative of a location of the device.

According to Clause 94, a device includes: means for selecting an acoustic model corresponding to a particular room, of a building, in which the device is located; and means for processing an input audio signal using the acoustic model.

According to Clause 95, a non-transitory computer-readable storage medium includes instructions that, when executed by a processor of a device, cause the processor to: select an acoustic model corresponding to a particular room, of a building, in which the device is located; and process an input audio signal using the acoustic model.

Particular aspects of the disclosure are described below in a seventh set of interrelated clauses:

According to Clause 96, a device includes: one or more processors configured to: in response to the device entering a vehicle, select a personalized acoustic model for a user of the device from among multiple personalized acoustic models corresponding to the vehicle; and process an input audio signal using the personalized acoustic model.

Clause 97 includes the device of Clause 96, wherein the one or more processors are configured to download the personalized acoustic model from a library of acoustic models in response to a determination that the device has entered the vehicle.

Clause 98 includes the device of Clause 96 or Clause 97, wherein the one or more processors are further configured to remove the personalized acoustic model in response to the device leaving the vehicle.

Clause 99 includes the device of any of Clauses 96 to 98, and further includes one or more microphones configured to generate the input audio signal.

Clause 100 includes the device of any of Clauses 96 to 99, and further includes one or more sensor devices coupled to the one or more processors and configured to generate sensor data indicative of a location of the device, and wherein the one or more processors are configured to determine the device has entered the vehicle based on the sensor data.

Clause 101 includes the device of any of Clauses 96 to 100, and further includes a modem coupled to the one or more processors and configured to receive location data indicative of a location of the device, and wherein the one or more processors are configured to determine the device has entered the vehicle based on the location data.

Particular aspects of the disclosure are described below in an eighth set of interrelated clauses:

According to Clause 102, a method includes: selecting, at one or more processors of a device and in response to detecting a user entering a vehicle, a personalized acoustic model for the user from among multiple personalized acoustic models corresponding to the vehicle; and processing, at the one or more processors, an input audio signal using the personalized acoustic model.

Clause 103 includes the method of Clause 102, and further includes downloading the personalized acoustic model from a library of acoustic models in response to a determination that the user has entered the vehicle.

Clause 104 includes the method of Clause 102 or Clause 103, and further includes removing the personalized acoustic model in response to the user leaving the vehicle.

Clause 105 includes the method of any of Clauses 102 to 104, and further includes determining the user has entered the vehicle based on sensor data indicative of a location of the user.

According to Clause 106, a device includes: means for selecting, in response to detecting a user entering a vehicle, a personalized acoustic model for the user from among multiple personalized acoustic models corresponding to the vehicle; and means for processing an input audio signal using the personalized acoustic model.

According to Clause 107, a non-transitory computer-readable storage medium includes instructions that, when executed by a processor of a device, cause the processor to: select, in response to detecting a user entering a vehicle, a personalized acoustic model for the user from among multiple personalized acoustic models corresponding to the vehicle; and process an input audio signal using the personalized acoustic model.

Particular aspects of the disclosure are described below in a ninth set of interrelated clauses:

According to Clause 108, a device includes: one or more processors configured to: download an acoustic model corresponding to a particular location in which the device is located; process an input audio signal using the acoustic model; and remove the acoustic model in response to the device exiting the location.

Clause 109 includes the device of Clause 108, wherein the location corresponds to a particular restaurant, and wherein the acoustic model is downloaded from a library of acoustic models in response to a determination that the device has entered the particular restaurant.

Clause 110 includes the device of Clause 108 or 109, and further includes one or more microphones configured to generate the input audio signal.

Clause 111 includes the device of any of Clauses 108 to 110, and further includes one or more sensor devices coupled to the one or more processors and configured to generate sensor data indicative of a location of the device, and wherein the one or more processors are configured to determine the device has entered the particular location based on the sensor data.

Clause 112 includes the device of any of Clauses 108 to 110, and further includes a modem coupled to the one or more processors and configured to receive location data indicative of a location of the device, and wherein the one or more processors are configured to determine the device has entered the particular location based on the location data.

Particular aspects of the disclosure are described below in a tenth set of interrelated clauses:

According to Clause 113, a method includes: downloading, at one or more processors of a device, an acoustic model corresponding to a particular location in which the device is located; processing, at the one or more processors, an input audio signal using the acoustic model; and removing, at the one or more processors, the acoustic model in response to the device exiting the location.

Clause 114 includes the method of Clause 113, wherein the location corresponds to a particular restaurant, and wherein the acoustic model is downloaded from a library of acoustic models in response to a determination that the device has entered the particular restaurant.

Clause 115 includes the method of Clause 113 or Clause 114, and further includes determining that the device has entered the particular location based on sensor data indicative of a location of the device.

Clause 116 includes the method of Clause 113 or 114, and further includes determining that the device has entered the particular location based on location data indicative of a location of the device.

According to Clause 117, a device includes: means for downloading an acoustic model corresponding to a particular location in which the device is located; means for processing an input audio signal using the acoustic model; and means for removing the acoustic model in response to the device exiting the location.

According to Clause 118, a non-transitory computer-readable storage medium includes instructions that, when executed by a processor of a device, cause the processor to: download an acoustic model corresponding to a particular location in which the device is located; process an input audio signal using the acoustic model; and remove the acoustic model in response to the device exiting the location.

Particular aspects of the disclosure are described below in an eleventh set of interrelated clauses:

According to Clause 119, a device includes: one or more processors configured to: select an acoustic model corresponding to a particular location; receive an access permission for the acoustic model at least partially based on a location of the device matching the particular location; and process an input audio signal using the acoustic model.

Clause 120 includes the device of Clause 119, and further includes a modem, and wherein the one or more processors are further configured to receive the access permission via the modem in response to detection of the device within the particular location.

Particular aspects of the disclosure are described below in a twelfth set of interrelated clauses:

According to Clause 121, a method includes: selecting, at the one or more processors of a device, an acoustic model corresponding to a particular location; receiving, at the one or more processors, an access permission for the acoustic model at least partially based on a location of the device matching the particular location; and processing, at the one or more processors, an input audio signal using the acoustic model.

Clause 122 includes the method of Clause 121, and further includes receiving the access permission in response to detection of the device within the particular location.

According to Clause 123, a device includes: means for selecting an acoustic model corresponding to a particular location; means for receiving an access permission for the acoustic model at least partially based on a location of the device matching the particular location; and means for processing an input audio signal using the acoustic model.

According to Clause 124, a non-transitory computer-readable storage medium includes instructions that, when executed by a processor of a device, cause the processor to: select an acoustic model corresponding to a particular location; receive an access permission for the acoustic model at least partially based on a location of the device matching the particular location; and process an input audio signal using the acoustic model.

Particular aspects of the disclosure are described below in a thirteenth set of interrelated clauses:

According to Clause 125, a device includes one or more processors configured to: detect a context of the device; send a request that indicates the context to a remote device; receive a model corresponding to the context; use the model while the context remains detected; and prune the model in response to detecting a change of the context.

Clause 126 includes the device of Clause 125, wherein the one or more processors are further configured to generate at least one new sound class while the context remains detected, and wherein prune the model includes preserving the at least one new sound class.

Clause 127 includes the device of Clause 125 or Clause 126, wherein prune the model includes permanently deleting the model.

Clause 128 includes the device of any of Clauses 125 to 127, wherein the one or more processors are further configured to receive the model based on private access.

According to Clause 129, a method includes: detecting, at one or more processors of a device, a context of the device; sending a request that indicates the context to a remote device; receiving a model corresponding to the context; using, at the one or more processors, the model while the context remains detected; and pruning, at the one or more processors, the model in response to detecting a change of the context.

Clause 130 includes the method of Clause 129, further comprising generating at least one new sound class while the context remains detected, and wherein pruning the model includes preserving the at least one new sound class.

Clause 131 includes the method of Clause 129 or Clause 130, wherein pruning the model includes permanently deleting the model.

Clause 132 includes the method of any of Clauses 129 to 131, wherein the model is received based on private access.

According to Clause 133, an apparatus includes: means for detecting a context of a device; means for sending a request that indicates the context to a remote device; means for receiving a model corresponding to the context; means for using the model while the context remains detected; and means for pruning the model in response to detecting a change of the context.

According to Clause 134, a non-transitory computer-readable storage medium includes instructions that, when executed by a processor of a device, cause the processor to: detect a context of a device; send a request that indicates the context to a remote device; receive a model corresponding to the context; use the model while the context remains detected; and prune the model in response to detecting a change of the context.

The previous description of the disclosed aspects is provided to enable a person skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.