Extraction and analysis of buffered audio data using multiple codec rates each greater than a low-power processor rate

A processor is configured to transition in and out of a low-power state at a first rate and to operate in a first mode or a second mode. In a particular method, the processor while coupled to a coder/decoder (CODEC) retrieves audio feature data from a buffer after transitioning out of the low-power state. The CODEC is configured to operate at a second rate in the first mode and at a third rate in the second mode, the second rate and the third rate each greater than the first rate. The audio feature data indicates features of audio data received during the low-power state of the processor. A ratio of CODEC activity to processor activity in the second mode is less than the ratio in the first mode.

The present disclosure is generally related to extraction and analysis of audio feature data.

III. DESCRIPTION OF RELATED ART

Advances in technology have resulted in smaller and more powerful computing devices. For example, there currently exist a variety of portable personal computing devices, including wireless computing devices, such as portable wireless telephones, personal digital assistants (PDAs), and paging devices that are small, lightweight, and easily carried by users. More specifically, portable wireless telephones, such as cellular telephones and Internet Protocol (IP) telephones, can communicate voice and data packets over wireless networks. Further, many such wireless telephones include other types of devices that are incorporated therein. For example, a wireless telephone can also include a digital still camera, a digital video camera, a digital recorder, and an audio file player.

As the number of devices incorporated into a wireless telephone increases, battery resources at the wireless telephone may become scarcer. To conserve battery resources, a wireless telephone may transition into an “idle” or “sleep” mode after a period of inactivity. The wireless telephone may transition back into an “active” or “wake” mode in response to a network event (e.g., receiving a telephone call) or user input (e.g., a user pushing a button of the wireless telephone). Some devices may also include the ability to “wake up” in response to audio input, such as voice commands. However, to implement such functionality, processor(s) and other components of a device may run in an “always on” mode and may continuously consume power, which may decrease an overall battery life of the device.

A low-power system and method of extracting and analyzing audio feature data is disclosed. For example, the techniques disclosed herein may enable sound-sensing functionality in an electronic device (e.g., wireless telephone) with reduced power consumption. The electronic device may include a low-power coder/decoder (CODEC) coupled to a processor (e.g., an audio digital signal processor (DSP)). The system may have multiple operational modes, each mode corresponding to a different ratio of CODEC activity to processor activity. For example, in a first mode, the CODEC may operate continuously and the processor may be duty-cycled at a first rate. For example, the processor may operate in accordance with a 10% duty cycle (i.e., active 10% of the time and idle 90% of the time). In a second mode, the CODEC may also be duty-cycled. The CODEC may be duty-cycled at different rates in different modes. In some modes, the CODEC's activity may be greater than or equal to the processor's activity. In other modes, such as when the processor has a heavy computational load, the processor's activity may be greater than the CODEC's activity. The CODEC may receive audio data (e.g., from a microphone of the device) and extract audio features from the audio data. The processor may analyze the audio features and may perform one or more actions based on the analysis. For example, the processor may activate one or more other components of the electronic device based on the analysis.

In a particular embodiment, a method includes transitioning out of a low-power state at a processor. The method also includes the processor retrieving audio feature data from a buffer after transitioning out of the low-power state. The audio feature data indicates features of audio data received during the low-power state of the processor. In some embodiments, the audio data may have been received and the audio feature data may have been extracted by a CODEC coupled to the processor while the processor was in the low-power state.

In another particular embodiment, a method includes receiving a frame of audio data at a CODEC. The method also includes extracting audio feature data from the frame of audio data. The method further includes storing the extracted audio feature data in a buffer to be accessible by a duty-cycled processor during an active state of the duty-cycled processor.

In another particular embodiment, an apparatus includes a processor and a plurality of filters configured to filter one or more frames of audio data to produce energies of filtered audio data (independently of whether the processor is in a low-power state or in an active state). The apparatus also includes a converter configured to generate audio feature data based on the energies of the filtered audio data. The apparatus further includes a transformer configured to apply a transform function to the audio feature data to generate transformed data. The processor is configured to perform one or more operations on the transformed data after transitioning out of the low-power state to the active state.

In another particular embodiment, an apparatus includes a processor configured to dynamically switch between operating in a first mode and operating in a second mode based on an application context of the processor. The processor is also configured to retrieve and process audio feature data from a buffer after transitioning out of a low-power state. The audio feature data indicates features of audio data received by a CODEC while the processor is in the low-power state. A ratio of CODEC activity to processor activity in the first mode is greater than a ratio of CODEC activity to processor activity in the second mode.

In another particular embodiment, a non-transitory processor-readable medium includes instructions that, when executed by a processor, cause the processor to dynamically switch between operating in a first mode and operating in a second mode. A ratio of CODEC activity to processor activity in the first mode is greater than a ratio of CODEC activity to processor activity in the second mode. The instructions, when executed, also cause the processor to transition out of a lower-power state during a duty cycle and to analyze audio feature data that is extracted during the low-power state. The instructions, when executed, further cause the processor to transition back into the low-power state.

Particular advantages provided by at least one of the disclosed embodiments include an ability of an electronic device to extract and analyze audio feature data by use of an always on low-power CODEC (or a duty-cycled CODEC) and a duty-cycled processor. For example, the audio feature data may indicate characteristics of audio data received by the CODEC while the duty-cycled processor is in a low-power state. The extraction and analysis of the audio feature data may be performed with reduced power consumption compared to systems that include an always on CODEC and an always on audio processor. The analysis of the audio feature data may trigger various operations, such as activating a touchscreen or other component of the electronic device.

VI. DETAILED DESCRIPTION

Referring toFIG. 1, a particular embodiment of a system that is operable to extract and analyze audio feature data is shown and generally designated100. The system100includes a coder/decoder (CODEC)120coupled to a processor150. In a particular embodiment, the processor150may be a digital signal processor (DSP), such as an audio DSP. In some embodiments, a buffer140may be located between the CODEC120and the processor150, as shown. In alternate embodiments, the buffer140may be internal to the CODEC120or the processor150, as further described with reference toFIGS. 2-3.

In a particular embodiment, the CODEC120may operate continuously and receive audio data110. For example, the audio data110may be generated by a microphone or other sound input device coupled to the CODEC120. The audio data110may be “raw” (i.e., unprocessed and/or uncompressed) audio data. The CODEC120may be configured to extract audio features from the audio data110, thereby generating audio feature data130. In a particular embodiment, the audio feature data130may be substantially smaller in size than the audio data110. The CODEC120may store the audio feature data130in the buffer140(e.g., a random access memory (RAM) buffer). In a particular embodiment, the audio feature data130may indicate particular characteristics of the audio data110, such as pitch, tone, volume, and/or rhythmic characteristics. The CODEC120may also discard the audio data110after extracting the audio feature data130.

The processor150may operate in accordance with a duty cycle. To illustrate, if the processor150operates in accordance with a 10% duty cycle, the processor150is “active” (i.e., in a high-power state) 10% of the time and is “idle” (i.e., in a low-power state) 90% of the time. In a particular embodiment, the processor150may periodically transition between the active state and the idle state in response to expiration of a programmable time period (e.g., the duty cycle of the processor150may be programmable). The duty-cycled processor150may thus consume less power than an “always on” processor.

After transitioning out of the low-power state, the processor150may retrieve the audio feature data130from the buffer140and analyze the retrieved audio feature data130. The processor150may perform one or more operations based on a result of the analysis. For example, when the system100is integrated into an electronic device, such as a wireless telephone, the processor150may generate an activation signal160based on the analysis of the audio feature data130to activate one or more components of the electronic device (e.g., an application processor or a portion of a mobile station modem (MSM), as further described with reference toFIG. 10).

During operation, the CODEC120may continuously receive frames of the audio data110and store the audio feature data130extracted from the audio data110in the buffer140. For example, each frame of the audio data110may be 20 ms long. In a particular embodiment, newer audio feature data130may overwrite older audio feature data130in the buffer140in accordance with a first-in-first-out policy.

It should be noted that instead of operating continuously as depicted inFIG. 1, the CODEC120may instead be duty-cycled. For example, if the CODEC120is less power-efficient than desired or is a “legacy” CODEC, the CODEC120may be duty-cycled. Generally, even though the CODEC120is duty-cycled, the CODEC120may be more active than the processor150. Thus, the system100may support multiple operational modes. In a first mode, the CODEC120may perform more frequent audio signal processing and may presumably consume more power. In a second mode, the CODEC120may perform less frequent audio signal processing and may presumably consume less power. The processor150may have the same duty cycle in the first mode and in the second mode.

It will be appreciated that various implementations may be supported by the dual-mode (or multi-mode) system100, each mode having a different ratio of CODEC activity to processor activity. For example, a higher activity mode may involve the CODEC120operating continuously and the processor150duty-cycled at a first rate (e.g., D1), and a lower activity mode may involve the CODEC120duty-cycled at a second rate (e.g., D2) that is greater than or equal to the first rate (e.g., D2>=D1). As another example, the higher activity mode may involve the CODEC120duty-cycled at a first rate (e.g., D1) and the processor150duty-cycled at a second rate (e.g., D2), and the lower activity mode may involve the CODEC120duty-cycled at a third rate (e.g., D3) and the processor150duty-cycled at the second rate (e.g., D2). The first rate may be substantially greater than the second rate (e.g., D1>>D2) and the third rate may be greater than or equal to the second rate (e.g., D3>=D2). Selected implementations may also support modes in which CODEC activity is less than or equal to processor activity, such as during periods of heavy processor computational load. For example, the third rate may be less than or equal to the second rate (e.g., D3<=D2).

Depending on how frequently the CODEC120and the processor150are active, the system100may be effectively working in a store-and-forward mode or in a direct transfer mode. In the store-and-forward mode, the processor150may empty the buffer140upon transitioning out of the low-power state. That is, the processor150may retrieve audio feature data130corresponding to every frame (or multiple frames) of audio data110received by the CODEC120while the processor150was in the low-power mode. In the direct transfer mode, the processor150may retrieve audio feature data130corresponding to a single frame of the audio data110(e.g., a most recently received frame of the audio data110). In a particular embodiment, the processor150may dynamically switch between operating in the store-and-forward mode and in the direct transfer mode, and/or between a higher activity mode and a lower activity mode (where the higher activity mode has a higher CODEC activity to processor activity ratio than the lower activity mode) based on an application context of the processor150, as further described with reference toFIGS. 2 and 4.

After retrieving the audio feature data130, the processor150may analyze the audio feature data130and may generate the activation signal160based on the analysis. For example, when the analysis of the audio feature data130identifies a particular voice input command (e.g., “wake up”), the processor150may generate the activation signal160to activate various components of an electronic device.

The system100ofFIG. 1, which includes a duty-cycled processor, may thus enable audio feature extraction and analysis at lower power than a system having an always-on CODEC and an always-on processor. Further, by buffering audio features instead of raw audio data, the system100ofFIG. 1may perform audio analysis with a reduced amount of memory usage.

Referring toFIG. 2, another particular embodiment of a system that is operable to extract and analyze audio feature data is shown and generally designated200. The system200may include a CODEC220(e.g., the CODEC120ofFIG. 1) coupled to a processor230(e.g., the processor150ofFIG. 1). The CODEC220may also be coupled to a sound input device, such as an illustrative microphone210.

The CODEC220may include an analog-to-digital converter (ADC)221that receives analog audio data212from the microphone210and converts the analog audio data212into digital audio data. In an alternate embodiment where the microphone210produces digital audio data, the ADC may not be present.

The CODEC220may also include a feature extractor222configured to extract audio features226from the audio data212. In a particular embodiment, the feature extractor222may include a plurality of filters223that filter the audio data212to generate energies224(e.g., mel-band energies) of filtered audio data. For example, the filters223may be mel-band filters, where each mel-band filer corresponds to a different portion of a human perception frequency scale (e.g., octave). To illustrate, the filters223may include 22 mel-band filters that generate mel-band energies224corresponding to 22 octaves. In an alternate embodiment, the feature extractor222may perform fast Fourier transform (FFT)-based feature extraction.

The feature extractor222may also include a log converter225. The log converter225may apply a logarithm function to the energies224of the filtered audio data to generate the extracted audio features226. The extracted audio features226may be stored in a buffer (e.g., RAM buffer)227. The extracted audio features226may be substantially smaller in size than the audio data212with compactly designed audio features (e.g., 22 log mel-band energies from each 20 ms frame). To illustrate, the audio data212may have a 16 kHz, 16 bit resolution. 200 ms (e.g., corresponding to 10 frames) of the audio data212may occupy 6400 bytes of space. However, extracted audio features226for the 10 frames may occupy only 220 bytes of space (10 frames×22 features per frame×1 byte per feature). Thus, by storing the extracted audio features226instead of the raw audio data212in the buffer227, the buffer227may be kept relatively small and may consume relatively less power.

The processor230may include state transition logic231. In a particular embodiment, the state transition logic231may transition the processor230in and out of a low-power state (e.g., in accordance with a duty cycle). Upon transitioning out of the low-power state, the processor230may retrieve the extracted audio features226from the buffer227. A transformer233may apply a transform function to the extracted audio features226to generate transformed audio feature data234. In a particular embodiment, the transformer233may be configured to apply a discrete cosine transform (DCT) function. To illustrate, transforming the extracted audio features226, where the extracted audio features226include features corresponding to 22 mel-bands per frame, may generate 12 mel-frequency cepstral coefficients (MFCCs) per frame by taking 12 elements of DCT coefficients.

The processor230may also include one or more sound recognition modules241-245configured to analyze the transformed audio feature data234. In a particular embodiment, which sound recognition modules241-245are active may depend on what mode the processor230is operating in. To illustrate, dynamic mode-switching logic232at the processor230may dynamically switch operation of the processor230based on context (e.g., application context). For example, when a device including the system200ofFIG. 2executes an application or other operation that involves listen location, continuous audio fingerprinting, and/or continuous keyword detection, the logic232may cause the processor230to operate in a store-and-forward mode (e.g., in which features from multiple frames of audio data are processed each time the processor230is active) and the modules241-243may be active. As another example, when the device executes an application that involves target sound detection (e.g., detection of specific music or speech) and/or novelty detection, the logic232may cause the processor230to operate in either the store-and-forward-mode or in a direct transfer mode (e.g., in which features from a single frame of audio data are processed each time the processor is active), and the modules244-245may be active. In alternate embodiments, the dynamic mode-switching logic232may switch operation of the processor230based on other factors, including, for example, characteristics of the audio data212and/or the audio features226.

The listen location module241may convert input sound into audio signatures. The signatures may be sent to a server (not shown), and the server may compare the signatures to signatures received from other devices. If signatures from different devices are similar, the server may determine that the different devices are in the same acoustical space, which may indicate that the different devices are in the same physical location, listening to the same content, or have a similar context as determined by surrounding sound. For example, listen location may be used in a social network service to group people and/or share an item with a group of people.

The continuous audio fingerprinting module242may attempt to detect the existence of pre-enrolled (e.g., predetermined) sound snapshots. Unlike target sound or environment detection, continuous audio fingerprinting may robustly detect perceptually identical sound snapshots in the presence of sound-quality distortions, such as distortion related to channel degradation, equalization, speed change, digital-to-analog or analog-to-digital conversion, etc. Continuous audio fingerprinting may thus find application in music and broadcast identification scenarios.

The continuous keyword detection module243may receive sound input and may detect the existence of pre-enrolled (e.g., predetermined) keyword sets. Continuous keyword detection may be performed in a relatively low-power state and may activate predefined applications based on detected keywords. The predetermined keyword sets may be programmable by an application processor. In a particular embodiment, models for keywords may be downloaded by the application processor. Continuous keyword detection may thus enable voice-activation commands without the use of a dedicated voice command button or non-verbal user input.

The target sound detection module244may detect a type of sound and may notify corresponding applications to respond to the sound. For example, upon detecting speech, target sound detection may cause a voice recording application to record the speech. As another example, upon detecting music, target sound detection may cause an application to identify properties of the music, such as song title, artist name, and album name.

The novelty detection module245may detect changes in input audio that correspond to changes in location and/or changes in activity. Novelty detection may be used in conjunction with other sound recognition operations (e.g., listen location and target sound detection) to identify location and sound activity, and to log the corresponding time for subsequent usage and analysis. Novelty detection may also be used to activate other sound recognition operations when there is a noticeable change in environmental acoustics.

During operation, the CODEC220may continuously receive frames of the audio data212from the microphone, extract the audio features226from the audio data212, and store the audio features226in the buffer227. The processor230may transition in and out of a low-power state in accordance with a duty cycle. After transitioning out of the low-power state, the processor230may retrieve and transform audio features226corresponding to a plurality of frames of the audio data212(when operating in the store-and-forward mode) or corresponding to a single frame of audio data212(when operating in the direct transfer mode). The processor230may also transition between operating in a higher activity mode and in a lower activity mode, as described with reference toFIG. 1. When active, the processor230may analyze the transformed audio feature data234via one or more of the sound recognition modules241-245, and may determine whether to activate an application processor and/or component(s) of a mobile station modem (MSM) or other component based on the analysis.

In a particular embodiment, the system200ofFIG. 2may provide a common listening service that can serve multiple higher-level applications (e.g., a music recognition application, a keyword detection application, etc.). For example, the common listening service may provide (e.g., via an application programming interface (API), shared memory, etc.) higher-level applications with the results of sound recognition operations performed by the processor230. The common listening service may reduce interoperability issues and may be more power-efficient than systems in which each higher-level application has its own listening engine.

The system200ofFIG. 2may thus enable audio feature extraction and analysis with reduced power consumption. For example, relatively low-power operations, such as analog-to-digital conversion and feature extraction, may be incorporated into a low-power always-on CODEC (or a duty-cycled CODEC), and higher-power operations, such as data transformation and sound recognition, may be incorporated into a duty-cycled DSP and may be performed intermittently.

In a particular embodiment, the system200ofFIG. 2may provide a low-power user interface at an electronic device that includes activation of high-power components by low-power components. To illustrate, the system200may support audible (e.g., 0-16 kHz sampling rate), beacon (e.g., 16-24 kHz sampling rate), and ultrasound (e.g., >24 kHz sampling rate) input. To support multiple types of input, the microphone210may be capable of receiving audio, beacon, and ultrasound signals. Alternately, additional microphones or components may be incorporated into the system200for ultrasound and/or beacon detection. Components used to convert sound signals to electrical signals may include, but are not limited to, microphones, piezoelectric sensors, and ultrasound transducers. The low-power CODEC220may perform coarse detection/classification290on received signals. It should be noted that althoughFIG. 2illustrates the coarse detection/classification290being performed on the output of the analog-to-digital converter221, alternate embodiments may include performing the coarse detection/classification290on analog signals instead. Depending on the results of the coarse detection/classification290, the CODEC220may activate the higher-power processor230via an activation signal292. For example, the processor230may be activated if the coarse detection/classification290indicates that ultrasound input has been received.

It should be noted that althoughFIG. 2illustrates a two-level activation hierarchy (i.e., the CODEC220and the processor230), any number of levels may be implemented. For example, in a three level hierarchy, a low-power digital/analog circuit may perform coarse detection to determine whether to activate a higher-power front-end processing unit, and the front-end processing unit may perform fine detection to determine whether to activate an even higher-power main processing unit that performs final detection and executes applications/user interface components. In a particular embodiment, the digital/analog circuit and the front-end processing unit may be integrated into the CODEC220and the main processing unit may be integrated into the processor230. To illustrate, the coarse detection/classification block290may be integrated into a digital/analog circuit of the CODEC220and may selectively activate a fine detection/classification block296in a front-end unit of the CODEC220via a first activation signal294. The fine detection/classification block296may activate a final detection/classification block at the processor230via a second activation signal298. Staggered hierarchical activation of higher-power components by lower-power components may improve battery life at an electronic device.

Various detection and classification methods may be used at the system200, and more than one method may be used at once. In a particular embodiment, root mean square (RMS) or band-power classification may be used to determine whether a received signal includes data in the audio, beacon, and/or ultrasound ranges. A time domain method may include use of filter banks with signal level detection, where each filter is designed to extract a particular type of sound and where filter output levels are compared to thresholds to qualify sounds. A frequency domain method may include performing a FFT of mel-spaced cepstral coefficients to derive frequencies used to classify the input signal. A sound content method may involve pattern matching by correlating input signals with a known pattern (e.g., to determine whether input signals are received from an ultrasound digital stylus). A model-based approach may include computing a probability that the input signal matches a predetermined music or speech model. Novelty detection may involve detecting changes in input sound characteristics. When a change is detected, applications may be notified to update context information (e.g., whether a device is indoors or outdoors). For example, when a user goes from an indoor environment to an outdoor environment, the resulting change in input sound characteristics may result in an application at the user's mobile phone increasing a ringer volume of the phone.

Examples of use cases for the system200ofFIG. 2and/or components thereof include, but are not limited to: voice recognition to control devices (e.g., televisions, game consoles, computers, and phones), audio recognition for contextual awareness, acoustic and pulse recognition for a digital stylus (e.g., an ultrasound digital stylus for handwriting input to digital devices via transmission of ultrasound), ultrasound gesture or proximity detection, device-to-device positioning using ultrasound, acoustic touch detection, sound beacons to identify locations of devices, content identification by audio fingerprinting, peer discovery and proximity sensing by sound matching, and location estimation by sound matching.

It should be noted that althoughFIGS. 1-2depict feature extraction performed by a CODEC and data transformation performed by a processor, this is for illustration only. In alternate embodiments, different functionality may be performed by different hardware components. For example, referring toFIG. 3, particular embodiments of dividing operations between the CODEC220ofFIG. 2and the processor (e.g., DSP)230ofFIG. 2are shown and generally designated300.

In a first embodiment, the CODEC/DSP boundary may be located at302. In this first embodiment, the CODEC may include an ADC321and the output of the ADC321may be buffered. The DSP may perform feature extraction (e.g., via mel-band filters323and a log converter325), data transformation (e.g., via a DCT transformer333), and sound recognition (e.g., via sound recognition modules340).

In a second embodiment, the CODEC/DSP boundary may be located at304. Thus, in this second embodiment, feature extraction may be partially performed by the CODEC and partially performed by the DSP. The output of the mel-band filters232may be buffered. Data transformation and sound recognition may be performed by the DSP.

In a third embodiment, the CODEC/DSP boundary may be located at306. It will be noted that the third embodiment may correspond to the system100ofFIG. 1and the system200ofFIG. 2. In this third embodiment, feature extraction may completely be performed by the CODEC, and the output of the log converter325may be buffered. Data transformation and sound recognition may be performed by the DSP.

In a fourth embodiment, the CODEC/DSP boundary may be located at308. In this fourth embodiment, both feature extraction and data transformation may be performed by the CODEC, and the output of the DCT transformer333may be buffered. Sound recognition may be performed by the DSP.

As described with reference toFIGS. 1-2, the disclosed techniques may involve use of an always-on low-power CODEC (or a duty-cycled CODEC) and a duty-cycled processor that consumes more power than the CODEC when “active.” Thus, it may be desirable to incorporate relatively low-power functionality into the CODEC and leave relatively high-power functionality in the DSP. As shown inFIG. 3, the CODEC/DSP boundary and buffering point may be flexibly located in any of multiple locations. In a particular embodiment, the location of the CODEC/DSP boundary may be determined during design and testing of an electronic device and may be based on factors such as overall power consumption and performance of the electronic device.

Referring toFIG. 4, a particular illustration of operation at the system100ofFIG. 1or the system200ofFIG. 2is shown and generally designated400. For example,FIG. 4compares DSP operation in store-and-forward mode and in direct transfer (e.g., real-time or near real-time) mode.

When the DSP operates in store-and-forward mode, a CODEC including a plurality of filters (e.g., 22 mel-band filters) may extract and accumulate 22 features per frame for each frame of received audio data, as indicated at402, while the DSP is in a low-power state. When the DSP transitions out of the low-power state, the DSP may retrieve and analyze the accumulated features, as indicated at412. In the particular embodiment illustrated inFIG. 4, the DSP transitions out of the low-power state after audio features corresponding to 10 frames of audio data have been extracted by the CODEC. Thus, in the store-and-forward mode, the DSP may retrieve and process220audio features (corresponding to 10 frames) prior to transitioning back to the low-power state. This process may continue, as indicated by a subsequent extraction of features, at404, and processing of retrieved features, at414.

To avoid or reduce audio feature loss and buffer overflow, when operating in the store-and-forward mode, the DSP may transition out of the low-power state in accordance with a programmable time period. The programmable time period may be less than or equal to a maximum time period that is based on the size of the buffer. Thus, in the store-and-forward mode, audio features from each frame received by the CODEC may eventually be analyzed by the DSP. In a particular embodiment, DSP-CODEC handshaking or another technique may be utilized to maintain synchronization between the DSP and the CODEC and to reduce buffer overflow/underflow.

When the DSP operates in the direct transfer mode, audio features (indicated at406) corresponding to a most recently received audio frame may be retrieved and processed by the DSP, as indicated at416. Because there is effectively a “direct transfer” of audio features to the DSP, the audio features may be buffered for a very short amount of time or may not be buffered at all, and the duty cycle of DSP may be programmed independent of the size of the buffer. Thus, in the direct transfer mode, the DSP may retrieve and process22audio features (corresponding to a single audio frame), prior to transitioning back to the low-power state. This process may continue, as indicated by subsequent extracted features, at408, and retrieved features, at418. Thus, in the direct transfer mode, audio features from only a subset of frames (e.g., one out of every ten frames in the embodiment ofFIG. 4) received by the CODEC may be analyzed by the DSP.

It should be noted that the CODEC and the DSP may support additional operating modes as well. Typically, activity of the CODEC may be greater than or equal to activity of the DSP. The various operating modes may correspond to different ratios of CODEC activity to processor activity. Each operating mode may include different settings for the duty cycle of the CODEC (where 100% corresponds to always on), the duty cycle of the DSP, and/or how many frames of audio data are analyzed each time the processor wakes up. The details of the supported operating modes may be determined at design-time and/or manufacturing-time. Which particular operating mode is selected may be determined at run-time based on factors such as application context.

Referring toFIG. 5, a particular illustration of power consumption at various sound-sensing systems is shown and generally designated500. More particularly, the left-hand side ofFIG. 5illustrates power consumption at a system that includes an always on CODEC and an always on DSP and the right-hand side ofFIG. 5illustrates power consumption at a system in accordance with the disclosed techniques; such as the system100ofFIG. 1or the system200ofFIG. 2.

The sound-sensing system to the left may include an always on CODEC502. The system may also include an always on DSP, including always on DSP feature extraction504and always on DSP analysis506. Because the CODEC and the DSP are always on, the power consumed by the system may be represented by a relatively flat curve, as shown at508.

The sound-sensing system to the right (e.g., the system100ofFIG. 1or the system200ofFIG. 2) may include an always on low-power CODEC512and CODEC feature extraction, at514. The system may also include a duty-cycled DSP. For example, in the particular embodiment ofFIG. 5, the DSP has a 20 ms active time and a 200 ms idle time. Thus, although the combination of the CODEC duty-cycled DSP on the right-hand side may consume more power than the system on the left-hand side during the 20 ms active times516,518, the combination may consume substantially less power during the 200 ms idle time of the duty-cycled DSP. Power consumption of the right-hand side system may be illustrated by the curve518. It will be appreciated that average power consumption of the system on the right-hand side ofFIG. 5, illustrated by the curve519, may thus be substantially less than the power consumption of the system on the left-hand side ofFIG. 5, illustrated by the curve508. In some implementations, the CODEC512may be duty-cycled as well, as described with reference toFIGS. 1-4.

Referring toFIG. 6, a particular embodiment of a method of performing sound recognition on audio feature data at a duty-cycled processor is shown and generally designated600. In an illustrative embodiment, the method600may be performed by the processor150ofFIG. 1or the processor230ofFIG. 2.

The method600may include transitioning out of a low-power state at a processor during a duty cycle of the processor, at602. In a particular embodiment, the processor may be a digital signal processor (DSP) having a 10% duty cycle. For example, inFIG. 2, the processor230may transition out of a low-power state during a duty cycle (e.g., a transition from idle to active).

The method600may also include retrieving audio feature data from a buffer, where the audio feature data indicates features of audio data received during the low-power state of the processor. When the processor is operating in a store-and-forward mode, the audio feature data may correspond to a plurality of audio frames, at604. Alternately, when the processor is operating in a direct transfer mode, the audio feature data may correspond to a single audio frame, at606. For example, inFIG. 2, the processor may retrieve the extracted audio features226from the buffer227.

The method600may further include transforming the retrieved audio feature data to generate transformed audio feature data, at608, and performing one or more sound recognition operations on the transformed audio feature data, at610. In a particular embodiment, the audio feature data may be transformed via a discrete cosine transform (DCT) transformer and the resulting transformed audio feature data may include a plurality of mel-frequency cepstral coefficients (MFCCs). For example, inFIG. 2, the transformer233may transform the retrieved audio features226to generate the transformed audio feature data234, and one or more of the sound recognition modules241-245may perform one or more sound recognition operations (e.g., listen location, continuous audio fingerprinting, continuous keyword detection, target sound detection, and/or novelty detection) on the transformed audio feature data234.

The method600may include determining whether to activate an application processor and/or a portion of a mobile station modem, or other component, based on a result of the one or more sound recognition operations, at612, prior to transitioning back to the low-power state, at614. For example, inFIG. 2, the processor230may determine, based on analysis performed by one or more of the sound recognition modules241-245, whether to activate an application processor and/or a portion of a mobile station modem prior to transitioning back into the low-power state.

In particular embodiments, the method600ofFIG. 6may be implemented via hardware (e.g., a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), etc.) of a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), or a controller, via a firmware device, or any combination thereof. As an example, the method600ofFIG. 6can be performed by a processor that executes instructions, as described with respect toFIG. 10.

Referring toFIG. 7, a particular embodiment of a method of extracting audio feature data at a CODEC is shown and generally designated700. In an illustrative embodiment, the method700may be performed by the CODEC120ofFIG. 1or the CODEC220ofFIG. 2.

The method700may include receiving a frame of audio data at a CODEC, at702. For example, inFIG. 2, the CODEC220may receive a frame of the audio data212. The method700may also include extracting audio feature data from the frame, at704. To illustrate, extracting audio feature data may include computing energies of filtered audio data of the frame via a plurality of mel-band filters, at706, and applying a logarithm function to the computed energies, at708. For example, inFIG. 2, the feature extractor222may filter the audio data212using the filters223to generate the energies224of filtered audio data and may apply a logarithm function using the log converter225to generate the extracted audio features226.

The method700may further include storing the extracted audio feature data in a buffer to be accessible by a duty-cycled processor during an active state of the duty-cycled processor, at710, and discarding the frame of audio data, at712. For example, inFIG. 2, the extracted audio features226may be stored in the buffer227and the frame of the audio data212may be discarded by the CODEC220. The method700may be repeated for subsequent frames of audio received by the CODEC while the duty-cycled processor is in a low-power state.

In particular embodiments, the method700ofFIG. 7may be implemented via hardware (e.g., a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a controller, etc.) of a CODEC, via a firmware device, or any combination thereof. As an example, the method700ofFIG. 7can be performed by a CODEC (or processor therein) that executes instructions, as described with respect toFIG. 10.

Referring toFIG. 8, a particular embodiment of a method of dynamically switching between operating in a store-and-forward mode and in a direct transfer mode at a processor is shown and generally designated800. In an illustrative embodiment, the method800may be performed by the processor150ofFIG. 1or by the processor230ofFIG. 2.

The method800may include, at a processor, dynamically switching between operating in a first mode and operating in a second mode based on an application context of the processor, at802. A ratio of CODEC activity to processor activity in the first mode may be greater than a ratio of CODEC activity to processor activity in the second mode. For example, inFIG. 2, the dynamic mode-switching logic232may dynamically switch operation of the processor230between various modes based on an application context of the processor230. Similar dynamic mode-switching logic may also be present in the CODEC220ofFIG. 2. Alternatively, dynamic mode-switching logic that controls both the CODEC220and the processor230ofFIG. 2may be in an external component (e.g., integrated into a controller). The method800may also include transitioning out of a low-power state at the processor during a duty cycle of the processor, at804. For example, inFIG. 2, the processor230may transition out of a low-power state during a duty cycle.

The method800may include analyzing the retrieved audio feature data, at806, and transitioning back to the low-power state, at808. For example, inFIG. 2, one or more of the sound recognition modules441-445may analyze the retrieved audio feature data prior to the processor230transitioning back to the low-power state. In a particular embodiment, the processor230may also determine whether or not to activate other system components, such as an application processor and/or portion of a mobile station modem (MSM) based on the analysis. For example, the processor230may generate an activation signal based on the analysis, as described with reference to the activation signal160ofFIG. 1.

In particular embodiments, the method800ofFIG. 8may be implemented via hardware (e.g., a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), etc.) of a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), or a controller, via a firmware device, or any combination thereof. As an example, the method800ofFIG. 8can be performed by a processor that executes instructions, as described with respect toFIG. 10.

Referring toFIG. 9, a particular embodiment of a method of performing staggered hierarchical activation of higher-power components of an electronic device by lower-power components of the electronic device is shown and generally designated900. In an illustrative embodiment, the method900may be performed at the system200ofFIG. 2.

The method900may include receiving sound data at a first component of an electronic device, at902. The first component may be at a digital/analog circuit of a CODEC. For example, inFIG. 2, the coarse detection/classification block290may receive sound data. The method900may also include performing, at the first component, at least one signal detection operation on the sound data, at904. For example, inFIG. 2, the coarse detection/classification block290may perform a signal detection operation (e.g., a RMS operation or a band-power operation) to determine whether the sound data includes audio, beacon, or ultrasound data.

The method900may further include selectively activating a second component of the electronic device based on a result of the at least one signal detection operation, at906. The second component when active may consume more power at the electronic device than the first component when active. In a particular embodiment, the second component may be at a front-end unit of the CODEC. For example, inFIG. 2, the coarse detection/classification block290may selectively activate the fine detection/classification block296via the first activation signal294.

The method900may include performing, at the second component, at least one second signal detection operation, at908. The method900may include selectively activating a third component of the electronic device based on a result of the at least one second signal detection operation. The third component when active may consume more power at the electronic device than the second component when active. In a particular embodiment, the third component may be incorporated into a DSP. For example, inFIG. 2, the fine detection/classification block296may selectively activate a final detection/classification block at the processor230via the second activation signal298.

In particular embodiments, the method900ofFIG. 9may be implemented via hardware (e.g., a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), etc.) of a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), or a controller, via a firmware device, or any combination thereof. As an example, the method900ofFIG. 9can be performed by a processor that executes instructions, as described with respect toFIG. 10.

Referring toFIG. 10, a block diagram of a particular illustrative embodiment of a wireless communication device is depicted and generally designated1000. The device1000includes an application processor1010and a digital signal processor (DSP)1080, both of which are coupled to a memory1032. In an illustrative embodiment, the DSP1080may be the processor150ofFIG. 1or the processor230ofFIG. 2. The memory1032may include instructions1060executable by the DSP1010to perform methods and processes disclosed herein, such as the method600ofFIG. 6and the method800ofFIG. 8. The instructions may also be executable by a coder/decoder (CODEC)1034to perform methods and processes disclosed herein, such as the method700ofFIG. 7. The instructions may also be executable by the CODEC1034and the DSP1080to perform the method900ofFIG. 9.

FIG. 10also shows a display controller1026that is coupled to the application processor1010and to a display1028. The CODEC1034may be coupled to the DSP1080, as shown. A speaker1036and a microphone1038can be coupled to the CODEC1034. For example, the microphone1038may be the microphone210ofFIG. 2.FIG. 10also indicates that a wireless controller1040can be coupled to the processors1010,1080and to a wireless antenna1042.

The CODEC1034may include an analog-to-digital converter (ADC)1071, a plurality of filters1072, and a log converter1073. For example, the ADC1071may be the ADC221ofFIG. 2, the filters1072may be the filters223ofFIG. 2, and the log converter1073may be the log converter225ofFIG. 2. In a particular embodiment, the CODEC1034may also include a buffer1074(e.g., as described with reference to the buffer227ofFIG. 2). Alternately, the buffer1074may be external to the CODEC1034and to the DSP1080(e.g., as described with reference to the buffer140ofFIG. 1). The DSP1080may include a transformer1082(e.g., the transformer233ofFIG. 2) and one or more sound recognition modules1083(e.g., the sound recognition modules241-245ofFIG. 2) configured to perform one or more sound recognition operations. In a particular embodiment, the transformer1082and the sound recognition module(s)1083may be included in a low-power audio sub-system (LPASS)1081of the DSP1080.

In a particular embodiment, the processors1010,1080, the display controller1026, the memory1032, the CODEC1034, and the wireless controller1040are included in a system-in-package or system-on-chip device (e.g., a mobile station modem (MSM))1022. In a particular embodiment, an input device1030, such as a touchscreen and/or keypad, and a power supply1044are coupled to the system-on-chip device1022. Moreover, in a particular embodiment, as illustrated inFIG. 10, the display1028, the input device1030, the speaker1036, the microphone1038, the wireless antenna1042, and the power supply1044are external to the system-on-chip device1022. However, each of the display1028, the input device1030, the speaker1036, the microphone1038, the wireless antenna1042, and the power supply1044can be coupled to a component of the system-on-chip device1022, such as an interface or a controller.

In conjunction with the described embodiments, an apparatus is disclosed that includes means for receiving one or more frames of audio data. For example, the means for receiving may include the CODEC120ofFIG. 1, the microphone210ofFIG. 2, the microphone310ofFIG. 3, the microphone1038ofFIG. 10, one or more devices configured to receive frames of audio data, or any combination thereof. The apparatus may also include means for filtering the one or more frames of audio data to produce filtered audio data independent of whether a processor is in a low-power state or in an active state. For example, the means for filtering may include the CODEC120ofFIG. 1, the filters223ofFIG. 2, the filters323ofFIG. 3, the filters1072ofFIG. 10, one or more devices configured to filter frames of audio data, or any combination thereof.

The apparatus may further include means for generating audio feature data based on the energies of the filtered audio data. For example, the means for generating may include the CODEC120ofFIG. 2, the log converter225ofFIG. 2, the log converter325ofFIG. 3, the log converter1073ofFIG. 10, one or more devices configured to generate audio feature data, or any combination thereof. The apparatus may include means for transforming the audio feature data to generate transformed data. For example, the means for transforming may include the processor150ofFIG. 1, the transformer233ofFIG. 2, the DCT333ofFIG. 3, the transformer1082ofFIG. 10, one or more devices configured to transform audio feature data, or any combination thereof.

The apparatus may also include means for performing one or more operations on the transformed data after the processor transitions out of the low-power state to the active state. For example, the means for performing may include the processor150ofFIG. 1, one or more of the sound recognition modules241-245ofFIG. 2, one or more of the sound recognition modules340ofFIG. 3, one or more of the sound recognition module(s)1083ofFIG. 10, one or more devices configured to perform operations on transformed data, or any combination thereof. The apparatus may further include means for buffering an output of at least one of the means for filtering, the means for generating, and the means for transforming. For example, the means for buffering may include the buffer140ofFIG. 1, the buffer227ofFIG. 2, a buffer at one or more of the buffering points302-308ofFIG. 3, the buffer1074ofFIG. 10, one or more devices configured to buffer data, or any combination thereof.

Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software executed by a processing device such as a hardware processor, or combinations of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or executable software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.