Patent Description:
The present disclosure relates to audio processing, and in particular, to audio content identification.

Audio playback has become popular with the rise of consumer entertainment devices, e.g., smart phones, tablets, PCs etc. There are also tens of thousands of audio application scenarios, such as high-fidelity playback, streaming, gaming, podcast, short video and live broadcast of users, etc. Thus, in order to improve the overall quality of the audio and provide different user experiences, there are different audio processing algorithms to enhance audio signals for various purposes. Some typical examples of audio processing algorithms include dialogue enhancement and intelligent equalization.

Dialogue enhancement generally enhances speech signals. Dialogue is an important component in a movie to understand the story. Dialogue enhancement implements methods to enhance the dialogue in order to increase their clarity and their intelligibility, especially for elders with decreased hearing capability.

Intelligent equalization generally performs dynamic adjustment of the audio tone. Intelligent equalization is often applied in a music content in order to provide consistency of spectral balance, as known as "tone" or "timbre". It achieves this consistency by continuously monitoring the spectral balance of the audio, comparing it to a desired tone, and dynamically adjusting an equalization filter to transform the audio's original tone into the desired tone.

In general, an audio processing algorithm has its own application scenario/context. That is, an audio processing algorithm may be suitable for only a certain set of content but not for all the possible audio signals, since different content may need to be processed in different ways. For example, a dialogue enhancement method is usually applied on movie content. If it is applied on music in which there is no dialogue, it may falsely boost some frequency sub-bands and introduce heavy timbre change and perceptual inconsistency. Similarly, if an intelligent equalization method is applied on movie contents, timbre artifacts will be audible. However, for an audio processing system, its input could be any of the possible types of audio signals. Thus, identifying or differentiating the content being processed becomes important, in order to apply the most appropriate algorithms (or the most appropriate parameters of each algorithm) on the corresponding content.

A general content-adaptive audio processing system (see e.g. <CIT>) includes three functions: audio content identification, steering, and audio processing.

Audio content identification automatically identifies the audio types of the content on playback. Audio classification technologies, through signal processing, machine learning, and pattern recognition, can be applied to identify audio content. Confidence scores, which represent the probabilities of the audio content regarding a set of pre-defined target audio types, are estimated.

Steering generally steers the behavior of audio processing algorithms. It estimates the most suitable parameters of the corresponding audio processing algorithm based on the results obtained from audio content identification.

Audio processing generally applies audio processing using the estimated parameters to an input audio signal to generate an output audio signal.

As the ever-changing audio contents and new applications increase, especially for user-generated content and the corresponding applications (e.g., chatting, streaming, live broadcast, short video, etc.), it is an inevitable consequence of improving the audio identifiers (classifiers) and the steering algorithms in existing systems to meet the performance requirement on new contents or new use cases. Taking music for example, pop music including jazz, country, rock and latin music used to be mainstream across over different applications. Thus, the general music classifier in many existing systems is mainly targeted for identifying the above music genres and generated confidence scores precisely for subsequent steering algorithms and audio processing algorithms. With the changes of the trend of fashions, many people prefer to listen to different music genres, such as rap/hip-hop, electronic music or the combinations between different music styles. In particular, rap music mainly consists of (rhythmic-) talking, which is hard to distinguish from common dialogue speaking. In many existing cases, the original music classifier is usually not capable of providing enough accuracy on the rap music or a cappella music classification. As a result, some segments/frames of rap music would be falsely identified as speech and then boosted by the dialogue enhancer, triggering audible artifacts.

Moreover, with the increasing needs from customers, the audio processing system may need to provide new functionalities, which further requires the audio classifier identify certain audio content types. Both of the above scenarios need a new classifier. While the new audio classifier provides more classification results, it is also hoped that the classification results on originally supported content types (such as dialogue or music) could be still similar to those from the old classifier so that other audio processing algorithms, such as dialogue enhancement and intelligent equalization, do not need to be heavily tuned after the new classifier is used.

Given the above, there is a need to add a new classifier to an existing classification system while still keeping the original audio processing behavior close to the original. Whatever improving the original classifiers on specific new contents or adding new functionalities, it is usually not trivial to transparently update or replace the old classifier with the new classifier. The whole system may not straightforwardly work optimally after the identifier replacement. In many cases, after an identifier is updated, the subsequent steering algorithms and audio processing algorithms may also need corresponding refinement or tuning; moreover, what the user expects to keep in the original music identifier for the behaviors testing on previous contents may be not suitable anymore. This may introduce a large amount of extra efforts on retuning in order to fully integrate the new component, which is undesirable.

In this disclosure, we propose a method of improving the original content identification on new contents while minimizing the extra efforts on developing or verification. Described herein are techniques related to using a two-stage audio classifier.

According to the invention there are provided a method as set forth in claim <NUM>, a non-transitory computer readable medium as set forth in claim <NUM> and an apparatus as set forth in claim <NUM>.

The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of various implementations.

Described herein are techniques related to audio content identification. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure.

In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.

In this document, the terms "and", "or" and "and/or" are used. Such terms are to be read as having an inclusive meaning. For example, "A and B" may mean at least the following: "both A and B", "at least both A and B". As another example, "A or B" may mean at least the following: "at least A", "at least B", "both A and B", "at least both A and B". As another example, "A and/or B" may mean at least the following: "A and B", "A or B". When an exclusive-or is intended, such will be specifically noted (e.g., "either A or B", "at most one of A and B").

This document describes various processing functions that are associated with structures such as blocks, elements, components, circuits, etc. In general, these structures may be implemented by a processor that is controlled by one or more computer programs.

<FIG> is a block diagram of an audio classifier <NUM>. The audio classifier <NUM> generally receives an input audio signal <NUM>, performs classification of the input audio signal <NUM> using various models, and outputs a confidence score <NUM>. The audio classifier <NUM> includes a feature extractor <NUM>, a first set of classifiers <NUM> (also referred to as the original classifiers), a second set of classifiers <NUM> (also referred to as the new classifiers), a context detector <NUM>, and a confidence decider <NUM>. The audio classifier <NUM> may also be generally referred to as a two-stage audio classifier or a two-stage music classifier. Alternatively, the classifiers <NUM> and <NUM>, the context detector <NUM> and the confidence decider <NUM> (e.g., excluding the feature extractor <NUM>) may collectively be referred to as a two-stage audio classifier or a two-stage music classifier.

The feature extractor <NUM> receives the audio signal <NUM>, performs feature extraction on the audio signal <NUM>, and generates extracted features <NUM>. The particular features extracted generally are selected according to the particular features that are relevant to the models implemented by the classifiers <NUM> and <NUM>. As an example, the extracted features <NUM> may correspond to the spectral energy in various frequency bands of the audio signal <NUM>.

The classifiers <NUM> generally comprise one stage of the audio classifier <NUM>. The classifiers <NUM> receive the extracted features <NUM>, perform classification of the extracted features <NUM> using one or more models, and generate a set of confidence scores <NUM> (also referred to as the original confidence score). The set of confidence scores <NUM> may include one or more confidence scores, e.g. corresponding to the one or more models.

The classifiers <NUM> generally correspond to an existing set of classifiers. In general, the existing set of classifiers have been developed to classify existing genres of audio, but may be less able to accurately classify new genres of audio. The classifiers <NUM> may include one or more classifiers, including a speech classifier, a music classifier, a sound effect classifier, a noise classifier, etc. The classifiers <NUM> may include one or more different types of each classifier, for example two or more types of music classifiers, each developed to classify a specific genre of music (e.g., a jazz classifier, a rock classifier, etc.). The speech classifier generally assesses whether the audio signal <NUM> corresponds to speech (e.g., dialogue) as opposed to music, sound effects, etc. The sound effect classifier generally assesses whether the audio signal <NUM> corresponds to sound effects (e.g., movie sound effects such as car crashes, explosions, etc.) as opposed to speech (e.g., dialogue) or music (e.g., background music, mood music, etc.). The noise classifier generally assesses whether the audio signal <NUM> contains noise (e.g., a constant or repetitive sound such as a hum, buzz, whine, jackhammer, siren, waterfall, rainfall, etc.).

The classifiers <NUM> may be implemented by a machine learning system that uses various models of the various types of audio to perform the various classifications. The classifiers <NUM> may implement an adaptive boosting (AdaBoost) or a deep neural network machine learning process. The AdaBoost process may be implemented in devices that use a small model size or have limited capability to perform complex computations. The deep neural network process may be implemented in devices that allow a greater model size and have greater ability to execute complex computations. In general, the models of the classifiers <NUM> are developed offline by performing machine learning on a set of training data.

The classifiers <NUM> generally comprise a second stage of the audio classifier <NUM>. The classifiers <NUM> receive the extracted features <NUM>, perform classification of the extracted features <NUM> using one or more models, and generate a set of confidence scores <NUM> (also referred to as the new confidence score). The confidence score <NUM> may include one or more confidence scores, e.g. corresponding to the one or more models.

The classifiers <NUM> generally correspond to a new set of classifiers. In general, the new classifiers have been developed to classify new genres of audio. For example, the training data used to develop the models for the original classifiers <NUM> may not have included audio data for new genres of music, making the original classifiers <NUM> not perform well in identifying the new genres. As described in more detail below, the new classifiers <NUM> include a rap classifier.

The classifiers <NUM> may be implemented by a machine learning system that uses various models of the various types of audio to perform the various classifications. The classifiers <NUM> may implement an adaptive boosting (AdaBoost) or a deep neural network machine learning process. In general, the models of the classifiers <NUM> are developed offline by performing machine learning on a set of training data.

The classifiers <NUM> may also receive information from the classifiers <NUM>, such as the set of confidence scores <NUM>. For example, the classifiers <NUM> may receive an indication from the classifiers <NUM> that the audio signal <NUM> corresponds to speech or music (as opposed to sound effects or noise).

The context detector <NUM> receives the set of confidence scores <NUM> and generates a steering signal <NUM>. The context detector <NUM> may receive information from the classifiers <NUM> that indicates that the audio signal <NUM> contains neither speech nor music. In general, the context detector <NUM> evaluates the components of the set of confidence scores <NUM> over various time frames, and uses the smoothed confidence scores to reduce the impact of misclassifications over the short term. The context detector <NUM> generates the steering signal <NUM> to weight the impact of the sets of confidence scores <NUM> and <NUM> by subsequent components. Further details of the context detector <NUM> and the steering signal <NUM> are provided below.

The confidence decider <NUM> receives the sets of confidence scores <NUM> and <NUM> and the steering signal <NUM>, and generates a final confidence score <NUM>. In general, the confidence detector <NUM> smoothly transitions the audio classifier <NUM> from using only the classifiers <NUM> to also using the classifiers <NUM>, when appropriate according to the confidence score <NUM>. Further details of the confidence decider <NUM> are provided below.

The following sections discuss a specific use case of rap music classification for the classifiers <NUM>. As compared to existing music genres, rap music has similarities to both dialogue and music. Using existing classifiers thus risks classifying rap music as either dialogue and applying one set of audio processing algorithms, or music and applying another set of audio processing algorithms, neither of which may be appropriate for rap music. In addition, existing classifiers may rapidly switch between the classifications of dialogue and music, resulting in rapidly switching between the two processing algorithms, resulting in an inconsistent listening experience. Adding a rap classifier, and integrating the rap classifier with existing classifiers to form a two-stage classifier, results in an improved listening experience without disrupting the existing classifiers.

For rap music, the new features extracted by the feature extractor <NUM> are developed based on spectral energy, which shows the energy fluctuation characteristics of different contents in frequency domain. First, the input audio signal is transformed to spectral coefficients by time frequency conversion tool (e.g., quadrature mirror filter (QMF), fast Fourier transform (FFT), etc.), then the energy spectrum is calculated by above spectral coefficients, here the whole energy spectrum is further divided into four sub-bands in this disclosure.

The first sub-band energy, representing the energy distribution of low frequency below <NUM>, is used to detect the onset of bass or drums. The second sub-band energy, representing the energy distribution between <NUM> and <NUM>, is used to measure the fluctuation of vocal pitch. The third sub-band energy, representing the energy distribution between <NUM> and <NUM>, is used to measure the fluctuation of vocal harmonic. The fourth sub-band energy, representing the energy distribution between <NUM> and <NUM>, is used to detect the fluctuation of unvoiced signal or snare drum.

All the sub-band spectral energies are calculated in short-term frames, e.g. <NUM>, and then stored in a memory buffer until it meets the expected window length, e.g. <NUM>. Finally the high-level features could be derived based on above window-length spectral energy.

The number of sub-bands, the frequency range of each sub-band, the frame length, and the window length may be adjusted as desired. For example, to classify a different new genre, the sub-bands appropriate for that new genre may be used to generate a model for another new classifier <NUM>.

Compared to general music, the typical rap music has a few prominent differences, including the vocal tempo, the rhythmic lyrics, the regularity of musical bars, etc. Based on the above sub-band spectral energy, we introduce the peak / valley tracking method to find the cues to reflect the characteristics of vocal tempo, the rhythmic meter and the regularity of musical bars.

For a typical rap music, the general tempo is around <NUM> to <NUM> beats per minute (BPM), and typically with <NUM>/<NUM>-time signature; the lyrics are often sung regularly over a fixed period so that the number of syllables in each sentence is almost similar. Thus, some new features are deduced accordingly:.

A first feature is the statistical characteristics of sub-band spectral energy distribution. Over a fixed period, the spectral energy parameters are divided into several musical bars; in each bar, the peak / valley spectral energy may be calculated and the number of peak / valley are also counted. The features indicating the statistical characteristics of the above spectral energy (e.g., mean, standard deviation, etc.) may be used for distinguishing rap music from general speech content.

A second feature is the peak / valley location intervals of sub-band spectral energy. The vocals or syllables consist of voiced sound and unvoiced sound, to some extent which are related to the peak and valley of spectral energy so that the locations of peak / valley are at regular intervals for general rap music. However, for natural dialogue speaking, there is not obvious and regular intervals between voiced and unvoiced sound. Therefore, here the locations of peak / valley, represented by the index in the window-length spectral energy, is recorded in a continuous manner and then each interval of adjacent peak locations is calculated. Finally, the even distributions of these intervals are used as the key features of rap music.

A third feature is the contrast of peak and valley spectral energy. Compared to general speech or dialogue in movies or shows, the contrast of peak vocal energy and valley vocal energy in rap music is not much different, which also may be used as an important cue to indicate if the audio content is dialogue content or not.

A fourth feature is the rhyme features. Most of the lyrics of rap music are written in certain meters and rhyme schemes. Unfortunately, it may be computationally infeasible to segment the lyrics correctly based on syllable unit without semantic recognition. In addition, sometimes the rhyme is incomplete in rap music, especially when lacking one or more syllables in the final metrical foot.

A fifth feature is the rhythmic features. Rhythmic features, representing the frequency and strength of musical onset and the rhythmic regularity and contrast, are calculated on the sub-band energy of various spectral range mentioned above. One measurement may be based on the 1st / 4th sub-band, and the other may be based on the 2nd / 3rd sub-band spectral energy, respectively.

Before training the rap classifier, it is necessary to prepare a set of training data and finalize the features and classifier algorithms. The training database consists of various content types such as speech, rap music, non-rap music, sound effects, noise, etc., which are collected from various applications and hand-labelled to represent their corresponding audio types over time. These labels represent the ground truth of the audio content. In order to meet the requirements of different application scenarios, the feature set may be selected jointly or separately between old features and new features. In a similar way, the new model may also be trained independently or jointly with multiple models by using different learning algorithms.

There are different combinations of old features / training data and new features / training data, depending on the requirement of the new classifier and the system tolerance. Unfortunately, it is hard to find the optimal solution of the above combinations since we cannot enumerate all the selection possibilities. In practice, we manually split the training data set into two data chucks, one data chuck representing rap music content genre and the other data chuck representing non-rap. For feature set, we select both the original and new features for training rap music classifier, meanwhile remaining the old features for the old music classifier. Therefore, there are two independent music classifiers: One is the original music classifier as the first-staged music classifier for general music content identification (e.g., the set of classifiers <NUM>) and the other is the new-trained rap music classifier as the second-staged music classifier (e.g., the set of classifiers <NUM>), which is specifically for identifying the audio content between rap songs and dialogue contents.

<FIG> is a block diagram showing an arrangement of the classifiers <NUM> and <NUM> (see <FIG>) into a two-stage classifier <NUM>. The classifiers <NUM> form a first stage, and include a speech classifier <NUM>, a music classifier <NUM>, a sound effect classifier <NUM>, and a noise classifier <NUM>. The classifiers <NUM> receive the extracted features <NUM> and respectively generate a speech confidence score <NUM>, a music confidence score <NUM>, a sound effect confidence score <NUM>, and a noise confidence score <NUM>, which collectively comprise the set of confidence scores <NUM>.

The classifiers <NUM> form a second stage and include a rap classifier <NUM>. The second stage also includes a decision stage <NUM>. The decision stage <NUM> receives the set of confidence scores <NUM>. When the set of confidence scores <NUM> indicates that the audio signal <NUM> does not correspond to speech or music (e.g., low values for the speech confidence score <NUM> and the music confidence score <NUM>, or a high value for the sound effect confidence score <NUM> or the noise confidence score <NUM>), the two-stage classifier <NUM> outputs the set of confidence scores <NUM>. When the set of confidence scores <NUM> indicates that the audio signal <NUM> does correspond to speech or music (e.g., a high value for the speech confidence score <NUM> or the music confidence score <NUM>), the decision stage indicates this information to the rap classifier <NUM>.

The rap classifier <NUM> receives the extracted features <NUM> and the indication of speech or music from the decision stage <NUM>. To effectively reduce the computational complexity, it is not necessary to run the rap classifier <NUM> all the time for all the contents. Instead, the classifiers <NUM> and the classifiers <NUM> are arranged as a two-stage cascaded classifier. Firstly, the confidence scores for each audio type are calculated in the first stage, which determines the corresponding audio type with the maximal confidence score. If the audio type is speech or music type, then the condition is met and the indication is provided to the rap classifier <NUM> to perform further identification. The two-stage classifier <NUM> then outputs the confidence score <NUM> resulting from the operation of the rap classifier <NUM>. If the output type of first-stage classifiers is sound effect or noise, the rap classifier <NUM> may be bypassed.

The context detector <NUM> (see <FIG>) generally monitors the changes in the confidence values over time. Both the original classifiers (e.g., the classifier <NUM>) and new classifiers (e.g., the classifier <NUM>) may make mistakes in a short period. Thus, the context detector <NUM> assesses the continuous context information in a longer term. For example, listening to music over a period of several minutes results in the context information tending to have high confidence score of the music type at the end of this period, which could help correct sudden false alarm by misclassification over a short time period. The context detector <NUM> takes both long-term context and short-term context into consideration. The long-term context information is the music confidence score (e.g., the music confidence score <NUM>) that is slowly smoothed. For example, the slow smoothing may be determined over <NUM> to <NUM> seconds, e.g., <NUM> seconds. The long-term context information p(t) may then be calculated according to Equation (<NUM>): <MAT> where p(t) is the confidence score of the music classifier (e.g., the music confidence score <NUM>) at the current frame t of the audio signal <NUM>, and αcontext is the long-term smoothing coefficient.

In a similar way, the short-term context information is the non-music confidence score (e.g., the greater of the sound effect confidence score <NUM> and the noise confidence score <NUM>) that is quickly smoothed. For example, the quick smoothing may be determined over <NUM> to <NUM> seconds, e.g., <NUM> seconds. The short-term context information q(t) may then be calculated according to Equation (<NUM>): <MAT> where q(t) is the maximum of the sound effect confidence score <NUM> and the noise confidence score <NUM> at the current frame t of the audio signal <NUM>, and βcontext is the short-term smoothing coefficient.

Given the above context signals p(t) and q(t), a steering signal s(t) can be determined by a non-linear function h(). For example, a sigmoid function may be used to map the obtained context signal to the expected steering signal (from <NUM> to <NUM>), according to Equation (<NUM>): <MAT> where h<NUM> and h<NUM> are sigmoid functions as per Equation (<NUM>): <MAT> where x is the output obtained context confidence (e.g., p(t) or q(t)), and A and B are two parameters.

The output of the context detector <NUM> is the steering signal <NUM>, which is used as a weighting factor for subsequent processing by the confidence decider <NUM>. The range of the steering signal <NUM> is a soft value from <NUM> to <NUM>, where the value <NUM> indicates a non-music context while the value <NUM> indicates a music context. Between <NUM> and <NUM>, the larger the value is, the more likely the audio signal <NUM> is in music context.

The confidence decider <NUM> (see <FIG>) generates the final music confidence score <NUM> by jointly considering the steering signal <NUM>, the set of confidence scores <NUM> and the confidence score <NUM>. To achieve a smooth transition between rap music classification on/off, a mixing procedure will be taken if w(t) ∈ (<NUM>,<NUM>). That is, the final output will be a mixed confidence score of the old music classifier (e.g., only the confidence score <NUM>) and the new music classifier (e.g., a combination of both the confidence scores <NUM> and <NUM>). Given the confidence score of new music classifier xnew (t), the confidence score of old music classifier xold(t) [e.g., the confidence score <NUM>], and the steering signal s(t) [e.g., the steering signal <NUM>] discussed above, xnew(t) can be calculated according to Equation (<NUM>): <MAT> where new_conf(t) is the second-stage (rap) music confidence output (e.g., the confidence score <NUM>).

Then the final output confidence score y(t) [e.g., the final confidence score <NUM>] can be represented according to Equations (<NUM>) and (<NUM>): <MAT> <MAT>.

The threshold may be determined via a summary of statistics of the training data; according to an embodiment, a threshold of <NUM> works well.

In this disclosure, the rap classifier is detailed as an example use case of building a two-stage music classifier, which not only keeps the original behavior on the existing audio content, such as speech, non-rap music, sound effect and noise, but also improves the overall listening experience of rap music by greatly improving the classification accuracy on rap songs. It is noted that, the proposed method could be easily extended or directly applied to audio systems for various use cases of music content classification, such as building a new classifier for a capella music, certain background music in game and reverbed speech in podcast. More broadly, the proposed method could be also extended to general audio systems for general content classification. The following paragraphs discuss show several specific use cases, scenarios, and applications where an old content identifier needs to be extended by a new one.

One example use case is reverb detection. For example, there is a need to specifically process the reverberation speech and then encode to bit stream such as podcast or user-generated audio content. While supporting new types of data, the new detector may need to generate similar results on the old types of data to keep backward compatibility. In this case, a reverb speech classifier may be added to the classifiers <NUM> (see <FIG>).

Another example use case is gunshot detection. In a gaming application, the sound effect detector may be updated with additional types of sound effects, for example gunshot sounds. In this case, a gunshot classifier may be added to the classifiers <NUM>.

Another example use case is noise detection. With the increasing needs from customer, the audio processing system may need to provide more functionalities (e.g., noise compensation for mobile devices), which further requires the noise classifier identify more audio content types (e.g., stationary noise in mobile). While the new noise classifier provides more classification results, it is hoped that the classification results on originally supported content types (such as noise or sound effects) could be still similar to those from the old classifier so that other audio processing algorithms, such as noise suppression and volume leveler, do not need to be heavily tuned after the new classifier is used. In this case, a new noise classifier may be added to the classifiers <NUM>.

In summary, when a new classifier needs to be built or improved, the proposed method could be generalized from the following four considerations.

The first consideration is the relationship of old and new use case. This consideration makes clear the relationship of old and new classifiers, which decides the structure of the model combination. When the new use case is a type subset of old use case, the new classifier may be combined with old classifier as a cascaded multi-staged structure. If the new use case is an independent requirement, the new classifier may be in parallel with the old classifier. Moreover, this consideration helps to decide when the new classifier is triggered or activated and how the outcome of new classifier is combined with the confidence scores of old classifiers in the original system.

The second consideration is the new characteristics of new use case. This consideration aims to find the typical features, representing the essential characteristics of new pattern, which is used to discriminate the targeted type from other content type.

The third consideration in the training model of new use case. This consideration prepares for the training data and labelling data as the target audio type according to the new requirements, then extracts the features and trains the model of new classifier by corresponding machine learning techniques in an offline manner.

The fourth consideration is the integration of the new classifier. This consideration aims to integrate the new features and classifiers into the original system and tune the appropriate parameters to minimize the behavior differences of old use cases.

In order to differentiate audio content and apply the best parameters or the best audio processing algorithms correspondingly, different use case profiles may be needed and pre-designed, and system developers may choose a profile for the application context being deployed. A profile usually encodes a set of audio processing algorithms and/or their best parameters that will be applied, such as a 'File-based' profile and a 'Portable' profile which is specifically designed for high performance application or resource-limited use case, e.g. mobile. A major difference between file-based profile and portable profile is on the computational complexity by feature selection and model selection, which extended functionalities are enabled in file-based profile and disabled in portable profile.

When we extend the original system with new request, the new system should not have huge impact on the existing application use case. This suggests the following three recommendations.

The first recommendation concerns the feature / model selection of the old use case. The general goal is to keep the original features and classifiers unchanged if possible and add or train the separated classifier for new request, which is the essential guarantee for avoiding the big impact on the existing use case.

The second recommendation concerns the determination to use the new classifier. In order to reduce the unnecessary false alarm, the determination condition to use the new classifier should be fine-tuned, which indicates that for the old use case the original classifier is used to calculate the confidence score and output the outcome and only for the new use will the new classifier be used for identifying the audio content type.

The third recommendation concerns the confidence decider between old and new classifiers. Different smoothing schemes may be used for the determination of final output between the old confidence score and the new outcome. For example, confidence score can be further smoothed in order to avoid abrupt changes and to make more smooth estimation of the parameters in audio processing algorithms. A common smoothing method is based on weighted average, for example according to Equations (<NUM>) and (<NUM>): <MAT> <MAT> where t is a timestamp, α, β is the weight, conf and smoothConf is the confidence before and after smoothing, respectively.

The smoothing algorithm can be also 'asymmetric', by using different smoothing weight for different cases. For example, suppose we care more about the original output when old confidence score increases, we can design the smoothing algorithm according to Equation (<NUM>): <MAT>.

The above formula allows the smoothed confidence score to quickly responds to the current state when the old confidence score increases, and to slowly smooth away when the old confidence score decreases. Variants of the smoothing functions may be made in similar ways.

<FIG> is a block diagram of an audio processing system <NUM>. The audio processing system <NUM> includes the audio classifier <NUM> (see <FIG>) and processing components <NUM>, including a dialogue enhancer <NUM>, an intelligent equalizer <NUM>, and a rap music enhancer <NUM>.

The audio classifier <NUM> receives the audio signal <NUM> and operates as discussed above to generate the final confidence score <NUM>. The processing components <NUM> receive the final confidence score and process the audio signal <NUM> using the appropriate components based on the final confidence score <NUM>. For example, when the final confidence score <NUM> indicates the audio signal <NUM> is dialogue, the dialogue enhancer <NUM> may be used to process the audio signal <NUM>. When the final confidence score <NUM> indicates the audio signal <NUM> has an unbalanced spectral balance, the intelligent equalizer <NUM> may be used to process the audio signal <NUM>. When the final confidence score <NUM> indicates the audio signal <NUM> is rap music, the rap music enhancer <NUM> may be used to process the audio signal <NUM>. The processing components <NUM> generate the processed audio signal <NUM> corresponding to the audio signal <NUM> having been processed by the selected components.

<FIG> is a block diagram of a device <NUM> that may be used to implement the audio classifier <NUM> (see <FIG>), the two-stage classifier <NUM> (see <FIG>), the audio processing system <NUM> (see <FIG>), etc. The device <NUM> may be a computer (a desktop computer, a laptop computer, etc.), a gaming console, a portable device (e.g., a mobile telephone, a media player, etc.), etc. The device <NUM> includes a processor <NUM>, a memory <NUM>, one or more input components <NUM>, one or more output components <NUM>, and one or more communication components <NUM>, connected by a bus <NUM>.

The processor <NUM> generally controls the operation of the device <NUM>, for example according to the execution of one or more computer programs. The processor <NUM> may implement one or more of the functions described herein, such as those of the features extractor <NUM> (see <FIG>), the classifiers <NUM> and <NUM>, the context detector <NUM>, the confidence decider <NUM>, the audio processing components <NUM> (see <FIG>), the equations (<NUM>) through (<NUM>), the method <NUM> (see <FIG>), etc. The processor <NUM> may interact with the memory <NUM> to store data, computer programs, etc..

The memory <NUM> generally stores data operated on by the device <NUM>. For example, the memory <NUM> may store the input signal <NUM> (see <FIG>; e.g., as data frames of a streaming signal, as a stored data file, etc.), the extracted features <NUM>, the models used by the classifiers <NUM> and <NUM>, the confidence scores <NUM> and <NUM>, the steering signal <NUM>, the final confidence score <NUM>, the results of Equation (<NUM>) through Equation (<NUM>), etc. The memory <NUM> may also store the computer programs executed by the processor <NUM>.

The input components <NUM> generally enable input into the device <NUM>. The specifics of the input components <NUM> may vary based on the particular form factor of the device <NUM>. For example, the input components <NUM> of a mobile telephone may include a touchscreen, a microphone, motion sensors, a camera, control buttons, etc. The input components <NUM> of a gaming console may include control buttons, kinetic motion sensors, a microphone, gaming controllers, etc..

The output components <NUM> generally enable output from the device <NUM>. The specifics of the output components <NUM> may vary based on the particular form factor of the device <NUM>. For example, the output components <NUM> of a mobile telephone may include a screen, a speaker, haptic mechanisms, light emitting diodes, etc. The output components <NUM> of a gaming console may include a screen, a speaker, etc..

The communication components <NUM> generally enable wired or wireless communication between the device <NUM> and other devices. As such, the communication components <NUM> include additional input and output components similar to the input components <NUM> and the output components <NUM>. Wireless components include radios, such as cellular radios, IEEE <NUM>. <NUM> radios (e.g., Bluetooth™ radios), IEEE <NUM> radios (e.g., Wi-Fi™ radios), etc. Wired components include keyboards, mice, gaming controllers, sensors, etc. The specifics of the input components <NUM> and the output components <NUM> may vary based on the particular form factor of the device <NUM>. For example, a mobile telephone may include a cellular radio to receive the input signal <NUM> as a streaming signal, and an IEEE <NUM>. <NUM> radio to transmit the processed audio signal <NUM> to a pair of wireless earbuds for output as sound.

<FIG> is a flow diagram of a method <NUM> of audio processing. The method <NUM> may be implemented by a device (e.g., the device <NUM> of <FIG>), as controlled by the execution of one or more computer programs.

At <NUM>, an audio signal is received. For example, the audio signal <NUM> (see <FIG>) may be received by the communication components <NUM> (see <FIG>) of the device <NUM>. As another example, the audio signal <NUM> may be received from the memory <NUM>, having been stored there previously.

At <NUM>, feature extraction is performed on the audio signal to extract a plurality of features. For example, the feature extractor <NUM> (see <FIG>) may perform feature extraction on the audio signal <NUM>, to generate the extracted features <NUM>. The specifics of the feature extraction performed, and the resulting extracted features, may vary based on the relevance of those particular features to the models used for classification. For example, the sub-band energies of the input signal <NUM> may be relevant to the rap classification model.

At <NUM>, the plurality of features is classified according to a first audio classification model to generate a first set of confidence scores. For example, the classifiers <NUM> (see <FIG>) may classify the extracted features <NUM> according to a music classification model, a speech classification model, a noise classification model, a sound effect classification model, etc., generating respective confidence scores <NUM>.

At <NUM>, the plurality of features is classified according to a second audio classification model to generate a second confidence score. For example, the classifiers <NUM> (see <FIG>) may classify the extracted features <NUM> according to a rap classification model to generate a rap confidence score <NUM>.

At <NUM>, a steering signal is calculated by combining a first component of the first set of confidence scores smoothed over a first time period and a second component of the first set of confidence scores smoothed over a second time period, where the second time period is shorter than the first time period. For example, the context detector <NUM> (see <FIG>) may generate the steering signal <NUM> according to Equation (<NUM>), using long-term context information according to Equation (<NUM>) and short-term context information according to Equation (<NUM>).

At <NUM>, a final confidence score is calculated according to the steering signal, the first set of confidence scores, and the second confidence score. For example, the confidence decider <NUM> (see <FIG>) may generate the final confidence score <NUM> according to the steering signal <NUM>, the confidence scores <NUM> and the confidence scores <NUM>. The final confidence score may correspond to a weighted combination of the confidence scores <NUM> and <NUM>, e.g., computed according to Equation (<NUM>).

At <NUM>, a classification of the audio signal is output according to the final confidence score. For example, the confidence decider <NUM> (see <FIG>) may output the final confidence score <NUM> for use by other components of the device <NUM>.

At <NUM>, one of a first process and a second process selectively performing on the audio signal, based on the classification, to generate a processed audio signal, where the first process is performed when the classification is a first classification and the second process is performed when the classification is a second classification. For example, when the audio signal <NUM> (see <FIG>) corresponds to speech, the dialogue enhancer <NUM> (see <FIG>) may be used to generate the processed audio signal <NUM>. When the audio signal <NUM> corresponds to rap, the rap music enhancer <NUM> may be used to generate the processed audio signal <NUM>.

At <NUM>, the processed audio signal is output as sound. For example, a speaker of the device <NUM> may output the processed audio signal <NUM> as audible sound.

An embodiment may be implemented in hardware, executable modules stored on a computer readable medium, or a combination of both (e.g., programmable logic arrays). Unless otherwise specified, the steps executed by embodiments need not inherently be related to any particular computer or other apparatus, although they may be in certain embodiments. In particular, various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatus (e.g., integrated circuits) to perform the required method steps. Thus, embodiments may be implemented in one or more computer programs executing on one or more programmable computer systems each comprising at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices, in known fashion.

Each such computer program is preferably stored on or downloaded to a storage media or device (e.g., solid state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer system to perform the procedures described herein. The inventive system may also be considered to be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer system to operate in a specific and predefined manner to perform the functions described herein. (Software per se and intangible or transitory signals are excluded to the extent that they are unpatentable subject matter.

Claim 1:
A method of audio processing, the method comprising:
receiving an audio signal;
performing feature extraction on the audio signal to extract a plurality of features;
classifying the plurality of features according to a first audio classification model to generate a first set of confidence scores;
classifying the plurality of features according to a second audio classification model to generate a second confidence score;
calculating a steering signal by combining a first confidence score of the first set of confidence scores and a further confidence score of the first set of confidence scores;
calculating a final confidence score according to the steering signal, the first set of confidence scores, and the second confidence score; and
outputting a classification of the audio signal according to the final confidence score.