Patent Description:
The present disclosure relates to audio processing, and in particular, to bass enhancement.

Bass effect is a desirable user experience and user evaluation indicator for mobile devices such as mobile telephones, media players, tablet computers, laptop computers, headsets, earbuds, etc. Due to the physical constraints of the transducers in mobile devices (e.g., diaphragm size, magnet weight, etc.) it is challenging for the loudspeaker of the mobile device to fully reproduce the acoustics of the original bass sound. As a result, mobile devices often implement audio processing techniques (e.g., using software processes, etc.) to improve the bass sound. These bass enhancement processes may be broadly referred to as "virtual bass" techniques.

<CIT> discloses a virtual bass synthesis method, system and device using harmonic transposition. Further, <CIT> discloses performing harmonic transposition on low frequency components of an input audio signal by performing a frequency domain oversampled transform on the input audio signal.

<CIT> discloses a latency reduction system in a virtual bass processing system performs harmonic transposition on low frequency components of an audio signal to generate transposed data indicative of harmonics of the audio signal. The low frequency components are generated by performing a frequency domain oversampled transform on the audio signal.

<CIT> discloses an apparatus and method for conveying to a listener a pseudo low frequency psycho-acoustic sensation) (Pseudo-LFPS) of a sound signal, including: frequency unit capable of deriving from the sound signal high frequency signal and low frequency signal (LF signal) that extends over a low frequency range of interest.

<CIT> discloses a method for conveying to a listener a directionality-preserving pseudo low frequency psycho-acoustic sensation of a multichannel sound signal.

<CIT> discloses systems, methods and techniques for processing an audio signal to add virtual bass.

One issue with existing bass enhancement systems is that they may have a high computational complexity. Given the above, there may be a need to implement bass enhancement with reduced computational complexity.

As discussed in more detail herein, embodiments discuss techniques for bass enhancement based on the principle of the "missing fundamental". This principle states in a psychoacoustics way that if a human listens to harmonics of a low frequency signal rather than the low frequency signal (fundamental) itself, the listener's brain is able to extrapolate and hence perceive the absent low frequency signal. Hence, for loudspeakers that are physically inadequate to reproduce low frequency signals (bass), a way to psycho-acoustically improve the quality is to generate harmonics to the low frequency range to enhance the bass effect.

The bass enhancement technique disclosed in this specification is less computationally complex as compared to conventional virtual bass technologies but reaches a similar effect. Hence, embodiments save computational complexity. In addition, the reduced complexity allows for lower latency. The technique may also include loudness adjustment schemes to adjust the power of the generated harmonics, which causes the perception of the resulting loudness to be more realistic and the bass effect to be more compelling.

The techniques disclosed in this specification may be used to enhance the output from mid-sized speakers and smaller transducers, e.g. mobile phone loudspeakers, wireless loudspeakers, etc..

According to the invention, there is provided a computer-implemented method of audio processing as defined by claim <NUM>, a non-transitory computer readable medium storing a computer program as defined by claim <NUM>, and an apparatus for audio processing as defined by claim <NUM>.

Described herein are techniques related to bass enhancement. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure.

In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps, and may occur in parallel with other steps.

This document describes various processing functions that are associated with structures such as blocks, elements, components, circuits, etc. In general, these structures may be implemented by a processor that is controlled by one or more computer programs.

<FIG> is a block diagram of an audio processing system <NUM>. The audio processing system <NUM> generally receives an input audio signal <NUM>, processes the input audio signal <NUM> according to the bass enhancement processes described herein, and generates an output audio signal <NUM>. The audio processing system <NUM> includes a signal transform system <NUM>, a bass enhancement system <NUM>, an additional processing system <NUM> (optional), and an inverse signal transform system <NUM>. The audio processing system <NUM> may include other components that (for brevity) are not discussed in detail. The components of the audio processing system <NUM> may be implemented by one or more computer programs that are executed by a processor.

The signal transform system <NUM> receives the input audio signal <NUM>, performs a signal transform process, and generates a transformed audio signal <NUM>. The input audio signal <NUM> may be a digital time domain signal that includes a number of samples that correspond to audio (e.g., sound in waveform pulse-code modulation (PCM) format). The input audio signal <NUM> may have a sample rate of <NUM>, <NUM>, <NUM>, <NUM>, etc. The input audio signal <NUM> may originate from a variety of formats, including the Advanced Television Systems Committee (ATSC) Digital Audio Compression (AC-<NUM>, E-AC-<NUM>) Standard. As a specific example, the input audio signal <NUM> may originate from a Dolby Digital Plus™ signal with a sample rate of <NUM>.

The signal transform system <NUM> may perform a variety of signal transform processes. In general, the signal transform process transforms the input audio signal <NUM> from a first signal domain to a second signal domain. For example, the first domain may be the time domain, and the second signal domain may be the frequency domain, the quadrature mirror frequency (QMF) domain, the complex quadrature mirror frequency (CQMF) domain, the hybrid complex quadrature mirror frequency (HCQMF) domain, etc. The transform from the first signal domain to the second signal domain may also be referred to as "analysis", e.g. transform analysis, signal analysis, filter bank analysis, QMF analysis, CQMF analysis, HCQMF analysis, etc..

In general, QMF domain information is generated by a filter whose frequency response is the mirror image around π/<NUM> of that of another filter; together these filters are known as a QMF pair. QMF theory also comprises filter banks with more channels than two (e.g., <NUM> channels); these may be referred to as M-channel QMF banks. QMF theory further teaches M-channel Pseudo QMF banks of the class referred to as modulated filter banks. In general, "CQMF" domain information results from a complex-modulated discrete Fourier transform (DFT) filter bank applied to a time-domain signal. The CQMF is a "complex" signal because it includes complex valued signals, e.g. signals that include an imaginary part in addition to the real part. In general, "HCQMF" domain information corresponds to CQMF domain information in which the CQMF filter bank has been extended to a hybrid structure to obtain an efficient non-uniform frequency resolution that better matches the frequency resolution of the human auditory system. In general, the term "hybrid" refers to a structure in which at least one frequency band is split into sub-bands.

According to a specific HCQMF implementation, the HCQMF information is generated into <NUM> frequency bands, where the lower CQMF bands are further split into sub-bands in order to obtain a higher frequency resolution for the lower frequencies. According to a further specific implementation, the signal transform system <NUM> transforms each channel of the input audio signal <NUM> into <NUM> CQMF bands, and further divides the lowest <NUM> bands into sub-bands as follows: the first band is divided into <NUM> sub-bands, and the second and third bands are each divided into <NUM> sub-bands. (This hybrid splitting of the lowest bands into sub-bands is to improve the low-frequency resolution of these bands. ) The signal transform system <NUM> may include Nyquist filters to split the bands into sub-bands. The <NUM> HCQMF bands then correspond to the <NUM> highest CQMF bands, plus the <NUM> sub-bands (<NUM>+<NUM>+<NUM>) from the lowest <NUM> CQMF bands. The sub-bands and bands may be numbered from <NUM> to <NUM>, with the lowest frequency sub-band being number <NUM>. The other sub-bands are then numbered from <NUM> to <NUM>, and the remaining bands are numbered from <NUM> to <NUM>. These <NUM> HCQMF bands may then be referred to as "hybrid bands" or "channels" along with their number, e.g., hybrid band <NUM>, hybrid band <NUM>, hybrid band <NUM>, channel <NUM>, channel <NUM>, channel <NUM>, etc. The hybrid bands <NUM>-<NUM> may also be referred to as "sub-bands" along with their number, e.g., sub-band <NUM>, sub-band <NUM>, sub-band <NUM>, etc. The hybrid bands <NUM>-<NUM> may also be referred to as "bands" along with their number, e.g., band <NUM>, band <NUM>, band <NUM>, etc. The channels <NUM> and <NUM> may have passbands on the negative frequency axis, but generally the other channels do not.

(Note that the terms QMF, CQMF and HCQMF are used a bit colloquially herein. Specifically, the terms QMF/CQMF may be used colloquially to refer to a DFT filter bank that may include more than two bands. The term HCQMF may be used colloquially to refer to a non-uniform DFT filter bank that may include more than two bands.

As a specific example, the signal transform system <NUM> performs a HCQMF transform on the input audio signal <NUM> to generate the transformed audio signal <NUM> having <NUM> frequency bands. In this case, the signal domain of the transformed audio signal <NUM> may be referred to as the HCQMF domain or the hybrid domain, and the HCQMF transform may be referred to as HCQMF analysis.

The bandwidth and the sampling frequency of the bands will depend upon the sampling frequency of the input audio signal <NUM>. For example, when the input audio signal <NUM> has a sampling frequency of <NUM> (corresponding to a maximum bandwidth of <NUM>), the hybrid structure with <NUM> bands discussed above results in a sampling frequency of <NUM> for all bands. The <NUM> bands with the highest frequencies have a passband bandwidth of <NUM>; the <NUM> lowest-frequency sub-bands have a passband bandwidth of <NUM>; and the next-lowest-frequency sub-bands have a passband bandwidth of <NUM>.

The bass enhancement system <NUM> receives the transformed audio signal <NUM>, performs bass enhancement, and generates an enhanced audio signal <NUM>. In general, the bass enhancement system <NUM> generates harmonics to the transformed audio signal <NUM> in order for the listener to psycho-acoustically perceive the missing fundamental. Further details of the bass enhancement system <NUM> are provided below (e.g., with reference to <FIG>, etc.).

The additional processing system <NUM> is optional. When present, the additional processing system <NUM> receives the enhanced audio signal <NUM>, performs additional signal processing, and generates a processed audio signal <NUM>. Alternatively, the additional processing system <NUM> may operate on the transformed audio signal <NUM> prior to the operation of the bass enhancement system <NUM>, in which case the bass enhancement system <NUM> receives as its input the signal output from the additional processing system <NUM> (instead of receiving the output signal directly from the signal transform system <NUM>). As another option, the additional processing system <NUM> may be multiple additional processing systems that operate both before and after the bass enhancement system <NUM>. The specific arrangement of the additional processing system <NUM> within the audio processing system <NUM> may vary according to the specific types of additional processing that the additional processing system <NUM> performs.

In general, the additional processing system <NUM> performs additional processing of the input audio signal <NUM> in the transform domain. This allows the bass enhancement system <NUM> to operate in combination with existing audio processing techniques that are implemented in the transform domain. Examples of the additional processing include dialogue enhancement, intelligent equalization, volume leveling, spectral limiting, etc. Dialogue enhancement refers to enhancing speech signals (e.g., as compared to sound effects), in order to improve the intelligibility of the speech. Intelligent equalization refers to performing dynamic adjustment of the audio tone, e.g. to provide consistency of spectral balance (also known as "tone" or "timbre"). Volume leveling refers to increasing the volume of quiet audio and decreasing the volume of loud audio, e.g. to reduce the need for a listener to perform manual adjustment of the volume. Spectral limiting refers to limiting selected frequencies or frequency bands, e.g. to limit the lowest frequencies that are difficult to output from small loudspeakers.

The inverse signal transform system <NUM> receives the enhanced audio signal <NUM> (or optionally the processed audio signal <NUM>), performs an inverse transform, and generates the output audio signal <NUM>. The inverse transform generally converts a signal from the second signal domain back into the first signal domain. In general, the inverse transform is an inverse of the signal transform process performed by the signal transform system <NUM>. For example, when the signal transform system <NUM> performs a HCQMF transform, the inverse signal transform system <NUM> performs an inverse HCQMF transform. The transform from the second signal domain back to the first signal domain may also be referred to as "synthesis", e.g. transform synthesis, signal synthesis, filter bank synthesis, etc.; and the inverse HCQMF transform may be referred to as HCQMF synthesis.

In this manner, the output audio signal <NUM> corresponds to the input audio signal <NUM>, with the addition of the bass enhancement and/or additional signal enhancements. The output audio signal <NUM> may then be output by a loudspeaker and perceived as sound by the listener.

As discussed above and in more detail below, the bass enhancement system <NUM> is suitable for small to mid-sized speakers. The processes implemented by the bass enhancement system <NUM> may be simpler than many existing bass enhancement methods; as compared to these existing methods, the bass enhancement system <NUM> has lower computational complexity and allows for short latency, while still retaining the audio quality. The bass enhancement system <NUM> is well suited for mid-sized speakers in e.g. TV sets or wireless speakers, and is also efficient for bass improvement of small transducers, e.g. for mobile phones, laptops and tablets. The bass enhancement system <NUM> in one mode of operation not only adds harmonics to the mix, but also adds the (dynamically changed) original bass, i.e. it may be operated to have an inherent bass boost.

<FIG> is a block diagram of a bass enhancement system <NUM>. The bass enhancement system <NUM> may be used as the bass enhancement system <NUM> (see <FIG>). For brevity, the description of <FIG> focuses on a single signal processing path in order to describe the general operation of bass enhancement system <NUM>; additional signal processing paths may also be implemented in variations of the bass enhancement systems described herein (see, e.g., <FIG>). The additional signal processing paths will also be briefly described here.

The bass enhancement system <NUM> receives the transformed audio signal <NUM> (see <FIG>). As discussed above, the transformed audio signal <NUM> is a hybrid complex transform domain signal (e.g., a HCQMF domain signal) with a number of bands (e.g., <NUM> hybrid bands, with the <NUM> lowest-frequency bands split into sub-bands). As a complex signal, the transformed audio signal <NUM> has complex values, e.g. both real values and imaginary values. Each sub-band may be processed in its own processing path, so the following description focuses on processing one sub-band (e.g., one of sub-bands <NUM>, <NUM>, <NUM>, <NUM>, etc.). The bass enhancement system <NUM> includes an upsampler <NUM>, a harmonics generator <NUM>, a dynamics processor <NUM> (optional), a converter <NUM> (optional), a filter <NUM>, a delay <NUM>, and a mixer <NUM>.

The upsampler <NUM> receives the transformed audio signal <NUM>, performs upsampling, and generates an upsampled signal <NUM>. As an example, when the input audio signal <NUM> (see <FIG>) has a sampling frequency of <NUM>, and the transformed audio signal <NUM> is processed into <NUM> bands, each band has a sampling frequency of <NUM>. The upsampler <NUM> may upsample the selected sub-band of the transformed audio signal <NUM> by 2x, 3x, 4x, 5x, 6x, etc. A suitable amount of upsampling is 4x, e.g. so that the upsampled signal <NUM> has a sampling frequency of <NUM> when the selected sub-band of the transformed audio signal <NUM> has a sampling frequency of <NUM>. The upsampled signal <NUM> is a complex transform domain signal. The upsampled signal <NUM> has a bandwidth that corresponds to the bandwidth of the selected sub-band of the transformed audio signal <NUM>. As an example, when the selected sub-band <NUM> having a passband bandwidth of <NUM> is input to the upsampler, the upsampled signal <NUM> likewise has a bandwidth of <NUM>.

The upsampler <NUM> may be implemented by performing CQMF synthesis. As an example, to upsample sub-band <NUM> from <NUM> to <NUM> (4x upsampling), the upsampler may implement <NUM>-channel CQMF synthesis, with one input being the sub-band <NUM> and the other <NUM> inputs being zero (null). The synthesis is configured as to maintain the signal <NUM> being a complex-valued time domain signal.

In general, the upsampler <NUM> provides additional headroom when generating the harmonics (see the harmonics generator <NUM>), to allow bandwidth extension without aliasing (also referred to as spectral folding). The upsampler <NUM> may be omitted when processing one or more of the lowest frequency sub-bands. For example, when processing the lowest band (e.g., sub-band <NUM>) only, the upsampler <NUM> may be omitted, as up to (at least) <NUM>th order harmonics may be generated without folding. Processing the lowest two bands (e.g., sub-bands <NUM> and <NUM>), the upsampler <NUM> may be omitted if only <NUM>nd and <NUM>rd order harmonics are generated. Processing the lowest three bands (e.g., sub-bands <NUM>, <NUM> and <NUM>), only <NUM>nd order harmonics may be generated without aliasing. This is discussed in more detail with reference to the harmonics generator <NUM>.

The harmonics generator <NUM> receives the upsampled signal <NUM> and generates harmonics thereof to result in a signal <NUM>. As mentioned with reference to the upsampler <NUM>, the harmonics generator <NUM> extends the bandwidth of its input signal when generating the harmonics for the signal <NUM>. For example, when sub-band <NUM> covers <NUM> to <NUM>, the sampling frequency of <NUM> may be sufficient to avoid aliasing of the generated harmonics. Similarly, when sub-band <NUM> covers <NUM> to <NUM>, the sampling frequency of <NUM> may be sufficient to avoid aliasing of the generated harmonics. However, when sub-band <NUM> covers <NUM> to <NUM>, the harmonics are approaching the Nyquist frequency of the original signal (with the sampling frequency of <NUM>), so upsampling is recommended for sub-bands <NUM>, <NUM>, etc. The signal <NUM> is a complex transform domain signal. The signal <NUM> has a bandwidth that is greater than the bandwidth of the input to the harmonics generator <NUM>, due to the addition of the harmonic frequencies. For example, when the upsampled signal <NUM> has a bandwidth of <NUM>, the signal <NUM> may have a bandwidth that exceeds <NUM>.

The harmonics generator <NUM> uses a non-linear process to generate the harmonics. In general, a non-linear process applies different gains to different components of the signal. Examples of the non-linear processes include multiplication, a feedback delay loop, rectification, etc. as further detailed below with reference to <FIG>, <FIG>, <FIG> and <FIG>.

The harmonics generator <NUM> may also perform loudness expansion when generating the signal <NUM>. Because the sound pressure level for a fixed loudness range (in phon) is increasing with frequency in the bass/mid range (e.g., less than <NUM>), the harmonics generator <NUM> performs expansion in dynamics when generating the signal <NUM>. Examples of loudness expansion processes include dynamic compression and loudness correction. Further details of the loudness expansion are provided with reference to <FIG> below.

The dynamics processor <NUM> receives the signal <NUM>, performs dynamics processing, and generates a signal <NUM>. The signal <NUM> is a complex transform domain signal. In general, the dynamics processor <NUM> implements dynamics processing by performing compression on the signal <NUM>, in order to control the transient to tonal ratio of the signal <NUM>. The dynamics processor <NUM> may implement an attack time that is relatively longer (e.g., between 4x to 12x longer, such as 8x longer) than the release time. For example, the attack time may be between <NUM> and <NUM> (e.g., <NUM>) and the release time may be between <NUM> and <NUM> (e.g., <NUM>). The dynamics processor <NUM> may implement de-coupled smooth peak detection using feed-forward topology. The dynamics processor <NUM> may implement compression similar to the compression performed by the harmonics generator (described in more detail with reference to <FIG>, <FIG>).

The dynamics processor <NUM> is optional. When the dynamics processor <NUM> is omitted, the converter <NUM> receives the signal <NUM> instead of the signal <NUM>.

The converter <NUM> receives the signal <NUM> (or the signal <NUM> when the dynamics processor <NUM> is omitted), drops the imaginary part from the signal <NUM>, and generates a signal <NUM>. In general, dropping the imaginary part lowers the computational complexity of subsequent analysis filter banks (e.g., the filter <NUM>), due to processing real-valued signals instead of complex-valued signals. As discussed above, the signal <NUM> is a complex transform domain signal that has complex values, e.g. both real values and imaginary values. The converter <NUM> may drop the imaginary part of the signal <NUM> by taking the real part of the complex-valued signal. The signal <NUM> is a real-valued transform domain signal.

The converter <NUM> is optional and may be omitted in some embodiments of the bass enhancement system <NUM>.

The filter <NUM> receives the signal <NUM> (or the signal <NUM> when the converter <NUM> is omitted, or the signal <NUM> when the dynamics processor <NUM> and the converter <NUM> are omitted), performs filtering of the input, and generates a signal <NUM>. The signal <NUM> is a complex-valued transform domain signal. The filtering generally splits the signal <NUM> into sub-bands as one of the inputs to the mixer <NUM>. The specifics of the filtering will depend upon whether or not upsampling was performed (see the upsampler <NUM>).

When the upsampler <NUM> is not present, the filter <NUM> may be implemented by feeding the input signal (e.g., the signal <NUM>) into an <NUM>-channel Nyquist filter bank to generate the signal <NUM> that has hybrid sub-bands <NUM>-<NUM>.

When the upsampler <NUM> is present, the filter <NUM> may be implemented by a CQMF analysis filter bank and two or more Nyquist filters. The real part of the input signal (e.g., the signal <NUM>) is fed into the CQMF analysis filter bank; the CQMF analysis filter bank has an appropriate number of channels to generate the signal <NUM> having sub-band signals of <NUM> sampling frequency. The appropriate number of channels then depends on the upsampling performed. For example, when 4x upsampling is performed, and hence a <NUM> channel CQMF analysis bank is used in the filter <NUM>, the three lowest frequency CQMF sub-band signals are each fed into a corresponding Nyquist filter (one generating hybrid sub-bands <NUM>-<NUM>, one generating hybrid sub-bands <NUM>-<NUM>, and one generating hybrid sub-bands <NUM>-<NUM>). As another example, when 2x upsampling is performed, and hence a <NUM> channel CQMF analysis bank is used in the filter <NUM>, the two CQMF sub-band signals are each fed into a corresponding Nyquist filter (one generating hybrid sub-bands <NUM>-<NUM>, and one generating hybrid sub-bands <NUM>-<NUM>). The remaining CQMF channels, if any, are provided to the mixer <NUM> (with an appropriate delay corresponding to the delay of the Nyquist filters).

The filter <NUM> may be implemented with filters similar to those used by the signal transform system <NUM> (see <FIG>). For example, a first Nyquist analysis filter with <NUM> channels may generate the sub-bands <NUM>-<NUM>, a second Nyquist analysis filter with <NUM> channels may generate the sub-bands <NUM>-<NUM>, and a third Nyquist analysis filter with <NUM> channels may generate the sub-bands <NUM>-<NUM>.

The delay <NUM> receives the transformed audio signal <NUM>, implements a delay period, and generates a signal <NUM>. The signal <NUM> corresponds to a delayed version of the transformed audio signal <NUM> according to the delay period. The delay <NUM> may be implemented using a memory, a shift register, etc. The delay period corresponds to the processing time of the other components in the signal processing chain, e.g. the upsampler <NUM>, the harmonics generator <NUM>, the dynamics processor <NUM>, the converter <NUM>, the filter <NUM>, etc. Because some of these other components are optional, the delay period decreases as more of the optional components are omitted. In one example, the delay period is <NUM> samples, of which <NUM> correspond to the upsampling, and <NUM> correspond to the remaining components, e.g. the Nyquist filters. As another example, the delay period is <NUM> samples when the upsampler <NUM> is omitted.

The mixer <NUM> receives the signal <NUM> and the signal <NUM>, performs mixing, and generates the enhanced audio signal <NUM> (see <FIG>). The enhanced audio signal <NUM> is a transform domain signal. The mixer <NUM> mixes the signals on a per-band basis. For example, the signal <NUM> and the signal <NUM> may each have <NUM> hybrid bands (e.g., <NUM>+<NUM>+<NUM>+<NUM> HCQMF bands), and the mixer <NUM> mixes sub-band <NUM> of the signal <NUM> with sub-band <NUM> of the signal <NUM>, mixes sub-band <NUM> of the signal <NUM> with sub-band <NUM> of the signal <NUM>, etc. The mixer <NUM> need not mix all the bands; one or more of the bands of the signal <NUM> may be passed through when generating the enhanced audio signal <NUM>. For example, the highest frequency bands (e.g., one or more of the hybrid bands <NUM>-<NUM>) of the signal <NUM> may be passed through without mixing.

Further details of the bass enhancement system <NUM> are provided below. First, various options for the harmonics generator <NUM> are discussed, with reference to <FIG>.

<FIG> is a block diagram of a harmonics generator <NUM>. The harmonics generator <NUM> may be used as the harmonics generator <NUM> (see <FIG>). In general, the harmonics generator <NUM> generates each consecutive harmonic by multiplication (e.g., using direct signal multiplication) of the input signal and the preceding harmonics.

The harmonics generator <NUM> includes one or more multipliers <NUM> (two shown: 302a and 302b), two or more gain stages <NUM> (three shown: 304a, 304b and 304c), two or more compressors <NUM> (three shown: 306a, 306b and 306c), and two or more adders <NUM> (three shown: 308a, 308b and 308c). In general, each row of components in the harmonics generator <NUM> corresponds to one of the generated harmonics, so the number of rows (and the corresponding number of components) may be adjusted to implement the desired number of harmonics. The first processing row includes the gain stage 304a, the compressor 306a, and the adder 308a. The second processing row includes the multiplier 302a, the gain stage 304b, the compressor 306b, and the adder 308b. The third processing row includes the multiplier 302b, the gain stage 304c, the compressor 306c, and the adder 308c. Additional rows may be added to generate additional harmonics, with each new row connected to the previous row in a manner similar to what is shown in the figure.

The harmonics generator <NUM> receives an input signal <NUM>, also denoted as "x". The input signal <NUM> corresponds to the upsampled signal <NUM> (see <FIG>) when the upsampler <NUM> is present, or to the transformed audio signal <NUM> when the upsampler <NUM> is not present. The input signal <NUM> is a complex transform domain signal. For example, the input signal <NUM> may correspond to a HCQMF band (e.g., hybrid sub-band <NUM>, hybrid sub-band <NUM>, hybrid sub-band <NUM>, hybrid sub-band <NUM>, etc.). The harmonics generator <NUM> generates the signal <NUM> (see <FIG>).

Starting with the multipliers <NUM>, the multiplier 302a receives the input signal <NUM>, performs multiplication of the input signal <NUM> with itself, and generates a signal 322a, also denoted as "x<NUM>". The multiplier 302b receives the input signal <NUM> and the signal 322a, performs multiplication of the input signal <NUM> with the signal 322a, and generates a signal 322b, also denoted as "x<NUM>". Note that the output of a given multiplier is provided as an input to the multiplier in the subsequent processing row: The signal 322a is provided to the multiplier 302b, the signal 322b is provided to the multiplier in the subsequent row (shown with a dotted line), etc..

Turning to the gain stages <NUM>, the gain stage 304a receives the input signal <NUM>, applies a gain g<NUM>, and generates a signal 324a. The gain stage 304b receives the signal 322a, applies a gain g<NUM>, and generates a signal 324b. The gain stage 304c receives the signal 322b, applies a gain g<NUM>, and generates a signal 324c. The gains g<NUM>, g<NUM>, g<NUM>, etc. may be adjusted as desired, generally as a tuning exercise for each specific device that implements the harmonics generator <NUM>. In general, the gain g<NUM> may be much smaller than the other gains (e.g., less than <NUM>% of the other gains). Setting the gain g<NUM> to a small value reduces what is referred to as the direct signal corresponding to the original bass harmonic, which is undesired in small loudspeakers that are physically inadequate to reproduce any signal in the direct signal frequency range. If so desired, the gain g<NUM> may be set to zero to eliminate the direct signal.

Turning to the compressors <NUM>, the compressor 306a receives the signal 324a, performs dynamic compression, and generates a signal 326a. The compressor 306b receives the signal 324b, performs dynamic compression, and generates a signal 326b. The compressor 306c receives the signal 324c, performs dynamic compression, and generates a signal 326c. The dynamic compression generally corresponds to an equation yr, where y corresponds to the input signal (e.g., the signal 324a) and r is the compression ratio, where r is less than <NUM>. The compression ratio r may differ for each harmonic (e.g., each row). For example, the compression ratio r<NUM> for the compressor 306a may differ from the compression ratio r<NUM> for the compressor 306b, which may differ from the compression ratio r<NUM> for the compressor 306c, etc. The compression ratios may be adjusted as tuning parameters based on the specific physical characteristics of the device implementing the harmonics generator <NUM>. Further details of the compressors <NUM> are provided below in the discussion regarding loudness expansion.

Turning to the adders <NUM>, the adder 308c receives the signal 326c (and any output signal from the adder in any additional row), performs addition, and generates a signal 328b. The adder 308b receives the signal 326b and the signal 328b, performs addition, and generates a signal 328a. The adder 308a receives the signal 326a and the signal 328a, performs addition, and generates the signal <NUM> (see <FIG>). Note that one of the inputs to a given adder is provided by the adder in the subsequent processing row: The adder 308c receives the output of the adder in the subsequent processing row (shown with a dotted line), the adder 308b receives the output of the adder 308c, the adder 308a receives the output of the adder 308b, etc..

The harmonics generator <NUM> is processing complex valued signals, e.g. signals with very low contribution from negative frequencies. Hence, when generating harmonics by multiplying the complex-valued signal with itself, a much cleaner output is obtained than if the input signal is real-valued, e.g. it results in less intermodulation distortion. In the complex-valued case, for an input signal consisting of plural frequencies, only the wanted terms plus the terms from frequency sums are generated, but not the terms from frequency differences, as would be the case for real-valued processing. The difference terms are, although usually of low frequencies, more perceptually offensive than the summation terms. The summation terms may actually be desirable, e.g. when the input signal contains a harmonic series.

<FIG> is a block diagram of a harmonics generator <NUM>. The harmonics generator <NUM> may be used as the harmonics generator <NUM> (see <FIG>). In general, the harmonics generator <NUM> generates harmonics by applying a feedback delay loop to the input signal. The harmonics generator <NUM> includes a multiplier <NUM>, a gain stage <NUM>, an addition stage <NUM>, a compressor <NUM>, a delay stage <NUM>, a gain stage <NUM>, and a gain stage <NUM>.

The harmonics generator <NUM> receives an input signal <NUM>. The input signal <NUM> corresponds to the upsampled signal <NUM> (see <FIG>) when the upsampler <NUM> is present, or to the transformed audio signal <NUM> when the upsampler <NUM> is not present. The input signal <NUM> is a complex transform domain signal. For example, the input signal <NUM> may correspond to a HCQMF band (e.g., hybrid sub-band <NUM>, hybrid sub-band <NUM>, hybrid sub-band <NUM>, hybrid sub-band <NUM>, etc.). The harmonics generator <NUM> generates the signal <NUM> (see <FIG>).

The multiplier <NUM> receives the input signal <NUM>, multiplies the input signal <NUM> with a signal <NUM>, and generates a signal <NUM>. The signal <NUM> may also be referred to as the feedback signal <NUM>, and is discussed in more detail below with reference to the gain stage <NUM>.

The gain stage <NUM> receives the input signal <NUM>, applies a gain a, and generates a signal <NUM>. The gain a may also be referred to as the blend gain. The value of the gain a may be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonics generator <NUM>.

The addition stage <NUM> receives the signal <NUM> and the signal <NUM>, performs addition, and generates a signal <NUM>. The combination of the gain stage <NUM> and the addition stage <NUM>, when added to the signal <NUM>, is used to help get the feedback loop started (e.g., when the signal <NUM> is initially zero) and otherwise helps to keep the feedback loop alive.

The compressor <NUM> receives the signal <NUM>, performs dynamic compression, and generates a signal <NUM>. The dynamic compression generally corresponds to an equation yr, where y corresponds to the input signal (e.g., the signal <NUM>) and r is the compression ratio, where r is less than <NUM>. The compression ratio may be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonics generator <NUM>. Further details of the compressor <NUM> are provided below in the discussion regarding loudness expansion.

The delay stage <NUM> receives the signal <NUM>, performs a delay operation, and generates a signal <NUM>. The delay stage <NUM> may be implemented using a memory.

The gain stage <NUM> receives the signal <NUM>, applies a gain g, and generates the signal <NUM>. The gain g may also be referred to as the feedback gain. As discussed above regarding the multiplier <NUM>, the signal <NUM> is multiplied with the input signal <NUM> to generate harmonics of theoretically indefinite order.

The gain stage <NUM> receives the signal <NUM>, applies a gain h, and generates the signal <NUM> (see <FIG>). The gain h may also be referred to as the output gain. The value of the gain h may be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonics generator <NUM>.

As with the harmonics generator <NUM>, the harmonics generator <NUM> generates a direct signal corresponding to the original bass harmonic. The direct signal may be reduced, as desired, by adjusting the values of the gain a and the compression ratio r.

As with the harmonics generator <NUM>, the harmonics generator <NUM> is processing complex valued signals, and when generating harmonics by multiplying the complex-valued signal with itself, a much cleaner output is obtained than if the input signal is real-valued.

<FIG> is a block diagram of a harmonics generator <NUM>. The harmonics generator <NUM> may be used as the harmonics generator <NUM> (see <FIG>). The harmonics generator <NUM> is similar to the harmonics generator <NUM> (see <FIG>), but with the blend gain signal added after the compressor. The harmonics generator <NUM> includes a multiplier <NUM>, a compressor <NUM>, a gain stage <NUM>, an addition stage <NUM>, a delay stage <NUM>, a gain stage <NUM>, and a gain stage <NUM>.

As compared to the harmonics generator <NUM> (see <FIG>) and the harmonics generator <NUM> (see <FIG>), the harmonics generator <NUM> avoids the direct signal path by adding the input signal <NUM> later in the loop (e.g., as the signal <NUM>). In such an arrangement, the input signal <NUM> passes through the multiplier <NUM> (in contrast to the adder <NUM> in <FIG>) as part of generating the signal <NUM>, so the signal <NUM> contains no direct signal.

As with the harmonics generator <NUM> and the harmonics generator <NUM>, the harmonics generator <NUM> is processing complex valued signals, and when generating harmonics by multiplying the complex-valued signal with itself, a much cleaner output is obtained than if the input signal is real-valued.

As discussed above, because the sound pressure level for a fixed loudness range (in phon) is increasing with frequency in the bass/mid range (e.g., less than <NUM>), the harmonics generators (e.g., the harmonics generator <NUM> of <FIG>, the harmonics generator <NUM> of <FIG>, the harmonics generator <NUM> of <FIG>, the harmonics generator <NUM> of <FIG>, etc.) perform expansion in dynamics when generating their output signals. The harmonics generators may use compressors (e.g., the compressors <NUM> of <FIG>, the compressor <NUM> of <FIG>, the compressor <NUM> of <FIG>, etc.) when performing loudness expansion. Examples of loudness expansion processes include dynamic compression and loudness correction.

The harmonics generators may generate nth order harmonics using an operation corresponding to Equation (<NUM>): <MAT>.

In Equation (<NUM>), n is the order of harmonic, y is the output signal, x is the input signal, ejnφ is a complex exponential function, j is an imaginary number, and φ is the phase. The output signal is generated by multiplying the input signal by itself n times. Accordingly, increasing n increases the order of the generated harmonic. (The right-hand side of Equation (<NUM>) serves later herein as illustration why dynamic expansion ultimately results in dynamic compression when signals have been multiplied with themselves.

<FIG> is a graph <NUM> showing equal loudness curves. In the graph <NUM>, the x-axis is the frequency in Hz and the y-axis is the sound pressure level (SPL) in dB. The graph <NUM> includes <NUM> plots 602a, 602b, 602c, 602d, 602e and 602f (collectively, plots <NUM>). Each of the plots <NUM> corresponds to a loudness level in phon, which is a logarithmic measurement of perceived sound magnitude. Each of the plots <NUM> may also be referred to as an equal loudness curve. The plot 602a corresponds to the perception threshold, the plot 602b corresponds to <NUM> phon, the plot 602c corresponds to <NUM> phon, the plot 602d corresponds to <NUM> phon, the plot 602e corresponds to <NUM> phon, and the plot 602f corresponds to <NUM> phon,.

When generating harmonics by the operation described by Equation (<NUM>), the dynamics are expanded by a ratio of n. Given this information, the equal loudness plots <NUM> suggest the relationship of Equation (<NUM>): <MAT>.

In Equation (<NUM>), the term κ(f, n) is a residue expansion ratio that is related to the fundamental frequency f and the order of the harmonics n. The residue expansion ratio κ(f, n) is typically in the range of <NUM> - <NUM> depending on the fundamental frequency f and the order of the harmonics n. When the harmonics are generated according to Equation (<NUM>), the desired expansion ratio κ(f, n) may be achieved by compression of the output from the harmonic generator by a factor κ(f, n)/n. (As an aside, the terms expansion and compression may be generally used as synonyms, with "compression" used when the ratio is less than <NUM> and "expansion" used when the ratio is greater than <NUM>. So the factor κ(f, n)/n may be referred to as "compression" due to the divisor n.

In the graph <NUM>, the lines <NUM> and <NUM> illustrate an example of loudness expansion. The line <NUM> indicates a loudness range between <NUM> and <NUM> phon for a fundamental frequency of <NUM>. The line <NUM> corresponds to generating a <NUM> <NUM>th order harmonic of <NUM> having the same loudness range. An arrow <NUM> from <NUM> to <NUM> indicates generating the <NUM>th order harmonic. The dynamic SPL range of the fundamental frequency (line <NUM>) is approximately <NUM> dB within the loudness range of <NUM> to <NUM> phon, and the dynamic SPL range of the <NUM>th order harmonic (line <NUM>) is approximately <NUM> dB for the same loudness range. Hence, when generating a <NUM>th order harmonic from an <NUM> phon <NUM> fundamental, the harmonic needs to be attenuated by approximately <NUM> dB. When the fundamental instead has a loudness of <NUM> phon, the harmonic needs to be attenuated by almost <NUM> dB, an increase in the needed attenuation by approximately <NUM> dB.

The SPL-to-phon expansion ratio, also referred to as the loudness expansion, may be approximated according to Equation (<NUM>): <MAT>.

In Equation (<NUM>), R(f) is the SPL-to-phon expansion ratio, which has an inverse relation to the frequency f.

The residue expansion ratio κ(f, n), is given by Equation (<NUM>): <MAT>.

In Equation (<NUM>), the residue expansion ratio κ(f, n) corresponds to a ratio between the SPL-to-phon expansion ratio of the fundamental frequency f and the SPL-to-phon expansion ratio of the harmonic n · f, which corresponds to a ratio between the natural logarithm of n (the harmonic order) and a natural logarithm of f (the fundamental frequency). In other words, the residue expansion ratio κ(f, n) determines the factor needed when generating the nth harmonic from a fundamental frequency at f (in Hz). Equations (<NUM>) and (<NUM>) have good agreement to the equal loudness curves of <FIG> in the range <NUM>-<NUM> phon and between <NUM> and <NUM>. When using the harmonics generator <NUM> (see <FIG>) or the harmonics generator <NUM> (see <FIG>), the dynamic compression needed can be performed with sufficient accuracy using one simple compressor having a constant ratio (e.g., as the compressor <NUM> or the compressor <NUM>).

The compressor may apply the dynamic compression using a first-order averaging filter to avoid distortion due to per-sample normalization. The first-order averaging filter may process a control signal s, which may be calculated according to Equation (<NUM>): <MAT>.

In Equation (<NUM>), m is the sample number, c is a compression gain, and α is a weight between the value of the control signal for the previous sample versus the value of the compression gain for the current sample. The weight α may also be referred to as an exponential smoothing factor, and corresponds to the pole in the first order low-pass system.

The weight α may be calculated using Equation (<NUM>): <MAT>.

In Equation (<NUM>), fs is the sampling frequency and τ is a time constant.

The compression gain c may be calculated using Equation (<NUM>): <MAT>.

In Equation (<NUM>), a and b are polynomial coefficients that are applied to each magnitude order of the sample m of the input signal x. Applying the compression gain c (or the smoothed version s of Equation (<NUM>)) to a signal x as c · x (or s · x) corresponds to a rational approximation of sign(x) · |x|r, which is the absolute value of signal x subject to a compression ratio r multiplied by the signum function of x.

<FIG> is a graph <NUM> showing various compression gains c. In the graph <NUM>, the x-axis is the input power (of the input signal x) in dB and the y-axis is the compression gain c in dB. Various curves are shown, each curve corresponding to a value for the compression ratio r. Specifically, <NUM> values for r in the range from <NUM> to <NUM> are given: <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and <NUM>, with each value corresponding to one of the curves in the graph <NUM> (e.g., the value for r of <NUM> corresponds to the top curve). Note that the indicated gains of <FIG> are not exact; it is merely an illustration of the general concept. Also notable from the graph <NUM> is that the gain is limited for low input power and given by the ratio b(<NUM>)/a(<NUM>). This prevents excessive gain from being applied in circumstances such as transient onsets after quiet periods of the signal. (Instead this gain in combination with the time constant in Equation (<NUM>) allows more energy to pass through the compressor during e.g., percussive onsets, contributing to the perception of "punchiness" in the bass signal.

An alternative approach to achieve loudness expansion is by applying normalization of the input signal in a first step, before the harmonic generation, followed by a gain adjustment stage. This is referred to as loudness correction.

<FIG> is a block diagram of a harmonics generator <NUM>. The harmonics generator <NUM> generally performs loudness correction using normalization of input signals. The amplitude normalization theoretically avoids the dynamic expansion of the harmonics (by the ratio n, as n ≥ <NUM>) when generated according to Equation (<NUM>).

The harmonics generator <NUM> includes two or more normalization stages <NUM> (two shown: 802a and 802b), two or more multipliers <NUM> (two shown: 804a and 804b), two or more loudness correction stages <NUM> (two shown: 806a and 806b), two or more adders <NUM> (two shown: 808a and 808b), and an adder <NUM>. In general, each row of components in the harmonics generator <NUM> corresponds to one of the generated harmonics, so the number of rows (and the corresponding number of components) may be adjusted to implement the desired number of harmonics. The first processing row includes the normalization stage 802a, the multiplier 804a, the loudness correction stage 806a, and the adder 808a. The second processing row includes the normalization stage 802b, the multiplier 804b, the loudness correction stage 806b, and the adder 808b. Additional rows may be added to generate additional harmonics, with each new row connected to the previous row in a manner similar to what is shown in the figure.

Starting with the normalization stages <NUM>, the normalization stage 802a receives the input signal <NUM>, performs normalization, and generates a signal 822a. The normalization stage 802b receives the input signal <NUM>, performs normalization, and generates a signal 822b. Similarly to Equation (<NUM>), each of the normalization stages <NUM> may perform normalization using a first order smoothing filter to avoid distortion caused by sample-to-sample normalization. The normalization stages <NUM> may perform normalization in a manner described by Equation (<NUM>): <MAT>.

In Equation (<NUM>), x̂(m) is the current sample m of the normalized version of the input signal x, x̂(m - <NUM>) is the previous sample of the normalized version of the input signal, α is a smoothing factor, and x(m) is given by Equation (<NUM>): <MAT>.

In Equation (<NUM>), x(m) corresponds to the ratio between the complex value of the current sample of the input signal and the magnitude (also referred to as the absolute value) of the current sample of the input signal. The smoothing factor α may be adjusted as desired to control the desired smoothing time, and is dependent on the dynamics of the input signal. A smaller α is applied during attack events (e.g., when there is rapidly increasing signal energy) than under stationary or decreasing energy conditions, in order to avoid signal clipping.

Alternatively, the harmonics generator may use a single normalization stage (e.g., 802a), with the output signal (e.g., 822a) provided as an input to each of the multipliers <NUM>.

Turning to the multipliers <NUM>, the multiplier 804a receives the input signal <NUM> and the signal 822a, multiplies these signals together, and generates a signal 824a. The multiplier 804b receives the signal 822b and the signal 824a, multiplies these signals together, and generates a signal 824b. The signal 824a corresponds to the second harmonic, the signal 824b corresponds to the third harmonic, etc. Note that the output of a given multiplier is provided as an input to the multiplier in the subsequent processing row: The signal 824a is provided to the multiplier 804b, the signal 824b is provided to the multiplier in the subsequent row (shown with a dotted line), etc..

Turning to the loudness correction stages <NUM>, the loudness correction stage 806a receives the signal 824a, performs loudness correction, and generates the signal 826a. The loudness correction stage 806b receives the signal 824b, performs loudness correction, and generates the signal 826b. In general, the loudness correction stages <NUM> apply dynamic expansion and attenuation of the normalized energy of the generated harmonics, in line with the equal loudness curves of <FIG>, in order to maintain the loudness as compared to the fundamental. To adjust the loudness, a correction factor k is defined, where k is a function of the order of harmonic n, the smoothed magnitude of the fundamental x̂ (see Equation (<NUM>)) and the hybrid band index b. This correction factor k is applied according to Equation (<NUM>): <MAT>.

In Equation (<NUM>), h̃n(m) is the loudness corrected harmonic and hn(m) is the normalized harmonic, for each harmonic respectively.

As discussed above, the bass enhancement processes may be performed on one or more hybrid bands (e.g., one or more of sub-bands <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, etc.). Several harmonics, e.g. <NUM>nd, <NUM>rd and <NUM>th, are generated in every band. If we let the center frequency approximate the fundamental frequency in each band, we may calculate the SPL-to-phon relationship using one parameter: the order or the harmonics n. As an example, the first hybrid band (e.g., sub-band <NUM>) has a center frequency of <NUM> (e.g., approximately <NUM>) and the corresponding values from the ELC curves in <FIG> are listed in TABLE <NUM>:.

In TABLE <NUM>, the value between parenthesis is the SPL difference as compared to the fundamental. A function representing the SPL difference of a harmonic and its fundamental may be calculated according to Equation (<NUM>): <MAT>.

In Equation (<NUM>), Kb,n is a gain value in dB, Ab is a minimum attenuation value, X is a smoothed input fundamental energy on a logarithmic scale, while βb,n is a harmonic order n dependent scaling parameter of the input energy. βb,n may be calculated according to Equation (<NUM>): <MAT>.

The correction factor on a linear scale may be calculated according to Equation (<NUM>): <MAT>.

In Equations (<NUM>) and (<NUM>), Ab, εb and ηb are all hybrid band based constants and may be estimated for an optimal fit to the ELC curves of <FIG>. The parameters listed in TABLE <NUM> will result in adequate accuracy for the first six hybrid bands and the resulting loudness correction factors are visualized in <FIG>. For bands <NUM>, <NUM> and <NUM>, the generated harmonics are in the <NUM> to <NUM> frequency range, where the ELC curves are assumed to be flat. The loudness correction stages <NUM> may calculate the loudness correction factors using segmental linear approximation to save computational complexity.

<FIG>, <FIG>, <FIG> show a set of graphs 900a-900f. In each graph, the x-axis is the magnitude of the normalized harmonic signal into the loudness correction stage (e.g., the signal 824a input into the loudness correction stage 806a, etc.) and the y-axis is the correction factor k. The graph 900a corresponds to hybrid band <NUM>, the graph 900b corresponds to hybrid band <NUM>, the graph 900c corresponds to hybrid band <NUM>, the graph 900d corresponds to hybrid band <NUM>, the graph 900e corresponds to hybrid band <NUM>, and the graph 900f corresponds to hybrid band <NUM>. The lines for three harmonics (the <NUM>nd, <NUM>rd and <NUM>th) are shown in each graph, but the lines are overlapping in the graphs 900d, 900e and 900f as the lines converge with the increasing hybrid band number. In general, the lines show the loudness correction factors k for the first <NUM> hybrid bands when using the hybrid band based constants listed in TABLE <NUM>.

Returning to <FIG> and the adders <NUM>, the adder 808b receives the signal 826b (and any signal received from the subsequent processing row, shown with a dotted line), performs addition, and generates a signal 828b. The adder 808b receives the signal 826a and the signal 828b, performs addition, and generates a signal 828a. Note that one of the inputs to a given adder is provided by the adder in the subsequent processing row: The adder 808b receives the output of the adder in the subsequent processing row (shown with a dotted line), the adder 808a receives the output of the adder 808b, etc..

The adder <NUM> receives the input signal <NUM> and the signal 828a, performs addition, and generates the signal <NUM> (see <FIG>).

Although the description for the bass enhancement system <NUM> (see <FIG>) focused on processing a single hybrid band, similar processing may be performed on multiple hybrid bands. For example, the bass enhancement system <NUM> (see <FIG>) may be performed on four hybrid bands (e.g., sub-bands <NUM>, <NUM>, <NUM> and <NUM>), six hybrid bands (e.g., sub-bands <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and <NUM>), etc. Several harmonics (e.g., <NUM>nd, <NUM>rd, <NUM>th, etc.) are generated in every band.

<FIG> is a block diagram of a bass enhancement system <NUM>. The bass enhancement system <NUM> may be used as the bass enhancement system <NUM> (see <FIG>). The bass enhancement system <NUM> is similar to the bass enhancement system <NUM> (see <FIG>), with similar components having similar names and reference numerals, plus the addition of explicit multiple processing paths. Each processing path corresponds to processing a hybrid sub-band signal. As a specific example, four processing paths are shown (e.g., to process hybrid sub-bands <NUM>, <NUM>, <NUM> and <NUM>). The number of processing paths may be increased or decreased as desired. For example, six processing paths may be used to process the hybrid sub-bands <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and <NUM>.

The bass enhancement system <NUM> receives the transformed audio signal <NUM> (see <FIG>). As discussed above, the transformed audio signal <NUM> is a hybrid complex transform domain signal with hybrid bands. Four of the hybrid bands of the transformed audio signal <NUM> are shown as the inputs to the bass enhancement system <NUM>: sub-band <NUM> (labeled 1002a), sub-band <NUM> (1002b), sub-band <NUM> (1002c) and sub-band <NUM> (1002d). Each sub-band corresponds to one of the processing paths. The bass enhancement system <NUM> includes upsamplers <NUM> (four shown: 1010a, 1010b, 1010c and 1010d), harmonics generators <NUM> (four shown: 1012a, 1012b, 1012c and 1012d), an adder <NUM>, a dynamics processor <NUM> (optional), a converter <NUM> (optional), a filter <NUM>, a delay <NUM>, and a mixer <NUM>.

The upsampler 1010a receives the signal 1002a, performs upsampling, and generates an upsampled signal 1030a. The upsampler 1010b receives the signal 1002b, performs upsampling, and generates an upsampled signal 1030b. The upsampler 1010c receives the signal 1002c, performs upsampling, and generates an upsampled signal 1030c. The upsampler 1010d receives the signal 1002d, performs upsampling, and generates an upsampled signal 1030d. The signals 1030a, 1030b, 1030c and 1030d are complex transform domain signals. The upsamplers <NUM> are otherwise similar to that described above regarding the upsampler <NUM> (see <FIG>).

The harmonics generator 1012a receives the upsampled signal 1030a and generates harmonics thereof to result in a signal 1032a. The harmonics generator 1012b receives the upsampled signal 1030b and generates harmonics thereof to result in a signal 1032b. The harmonics generator 1012c receives the upsampled signal 1030c and generates harmonics thereof to result in a signal 1032c. The harmonics generator 1012d receives the upsampled signal 1030d and generates harmonics thereof to result in a signal 1032d. The signals 1032a, 1032b, 1032c and 1032d are complex transform domain signals. The harmonics generators <NUM> are otherwise similar to the harmonics generator <NUM> (see <FIG>). For example, one or more of the harmonics generators <NUM> may be implemented using the harmonics generator <NUM> (see <FIG>), the harmonics generator <NUM> (see <FIG>), the harmonics generator <NUM> (see <FIG>), the harmonics generator <NUM> (see <FIG>), etc..

The adder <NUM> receives the signals 1032a, 1032b, 1032c and 1032d, performs addition, and generates a signal <NUM>. The signal <NUM> is a complex transform domain signal.

The dynamics processor <NUM> receives the signal <NUM>, performs dynamics processing, and generates a signal <NUM>. The signal <NUM> is a complex transform domain signal. The dynamics processor <NUM> is otherwise similar to the dynamics processor <NUM> (see <FIG>). The dynamics processor <NUM> is optional. When the dynamics processor <NUM> is omitted, the converter <NUM> receives the signal <NUM> instead of the signal <NUM>.

The converter <NUM> receives the signal <NUM> (or the signal <NUM> when the dynamics processor <NUM> is omitted), drops the imaginary part from the signal <NUM>, and generates a signal <NUM>. The signal <NUM> is a transform domain signal. The converter <NUM> is otherwise similar to the converter <NUM> (see <FIG>), including being optional.

The filter <NUM> receives the signal <NUM> (or the signal <NUM> when the converter <NUM> is omitted, or the signal <NUM> when the dynamics processor <NUM> and the converter <NUM> are omitted), performs filtering, and generates a signal <NUM>. The signal <NUM> is a transform domain signal. The filter <NUM> is otherwise similar to the filter <NUM> (see <FIG>).

The delay <NUM> receives the signal <NUM>, implements a delay period, and generates a signal <NUM>. The signal <NUM> corresponds to a delayed version of the transformed audio signal <NUM> according to the delay period. The delay <NUM> may be implemented using a memory, a shift register, etc. The delay period corresponds to the processing time of the other components in the signal processing chain; because some of these other components are optional, the delay period decreases when the optional components are omitted. The delay <NUM> is otherwise similar to the delay <NUM> (see <FIG>).

The mixer <NUM> receives the signal <NUM> and the signal <NUM>, performs mixing, and generates the enhanced audio signal <NUM> (see <FIG>). The mixer <NUM> is otherwise similar to the mixer <NUM> (see <FIG>).

<FIG> is a mobile device architecture <NUM> for implementing the features and processes described herein, according to an embodiment. The architecture <NUM> may be implemented in any electronic device, including but not limited to: a desktop computer, consumer audio/visual (AV) equipment, radio broadcast equipment, mobile devices (e.g., smartphone, tablet computer, laptop computer, wearable device), etc.. In the example embodiment shown, the architecture <NUM> is for a laptop computer and includes processor(s) <NUM>, peripherals interface <NUM>, audio subsystem <NUM>, loudspeakers <NUM>, microphone <NUM>, sensors <NUM> (e.g., accelerometers, gyros, barometer, magnetometer, camera), location processor <NUM> (e.g., GNSS receiver), wireless communications subsystems <NUM> (e.g., Wi-Fi, Bluetooth, cellular) and I/O subsystem(s) <NUM>, which includes touch controller <NUM> and other input controllers <NUM>, touch surface <NUM> and other input/control devices <NUM>. Other architectures with more or fewer components can also be used to implement the disclosed embodiments.

Memory interface <NUM> is coupled to processors <NUM>, peripherals interface <NUM> and memory <NUM> (e.g., flash, RAM, ROM). Memory <NUM> stores computer program instructions and data, including but not limited to: operating system instructions <NUM>, communication instructions <NUM>, GUI instructions <NUM>, sensor processing instructions <NUM>, phone instructions <NUM>, electronic messaging instructions <NUM>, web browsing instructions <NUM>, audio processing instructions <NUM>, GNSS/navigation instructions <NUM> and applications/data <NUM>. Audio processing instructions <NUM> include instructions for performing the audio processing described herein.

<FIG> is a flowchart of a method <NUM> of audio processing. The method <NUM> may be performed by a device (e.g., a laptop computer, a mobile telephone, etc.) with the components of the architecture <NUM> of <FIG>, to implement the functionality of the audio processing system <NUM> (see <FIG>), the bass enhancement system <NUM> (see <FIG>), the bass enhancement system <NUM> (see <FIG>), etc., for example by executing one or more computer programs. In general, the method <NUM> performs audio signal processing in a complex-valued sub-band domain (e.g., the HCQMF domain).

At <NUM>, a first transform domain signal is received. The first transform domain signal is a hybrid complex transform domain signal having a number of bands. At least one of the bands has a number of sub-bands. The first transform domain signal has a first plurality of harmonics. For example, the bass enhancement system <NUM> (see <FIG>) may receive the transformed audio signal <NUM>. The first transform domain signal may have <NUM> hybrid bands numbered <NUM>-<NUM>, where bands <NUM>-<NUM> are sub-bands that result from splitting one or several larger bands. The first transform domain signal may be a CQMF domain signal. The first transform domain signal may be a HCQMF signal generated by splitting (e.g., by using Nyquist filter banks) a subset of the channels of a CQMF domain signal into sub-bands to increase the frequency resolution for the lowest frequency range.

At <NUM>, a second transform domain signal is generated based on the first transform domain signal. The second transform domain signal is generated by generating harmonics to of the first transform domain signal according to a non-linear process. The second transform domain signal has a second plurality of harmonics that differs from the first plurality of harmonics, and the second transform domain signal is a complex-valued signal having an imaginary part. The second transform domain signal is further generated by performing loudness expansion on the second plurality of harmonics. For example, the harmonics generator <NUM> (see <FIG>), the harmonics generator <NUM> (see <FIG>), the harmonics generator <NUM> (see <FIG>), the harmonics generator <NUM> (see <FIG>), the harmonics generator <NUM> (see <FIG>), etc. may generate the second transform domain signal (e.g., the signal <NUM>) based on the first transform domain signal (e.g., the signal <NUM>, etc.).

At <NUM>, a third transform domain signal is generated by filtering the second transform domain signal. The third transform domain signal has a number of bands, and at least one of the bands has a number of sub-bands. For example, the filter <NUM> (see <FIG>) may filter the signal <NUM> (or the signal <NUM>) to generate the signal <NUM>. As another example, the filter <NUM> (see <FIG>) may filter the signal <NUM> to generate the signal <NUM>. The third transform domain signal may have <NUM> hybrid bands numbered <NUM>-<NUM>, where bands <NUM>-<NUM> are sub-bands that result from splitting one or several larger bands. The third transform domain signal may be a HCQMF domain signal.

At <NUM>, a fourth transform domain signal is generated by mixing the third transform domain signal with a delayed version of the first transform domain signal. A given sub-band of the third transform domain signal is mixed with a corresponding sub-band of the delayed version of the first transform domain signal. For example, the mixer <NUM> (see <FIG>) may mix the signal <NUM> with the delayed signal <NUM>. As another example, the mixer <NUM> (see <FIG>) may mix the signal <NUM> with the delayed signal <NUM>. The input signals may have <NUM> hybrid bands numbered <NUM>-<NUM>, where a given band of one input signal (e.g., band <NUM>) is mixed with the corresponding band of the other input signal (e.g., band <NUM>).

The method <NUM> may include additional steps corresponding to the other functionalities of the bass enhancement system <NUM>, the bass enhancement system <NUM>, etc. as described herein. For example, the fourth transform domain signal may be outputted by a loudspeaker, such as the loudspeakers <NUM> (see <FIG>). As another example, the transform domain signals may be upsampled (e.g., using the upsampler <NUM>, the upsamplers <NUM>) prior to generating the harmonics at <NUM>. As another example, dynamics processing may be applied to the transform domain signals, e.g. using the dynamics processor <NUM> or the dynamics processor <NUM>. As another example, generating the harmonics may include performing multiplication, using a feedback delay loop, etc. As another example, the second transform domain signal may be a number of second transform domain signals, each of which corresponds to a hybrid band of the first transform domain signal. As another example, the imaginary part of the second transform domain signal may be dropped prior to generating the third transform domain signal.

An embodiment may be implemented in hardware, executable modules stored on a computer readable medium, or a combination of both (e.g., programmable logic arrays). Unless otherwise specified, the steps executed by embodiments need not inherently be related to any particular computer or other apparatus, although they may be in certain embodiments. In particular, various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatus (e.g., integrated circuits) to perform the required method steps. Thus, embodiments may be implemented in one or more computer programs executing on one or more programmable computer systems each comprising at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices, in known fashion.

Each such computer program is preferably stored on or downloaded to a storage media or device (e.g., solid state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer system to perform the procedures described herein. The inventive system may also be considered to be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer system to operate in a specific and predefined manner to perform the functions described herein. (Software per se and intangible or transitory signals are excluded to the extent that they are unpatentable subject matter.

Aspects of the systems described herein may be implemented in an appropriate computer-based sound processing network environment for processing digital or digitized audio files. Portions of the adaptive audio system may include one or more networks that comprise any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route the data transmitted among the computers. Such a network may be built on various different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

Claim 1:
A computer-implemented method of audio processing, the method (<NUM>) comprising:
receiving a first transform domain signal (<NUM>), wherein the first transform domain signal is a hybrid complex transform domain signal having a plurality of bands, wherein at least one of the plurality of bands has a plurality of sub-bands, wherein the first transform domain signal has a first plurality of harmonics;
generating an upsampled transform domain signal by upsampling the first transform domain signal, wherein the upsampled signal is a complex-valued time domain signal;
generating a second transform domain signal (<NUM>) based on the upsampled first transform domain signal by:
generating a second plurality of harmonics to the upsampled first transform domain signal according to a non-linear process, wherein the second transform domain signal has the second plurality of harmonics that differs from the first plurality of harmonics; and
performing loudness expansion or compression on the second plurality of harmonics, wherein the second transform domain signal is a complex-valued signal having an imaginary part;
filtering the second transform domain signal to split the second transform domain signal into a plurality of sub-bands and generate a third transform domain signal (<NUM>), wherein the third transform domain signal has a plurality of bands, wherein at least one of the plurality of bands has the plurality of sub-bands; and
generating a fourth transform domain signal (<NUM>) by mixing the third transform domain signal with a delayed version of the first transform domain signal, wherein a given sub-band of the third transform domain signal is mixed with a corresponding sub-band of the delayed version of the first transform domain signal.