Bass enhancement for loudspeakers

A method of audio processing includes generating harmonics in a hybrid complex quadrature mirror filter domain. Generating the harmonics may include multiplication, using a feedback delay loop, and dynamic compression. The harmonics may be generated based on one or more hybrid sub-bands of the complex transform domain signal.

FIELD

The present disclosure relates to audio processing, and in particular, to bass enhancement.

BACKGROUND

Bass effect is a desirable user experience and user evaluation indicator for mobile devices such as mobile telephones, media players, tablet computers, laptop computers, headsets, earbuds, etc. Due to the physical constraints of the transducers in mobile devices (e.g., diaphragm size, magnet weight, etc.) it is challenging for the loudspeaker of the mobile device to fully reproduce the acoustics of the original bass sound. As a result, mobile devices often implement audio processing techniques (e.g., using software processes, etc.) to improve the bass sound. These bass enhancement processes may be broadly referred to as “virtual bass” techniques.

SUMMARY

One issue with existing bass enhancement systems is that they may have a high computational complexity. Given the above, there may be a need to implement bass enhancement with reduced computational complexity.

As discussed in more detail herein, embodiments discuss techniques for bass enhancement based on the principle of the “missing fundamental”. This principle states in a psychoacoustics way that if a human listens to harmonics of a low frequency signal rather than the low frequency signal (fundamental) itself, the listener's brain is able to extrapolate and hence perceive the absent low frequency signal. Hence, for loudspeakers that are physically inadequate to reproduce low frequency signals (bass), a way to psycho-acoustically improve the quality is to generate harmonics to the low frequency range to enhance the bass effect.

The bass enhancement technique disclosed in this specification is less computationally complex as compared to conventional virtual bass technologies but reaches a similar effect. Hence, embodiments save computational complexity. In addition, the reduced complexity allows for lower latency. The technique may also include loudness adjustment schemes to adjust the power of the generated harmonics, which causes the perception of the resulting loudness to be more realistic and the bass effect to be more compelling.

The techniques disclosed in this specification may be used to enhance the output from mid-sized speakers and smaller transducers, e.g. mobile phone loudspeakers, wireless loudspeakers, etc.

According to an embodiment, a computer-implemented method of audio processing includes receiving a first transform domain signal. The first transform domain signal is a hybrid complex transform domain signal having a plurality of bands. At least one of the plurality of bands has a plurality of sub bands, and the first transform domain signal has a first plurality of harmonics.

The method further includes generating a second transform domain signal based on the first transform domain signal. The second transform domain signal is generated by generating harmonics to the first transform domain signal according to a non-linear process. The second transform domain signal has a second plurality of harmonics that differs from the first plurality of harmonics. The second transform domain signal is further generated by performing loudness expansion on the second plurality of harmonics. The second transform domain signal is a complex-valued signal having an imaginary part.

The method further includes generating a third transform domain signal by filtering the second transform domain signal. The third transform domain signal has a plurality of bands, and at least one of the plurality of bands has a plurality of sub-bands. The method further includes generating a fourth transform domain signal by mixing the third transform domain signal with a delayed version of the first transform domain signal, where a given sub-band of the third transform domain signal is mixed with a corresponding sub-band of the delayed version of the first transform domain signal.

According to another embodiment, an apparatus includes a loudspeaker and a processor. The processor is configured to control the apparatus to implement one or more of the methods described herein. The apparatus may additionally include similar details to those of one or more of the methods described herein.

According to another embodiment, a non-transitory computer readable medium stores a computer program that, when executed by a processor, controls an apparatus to execute processing including one or more of the methods described herein.

The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of various implementations.

DETAILED DESCRIPTION

In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.

In this document, the terms “and”, “or” and “and/or” are used. Such terms are to be read as having an inclusive meaning. For example, “A and B” may mean at least the following: “both A and B”, “at least both A and B”. As another example, “A or B” may mean at least the following: “at least A”, “at least B”, “both A and B”, “at least both A and B”. As another example, “A and/or B” may mean at least the following: “A and B”, “A or B”. When an exclusive-or is intended, such will be specifically noted (e.g., “either A or B”, “at most one of A and B”).

This document describes various processing functions that are associated with structures such as blocks, elements, components, circuits, etc. In general, these structures may be implemented by a processor that is controlled by one or more computer programs.

FIG.1is a block diagram of an audio processing system100. The audio processing system100generally receives an input audio signal102, processes the input audio signal102according to the bass enhancement processes described herein, and generates an output audio signal104. The audio processing system100includes a signal transform system110, a bass enhancement system120, an additional processing system130(optional), and an inverse signal transform system140. The audio processing system100may include other components that (for brevity) are not discussed in detail. The components of the audio processing system100may be implemented by one or more computer programs that are executed by a processor.

The signal transform system110receives the input audio signal102, performs a signal transform process, and generates a transformed audio signal112. The input audio signal102may be a digital time domain signal that includes a number of samples that correspond to audio (e.g., sound in waveform pulse-code modulation (PCM) format). The input audio signal102may have a sample rate of 32 kHz, 44.1 kHz, 48 kHz, 192 kHz, etc. The input audio signal102may originate from a variety of formats, including the Advanced Television Systems Committee (ATSC) Digital Audio Compression (AC-3, E-AC-3) Standard. As a specific example, the input audio signal102may originate from a Dolby Digital Plus™ signal with a sample rate of 48 kHz.

The signal transform system110may perform a variety of signal transform processes. In general, the signal transform process transforms the input audio signal102from a first signal domain to a second signal domain. For example, the first domain may be the time domain, and the second signal domain may be the frequency domain, the quadrature mirror frequency (QMF) domain, the complex quadrature mirror frequency (CQMF) domain, the hybrid complex quadrature mirror frequency (HCQMF) domain, etc. The transform from the first signal domain to the second signal domain may also be referred to as “analysis”, e.g. transform analysis, signal analysis, filter bank analysis, QMF analysis, CQMF analysis, HCQMF analysis, etc.

In general, QMF domain information is generated by a filter whose frequency response is the mirror image around π/2 of that of another filter; together these filters are known as a QMF pair. QMF theory also comprises filter banks with more channels than two (e.g., 64 channels); these may be referred to as M-channel QMF banks. QMF theory further teaches M-channel Pseudo QMF banks of the class referred to as modulated filter banks. In general, “CQMF” domain information results from a complex-modulated discrete Fourier transform (DFT) filter bank applied to a time-domain signal. The CQMF is a “complex” signal because it includes complex valued signals, e.g. signals that include an imaginary part in addition to the real part. In general, “HCQMF” domain information corresponds to CQMF domain information in which the CQMF filter bank has been extended to a hybrid structure to obtain an efficient non-uniform frequency resolution that better matches the frequency resolution of the human auditory system. In general, the term “hybrid” refers to a structure in which at least one frequency band is split into sub-bands.

According to a specific HCQMF implementation, the HCQMF information is generated into 77 frequency bands, where the lower CQMF bands are further split into sub-bands in order to obtain a higher frequency resolution for the lower frequencies. According to a further specific implementation, the signal transform system110transforms each channel of the input audio signal102into 64 CQMF bands, and further divides the lowest 3 bands into sub-bands as follows: the first band is divided into 8 sub-bands, and the second and third bands are each divided into 4 sub-bands. (This hybrid splitting of the lowest bands into sub-bands is to improve the low-frequency resolution of these bands.) The signal transform system110may include Nyquist filters to split the bands into sub-bands. The 77 HCQMF bands then correspond to the 61 highest CQMF bands, plus the 16 sub-bands (8+4+4) from the lowest 3 CQMF bands. The sub-bands and bands may be numbered from 0 to 76, with the lowest frequency sub-band being number 0. The other sub-bands are then numbered from 1 to 15, and the remaining bands are numbered from 16 to 76. These 77 HCQMF bands may then be referred to as “hybrid bands” or “channels” along with their number, e.g., hybrid band 0, hybrid band 1, hybrid band 76, channel 0, channel 1, channel 76, etc. The hybrid bands 0-15 may also be referred to as “sub-bands” along with their number, e.g., sub-band 0, sub-band 1, sub-band 15, etc. The hybrid bands 16-76 may also be referred to as “bands” along with their number, e.g., band 16, band 17, band 76, etc. The channels 1 and 3 may have passbands on the negative frequency axis, but generally the other channels do not.

(Note that the terms QMF, CQMF and HCQMF are used a bit colloquially herein. Specifically, the terms QMF/CQMF may be used colloquially to refer to a DFT filter bank that may include more than two bands. The term HCQMF may be used colloquially to refer to a non-uniform DFT filter bank that may include more than two bands.)

As a specific example, the signal transform system110performs a HCQMF transform on the input audio signal102to generate the transformed audio signal112having 77 frequency bands. In this case, the signal domain of the transformed audio signal112may be referred to as the HCQMF domain or the hybrid domain, and the HCQMF transform may be referred to as HCQMF analysis.

The bandwidth and the sampling frequency of the bands will depend upon the sampling frequency of the input audio signal102. For example, when the input audio signal102has a sampling frequency of 48 kHz (corresponding to a maximum bandwidth of 24 kHz), the hybrid structure with 77 bands discussed above results in a sampling frequency of 750 Hz for all bands. The 61 bands with the highest frequencies have a passband bandwidth of 375 Hz; the 8 lowest-frequency sub-bands have a passband bandwidth of 93.75 Hz; and the next-lowest-frequency sub-bands have a passband bandwidth of 187.5 Hz.

The bass enhancement system120receives the transformed audio signal112, performs bass enhancement, and generates an enhanced audio signal122. In general, the bass enhancement system120generates harmonics to the transformed audio signal112in order for the listener to psycho-acoustically perceive the missing fundamental. Further details of the bass enhancement system120are provided below (e.g., with reference toFIG.2, etc.).

The additional processing system130is optional. When present, the additional processing system130receives the enhanced audio signal122, performs additional signal processing, and generates a processed audio signal132. Alternatively, the additional processing system130may operate on the transformed audio signal112prior to the operation of the bass enhancement system120, in which case the bass enhancement system120receives as its input the signal output from the additional processing system130(instead of receiving the output signal directly from the signal transform system110). As another option, the additional processing system130may be multiple additional processing systems that operate both before and after the bass enhancement system120. The specific arrangement of the additional processing system130within the audio processing system100may vary according to the specific types of additional processing that the additional processing system130performs.

In general, the additional processing system130performs additional processing of the input audio signal102in the transform domain. This allows the bass enhancement system120to operate in combination with existing audio processing techniques that are implemented in the transform domain. Examples of the additional processing include dialogue enhancement, intelligent equalization, volume leveling, spectral limiting, etc. Dialogue enhancement refers to enhancing speech signals (e.g., as compared to sound effects), in order to improve the intelligibility of the speech. Intelligent equalization refers to performing dynamic adjustment of the audio tone, e.g. to provide consistency of spectral balance (also known as “tone” or “timbre”). Volume leveling refers to increasing the volume of quiet audio and decreasing the volume of loud audio, e.g. to reduce the need for a listener to perform manual adjustment of the volume. Spectral limiting refers to limiting selected frequencies or frequency bands, e.g. to limit the lowest frequencies that are difficult to output from small loudspeakers.

The inverse signal transform system140receives the enhanced audio signal122(or optionally the processed audio signal132), performs an inverse transform, and generates the output audio signal104. The inverse transform generally converts a signal from the second signal domain back into the first signal domain. In general, the inverse transform is an inverse of the signal transform process performed by the signal transform system110. For example, when the signal transform system110performs a HCQMF transform, the inverse signal transform system140performs an inverse HCQMF transform. The transform from the second signal domain back to the first signal domain may also be referred to as “synthesis”, e.g. transform synthesis, signal synthesis, filter bank synthesis, etc.; and the inverse HCQMF transform may be referred to as HCQMF synthesis.

In this manner, the output audio signal104corresponds to the input audio signal102, with the addition of the bass enhancement and/or additional signal enhancements. The output audio signal104may then be output by a loudspeaker and perceived as sound by the listener.

As discussed above and in more detail below, the bass enhancement system120is suitable for small to mid-sized speakers. The processes implemented by the bass enhancement system120may be simpler than many existing bass enhancement methods; as compared to these existing methods, the bass enhancement system120has lower computational complexity and allows for short latency, while still retaining the audio quality. The bass enhancement system120is well suited for mid-sized speakers in e.g. TV sets or wireless speakers, and is also efficient for bass improvement of small transducers, e.g. for mobile phones, laptops and tablets. The bass enhancement system120in one mode of operation not only adds harmonics to the mix, but also adds the (dynamically changed) original bass, i.e. it may be operated to have an inherent bass boost.

FIG.2is a block diagram of a bass enhancement system200. The bass enhancement system200may be used as the bass enhancement system120(seeFIG.1). For brevity, the description ofFIG.2focuses on a single signal processing path in order to describe the general operation of bass enhancement system200; additional signal processing paths may also be implemented in variations of the bass enhancement systems described herein (see, e.g.,FIG.10). The additional signal processing paths will also be briefly described here.

The bass enhancement system200receives the transformed audio signal112(seeFIG.1). As discussed above, the transformed audio signal112is a hybrid complex transform domain signal (e.g., a HCQMF domain signal) with a number of bands (e.g., 77 hybrid bands, with the 3 lowest-frequency bands split into sub-bands). As a complex signal, the transformed audio signal112has complex values, e.g. both real values and imaginary values. Each sub-band may be processed in its own processing path, so the following description focuses on processing one sub-band (e.g., one of sub-bands 0, 2, 4, 6, etc.). The bass enhancement system200includes an upsampler (optional)202, a harmonics generator204, a dynamics processor206(optional), a converter208(optional), a filter212, a delay214, and a mixer216.

The upsampler202receives the transformed audio signal112, performs upsampling, and generates an upsampled signal220. As an example, when the input audio signal102(seeFIG.1) has a sampling frequency of 48 kHz, and the transformed audio signal112is processed into 64 bands, each band has a sampling frequency of 750 Hz. The upsampler202may upsample the selected sub-band of the transformed audio signal112by 2×, 3×, 4×, 5×, 6×, etc. A suitable amount of upsampling is 4×, e.g. so that the upsampled signal220has a sampling frequency of 3 kHz when the selected sub-band of the transformed audio signal112has a sampling frequency of 750 Hz. The upsampled signal220is a complex transform domain signal. The upsampled signal220has a bandwidth that corresponds to the bandwidth of the selected sub-band of the transformed audio signal112. As an example, when the selected sub-band 0 having a passband bandwidth of 93.75 Hz is input to the upsampler, the upsampled signal220likewise has a bandwidth of 93.75 Hz.

The upsampler202may be implemented by performing CQMF synthesis. As an example, to upsample sub-band 0 from 750 Hz to 3000 Hz (4× upsampling), the upsampler may implement 4-channel CQMF synthesis, with one input being the sub-band 0 and the other 3 inputs being zero (null). The synthesis is configured as to maintain the signal220being a complex-valued time domain signal.

The upsampler202is optional. In general, the upsampler202provides additional headroom when generating the harmonics (see the harmonics generator204), to allow bandwidth extension without aliasing (also referred to as spectral folding). The upsampler202may be omitted when processing one or more of the lowest frequency sub-bands. For example, when processing the lowest band (e.g., sub-band 0) only, the upsampler202may be omitted, as up to (at least) 6thorder harmonics may be generated without folding. Processing the lowest two bands (e.g., sub-bands 0 and 2), the upsampler202may be omitted if only 2ndand 3rdorder harmonics are generated. Processing the lowest three bands (e.g., sub-bands 0, 2 and 4), only 2ndorder harmonics may be generated without aliasing. This is discussed in more detail with reference to the harmonics generator204.

The harmonics generator204receives the upsampled signal220(or the selected sub-band signal of the transformed audio signal112when the upsampler202is omitted) and generates harmonics thereof to result in a signal222. As mentioned with reference to the upsampler202, the harmonics generator204extends the bandwidth of its input signal when generating the harmonics for the signal222. For example, when sub-band 0 covers 0 to 93.75 Hz, the sampling frequency of 750 Hz may be sufficient to avoid aliasing of the generated harmonics. Similarly, when sub-band 2 covers 93.75 to 187.5 Hz, the sampling frequency of 750 Hz may be sufficient to avoid aliasing of the generated harmonics. However, when sub-band 4 covers 187.5 to 281.25 Hz, the harmonics are approaching the Nyquist frequency of the original signal (with the sampling frequency of 750 Hz), so upsampling is recommended for sub-bands 4, 6, etc. The signal222is a complex transform domain signal. The signal222has a bandwidth that is greater than the bandwidth of the input to the harmonics generator204, due to the addition of the harmonic frequencies. For example, when the upsampled signal220has a bandwidth of 93.75 Hz, the signal222may have a bandwidth that exceeds 300 Hz.

The harmonics generator204uses a non-linear process to generate the harmonics. In general, a non-linear process applies different gains to different components of the signal. Examples of the non-linear processes include multiplication, a feedback delay loop, rectification, etc. as further detailed below with reference toFIGS.3,4,5and8.

The harmonics generator204may also perform loudness expansion when generating the signal222. Because the sound pressure level for a fixed loudness range (in phon) is increasing with frequency in the bass/mid range (e.g., less than 800 Hz), the harmonics generator204performs expansion in dynamics when generating the signal222. Examples of loudness expansion processes include dynamic compression and loudness correction. Further details of the loudness expansion are provided with reference toFIG.6below.

The dynamics processor206receives the signal222, performs dynamics processing, and generates a signal224. The signal224is a complex transform domain signal. In general, the dynamics processor206implements dynamics processing by performing compression on the signal222, in order to control the transient to tonal ratio of the signal224. The dynamics processor206may implement an attack time that is relatively longer (e.g., between 4× to 12× longer, such as 8× longer) than the release time. For example, the attack time may be between 140 and 180 ms (e.g., 160 ms) and the release time may be between 15 and 25 ms (e.g., 20 ms). The dynamics processor206may implement de-coupled smooth peak detection using feed-forward topology. The dynamics processor206may implement compression similar to the compression performed by the harmonics generator (described in more detail with reference toFIGS.3,4and5).

The dynamics processor206is optional. When the dynamics processor206is omitted, the converter208receives the signal222instead of the signal224.

The converter208receives the signal224(or the signal222when the dynamics processor206is omitted), drops the imaginary part from the signal224, and generates a signal228. In general, dropping the imaginary part lowers the computational complexity of subsequent analysis filter banks (e.g., the filter212), due to processing real-valued signals instead of complex-valued signals. As discussed above, the signal224is a complex transform domain signal that has complex values, e.g. both real values and imaginary values. The converter208may drop the imaginary part of the signal224by taking the real part of the complex-valued signal. The signal228is a real-valued transform domain signal.

The converter208is optional and may be omitted in some embodiments of the bass enhancement system200. When the upsampler202is omitted, the converter208should also be omitted, in order for the imaginary part to remain in the signal processing path for use by subsequent components.

The filter212receives the signal228(or the signal224when the converter208is omitted, or the signal222when the dynamics processor206and the converter208are omitted), performs filtering of the input, and generates a signal230. The signal230is a complex-valued transform domain signal. The filtering generally splits the signal228into sub-bands as one of the inputs to the mixer216. The specifics of the filtering will depend upon whether or not upsampling was performed (see the upsampler202).

When the upsampler202is not present, the filter212may be implemented by feeding the input signal (e.g., the signal228) into an 8-channel Nyquist filter bank to generate the signal230that has hybrid sub-bands 0-7.

When the upsampler202is present, the filter212may be implemented by a CQMF analysis filter bank and two or more Nyquist filters. The real part of the input signal (e.g., the signal228) is fed into the CQMF analysis filter bank; the CQMF analysis filter bank has an appropriate number of channels to generate the signal230having sub-band signals of 750 Hz sampling frequency. The appropriate number of channels then depends on the upsampling performed. For example, when 4× upsampling is performed, and hence a 4 channel CQMF analysis bank is used in the filter212, the three lowest frequency CQMF sub-band signals are each fed into a corresponding Nyquist filter (one generating hybrid sub-bands 0-7, one generating hybrid sub-bands 8-11, and one generating hybrid sub-bands 12-15). As another example, when 2× upsampling is performed, and hence a 2 channel CQMF analysis bank is used in the filter212, the two CQMF sub-band signals are each fed into a corresponding Nyquist filter (one generating hybrid sub-bands 0-7, and one generating hybrid sub-bands 8-11). The remaining CQMF channels, if any, are provided to the mixer216(with an appropriate delay corresponding to the delay of the Nyquist filters).

The filter212may be implemented with filters similar to those used by the signal transform system110(seeFIG.1). For example, a first Nyquist analysis filter with 8 channels may generate the sub-bands 0-7, a second Nyquist analysis filter with 4 channels may generate the sub-bands 8-11, and a third Nyquist analysis filter with 4 channels may generate the sub-bands 12-15.

The delay214receives the transformed audio signal112, implements a delay period, and generates a signal232. The signal232corresponds to a delayed version of the transformed audio signal112according to the delay period. The delay214may be implemented using a memory, a shift register, etc. The delay period corresponds to the processing time of the other components in the signal processing chain, e.g. the upsampler202, the harmonics generator204, the dynamics processor206, the converter208, the filter212, etc. Because some of these other components are optional, the delay period decreases as more of the optional components are omitted. In one example, the delay period is 961 samples, of which577correspond to the upsampling, and384correspond to the remaining components, e.g. the Nyquist filters. As another example, the delay period is 384 samples when the upsampler202is omitted.

The mixer216receives the signal230and the signal232, performs mixing, and generates the enhanced audio signal122(seeFIG.1). The enhanced audio signal122is a transform domain signal. The mixer216mixes the signals on a per-band basis. For example, the signal230and the signal232may each have 77 hybrid bands (e.g., 8+4+4+61 HCQMF bands), and the mixer216mixes sub-band 0 of the signal230with sub-band 0 of the signal232, mixes sub-band 1 of the signal230with sub-band 1 of the signal232, etc. The mixer216need not mix all the bands; one or more of the bands of the signal232may be passed through when generating the enhanced audio signal122. For example, the highest frequency bands (e.g., one or more of the hybrid bands 16-77) of the signal232may be passed through without mixing.

Further details of the bass enhancement system200are provided below. First, various options for the harmonics generator204are discussed, with reference toFIGS.3-5.

FIG.3is a block diagram of a harmonics generator300. The harmonics generator300may be used as the harmonics generator204(seeFIG.2). In general, the harmonics generator300generates each consecutive harmonic by multiplication (e.g., using direct signal multiplication) of the input signal and the preceding harmonics.

The harmonics generator300includes one or more multipliers302(two shown:302aand302b), two or more gain stages304(three shown:304a,304band304c), two or more compressors306(three shown:306a,306band306c), and two or more adders308(three shown:308a,308band308c). In general, each row of components in the harmonics generator300corresponds to one of the generated harmonics, so the number of rows (and the corresponding number of components) may be adjusted to implement the desired number of harmonics. The first processing row includes the gain stage304a, the compressor306a, and the adder308a. The second processing row includes the multiplier302a, the gain stage304b, the compressor306b, and the adder308b. The third processing row includes the multiplier302b, the gain stage304c, the compressor306c, and the adder308c. Additional rows may be added to generate additional harmonics, with each new row connected to the previous row in a manner similar to what is shown in the figure.

The harmonics generator300receives an input signal320, also denoted as “x”. The input signal320corresponds to the upsampled signal220(seeFIG.2) when the upsampler202is present, or to the transformed audio signal112when the upsampler202is not present. The input signal320is a complex transform domain signal. For example, the input signal320may correspond to a HCQMF band (e.g., hybrid sub-band 0, hybrid sub-band 2, hybrid sub-band 4, hybrid sub-band 6, etc.). The harmonics generator300generates the signal222(seeFIG.2).

Starting with the multipliers302, the multiplier302areceives the input signal320, performs multiplication of the input signal320with itself, and generates a signal322a, also denoted as “x2”. The multiplier302breceives the input signal320and the signal322a, performs multiplication of the input signal320with the signal322a, and generates a signal322b, also denoted as “x3”. Note that the output of a given multiplier is provided as an input to the multiplier in the subsequent processing row: The signal322ais provided to the multiplier302b, the signal322bis provided to the multiplier in the subsequent row (shown with a dotted line), etc.

Turning to the gain stages304, the gain stage304areceives the input signal320, applies a gain g1, and generates a signal324a. The gain stage304breceives the signal322a, applies a gain g2, and generates a signal324b. The gain stage304creceives the signal322b, applies a gain g3, and generates a signal324c. The gains g1, g2, g3, etc. may be adjusted as desired, generally as a tuning exercise for each specific device that implements the harmonics generator300. In general, the gain g1may be much smaller than the other gains (e.g., less than 50% of the other gains). Setting the gain g1to a small value reduces what is referred to as the direct signal corresponding to the original bass harmonic, which is undesired in small loudspeakers that are physically inadequate to reproduce any signal in the direct signal frequency range. If so desired, the gain g1may be set to zero to eliminate the direct signal.

Turning to the compressors306, the compressor306areceives the signal324a, performs dynamic compression, and generates a signal326a. The compressor306breceives the signal324b, performs dynamic compression, and generates a signal326b. The compressor306creceives the signal324c, performs dynamic compression, and generates a signal326c. The dynamic compression generally corresponds to an equation yr, where y corresponds to the input signal (e.g., the signal324a) and r is the compression ratio, where r is less than 1. The compression ratio r may differ for each harmonic (e.g., each row). For example, the compression ratio r1for the compressor306amay differ from the compression ratio r2for the compressor306b, which may differ from the compression ratio r3for the compressor306c, etc. The compression ratios may be adjusted as tuning parameters based on the specific physical characteristics of the device implementing the harmonics generator300. Further details of the compressors306are provided below in the discussion regarding loudness expansion.

Turning to the adders308, the adder308creceives the signal326c(and any output signal from the adder in any additional row), performs addition, and generates a signal328b. The adder308breceives the signal326band the signal328b, performs addition, and generates a signal328a. The adder308areceives the signal326aand the signal328a, performs addition, and generates the signal222(seeFIG.2). Note that one of the inputs to a given adder is provided by the adder in the subsequent processing row: The adder308creceives the output of the adder in the subsequent processing row (shown with a dotted line), the adder308breceives the output of the adder308c, the adder308areceives the output of the adder308b, etc.

The harmonics generator300is processing complex valued signals, e.g. signals with very low contribution from negative frequencies. Hence, when generating harmonics by multiplying the complex-valued signal with itself, a much cleaner output is obtained than if the input signal is real-valued, e.g. it results in less intermodulation distortion. In the complex-valued case, for an input signal consisting of plural frequencies, only the wanted terms plus the terms from frequency sums are generated, but not the terms from frequency differences, as would be the case for real-valued processing. The difference terms are, although usually of low frequencies, more perceptually offensive than the summation terms. The summation terms may actually be desirable, e.g. when the input signal contains a harmonic series.

FIG.4is a block diagram of a harmonics generator400. The harmonics generator400may be used as the harmonics generator204(seeFIG.2). In general, the harmonics generator400generates harmonics by applying a feedback delay loop to the input signal. The harmonics generator400includes a multiplier402, a gain stage404, an addition stage406, a compressor408, a delay stage410, a gain stage412, and a gain stage414.

The harmonics generator400receives an input signal420. The input signal420corresponds to the upsampled signal220(seeFIG.2) when the upsampler202is present, or to the transformed audio signal112when the upsampler202is not present. The input signal420is a complex transform domain signal. For example, the input signal420may correspond to a HCQMF band (e.g., hybrid sub-band 0, hybrid sub-band 2, hybrid sub-band 4, hybrid sub-band 6, etc.). The harmonics generator400generates the signal222(seeFIG.2).

The multiplier402receives the input signal420, multiplies the input signal420with a signal432, and generates a signal422. The signal432may also be referred to as the feedback signal432, and is discussed in more detail below with reference to the gain stage412.

The gain stage404receives the input signal420, applies a gain a, and generates a signal424. The gain a may also be referred to as the blend gain. The value of the gain a may be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonics generator400.

The addition stage406receives the signal422and the signal424, performs addition, and generates a signal426. The combination of the gain stage404and the addition stage406, when added to the signal422, is used to help get the feedback loop started (e.g., when the signal432is initially zero) and otherwise helps to keep the feedback loop alive.

The compressor408receives the signal426, performs dynamic compression, and generates a signal428. The dynamic compression generally corresponds to an equation yr, where y corresponds to the input signal (e.g., the signal426) and r is the compression ratio, where r is less than 1. The compression ratio may be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonics generator400. Further details of the compressor408are provided below in the discussion regarding loudness expansion.

The delay stage410receives the signal428, performs a delay operation, and generates a signal430. The delay stage410may be implemented using a memory.

The gain stage412receives the signal430, applies a gain g, and generates the signal432. The gain g may also be referred to as the feedback gain. As discussed above regarding the multiplier402, the signal432is multiplied with the input signal420to generate harmonics of theoretically indefinite order.

The gain stage414receives the signal428, applies a gain h, and generates the signal222(seeFIG.2). The gain h may also be referred to as the output gain. The value of the gain h may be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonics generator400.

As with the harmonics generator300, the harmonics generator400generates a direct signal corresponding to the original bass harmonic. The direct signal may be reduced, as desired, by adjusting the values of the gain a and the compression ratio r.

As with the harmonics generator300, the harmonics generator400is processing complex valued signals, and when generating harmonics by multiplying the complex-valued signal with itself, a much cleaner output is obtained than if the input signal is real-valued.

FIG.5is a block diagram of a harmonics generator500. The harmonics generator500may be used as the harmonics generator204(seeFIG.2). The harmonics generator500is similar to the harmonics generator400(seeFIG.4), but with the blend gain signal added after the compressor. The harmonics generator500includes a multiplier502, a compressor504, a gain stage506, an addition stage508, a delay stage510, a gain stage512, and a gain stage514.

The harmonics generator500receives an input signal520. The input signal520corresponds to the upsampled signal220(seeFIG.2) when the upsampler202is present, or to the transformed audio signal112when the upsampler202is not present. The input signal520is a complex transform domain signal. For example, the input signal520may correspond to a HCQMF band (e.g., hybrid sub-band 0, hybrid sub-band 2, hybrid sub-band 4, hybrid sub-band 6, etc.). The harmonics generator500generates the signal222(seeFIG.2).

The multiplier502receives the input signal520, multiplies the input signal520with a signal532, and generates a signal522. The signal532may also be referred to as the feedback signal532, and is discussed in more detail below with reference to the gain stage512.

The compressor504receives the signal522, performs dynamic compression, and generates a signal524. The dynamic compression generally corresponds to an equation yr, where y corresponds to the input signal (e.g., the signal522) and r is the compression ratio, where r is less than 1. The compression ratio may be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonics generator500. Further details of the compressor504are provided below in the discussion regarding loudness expansion.

The gain stage506receives the input signal520, applies a gain a, and generates a signal526. The gain a may also be referred to as the blend gain. The value of the gain a may be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonics generator500.

The addition stage508receives the signal524and the signal526, performs addition, and generates a signal528. The combination of the gain stage506and the addition stage508, when added to the signal524, is used to help get the feedback loop started (e.g., when the signal532is initially zero) and otherwise helps to keep the feedback loop alive.

The delay stage510receives the signal528, performs a delay operation, and generates a signal530. The delay stage510may be implemented using a memory.

The gain stage512receives the signal530, applies a gain g, and generates the signal532. The gain g may also be referred to as the feedback gain. As discussed above regarding the multiplier502, the signal532is multiplied with the input signal520to generate harmonics of theoretically indefinite order.

The gain stage514receives the signal524, applies a gain h, and generates the signal222(seeFIG.2). The gain h may also be referred to as the output gain. The value of the gain h may be adjusted as a tuning parameter based on the specific physical characteristics of the device implementing the harmonics generator500.

As compared to the harmonics generator300(seeFIG.3) and the harmonics generator400(seeFIG.4), the harmonics generator500avoids the direct signal path by adding the input signal520later in the loop (e.g., as the signal526). In such an arrangement, the input signal520passes through the multiplier502(in contrast to the adder406inFIG.4) as part of generating the signal222, so the signal222contains no direct signal.

As with the harmonics generator300and the harmonics generator400, the harmonics generator500is processing complex valued signals, and when generating harmonics by multiplying the complex-valued signal with itself, a much cleaner output is obtained than if the input signal is real-valued.

Loudness Expansion

As discussed above, because the sound pressure level for a fixed loudness range (in phon) is increasing with frequency in the bass/mid range (e.g., less than 800 Hz), the harmonics generators (e.g., the harmonics generator204ofFIG.2, the harmonics generator300ofFIG.3, the harmonics generator400ofFIG.4, the harmonics generator500ofFIG.5, etc.) perform expansion in dynamics when generating their output signals. The harmonics generators may use compressors (e.g., the compressors306ofFIG.3, the compressor408ofFIG.4, the compressor504ofFIG.5, etc.) when performing loudness expansion. Examples of loudness expansion processes include dynamic compression and loudness correction.

Dynamic Compression

The harmonics generators may generate nthorder harmonics using an operation corresponding to Equation (1):
yn=xn=|x|n·ejnφ(1)

In Equation (1), n is the order of harmonic, y is the output signal, x is the input signal, ejnφis a complex exponential function, j is an imaginary number, and φ is the phase. The output signal is generated by multiplying the input signal by itself n times. Accordingly, increasing n increases the order of the generated harmonic. (The right-hand side of Equation (1) serves later herein as illustration why dynamic expansion ultimately results in dynamic compression when signals have been multiplied with themselves.)

FIG.6is a graph600showing equal loudness curves. In the graph600, the x-axis is the frequency in Hz and the y-axis is the sound pressure level (SPL) in dB. The graph600includes 6 plots602a,602b,602c,602d,602eand602f(collectively, plots602). Each of the plots602corresponds to a loudness level in phon, which is a logarithmic measurement of perceived sound magnitude. Each of the plots602may also be referred to as an equal loudness curve. The plot602acorresponds to the perception threshold, the plot602bcorresponds to 20 phon, the plot602ccorresponds to 40 phon, the plot602dcorresponds to 60 phon, the plot602ecorresponds to 80 phon, and the plot602fcorresponds to 100 phon,

When generating harmonics by the operation described by Equation (1), the dynamics are expanded by a ratio of n. Given this information, the equal loudness plots602suggest the relationship of Equation (2):
yn=|x|κ(f,n)·ejnφ(2)

In Equation (2), the term κ(f, n) is a residue expansion ratio that is related to the fundamental frequency f and the order of the harmonics n. The residue expansion ratio κ(f, n) is typically in the range of 1.1-1.4 depending on the fundamental frequency f and the order of the harmonics n. When the harmonics are generated according to Equation (1), the desired expansion ratio κ(f, n) may be achieved by compression of the output from the harmonic generator by a factor κ(f, n)/n. (As an aside, the terms expansion and compression may be generally used as synonyms, with “compression” used when the ratio is less than 1 and “expansion” used when the ratio is greater than 1. So the factor κ(f, n)/n may be referred to as “compression” due to the divisor n.)

In the graph600, the lines610and612illustrate an example of loudness expansion. The line610indicates a loudness range between 20 and 80 phon for a fundamental frequency of 50 Hz. The line612corresponds to generating a 50 Hz 4thorder harmonic of 400 Hz having the same loudness range. An arrow614from610to612indicates generating the 4th order harmonic. The dynamic SPL range of the fundamental frequency (line610) is approximately 38 dB within the loudness range of 20 to 80 phon, and the dynamic SPL range of the 4thorder harmonic (line612) is approximately 50 dB for the same loudness range. Hence, when generating a 4thorder harmonic from an 80 phon 50 Hz fundamental, the harmonic needs to be attenuated by approximately 20 dB. When the fundamental instead has a loudness of 20 phon, the harmonic needs to be attenuated by almost 40 dB, an increase in the needed attenuation by approximately 20 dB.

The SPL-to-phon expansion ratio, also referred to as the loudness expansion, may be approximated according to Equation (3):

In Equation (3), R(f) is the SPL-to-phon expansion ratio, which has an inverse relation to the frequency f.

The residue expansion ratio κ(f, n), is given by Equation (4):

In Equation (4), the residue expansion ratio κ(f, n) corresponds to a ratio between the SPL-to-phon expansion ratio of the fundamental frequency f and the SPL-to-phon expansion ratio of the harmonic n·f, which corresponds to a ratio between the natural logarithm of n (the harmonic order) and a natural logarithm of f (the fundamental frequency). In other words, the residue expansion ratio κ(f, n) determines the factor needed when generating the nthharmonic from a fundamental frequency at f (in Hz). Equations (3) and (4) have good agreement to the equal loudness curves ofFIG.6in the range 20-80 phon and between 20 and 1000 Hz. When using the harmonics generator400(seeFIG.4) or the harmonics generator500(seeFIG.5), the dynamic compression needed can be performed with sufficient accuracy using one simple compressor having a constant ratio (e.g., as the compressor408or the compressor504).

The compressor may apply the dynamic compression using a first-order averaging filter to avoid distortion due to per-sample normalization. The first-order averaging filter may process a control signal s, which may be calculated according to Equation (5):
s(m)=α·s(m−1)+(1−α)·c(m)  (5)

In Equation (5), m is the sample number, c is a compression gain, and a is a weight between the value of the control signal for the previous sample versus the value of the compression gain for the current sample. The weight a may also be referred to as an exponential smoothing factor, and corresponds to the pole in the first order low-pass system.

The weight a may be calculated using Equation (6):
α=e−1/(τfs) and τ≈20e−3s(6)

In Equation (6), fsis the sampling frequency and τ is a time constant.

The compression gain c may be calculated using Equation (7):

In Equation (7), a and b are polynomial coefficients that are applied to each magnitude order of the sample m of the input signal x. Applying the compression gain c (or the smoothed version s of Equation (5)) to a signal x as c·x (or s·x) corresponds to a rational approximation of sign(x)·|x|r, which is the absolute value of signal x subject to a compression ratio r multiplied by the signum function of x.

FIG.7is a graph700showing various compression gains c. In the graph700, the x-axis is the input power (of the input signal x) in dB and the y-axis is the compression gain c in dB. Various curves are shown, each curve corresponding to a value for the compression ratio r. Specifically, 9 values for r in the range from 0.5 to 1.0 are given: 0.5, 0.6, 0.65, 0.7, 0.73, 0.77, 0.8, 0.9 and 1.0, with each value corresponding to one of the curves in the graph700(e.g., the value for r of 0.5 corresponds to the top curve). Note that the indicated gains ofFIG.7are not exact; it is merely an illustration of the general concept. Also notable from the graph700is that the gain is limited for low input power and given by the ratio b(0)/a(0). This prevents excessive gain from being applied in circumstances such as transient onsets after quiet periods of the signal. (Instead this gain in combination with the time constant in Equation (6) allows more energy to pass through the compressor during e.g., percussive onsets, contributing to the perception of “punchiness” in the bass signal.)

Loudness Correction

An alternative approach to achieve loudness expansion is by applying normalization of the input signal in a first step, before the harmonic generation, followed by a gain adjustment stage. This is referred to as loudness correction.

FIG.8is a block diagram of a harmonics generator800. The harmonics generator800generally performs loudness correction using normalization of input signals. The amplitude normalization theoretically avoids the dynamic expansion of the harmonics (by the ratio n, as n≥2) when generated according to Equation (1).

The harmonics generator800includes two or more normalization stages802(two shown:802aand802b), two or more multipliers804(two shown:804aand804b), two or more loudness correction stages806(two shown:806aand806b), two or more adders808(two shown:808aand808b), and an adder810. In general, each row of components in the harmonics generator800corresponds to one of the generated harmonics, so the number of rows (and the corresponding number of components) may be adjusted to implement the desired number of harmonics. The first processing row includes the normalization stage802a, the multiplier804a, the loudness correction stage806a, and the adder808a. The second processing row includes the normalization stage802b, the multiplier804b, the loudness correction stage806b, and the adder808b. Additional rows may be added to generate additional harmonics, with each new row connected to the previous row in a manner similar to what is shown in the figure.

The harmonics generator800receives an input signal820. The input signal820corresponds to the upsampled signal220(seeFIG.2) when the upsampler202is present, or to the transformed audio signal112when the upsampler202is not present. The input signal820is a complex transform domain signal. For example, the input signal820may correspond to a HCQMF band (e.g., hybrid sub-band 0, hybrid sub-band 2, hybrid sub-band 4, hybrid sub-band 6, etc.). The harmonics generator800generates the signal222(seeFIG.2).

Starting with the normalization stages802, the normalization stage802areceives the input signal820, performs normalization, and generates a signal822a. The normalization stage802breceives the input signal820, performs normalization, and generates a signal822b. Similarly to Equation (5), each of the normalization stages802may perform normalization using a first order smoothing filter to avoid distortion caused by sample-to-sample normalization. The normalization stages802may perform normalization in a manner described by Equation (8):
{circumflex over (x)}(m)=α·{circumflex over (x)}(m−1)+(1−α)·x(m)  (8)

In Equation (8), {circumflex over (x)}(m) is the current sample m of the normalized version of the input signal x, {circumflex over (x)}(m−1) is the previous sample of the normalized version of the input signal, α is a smoothing factor, andx(m) is given by Equation (9):

In Equation (9),x(m) corresponds to the ratio between the complex value of the current sample of the input signal and the magnitude (also referred to as the absolute value) of the current sample of the input signal. The smoothing factor α may be adjusted as desired to control the desired smoothing time, and is dependent on the dynamics of the input signal. A smaller α is applied during attack events (e.g., when there is rapidly increasing signal energy) than under stationary or decreasing energy conditions, in order to avoid signal clipping.

Alternatively, the harmonics generator may use a single normalization stage (e.g.,802a), with the output signal (e.g.,822a) provided as an input to each of the multipliers804.

Turning to the multipliers804, the multiplier804areceives the input signal820and the signal822a, multiplies these signals together, and generates a signal824a. The multiplier804breceives the signal822band the signal824a, multiplies these signals together, and generates a signal824b. The signal824acorresponds to the second harmonic, the signal824bcorresponds to the third harmonic, etc. Note that the output of a given multiplier is provided as an input to the multiplier in the subsequent processing row: The signal824ais provided to the multiplier804b, the signal824bis provided to the multiplier in the subsequent row (shown with a dotted line), etc.

Turning to the loudness correction stages806, the loudness correction stage806areceives the signal824a, performs loudness correction, and generates the signal826a. The loudness correction stage806breceives the signal824b, performs loudness correction, and generates the signal826b. In general, the loudness correction stages806apply dynamic expansion and attenuation of the normalized energy of the generated harmonics, in line with the equal loudness curves ofFIG.6, in order to maintain the loudness as compared to the fundamental. To adjust the loudness, a correction factor k is defined, where k is a function of the order of harmonic n, the smoothed magnitude of the fundamental {circumflex over (x)} (see Equation (8)) and the hybrid band index b. This correction factor k is applied according to Equation (10):
{tilde over (h)}n(m)=k(n,{circumflex over (x)},b)·hn(m)  (10)

In Equation (10), {tilde over (h)}n(m) is the loudness corrected harmonic and hn(m) is the normalized harmonic, for each harmonic respectively.

As discussed above, the bass enhancement processes may be performed on one or more hybrid bands (e.g., one or more of sub-bands 0, 2, 4, 6, 7, 9, etc.). Several harmonics, e.g. 2nd, 3rdand 4th, are generated in every band. If we let the center frequency approximate the fundamental frequency in each band, we may calculate the SPL-to-phon relationship using one parameter: the order or the harmonics n. As an example, the first hybrid band (e.g., sub-band 0) has a center frequency of 46.875 Hz (e.g., approximately 47 Hz) and the corresponding values from the ELC curves inFIG.6are listed in TABLE 1:

In TABLE 1, the value between parenthesis is the SPL difference as compared to the fundamental. A function representing the SPL difference of a harmonic and its fundamental may be calculated according to Equation (11):
Kb,n=Ab+βb,nX(11)

In Equation (11), Kb,nis a gain value in dB, Abis a minimum attenuation value, X is a smoothed input fundamental energy on a logarithmic scale, while βb,nis a harmonic order n dependent scaling parameter of the input energy. βb,nmay be calculated according to Equation (12):
βb,n=εbn+ηb(12)

The correction factor on a linear scale may be calculated according to Equation (13):

In Equations (12) and (13), Ab, εband ηbare all hybrid band based constants and may be estimated for an optimal fit to the ELC curves ofFIG.6. The parameters listed in TABLE 2 will result in adequate accuracy for the first six hybrid bands and the resulting loudness correction factors are visualized inFIG.9. For bands 6, 7 and 9, the generated harmonics are in the 700 to 2000 Hz frequency range, where the ELC curves are assumed to be flat. The loudness correction stages806may calculate the loudness correction factors using segmental linear approximation to save computational complexity.

FIGS.9A,9B,9C,9D,9E and9Fshow a set of graphs900a-900f. In each graph, the x-axis is the magnitude of the normalized harmonic signal into the loudness correction stage (e.g., the signal824ainput into the loudness correction stage806a, etc.) and the y-axis is the correction factor k. The graph900acorresponds to hybrid band 0, the graph900bcorresponds to hybrid band 2, the graph900ccorresponds to hybrid band 4, the graph900dcorresponds to hybrid band 6, the graph900ecorresponds to hybrid band 7, and the graph900fcorresponds to hybrid band 9. The lines for three harmonics (the 2nd, 3rdand 4th) are shown in each graph, but the lines are overlapping in the graphs900d,900eand900fas the lines converge with the increasing hybrid band number. In general, the lines show the loudness correction factors k for the first 6 hybrid bands when using the hybrid band based constants listed in TABLE 2.

Returning toFIG.8and the adders808, the adder808breceives the signal826b(and any signal received from the subsequent processing row, shown with a dotted line), performs addition, and generates a signal828b. The adder808breceives the signal826aand the signal828b, performs addition, and generates a signal828a. Note that one of the inputs to a given adder is provided by the adder in the subsequent processing row: The adder808breceives the output of the adder in the subsequent processing row (shown with a dotted line), the adder808areceives the output of the adder808b, etc.

The adder810receives the input signal820and the signal828a, performs addition, and generates the signal222(seeFIG.2).

Multiple Hybrid Bands Processing

Although the description for the bass enhancement system200(seeFIG.2) focused on processing a single hybrid band, similar processing may be performed on multiple hybrid bands. For example, the bass enhancement system120(seeFIG.1) may be performed on four hybrid bands (e.g., sub-bands 0, 2, 4 and 6), six hybrid bands (e.g., sub-bands 0, 2, 4, 6, 7 and 9), etc. Several harmonics (e.g., 2nd, 3rd, 4th, etc.) are generated in every band.

FIG.10is a block diagram of a bass enhancement system1000. The bass enhancement system1000may be used as the bass enhancement system120(seeFIG.1). The bass enhancement system1000is similar to the bass enhancement system200(seeFIG.2), with similar components having similar names and reference numerals, plus the addition of explicit multiple processing paths. Each processing path corresponds to processing a hybrid sub-band signal. As a specific example, four processing paths are shown (e.g., to process hybrid sub-bands 0, 2, 4 and 6). The number of processing paths may be increased or decreased as desired. For example, six processing paths may be used to process the hybrid sub-bands 0, 2, 4, 6, 7 and 9.

The bass enhancement system1000receives the transformed audio signal112(seeFIG.1). As discussed above, the transformed audio signal112is a hybrid complex transform domain signal with hybrid bands. Four of the hybrid bands of the transformed audio signal112are shown as the inputs to the bass enhancement system1000: sub-band 0 (labeled1002a), sub-band 2 (1002b), sub-band 4 (1002c) and sub-band 6 (1002d). Each sub-band corresponds to one of the processing paths. The bass enhancement system1000includes upsamplers1010(four shown:1010a,1010b,1010cand1010d), harmonics generators1012(four shown:1012a,1012b,1012cand1012d), an adder1014, a dynamics processor1016(optional), a converter1018(optional), a filter1022, a delay1024, and a mixer1026.

The upsampler1010areceives the signal1002a, performs upsampling, and generates an upsampled signal1030a. The upsampler1010breceives the signal1002b, performs upsampling, and generates an upsampled signal1030b. The upsampler1010creceives the signal1002c, performs upsampling, and generates an upsampled signal1030c. The upsampler1010dreceives the signal1002d, performs upsampling, and generates an upsampled signal1030d. The signals1030a,1030b,1030cand1030dare complex transform domain signals. The upsamplers1010are otherwise similar to that described above regarding the upsampler202(seeFIG.2).

The harmonics generator1012areceives the upsampled signal1030aand generates harmonics thereof to result in a signal1032a. The harmonics generator1012breceives the upsampled signal1030band generates harmonics thereof to result in a signal1032b. The harmonics generator1012creceives the upsampled signal1030cand generates harmonics thereof to result in a signal1032c. The harmonics generator1012dreceives the upsampled signal1030dand generates harmonics thereof to result in a signal1032d. The signals1032a,1032b,1032cand1032dare complex transform domain signals. The harmonics generators1012are otherwise similar to the harmonics generator204(seeFIG.2). For example, one or more of the harmonics generators1012may be implemented using the harmonics generator300(seeFIG.3), the harmonics generator400(seeFIG.4), the harmonics generator500(seeFIG.5), the harmonics generator800(seeFIG.8), etc.

The adder1014receives the signals1032a,1032b,1032cand1032d, performs addition, and generates a signal1034. The signal1034is a complex transform domain signal.

The dynamics processor1016receives the signal1034, performs dynamics processing, and generates a signal1036. The signal1036is a complex transform domain signal. The dynamics processor1016is otherwise similar to the dynamics processor206(seeFIG.2). The dynamics processor1016is optional. When the dynamics processor1016is omitted, the converter1018receives the signal1034instead of the signal1036.

The converter1018receives the signal1036(or the signal1034when the dynamics processor1016is omitted), drops the imaginary part from the signal1036, and generates a signal1040. The signal1040is a transform domain signal. The converter1018is otherwise similar to the converter208(seeFIG.2), including being optional.

The filter1022receives the signal1040(or the signal1036when the converter1018is omitted, or the signal1034when the dynamics processor1016and the converter1018are omitted), performs filtering, and generates a signal1042. The signal1042is a transform domain signal. The filter1022is otherwise similar to the filter212(seeFIG.2).

The delay1024receives the signal1042, implements a delay period, and generates a signal1044. The signal1044corresponds to a delayed version of the transformed audio signal112according to the delay period. The delay1024may be implemented using a memory, a shift register, etc. The delay period corresponds to the processing time of the other components in the signal processing chain; because some of these other components are optional, the delay period decreases when the optional components are omitted. The delay1024is otherwise similar to the delay214(seeFIG.2).

The mixer1026receives the signal1042and the signal1044, performs mixing, and generates the enhanced audio signal122(seeFIG.1). The mixer1026is otherwise similar to the mixer216(seeFIG.2).

FIG.11is a mobile device architecture1100for implementing the features and processes described herein, according to an embodiment. The architecture1100may be implemented in any electronic device, including but not limited to: a desktop computer, consumer audio/visual (AV) equipment, radio broadcast equipment, mobile devices (e.g., smartphone, tablet computer, laptop computer, wearable device), etc. In the example embodiment shown, the architecture1100is for a laptop computer and includes processor(s)1101, peripherals interface1102, audio subsystem1103, loudspeakers1104, microphone1105, sensors1106(e.g., accelerometers, gyros, barometer, magnetometer, camera), location processor1107(e.g., GNSS receiver), wireless communications subsystems1108(e.g., Wi-Fi, Bluetooth, cellular) and I/O subsystem(s)1109, which includes touch controller1110and other input controllers1111, touch surface1112and other input/control devices1113. Other architectures with more or fewer components can also be used to implement the disclosed embodiments.

Memory interface114is coupled to processors1101, peripherals interface1102and memory1115(e.g., flash, RAM, ROM). Memory1115stores computer program instructions and data, including but not limited to: operating system instructions1116, communication instructions1117, GUI instructions1118, sensor processing instructions1119, phone instructions1120, electronic messaging instructions1121, web browsing instructions1122, audio processing instructions1123, GNSS/navigation instructions1124and applications/data1125. Audio processing instructions1123include instructions for performing the audio processing described herein.

FIG.12is a flowchart of a method1200of audio processing. The method1200may be performed by a device (e.g., a laptop computer, a mobile telephone, etc.) with the components of the architecture1100ofFIG.11, to implement the functionality of the audio processing system100(seeFIG.1), the bass enhancement system200(seeFIG.2), the bass enhancement system1000(seeFIG.10), etc., for example by executing one or more computer programs. In general, the method1200performs audio signal processing in a complex-valued sub-band domain (e.g., the HCQMF domain).

At1202, a first transform domain signal is received. The first transform domain signal is a hybrid complex transform domain signal having a number of bands. At least one of the bands has a number of sub-bands. The first transform domain signal has a first plurality of harmonics. For example, the bass enhancement system200(seeFIG.2) may receive the transformed audio signal112. The first transform domain signal may have 77 hybrid bands numbered 0-76, where bands 0-15 are sub-bands that result from splitting one or several larger bands. The first transform domain signal may be a CQMF domain signal. The first transform domain signal may be a HCQMF signal generated by splitting (e.g., by using Nyquist filter banks) a subset of the channels of a CQMF domain signal into sub-bands to increase the frequency resolution for the lowest frequency range.

At1204, a second transform domain signal is generated based on the first transform domain signal. The second transform domain signal is generated by generating harmonics to of the first transform domain signal according to a non-linear process. The second transform domain signal has a second plurality of harmonics that differs from the first plurality of harmonics, and the second transform domain signal is a complex-valued signal having an imaginary part. The second transform domain signal is further generated by performing loudness expansion on the second plurality of harmonics. For example, the harmonics generator204(seeFIG.2), the harmonics generator300(seeFIG.3), the harmonics generator400(seeFIG.4), the harmonics generator500(seeFIG.5), the harmonics generator800(seeFIG.8), etc. may generate the second transform domain signal (e.g., the signal222) based on the first transform domain signal (e.g., the signal220, etc.).

At1206, a third transform domain signal is generated by filtering the second transform domain signal. The third transform domain signal has a number of bands, and at least one of the bands has a number of sub-bands. For example, the filter212(seeFIG.2) may filter the signal228(or the signal226) to generate the signal230. As another example, the filter1022(seeFIG.10) may filter the signal1040to generate the signal1042. The third transform domain signal may have 77 hybrid bands numbered 0-76, where bands 0-15 are sub-bands that result from splitting one or several larger bands. The third transform domain signal may be a HCQMF domain signal.

At1208, a fourth transform domain signal is generated by mixing the third transform domain signal with a delayed version of the first transform domain signal. A given sub-band of the third transform domain signal is mixed with a corresponding sub-band of the delayed version of the first transform domain signal. For example, the mixer216(seeFIG.2) may mix the signal230with the delayed signal232. As another example, the mixer1026(seeFIG.10) may mix the signal1042with the delayed signal1044. The input signals may have 77 hybrid bands numbered 0-76, where a given band of one input signal (e.g., band 0) is mixed with the corresponding band of the other input signal (e.g., band 0).

The method1200may include additional steps corresponding to the other functionalities of the bass enhancement system200, the bass enhancement system1000, etc. as described herein. For example, the fourth transform domain signal may be outputted by a loudspeaker, such as the loudspeakers1104(seeFIG.11). As another example, the transform domain signals may be upsampled (e.g., using the upsampler202, the upsamplers1010) prior to generating the harmonics at1204. As another example, dynamics processing may be applied to the transform domain signals, e.g. using the dynamics processor206or the dynamics processor1016. As another example, generating the harmonics may include performing multiplication, using a feedback delay loop, etc. As another example, the second transform domain signal may be a number of second transform domain signals, each of which corresponds to a hybrid band of the first transform domain signal. As another example, the imaginary part of the second transform domain signal may be dropped prior to generating the third transform domain signal.

Implementation Details

An embodiment may be implemented in hardware, executable modules stored on a computer readable medium, or a combination of both (e.g., programmable logic arrays). Unless otherwise specified, the steps executed by embodiments need not inherently be related to any particular computer or other apparatus, although they may be in certain embodiments. In particular, various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatus (e.g., integrated circuits) to perform the required method steps. Thus, embodiments may be implemented in one or more computer programs executing on one or more programmable computer systems each comprising at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices, in known fashion.

Each such computer program is preferably stored on or downloaded to a storage media or device (e.g., solid state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer system to perform the procedures described herein. The inventive system may also be considered to be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer system to operate in a specific and predefined manner to perform the functions described herein. (Software per se and intangible or transitory signals are excluded to the extent that they are unpatentable subject matter.)

Aspects of the systems described herein may be implemented in an appropriate computer-based sound processing network environment for processing digital or digitized audio files. Portions of the adaptive audio system may include one or more networks that comprise any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route the data transmitted among the computers. Such a network may be built on various different network protocols, and may be the Internet, a Wide Area Network (WAN), a Local Area Network (LAN), or any combination thereof.

One or more of the components, blocks, processes or other functional components may be implemented through a computer program that controls execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and/or as data and/or instructions embodied in various machine-readable or computer-readable media, in terms of their behavioral, register transfer, logic component, and/or other characteristics. Computer-readable media in which such formatted data and/or instructions may be embodied include, but are not limited to, physical (non-transitory), non-volatile storage media in various forms, such as optical, magnetic or semiconductor storage media.