System and method for leveling loudness variation in an audio signal

Systems and methods for leveling loudness variation in an audio signal are described. Embodiments use both a perceptual leveling algorithm and a standards-based loudness measure together to minimize audio process artifacts and ensure that the measured loudness of the processed audio is close to a required measure, according to a particular standard measurement of loudness. These systems and methods can be used either offline or in real-time.

FIELD

The present disclosure relates generally to signal processing, and more particularly, to systems and methods for leveling loudness variation in an audio signal.

BACKGROUND

Loudness leveling, or the automatic reduction of the range between loud and soft sections of an audio signal, is desirable in many situations, particularly with respect to broadcast television networks. Poor authoring practices lead to wide variation in loudness levels within a television program, advertisement or both, and programs or advertisements with significantly different loudness levels are frequently concatenated. Television viewers often find themselves adjusting the volume control on their television or sound playback system to compensate for these variations; however, the reaction time of such viewers is often not fast enough to avoid annoyance. In addition, some programs (such as movies) have a very high dynamic range. These ranges are often too wide for home listening, where the volume position required to hear dialogue may result in very loud levels for sound effects and music, which can disturb others in the household.

Existing methods for loudness leveling include compressor and limiter methods. These methods integrate audio signal level or power over time. The shorter the integration time, the faster the algorithm can measure and adjust short term fluctuations in loudness. The longer the integration time, the more the average loudness is affected, but short term fluctuations persist. These methods typically operate by gain adjusting the whole audio signal, i.e., all frequencies at once. This can result in audible artifacts, such as “breathing” or “pumping”. More recently, psychoacoustic methods for loudness measurement and adjustment have been developed, such as those described in U.S. Patent Application Publication No. 2007/0092089 A1 to Seefeldt et al., published Apr. 26, 2007, herein incorporated by reference in its entirety. These algorithms use spectral analysis and models of human hearing to adjust the audio in ways that vary across frequency and vary with measured loudness level. These methods work well in adjusting short term loudness on millisecond to second timescales, with very few audible artifacts.

Separate from loudness levels, methods also exist for objectively measuring the perceived loudness of audio signals. Examples include A-, B-, and C-weighted power measures, as well as psychoacoustic models of loudness, such as those described in “Acoustics—Method for calculating loudness level,” ISO 532 (1997), and in U.S. Patent Application Publication No. 2007/0092089 A1. Weighted power measures operate by taking an input audio signal, applying a known filter that emphasizes more perceptibly sensitive frequencies, while deemphasizing less perceptibly sensitive frequencies, then averaging the power of the filtered signal over a predetermined length of time. The recently developed ITU-R BS.1770-2 objective loudness measurement standard uses a weighting filter similar to B-weighting, and removes parts of the audio signal that are quiet or silent from the final average power calculation.

Psychoacoustic methods are typically more complex and aim to better model the workings of the human ear. Such psychoacoustic methods divide the signal into frequency bands that mimic the frequency response and sensitivity of the ear, then manipulate and integrate such bands while taking into account psychoacoustic phenomena, such as frequency and temporal masking, as well as the non-linear perception of loudness with varying signal intensity. The aim of all such methods is to derive a numerical measurement that closely matches the subjective impression of the audio signal. These methods are typically useful for measuring the longer term perceived loudness of an audio signal, e.g., where the audio signal length is 30 seconds or more, and typically minutes or hours. Over many years, the development and acceptable of these objective measurement algorithms has been accompanied by subjective testing, i.e., comparing the objective algorithm's measurements to human listening.

Recently, there has been a growing need for broadcast television audio signals to maintain a consistent loudness, particularly with respect to commercials. This need has been driven by government regulation, such as by Federal Communications Commission Publication No. FCC 11-84, “Notice of Proposed Rulemaking: Implementation of the Commercial Advertisement Loudness Mitigation (CALM) Act”. Since broadcasters have a mixture of well-authored content with known average loudness and dynamics and unknown content with unknown average loudness and possibly wide dynamics, they frequency use loudness leveling equipment in-line with the real-time audio signal that eventually makes its way to the television viewer. However, loudness levelers are typically optimized for short-term behavior to minimize artifacts when level-adjusting the audio signal, and as a result, the leveled audio signal is not necessarily consistent when measured using longer term measures. That is, the measured loudness of the sections of the leveled audio, e.g., 30 seconds or more, is not consistent.

SUMMARY

Thus, there is a need in the art for a real-time leveling method that can perform short-term artifact-free loudness leveling, while ensuring that the longer term loudness of the leveled audio matches a known measurement standard. Embodiments of the invention meet this need and others by providing a system and method for leveling loudness variation in an audio signal.

According to one embodiment, a method for leveling loudness variation in an audio signal is described. The method comprises receiving the audio signal and a desired loudness of the audio signal, removing artifacts from the audio signal, measuring an actual loudness of the audio signal, calculating a gain value using a difference between the desired loudness and the actual loudness of the audio signal, and modifying the audio signal using the gain value.

According to another embodiment, a system for leveling loudness variation in an audio signal is described. The system comprises a short term loudness leveling module configured to receive the audio signal and a desired loudness of the audio signal and to remove artifacts from the audio signal, a long term loudness leveling module configured to measure an actual loudness of the audio signal, a loudness to gain module configured to calculate a gain value using a difference between the desired loudness and the actual loudness of the audio signal, and an audio modification module configured to modify the audio signal using the gain value. In one embodiment, at least one of the short term loudness leveling module, the long term loudness leveling module, the loudness to gain module and the audio modification module are comprised in a processor.

According to another embodiment, a computer readable medium having computer executable instructions embedded thereon is described for performing the steps of receiving the audio signal and a desired loudness of the audio signal, removing artifacts from the audio signal, measuring an actual loudness of the audio signal, calculating a gain value using a difference between the desired loudness and the actual loudness of the audio signal, and modifying the audio signal using the gain value.

DETAILED DESCRIPTION

A system and method for leveling loudness variation in an audio signal is described. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the exemplary embodiments. It is apparent to one skilled in the art, however, that embodiments can be practiced without these specific details or with an equivalent arrangement. In some instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the embodiments.

Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views,FIG. 1is a schematic functional block diagram illustrating a system for leveling loudness variation in real-time according to one embodiment. An audio signal110and a desired loudness140of the audio signal are input into short term loudness leveling module120. Short term loudness leveling module120adjusts audio signal110to minimize audible leveling artifacts, such as “pumping” and “breathing”. Short term loudness leveling module120may accomplish this by the methods described in U.S. Patent Application Publication No. 2007/0092089 A1, herein incorporated by reference in its entirety.

In one embodiment, short term loudness leveling module120operates on time intervals of 5 milliseconds of digital pulse code modulation (“PCM”) audio, i.e., a 5 millisecond period. In another embodiment, short term loudness leveling module120operates continuously, as in an analog system. In still another embodiment, short term loudness leveling module120operates on every PCM sample in a digital system.

Short term loudness leveling module120also takes desired loudness140of the audio signal as input. In a digital system where the audio signal is a PCM signal, desired loudness140is expressed in units of decibels (dB) relative to the full-scale PCM. For example, desired loudness140may be −24 dB FS (full-scale).

The audio signal processed by short term loudness leveling module120is then passed to long term loudness leveling module130. Long term loudness leveling module130measures the actual loudness of the audio signal according to a chosen objective loudness measurement standard, such as, for example, ITU-R BS.1770-3 (August 2012). Long term loudness leveling module130outputs a single loudness number for every period, i.e., every time the actual loudness of the audio signal is measured. The loudness number represents the actual loudness of the audio signal. In a digital system where the audio signal is a PCM signal, the loudness number is expressed in units of decibels (dB) relative to full-scale PCM.

Long term loudness leveling module130is set to a nominal integration time, such as, for example, 30 seconds, in one embodiment. This integration time is the time interval of PCM samples which long term loudness leveling module130uses to calculate a single loudness number. This function may operate more frequently than the integration time interval; however, each time is operates, it outputs a number representing a measurement over the last “integration time interval” PCM samples it has received.

In one embodiment, long term loudness leveling module130operates at the same rate as short term loudness leveling module120, such as, for example, every 5 milliseconds. In other embodiments, long term loudness leveling module130operates at slower or faster rates than short term loudness leveling module120. For example, long term loudness leveling module130may operate every 20 milliseconds, or once for every four times the short term loudness leveling module120performs its function.

Long term loudness leveling module130may include, in one embodiment, a mode of operation in which the integration of PCM samples (to calculate a single loudness number) only occurs when a block of PCM samples, typically representing a time interval of 0.5 seconds, are classified as “dialogue” by the algorithm described in ITU-R BS.1770-1 (2006-2007), which is herein incorporated by reference in its entirety. This approach gates the loudness measure on the dialogue portions of the audio signal only. Alternatively, for geographical regions implementing regulatory requirements for loudness measurement and control (e.g., the Netherlands, Germany, Austria, France), long term loudness leveling module130can be operated in a mode that solely complies with ITU-R BS.1770-2 (March 2011) or ITU-R BS.1770-3 (August 2012), which specify a level-based gating technique, and are herein incorporated by references in their entireties.

Loudness to gain module150takes the loudness number and compares it with the desired loudness140. Specifically, loudness to gain module150calculates the difference between desired loudness140and the loudness number, and outputs a gain value. This gain, which is applied to the leveled audio by audio modification module170to create the modified audio signal190, brings the long term measure of the leveled audio signal closer to the desired loudness140.

The calculation of gain value includes some rate limiting to prevent the gain value from changing too quickly and causing audible artifacts. This is particularly important when the process starts and the long term loudness leveling module130has not yet received sufficient audio samples to output a relevant loudness number. In one embodiment, loudness to gain module150operates at the same rate as long term loudness leveling module130, such as, for example, 20 milliseconds, and the actual gain applied to the leveled audio signal is linearly interpolated between each gain value that is calculated every 20 milliseconds.

The gain value (in decibels) may be calculated according to the following equations.

FIG. 2is a schematic functional block diagram illustrating a system for leveling loudness variation offline according to one embodiment. In offline leveling, the whole audio signal is captured and stored, then processed through the method described with respect toFIG. 1. Audio signal210is processed by short term loudness leveling module220, and stored in memory225. Long term loudness leveling module230calculates a single loudness number over the whole audio signal or program. Loudness to gain module250calculates the gain value as the difference between desired loudness240and the loudness number. This gain value is applied at audio modification module270to produce modified audio signal290.

Input Signal Adjustment

The audio signals input into the systems described above with respect toFIGS. 1 and 2may have an average level that is well away from the desired loudness. In this situation, the system is almost always consistently either boosting or attenuating the audio signal. Audio signals such as these may have been authored well, i.e., they may have a fairly consistent average level, but their overall level may be shifted, e.g., due to a mismatch in reference levels when connecting and configuring various broadcast audio equipment. For example, the desired loudness may be −24 dB FS (full-scale), and the average level of the input audio signal consistently around −36 dB FS (full-scale). In such an example, the short term loudness leveling module is always boosting the audio signal by approximately 12 dB.

In one embodiment, the audio signal is not adjusted by the system if it is already close to the desired loudness. In other words, the system may implement a null-band. Thus, only signals outside of the null-band, i.e., that are significantly louder or quieter than the desired loudness, are modified to bring the signal closer in level to the desired loudness. An example null-band is a range between 4 dB quieter than the desired loudness to 4 dB louder than the desired loudness.

If an audio program is authored well but simply shifted, a slow gain adjustment may be applied according to one embodiment to move the level of the audio signal so that its average level is approximately the same as the desired loudness, before the signal is passed to the short term loudness leveling module. Thus, the short term loudness leveling module will not affect the audio signal as much, and only attenuates short term excessively loud signals, and only boosts short term very quiet signals.

FIGS. 3 and 4are schematic functional block diagrams illustrating systems for leveling loudness variation with input reference level adjustment in real-time and offline, respectively, according to an embodiment. Audio signal310is passed to a first long term loudness leveling module313, which outputs a loudness number. In a digital system where the audio is a PCM signal, this number is typically expressed in units of dB relative to full-scale PCM. In one embodiment, the objective loudness measurement method used by long term loudness leveling module313is the same as is used by long term loudness leveling module130ofFIG. 1.

Long term loudness leveling module313is set to a nominal integration time, e.g., 10 seconds. This integration time is the time interval of PCM samples which the module uses to calculate a single loudness number. Long term loudness leveling module313may operate more frequently than the integration time interval; however, each time it operates, it outputs a number representing a measurement over the last “integration time interval” PCM samples it has received. Long term loudness leveling module313can be operated at the same rate as short term loudness leveling module120, e.g., every 5 milliseconds, or at a slower or faster rate. In one example, long term loudness leveling module313is operated every 20 milliseconds, or once for every 4 times short term loudness leveling module120is operated.

A first loudness to gain module314takes the loudness number received from long term loudness leveling module313and compares it to the desired loudness140. Loudness to gain module314calculates the difference between the desired loudness and the loudness number, and outputs a first gain value. This gain, which is applied to audio signal310by a first audio modification module317, bring the long term measure of audio signal310closer to desired loudness140.

The calculation of the first gain value includes rate limiting to prevent the gain value from changing too quickly and causing audible artifacts. This is particularly important when the system begins operation, as long term loudness leveling module313has not yet received enough audio samples to output a relevant loudness number. In one embodiment, loudness to gain module315operates at the same rate as long term loudness leveling module313, e.g., every 20 milliseconds, and the actual gain applied to the leveled audio signal is linearly interpolated between each of the first gain values calculated every 20 milliseconds. The first gain value can be calculated according to Equations 1-4.

In real-time leveling, after audio signal310is gain adjusted by the first gain value, it is passed to short term loudness leveling module120, and processed according to the system described with respect toFIG. 1for real-time leveling. That is, the adjusted audio signal is passed to short term loudness leveling module120; long term loudness leveling module130; loudness to gain module150, and audio modification module170, to produce modified audio signal391.

In an embodiment in which offline leveling is used, long term loudness leveling module313can measure one loudness number for the whole audio signal. The whole audio signal can then be gain adjusted by the difference between the desired loudness140and the measurement from long term loudness leveling module313. Once adjusted, audio signal310is then processed according to the system described with respect toFIG. 2. That is, the adjusted audio signal is passed to short term loudness leveling module220, memory225, long term loudness leveling module230, loudness to gain module250, and audio modification module270, to produce modified audio signal392.

Input Reference Level

In one embodiment, the short term loudness leveling modules described with respect toFIGS. 1-4has an input called an input reference level. The input reference level indicates the nominal or average level of the audio signal that is passed into the short term loudness leveling module. The short term loudness leveling module performs two operations: adjusting or leveling the audio signal around the input reference level, and applying a fixed gain established by the difference between the desired loudness and the input reference level.

In one example, 85 corresponds to a nominal or average level of −31 dB full-scale (FS). The input reference level (IRL) is in dB's and may be calculated by Equation 5 below.
IRL=54−AverageLevel(dBFS)  [5]
For example, a nominal value of −20 dB full-scale (FS) corresponds to an input reference level of 74.

By way of example, assuming digital PCM signals are used, if the desired loudness is −24 dB FS and the nominal level of the input audio signal is indicated by the input reference level as −10 dB FS, the short term loudness leveling module: (a) levels the audio either side of −10 dB FS, i.e., parts of the audio signal that are significantly quieter than −10 dB FS are boosted and parts that are significantly louder than −10 dB are attenuated, and (b) applies a bulk gain of −24−−10=−14 dB, i.e., 14 dB of attenuation. In other words, the null-band of the short term loudness leveling module is moved to be centered at the input reference level, or −10 dB FS in this example.

FIGS. 5 and 6are a schematic functional block diagrams illustrating systems for leveling loudness variation with input reference level adjustment in real-time and offline, respectively, according to an embodiment. Audio signal410is passed to a first long term loudness leveling module413. In a digital system where the audio is a PCM signal, this number is typically expressed in units of dB relative to full-scale PCM. In one embodiment, the objective loudness measurement method used by long term loudness leveling module413is the same as is used by long term loudness leveling module130ofFIG. 1.

Long term loudness leveling module413is set to a nominal integration time, e.g., 10 seconds. This integration time is the time interval of PCM samples which the module uses to calculate a single loudness number. Long term loudness leveling module413may operate more frequently than the integration time interval; however, each time it operates, it outputs a number representing a measurement over the last “integration time interval” PCM samples it has received. Long term loudness leveling module413can be operated at the same rate as short term loudness leveling module120, e.g., every 5 milliseconds, or at a slower or faster rate. In one example, long term loudness leveling module413is operated every 20 milliseconds, or once for every 4 times short term loudness leveling module120is operated.

A loudness to reference level module415performs the calculations of Equations 1-4, then converts the gain value to an input reference level using the desired loudness440according to the following equation:
IRL=54−DesiredLoudness+GV[6]

In real-time leveling, after audio signal410is adjusted by the input reference level, it is passed to short term loudness leveling module120, and processed according to the system described with respect toFIG. 1for real-time leveling. That is, the adjusted audio signal is passed to short term loudness leveling module120; long term loudness leveling module130; loudness to gain module150, and audio modification module170, to produce modified audio signal491.

In offline leveling, after audio signal410is adjusted by the input reference level, it is processed according to the system described with respect toFIG. 2. That is, the adjusted audio signal is passed to short term loudness leveling module220, memory225, long term loudness leveling module230, loudness to gain module250, and audio modification module270, to produce modified audio signal492.

According to some embodiments, computer system500comprises processor550(e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both), main memory560(e.g., read only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.) and/or static memory570(e.g., flash memory, static random access memory (SRAM), etc.), which communicate with each other via bus595.

According to some embodiments, computer system500may further comprise video display unit510(e.g., a liquid crystal display (LCD), a light-emitting diode display (LED), an electroluminescent display (ELD), plasma display panels (PDP), an organic light-emitting diode display (OLED), a surface-conduction electron-emitted display (SED), a nanocrystal display, a 3D display, or a cathode ray tube (CRT)). According to some embodiments, computer system500also may comprise alphanumeric input device515(e.g., a keyboard), cursor control device520(e.g., a controller or mouse), disk drive unit530, signal generation device540(e.g., a speaker), and/or network interface device580.

Disk drive unit530includes computer-readable medium534on which is stored one or more sets of instructions (e.g., software536) embodying any one or more of the methodologies or functions described herein. Software536may also reside, completely or at least partially, within main memory560and/or within processor550during execution thereof by computer system500, main memory560and processor550. Processor550and main memory560can also constitute computer-readable media having instructions554and564, respectively. Software536may further be transmitted or received over network590via network interface device580.

It should be understood that processes and techniques described herein are not inherently related to any particular apparatus and may be implemented by any suitable combination of components. Further, various types of general purpose devices may be used in accordance with the teachings described herein. It may also prove advantageous to construct a specialized apparatus to perform the methods described herein. Those skilled in the art will appreciate that many different combinations of hardware, software, and firmware will be suitable for practicing the disclosed embodiments.

Embodiments have been described in relation to particular examples, which are intended in all respects to be illustrative rather than restrictive. Further, while embodiments have been described in connection with a number of examples and implementations, it is understood that various modifications and equivalent arrangements can be made to the examples while remaining within the scope of the inventive embodiments.