Comfort noise generation based on noise estimation

Features are disclosed for generating comfort noise that matches a frequency spectrum of original background noise. For example, a spectral shape of an estimated noise component can be determined. A frame of white noise can be modified based at least in part on the spectral shape of the noise component. The modified frame of white noise can be converted to a time-domain noise signal. The level of the time-domain noise signal can be adjusted to match an original level of the noise after noise reduction. Residual echo suppression can sometimes cause background noise to be eliminated, causing silence. The adjusted time-domain noise signal can be added after residual echo suppression to maintain continuity of background noise levels.

BACKGROUND

Many communication devices configured to obtain audio data of user utterances include both a loudspeaker and a microphone. The loudspeaker is used to play audio signals, such as speech from a remote source during a telephone call, audio content presented from local storage or streamed from a network etc. The microphone is used to capture audio signals from a local source, such as a user speaking voice commands or other utterances. An acoustic echo occurs when the remote signal emitted by the loudspeaker is captured by the microphone, after undergoing reflections in the local environment.

An acoustic echo canceller (“AEC”) may be used to remove acoustic echo from an audio signal captured by a microphone in order to facilitate improved communication. For example, the AEC may filter the microphone signal by determining an estimate of the acoustic echo (e.g., the remote audio signal emitted from the loudspeaker and reflected in the local environment). The AEC can then subtract the estimate from the microphone signal to produce an approximation of the true local signal (e.g., the user's utterance). The estimate can be obtained by applying a transformation to a reference signal that corresponds to the remote signal emitted from the loudspeaker. In addition, the transformation can be implemented using an adaptive algorithm. For example, adaptive transformation relies on a feedback loop, which continuously adjusts a set of coefficients that are used to calculate the estimated echo from the far-end signal. Different environments produce different acoustic echoes from the same loudspeaker signal, and any change in the local environment may change the way that echoes are produced. By using a feedback loop to continuously adjust the coefficients, an AEC to can adapt its echo estimates to the local environment in which it operates.

Many communication devices also include a noise reduction (“NR”) module. In addition to user utterances and acoustic echo, background noise is typically present in any environment. The NR module can use a noise reduction algorithm to reduce the level of background noise present in an audio signal. Typically, the NR module reduces but does not entirely eliminate the level of noise in the audio signal.

In addition, communication devices may also use a residual echo suppressor (“RES”). Various factors, including nonlinearity and noise, can cause an echo to not be completely eliminated by an acoustic echo canceller. A residual echo suppressor may be used to further reduce the level of echo that remains after processing by an acoustic echo canceller. For example, residual echo suppressors may use non-linear processing to further reduce the echo level. In addition to echo, however, processing by a residual echo suppressor often eliminates noise as well. For example, a residual echo suppressor can receive an audio signal that already has reduced levels of noise after processing from the NR module and further process the signal so that the level of noise is wiped out completely.

This processing by the residual echo suppressor can have the undesirable effect of creating silence in the audio output signal. For example, when a user is speaking an utterance, the residual echo suppressor further reduces residual echo, but a level of background noise remains present in the output signal. However, when a user stops speaking an utterance, the residual echo suppressor can eliminate any residual echo as well as the background noise that was present. The abrupt transition in an audio output signal that includes some level of background noise and one that does not (e.g., silence) can cause a listener to mistakenly believe that the communication link is dead. In addition, the frequent changes between listening to a signal that includes some level of background noise and one that does not can cause distraction to a listener.

DETAILED DESCRIPTION

Introduction

Communication devices, such as telephones and computing devices that present audio output and accept spoken input, may receive an acoustic echo of audio output as well as background noise in combination with spoken input. To a consumer of the spoken input, such as another party to a telephone call or an automatic speech recognition system, the noise and acoustic echo can interfere with spoken input and make it difficult to understand. Devices often include acoustic echo cancellers (“AECs”) that attempt to remove acoustic echo from audio input, leaving only the spoken input (and environmental noise). Communication devices also often include a noise reduction (“NR”) module to reduce the level of background noise present in an audio signal. Communication devices may also use a residual echo suppressor (“RES”) to further reduce the level of echo that remains after processing by an acoustic echo canceller. In addition to echo, however, the processing by a residual echo suppressor often eliminates noise as well. This processing by the residual echo suppressor can have the undesirable effect of creating abrupt changes to silence in the audio output signal, which can cause a listener to mistakenly believe that the communication link is dead.

Aspects of the present disclosure relate to introducing comfort noise back into an audio signal processed by a residual echo suppressor. The comfort noise can match a frequency spectrum and amplitude level of the original background noise as reduced by the noise reduction module. By adding the comfort noise to the output of the residual echo suppressor, a listener can perceive a substantially constant level of background noise whether or not a user is speaking an utterance. By maintaining a substantially constant level of background noise, a listener will be informed that the communication link remains active.

In some aspects of the present disclosure, a communication device includes a microphone configured to detect sound as an audio input signal. The communication device can further include a memory configured to store a plurality of frames of white noise. In some embodiments, each frame of white noise can correspond to an index. One or more processors may be in communication with the microphone and the memory. The one or more processors may be configured to perform acoustic echo cancellation on the audio input signal to generate an echo-reduced signal. The one or more processors also may be configured to perform noise reduction to reduce a noise component from the echo-reduced signal and generate a noise-reduced echo-reduced signal. In addition, the one or more processors may be configured to perform residual echo suppression on the noise-reduced echo-reduced signal to further reduce echo of the noise-reduced echo-reduced signal and generate a residual-echo-reduced signal. The one or more processors also may be configured to determine a spectral shape of the noise component. For example, determining the spectral shape may include determining a spectral gain for a plurality of frequency bins. In addition, the one or more processors may be configured to modify a frame of noise based at least in part on the spectral shape of the noise component. In some embodiments, the frame of noise is received from the memory, the frame of noise corresponding to a generated random index. In addition, the one or more processors may be configured to modify the frame of white noise based at least in part on the spectral shape of the noise component. The one or more processors also may be configured to generate a modified noise signal using the spectral shape. For example, the modified noise signal may be created by computing a spectrum of the frame of noise, applying the spectral shape, and converting the modified noise signal back into the time domain to obtain a modified noise signal. In addition, the one or more processors may be configured to combine the modified noise signal with the residual-echo-reduced signal to generate an output signal.

In some embodiments, the one or more processors may be configured to estimate a level of noise in the noise-reduced echo-reduced signal. In addition, the one or more processors may be configured to adjust a level of the time-domain noise signal to approximate the level of noise in the noise-reduced echo-reduced signal.

In some embodiments, the one or more processors may be configured to determine a power spectrum density of the noise component at a plurality of frequency bands. For example, in some embodiments, the plurality of frequency bands comprises approximately thirty frequency bands spaced between zero and eight kilohertz. The one or more processors also may be configured to determine which of the plurality of frequency bands corresponds to a maximum of the power spectrum density. In addition, the one or more processors may be configured to normalize each of the plurality of frequency bands of the power spectrum density relative to the maximum to generate a normalized noise power spectrum density. The one or more processors also may be configured to calculate a square root of the normalized noise power spectrum density to determine the spectral shape of the noise component.

Although aspects of the embodiments described in the disclosure will focus, for the purpose of illustration, with respect to a local device using comfort noise generation, one skilled in the art will appreciate that the techniques disclosed herein may be applied to any number of processes or applications performing comfort noise generation. Various aspects of the disclosure will now be described with regard to certain examples and embodiments, which are intended to illustrate but not limit the disclosure.

Communication System Environment

With reference to an illustrative embodiment,FIG. 1shows example interactions and data flows in a communication environment100between communication devices102and102′, communication link104, and a users106and106′. In particular, a communication device102generally can comprise a microphone110and a speaker112. Similarly, communication device102′ comprises a microphone110′ and a speaker112′. Both communication devices102and102′ can communicate with communication link104.

The microphone110of communication device102generally detects sound from at least three sources when generating an audio input signal. For example, when user106speaks an utterance, the user's voice v is detected by the microphone110. In addition, background noise n present in the user's environment is detected by the microphone110. In addition, speaker112produces sound, creating an acoustic echo e that is detected by the microphone110. Thus, an audio input signal detected by the microphone110includes components attributable to the user's voice v, the background noise n, and the acoustic echo e.

After processing the audio input signal, the communication device102generally sends an output signal through the communication link104to the receiving communication device102′. The communication link104may be a telephone network, a data network, or a combination of the two. In addition, the communication link104may be a wired network, a wireless network, or a combination of the two. For example, the communication network may be a landline telephone network, a cellular network, or combinations of the same. The communication link104may be a personal area network, a local area network (LAN), a wide area network (WAN), or combinations of the same. Protocols and components for communicating via any of the other aforementioned types of communication networks, including TCP/IP protocols, can be used in the communication link104.

Upon receipt of the audio output signal, the receiving communication device102′ generally presents the received audio signal through speaker112′. As the output of the speaker reverberates around the environment of the receiving user106′, and acoustic echo e′ may be generated. In addition, the user106′ may speak an utterance, producing sound v′. Also, the environment of the user106′ has background noise n′. Each of the sounds e′, v′, and n′ may be detected by the microphone110′.

For example, when user106speaks an utterance, the user's voice v is detected by the microphone110. In addition, background noise n present in the user's environment is detected by the microphone110. In addition, speaker112produces sound, creating an acoustic echo e that is detected by the microphone110. Thus, an audio input signal detected by the microphone110includes components attributable to the user's voice v, the background noise n, and the acoustic echo e. After processing the audio input signal, the communication device102′ generally sends an output signal through the communication link104to communication device102.

To the user106′ at the receiving communication device102′, the noise n and acoustic echo e at the sending communication device102can interfere with spoken input v and make it difficult to understand. The communication device102may include an acoustic echo canceller that attempts to remove acoustic echo from audio input. The communication device102also may include a noise reduction module to reduce the level of noise n present in an audio signal. In addition, communication device102may also include a residual echo suppressor to further reduce the level of echo that remains after processing by an acoustic echo canceller. In addition to echo, however, the processing by a residual echo suppressor often eliminates noise as well. This processing by the residual echo suppressor can have the undesirable effect of creating abrupt changes to silence in the audio output signal sent from the communication device102to the communication device102′, which can cause the user106′ to mistakenly believe that the communication link4is disconnected, or dead.

FIG. 2illustrates a communication device102according to an embodiment that comprises an acoustic echo cancellation module120, a noise reduction module122, a residual echo suppression module124, a noise generator module126, and a summation module128. The communication system102can correspond to a wide variety of electronic devices or some combination thereof. In some embodiments, the communication system102may be a computing device that includes one or more processors and a memory130which may contain software applications executed by the processors. For example, each of the acoustic echo cancellation module120, noise reduction module122, residual echo suppression module124, noise generator module126, and summation module128may be implemented by one or more processors running software applications executed by the processors.

The communication system102may include a microphone110or other audio input component for accepting speech input. The audio input signal i detected by the microphone110includes components attributable to user106's voice v, the background noise n, and the acoustic echo e. For example, the acoustic echo e may be generated when the speaker112reproduces audio signal x, which may be received from another communication device.

The communication system102may include an acoustic echo cancellation module120to cancel acoustic echoes in the audio signal obtained from the microphone110. For example, the acoustic echo cancellation module120receives the audio input signal i. The acoustic echo cancellation module120performs acoustic echo cancellation on the audio input signal i to generate an echo-reduced signal er.

The communication device102may include a noise reduction module122to reduce the level of noise n present in an audio signal. For example, the noise reduction module122receives the echo-reduced signal er. The noise reduction module122performs noise reduction to reduce a noise component nc from the echo-reduced signal er and generate a noise-reduced echo-reduced signal nr. Although the noise reduction module122reduces the level of noise present in the noise-reduced echo-reduced signal nr compared to full level of background noise n present in the echo-reduced signal er, the noise-reduced echo-reduced signal nr generally still contains an appreciable level of noise. For example, if the noise-reduced echo-reduced signal nr was played by a speaker, a listener generally would be able to perceive an amount of noise.

As part of the process of performing noise reduction, noise reduction module122generally estimates the noise component nc that is being reduced. The noise component nc approximately shares the same frequency characteristics as the background noise n.

In addition, communication device102may include a residual echo suppression module124to further reduce the level of echo that remains after processing by an acoustic echo canceller. For example, the residual echo suppression module124receives the noise-reduced echo-reduced signal nr from the noise reduction module124. The residual echo suppression module124performs residual echo suppression on the noise-reduced echo-reduced signal to: (1) further reduce echo of the noise-reduced echo-reduced signal nr and (2) generate a residual-echo-reduced signal rer. In addition to further reducing echo, however, the residual echo suppression module124often eliminates noise as well. For example, the residual echo suppression module124sometimes eliminates the level of noise that is present in the noise-reduced echo-reduced signal nr. As a result, without added noise from the noise generator module126, a listener may mistakenly believe that the communication link is disconnected, or dead.

To address this problem, the communication device102includes a noise generator module126to generate comfort noise that corresponds to a frequency spectrum of original background noise n, as described in greater detail below. However, it may not be desirable to for the noise generator module126to generate comfort noise for each frame. For example, when a signal level of the residual-echo-reduced signal rer generated by the residual echo suppression module drops below a threshold level, without comfort noise introduced into an output signal, a listener may hear a discontinuity in the output signal or mistakenly believe the communication link is dead. In particular, it may be desirable to generate comfort noise when it is determined that a level of near-end user speech in the audio input signal is below a threshold. For example, it would be desirable to generate comfort noise when there is no near-end speech at all. On the other hand, when a signal level of the residual-echo-reduced signal rer generated by the residual echo suppression module remains above a threshold level, a listener may be able to listen to the output signal without perceiving any discontinuities in the background noise level. For example, if it is determined that a level of near-end speech in the audio input signal is not below a threshold, then it may not be desirable to generate comfort noise.

In some embodiments, the residual echo suppression module may provide signal-level information sl of the residual-echo-reduced signal rer to the noise generator module126. For example, the signal-level information sl may comprise a yes or no instruction to the noise generator module126regarding whether to generate comfort noise (e.g., with respect to a particular frame). The yes or no instruction whether to generate comfort noise may be based on a threshold level of the residual-echo-reduced signal rer. For example, if the signal level of the residual-echo-reduced signal rer drops below a signal level of the noise-reduced echo-reduced signal nr, the residual echo suppression module may provide a yes instruction to the noise generator module126to generate a frame of noise. Alternatively, the signal-level information sl may comprise information based on the signal level of the residual-echo-reduced signal rer, and the noise generator module126may use the signal-level information sl to decide whether to generate a frame of noise. In addition, the noise generator module126may receive the signal-level information sl for each frame of sound, and the noise generator module may be able to rapidly react to moments in which generating comfort noise is desirable and moments in which it is not.

In various embodiments, the noise generator module126may perform the processes described with respect toFIGS. 3 and 4. Generally, the noise generator module126receives the noise component nc estimated by the noise reduction module122. The noise generator module126generates comfort noise that shares substantially the same frequency characteristics as the noise present in the noise-reduced echo-reduced signal nr. For example, the noise generator module126can determine a spectral shape of the noise component nc. In addition, the noise generator module126can modify a frame of white noise based on the spectral shape of the noise component nc. Furthermore, the noise generator module126can generate a time-domain noise signal based on the modified frame of white noise. In addition, the noise generator module126can adjust a level of the generated comfort noise to approximate the level of noise in the noise-reduced echo-reduced signal.

The communication device102may include a summation module128that combines the output of the residual echo suppression module124and the noise generator module126. For example, the summation module128can combine the time-domain noise signal mn with the residual-echo-reduced signal rer to generate an output signal o. In some embodiments, the summation module128simply sums the time-domain noise signal mn and the residual-echo-reduced signal rer to generate an output signal o. The output signal o may be sent directly through a communication link104to another communication device. Alternatively, the output signal o may be provided to a speech recognition module. In addition, the output signal o may be provided to a gain control module (e.g., if a destination for the output signal o was a wireless headset (e.g., a Bluetooth device), it may be desirable to increase the gain of the output signal o before sending to the headset).

Additional hardware and/or software modules or components may be included in the communication system102. For example, the communication device102may include an automatic speech recognition (“ASR”) module (not shown) for performing speech recognition on an audio signal that corresponds to a user utterance. The communication device102may also include a network communication module (not shown) for establishing communications over communication networks, such as communication link104, or directly with other computing devices.

Illustratively, the communication system102may be (or be part of) a personal computing device, laptop computing device, hand held computing device, terminal computing device, server computing device, mobile device (e.g., mobile phones or tablet computing devices), wearable device configured with network access and program execution capabilities (e.g., “smart eyewear” or “smart watches”), wireless device, electronic reader, media player, home entertainment system, gaming console, set-top box, television configured with network access and program execution capabilities (e.g., “smart TVs”), telephone, or some other electronic device or appliance.

Process for Generating Comfort Noise

With reference now toFIG. 3, an example process300for performing comfort noise generation according to an embodiment will be described. The process300begins at block302. For example, if the noise generator module126receives signal level information sl indicating to generate comfort noise, the process300may begin. The process300may be embodied in hardware, a set of executable program instructions, or a combination of hardware and executable program instructions. The process300may be performed, for example, by the noise generator module126of the communication device102ofFIG. 2. Although the process300ofFIG. 3will be described with respect to the components ofFIG. 2, the process300is not limited to implementation by, or in conjunction with, any specific component shown inFIG. 2. In some embodiments, the process300, or some variant thereof, may be implemented by alternative components, by the components ofFIG. 2in a different configuration, etc.

At block304, the noise generator module126receives the noise component nc estimated by the noise reduction module122. At block306, the noise generator module determines the spectral shape of the noise component nc. For example, the noise generator module may determine the spectral shape of the power spectrum density of the noise component nc, as described below with respect toFIG. 4. In general, the spectral shape provides information with respect to the frequency characteristics of the noise component nc.

At block308, the noise generator module126receives a frame of noise. For example, the communication device102may include a memory130configured to store a plurality of frames of noise. A frame of noise can be any plurality of samples of noise. In some embodiments, a frame of noise comprises 128 samples. The noise generator module126may receive the frame of noise from the memory130. In order to not select the same frame of noise each time the process300is executed, which could create a pattern in the audio output signal that is perceivable to a listener, the noise generator module can randomly select the frame of noise from the memory130(e.g., based on an index). In alternative embodiments, the noise generator module126, or another component of the communication device102, may generate the received frame of noise. However, generating the frame of noise typically would require more computational resources than reading a randomly selected frame of noise from the memory130.

In some embodiments, the received frame of noise is white noise. White noise generally has a substantially flat power spectral density. For example, white noise has approximately equal power within any frequency band of a fixed width.

At block310, the noise generator module126may transform the received frame of noise to the frequency domain. For example, the noise generator module126may process a fast Fourier transform of the received frame of noise.

At block312, the noise generator module126modifies the received frame of noise based at least in part on the spectral shape of the noise component nc. For example, the noise generator module126modifies the substantially flat power spectral density of the received frame of white noise to have approximately the same power spectral density as determined for the noise component nc. In other embodiments, the received frame of noise may not correspond to white noise. In such embodiments, the noise generator module126could modify the non-flat power spectral density of the received frame of white noise to have approximately the same power spectral density of the noise component nc by applying different weights that compensate for the received frame of noise not having flat power spectral density.

At block314, the noise generator module126generates a time-domain noise signal mn based on the modified frame of noise. The noise generator module126may generate the time-domain noise signal mn by converting the modified frame of noise from the frequency domain to the time domain (e.g., by taking the inverse Fourier transform).

At block316, the noise generator module126determines whether there are more audio frames to be processed. If so, the process repeats, beginning at blocks304and308. For example, blocks304,306,308,310,312,314, and316may be executed in a continuous or substantially continuous loop until there are no more frames to be processed. For example, if the noise generator module126receives signal level information sl indicating to generate another frame of comfort noise, the process300may be repeated, beginning at blocks304and308. After there are no more frames to be processed, the process300ends at block318.

Turning now toFIG. 4, another example process400for performing comfort noise generation according to an embodiment will be described. The process400begins at block402. The process400may be embodied in hardware, a set of executable program instructions, or a combination of hardware and executable program instructions. The process400may be performed, for example, by the noise generator module126of the communication device102ofFIG. 2. Although the process400ofFIG. 4will be described with respect to the components ofFIG. 2, the process400is not limited to implementation by, or in conjunction with, any specific component shown inFIG. 2. In some embodiments, the process400, or some variant thereof, may be implemented by alternative components, by the components ofFIG. 2in a different configuration, etc.

At block404, the noise generator module126receives the noise component nc estimated by the noise reduction module122. At block406, the noise generator module126determines the power spectrum density (“PSD”) of the noise component nc. In determining the PSD of the noise component nc, the noise generator module generally converts the noise component nc from a time domain signal to a frequency domain signal. For example, the noise generator module126determines a power spectrum density of the noise component nc at a plurality of frequency bands. In some embodiments, the plurality of frequency bands comprises approximately thirty frequency bands spaced between zero and eight kilohertz. In other embodiments, the plurality of frequency bands can comprise fewer or more frequency bands spaced between a smaller or larger frequency range. For example, in other embodiments, the plurality of frequency bands can comprise 128 frequency bands spaced between zero and 10 kHz. In other embodiments, the plurality of frequency bands can comprise 256 frequency bands spaced between zero and 20 kHz. Generally, as the number of frequency bands increases, the computational complexity increases.

At block408, the noise generator module126determines which of the plurality of frequency bands corresponds to a maximum of the power spectrum density of the noise component nc. For example, if the plurality of frequency bands comprises 30 frequency bands, the noise generator module126may determine which of the 30 frequency bands has a maximum amplitude.

At block410, the noise generator module126normalizes each of the plurality of frequency bands of the power spectrum density of the noise component nc relative to the maximum as determined at block408, in order to generate a normalized PSD noise component nc. For example, the noise generator module126may normalize by dividing the magnitude of the PSD noise component nc at each of the plurality of frequency bands by the maximum as determined at block408. After normalizing, the magnitude of the PSD noise component nc at each of the plurality of frequency bands has a value of between zero and one.

At block412, the noise generator module126uses the normalized PSD noise component nc to calculate a spectral shape of the PSD noise component nc. For example, in some embodiments, the noise generator module calculates a spectral shape of the PSD noise component nc by calculating a square root of the normalized noise PSD noise component nc at each of the plurality of frequency bands.

Turning to block414, the noise generator module126may generate a random index. For example, the random index may be a number that corresponds to an index value at which white noise is stored in memory130. In some embodiments, 10 frames of white noise can be stored in memory130, and the generated random index would take on values between 1 and 10. In other embodiments, any number of frames of white noise can be stored in memory130. For example, 100 frames of white noise can be stored in memory130, and the generated random index would take on values between 1 and 100. In addition, a frame of white noise can include any number of samples. For example, in some embodiments, a frame of white noise can include 128 samples. In some embodiments, the generated random index corresponds to a sample of a frame of noise. For example, if in an embodiment a memory stores 100 frames of noise, and each frame of noise includes 128 samples, then the generated random index may be configured to specify one of the 12,800 sample values of noise. In other embodiments, the generated random index corresponds to a frame of noise.

At block416, the noise generator module126receives a frame of white noise from the memory130, the index of the received frame of white noise corresponding to the random index generated at block414. By selecting the received frame of white noise based on a generated random number, the process400utilizes a technique that ensures that the same frame of white noise will not be selected each time the process300is executed, while reducing computational complexity compared to generating white noise on the fly. However, in alternative embodiments, the noise generator module126, or another component of the communication device102, may generate the received frame of white noise.

At block418, the noise generator module126transforms the received frame of white noise from the time domain to the frequency domain. For example, the noise generator module126can perform spectral analysis, using the same technique applied to the noise component nc, to obtain the power density spectrum of the received frame of white noise. Although blocks404,406,408,410, and412have been described before blocks414,416, and418, it should be understood that404,406,408,410, and412and blocks414,416, and418are independent and can be performed in either order.

At block420, the noise generator module126modifies the received frame of noise based on the spectral shape of the noise component nc. For example, the noise generator module126modifies the substantially flat power spectral density of the received frame of white noise to have approximately the same power spectral density as determined for the noise component nc. To modify, the noise generator module126may, for each of the plurality of frequency bands, multiply the magnitude of the PSD of the frame of white noise by the value of the PSD of the noise component nc.

At block422, the noise generator module126generates a time-domain noise signal mn based on the modified frame of noise from block420. For example, the noise generator module126may generate the time-domain noise signal mn by converting the modified frame of noise from the frequency domain to the time domain (e.g., by taking the inverse Fourier transform).

At block424, the noise generator module126can determine a target noise amplitude level for the time-domain noise signal mn. For example, in some embodiments, the target noise amplitude level is the same as the level of noise present in the noise-reduced echo-reduced signal nr, which can be indicated by the noise component nc estimated by noise reduction module122. In other embodiments, a noise control knob, either physical or virtual, may be provided on the communication device102whereby a user106may adjust the amplitude of the time-domain noise signal mn. For example, if the volume of the time-domain noise signal mn is such that a listener may not be able to hear it, and may inadvertently think there may be a problem with the communication link104, the user106can adjust the noise control knob to increase the amplitude of the comfort noise.

At block426, the noise generator module126can adjust a level of the time-domain noise signal mn based on the target noise level from block424. For example, the noise generator module can adjust a level of the time-domain noise signal mn to approximate the level of noise in the noise-reduced echo-reduced signal. By modifying the level of the time-domain noise signal mn to approximate the level of noise in the noise-reduced echo-reduced signal nr, the noise generator module126provides for a continuous level of noise output in the output signal o, even if the residual echo suppression module124had removed all noise from the residual-echo-reduced signal rer.

At block428, the noise generator module126determines whether there are more audio frames to be processed. If so, the process repeats, beginning at blocks404and414. The blocks of process400may be executed in a continuous or substantially continuous loop until there are no more frames to be processed. For example, if the noise generator module126receives signal level information sl indicating to generate another frame of comfort noise, the process400may be repeated, beginning at blocks404and414. After there are no more frames to be processed, the process400ends at block430.

Turning now toFIG. 5, an example process500for processing an audio signal according to an embodiment will be described. The process500begins at block502. The process500may be embodied in hardware, a set of executable program instructions, or a combination of hardware and executable program instructions. The process500may be performed, for example, by the noise generator module126, as well as the acoustic echo cancellation module120, noise reduction module122, and/or residual echo suppression module124, of the communication device102ofFIG. 2. Although the process500ofFIG. 5will be described with respect to the components ofFIG. 2, the process500is not limited to implementation by, or in conjunction with, any specific component shown inFIG. 2. In some embodiments, the process500, or some variant thereof, may be implemented by alternative components, by the components ofFIG. 2in a different configuration, etc.

At block504, the audio input signal is received. In some embodiments, the audio input signal includes user speech and a noise component nc. For example, the noise component nc may have been estimated by the noise reduction module122. In addition, the audio input signal may have been generated by the noise reduction module122.

At block506, the audio input signal is processed. For example, in some embodiments, the processing in block506can be performed by the residual echo suppression module124.

At block508, it is determined whether a level of the user speech is below a threshold. For example, in a single talk condition in which a user106is not speaking, the level of the user speech will be below the threshold, and the residual echo suppression module124may be likely to remove the noise component in the audio input signal during the processing at block506. Accordingly, if it is determined that a level of the user speech is below a threshold, it is desirable to add comfort noise back into the audio signal, and the routine proceeds to block510. If it is determined that a level of the user speech is not below a threshold, the residual echo suppression module124may be unlikely to entirely remove the noise component in the audio input signal during the processing at block506. Accordingly, it is not necessary to add comfort noise back into the audio signal, and the routine may proceed back to block504.

At block510, the noise generator module126receives the noise component nc. For example, the noise generator module126may receive the noise component estimated by the noise reduction module122. At block512, the noise generator module determines the spectral shape of the noise component nc. For example, the noise generator module may determine the spectral shape of the power spectrum density of the noise component nc, as described above with respect toFIG. 4.

At block514, the noise generator module514receives a plurality of samples of noise. For example, the communication device102may include a memory130configured to store a plurality of samples of noise. The noise generator module126may receive the frame of noise from the memory130. In order to not select the same starting sample of noise each time the process500is executed, which could create a pattern in the audio output signal that is perceivable to a listener, the noise generator module126can randomly select the starting sample of noise from the memory130. For example, the starting sample of noise from the memory can be based on a generated random index. In alternative embodiments, the noise generator module126, or another component of the communication device102, may generate the received frame of noise. However, generating the frame of noise typically would require more computational resources than reading a randomly selected frame of noise from the memory130.

In some embodiments, the received frame of noise is white noise. White noise generally has a substantially flat power spectral density. For example, white noise has approximately equal power within any frequency band of a fixed width.

At block516, the noise generator module126may transform the received frame of noise to the frequency domain. For example, the noise generator module126may process a fast Fourier transform of the received frame of noise.

At block518, the noise generator module126modifies the received samples of noise based at least in part on the spectral shape of the noise component nc. For example, the noise generator module126modifies the substantially flat power spectral density of the received samples of white noise to have approximately the same power spectral density as determined for the noise component nc. In other embodiments, the received samples of noise may not correspond to white noise. In such embodiments, the noise generator module126could modify the non-flat power spectral density of the received frame of white noise to have approximately the same power spectral density of the noise component nc by applying different weights that compensate for the received samples of noise not having flat power spectral density.

At block520, the noise generator module126generates a time-domain noise signal mn based on the modified samples of noise. The noise generator module126may generate the time-domain noise signal mn by converting the modified samples of noise from the frequency domain to the time domain (e.g., by taking the inverse Fourier transform).

At block522, the time domain noise signal mn is combined with the processed audio signal. For example, the summation module128may combine the time domain noise signal mn and the processed audio signal (e.g., the output of the residual echo suppression module124). By combining, the summation module128may add comfort noise to compensate for noise unintentionally removed during processing of the audio signal.

At block524, the noise generator module126determines whether there are more audio frames to be processed. If so, the process repeats, beginning at block504. For example, blocks504through522may be executed in a continuous or substantially continuous loop until there are no more frames to be processed. After there are no more frames to be processed, the process500ends at block526.

Noise PSD Examples

FIG. 6Ais a diagram illustrating an example of a PSD of background noise n in a particular environment before noise reduction. The vertical axis indicates the amplitude of the PSD at a particular frequency. The background noise n includes measured frequencies between zero and eight kHz. To reduce computation complexity, the PSD between eight kHz and sixteen kHz is symmetric to the PSD between zero and eight kHz.

FIG. 6Bis a diagram illustrating an example of a power spectrum density of comfort noise generated by a noise generator according to an embodiment. As shown, the spectral density of the PSD of the generated noise is approximately the same as the PSD of the measured background noise shown inFIG. 6A. However, the spectral density of the PSD of the generated noise, as shown, has a reduced amplitude compared to the PSD of the measured background noise. This is because the level of the time-domain noise signal mn is set to approximate the lower level of noise in the noise-reduced echo-reduced signal produced by the noise reduction module122, not the background noise level n.

Terminology