Aspects of the subject technology relate to a device including a microphone, a filter and a processor. The filter receives an audio signal including ambient noise and a voice of a user of the device from the microphone. At least a portion of ambient noise is filtered from the audio signal. The processor determines a level of the ambient noise in the received audio signal and dynamically adjusts a gain applied to the filtered audio signal based on the level of the ambient noise.

TECHNICAL FIELD

The present description relates generally to audio processing and more particularly, but not exclusively, to audio processing for self-voice adaptation.

BACKGROUND

With mobile phones, in order to make it more comfortable for a user talking on a phone, a provision is made for the user to hear a weak version of her/his own voice, which is referred to as sidetone. The purpose of the sidetone, which is usually provided at a fairly low volume, is to allow a caller to get a sense of the volume of her/his voice. When the microphone is too close to the speaker, the sidetone may cause unwanted feedback (echo).

DETAILED DESCRIPTION

When a user of a mobile device uses a headphone or an earbud, the user may have very little feedback from her/his own voice. For example, noise-controlled headsets and/or headphones have provisions for controlling the ambient noise that can also muffle or reduce the user's own voice while keeping the ambient noise low during a phone call (and/or other audio communication sessions). The phone call may sound unnatural and/or unusual to the user when they cannot hear their own voice. However, allowing a sidetone (including the user's voice) to pass through the audio path may result in a loss and/or degradation of the active noise cancellation.

The subject technology is directed to self-voice adaptation, where the ambient noise level, excluding the speech signal, is tracked and used to set the sidetone level. For example, during a high level of ambient noise, the user may prefer to have the noise reduced rather than hear her/his own voice, e.g., the user may not notice their own voice during high levels of ambient noise. On the other hand, when the level of the ambient noise is low, the user may have interest in hearing their voice as the user's voice may be more noticeable during low levels of ambient noise. The subject technology tracks and analyzes the ambient noise and adjusts the level of the sidetone (that includes the user's voice) based on the result of the analysis. For instance, when the ambient noise level is below a predefined level, the level of the sidetone is set to a higher value than when the ambient noise level is high.

FIG.1illustrates a high-level block diagram of an example of a device100with self-voice adaptation, in accordance with various aspects of the subject technology. The device100includes a microphone (Mic)102, an active noise cancellation filter110, a sidetone filter120, an ambient noise processor130, a gain stage140, a summing circuit150and a speaker160. In one or more implementations, the device100can be a headset or an earbud that is coupled, wirelessly or via a wired link, to a handheld device, such as a smartphone, a smartwatch or a tablet, or to a computing device such as a laptop or desktop computer.

The device100has an active noise cancellation feature that is supported by the active noise cancellation filter110. The active noise cancellation filter110is a variation of optimal filtering that can produce an estimate of the noise by filtering the input signal104received from the microphone102and then subtracting the estimated noise from the input signal104. An estimate of the noise can be produced, for example, by adaptive prediction or by using a prediction filter that exploits a low-pass characteristic of the input signal104. In one or more implementations, the active noise cancellation filter110can be implemented at least partially in hardware, firmware or software. The output of the active noise cancellation filter110is a substantially noise-free audio signal112. In some aspects, the active noise cancellation filter110may not be able to distinguish between the noise and the user's self-voice. Therefore, the user's self-voice may be at least partially removed as well.

The sidetone filter120is used to filter the noise from the input signal104received from the microphone102, for example, the ambient noise, and to produce an audio signal122that replicates the voice of a user speaking to the microphone. In some implementations of the subject technology, the sidetone filter120can sample the input signal104at a very high rate (e.g., thousands of samples per second). The audio signal122of the sidetone filter120is amplified by the gain stage140, which is a variable gain amplifier. The variable gain of the gain stage140is controlled via a control signal132produced by the ambient noise processor130.

The ambient noise processor130receives and analyzes the input signal104from the microphone102, estimates the level of ambient noise in the input signal104, and produces the control signal132based on the level of the estimated ambient noise. For example, when the level of the ambient noise is high, such as above a preset threshold (e.g., 65 dB sound-pressure level (SPL)), the gain of the gain stage140is set to a lower value (e.g., −40 dB). This is because the user may prefer to have the noise reduced rather than hear her/his own voice. In one or more implementations, the ambient noise processor130may use different threshold values to set the gain of the gain stage140. For example, the ambient noise processor130may set the gain of the gain stage140to a medium level (e.g., −36 dB) when the level of ambient noise is in a medium range (e.g., between 55-65 dB SPL) and set the gain of the gain stage140to a higher level (e.g., −32 dB) when the level of ambient noise is low (e.g., less than 55 dB SPL). This feature of the subject technology may result in an improved user experience during phone calls (and/or other communication sessions), as the user is allowed to hear more of their own voice during the phone call when the ambient noise level is low.

In one or more implementations, the ambient noise processor130is implemented at least partially by hardware, firmware, or software. In some implementations, some of the functionalities of ambient noise processor130can be performed by a processor of a host device, such as a smartphone or a smartwatch. The summing circuit150adds the amplified audio signal142of the gain stage140to the noise-free audio signal112and provides an audio signal152to the speaker160. The audio signal152includes the noise-free audio from the microphone102and the self-voice of the user at a controlled level, based on the level of the ambient noise in the input signal104.

FIG.2illustrates a high-level block diagram of an example of a device200with self-voice adaptation, in accordance with various aspects of the subject technology. The device200includes the microphone102, the active noise cancellation filter110, a variable sidetone filter220, the ambient noise processor130, the gain stage140, the summing circuit150and the speaker160. In one or more implementations, the device200can be a headset or an earbud that is coupled, wirelessly or via a wired link, to a handheld device, such as a smartphone, a smartwatch or a tablet, or to a computing device such as a laptop or desktop computer.

The descriptions of the active noise cancellation filter110, the ambient noise processor130, the gain stage140, the summing circuit150and the speaker160are the same as provided with respect toFIG.1and are skipped here for brevity. The variable sidetone filter220is different from the sidetone filter120ofFIG.1in the sense that it has variable frequency characteristics, which can include specifics of a passband such as a mid-band frequency and a bandwidth or the lower and/or the upper frequencies of the passband or other frequency characteristics.

In some implementations, the frequency characteristics of the variable sidetone filter220can be controlled by a control signal134provided by the ambient noise processor130. In other words, the ambient noise processor130can estimate a level and a frequency spectrum of the ambient noise in the input signal104and, based on the estimated ambient noise level and frequency spectrum, generate the control signals132and134. The control signal132controls the gain of the gain stage140to provide a suitable self-voice level based on the level of ambient noise, as discussed above with respect toFIG.1. The control signal134controls the frequency characteristics of the variable sidetone filter220based on the estimated frequency spectrum of the ambient noise. This enables the variable sidetone filter220to be more effective in distinguishing the user's self-voice from the ambient noise and to provide a rather noise-free self-voice signal222to the gain stage140.

FIG.3illustrates a block diagram of an example of a device300with self-voice adaptation based on the ambient noise level, in accordance with various aspects of the subject technology. The device300includes the microphone102(Mic1), the active noise cancellation filter110, a transparency filter320, the ambient noise processor130, the gain stage140, a cross fader325, the summing circuit150, an active noise cancellation filter350, a soft clip360, a downlink370, the speaker160and a microphone302(Mic2). In one or more implementations, the device300can be a headset or an earbud that is coupled, wirelessly or via a wired link, to a handheld device, such as a smartphone, a smartwatch or a tablet, or to a computing device such as a laptop or desktop computer.

The descriptions of the active noise cancellation filter110, gain stage140and the summing circuit150are the same as discussed with respect toFIG.1and are skipped herein for brevity. The transparency filter320can analyze the input signal104received from the microphone102and reproduce an ambient sound environment322by suitably filtering the ambient noise components of the input signal104. Examples of the ambient sound environment322include, announcements at a sports field or a speech broadcasted over a loudspeaker. Furthermore, when the user is speaking, the ambient sound environment322may include the user's voice. The acoustical transparency (also referred to as transparent hearing, or hear-through mode) function may be desirable in some usage scenarios to reproduce the ambient sound environment through the earpiece speaker drivers of the headset.

In some implementations, a data processor may perform an algorithm that adjusts the filters of the transparency filter320and reduces the acoustic occlusion due to an earpiece of the headset, while also preserving the spatial filtering effect of the wearer's anatomical features (e.g., head, pinna, shoulder). The filters may help preserve the timbre and spatial cues associated with the actual ambient sound.

As described above with respect toFIG.1, the output of the active noise cancellation filter110is a substantially noise-free audio signal112, which is mixed with the surround sound322by the cross-fader (XF)325. In mixing the noise-free audio signal112and the surround sound322, the XF325may allow the level of the former to fade in while the level of the latter fades out, or vice versa. The ambient sound environment322is amplified by the gain stage140, the gain of which is controlled by the control signal332of the dynamic ambient noise processor330.

The dynamic ambient noise processor330analyzes the ambient noise level of the input signal104, estimates the level of the ambient noise and produces the control signal332based on the level of the estimate of the ambient noise. Therefore, the ambient sound environment322is amplified by the gain stage140having a gain that is controlled based on the ambient noise. That is to say, if the level of the ambient noise is high, the ambient sound environment322including the user's voice when speaking may be amplified with a higher gain so that the ambient noise can be overcome. The dynamic ambient noise processor330can periodically estimate the level of the ambient noise. The dynamic nature of the dynamic ambient noise processor330enables operation in an environment that the level and nature of surround sound may vary with time.

The summing circuit150mixes the output of the XF325and the amplified surround sound342with a feedback signal352. The feedback signal352is generated by the active noise cancellation filter350, which is similar to the active noise cancellation filter110, and can filter noise of an audio signal produced by the microphone302. The microphone302is located close to the speaker160and can pick up an output of the speaker160, which receives mixed audio signals including an output of the summing circuit150and a downlink signal372through the soft clip360. The downlink signal372is generated by the downlink370, which receives an audio output from another speaker (e.g., a right/left speaker of a stereo headset, not shown for simplicity). The soft clip360can create a type of distortion effect where the signal amplitude is saturated along a smooth curve, rather than the abrupt shape used in hard-clipping.

FIG.4illustrates a schematic diagram of an example of a device400with self-voice adaptation based on the ambient noise level, in accordance with various aspects of the subject technology. The device400is similar to the device300ofFIG.3, except that an equalizer432is added after the dynamic ambient noise processor330, and the input signal of the gain stage440is received from the equalizer432rather than the transparency filter320. The dynamic ambient noise processor330not only produces the control signal332that controls the gain of the gain stage440based on the level of the ambient noise, as described above with respect theFIG.3, but it also provides the sidetone signal334to the equalizer432.

The equalizer432performs the process of adjusting the balance between frequency components of the sidetone signal334by strengthening or weakening the energy of specific frequency bands of the sidetone signal334to provide a filtered sidetone signal436as the input signal to the gain stage440. The output signal442of the gain stage440is an amplified version of the filtered sidetone signal436, which is amplified based on the level of the ambient noise. For example, if the level of the ambient noise is lower than a threshold value, the gain of the gain stage440is set to a higher value, as the user prefers to hear more of her/his own voice while on a call when the noise level is low. In some implementations, the gain of the gain stage440can be set using a look-up table (LUT).

The descriptions of the functionalities of the active noise cancellation filters110and350, the transparency filter320, the XF325, the summing circuit150, the soft clip360and the downlink370are similar to the descriptions provided with respect toFIG.3and are skipped herein for brevity.

FIG.5illustrates a flow diagram of an example process500of self-voice adaptation, in accordance with various aspects of the subject technology. For explanatory purposes, the process500is primarily described herein with reference to the device100ofFIG.1. However, the process500is not limited to the device100ofFIG.1, and one or more blocks (or operations) of the process500may be performed by one or more other components of other suitable devices, such as earbuds, headphones, headsets, and the like. Further for explanatory purposes, the blocks of the process500are described herein as occurring in serial, or linearly. However, multiple blocks of the process500may occur in parallel. In addition, the blocks of the process500need not be performed in the order shown and/or one or more blocks of the process500need not be performed and/or can be replaced by other operations.

The process500includes receiving an audio signal (e.g.,104ofFIG.1) corresponding to a microphone (e.g.,102ofFIG.1) (510). The audio signal includes ambient noise and one or more audio components. At least a portion of the ambient noise is filtered (e.g., by102ofFIG.1) from the audio signal to generate a filtered audio signal (e.g.,122ofFIG.1) (520). A level of the ambient noise in the received audio signal is determined (530). A gain applied (e.g., by140ofFIG.1) to the filtered audio signal is dynamically adjusted based on the level of the ambient noise (540).

FIG.6illustrates a flow diagram of an example process600of self-voice adaptation, in accordance with various aspects of the subject technology. For explanatory purposes, the process600is primarily described herein with reference to the device100ofFIG.1. However, the process600is not limited to the device100ofFIG.1, and one or more blocks (or operations) of the process600may be performed by one or more other components of other suitable devices, such as earbuds, headphones, headsets, and the like. Further for explanatory purposes, the blocks of the process600are described herein as occurring in serial, or linearly. However, multiple blocks of the process600may occur in parallel. In addition, the blocks of the process600need not be performed in the order shown and/or one or more blocks of the process600need not be performed and/or can be replaced by other operations.

The process600begins when the device100, such as a headset, determines whether it is paired with a handheld communication device, such as a smartphone or a smartwatch of a user of the headset (610). Next, the headset is set to a hand-free profile (HPF) mode (620), and an estimate of the near-end noise is received from a processor (e.g.,130ofFIG.2) (630). Next, the input-sound pressure level (SPL) smoothing is performed by a variable sidetone filter (e.g., by220ofFIG.2) (640).

The sidetone gain of a gain stage (e.g.,140ofFIG.1) is set by using an LUT (660). Then, it is checked whether the near-end noise has changed from a previous estimated value (670). If no change is observed, the sidetone gain of the gain stage remains unchanged, and control is passed to the operation block690(680). If, at operation block670, it is determined that the near-end noise has changed, control is passed to operation block640. Next, it is checked whether the headset is still in HPF mode (690). If the headset is still in HPF mode, control is passed to operation block670. If the headset is not in HPF mode any longer, the process600ends.

FIG.7illustrates a wireless communication device700within which some aspects of the subject technology are implemented. In one or more implementations, the wireless communication device700can be a headset or an earbud device of the subject technology, for example, any of the devices100,200,300or400ofFIG.1,2,3or4. The wireless communication device700may comprise a radio-frequency (RF) antenna710, a duplexer712, a receiver720, a transmitter730, a baseband processing module740, a memory750, a processor760and a local oscillator generator (LOGEN)770. In various aspects of the subject technology, one or more of the blocks represented inFIG.7may be integrated on one or more semiconductor substrates. For example, the blocks720-770may be realized in a single chip or a single system on a chip, or may be realized in a multichip chipset.

The receiver720may comprise suitable logic circuitry and/or code that may be operable to receive and process signals from the RF antenna710. The receiver720may, for example, be operable to amplify and/or down-convert received wireless signals. In various aspects of the subject technology, the receiver720may be operable to cancel noise in received signals and may be linear over a wide range of frequencies. In this manner, the receiver720may be suitable for receiving signals in accordance with a variety of wireless standards, such as Wi-Fi, WiMAX, Bluetooth, and various cellular standards. In various aspects of the subject technology, the receiver720may not use any sawtooth acoustic wave (SAW) filters and few or no off-chip discrete components such as large capacitors and inductors.

The transmitter730may comprise suitable logic circuitry and/or code that may be operable to process and transmit signals from the RF antenna710. The transmitter730may, for example, be operable to upconvert baseband signals to RF signals and amplify RF signals. In various aspects of the subject technology, the transmitter730may be operable to upconvert and amplify baseband signals processed in accordance with a variety of wireless standards. Examples of such standards may include Wi-Fi, WiMAX, Bluetooth, and various cellular standards. In various aspects of the subject technology, the transmitter730may be operable to provide signals for further amplification by one or more power amplifiers.

The duplexer712may provide isolation in the transmit band to avoid saturation of the receiver720or damaging parts of the receiver720, and to relax one or more design requirements of the receiver720. Furthermore, the duplexer712may attenuate the noise in the receive band. The duplexer712may be operable in multiple frequency bands of various wireless standards.

The baseband processing module740may comprise suitable logic, circuitry, interfaces, and/or code that may be operable to perform the processing of baseband signals. The baseband processing module740may, for example, analyze received signals and generate control and/or feedback signals for configuring various components of the wireless communication device700, such as the receiver720. The baseband processing module740may be operable to encode, decode, transcode, modulate, demodulate, encrypt, decrypt, scramble, descramble, and/or otherwise process data in accordance with one or more wireless standards.

The processor760may comprise suitable logic, circuitry, and/or code that may enable processing data and/or controlling operations of the wireless communication device700. In this regard, the processor760may be enabled to provide control signals to various other portions of the wireless communication device700. The processor760may also control transfer of data between various portions of the wireless communication device700. Additionally, the processor760may enable implementation of an operating system or otherwise execute code to manage operations of the wireless communication device700. In one or more implementations, the processor760can be used to perform some of the functionalities of the subject technology.

The memory750may comprise suitable logic, circuitry, and/or code that may enable storage of various types of information such as received data, generated data, code, and/or configuration information. The memory750may comprise, for example, RAM, ROM, flash, and/or magnetic storage. In various aspects of the subject technology, information stored in the memory750may be utilized for configuring the receiver720and/or the baseband processing module740. In some implementations, the memory750may store image information from processed and/or unprocessed fingerprint images of the under-display fingerprint sensor of the subject technology.

The LOGEN770may comprise suitable logic, circuitry, interfaces, and/or code that may be operable to generate one or more oscillating signals of one or more frequencies. The LOGEN770may be operable to generate digital and/or analog signals. In this manner, the LOGEN770may be operable to generate one or more clock signals and/or sinusoidal signals. Characteristics of the oscillating signals such as the frequency and duty cycle may be determined based on one or more control signals from, for example, the processor760and/or the baseband processing module740.

In operation, the processor760may configure the various components of the wireless communication device700based on a wireless standard according to which it is desired to receive signals. Wireless signals may be received via the RF antenna710, amplified, and downconverted by the receiver720. The baseband processing module740may perform noise estimation and/or noise cancellation, decoding, and/or demodulation of the baseband signals. In this manner, information in the received signal may be recovered and utilized appropriately. For example, the information may be audio and/or video to be presented to a user of the wireless communication device700, data to be stored to the memory750, and/or information affecting and/or enabling operation of the wireless communication device700. The baseband processing module740may modulate, encode, and perform other processing on audio, video, and/or control signals to be transmitted by the transmitter730in accordance with various wireless standards.

Various functions described above can be implemented in digital electronic circuitry, as well as, in computer software, firmware or hardware. The techniques can be implemented by using one or more computer program products. Programmable processors and computers can be included in or packaged as mobile devices. The processes and logic flows can be performed by one or more programmable processors and by one or more programmable logic circuitries. General and special-purpose computing devices and storage devices can be interconnected through communication networks.

Some implementations include electronic components, such as microprocessors and storage and memory that store computer program instructions in a machine-readable or computer-readable medium (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM or flash memory. The computer-readable media can store a computer program that is executable by at least one processing unit and include sets of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter.

Many of the above-described features and applications are implemented as software processes that are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more processing unit(s) (e.g., one or more processors, cores of processors, or other processing units), they cause the processing unit(s) to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, flash drives, RAM chips, hard drives, EPROMs, etc. The computer-readable media does not include carrier waves and electronic signals passing wirelessly or over wired connections.

All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims. No claim element is to be construed under the provisions of 35 U.S.C. § 112(f) unless the element is expressly recited using the phrase “means for,” or, in the case of a method claim, the element is recited using the phrase “step for.” Furthermore, to the extent that the terms “include,” “have,” or the like are used in the description or the claims, such terms are intended to be inclusive in a manner similar to the term “comprise,” as “comprise” is interpreted when employed as a transitional word in a claim.