Patent Description:
<CIT> discloses a method for keystroke sound suppression.

<CIT> discloses a "running range normalization" method.

This disclosure describes apparatuses and techniques of adaptive energy limiting for transient noise suppression. The invention is defined in the independent claims and the dependent claims define embodiments of the invention.

In some aspects, a method for adaptive energy limiting includes setting a limiter ceiling for an audio signal to full scale and receiving a portion of the audio signal. The method then determines a maximum amplitude of the portion of the
audio signal and evaluates the portion of the audio signal with a neural network to provide a voice likelihood estimate for the portion of the audio signal. Based on the maximum amplitude and the voice likelihood estimate, the method determines that the portion of the audio signal includes noise. In response to determining that the portion of the audio signal includes noise, the method decreases the limiter ceiling. The limiter ceiling is then provided to a limiter module through which the audio signal passes to limit an amount of energy of the audio signal. By so doing, the audio signal may be prevented from carrying full energy transient noise into conference audio or subsequent audio processes, such as speaker selection for video conferencing.

In other aspects, an apparatus includes a network interface to receive or transmit an audio signal over a data network and a limiter module to limit energy of the audio signal. The apparatus also includes a hardware-based processor associated with the data interface and storage media storing processor-executable instructions for an adaptive energy limiter. The adaptive energy limiter is implemented to set a limiter ceiling for the audio signal to full scale and provide, from the audio signal, a frame of audio that corresponds to a duration of audio from the audio signal. The adaptive energy limiter then determines, for the frame of audio, a maximum amplitude of the audio signal and evaluates the frame of audio with a neural network to provide a voice likelihood estimate for the frame of audio. Based on the maximum amplitude and the voice likelihood estimate, the adaptive energy limiter determines that the frame of audio includes noise. The adaptive energy limiter then decreases the limiter ceiling in response to the determination that the frame of audio includes noise and provides, to the limiter module, the limiter ceiling to reduce the energy of the audio signal.

In yet other aspects, a system comprises a hardware-based processor operably associated with an audio interface or a data interface by which an audio signal is received and storage media storing processor-executable instructions for an adaptive energy limiter. The adaptive energy limiter is implemented to set a limiter ceiling for the audio signal to full scale and generate, based on the audio signal, a frame of audio that corresponds to a duration of audio from the audio signal. The adaptive energy limiter then determines, for the frame of audio, a maximum amplitude of the audio signal and evaluates the frame of audio with a neural network to provide a voice likelihood estimate for the frame of audio. Based on the maximum amplitude and the voice likelihood estimate, the adaptive energy limiter determines that the frame of audio includes noise. The adaptive energy limiter then decreases the limiter ceiling in response to the determination that the frame of audio includes noise and provides, to a limiter module, the limiter ceiling to reduce the energy of the audio signal.

The details of one or more implementations of adaptive energy limiting for transient noise suppression are set forth in the accompanying drawings and the following description. This summary is provided to introduce subject matter that is further described in the Detailed Description and Drawings. Accordingly, this summary should not be considered to describe essential features nor used to limit the scope of the claimed subject matter.

This specification describes apparatuses and techniques of adaptive energy limiting for transient noise suppression with reference to the following drawings. The same numbers are used throughout the drawings to reference like features and components:.

Audio conferences or video conferences often include many participants, with one or few of the participants actively speaking at any given time. When not speaking, the other participants typically produce noise, which may be picked up by their microphones and fed into the audio of the conference for all participants to hear. Example noises generated by conference participants may include typing on a keyboard, placing a coffee cup on a table, shuffling paper, moving chairs, shutting doors, and so on. Some of these noises have a transient characteristic that, unlike static or recurrent noise, prevent suppression through conventional noise reduction techniques. Additionally, audio energy of transient noise is typically as high, or higher, than energy levels associated with speech of the conference participants. As such, these transient noises are often fed into the conference audio as raw unsuppressed energy, resulting in noise that may disrupt the speaker and listeners, overpower the speaker's voice, trigger residual echo suppression, falsely trigger audio or video switch schemes, or the like.

Because conventional techniques of noise reduction are unable to mitigate transient noise, there are multiple negative consequences that affect conference call participants. Generally, the unsuppressed noise is let through to the other end of the call, disturbing both the speaker and other listeners. This unsuppressed noise may also, when let through to a current speaker in the call, trigger residual echo suppression that dampens the speaker's voice or affect backend speaker selection schemes such as top-<NUM> filtering (e.g., passing through respective audio of the three call participants with the most energy). Additionally, the conference system may incorrectly prioritize noisy participants over actively speaking participants or interrupt video switching schemes by switching a video feed of the speaker to the participant generating the noise.

Some conventional techniques involve having participants that are not currently speaking manually mute their respective microphone. Muting solutions, however, are undesirable and inconvenient because these solutions result in unnatural conversational flow and often cause issues when participants forget to unmute their microphone before speaking. Manually muting microphones can be especially frustrating in a large meeting room in which many participants take turns speaking such that muting occurs very frequently. For example, anytime someone wants to speak to the other participants, that person would need to reach for a remote control or button on a device to unmute their microphone, and then remember to mute again afterwards. As such, manual muting that relies on timely manual interaction from all participants is inconvenient and often ineffective at suppressing transient noise.

Other conventional techniques also typically fail at preventing transient noise from entering the conference audio or do so at the cost of other impairments to call flow or quality. For example, some phones include noise gates that auto-mute unless there is strong energy present in an audio stream. These noise gates, however, lead to choppy quality audio and often let high-energy noise through to the conference audio. Other noise reduction techniques only work for stationary or slightly non-stationary noise (e.g., fans, traffic, background babble), not transient noise, which is sudden, non-constant, and high energy. In other cases, keyboard suppression predicts when keyboard sounds will occur and selectively suppresses these sounds. This suppression is limited to cases where the typing happens on the same laptop that is hosting the meeting, and only works for keyboard noise. Accordingly, conventional noise suppression techniques for conferences calls are unable to suppress or limit transient noise, which often interferes with call flow and quality.

This document describes apparatuses and techniques for adaptive energy limiting for transient noise suppression. As described, participants of a conference call may generate transient noise that, when allowed into the conferenced audio, often disrupts the speaker and other participants. Transient noise may also interfere with or degrade conference service processes for audio and video features, such as audio stream or video stream selection (e.g., active speaker) for presentation to other participants. Generally, aspects of adaptive energy limiting manage or control a maximum level of energy that a participant is allowed to contribute based on the participant's recent history of producing noise or speech. In various aspects, an adaptive energy limiter of a user device or conference system sets a limiter ceiling for an audio signal to full scale and receives a portion of the audio signal. For the portion of the audio signal, the adaptive energy limiter determines a maximum amplitude and evaluates the portion with a neural network to provide a voice likelihood estimate. Based on the maximum amplitude and the voice likelihood estimate, the adaptive energy limiter determines that the portion of the audio signal includes noise. In response to determining that the portion of the audio signal includes noise, the adaptive energy limiter decreases the limiter ceiling and provides the limiter ceiling to a limiter module effective to limit an amount of energy of the audio signal. By so doing, the adaptive energy limiter may prevent the audio signal from carrying full energy transient noise into conference audio or subsequent audio processes, such as speaker selection for video conferencing.

By way of example, if a participant makes noise, the ceiling of energy let through by the adaptive energy limiter will gradually be decreased. Generally, this will result in future sudden noise generated by that participant being less intrusive, and more easily ignored by other conference service algorithms, such as speaker selection for video conferencing. In some aspects, the ceiling of audio energy decreases to a minimum level after approximately <NUM> to <NUM> seconds of medium or high-energy noise, after which audio energy (e.g., noise energy) from that participant will be very limited. When that participant does start to speak, the adaptive energy limiter may reset the ceiling of audio energy to a maximum level (e.g., speech level or full scale), to let speech audio of the participant through to the other conference participants. The adaptive energy limiter does so quickly, such that the transient noise suppression provided by the adaptive energy limiter has little detrimental effect to the speech audio of the conference call. Alternately or additionally, if a participant is silent, quiet, or making low energy background noise, the adaptive energy limiter may maintain or leave the ceiling of audio energy high, as to not effect speech audio when the participant begins to speak.

Generally, aspects of adaptive energy limiting for transient noise suppression limit energy of transient noise without impairing quality of speech audio of a conference call or voice call. For example, by using long-term statistical properties of noise and/or speech in the context of audio or video conferencing scenarios, the adaptive energy limiter may substantially reduce an amount or effects of transient noise while minimally affecting speech. In other words, the adaptive noise limiter does not attempt to remove noise from concurrent noise and speech, which is otherwise a typical issue with conventional noise reduction techniques when trying to remove noise, particularly noise that may be confused with speech.

In various aspects of adaptive energy limiting, amplitude of an audio signal is measured for a time, and together with the other described utilizations of statistical properties, a limiter ceiling for audio energy is configured to prevent or suppress transient noise from entering a conference call. In some cases, a neural network is implemented to provide statistical properties about the audio signal. In accordance with various aspects, a small neural network has sufficient accuracy for such a task, such that no special acceleration hardware is needed, and speech quality does not suffer by the limits in accuracy of the neural network or associated voice activity detector (VAD). Alternately or additionally, an adaptive energy manager may be implemented to adjust or manage gain or sub-band gain of an audio signal based on the audio signal evaluations described herein.

As such, various aspects of energy limiting (or energy management) may be implemented to limit or reduce an amount of energy an audio signal is able to carry into a conference service, through a conference call, or out to conference call participants. In other words, for each participant, an adaptive energy limiter may track a noise debt that builds up as a participant continues to make noise. As the noise debt builds (or energy limit decreases), the adaptive energy limiter prevents or disallows that participant from sending a lot of energy into a call until that participant proves that they're sending speech (e.g., by sending a statistically significant amount of speech audio). The adaptive energy limiter may also effectively suppress transient noise by using (e.g., via the neural network) statistical energy differences of transient noises (e.g., high energy), vowels (e.g., medium energy) and consonants (e.g., low energy) to allow speech (e.g., consonants) to pass through perceptually unaffected even when transient noises are reduced <NUM> dB or more. Aspects of adaptive energy limiting may achieve such an effect through use of the limiter ceiling of audio signal energy and/or through management of sub-band gains used to process the audio signals of participants.

While any number of different environments, systems, devices, and/or various configurations can implement features and concepts of the described techniques and apparatuses for adaptive energy limiting for transient noise suppression, aspects of adaptive energy limiting for transient noise suppression are described in the context of the following example environment, devices, configuration, methods, and system.

<FIG> illustrates an example environment <NUM> in which various aspects of adaptive energy limiting for transient noise suppression can be implemented. In the example environment <NUM>, user devices <NUM> may communicate audio and/or video through a conference system <NUM> in which access to the system is provided by a conference service <NUM> (e.g., cloud-based meeting or conference service). User devices <NUM> in this example include a smartphone <NUM>-<NUM>, laptop computer <NUM>-<NUM>, tablet computer <NUM>-<NUM>, smartwatch <NUM>-<NUM>, telephone <NUM>-<NUM>, conference bridge <NUM>-<NUM>, and video conference display <NUM>-<NUM>. Although illustrated as devices, a user device may be implemented as any suitable computing or electronic device, such as a mobile communication device, a computing device, a client device, an entertainment device, a gaming device, a mobile gaming console, a personal media device, a media playback device, a charging station, an Advanced Driver Assistance System (ADAS), a point-of-sale (POS) transaction system, a health monitoring device, a drone, a camera, a wearable smart-device, a navigation device, a mobile-internet device (MID), an Internet home appliance capable of wireless Internet access and browsing, an Internet-ofThings (IoT) device, a Fifth Generation New Radio (5GNR) user equipment, and/or other types of user devices.

Generally, a respective user of a user device <NUM> may interact with other users through audio and/or video data exchanged through a data or voice connection to the conference service <NUM>. In some aspects, each user device <NUM> participating in an instance of a conference call facilitated by the conference service <NUM> provides an audio signal <NUM> and/or video signal through a respective connection with the conference service. For example, any or all of the user devices <NUM> may provide a channel of audio signals <NUM> (or audio data) that corresponds to audio captured by a microphone of that device. During a conference call, participants typically take turns speaking while other inactive or non-speaking participants listen or watch. Some of the participants, however, may choose to move a chair, write an e-mail, or take notes on a computer. Such moving and typing activities may generate transient noise, which may include a sound or sound wave with a short, pulse-like signal characteristic. Other potential sources of transient noise may include clicking noise from a computer mouse, moving items on a table or work surface, doors closing, phone keypad or ring tones, or the like. For example, if two participants, each at a respective endpoint of a conference or voice call are situated proximate each other in an open-plan office, one of the participants using a smartphone <NUM>-<NUM> and the other using a laptop computer <NUM>-<NUM>, potential transient noise may be generated at both endpoints when the participant using the laptop computer <NUM>-<NUM> starts typing.

In aspects of adaptive energy limiting for transient noise suppression, the conference service <NUM> includes an instance of an adaptive energy limiter <NUM> (adaptive limiter <NUM>), which may limit or manage energy of an audio signal to suppress various forms of transient noise. Although illustrated with reference to the conference service <NUM>, any or all of the user devices <NUM> may also include an instance of the adaptive energy limiter <NUM>. Thus, an adaptive energy limiter <NUM> may limit or manage energy of an audio signal sent to the conference service <NUM>, processed by the conference service <NUM>, or sent by the conference service to other user devices <NUM>. The adaptive energy limiter <NUM> is associated with or has access to a neural network <NUM>, which may be implemented as a recurrent neural network (RNN). In this example, the neural network <NUM> includes a voice activity detector <NUM> (VAD <NUM>) that may be configured to provide indications of voice likelihood for audio signals or frames of audio. For example, the adaptive energy limiter <NUM> may use the voice activity detector <NUM> to obtain an indication of voice likelihood for a frame of audio. Such an indication may be useful to determine whether the audio signal or frame of audio is more likely speech or noise. Alternately or additionally, the voice activity detector <NUM> can be implemented as a neural network-enabled voice activity detector that uses a neural network to determine or provide a voice likelihood measurement for a sample of audio signal or audio frame.

<FIG> illustrates at <NUM> example device diagrams of a user device <NUM> and a conference device <NUM>, which may provide the conference service <NUM>. Although each device is shown with an instance of an adaptive energy limiter, aspects of adaptive energy limiting may be implemented on one device, both devices, or in coordination between devices. For example, an adaptive energy limiter <NUM> of a user device <NUM> may interact with adaptive energy limiter <NUM> or neural network <NUM> of the conference device <NUM> to set a limiter ceiling value at the user device <NUM>. Shown in exemplary configurations, the user device <NUM> or the conference device <NUM> may also include additional functions, components, or interfaces omitted from <FIG> for the sake of clarity or visual brevity. Alternately or additionally, any respective components of the user device <NUM> or the conference device <NUM> may be implemented in whole or part as hardware logic or circuitry integrated with or separate from other components.

In this example, the user device <NUM> includes network interfaces <NUM> for exchanging data, such as audio signals or video streams, over various types of networks or communication protocols. Generally, the network interfaces <NUM> can be implemented as any one or more of a serial and/or parallel interface, a wireless interface, a wired interface, or a modem for transmitting or receiving data or signals. In some cases, the network interfaces <NUM> provide a connection and/or communication link between the user device <NUM> and a communication network by which other user devices <NUM>, and the conference device <NUM>, communicate audio signals <NUM>, video data, or the like for conferenced media communication. The user device <NUM> also includes at least one microphone <NUM> to capture audio (e.g., speech, sound, or noise) from an environment of the user device <NUM> and at least one speaker <NUM> to generate audio or sound based on audio data of the user device <NUM>. In some aspects, the microphone captures audio generated by a user, such as speech, and provides an audio signal to audio circuitry (not shown) of the user device <NUM> for encoding or other signal-processing.

The user device <NUM> also includes processor(s) <NUM> and computer-readable storage media <NUM> (CRM <NUM>). The processor(s) <NUM> may be a single core processor or a multiple core processor composed of a variety of materials, such as silicon, polysilicon, high-K dielectric, or the like. The computer-readable storage media <NUM> is configured as storage, and thus does not include transitory signals or carrier waves. The CRM <NUM> may include any suitable memory or storage device such as random-access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), non-volatile RAM (NVRAM), read-only memory (ROM), or Flash memory useable to store device data <NUM> of the user device <NUM>.

The device data <NUM> may include user data, multimedia data (e.g., audio data or video data), applications <NUM> (e.g., media conference client application), user interface(s) <NUM>, and/or an operating system of the user device <NUM>, which are accessible to or executable by processor(s) <NUM> to enable audio or video conferencing and/or other user interaction with the user device <NUM>. The user interface <NUM> can be configured to receive inputs from a user of the user device <NUM>, such as to receive input from a user that may define and/or facilitate one or more aspects of adaptive energy limiting for transient noise suppression. The user interface <NUM> can include a graphical user interface (GUI) that receives the input information via a touch input. In other instances, the user interface <NUM> includes an intelligent assistant that receives the input information via an audible input. Alternately or additionally, the operating system of the user device <NUM> may be maintained as firmware or an application on the CRM <NUM> and executed by the processor(s) <NUM>.

The CRM <NUM> also includes an adaptive energy limiter <NUM>, neural network <NUM>, and voice activity detector <NUM>. In various aspects, the adaptive energy limiter <NUM> utilizes the neural network <NUM> and/or voice activity detector <NUM> (VAD <NUM>) to determine whether an audio signal comprises speech or noise. Based on this determination, the adaptive energy limiter <NUM> may decrease a limiter ceiling to limit energy of noise that would otherwise disrupt a conference call or voice call if allowed to pass through at full energy. The implementations and uses of the adaptive energy limiter <NUM>, neural network <NUM>, and/or voice activity detector <NUM> vary and are described throughout the disclosure.

Aspects and functionalities of the user device <NUM> may be managed via operating system controls presented through at least one application programming interface <NUM> (API <NUM>). In some aspects, the adaptive energy limiter <NUM> or an application of the user device <NUM> accesses an API <NUM> or an API service of the user device <NUM> to control aspects and functionalities of audio or video conference applications. For example, the adaptive energy limiter <NUM> may access low-level audio processor settings of the user device <NUM> to implement aspects of adaptive energy limiting, such as to set a minimum limiter ceiling level, adjust audio gain setting, manage respective signal levels of incoming and outgoing audio signals, or the like. The CRM <NUM> of the user device <NUM> may also include a user device manager <NUM>, which can be implemented in whole or part as hardware logic or circuitry integrated with or separate from other components of the user device <NUM>. In at least some aspects, the user device manager <NUM> configures the microphone <NUM> and other audio circuitry of the user device <NUM> to implement the techniques for transient noise suppression as described herein.

The user device <NUM> also includes a display <NUM> for displaying and/or providing information or a video feed to a user. For example, through the display <NUM>, the user device <NUM> may provide the user with a video feed from a video conference enabled by the conference service <NUM>. Alternately or additionally, the user device <NUM> may also include a camera (not shown) to enable generation of a video feed from the user device <NUM> for multimedia conferencing.

The conference device <NUM> may be implemented as a computing device, server, cloud-based hardware, or other resources through which the conference service <NUM> is provided to the user devices <NUM>. Generally, the conference device <NUM> may serve as a collector and/or arbiter of multimedia data or streams for an instance of a conference call. As such, the conference device <NUM> may implement aspects of adaptive energy limiting with respect to inbound audio data received from user devices <NUM>, internal multimedia processing operations, or outbound audio data transmitted to the user devices <NUM> as part of a conference or voice call.

In this example, the conference device <NUM> includes network interfaces <NUM> for exchanging data, such as audio signals or video streams, over various types of networks or communication protocols. Generally, the network interfaces <NUM> can be implemented as any one or more of a serial and/or parallel interface, a wireless interface, a wired interface, or a modem for transmitting or receiving data or signals. In some cases, the network interfaces <NUM> provide a connection and/or communication link between the conference device <NUM> and a communication network by which the user devices <NUM> communicate audio signals <NUM>, video data, or the like to for conferenced media communication.

In this example, the conference device <NUM> also includes processor(s) <NUM>, or compute resources, and computer-readable storage media <NUM> (CRM <NUM>). The computer-readable storage media <NUM> is configured as storage, and thus does not include transitory signals or carrier waves. The CRM <NUM> may include any suitable memory or storage device such as RAM, SRAM, DRAM, NVRAM, ROM, or Flash memory useable to store multimedia data <NUM> of the conference device <NUM>.

The multimedia data <NUM> of the conference device <NUM> may include audio data, audio signals, or video data useful to facilitate conference calls through an instance of the conference service <NUM>. The multimedia data <NUM> and conference service <NUM>, as well as other applications (e.g., media conference server applications) and/or an operating system of the conference device <NUM> may be accessible to or executable by processor(s) <NUM> to enable audio or video conferencing for multiple user devices <NUM>.

In this example, the CRM <NUM> also includes an instance of the adaptive energy limiter <NUM>, neural network <NUM>, and voice activity detector <NUM>. As noted, aspects of adaptive energy limiting may be implemented by a user device <NUM>, conference device <NUM>, or a combination of both devices. In various aspects, the adaptive energy limiter <NUM> utilizes the neural network <NUM> and/or voice activity detector <NUM> to determine whether one or more audio signals comprise speech or noise. Based on this determination, the adaptive energy limiter <NUM> of the conference device <NUM> may decrease a limiter ceiling for a respective audio signal or audio feed to limit energy of noise that would otherwise disrupt a conference call or voice call if allowed to pass through at full energy. The implementations and uses of the adaptive energy limiter <NUM>, neural network <NUM>, and/or voice activity detector <NUM> vary and are described throughout the disclosure.

Aspects and functionalities of the conference device <NUM> may be managed via system controls presented through at least one application programming interface (API) of an API library <NUM>. In some aspects, the adaptive energy limiter <NUM> or an application of the conference device <NUM> accesses an API or library of the API library <NUM> to implement aspects of transient noise limiting. For example, the adaptive energy limiter <NUM> may be implemented as part of or in conjunction with a web-based real-time communications library.

<FIG> illustrates at <NUM> an example configuration of components that are capable of implementing various aspects of adaptive energy limiting. Generally, the components of <FIG> may be embodied on a user device <NUM>, a conference device <NUM>, or a combination thereof. In some aspects, the components shown at <NUM> are implemented as an integrated component (e.g., system-on-chip) of one device and/or in combination with a memory storing processor-executable instructions to provide respective functionalities of one or more components. As such, the configuration of components shown in <FIG> is non-limiting and may be implemented on any suitable device, combination of devices, and/or as hardware (e.g., logic circuitry) combined with firmware or software to provide the described functionalities.

In some aspects, an audio signal <NUM> is sliced or partitioned into audio frames <NUM> that correspond to respective portions of the audio signal. For example, each of the audio frames <NUM> may correspond to a portion, segment, or duration of audio (e.g., speech and/or noise) of the audio signal <NUM>. In some cases, an audio frame <NUM> or frame of audio corresponds to a range of approximately five milliseconds to <NUM> milliseconds of audio (e.g., <NUM> millisecond of audio). Alternately or additionally, the audio frames <NUM> may be converted from a time domain to a frequency domain, such as to enable spectral analysis or other frequency domain-based processing.

As shown in <FIG>, the example components include an amplitude detector <NUM> and a neural network <NUM>, which includes or provides a voice activity detector <NUM> for processing the audio frames <NUM>. Generally, the amplitude detector <NUM> measures or determines an amplitude of the audio signal <NUM> that corresponds to an audio frame. For example, the amplitude detector <NUM> may generate or provide an indication of a maximum amplitude <NUM> for a frame of audio or portion of audio signal. In some aspects, the adaptive energy limiter <NUM> determines or updates an average amplitude <NUM> (e.g., moving average) for the audio signal <NUM> or audio frames <NUM> based on multiple maximum amplitudes <NUM>.

The neural network <NUM> may be implemented as a network that operates on a processor of a user device <NUM> to provide voice likelihood estimates for the audio frames <NUM>. Alternately or additionally, the neural network <NUM> may be implemented as a recurrent neural network (RNN) or machine-learned model with a memory (e.g., RNNoise). In some aspects, the voice activity detector <NUM> provides, for one or more of the audio frames, an instantaneous voice likelihood <NUM> (IVL <NUM>). Although described as a neural network-enabled voice activity detector, other types of voice activity detection or voice classification may be used.

For example, the neural network <NUM> and/or voice activity detector <NUM> may be implemented as a neural network (e.g., deep neural network (DNN)) comprising an input layer, an output layer, and one or more hidden intermediate layers positioned between the input layer and the output layer of the neural network. Any or all nodes of the neural network may be in turn fully connected or partially connected between the layers of the neural network. A voice activity detector <NUM> may be implemented with or through any type of neural network, such as a convolutional neural network (CNN) including GoogleNet or similar convolutional networks. Alternately or additionally, a voice activity detector <NUM> or machine-learned voice activity detection model may include any suitable recurrent neural network (RNN) or any variation thereof. Generally, the neural network <NUM> and/or voice activity detector <NUM> employed by the adaptive energy limiter may also include any other supervised learning, unsupervised learning, reinforcement learning algorithm, or the like.

In various aspects, a neural network <NUM> and/or voice activity detector <NUM> associated with the adaptive energy limiter <NUM> may be implemented as a recurrent neural network (RNN) with connections between nodes that form a cycle to retain information from a previous portion of an input data sequence for a subsequent portion of the input data sequence (e.g., previous audio frames of speech or noise generated by a participant). In other cases, a neural network <NUM> is implemented as a feed-forward neural network having connections between the nodes that do not form a cycle between input data sequences. Alternately, a neural network <NUM> may be implemented as a convolutional neural network (CNN) with multilayer perceptrons where each neuron in a particular layer is connected with all neurons of an adjacent layer. In various aspects of adaptive energy limiting, the neural network <NUM> and/or voice activity detector <NUM> may use previous determinations of noise or speech by a participant to predict or determine whether subsequent frames of an audio signal include speech or noise that may be suppressed.

Generally, the neural network <NUM> may enable the determination of voice likelihood estimations that quickly converge to high statistical confidence, particularly in the presence of vowel sounds. By way of review, transient noise often has more full-band energy than vowels, and even more so than consonants in speech. Thus, in utilizing a statistical confidence provided by the neural network <NUM>, the adaptive energy limiter is able to leverage historical noise or speech patterns of a participant to distinguish between noise, vowels, and consonants of speech. In other words, speech and noise tend to come in bursts, that is, a participant that has recently spoken is more likely to continue speaking in the near future (e.g., sub-second). Alternately, a participant that produced noise in the recent past is more likely to generate additional noise in the future. In some cases, any lag introduced by the adaptive energy limiter is imperceptible to conference call participants, yet the neural network <NUM> is able to more accurately determine in retrospect (e.g., a few <NUM> milliseconds) whether audio of the frame or signal is noise or speech, than to such in real-time.

Based on one or more of the instantaneous voice likelihoods <NUM>, the adaptive energy limiter <NUM> may determine an aggregate speech likelihood estimate <NUM> (ASLE <NUM>) for the audio signal <NUM> or audio <NUM>. The aggregate speech likelihood estimate <NUM> may be configured or updated based on a current aggregate speech likelihood estimate <NUM> and/or a threshold for detection of voice or noise. For example, in some cases, the adaptive energy limiter <NUM> increases the aggregate speech likelihood estimate <NUM> in response to an instantaneous voice likelihood <NUM> exceeding the current aggregate speech likelihood estimate <NUM>, as well as exceeding a threshold for voice detection. In other cases, the adaptive energy limiter <NUM> may decrease the aggregate speech likelihood estimate <NUM> in response to an instantaneous voice likelihood <NUM> not exceeding the current aggregate speech likelihood estimate <NUM> or not exceeding a threshold for voice detection.

The adaptive energy limiter <NUM> also includes or provides a limiter ceiling <NUM> by which energy of the audio signal <NUM> may be limited, such as to suppress energy of transient noise. Generally, the limiter ceiling <NUM> is provided to an audio signal limiter module <NUM> through which the audio signal <NUM> passes before transmission to other audio components or processes. The audio signal limiter module <NUM> may pass audio signal through at full scale (e.g., unreduced or not limited) or a reduced scale or reduced amplitude as specified by the limiter ceiling <NUM> set by the adaptive energy limiter <NUM>. In the context of <FIG>, based on the limiter ceiling <NUM> provided by the adaptive energy limiter <NUM>, the audio signal limiter module <NUM> limits or decreases energy of the audio signal <NUM> to provide or generate an energy-limited audio signal <NUM>. In various aspects, the adaptive energy limiter <NUM> limits the energy of an audio signal determined to be, or include, noise in order to suppress the noise and likely future noise. The energy-limited audio signal <NUM> may then be transmitted to audio-based processing <NUM> for subsequent processing or use for other features (e.g., speaker selection), before being included in conference audio <NUM>, which is shared with other participants of an audio or video conference call.

Example methods <NUM> and <NUM> are described with reference to <FIG>, <FIG>, and <FIG> in accordance with one or more aspects of adaptive energy limiting for transient noise suppression. Generally, methods <NUM> and <NUM> illustrate sets of operations (or acts) performed in, but not necessarily limited to, the order or combinations in which the operations are shown herein. Further, any of one or more of the operations may be repeated, combined, reorganized, skipped, or linked to provide a wide array of additional and/or alternate methods. In portions of the following discussion, reference may be made to example conference environment <NUM> of <FIG>, example devices of <FIG>, example components of <FIG>, example systems of <FIG>, and/or entities detailed in <FIG>, reference to which is made for example only. The techniques and apparatuses described in this disclosure are not limited to embodiment or performance by one entity or multiple entities operating on one device.

Method <NUM> is a method performed by a user device <NUM> or conference device <NUM>. The method <NUM> limits an amount of energy of an audio signal to mitigate effects associated with transient noise in conference environments or other audio processes (e.g., speaker selection for video conferencing). In some aspects, operations of the method <NUM> are implemented by or with an adaptive limiter <NUM>, neural network <NUM>, and/or voice activity detector <NUM> of the user device <NUM> or conference device <NUM>.

At <NUM>, a limiter ceiling for an audio signal is set to full scale. In some cases, the limiter ceiling or limiting value is set to full scale on initialization of the adaptive energy limiter or in response to speech by a participant for which an audio signal is being processed for noise suppression.

At <NUM>, a portion of the audio signal is received. The portion of the audio signal may include a frame of audio, audio frame, segment of the audio signal, or the like. In some cases, the audio signal is received and separated into frames of audio for analysis by the adaptive energy limiter. For example, a frame of the audio may correspond to a range of approximately five milliseconds to <NUM> milliseconds of audio. Alternately or additionally, the frame of audio can be converted from a time domain to a frequency domain to enable spectral analysis or other frequency domain-based processing.

At <NUM>, a maximum amplitude of the portion of the audio signal is determined. The maximum amplitude may be determined for the portion of audio signal that corresponds to a frame of audio or a duration of audio (e.g., <NUM> milliseconds). In some cases, the maximum amplitude of the audio signal is compared to a threshold to determine if a participant is silent, quiet, or otherwise not generating noise. Optionally, from operation <NUM>, the method <NUM> may return to operation <NUM> if the audio signal is quiet or silent. By so doing, energy of the silent participant's speech will not be reduced if and when the participant begins to speak.

At <NUM>, the portion of the audio signal is evaluated with a neural network to provide a voice likelihood estimate. In some aspects, the portion of the audio signal or a frame of audio is evaluated with the neural network or a neural network-enabled voice activity detector to provide an instantaneous voice likelihood for the portion of the audio signal or the audio frame. Generally, the instantaneous voice likelihood may indicate if the audio stream is more likely speech or more likely noise, which the adaptive energy limiter would suppress.

At <NUM>, a determination is made, based on the maximum amplitude and the voice likelihood estimate, as to whether the portion of the audio signal includes speech or noise. For example, if the maximum amplitude of the portion of the audio signal exceeds a moving average of the maximum amplitude (e.g., maximum average plus a small modifier) and the instantaneous voice likelihood is less than <NUM> or <NUM>% (indicating noise), it may be determined that the portion of audio includes or is noise. Alternately, if the maximum amplitude of the portion of the audio signal does not exceed the moving average of the maximum amplitude (e.g., maximum average plus a small modifier) or the instantaneous voice likelihood is greater than <NUM> or <NUM>%, it may be determined that the portion of audio is not noise or is speech (e.g., maximum average exceeded and IVL greater than <NUM>%). Optionally, from operation <NUM>, the method <NUM> may return to operation <NUM> if it is determined that the portion of the audio signal is or includes speech of the participant.

At <NUM>, the limiter ceiling for the audio signal is decreased in response to determining that the portion of the audio signal includes noise. In some aspects, the limiter ceiling is decreased by a specific rate or amount based on an aggregate speech likelihood estimate. For example, if the aggregate speech likelihood estimate is high, the ceiling limit is decreased by a small amount or slowly toward a minimum limiter ceiling value. In other cases, when the aggregate speech likelihood estimate is low, the ceiling limit may be decreased by a large amount or quickly toward the minimum limiter ceiling value. Alternately or additionally, the minimum limiter ceiling can be configured based on the aggregate speech likelihood estimate, an average of respective amplitudes of multiple portions of the audio signal, or an average of respective maximum amplitudes of multiple portions of the audio signal, such as to represent a portion of current energy estimated to be speech.

At <NUM>, the limiter ceiling is provided to a limiter module through which the audio signal passes. The limiter module limits, based on the limiter ceiling, the amount of energy of the audio signal. By limiting the energy that the audio signal is allowed to transmit or carry into a conferenced audio environment, aspects of adaptive energy limiting may prevent full energy transient noise from entering the conference audio and disrupting participants and/or other audio-based processes.

Method <NUM> of <FIG> and <FIG> is a method performed by a user device <NUM> or a conference device <NUM>. The method <NUM> scales an audio signal to not exceed a limiter ceiling, which may be effective to prevent the audio signal from carrying full energy transient noise into a conferenced audio environment. In some aspects, operations of the method <NUM> are implemented by or with an adaptive limiter <NUM>, neural network <NUM>, and/or voice activity detector <NUM> of the user device <NUM> or conference device <NUM>.

At <NUM>, a limiter ceiling for an audio signal is set to full scale (e.g., <NUM> or <NUM>%). The limiter ceiling or limiting value may be set to full scale on initialization of the adaptive energy limiter or reset to full scale in response to speech by a participant for which an audio signal is being processed for noise suppression.

At <NUM>, a frame of audio is generated that corresponds to a portion of the audio signal. In some cases, the audio signal is received and/or separated, sliced, or otherwise partitioned into frames of audio for analysis by the voice activity detector and/or adaptive energy limiter. In other cases, the audio frame may be received from an audio codec or other entity configured to provide frames from the audio signal. For example, a frame of the audio may correspond to a range of approximately five milliseconds to <NUM> milliseconds of audio (e.g., <NUM> milliseconds). Alternately or additionally, the frame of audio can be converted from a time domain to a frequency domain to enable spectral analysis or other frequency domain-based processing.

At <NUM>, the frame of audio is evaluated with a neural network-enabled voice activity detector to provide an instantaneous voice likelihood (IVL). In some aspects, the portion of the audio signal or a frame of audio is evaluated with the neural network or a neural network-enabled voice activity detector to provide an instantaneous voice likelihood for the portion of the audio signal or the audio frame. Generally, the instantaneous voice likelihood may indicate if the audio stream is more likely speech or more likely noise, which the adaptive energy limiter would suppress.

At <NUM>, a maximum amplitude of the audio signal is recorded from the frame of audio. The maximum amplitude may be determined or recorded for a duration of audio signal that corresponds to a frame of audio or a duration of audio (e.g., <NUM> milliseconds). In some cases, the maximum amplitude of the audio signal is compared to a threshold to determine if a participant is silent, quiet, or otherwise not generating noise. In such cases, the method <NUM> may return to operation <NUM> if the audio signal is quiet or silent.

At <NUM>, a moving average of maximum amplitudes for the audio signal is updated based on the recorded maximum amplitude for the frame of audio. The moving average of maximum amplitudes may correspond to any suitable number of audio frames or duration of audio, such as a range of approximately <NUM> milliseconds to <NUM> milliseconds.

As shown at <NUM> in <FIG>, operation <NUM> determines an aggregate speech likelihood estimate (ASLE) based on the instantaneous voice likelihood (IVL) of the frame of audio. The aggregate speech likelihood estimate may be determined or configured based on a current aggregate speech likelihood estimate and/or a threshold for detection of voice (or noise). In some cases, the aggregate speech likelihood estimate is increased in response to an instantaneous voice likelihood that exceeds the current aggregate speech likelihood estimate and the threshold for voice detection. In other cases, the aggregate speech likelihood estimate is decreased in response to an instantaneous voice likelihood that does not exceed the current aggregate speech likelihood estimate or the threshold for voice detection.

At <NUM>, a determination is made as to whether the maximum amplitude exceeds the moving average and the instantaneous voice likelihood indicates the frame of audio is noise. For example, if the maximum amplitude of the portion of the audio signal exceeds the moving average of the maximum amplitude (e.g., maximum average plus a small modifier) and the instantaneous voice likelihood is less than <NUM> or <NUM>% (indicating noise), the audio frame may include or be noise. Alternately, if the maximum amplitude of the portion of the audio signal does not exceed the moving average of the maximum amplitude (e.g., maximum average plus a small modifier) or the instantaneous voice likelihood is greater than <NUM> or <NUM>%, the audio frame may not include or be predominately noise.

Optionally at <NUM>, the limiter ceiling is not decreased in response to the maximum amplitude not exceeding the moving average and/or the instantaneous voice likelihood not indicating that the frame of audio is noise. Optionally at <NUM>, the limiter ceiling is decreased based on the aggregate speech likelihood estimate (ASLE). The limiter ceiling is decreased in response to the maximum amplitude exceeding the moving average and the IVL indicating that the frame of audio is noise. Generally, an amount by which or a rate at which the limiter ceiling is decreased is determined based on the aggregate speech likelihood estimate.

At <NUM>, a current value of the limiter ceiling is provided to a limiter module to scale the audio signal to not exceed the current value. The limiter module scales, based on the limiter ceiling, the amount of energy of the audio signal that passes through the limiter module. By scaling or limiting the energy that the audio signal is allowed to transmit or carry into a conferenced audio environment, aspects of adaptive energy limiting may prevent full energy transient noise from entering the conference audio and disrupting participants and/or other audio-based processes. From operation <NUM>, the method <NUM> may return to operation <NUM> to perform another iteration of the method <NUM> to further limit energy of the audio signal, reset the limiter ceiling, or maintain a current limiter ceiling. In some aspects, the method <NUM> or process for adaptive energy limiting is iterated or repeated approximately every five milliseconds to <NUM> milliseconds (e.g., <NUM> milliseconds) to provide responsive suppression of transient noise.

By way of example, consider <FIG> in which a graph <NUM> illustrates aspects of adaptive energy limiting. In the context of a limiter module, energy of an audio signal is passed at full scale <NUM> or limited to a minimum <NUM> of the limiter ceiling. In this example, assume the audio signal <NUM> is received from a participant that is constantly generating noise at a medium to high level (without speech). Here, the adaptive energy limiter <NUM> may quickly limit the energy of the audio signal that passes to the conference audio environment to prevent the noise of audio signal <NUM> from disrupting other participants of the conference call.

As another example, consider graph <NUM>, which includes an audio signal <NUM> of another participant of the conference call. Here, assume that the participant is not speaking, but also not making much noise. The adaptive energy limiter <NUM> gradually limits the audio signal <NUM> until the participant begins speaking at <NUM>. In response to detecting speech, the adaptive energy limiter <NUM> resets the limiter ceiling to full scale <NUM> at <NUM> and does not begin to limit energy of the audio signal <NUM> until the participant ceases to speak at <NUM>.

<FIG> illustrates various components of an example system <NUM> that can be implemented as any type of user device <NUM> or conference device <NUM> as described with reference to <FIG> to implement adaptive energy limiting for transient noise suppression. In some aspects, the system <NUM> is implemented as a component of or embodied on a user equipment device or base station. For example, the system <NUM> may be implemented as a system of hardware-based components, such as, and without limitation, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-chip (SoC), a system-in-package, a complex programmable logic device (CPLD), audio codec, audio processor, co-processor, context hub, communication co-processor, sensor co-processor, or the like.

The system <NUM> includes communication devices <NUM> that enable wired and/or wireless communication of system data <NUM> (e.g., encoded audio data or audio signals). The system data <NUM> or other system content can include configuration settings of the system, media content stored on the device, and/or information associated with a user of the device. Media content stored on the system <NUM> may include any type of audio, video, and/or image data. The system <NUM> includes one or more data inputs <NUM> via which any type of data, media content, and/or inputs can be received, such as human utterances, speech, interactions with a radar field, user-selectable inputs (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and/or image data received from any content and/or data source.

The system <NUM> also includes communication interfaces <NUM>, which can be implemented as any one or more of a serial and/or parallel interface, a wireless interface, a network interface, a modem, and as any other type of communication interface. Communication interfaces <NUM> provide a connection and/or communication links between the system <NUM> and a communication network by which other electronic, computing, and communication devices communicate data with the system <NUM>.

The system <NUM> includes one or more processors <NUM> (e.g., any of microprocessors, controllers, and the like), which process various computer-executable instructions to control the operation of the system <NUM> and to enable techniques for, or in which can be embodied, adaptive energy limiting for transient noise suppression. Alternately or additionally, the system <NUM> can be implemented with any one or combination of hardware, firmware, or fixed logic circuitry that is implemented in connection with processing and control circuits, which are generally identified at <NUM>. Although not shown, the system <NUM> can include a system bus or data transfer system that couples the various components within the device. A system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus that utilizes any of a variety of bus architectures.

The system <NUM> also includes computer-readable media <NUM> (CRM <NUM>), such as one or more memory devices that enable persistent and/or non-transitory data storage, and thus do not include transitory signals or carrier waves. Examples of the CRM <NUM> include random access memory (RAM), non-volatile memory (e.g., any one or more of a read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), or a disk storage device. A disk storage device may be implemented as a magnetic or an optical storage device, such as a hard disk drive, a recordable and/or rewriteable compact disc (CD), any type of a digital versatile disc (DVD), and the like. The system <NUM> can also include a mass storage media device (storage media) <NUM> or mass storage device interface. In this example, the system <NUM> also includes, or may be implemented as, an audio codec <NUM> to support the coding or decoding of audio signals or audio data, such as to encode audio from a microphone to provide audio signals or audio data for a conference service or voice call.

The computer-readable media <NUM> provides data storage mechanisms to store the device data, as well as various system applications <NUM> and any other types of information and/or data related to operational aspects of the system <NUM>. For example, an operating system <NUM> can be maintained as a computer application with the computer-readable media <NUM>, executed on the processors <NUM>. The system applications <NUM> may include a system manager, such as any form of a control application, software application, signal-processing and control module, code that is native to a particular device, an abstraction module or gesture module and so on. The system applications <NUM> also include system components and utilities to implement adaptive energy limiting for transient noise suppression, such as the adaptive limiter <NUM>, neural network <NUM>, and voice activity detector <NUM>. While not shown, one or more elements of the adaptive limiter <NUM>, neural network <NUM>, or voice activity detector <NUM> may be implemented, in whole or in part, through hardware or firmware.

According to an example, the present disclosure describes aspects of adaptive energy limiting for transient noise suppression. In some aspects, an adaptive energy limiter sets a limiter ceiling for an audio signal to full scale and receives a portion of the audio signal. For the portion of the audio signal, the adaptive energy limiter determines a maximum amplitude and evaluates the portion with a neural network to provide a voice likelihood estimate. Based on the maximum amplitude and the voice likelihood estimate, the adaptive energy limiter determines that the portion of the audio signal includes noise. In response to determining that the portion of the audio signal includes noise, the adaptive energy limiter decreases the limiter ceiling and provides the limiter ceiling to a limiter module effective to limit an amount of energy of the audio signal. This may be effective to prevent audio signals from carrying full energy transient noise into conference audio.

Although the above-described devices, systems, and methods are described in the context of adaptive energy limiting for transient noise suppression in an audio/video conference environment, the described devices, systems, or methods are non-limiting and may apply to other contexts, user equipment deployments, or audio-based communication environments.

Claim 1:
A method comprising:
setting (<NUM>) a ceiling level (<NUM>) of a limiter module (<NUM>) through which audio signals (<NUM>) pass to a full-scale level;
receiving (<NUM>), via a data interface, data that comprises a portion of an audio signal;
determining (<NUM>) a maximum amplitude of the portion of the audio signal;
evaluating (<NUM>) the portion of the audio signal with a machine learning algorithm (<NUM>) to provide a voice likelihood (<NUM>) estimate for the portion of the audio signal;
determining (<NUM>), based on the maximum amplitude (<NUM>) and the voice likelihood (<NUM>) estimate, that the portion of the audio signal includes noise;
decreasing (<NUM>) the ceiling level (<NUM>) of the limiter module (<NUM>) from the full-scale level to a decreased ceiling level in response to determining that the portion of the audio signal includes noise; and
providing (<NUM>), to the limiter module (<NUM>) through which the audio signals pass, the decreased ceiling level to limit an amount of energy of the audio signal.