Patent Description:
This disclosure relates generally to audio signal processing, and more particularly to processing user-generated content (UGC).

UGC is typically created by consumers and can include any form of content (e.g., images, videos, text, audio). UGC is typically posted by its creator to online platforms, including but not limited to social media, blogs, wikis and the like. One trend related to UGC is personal moment sharing in variable environments (e.g., indoors, outdoors, by the sea) by recording video and audio using a personal mobile device (e.g., smart phone, tablet computer, wearable devices). Most UGC content contains audio artifacts due to consumer hardware limitations and a non-professional recording environment. The traditional way of UGC processing is based on audio signal analysis or artificial intelligence (Al) based noise reduction and enhancement processing. One difficulty in processing UGC is how to treat different sound types in different audio environments while maintaining the creative objective of content creator.

Prior art document <CIT> discloses a method for denoising audio based on detected context using sensors. The processing parameters are selected according to the context and the parameters steer the denoising beam.

In the drawings, specific arrangements or orderings of schematic elements, such as those representing devices, units, instruction blocks and data elements, are shown for ease of description. However, it should be understood by those skilled in the art that the specific ordering or arrangement of the schematic elements in the drawings is not meant to imply that a particular order or sequence of processing, or separation of processes, is required. Further, the inclusion of a schematic element in a drawing is not meant to imply that such element is required in all embodiments or that the features represented by such element may not be included in or combined with other elements in some embodiments.

Further, in the drawings, where connecting elements, such as solid or dashed lines or arrows, are used to illustrate a connection, relationship, or association between or among two or more other schematic elements, the absence of any such connecting elements is not meant to imply that no connection, relationship, or association can exist. In other words, some connections, relationships, or associations between elements are not shown in the drawings so as not to obscure the disclosure. In addition, for ease of illustration, a single connecting element is used to represent multiple connections, relationships or associations between elements. For example, where a connecting element represents a communication of signals, data, or instructions, it should be understood by those skilled in the art that such element represents one or multiple signal paths, as may be needed, to affect the communication.

The same reference symbol used in various drawings indicates like elements.

In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the various described embodiments. It will be apparent to one of ordinary skill in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits, have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. Several features are described hereafter that can each be used independently of one another or with any combination of other features.

As used herein, the term "includes" and its variants are to be read as open-ended terms that mean "includes, but is not limited to. " The term "or" is to be read as "and/or" unless the context clearly indicates otherwise. The term "based on" is to be read as "based at least in part on. " The term "one example embodiment" and "an example embodiment" are to be read as "at least one example embodiment. " The term "another embodiment" is to be read as "at least one other embodiment. " The terms "determined," "determines," or "determining" are to be read as obtaining, receiving, computing, calculating, estimating, predicting or deriving. In addition, in the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.

<FIG> illustrates binaural recording using earbuds and a mobile device, according to an embodiment. System <NUM> includes a two-step process of recording video with a video camera of a mobile device <NUM> (e.g., a smartphone), and concurrently recording audio associated with the video recording. In an embodiment, the audio recording can be made by, for example, mobile device <NUM> recording audio signals output by microphones embedded in earbuds <NUM>. The audio signals can include but are not limited to comments spoken by a user and/or ambient sound. If both the left and right microphones are used then a binaural recording can be captured. In some implementations, microphones embedded or attached to mobile device <NUM> can also be used.

<FIG> is a block diagram of a system <NUM> for context aware audio processing, according to an embodiment. System <NUM> includes window processor <NUM>, spectrum analyzer <NUM>, band feature analyzer <NUM>, gain estimator <NUM>, machine learning model <NUM>, context analyzer <NUM>, gain analyzer/adjuster <NUM>, band gain to bin gain converter <NUM>, spectrum modifier <NUM>, speech reconstructor <NUM> and window overlap-add processor <NUM>.

Window processor <NUM> generates a speech frame comprising overlapping windows of samples of input audio <NUM> containing speech (e.g., an audio recording captured by mobile device <NUM>). The speech frame is input into spectrum analyzer <NUM> which generates frequency bin features and a fundamental frequency (F0). The analyzed spectrum information can be represented by: Fast Fourier transform (FFT) spectrum, Quadrature Mirror Filter (QMF) features or any other audio analysis process. The bins are scaled by spectrum modifier <NUM> and input into speech reconstructor <NUM> which outputs a reconstructed speech frame. The reconstructed speech frame is input into window overlap-add processor <NUM>, which generates output speech.

Referring back to step <NUM> the bin features and F0 are input into band feature analyzer <NUM>, which outputs band features and F0. In an embodiment, the band features are extracted based on FFT parameters. Band features can include but are not limited to: MFCC and BFCC. In an embodiment, a band harmonicity feature can be computed, which indicates how much a current frequency band is composed of a periodic signal. In an embodiment, the harmonicity feature can be calculated based on FFT frequency bins of a current speech frame. In other embodiments, the harmonicity feature is calculated by a correlation between the current speech frame and a previous speech frame.

The band features and F0 are input into gain estimator <NUM> which estimates gains (CGains) for noise reduction based on a model selected from model pool <NUM>. In an embodiment, the model is selected based on a model number output by context analyzer <NUM> in response to input visual information and other sensor information. In an embodiment, the model is a deep neural network (DNN) trained to estimate gains and VAD for each frequency band based on the band features and F0. The DNN model can be based on a fully connected neural network (FCNN), recurrent neural network (RNN) or convolutional neural network (CNN) or any combination of FCNN, RNN and CNN. In an embodiment, a Wiener Filter or other suitable estimator can be combined with the DNN model to get the final estimated gains for noise reduction.

The estimated gains, CGains, are input into gain analyzer/adjuster <NUM> which generates adjusted gains, AGains, based on an audio processing profile. The adjusted gains, AGains, is input into band gain to bin gain converter <NUM>, which generates adjusted bin gains. The adjusted bin gains are input spectrum modifier <NUM> which applies the adjusted bin gains to their corresponding frequency bins (e.g., scales the bin magnitudes by their respective adjusted bin gains). The adjusted bin features are then input into speech reconstructor <NUM>, which outputs a reconstructed speech frame. The reconstructed speech frame is input into window overlap-add processor <NUM>, which generates reconstructed output speech using an overlap and add algorithm.

In an embodiment, the model number is output by context analyzer <NUM> based on input audio <NUM> and input visual information and/or other sensors data <NUM>. Context analyzer <NUM> can include one or more audio scene classifiers trained to classify audio content into one or more classes representing recording locations. In an embodiment, the recording location classes are indoors, outdoors and transportation. For each class, a specific audio processing profile can be assigned. In another embodiment, context analyzer <NUM> is trained to classify a more specific recording location (e.g., sea bay, forest, concert, meeting room, etc.).

In another embodiment, context analyzer <NUM> is trained using visual information, such as digital pictures and video recordings, or a combination of an audio recording and visual information. In other embodiments, other sensor data can be used to determine context, such as inertial sensors (e.g., accelerometers, gyros) or position technologies, such as global navigation satellite systems (GNSS), cellular networks or WIFI fingerprinting. For example, the accelerometer and gyroscope and/or Global Position System (GPS) data can be used to determine a speed of mobile device <NUM>. The speed can be combined with the audio recording and/or visual information to determine whether the mobile device <NUM> is being transported (e.g., in a vehicle, bus, airplane, etc.).

In an embodiment, different models can be trained for different scenarios to achieve better performance. For example, for a sea bay recording location, the model can include the sound of tides. The training data can be adjusted to achieve different model behaviors. When a model is trained, the training data can be separated into two parts: (<NUM>) a target audio database containing signal portions of the input audio to be maintained in the output speech, and (<NUM>) a noise audio database which contains noise portions of the input audio that needs to be suppressed in the output speech. Different training data can be defined to train different models for different recording locations. For example, for the sea bay model, the sound of tides can be added to the target audio database to make sure the model maintains the sound of tides. After defining the specific training database, traditional training procedures can be used to train the models.

In an embodiment, the context information can be mapped to a specific audio processing profile. The specific audio processing profile can include a least a specific mixing ratio for mixing the input audio (e.g., the original audio recording) with the processed audio recording where noise was suppressed. The processed recording is mixed with the original recording to reduce quality degradation of the output speech. The mixing ratio is controlled by context analyzer <NUM> shown in <FIG>. The mixing ratio can be applied to the input audio in the time domain, or the CGains can be adjusted with the mixing ratio according to Equation [<NUM>] below using gain adjuster <NUM>.

Although a DNN based noise reduction algorithm can suppress noise significantly, the noise reduction algorithm may introduce significant artifacts in the output speech. Thus, to reduce the artifacts the processed audio recording is mixed with the original audio recording. In an embodiment, a fixed mixing ratio can be used. For example, the mixing ratio can be <NUM>.

However, a fixed mixing ratio may not work for different contexts. Therefore, in an embodiment the mixing ratio can be adjusted based on the recording context output by context analyzer <NUM>. To achieve this, the context is estimated based on the input audio information. For example, for the indoor class, a larger mixing ratio (e.g., <NUM>) can be used. For the outdoor case, a lower mixing ratio (e.g., <NUM>) can be used. For the transportation class, an even lower mixing ratio can be used (e.g., <NUM>). In an embodiment where a more specific recording location can be determined, a different audio processing profile can be used. For example, for meeting room, a small mixing ratio (e.g., <NUM>), can be used to remove more noise. For a concert, a larger mixing ratio such as <NUM> can be used to avoid degrading the music quality.

In an embodiment, mixing the original audio recording with the processed audio recording can be implemented by mixing the denoised audio file with the original audio file in the time domain. In another embodiment, the mixing can be implemented by adjusting the CGains with the mixing ration dMixRatio, according to Equation [<NUM>]: <MAT> where if AGains > <NUM>, AGains = <NUM>.

In an embodiment, the specific audio processing profile also includes an equalization (EQ) curve and/or a dynamic range control (DRC), which can be applied in post processing. For example, if the recording location is identified as a concert, a music specific equalization curve can be applied to the output of system <NUM> to preserve the timbre of various music instruments, and/or the dynamic range control can be configured to do less compressing to make sure the music level is within a certain loudness range suitable for music. In a speech dominant audio scene, the equalization curve could be configured to enhance speech quality and intelligibility (e.g., boost at <NUM>), and the dynamic range control can be configured to do more compressing to make sure the speech level is within a certain loudness range suitable for speech.

<FIG> is a flow diagram of process <NUM> of context aware audio processing, according to an embodiment. Process <NUM> can be implemented using, for example, device architecture <NUM> described in reference to <FIG>.

Process <NUM> includes the steps of receiving, with one or more sensors of a device, environment information about an audio recording captured by the device (<NUM>), detecting, with at least one processor of the device, a context of the audio recording based on the audio recording and the environment information (<NUM>), determining, with the at least one processor, a model based on the context (<NUM>), processing, with the at least one processor, the audio recording based on the model to produce a processed audio recording with suppressed noise (<NUM>), determining, with the at least one processor, an audio processing profile based on the context (<NUM>), and combining, with the at least one processor, the audio recording and the processed audio recording based on the audio processing profile (<NUM>). Each of these steps were previously described in detail above in reference to <FIG>.

<FIG> shows a block diagram of an example system <NUM> suitable for implementing example embodiments described in reference to <FIG>. System <NUM> includes a central processing unit (CPU) <NUM> which is capable of performing various processes in accordance with a program stored in, for example, a read only memory (ROM) <NUM> or a program loaded from, for example, a storage unit <NUM> to a random access memory (RAM) <NUM>. In the RAM <NUM>, the data required when the CPU <NUM> performs the various processes is also stored, as required. The CPU <NUM>, the ROM <NUM> and the RAM <NUM> are connected to one another via a bus <NUM>. An input/output (I/O) interface <NUM> is also connected to the bus <NUM>.

The following components are connected to the I/O interface <NUM>: an input unit <NUM>, that may include a keyboard, a mouse, or the like; an output unit <NUM> that may include a display such as a liquid crystal display (LCD) and one or more speakers; the storage unit <NUM> including a hard disk, or another suitable storage device; and a communication unit <NUM> including a network interface card such as a network card (e.g., wired or wireless).

In some embodiments, the input unit <NUM> includes one or more microphones in different positions (depending on the host device) enabling capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

In some embodiments, the output unit <NUM> include systems with various number of speakers. The output unit <NUM> can render audio signals in various formats (e.g., mono, stereo, immersive, binaural, and other suitable formats).

The communication unit <NUM> is configured to communicate with other devices (e.g., via a network). A drive <NUM> is also connected to the I/O interface <NUM>, as required. A removable medium <NUM>, such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive or another suitable removable medium is mounted on the drive <NUM>, so that a computer program read therefrom is installed into the storage unit <NUM>, as required. A person skilled in the art would understand that although the system <NUM> is described as including the above-described components, in real applications, it is possible to add, remove, and/or replace some of these components and all these modifications or alteration all fall within the scope of the present disclosure.

In accordance with example embodiments of the present disclosure, the processes described above may be implemented as computer software programs or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program including program code for performing methods. In such embodiments, the computer program may be downloaded and mounted from the network via the communication unit <NUM>, and/or installed from the removable medium <NUM>, as shown in <FIG>.

Generally, various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits (e.g., control circuitry), software, logic or any combination thereof. For example, the units discussed above can be executed by control circuitry (e.g., a CPU in combination with other components of <FIG>), thus, the control circuitry may be performing the actions described in this disclosure. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device (e.g., control circuitry). While various aspects of the example embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representation, it will be appreciated that the blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

Additionally, various blocks shown in the flowcharts may be viewed as method steps, and/or as operations that result from operation of computer program code, and/or as a plurality of coupled logic circuit elements constructed to carry out the associated function(s). For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine readable medium, the computer program containing program codes configured to carry out the methods as described above.

In the context of the disclosure, a machine readable medium may be any tangible medium that may contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine readable medium may be a machine readable signal medium or a machine readable storage medium. A machine readable medium may be non-transitory and may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Computer program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus that has control circuitry, such that the program codes, when executed by the processor of the computer or other programmable data processing apparatus, cause the functions/operations specified in the flowcharts and/or block diagrams to be implemented. The program code may execute entirely on a computer, partly on the computer, as a stand-alone software package, partly on the computer and partly on a remote computer or entirely on the remote computer or server or distributed over one or more remote computers and/or servers.

Claim 1:
An audio processing method, comprising:
receiving, with one or more sensors of a device, environment information about an audio recording captured by the device;
detecting, with at least one processor of the device, a context of the audio recording based on the audio recording and the environment information ;
determining, with the at least one processor, a model based on the context;
processing, with the at least one processor, the audio recording based on the model to produce a processed audio recording with suppressed noise;
characterised in that:
determining, with the at least one processor, an audio processing profile based on the context, wherein the audio processing profile includes at least a mixing ratio for mixing the audio recording with the processed audio recording and wherein the mixing ratio is controlled at least in part based on the context; and
combining, with the at least one processor, the audio recording and the processed audio recording based on the mixing ratio.