Patent Description:
An audio system is typically regarded as "high quality" if the ratio between the input signal and the added audio artefacts, which are a by-product of the system itself, is kept to a minimum. Such artefacts can be divided into noise, non-harmonic distortion and harmonic distortion. Sensing and quantifying such artefacts is needed both for designing better systems and for providing real-time control of automatic-tuning systems.

Techniques for sensing of distortion in audio signals have been previously proposed in the patent literature. For example, <CIT> describes systems and methods that provide distortion sensing, prevention, and/or distortion-aware bass enhancement in audio systems, that can be implemented in a variety of applications. Sensing circuitry can generate statistics based on an input signal received for which an acoustic output is generated. In some embodiments, the sensing circuitry is operable to compute a soft indicator corresponding to a likelihood of distortion or a degree of objectionable, perceptible, or measurable distortion, at an output of the speaker using a technique selected from a group including machine learning, statistical learning, predictive learning, or artificial intelligence.

An embodiment of the present invention that is described hereinafter provides a system including a memory and a processor. The memory is configured to store a machine learning (ML) model. The processor is configured to (i) obtain a set of training audio signals that are labeled with respective levels of distortion, (ii) convert the training audio signals into respective images, (iii) train the ML model to estimate the levels of the distortion based on the images, (iv) receive an input audio signal, (v) convert the input audio signal into an image, and (vi) estimate a level of the distortion in the input audio signal, by applying the trained ML model to the image.

In some embodiments, the distortion includes a Total Harmonic Distortion (THD).

In the invention, the processor is configured to convert a given training audio signal into a given image by setting pixel values of the given image to represent an amplitude of the given training audio signal as a function of time.

For example, the respective images and the image are two-dimensional (2D).

In another example, the respective images and the image are of three or more dimensions.

In an embodiment, the processor is configured to obtain the training audio signals by (i) receiving initial audio signals having first durations, and (ii) slicing the initial audio signals into slices having second, shorter durations, so as to produce the training audio signals.

In some embodiments, the ML model includes a convolutional neural network (CNN).

In some embodiments, the ML model includes a generative adversary network (GAN).

In an embodiment, the input audio signal is received from nonlinear audio processing circuitry.

In an embodiment, the ML model classifies the distortion according to the levels of distortion that label the training audio signal.

In another embodiment, the ML model estimates the level of distortion using regression.

In some embodiments, the processor is further configured to control, using the estimated level of the distortion, an audio system that produces the input audio signal.

There is additionally provided, in accordance with another embodiment of the present invention, a system including a memory and a processor. The memory is configured to store a machine learning (ML) model. The processor is configured to (i) obtain a plurality of initial audio signals, which have first durations in a first range of durations and which are labeled with respective levels of distortion, (ii) slice the initial audio signals into slices having second durations in a second range of durations, shorter than the first durations, so as to produce a set of training audio signals, (iii) train the ML model to estimate the levels of the distortion based on the training audio signals, (iv) receive an input audio signal having a duration in the second range of durations, and (v) estimate a level of the distortion in the input audio signal by applying the trained ML model to the input audio signal.

In some embodiments, the processor is configured to train the ML model by (i) converting the training audio signals into respective images and (ii) training the ML model to estimate the levels of the distortion based on the images.

In some embodiments, the processor is configured to estimate the level of the distortion in the input audio signal by (i) converting the input audio signal into an image and (ii) applying the trained ML model to the image.

In some embodiments, the respective images are two-dimensional (2D) images.

In some embodiments, the respective images are of three or more dimensionals.

There is further provided, in accordance with another embodiment of the present invention, a method including obtaining a set of training audio signals that are labeled with respective levels of distortion. The training audio signals are converted into respective two-dimensional (2D) images. A machine learning (ML) model is trained to estimate the levels of the distortion based on the 2D images. An input audio signal is received. The input audio signal is converted into a 2D image. A level of the distortion in the input audio signal is estimated by applying the trained ML model to the 2D image.

There is furthermore provided, in accordance with another embodiment of the present invention, a method corresponding to the system embodiments.

Audio (e.g., music or voice) is primarily a form of acoustic energy spread over a continuous or discrete range of frequencies. One technique to characterize the audio quality of an audio device is to measure the Total Harmonic Distortion (THD) that the device introduces into an input audio signal. THD is a measure of the harmonic distortion present in a signal, and is defined as the ratio of the sum of the powers of all harmonic components to the power of a fundamental frequency, the fundamental frequency being a sinewave.

When the main performance criterion is the "purity" of the original sine wave (in other words, the contribution of the original frequency with respect to its harmonics), the measurement is most commonly defined as the ratio of the RMS amplitude, A , of a set of higher harmonic frequencies to the RMS amplitude of the first harmonic, or fundamental, frequency: <MAT>.

In audio systems, a lower THD (i.e., lower distortion) means that audio components such as a loudspeaker, an amplifier, a signal processing unit, a microphone or other audio equipment, produce a more accurate reproduction of the original input audio.

The distortion of a waveform relative to a pure sinewave, for example, can be measured either by using a THD analyzer to analyze the output wave into its constituent harmonics and noting the amplitude of each harmonic relative to the fundamental, or by cancelling out the fundamental with a notch filter and measuring the remaining signal, which will be a total aggregate harmonic distortion plus noise.

Given a sine wave generator of very low inherent distortion, the generator's output can be used as an input to amplification equipment, whose distortion at different frequencies and signal levels can be measured by examining the output waveform. While dedicated electronic equipment can be used to both generate sinewaves and to measure distortion, a general-purpose digital computer equipped with a sound card and suitable software can carry out harmonic analysis.

Identifying various different frequencies from an incoming time-domain signal is typically done using a Fourier transform, which is based on mathematical integration. This process requires a signal with a minimal time duration to achieve a specific spectral resolution required of the measurement. Therefore, THD can only be well defined for a sufficient number of cycles of an incoming time-domain signal. For example, to measure a low frequency sine wave (e.g., a bass monotone at <NUM> and corresponding cycle of <NUM> mSec), the incoming time-domain signal must be stable over at least several hundred milliseconds (i.e., at least over several tens of cycles).

This means that THD cannot be estimated for an "instantaneous" audio signal, such as an audio performance during a sound-dominant portion of a beat of a drum that, typically, lasts a few tens of milliseconds at most. The human ear, on the other hand, can recognize distortion of such a drum beat.

In particular, the absence of a THD measurement precludes (a) using the measure to design a more linear system (when the distortion is unintentional), and (b) using the measure, including in real time, to control (e.g., limit) an amount of intentional distortion, such as that introduced by a non-linear audio processing element.

Embodiments of the present invention that are described herein provide systems and methods that define and estimate a level of the distortion in an audio signal, by applying a machine learning (ML) model (e.g., an artificial neural network (ANN)) and artificial intelligence (AI) techniques, such as using a trained ML model. Some embodiments define and estimate a harmonic distortion by defining and estimating "virtual THD' (vTHD), which can be described as a measure of an instantaneous THD. For audio signals for which THD is well defined, vTHD coincides with THD up to a given tolerance (e.g., allowing a classification error to a nearest labeled THD value, such as one smaller or one larger of the classified THD value). However, when THD fails for very short duration audio signals, vTHD provides a new standard for estimating audio quality based on the disclosed technique that estimates vTHD of such signals.

Some embodiments of the disclosed solution are focused on sensing and quantifying harmonic distortions, regardless of noise, in a very short time. This feature makes the disclosed techniques applicable to dynamic (i.e., rapidly varying) signals and provides a powerful tool for better system engineering.

The disclosed ML techniques are able to systematically quantify the so-called "instantaneous" THD (i.e., the entity vTHD) on complex signals (e.g., a drum beat) and at very short times (e.g., several milliseconds).

To illustrate the challenge and the capabilities of such an ML technique, one can consider, by way of example, a Dynamic Range Compressor (DRC) nonlinear audio device that maps an input dynamic range to a smaller dynamic range in the output side. This sort of compression is usually achieved by lowering the high energy parts of the signal.

There is a strong relation between the response times of a DRC to the amount of harmonic distortion it will create as a side effect. As a general example, a very fast response time (e.g., <NUM> mSec) setting on a very slow signal (e.g., <NUM>) will create distortions once the compressor significantly attenuates the output. A DRC might have different response-time operation profiles from which to select. So, with the disclosed technique, a designer and system architect of such a device can quantify, using a vTHD scale, the distortion level of one DRC design over another.

The disclosed technique is by no means restricted to DRCs. A DRC embodiment is described later in detail since DRCs are a very common tool and since DRC's distortion artefact is controllable, making this use-case a good tool for explaining the technique.

In some embodiments, the disclosed technique endeavors to detect audio distortion in an audio signal that is presented as a picture (e.g., into 2D information). To this end, the disclosed technique classifies a set of distortions according to a model trained by using signals that were sliced from longer signals having a measurable THD. In particular, the THD of the longer signals can be measured by a laboratory-grade analyzer. The technique trains an ML model with a set of short (e.g., sliced) signals to classify any short signal according to the sets of labels, where the label is now converted one-to-one from THD to vTHD, with the vTHD of a distortion determined only by inference.

One scenario that justifies this conjecture on conversion validity is to consider a long stable signal (e.g., lasting few hundred cycles) from which THD can be measured. By slicing only several cycles of the long signal, a very short signal is received, on which THD is undefined, but any distortion is still present, and therefore a valid definition of vTHD scale would follow the rule: <MAT>.

In one embodiment, a system is provided that includes a memory configured to store a machine learning (ML) model and a processor, which is configured to perform the following steps:.

In a particular embodiment, the processor is configured to train the ML model by (i) converting the training audio signals into respective images (e.g., two-dimensional (2D) images, also referred to a first image) and (ii) training the ML model to estimate the levels of the distortion based on the images (first images). The processor is configured to estimate the level of the distortion in the input audio signal (e.g., it's vTHD) by (i) converting the input audio signal into a 2D image (also referred to a second image) and (ii) applying the trained ML model to the 2D image (second image). Note, however, that the disclosed technique can convert audio signals into multi-dimensional mathematical structures (e.g., 3D and more), such as tensors, to, for example, utilize dedicated computing hardware such as graphics processing units (GPUs) or tensor processing units (TPUs). Moreover, given a type of ML model (e.g., a type of NN) which is optimized to another mathematical structure at its input, the disclosed technique can, mutatis mutandis, convert an audio signal to that structure, such as a 3D RGB image, and apply it the given type of the trained ML model.

The training audio signals are typically labeled according to a ground truth scale of the THD, to, for example, estimate and classify the new preprocessed audio signal, during inference, according to the different labels of THD. The processor runs the ML model to infer the new preprocessed audio signal and to classify the new audio signal according to the different labels of THD with the respective vTHD. However, as no actual THD measurement could have been performed, the ML model is trained to recognize a distortion pattern on brief signals. In this way, as noted above, the vTHD serves as a consistent scale for comparing audio processing performance of very short duration signals.

In one embodiment, the processor is configured to preprocess the training audio signals by converting each audio signal into a respective 2D image (first image). For example, the processor is configured to convert each audio signal into a respective black and white 2D image by binary coding the audio signals in a 2D plane comprising a temporal axis and a signal amplitude axis, which is manifested as encoding an area confined by the graph as black while encoding the rest of the 2D image is white, as described below.

In another embodiment, the training samples are sliced and used in this way as a 2D image input for training without further preprocessing (e.g., without the black and white area encoding), and a new signal is not preprocessed before the ML model runs inference on that audio signal.

In yet another embodiment, the ML model uses ANN as a generative adversary network (GAN) which is particularly flexible in learning and inferring arbitrary waveforms. In general, various ML models may be used with data format optimized (e.g., converted from the audio samples) for the given ML model.

Moreover, with the necessary changes being made, the disclosed technique can identify and estimate audio distortion other than harmonic ones. For example, the disclosed technique may be applied, mutatis mutandis, to identify and estimate one of phase noise, chirp, and damping in audio signals.

By providing a ML-based audio distortion scale called virtual THD, audio engineers can quantify audio performance that cannot be quantified using existing techniques.

The time duration needed for a DRC to respond to (i.e., compress) an increased input signal ("attack"), or for a DRC to stop its processing ("release"), is a crucial parameter to audio quality. A user cannot simply "set the attack and release" to a minimum, because an exceedingly short attack and release setting creates harmonic distortion. This artefact, e.g., THD, is a by-product of the DRC setting in conjunction with the input signal and its properties.

The THD of an output signal (i.e., a THD which is a by-product of the DRC setting) is easily noticeable by a human listener and hence each DRC has its attack and release knobs (or auto setting). Even more, THD is viewable on a waveform display.

Albeit being both audible and viewable to a human user, it is quite surprising to see that there is no measurement method which quantifies this distortion. This lack of quantification leads to a reality in which DRC engineers and system designers lack a scientific measurement tool which can help systemize the development process of future DRCs by means of quantifying the artefacts. As mentioned above, this is true not only for DRC, but in fact to any non-linear processor (Gates, Limiters, Saturators, etc.).

<FIG> is a graph <NUM> that shows the effect of audio compression on an audio signal, the compression performed by a Dynamic Range Compressor (DRC) configured with short and long response times, in accordance with an embodiment of the present invention.

In the shown embodiment, a compressor or a DRC maps an input dynamic range <NUM> of an incoming sinewave signal into a target dynamic range <NUM>, set by the user. This process involves setting (or auto setting) a threshold audio energy, above which the DRC will compress and under which the DRC will not alter the signal, the ratio of compression as well as the attack and release.

In the example of <FIG>, the input signal has a fixed frequency of <NUM> with an amplitude that can be varied below and above the threshold value of the DRC. In the example measurement of <FIG>, the DRC threshold is -<NUM> dB, with a compression ratio of <NUM>:<NUM>. with two different attack times (<NUM>µSec vs. <NUM> mSec) the output result distortion is very vivid visually. As seen, the short attack time results in a signal <NUM> that is highly distorted. On the other hand, a signal <NUM>, which results from the long attack time, is largely a sinewave, with some amplitude modulation. However, the different level of distortion exhibited by signals <NUM> and <NUM> is not quantifiable to date, as explained above. The present disclosure provides embodiments that can quantify the different short-duration audio distortions (e.g., distortions taking place over a time duration smaller than several milliseconds).

<FIG> is a block diagram schematically illustrating a system <NUM> for estimation of virtual total harmonic distortion (vTHD) of a short audio sample (<NUM>) outputted by an audio processing apparatus <NUM>, in accordance with an embodiment of the present invention.

As seen, system <NUM> is coupled to audio processing apparatus <NUM> that comprises a linear gain circuitry <NUM> that does not distort the input signal, and a non-linear processor <NUM>, such as the aforementioned DRC, that may distort the linearly amplified input signal. The output signal is directed to an output device <NUM>, such as a loudspeaker.

System <NUM> for estimation of vTHD is configured to estimate the nonlinear audio effect of audio processing apparatus <NUM>, and in particular of non-linear processor <NUM>, by providing a vTHD <NUM> grade of an unintentional distortion introduced by non-linear processor <NUM>. Using the estimated vTHD enables a user, or a processor, to optimize settings of apparatus <NUM> to optimize an intentional amount of distortion, such as to limit an intentional distortion to a desired level.

As further seen, system <NUM> is inputted with an audio signal <NUM> that is distorted after being processed by non-linear audio processing circuitry <NUM>.

A processor <NUM>, or a preprocessing circuitry <NUM>, performs preprocessing of audio signal <NUM> by converting (e.g., encoding) the 1D waveform of signal <NUM> into a 2D black and white image <NUM>, such as the images seen in <FIG>. In other words, processor <NUM> converts a given training audio signal into a given 2D image by setting pixel values of the 2D images to represent an amplitude of the given training audio signal as a function of time.

Then, processor <NUM> runs a trained ANN <NUM> (that can be a convolutional ANN (CNN) or a GAN, to name two options, that is held in a memory <NUM> to perform inference on image <NUM> to estimate vTHD <NUM> of signal <NUM>.

Finally, a feedback line <NUM> between processor <NUM> and non-linear processor <NUM> enables controlling the amount of artefacts in output audio signal <NUM>, based on the estimated vTHD. Such feedback line may alternatively, or additionally, be used between processor <NUM> and linear gain circuitry <NUM>.

The embodiment of <FIG> is depicted by way of example, purely for the sake of clarity. For example, preprocessing circuitry <NUM> may perform another type of preprocessing, or, for a given suitable ML model being used, perform no preprocessing of the training samples <NUM> (e.g., aside from slicing them after measuring THD).

The different elements of system <NUM> and audio processing apparatus <NUM> shown in <FIG> may be implemented using suitable hardware, such as one or more discrete components, one or more Application-Specific Integrated Circuits (ASICs) and/or one or more Field-Programmable Gate Arrays (FPGAs). Some of the functions of system <NUM> may be implemented in one or more general purpose processors programmed in software to carry out the functions described herein.

The software may be downloaded to the processors in electronic form, over a network or from a host, for example, or it may, alternatively or additionally, be provided and/or stored on non-transitory tangible media, such as magnetic, optical, or electronic memory.

<FIG> shows a set <NUM> of two-dimensional (2D) images used in training an artificial neural network (ANN) <NUM> in the system of <FIG>, in accordance with an embodiment of the present invention. As seen, images of set <NUM> are associated with progressively increasing THD levels. The THD was measured on training audio signals from which the 2D images were generated, the training audio signals being each of <NUM> cycle length (i.e., samples with duration of <NUM> mSec) at <NUM>). The preprocessed 2D images were generated after the training audio samples were truncated (e.g., sliced) to leave only five cycles. Thus, the training uses short duration samples (e.g., of five cycles of a <NUM> wave), with total duration of each sample being <NUM> milliseconds. This duration is considered very short and does not allow, for example, meaningful FFT analysis of harmonic distortion, as emphasized above. In principle, a signal can be truncated to as little as a fraction of a cycle (e.g., quarter cycle), and the disclosed technique will generate a vTHD scale of distortion using such ultrashort audio signals. Using truncated signals further allows to, for example, maximize tolerance of the disclosed technique to low signal-to-noise ratio, while gaining on the analysis of ultra-short duration audio harmonic distortions.

Set <NUM> of training images is a cascade of preprocessed sine-wave signals with the initial sine wave signals with an increasing "digital saturation" level that clips the sine wave at its minimum and maximum absolute values. As seen, the clipping is first none, i.e., starting with zero clipping having a THD=<NUM>, with the saturation effect increasing all the way to a maximal clipping that results in a rectangular wave-like waveform with a measure (e.g., ground truth) THD of <NUM>. In the given example, the actual testing starts, for simplicity of presentation, from <NUM>% THD (i.e., THD=<NUM>), as described in <FIG>.

The increased level of THD reflects a growing relative contribution to a signal of higher harmonics (3ω, 5ω, 7ω. ), pure, sinus harmonics at ω.

Each 2D image of set <NUM> is received from a 1D waveform similarly to how image <NUM> is received from respective waveform <NUM>, as described in <FIG>.

In particular, the preprocessing may use a code that blacks areas <NUM> between the envelope and the horizonal axis, and maintains white the rest of each image.

In the particular example exemplified by <FIG>, data preprocessing includes these steps:.

In the field of ML, and specifically the problem of statistical classification, a confusion matrix, also known as an error matrix, is a specific table layout that allows visualization of the performance of an algorithm, typically a supervised learning algorithm (i.e., one that uses labeled training data for learning). Each row of the matrix represents the instances in an actual class while each column represents the instances in a predicted class, or vice versa. The name stems from the fact that it makes it easy to see whether the system is confusing two classes (i.e., commonly mislabeling one as the other).

<FIG> illustrates a confusion matrix <NUM> comparing vTHD <NUM> estimated using system <NUM> of <FIG> to ground-truth THD of <FIG>, in accordance with an embodiment of the present invention. The number of samples inferenced at each THD level is indicated by a scale <NUM>, with number of samples ranging between few to more than <NUM>.

As seen, for THD><NUM>, the errors made during inference by the trained ANN model <NUM> are deviations by one class at most (for example, some audio samples with THD=j may have been classified as having VTHD=j+<NUM> or VTHD=j-<NUM>). The vast majority of audio samples were accurately classified by system <NUM>.

The shown example of <FIG> is brought by way of example. As another example, rather than use classification to estimate an error in vTHD compared to a ground truth THD, a ML model may use a regression-based scoring, as described below.

<FIG> is a flow chart that schematically illustrates a method for estimation of vTHD of a short audio sample using system <NUM> of <FIG>, in accordance with an embodiment of the present invention. The algorithm, according to the presented embodiment, carries out a process that is split between a training phase <NUM> and an inferencing phase <NUM>.

The training phase begins at an uploading step <NUM>, during which processor <NUM> uploads a set of short (e.g., sliced) training audio samples, like the <NUM>-cycle audio sample used in <FIG>, from memory <NUM>. Next, processing circuitry <NUM> converts the audio samples into black and white images, as shown in <FIG>, at a data format conversion step <NUM>.

In an ANN training step <NUM>, processor <NUM> trains ANN <NUM> using the black and white images to estimate a vTHD of an audio signal.

Inference phase <NUM> begins by system <NUM> receiving as an input a short time duration audio sample (e.g., of several milliseconds duration), at an audio sample inputting step <NUM>.

Next, processing circuitry <NUM> converts the short audio sample into a black and white image, at a data format conversion step <NUM>. Then, processor <NUM> runs the trained ANN <NUM> to estimate a vTHD value of the audio sample, at a vTHD estimation step <NUM>. Finally, at a vTHD outputting step <NUM>, processor <NUM> of system <NUM> outputs the estimated vTHD to a user, or to a processor, to, for example, adjust a nonlinear audio stage according to a desired vTHD value, such as to adjust a saturation level imposed by nonlinear audio processor <NUM> of audio processing apparatus <NUM>.

The flow chart of <FIG> is brought purely by way of example, for the sake of clarity. For example, other preprocessing steps, or fewer steps, may be used.

As noted above, a regression-based scoring may be used in addition, or as alternative to vTHD estimation by classification shown in <FIG>. In a regression-based scoring, the system uses the same processed data (either the white painted data and/or the black painted can be used). In this embodiment, the CNN uses a predicts mean squared error function as a loss function, to output a number that indicates how close the vTHD is to the ground truth THD value.

The accuracy of both the classification method and the regression-based method can be improved with data sampling precision, by, for example, using a <NUM>-bit digitization scheme instead of the <NUM>-bit used.

Note that, mathematically, the data set looks different for classification and regression problem in terms of Y vector (For classification - for every example Sj there is a 1D classification vector. For regression for every example Sj there is a scalar regression score.

Claim 1:
A system (<NUM>), comprising:
a memory (<NUM>) configured to store a machine learning, ML, model (<NUM>); and
a processor (<NUM>), which is configured to:
obtain a set of training audio signals that are labeled with respective levels of distortion;
convert the training audio signals into respective first images, wherein the conversion of a given audio training signal into respective first images comprises setting pixel values of the first images to represent an amplitude of the given training audio signal as a function of time;
train the ML model (<NUM>) to estimate the levels of the distortion based on the respective first images;
receive an input audio signal (<NUM>);
convert the input audio signal (<NUM>) into a second image (<NUM>);
estimate a level of the distortion in the input audio signal (<NUM>), by applying the trained ML model (<NUM>) to the second image (<NUM>).