Patent Description:
Embodiments of the invention refer to an audio generator, configured to generate an audio signal from an input signal and target data, the target data representing the audio signal. Further embodiments refer to methods for generating an audio signal, and methods for training an audio generator. Further embodiments refer to a computer program product.

In recent years, neural vocoders have surpassed classical speech synthesis approaches in terms of naturalness and perceptual quality of the synthesized speech signals. The best results can be achieved with computationally-heavy neural vocoders like WaveNet and WaveGlow, while light-weight architectures based on Generative Adversarial Networks, e.g. MelGAN and Parallel WaveGAN, are still inferior in terms of the perceptual quality.

Generative models using Deep Learning for generating audio waveforms, such as WaveNet, LPCNet, and WaveGlow, have provided significant advances in natural-sounding speech synthesis. These generative models called in Text-To-Speech (TTS) applications neural vocoders, outperform both parametric and concatenative synthesis methods. They can be conditioned using compressed representations of the target speech (e.g. mel-spectrogram) for reproducing a given speaker and a given utterance.

Prior works have shown that speech coding at very low bit-rate of clean speech can be achieved using such generative models at the decoder side. This can be done by conditioning the neural vocoders with the parameters from a classical low bit-rate speech coder.

Neural vocoders were also used for speech enhancement tasks, like speech denoising or dereverberation.

The main problem of these deep generative models is usually the high number of required parameters, and the resulting complexity both during training and synthesis (inference). For example, WaveNet, considered as state-of-the-art for the quality of the synthesized speech, generates sequentially the audio samples one by one. This process is very slow and computationally demanding, and cannot be performed in real time.

Recently, lightweight adversarial vocoders based on Generative Adversarial Networks (GANs), such as MelGAN and Parallel WaveGAN, have been proposed for fast waveform generation. However, the reported perceptual quality of the speech generated using these models is significantly below the baseline of neural vocoders like WaveNet and WaveGlow. A GAN for Text-to-Speech (GAN-TTS) has been proposed to bridge this quality gap, but still at a high computational cost.

There exists a great variety of neural vocoders, which all have drawbacks. Autoregressive vocoders, for example WaveNet and LPCNet, may have very high quality, and be suitable for optimization for inference on CPU, but they are not suitable for usage on GPUs, since their processing cannot be parallelized easily, and they cannot offer not real time processing without compromising the quality.

Normalizing flows vocoders, for example WaveGlow, may also have very high quality, and be suitable for inference on a GPU, but they comprise a very complex model, which takes a long time to train and optimize, it is also not suitable for embedded devices.

GAN vocoders, for example MelGAN and Parallel WaveGAN may be suitable for inference on GPUs and lightweight, but their quality is lower than autoregressive models.

In summary, there still does not exist a low complexity solution delivering high fidelity speech. GANs are the most studied approach to achieve such a goal. The present invention is an efficient solution for this problem.

It is an object of the present invention to provide a lightweight neural vocoder solution which generates speech at very high quality and is trainable with limited computational resources.

<CIT> discloses a multi-to-many conversion method for audio signals.

Embodiments according to the present invention will subsequently be described taking reference to the enclosed figures in which:.

In the figures, similar reference signs denote similar elements and features.

There is proposed an audio generator, configured to generate an audio signal from an input signal and target data, the target data representing the audio signal, comprising:
A first processing block, configured to receive first data derived from the input signal and to output first output data, wherein the first output data comprises a plurality of channels, and a second processing block, configured to receive, as second data, the first output data or data derived from the first output data, wherein the first processing block comprises for each channel of the first output data: a conditioning set of learnable layers configured to process the target data to obtain conditioning features parameters; and a styling element, configured to apply the conditioning feature parameters to the first data or normalized first data; and wherein the second processing block is configured to combine the plurality of channels of the second data to obtain the audio signal, characterized in that the first processing block further comprises:
a further set of learnable layers, configured to process data derived from the first data using a second activation function, wherein the second activation function is a gated activation function.

There is also proposed a method for generating an audio signal by an audio generator from an input signal and target data, the target data representing the audio signal, comprising:
receiving, by a first processing block, first data derived from the input signal; for each channel of a first output data: processing, by a conditioning set of learnable layers of the first processing block, the target data to obtain conditioning feature parameters; and applying, by a styling element of the first processing block, the conditioning feature parameters to the first data or normalized first data; outputting, by the first processing block, first output data comprising a plurality of channels; receiving, by a second processing block, as second data, the first output data or data derived from the first output data; and combining, by the second processing block, the plurality of channels of the second data to obtain the audio signal, characterized by:
processing, by a further set of learnable layers of the first processing block, data derived from the first data using a second activation function, wherein the second activation function a gated activation function.

It is in this context that we propose StyleMelGAN (the audio generator <NUM>), a light-weight neural vocoder, allowing synthesis of high-fidelity speech with low computational complexity. StyleMelGAN is a fully convolutional, feed-forward model that uses Temporal Adaptive DEnormalization, TADE, (60a and 60b in <FIG>, and <NUM> in <FIG>) to style (at <NUM>) a low-dimensional noise vector (e.g. a 128x1 vector) via the acoustic features of the target speech waveform. The architecture allows for highly parallelizable generation, several times faster than real time on both control processing units, CPUs, and graphic processing units, GPUs. For efficient and fast training, we may use a multi-scale spectral reconstruction loss together with an adversarial loss calculated by multiple discriminators (132a-132d) evaluating the speech signal <NUM> in multiple frequency bands and with random windowing (e.g., the windows 105a, 105b, 105c, 105d). MUSHRA and P. <NUM> listening tests show that StyleMelGAN (the audio generator <NUM>) outperforms known existing neural vocoders in both copy synthesis and TTS scenarios.

The present application proposes, inter alia, a neural vocoder for generating high quality speech <NUM>, which may be based on a generative adversarial network (GAN). The solution, here called StyleMelGAN (and implemented in the audio generator <NUM>), is a lightweight neural vocoder allowing synthesis of high-quality speech <NUM> at low computational complexity. StyleMelGAN is a feed-forward, fully convolutional model that uses temporal adaptive denormalization (TADE) for styling (at block <NUM>) a latent noise representation (e.g. <NUM>) using, for example the mel-spectrogram (<NUM>) of the target speech waveform. It allows highly parallelizable generation, which is several times faster than real time on both CPUs and GPUs. For training, it is possible to use multi-scale spectral reconstruction losses followed by adversarial losses. This enables to obtain a model able to synthesize high-quality outputs after less than <NUM> days of training on a single GPU.

<FIG> shows an example of an audio generator <NUM> which generates (e.g., synthesizes) an audio signal (output signal) <NUM>, e.g. according to StyleMelGAN. The output audio signal <NUM> is generated based on an input signal <NUM> (also called latent signal and which may be noise, e.g. white noise) and target data <NUM> (also called "input sequence"). The target data <NUM> may, for example, comprise (e.g. be) a spectrogram (e.g., a mel-spectrogram), the mel-spectrogram providing mapping, for example, of a sequence of time samples onto mel scale. In addition or alternatively, the target data <NUM> may comprise (e.g. be) a bitstream. For example, the target data may be or include text which is to be reproduced in audio (e.g., text-to-speech). The target data <NUM> is in general to be processed, in order to obtain a speech sound recognizable as natural by a human listener. The input signal <NUM> may be noise (which as such carries no useful information), e.g. white noise, but, in the generator <NUM>, a noise vector taken from the noise is styled (at <NUM>) to have a noise vector with the acoustic features conditioned by the target data <NUM>. At the end, the output audio signal <NUM> will be understood as speech by a human listener. The noise vector <NUM> may be, like in <FIG>, a 128x1 vector (one single sample, e.g. time domain samples or frequency domain samples, and <NUM> channels). Other length of the noise vector <NUM> could be used in other examples.

The first processing block <NUM> is shown in <FIG>. As will be shown (e.g., in <FIG>) the first processing block <NUM> may be instantiated by each of a plurality of blocks (in <FIG>, blocks 50a, 50b, 50c, 50d, 50e, 50f, <NUM>, <NUM>). The blocks 50a-<NUM> may be understood as forming one single block <NUM>. It will be shown that in the first processing block <NUM>, <NUM>, a conditioning set of learnable layers (e.g., <NUM>, <NUM>, <NUM>) is used to process the target data <NUM> and the input signal <NUM>. Accordingly, conditioning feature parameters <NUM>, <NUM> (also referred to as gamma, γ, and beta, β, in <FIG>) may be obtained, e.g. by convolution, during training. The learnable layers <NUM>-<NUM> may therefore be part of a weight layer of a learning network or, more in general, another learning structure. The first processing block <NUM>, <NUM> may include at least one styling element <NUM>. The at least one styling element <NUM> may output the first output data <NUM>. The at least one styling element <NUM> may apply the conditioning feature parameters <NUM>, <NUM> to the input signal <NUM> (latent) or the first data <NUM> obtained from the input signal <NUM>.

The first output data <NUM> at each block <NUM> are in a plurality of channels. The audio generator <NUM> may include a second processing block <NUM> (in <FIG> shown as including the blocks <NUM>, <NUM>, <NUM>). The second processing block <NUM> may be configured to combine the plurality of channels <NUM> of the first output data <NUM> (inputted as second input data or second data), to obtain the output audio signal <NUM> in one single channel, but in a sequence of samples.

The "channels" are not to be understood in the context of stereo sound, but in the context of neural networks (e.g. convolutional neural networks). For example, the input signal (e.g. latent noise) <NUM> may be in <NUM> channels (in the representation in the time domain), since a sequence of channels are provided. For example, when the signal has <NUM> samples and <NUM> channels, it may be understood as a matrix of <NUM> columns and <NUM> rows, while when the signal has <NUM> samples and <NUM> channels, it may be understood as a matrix of <NUM> columns and <NUM> rows (other schematizations are possible). Therefore, the generated audio signal <NUM> (which in <FIG> results in a 1x22528 row matrix) may be understood as a mono signal. In case stereo signals are to be generated, then the disclosed technique is simply to be repeated for each stereo channel, so as to obtain multiple audio signals <NUM> which are subsequently mixed.

The at least the original input signal <NUM> and/or the generated speech <NUM> may be a vector. To the contrary, the output of each the blocks <NUM> and 50a-<NUM>, <NUM>, <NUM> has in general a different dimensionality. The first data may have a first dimension or at least one dimension lower than that of the audio signal. The first data may have a total number of samples across all dimensions lower than the audio signal. The first data may have one dimension lower than the audio signal but a number of channels greater than the audio signal. At each block <NUM> and 50a-<NUM>, the signal, evolving from noise <NUM> towards becoming speech <NUM>, may be upsampled. For example, at the upsampling block <NUM> before the first block 50a among the blocks 50a-<NUM>, an <NUM>-times upsampling is performed. An example of upsampling may include, for example, the following sequence: <NUM>) repetition of same value, <NUM>) insert zeros, <NUM>) another repeat or insert zero + linear filtering, etc..

The generated audio signal <NUM> may generally be a single-channel signal (e.g. 1x22528). In case multiple audio channels are necessary (e.g., for a stereo sound playback) then the claimed procedure shall be in principle iterated multiple times.

Analogously, also the target data <NUM> can be, in principle, in one single channel (e.g. if it is text) or in multiple channels (e.g. in spectrograms). In any case, it may be upsampled (e.g. by a factor of two, a power of <NUM>, a multiple of <NUM>, or a value greater than <NUM>) to adapt to the dimensions of the signal (59a, <NUM>, <NUM>) evolving along the subsequent layers (50a-<NUM>, <NUM>), e.g. to obtain the conditioning feature parameters <NUM>, <NUM> in dimensions adapted to the dimensions of the signal.

When the first processing block <NUM> is instantiated in multiple blocks 50a-<NUM>, the number of channels may, for example, remain the same for the multiple blocks 50a-<NUM>. The first data may have a first dimension or at least one dimension lower than that of the audio signal. The first data may have a total number of samples across all dimensions lower than the audio signal. The first data may have one dimension lower than the audio signal but a number of channels greater than the audio signal.

The signal at the subsequent blocks may have different dimensions from each other. For example, the sample may be upsampled more and more times to arrive, for example, from <NUM> samples to <NUM>,<NUM> samples at the last block <NUM>. Analogously, also the target data <NUM> are upsampled at each processing block <NUM>. Accordingly, the conditioning features parameters <NUM>, <NUM> can be adapted to the number of samples of the signal to be processed. Accordingly, se-mantic information provided by the target data <NUM> is not lost in subsequent layers 50a-<NUM>.

It is to be understood that examples may be performed according to the paradigms of generative adversarial networks (GANs). A GAN includes a GAN generator <NUM> (<FIG>) and a GAN discriminator <NUM> (<FIG>). The GAN generator <NUM> tries to generate an audio signal <NUM>, which is as close as possible to a real signal. The GAN discriminator <NUM> shall recognize whether the generated audio signal is real (like the real audio signal <NUM> in <FIG>) or fake (like the generated audio signal <NUM>). Both the GAN generator <NUM> and the GAN discriminator <NUM> may be obtained as neural networks. The GAN generator <NUM> shall minimize the losses (e.g., through the method of the gradients or other methods), and update the conditioning features parameters <NUM>, <NUM> by taking into account the results at the GAN discriminator <NUM>. The GAN discriminator <NUM> shall reduce its own discriminatory loss (e.g., through the method of gradients or other methods) and update its own internal parameters. Accordingly, the GAN generator <NUM> is trained to provide better and better audio signals <NUM>, while the GAN discriminator <NUM> is trained to recognize real signals <NUM> from the fake audio signals generated by the GAN generator <NUM>. In general terms, it may be understood that the GAN generator <NUM> may include the functionalities of the generator <NUM>, without at least the functionalities of the GAN discriminator <NUM>. Therefore, in most of the foregoing, it may be understood that the GAN generator <NUM> and the audio generator <NUM> may have more or less the same features, apart from those of the discriminator <NUM>. The audio generator <NUM> may include the discriminator <NUM> as an internal component. Therefore, the GAN generator <NUM> and the GAN discriminator <NUM> may concur in constituting the audio generator <NUM>. In examples where the GAN discriminator <NUM> is not present, the audio generator <NUM> can be constituted uniquely by the GAN generator <NUM>.

As explained by the wording "conditioning set of learnable layers", the audio generator <NUM> may be obtained according to the paradigms of conditional GANs, e.g. based on conditional information. For example, conditional information may be constituted by target data (or upsampled version thereof) <NUM> from which the conditioning set of layers <NUM>-<NUM> (weight layer) are trained and the conditioning feature parameters <NUM>, <NUM> are obtained. Therefore, the styling element <NUM> is conditioned by the learnable layers <NUM>-<NUM>.

The examples may be based on convolutional neural networks. For example, a little matrix (e.g., filter or kernel), which could be a 3x3 matrix (or a 4x4 matrix, etc.), is convolved (convoluted) along a bigger matrix (e.g., the channel x samples latent or input signal and/or the spectrogram and/or the spectrogram or upsampled spectrogram or more in general the target data <NUM>), e.g. implying a combination (e.g., multiplication and sum of the products; dot product, etc.) between the elements of the filter (kernel) and the elements of the bigger matrix (activation map, or activation signal). During training, the elements of the filter (kernel) are obtained (learnt) which are those that minimize the losses. During inference, the elements of the filter (kernel) are used which have been obtained during training. Examples of convolutions are at blocks <NUM>-<NUM>, 61a, 61b, 62a, 62b (see below). Where a block is conditional (e.g., block <NUM> of <FIG>), then the convolution is not necessarily applied to the signal evolving from the input signal <NUM> towards the audio signal <NUM> through the intermediate signals 59a (<NUM>), <NUM>, etc., but may be applied to the target signal <NUM>. In other cases (e.g. at blocks 61a, 61b, 62a, 62b) the convolution may be not conditional, and may for example be directly applied to the signal 59a (<NUM>), <NUM>, etc., evolving from the input signal <NUM> towards the audio signal <NUM>. As can be seen from <FIG> and <FIG>, both conditional and no-conditional convolutions may be performed.

It is possible to have, in some examples, activation functions downstream to the convolution (ReLu, TanH, softmax, etc.), which may be different in accordance to the intended effect. ReLu may map the maximum between <NUM> and the value obtained at the convolution (in practice, it maintains the same value if it is positive, and outputs <NUM> in case of negative value). Leaky ReLu may output x if x><NUM>, and <NUM>*x if x≤<NUM>, x being the value obtained by convolution (instead of <NUM> another value, such as a predetermined value within <NUM> ± <NUM>, may be used in some examples). TanH (which may be implemented, for example, at block 63a and/or 63b) may provide the hyperbolic tangent of the value obtained at the convolution, e.g. <MAT> with x being the value obtained at the convolution (e.g. at block 61a and/or 61b). Softmax (e.g. applied, for example, at block 64a and/or 64b) may apply the exponential to each element of the elements of the result of the convolution (e.g., as obtained in block 62a and/or 62b), and normalize it by dividing by the sum of the exponentials. Softmax (e.g. at 64a and/or 64b) may provide a probability distribution for the entries which are in the matrix which results from the convolution (e.g. as provided at 62a and/or 62b). After the application of the activation function, a pooling step may be performed (not shown in the figures) in some examples, but in other examples it may be avoided.

<FIG> shows that it is also possible to have a softmax-gated TanH function, e.g. by multiplying (e.g. at 65a and7Or 65b) the result of the TanH function (e.g. obtained at 63a and/or 63b) with the result of the softmax function (e.g. obtain at 64a and/or 64b).

Multiple layers of convolutions (e.g. a conditioning set of learnable layers) may be one after another one and/or in parallel to each other, so as to increase the efficiency. If the application of the activation function and/or the pooling are provided, they may also be repeated in different layers (or maybe different activation functions may be applied to different layers, for example).

The input signal <NUM> (e.g. noise) is processed, at different steps, to become the generated audio signal <NUM> (e.g. under the conditions set by the conditioning sets of learnable layers <NUM>-<NUM>, and on the parameters <NUM>, <NUM> learnt by the conditioning sets of learnable layers <NUM>-<NUM>. Therefore, the input signal is to be understood as evolving in a direction of processing (from <NUM> to <NUM> in <FIG>) towards becoming the generated audio signal <NUM> (e.g. speech). The conditions will be substantially generated based on the target signal <NUM>, and on the training (so as to arrive at the most preferable set of parameters <NUM>, <NUM>).

It is also noted that the multiple channels of the input signal (or any of its evolutions) may be considered to have a set of learnable layers and a styling element associated thereto. For example, each row of the matrixes <NUM> and <NUM> is associated to a particular channel of the input signal (or one of its evolutions), and is therefore obtained from a particular learnable layer associated to the particular channel. Analogously, the styling element <NUM> may be considered to be formed by a multiplicity of styling elements (each for each row of the input signal x, c, <NUM>, <NUM>, <NUM>', <NUM>, 59a, 59b, etc.).

<FIG> shows an example of the audio generator <NUM> (which may embody the audio generator <NUM> of <FIG>), and which may also comprise (e.g. be) a GAN generator <NUM>. The target data <NUM> is indicated as mel-spectrogram, the input signal <NUM> may be a latent noise, and the output of the signal <NUM> may be speech (other examples are notwithstanding possible, as explained above). As can be seen, the input signal <NUM> has only one sample and <NUM> channels. The noise vector <NUM> may be obtained in a vector with <NUM> channels (but other numbers are possible) and may have a zero-mean normal distribution. The noise vector may follow the formula z ~ <IMG>(<NUM>, I<NUM>) The noise vector may be a random noise of dimension <NUM> with mean <NUM> generated, and with an autocorrelation matrix (square 128x128) is equal to the identity I (different choice may be made). Hence, in examples the generated noise can be completely decorrelated between the channels and of variance <NUM> (energy). <IMG>(<NUM>, I<NUM>) may be realized at every <NUM> generated samples (or other numbers may be chosen for different examples); the dimension may therefore be <NUM> in the time axis and <NUM> in the channel axis.

It will be shown that the noise vector <NUM> is step-by-step processed (e.g., at blocks 50a-<NUM>, <NUM>, <NUM>, <NUM>, etc.), so as to evolve from, e.g., noise <NUM> to, e.g., speech <NUM> (the evolving signal will be indicated, for example, with different signals <NUM>, 59a, x, c, <NUM>', <NUM>, 79a, 59b, 79b, <NUM>, etc.).

At block <NUM>, the input signal (noise) <NUM> may be upsampled to have <NUM> samples (different numbers are possible) and <NUM> channels (different numbers are possible).

As can be seen, eight processing blocks 50a, 50b, 50c, 50d, 50e, 50f, <NUM>, <NUM> (altogether embodying the first processing block <NUM> of <FIG>) may increase the number of samples by performing an upsampling (e.g., maximum <NUM>-upsampling). The number of channels may always remain the same (e.g., <NUM>) along blocks 50a, 50b, 50c, 50d, 50e, 50f, <NUM>, <NUM>. The samples may be, for example, the number of samples per second (or other time unit): we may obtain, at the output of block <NUM>, sound at more than <NUM>.

Each of the blocks 50a-<NUM> (<NUM>) can also be a TADEResBlock (residual block in the context of TADE, Temporal Adaptive DEnormalization). Notably, each block 50a-<NUM> may be conditioned by the target data (e.g., mel-spectrogram) <NUM>.

At a second processing block <NUM> (<FIG> and <FIG>), only one single channel may be obtained, and multiple samples are obtained in one single dimension. As can be seen, another TADEResBlock <NUM> (further to blocks 50a-<NUM>) is used (which reduces to one single channel). Then, a convolution layer <NUM> and an activation function (which may be TanH <NUM>, for example) may be performed. After that, the speech <NUM> is obtained (and, possibly, stored, rendered, encoded, etc.).

At least one of the blocks 50a-<NUM> (or each of them, in particular examples) may be, for example, a residual block. A residual block operates a prediction only to a residual component of the signal evolving from the input signal <NUM> (e.g. noise) to the output audio signal <NUM>. The residual signal is only a part (residual component) of the main signal. For example, multiple residual signals may be added to each other, to obtain the final output audio signal <NUM>.

<FIG> shows an example of one of the blocks 50a-<NUM> (<NUM>). As can be seen, each block <NUM> is inputted with a first data 59a, which is either the input signal <NUM>, (or the upsampled version thereof, such as that output by the upsampling block <NUM>) or the output from a preceding block. For example, the block 50b may be inputted with the output of block 50a; the block 50c may be inputted with the output of block 50b, and so on.

In <FIG>, it is therefore possible to see that the first data 59a provided to the block <NUM> (50a-<NUM>) is processed and its output is the output signal <NUM> (which will be provided as input to the subsequent block). As indicated by the line 59a', a main component of the first data 59a inputted to the first processing block 50a-<NUM> (<NUM>) actually bypasses most of the processing of the first processing block 50a-<NUM> (<NUM>). For example, blocks 60a, 61a, 62a, 63a, 65a, 60b, 61b, 62b, 63b, 64b, and 65b are bypassed by the bypassing line 59a'. The first data 59a will subsequently add to a residual portion 64b' at an adder 65c (which is indicated in <FIG>, but not shown). This bypassing line 59a' and the addition at the adder 65c may be understood as instantiating the fact that each block <NUM> (50a-<NUM>) processes operations to residual signals, which are then added to the main portion of the signal. Therefore, each of the blocks 50a-<NUM> can be considered a residual block.

Notably, the addition at adder 65c does not necessarily need to be performed within the residual block <NUM> (50a-<NUM>). A single addition of a plurality of residual signals 65b' (each outputted by each of residual blocks 50a-<NUM>) can be performed (e.g., at an adder block in the second processing block <NUM>, for example). Accordingly, the different residual blocks 50a-<NUM> may operate in parallel with each other.

In the example of <FIG>, each block <NUM> may repeat its convolution layers twice (e.g., first at replica <NUM>, including at least one of blocks 60a, 61a, 62a, 63a, 64a, 65a, and obtaining signal 59b; then, at replica <NUM>, including at least one of blocks 60b, 61b, 62b, 63b, 64b, 65b, and obtaining signal 65b', which may be added to the main component 59a').

For each replica (<NUM>, <NUM>), a conditioning set of learnable layers <NUM>-<NUM> and a styling element <NUM> is applied (e.g. twice for each block <NUM>) to the signal evolving from the input signal <NUM> to the audio output signal <NUM>. A first temporal adaptive denormalization (TADE) is performed at TADE block 60a to the first data 59a at the first replica <NUM>. The TADE block 60a performs a modulation of the first data 59a (input signal or, e.g., processed noise) under the conditions set out by the target data <NUM>. In the first TADE block 60a, an upsampling of the target data <NUM> may be performed at upsampling block <NUM>, to obtain an upsampled version <NUM>' of the target data <NUM>. The upsampling may be obtained through non-linear interpolation, e.g. using a factor of <NUM>, a power of <NUM>, a multiple of two, or another value greater than <NUM>. Accordingly, in some examples it is possible to have that the spectrogram <NUM>' has the same dimensions (e.g. conforms to) the signal (<NUM>, <NUM>', x, c, <NUM>, 59a, 59b, etc.) to be conditioned by the spectrogram. An application of stylistic information to the processed noise (first data) (<NUM>, <NUM>', x, c, <NUM>, 59a, 59b, etc.) may be performed at block <NUM> (styling element). In a subsequent replica <NUM>, another TADE block 60b may be applied to the output 59b of the first replica <NUM>. An example of the TADE block <NUM> (60a, 60b) is provided in <FIG> (see also below). After having modulated the first data 59a, convolutions 61a and 62a are carried out. Subsequently, activation functions TanH and softmax (e.g. constituting the softmax-gated TanH function) are also performed (63a, 64a). The outputs of the activation functions 63a and 64a are multiplied at multiplier block 65a (e.g. to instantiate the gating), to obtain a result 59b. In case of the use of two different replicas <NUM> and <NUM> (or in case of the use of more than two replicas), the passages of blocks 60a, 61a, 62a, 63a, 64a, 65a, are repeated.

In examples, the first and second convolutions at 61b and 62b, respectively downstream to the TADE block 60a and 60b, may be performed at the same number of elements in the kernel (e.g., <NUM>, e.g., 3x3). However, the second convolutions 61b and 62b may have a dilation factor of <NUM>. In examples, the maximum dilation factor for the convolutions may be <NUM> (two).

<FIG> shows an example of a TADE block <NUM> (60a, 60b). As can be seen, the target data <NUM> may be upsampled, e.g. so as to conform to the input signal (or a signal evolving therefrom, such as <NUM>, 59a, <NUM>', also called latent signal or activation signal). Here, convolutions <NUM>, <NUM>, <NUM> may be performed (an intermediate value of the target data <NUM> is indicated with <NUM>'), to obtain the parameters γ (gamma, <NUM>) and β (beta, <NUM>). The convolution at any of <NUM>, <NUM>, <NUM> may also require a rectified linear unit, ReLu, or rectify a leaky rectified linear unit, leaky ReLu. The parameters γ and β may have the same dimension of the activation signal (the signal being processed to evolve from the input signal <NUM> to the generated audio signal <NUM>, which is here represented as x, <NUM>, or <NUM>' when in normalized form). Therefore, when the activation signal (x, <NUM>, <NUM>') has two dimensions, also γ and β (<NUM> and <NUM>) have two dimensions, and each of them is superimposable to the activation signal (the length and the width of γ and β may be the same of the length and the width of the activation signal). At the stylistic element <NUM>, the conditioning feature parameters <NUM> and <NUM> are applied to the activation signal (which is the first data 59a or the 59b output by the multiplier 65a). It is to be noted, however, that the activation signal <NUM>' may be a normalized version (<NUM> at instance norm block <NUM>) of the first data <NUM>, 59a, 59b (<NUM>). It is also to be noted that the formula shown in stylistic element <NUM> (γx+β) may be an element-by-element product, and not a convolutional product or a dot product.

After stylistic element <NUM>, the signal is output. The convolutions <NUM> and <NUM> have not necessarily activation function downstream of them. It is also noted that the parameter γ (<NUM>) may be understood as a variance and β (<NUM>) as a bias. Also, block <NUM> of <FIG> may be instantiated as block <NUM> of <FIG>. Then, for example, a convolutional layer <NUM> will reduce the number of channels to <NUM> and, after that, a TanH <NUM> is performed to obtain speech <NUM>.

<FIG> shows an example of the evolution, in one of the replica <NUM> and <NUM> of one of blocks 50a-<NUM>.

The following procedure may be performed:.

The GAN discriminator <NUM> of <FIG> may be used during training for obtaining, for example, the parameters <NUM> and <NUM> to be applied to the input signal <NUM> (or a processed and/or normalized version thereof). The training may be performed before inference, and the parameters <NUM> and <NUM> may be, for example, stored in a non-transitory memory and used subsequently (however, in some examples it is also possible that the parameters <NUM> or <NUM> are calculated on line).

The GAN discriminator <NUM> has the role of learning how to recognize the generated audio signals (e.g., audio signal <NUM> synthesized as discussed above) from real input signals (e.g. real speech) <NUM>. Therefore, the role of the GAN discriminator <NUM> is mainly exerted during training (e.g. for learning parameters <NUM> and <NUM>) and is seen in counter position of the role of the GAN generator <NUM> (which may be seen as the audio generator <NUM> without the GAN discriminator <NUM>).

In general terms, the GAN discriminator <NUM> may be input by both audio signal <NUM> synthesized generated by the GAN generator <NUM>, and real audio signal (e.g., real speech) <NUM> acquired e.g., through a microphone, and process the signals to obtain a metric (e.g., loss) which is to be minimized. The real audio signal <NUM> can also be considered a reference audio signal.

During training, operations like those explained above for synthesizing speech <NUM> may be repeated, e.g. multiple times, so as to obtain the parameters <NUM> and <NUM>, for example.

In examples, instead of analyzing the whole reference audio signal <NUM> and/or the whole generated audio signal <NUM>, it is possible to only analyze a part thereof (e.g. a portion, a slice, a window, etc.). Signal portions generated in random windows (105a-105d) sampled from the generated audio signal <NUM> and from the reference audio signal <NUM> are obtained. For example random window functions can be used, so that it is not a priori pre-defined which window 105a, 105b, 105c, 105d will be used. Also the number of windows is not necessarily four, at may vary.

Within the windows (105a-105d), a PQMF (Quadrature Mirror Filter-bank (PQMF) <NUM> may be applied. Hence, subbands <NUM> are obtained. Accordingly, a decomposition (<NUM>) of the representation of the generated audio signal (<NUM>) or the representation of the reference audio signal (<NUM>) is obtained.

An evaluation block <NUM> may be used to perform the evaluations. Multiple evaluators 132a, 132b, 132c, 132d (complexively indicated with <NUM>) may be used (different number may be used). In general, each window 105a, 105b, 105c, 105d may be input to a respective evaluator 132a, 132b, 132c, 132d. Sampling of the random window (105a-105d) may be repeated multiple times for each evaluator (132a-132d). In examples, the number of times the random window (105a-105d) is sampled for each evaluator (132a-132d) may be proportional to the length of the representation of the generated audio signal or the representation of the reference audio signal (<NUM>). Accordingly, each of the evaluators (132a-132d) may receive as input one or several portions (105a-105d) of the representation of the generated audio signal (<NUM>) or the representation of the reference audio signal (<NUM>).

Each evaluator 132a-132d may be a neural network itself. Each evaluator 132a-132d may, in particular, follow the paradigms of convolutional neutral networks. Each evaluator 132a-132d may be a residual evaluator. Each evaluator 132a-132d may have parameters (e.g. weights) which are adapted during training (e.g., in a manner similar to one of those explained above).

As shown in <FIG>, each evaluator 132a-132d also performs a downsampling (e.g., by <NUM> or by another downsampling ratio). The number of channels increase for each evaluator 132a-132d (e.g., by <NUM>, or in some examples by a number which is the same of the downsampling ratio).

Upstream and/or downstream to the evaluators, convolutional layers <NUM> and/or <NUM> may be provided. An upstream convolutional layer <NUM> may have, for example, a kernel with dimension <NUM> (e.g., 5x3 or 3x5). A downstream convolutional layer <NUM> may have, for example, a kernel with dimension <NUM> (e.g., 3x3).

During training, a loss function (adversarial loss) <NUM> may be optimized. The loss function <NUM> may include a fixed metric (e.g. obtained during a pretraining step) between a generated audio signal (<NUM>) and a reference audio signal (<NUM>). The fixed metric may be obtained by calculating one or several spectral distortions between the generated audio signal (<NUM>) and the reference audio signal (<NUM>). The distortion may be measured by keeping into account:.

In examples, the adversarial loss may be obtained by randomly supplying and evaluating a representation of the generated audio signal (<NUM>) or a representation of the reference audio signal (<NUM>) by one or more evaluators (<NUM>). The evaluation may comprise classifying the supplied audio signal (<NUM>, <NUM>) into a predetermined number of classes indicating a pretrained classification level of naturalness of the audio signal (<NUM>, <NUM>). The predetermined number of classes may be, for example, "REAL" vs "FAKE".

Examples of losses may be obtained as <MAT> where:.

The spectral reconstruction loss <IMG> is still used for regularization to prevent the emergence of adversarial artifacts. The final loss is can be, for example: <MAT> where each i is the contribution at each evaluator 132a-132d (e.g.. each evaluator 132a-132d providing a different Di) and <IMG> is the pretrained (fixed) loss.

During training, there is a search foddr the minimum value of <IMG>, which may be expressed for example as <MAT>.

Other kinds of minimizations may be performed.

In general terms, the minimum adversarial losses <NUM> are associated to the best parameters (e.g., <NUM>, <NUM>) to be applied to the stylistic element <NUM>.

In the following, examples of the present disclosure will be described in detail using the accompanying descriptions. In the following description, many details are described in order to provide a more thorough explanation of examples of the disclosure. However, it will be apparent to those skilled in the art that other examples can be implemented without these specific details. Features of the different examples described can be combined with one another, unless features of a corresponding combination are mutually exclusive or such a combination is expressly excluded.

It should be pointed out that the same or similar elements or elements that have the same functionality can be provided with the same or similar reference symbols or are designated identically, with a repeated description of elements that are provided with the same or similar reference symbols or the same are typically omitted. Descriptions of elements that have the same or similar reference symbols or are labeled the same are interchangeable.

Neural vocoders have proven to outperform classical approaches in the synthesis of natural high-quality speech in many applications, such as text-to-speech, speech coding, and speech enhancement. The first groundbreaking generative neural network to synthesize high-quality speech was WaveNet, and shortly thereafter many other approaches were developed. These models offer state-of-the-art quality, but often at a very high computational cost and very slow synthesis. An abundance of models generating speech with low computational cost was presented in the recent years. Some of these are optimized versions of existing models, while others leverage the integration with classical methods. Besides, many completely new approaches were also introduced, often relying on GANs. Most GAN vocoders offer very fast generation on GPUs, but at the cost of compromising the quality of the synthesized speech.

One of the main objectives of this work is to propose a GAN architecture, which we call StyleMelGAN (and may be implemented, for example, in the audio generator <NUM>), that can synthesize very high-quality speech <NUM> at low computational cost and fast training. StyleMelGAN's generator network may contain <NUM> trainable parameters, and synthesize speech at <NUM> around <NUM>. 6x faster than real-time on CPU and more than 54x on GPU. The model may consist, for example, of eight up-sampling blocks, which gradually transform a low-dimensional noise vector (e.g., <NUM> in <FIG>) into the raw speech waveform (e.g.<NUM>). The synthesis may be conditioned on the mel-spectrogram of the target speech (or more in general by target data <NUM>), which may be inserted in every generator block (50a-<NUM>) via a Temporal Adaptive DEnormalization (TADE) layer (<NUM>, 60a, 60b). This approach for inserting the conditioning features is very efficient and, as far as we know, new in the audio domain. The adversarial loss is computed (e.g. through the structure of <FIG>, in GAN discriminator <NUM>) by an ensemble of four discriminators 132a-132d (but in some examples a different number of discriminators is possible), each operating after a differentiable Pseudo Quadrature Mirror Filter-bank (PQMF) <NUM>. This permits to analyze different frequency bands of the speech signal (<NUM> or <NUM>) during training. In order to make the training more robust and favor generalization, the discriminators (e.g. the four discriminators 132a-132d) are not conditioned on the input acoustic features used by the generator <NUM>, and the speech signal (<NUM> or <NUM>) is sampled using random windows (e.g. 105a-105d).

To summarize, StyleMelGAN is proposed, which is a low complexity GAN for high-quality speech synthesis conditioned on a mel-spectrogram (e.g. <NUM>) via TADE layers (e.g. <NUM>, 60a, 60b). The generator <NUM> may be highly parallelizable. The generator <NUM> may be completely convolutional. The aforementioned generator <NUM> may be trained adversarial with an ensemble of PQMF multi-sampling random window discriminators (e.g. 132a-132d), which may be regularized by multi-scale spectral reconstruction losses. The quality of the generated speech <NUM> can be assessed using both objective (e.g. Fréchet scores) and/or subjective assessments. Two listening tests were conducted, a MUSHRA test for the copy-synthesis scenario and a P. <NUM> ACR test for the TTS one, both confirming that StyleMelGAN achieves state-of-art speech quality.

Existing neural vocoders usually synthesize speech signals directly in time-domain, by modelling the amplitude of the final waveform. Most of these models are generative neural networks, i.e. they model the probability distribution of the speech samples observed in natural speech signals. They can be divided in autoregressive, which explicitly factorize the distribution into a product of conditional ones, and non-autoregressive or parallel, which instead model the joint distribution directly. Autoregressive models like WaveNet, SampleRNN and WaveRNN have been reported to synthesize speech signals of high perceptual quality. A big family of non-autoregressive models is the one of Normalizing Flows, e.g. WaveGlow. A hybrid approach is the use of Inverse Autoregressive Flows, which use a factorized transformation between a noise latent representation and the target speech distribution. Examples above mainly refer to autoregressive neural networks.

Early applications of GANs for audio include WaveGAN for unconditioned speech generation, and Gan-Synth for music generation. MelGAN learns a mapping between the mel-spectrogram of speech segments and their corresponding time-domain waveforms. It ensures faster than real-time generation and leverages adversarial training of multi-scale discriminators regularized by spectral reconstruction losses. GAN-TTS is the first GAN vocoder to use uniquely adversarial training for speech generation conditioned on acoustic features. Its adversarial loss is calculated by an ensemble of conditional and unconditional random windows discriminators. Parallel WaveGAN uses a generator, similar to WaveNet in structure, trained using an unconditioned discriminator regularized by a multi-scale spectral reconstruction loss. Similar ideas are used in Multiband-MelGAN, which generates each sub-band of the target speech separately, saving computational power, and then obtains the final waveform using a synthesis PQMF. Its multiscale discriminators evaluate the full-band speech waveform, and are regularized using a multi-bandscale spectral reconstruction loss. Research in this field is very active and we can cite the very recent GAN vocoders such as VocGan and HooliGAN.

<FIG> shows the generator architecture of StyleMelGAN implemented in the audio generator <NUM>. The generator model maps a noise vector z ~ <IMG>(<NUM>, I<NUM>) (indicated with <NUM> in <FIG>) into a speech waveform <NUM> (e.g. at <NUM>) by progressive up-sampling (e.g. at blocks 50a-<NUM>) conditioned on mel-spectrograms (or more in general target data) <NUM>. It uses Temporal Adaptive DE-Normalization, TADE (see blocks <NUM>, 60a, 60b), which may be a feature-wise conditioning based on linear modulation of normalized activation maps (<NUM>' in <FIG>). The modulation parameters γ (gamma, <NUM> in <FIG>) and β (beta, <NUM> in <FIG>) are adaptively learned from the conditioning features, and in one example have the same dimension as the latent signal. This delivers the conditioning features to all layers of the generator model hence preserving the signal structure at all up-sampling stages. In the formula z ~ <IMG>(<NUM>, I<NUM>), <NUM> is the number of channels for the latent noise (different numbers may be chosen in different examples). A random noise of dimension <NUM> with mean <NUM> may therefore be generated, and with an autocorrelation matrix (square <NUM> by <NUM>) is equal to the identity I. Hence, in examples the generated noise can be considered as completely decorrelated between the channels and of variance <NUM> (energy). <IMG>(<NUM>, I<NUM>) may be realized at every <NUM> generated samples (or other numbers may be chosen for different examples); the dimension may therefore be <NUM> in the time axis and <NUM> in the channel axis.

<FIG> shows the structure of a portion of the audio generator <NUM> and illustrates the structure of the TADE block <NUM> (60a, 60b). The input activation c (<NUM>') is adaptively modulated via c ⊙ γ + β, where ⊙ indicates elementwise multiplication (notably, γ and β have the same dimension of the activation map; it is also noted that c is the normalized version of the x of <FIG>, and therefore c ⊙ γ + β is the normalized version of x γ + β which could also be indicated with x ⊙ γ + β). Before the modulation at block <NUM>, an instance normalization layer <NUM> is used. Layer <NUM> (normalizing element) may normalize the first data to a normal distribution of zero-mean and unit-variance. Softmax-gated Tanh activation functions (e.g. the first instantiated by blocks 63a-64a-65a, and the second instantiated by blocks 63b-64b-65b at <FIG>) can be used, which reportedly performs better than rectified linear unit, ReLU, functions. Softmax gating (e.g. as obtained by multiplications 65a and 65b) allows for less artifacts in audio waveform generation.

<FIG> shows the structure of a portion of the audio generator <NUM> and illustrates the TADEResBlock <NUM> (which may be any of blocks 50a-<NUM>), which is the basic building block of the generator model. A complete architecture is shown in <FIG>. It includes eight up-sampling stages 50a-<NUM> (in other examples, other numbers are possible), consisting, for example, of a TADEResBlock and a layer <NUM> up-sampling the signal 79b by a factor of two, plus one final activation module <NUM> (in <FIG>). The final activation comprises one TADEResBlock <NUM> followed by a channel-change convolutional layer <NUM>, e.g. with tanh non-linearity <NUM>. This design permits to use, for example, a channel depth of <NUM> for the convolution operations, hence saving complexity. Moreover, this up-sampling procedure permits to keep the dilation factor lower than <NUM>.

<FIG> shows the architecture of a filter bank random window discriminators (FB-RWDs). StyleMelGAN may use multiple (e.g. four) discriminators 132a-132d for its adversarial training, wherein in examples the architecture of the discriminators 132a-132d has no average pooling down-sampling. Moreover, each discriminator (132a-132d) may operate on a random window (105a-105d) sliced from the input speech waveform (<NUM> or <NUM>). Finally, each discriminator (132a-132d) may analyze the sub-bands <NUM> of the input speech signal (<NUM> or <NUM>) obtained by an analysis PQMF (e.g. <NUM>). More precisely we may use, in examples, <NUM>, <NUM>, <NUM>, and <NUM> sub-bands calculated respectively from select random segments of respectively <NUM>, <NUM>, <NUM>, and <NUM> samples extracted from a waveform of one second. This enables a multi-resolution adversarial evaluation of the speech signal (<NUM> or <NUM>) in both time and frequency domains.

Training GANs is known to be challenging. Using random initialization of the weights (e.g. <NUM> and <NUM>), the adversarial loss (e.g. <NUM>) can lead to severe audio artifacts and unstable training. To avoid this problem, the generator <NUM> may be firstly pretrained using only the spectral reconstruction loss consisting of error estimates of the spectral convergence and the log-magnitude computed from different STFT analyses. The generator obtained in this fashion can generate very tonal signals although with significant smearing in high frequencies. This is nonetheless a good starting point for the adversarial training, which can then benefit from a better harmonic structure than if it started directly from a complete random noise signal. The adversarial training then drives the generation to naturalness by removing the tonal effects and sharpening the smeared frequency bands. The hinge loss <NUM> is used to evaluate the adversarial metric, as can be seen in equation <NUM> below. <MAT> where x is the real speech <NUM>, z is the latent noise <NUM> (or more in general the input signal), and s is the mel-spectrogram of x (or more in general the target signal <NUM>). It should be noted that the spectral reconstruction loss <IMG> (<NUM>) is still used for regularization to prevent the emergence of adversarial artifacts. The final loss (<NUM>) is according to equation <NUM>, which can be seen below.

Weight normalization may be applied to all convolution operations in G (or more precisely the GAN generator <NUM>) and D (or more precisely the discriminator <NUM>). In experiments, StyleMelGAN was trained using one NVIDIA Tesla V100 GPU on the LJSpeech corpus at <NUM>. The log-magnitude mel-spectrograms is calculated for <NUM> mel-bands and is normalized to have zero mean and unit variance. This is only one possibility of course; other values are equally possible. The generator is pretrained for <NUM> steps using Adam optimizer with learning rate (Irg) of <NUM>-<NUM>, β = {<NUM>, <NUM>}. When starting the adversarial training, the learning rate of G (Irg) is set to <NUM> * <NUM>-<NUM> and use FB-RWDs with the Adam optimizer with a discriminator learning rate (Ird) of <NUM> * <NUM>-<NUM> and the same β. The FB-RWDs repeat the random windowing for <NUM>/window_length, i.e. one second per window length, times at every training step to support the model with enough gradient updates. A batch size of <NUM> and segments with a length of <NUM>, i.e. one second, for each sample in the batch are used. The training lasts for about one and a half million steps, i.e. <NUM>. <NUM> steps.

The following lists the models used in experiments:.

Objective and subjective evaluations of StyleMelGAN against pretrained baseline vocoder models listed above have been performed. The subjective quality of the audio TTS outputs via a P. <NUM> listening test performed by listeners were evaluated in a controlled environment. The test set contains unseen utterances recorded by the same speaker and randomly selected from the LibriVox online corpus. These utterances test the generalization capabilities of the models, since they were recorder in slightly different conditions and present varying prosody. The original utterances were resynthesized using the GriffinLim algorithm and used these in the place of the usual anchor condition. This favors the use of the totality of the rating scale.

Traditional objective measures such as PESQ and POLQA are not reliable to evaluate speech waveforms generated by neural vocoders. Instead, the conditional Fréchet Deep Speech Distances (cFDSD) are used. The following cFDSD scores for different neural vocoders show that StyleMelGAN significantly outperforms the other models.

It can be seen that that StyleMelGAN outperforms other adversarial and non-adversarial vocoders.

A MUSHRA listening test with a group of <NUM> expert listeners was conducted. This type of test was chosen, because this allows to more precisely evaluate the quality of the generated speech. The anchor is generated using the Py-Torch implementation of the Griffin-Lim algorithm with <NUM> iterations. <FIG> shows the result of the MUSHRA test. It can be seen that StyleMelGAN significantly outperforms the other vocoders by about <NUM> MUSHRA points. The results also show that WaveGlow produces outputs of comparable quality to WaveNet, while being on par with Parallel WaveGAN.

The subjective quality of the audio TTS outputs can be evaluated via a P. <NUM> ACR listening test performed by <NUM> listeners in a controlled environment. The Transformer. v3 model of ESPNET can be used to generate mel-spectrograms of transcriptions of the test set. The same Griffin-Lim anchor can also be added, since this favors the use of the totality of the rating scale.

The following P800 mean opinion scores (MOS) for different TTS systems show the similar finding that StyleMelGAN clearly outperforms the other models:.

The following shows the generation speed in real-time factor (RTF) with number of parameters of different parallel vocoder models. StyleMelGAN provides a clear compromise between generation quality and inference speed.

Here is given, the number of parameters and real-time factors for generation on a CPU (e.g. Intel Core i7-<NUM><NUM>) and a GPU (e.g. Nvidia GeForce GTX1060) for various models under study.

Finally, <FIG> shows results of a MUSHRA expert listening test. It can be seen that StyleMelGAN outperforms state-of-the-art models.

This work presents StyleMelGAN, a lightweight and efficient adversarial vocoder for high-fidelity speech synthesis. The model uses temporal adaptive normalization (TADE) to deliver sufficient and accurate conditioning to all generation layers instead of just feeding the conditioning to the first layer. For adversarial training, the generator competes against filter bank random window discriminators that provide multiscale representations of the speech signal in both time and frequency domains. StyleMelGAN operates on both CPUs and GPUs by order of magnitude faster than real-time. Experimental objective and subjective results show that StyleMelGAN significantly outperforms prior adversarial vocoders as well as auto-regressive, flow-based and diffusion-based vocoders, providing a new state-of-the-art baseline for neural waveform generation.

To conclude, the embodiments described herein can optionally be supplemented by any of the important points or aspects described here. However, it is noted that the important points and aspects described here can either be used individually or in combination and can be introduced into any of the embodiments described herein, both individually and in combination.

Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a device or a part thereof corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding apparatus or part of an apparatus or item or feature of a corresponding apparatus.

The program code may for example be stored on a machine-readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine-readable carrier.

The methods described herein, or any parts of the methods described herein, may be performed at least partially by hardware and/or by software.

Claim 1:
Audio generator (<NUM>), configured to generate an audio signal (<NUM>) from an input signal (<NUM>) and target data (<NUM>), the target data (<NUM>) representing the audio signal (<NUM>), comprising:
A first processing block (<NUM>, <NUM>, 50a-<NUM>), configured to receive first data (<NUM>, 59a) derived from the input signal (<NUM>) and to output first output data (<NUM>), wherein the first output data (<NUM>) comprises a plurality of channels (<NUM>), and
a second processing block (<NUM>), configured to receive, as second data, the first output data (<NUM>) or data derived from the first output data (<NUM>),
wherein the first processing block (<NUM>) comprises for each channel of the first output data:
a conditioning set of learnable layers (<NUM>, <NUM>, <NUM>) configured to process the target data (<NUM>) to obtain conditioning features parameters (<NUM>, <NUM>); and
a styling element (<NUM>), configured to apply the conditioning feature parameters (<NUM>, <NUM>) to the first data (<NUM>, 59a) or normalized first data (<NUM>, <NUM>'); and
wherein the second processing block (<NUM>) is configured to combine the plurality of channels (<NUM>) of the second data (<NUM>) to obtain the audio signal (<NUM>),
characterized in that the first processing block (<NUM>, <NUM>, 50a-<NUM>) further comprises:
a further set of learnable layers (61a, 62a, 61b, 62b), configured to process data derived from the first data (<NUM>, <NUM>, 59a, 59b) using a second activation function (63a, 64a, 63b, 64b),
wherein the second activation function (63a, 64a, 63b, 64b) is a gated activation function.