Patent Description:
Embodiments of the invention refer to an audio decoder or generator, configured to decode or generate an audio signal from an input signal and a bitstream, the bitstream representing the audio signal. Further embodiments refer to methods for decoding or generating an audio signal, and methods for training an audio decoder or generator. Further embodiments refer to a computer program product.

Neural vocoders have proven to outperform classical approaches in the synthesis of natural high-quality speech in many applications, such as text-to-speech, speech coding, and speech enhancement. The first groundbreaking generative neural network to synthesize high-quality speech was WaveNet [<NUM>], and shortly there-after many other approaches were developed [<NUM>, <NUM>]. These models offer state-of-the-art quality, but often at a very high computational cost and very slow synthesis. An abundance of models generating speech with lowered computational cost was presented in the recent years. Some of these are optimized versions of existing models, while others leverage the integration with classical methods [<NUM>]. Be-sides, many completely new approaches were also introduced, often relying on GANs.

Existing solutions combining speech coding and neural vocoders are:.

GAN can't be directly used in speech coding application. The present invention aims to address this challenge.

<CIT> discloses an artificial-intelligence-based audio coding technique.

Embodiments according to the present invention will subsequently be described taking reference to the enclosed figures in which:.

In the figures, similar reference signs denote similar elements and features.

<FIG> shows an example of a vocoder system in which an audio signal <NUM> is encoded and an output audio signal <NUM> is decoded. In general, the audio signal is here considered to be speech (but could also be other kind of sound). The audio signal <NUM> is encoded by an encoder <NUM> to obtain a bitstream <NUM> (which may include data 3a and 3b, see below). The encoder <NUM> may be an LPC (linear prediction coding) encoder, or Spectrogrum-based coder or Cepstrum-based coder. The bitstream <NUM> may include several parameters associated to the audio signal <NUM> (and to the audio signal to be generated at the decoder). The bitstream <NUM> may be compressed with respect to the original version of the audio signal <NUM>. The bitstream <NUM> is decoded by a decoder <NUM>. The decoder <NUM> may also be considered as an audio generator, since it generates the audio signal <NUM> which in general shall be a representation of the original audio signal <NUM> and of the bitstream <NUM>. The decoder <NUM> is discussed in great detail below.

Here, the architecture of the encoder <NUM> is not restricted. The encoder <NUM> may perform operations like feature extraction, quantization and coding which are per se known. The bitstream <NUM> includes some parameters. The encoder <NUM> may be a straightforward, normal LPC encoder and the bitstream <NUM> may therefore an LPC bitstream. Below in table <NUM>, a possible allocation of bits for the bitstream <NUM> is provided. LPC coding is a known coding technique according towhich parameters are provided which include filter data (e.g. formant data) and excitation data (or stimulus data).

The encoder <NUM> may estimate the formants through by minimizing a prediction error power and estimating an auto-regressive (AR) model with for example the autocorrelation-based by optimizing the Yule-Walker equation system with for example the recursive method Levinson-Durbin and obtain parameters associated to the formants (filter data 3a). As an alternative, the spectral envelope and the format structure can be directly computed in frequency domain by smoothing the spectrum magnitudes or computed energy in frequency subbands. In parallel, the encoder <NUM> may subtract the formants by inverse filtering of the AR model for example (also known as LPC analysis filter) from the audio signal <NUM> and also obtain residual parameters (excitation data 3b). The filter parameters or data 3a may be understood as representing a filter generated by the positions taken by the muscles of the human mouth, which modify the excitation (represented by the excitation data 3b) consisting of the air flow passed through or stopped by the glottis. Basically, the LPC encoder <NUM> encodes the human voice by taking into account the physiology of the human voice production. In general, the LPC decoder <NUM> may obtain both the filter data 3a and the other data 3b (e.g. excitation data, pitch data. ) from the bitstream <NUM> and use them for generating the audio signal <NUM>. In particular, the other data (e.g. excitation data 3b) are decoded and reconstructed, and the filter data or spectral envelope data 3a are used for filtering (e.g., through a 16th-order prediction coefficient). The cepstrum is another representation of the speech, which decomposes into a sum a convolution that means a filtered signal. The filter data or spectral envelope data 3a and the excitation data (or more in general the other data 3b) can be deduced easily in such a domain. The first coefficients (MFCCs) of the cepstrum represent the spectral envelope and the formantic structure, and are called in the subsequent text, the cepstrum data. The cepstrum data can be seen as representing the filter coefficients of AR/MA system with zeros and poles. From the cepstrum data it is possible to obtain filter data or spectral envelope data 3a and/or the other data 3b (e.g. excitation data, pitch data etc.). Here, the encoder <NUM> may be a LPC encoder, spectrogrum-based encoder, cepstrum-based encoder, etc..

It has been understood that it is possible to decode a bitstream like the bitstream <NUM> according to the LPC technique, spectrogrum-based technique, cepstrum-based technique, etc., for generating audio signal <NUM> properly, in particular, by using learnable techniques (such as neural network techniques), and in particular using generative neural networks.

In general terms, the bitstream <NUM> (whether encoded though LPC, a spectrogrum-based technique, a cepstrum-based technique, or another technique) may include, in some examples:.

The filter data 3a may include Mel-frequency cepstrum or cepstral coefficients (MFCCs) data, even though other types of filter data (e.g. spectral envelope data) may be adopted.

Among the other data 3b, they may comprise the pitch data (e.g., pitch lag of the audio signal <NUM>). For example, the pitch data may include the pitch correlation (e.g. encoded in <NUM> bits) and/or pitch correlation (e.g. encoded in <NUM> bits). See, for example, Table <NUM> below.

Here below, the written proposed technique of an audio decoder for decoding a bitstream (e.g. <NUM>) representing the audio signal (e.g. <NUM>, <NUM>), the audio signal being subdivided in a sequence of frames, the audio decoder (e.g. <NUM>) comprising:.

There is also proposed a method for decoding an audio signal (e.g. <NUM>, <NUM>) from a bitstream (e.g. <NUM>) representing the audio signal (e.g. <NUM>), the method using an input signal (e.g. <NUM>), the audio signal (e.g. <NUM>, <NUM>) being subdivided into a plurality of frames, the method comprising:.

<FIG> shows an example of the decoder (generator) <NUM> (other examples are provided in <FIG> and <FIG>). The bitstream <NUM> comprises filter data (e.g. spectral envelope data) 3a and other data 3b (e.g. pitch data, and/or at least one of excitation data stimulus data, residual data, stimulus data, excitation data, harmonicity data, periodicity data, long-term prediction data). The other data 3b may be used in a first processing block <NUM> for generating first output data <NUM> (second data). The first output data <NUM> is in a plurality of channels. The first output data <NUM> is provided to a second processing block <NUM> which combines the plurality of channels of the first output data <NUM> providing an output audio signal <NUM> in one signal channel.

It is to be noted that, in general, the audio signal <NUM> (as well as the original audio signal <NUM> and its encoded version, the bitstream <NUM>) are subdivided according to a sequence of frames (in some examples, the frames do not overlap with each other, while in some other examples they may overlap). Each frame includes a sequence of samples. For example, each frame may be subdivided into <NUM> samples (but other resolutions are possible). A frame can be long, for example, <NUM> (in other cases <NUM> or <NUM> or other time lengths may be used), while the sample rate may be for example <NUM> (in other case <NUM>, <NUM> or <NUM>, or any other sampling rates), and the bit-rate for example, <NUM> kbps (kilobit per second). It is also noted that the multiple frames may be grouped in one single packet, e.g., for transmission or for storage. While the time length of one frame is in general considered fixed, the number of samples per frame may vary, and upsampling operations may be performed.

The sample-by-sample branch 10b may be updated for each sample at the output sampling rate and/or for each sample at a lower sampling-rate than the final output sampling-rate, e.g. using the excitation data or pitch data 3b or another input, e.g. noise <NUM> or another input taken from an external or internal source.

It is also to be noted that the bitstream <NUM> is here considered to encode mono signals and also the output audio signal <NUM> and the original audio signal <NUM> are considered to be mono signals. In the case of stereo signals or multi-channel signals like loudspeaker signal or Ambisonics signal for example, then all the techniques here are repeated for each audio channel (in stereo case, there are two input audio channels <NUM>, two output audio channels <NUM>, etc.).

In this document, when referring to "channels", it has to be understood in the context of convolutional neural networks, according to which a signal is seen as an activation map which has at least two dimensions:.

The first processing block <NUM> may operate like a conditional network, for which data from the bitstream <NUM> (e.g. filter data 3a, e.g. spectral envelope data) are provided for generating conditions which modify input data <NUM> (input signal). The input data <NUM> (in any of its evolutions) will be subjected to several processings, to arrive at the output audio signal <NUM>, which is intended to be a version of the original audio signal <NUM>. Both the conditions, the input data <NUM> and their subsequent processed versions may be represented as activation maps which are subjected to learnable layers, e.g. by convolutions. Notably, during its evolutions towards the speech <NUM>, the signal is subjected to an upsampling (e.g. from <NUM> sample to thousands of samples in <FIG>), but its number of channels is reduced (e.g. from <NUM> channels to <NUM> single channel in <FIG>).

In the sample-by-sample branch 10b, first data <NUM> may be obtained, for example, from the data encoded in the bitstream <NUM> such as the filter data (e.g. spectral envelope data) 3a ("first option"), from another input (such as noise or a signal from an external signal) ("second option"), or from other internal or external source(s). The first data <NUM> may be considered the input of the first processing block <NUM> and may be an evolution of the input signal <NUM> (or may be the input signal <NUM>). The first data <NUM> may be considered, in the context of conditional neural networks, as a latent signal or a prior signal. Basically, the first data <NUM> is modified according to the conditions set by the first processing block <NUM> to obtain the first output data <NUM>. The first data <NUM> is in multiple channels, but in one single sample. Also, the first data <NUM> as provided to the first processing block <NUM> may have the one sample resolution, but in multiple channels. The multiple channels may form a set of parameters, which may be associated to the coded parameters encoded in the bitstream <NUM>. During the processing in the first processing block <NUM> the number of samples per frame increases from a first number to a second, higher number (i.e. the bitrate increases from a first bitrate to a second, higher bitrate). On the other side, the number of channels may be reduced from a first number of channels to a second, lower number of channels. The conditions used in the first processing block (which are discussed in great detail below) can be indicated with <NUM> and <NUM> and are generated by target data <NUM>, which in turn are obtained from the bitstream (e.g., from filter data, such as spectral envelope data) 3a. It will be shown that also the conditions <NUM> and <NUM>, and/or the target data <NUM> may be subjected to upsampling, to conform (e.g. adapt) to the dimensions of the versions of the target data <NUM>. The unit that provides the first data <NUM> (either from an internal source, an external source, the bitstream <NUM>, etc.) is here called first data provisioner <NUM>.

As can be seen from <FIG>, the first processing block <NUM> includes a preconditioning learnable layer <NUM>. The preconditioning learnable layer <NUM> generate target data <NUM> for each frame. The target data <NUM> is at least <NUM>-dimensional: there are multiple samples for each frame in the abscissa direction and multiple channels for each frame in the ordinate direction. The target data <NUM> may be in the form of a spectrogram, which may be a mel-spectrogram, e.g. in case the frequency scale is non-uniform and is motivated by perceptual principles. In case the sampling rate corresponding to conditioning learnable layer to be fed is different from the frame rate, the target data <NUM> may be the same for all the samples of the same frame at a layer sampling rate. Another up-sampling strategy can also be applied. The target data <NUM> are provided to at least one conditioning learnable layer, which is here indicated as having the layer <NUM>, <NUM>, <NUM> (also see <FIG> and the discussion below). The conditioning learnable layer(s) <NUM>, <NUM>, <NUM> generate(s) conditions (some of which may be indicated as β and γ or the numbers <NUM> and <NUM>), which are also called conditioning feature parameters to be applied to the first data <NUM>, and any upsampled data derived from the first data. The conditioning learnable layer(s) <NUM>, <NUM>, <NUM> may be in the form of matrixes with multiple channels and multiple samples for each frame. The first processing block <NUM> may include a denormalization (or styling element) block <NUM>. For example, the styling element <NUM> applies the conditioning feature parameters <NUM> and <NUM> to the first data <NUM>. An example may be element wise multiplication of the values of the first data by the condition β (which may operates as bias) and an addition with the condition γ (which may operate as multiplier). The styling element <NUM> may produce a first output data <NUM> sample by sample.

The decoder <NUM> includes a second processing block <NUM>. The second processing block <NUM> combines the plurality of channels of the first output data <NUM>, to obtain the output audio signal <NUM>.

Reference is now mainly made to <FIG>. A bitstream <NUM> is subdivided onto a plurality of frames (here three frames are shown). Time evolves from left to right. Each frame is subdivided into a plurality of samples (not shown) in the abscissa direction. In the ordinate direction, many channels are provided, which may include LPC parameters. The LPC parameters may include excitation data 3b (e.g., pitch data, correlation data, residual data, stimulus data) and filter data 3a (e.g., spectral envelope data), such as MFCCs. The filter data 3a (or more in general data from the bitstream <NUM>) may be used by the preconditioning learnable layer(s) <NUM> to generate a spectrogram (e.g., a mel-spectrogram) or, more in general, target data <NUM>. The target data <NUM> may represent one single frame and evolve, in the abscissa direction (from left to right) with time. Several channels may be in the ordinate direction for each frame. For example, different coefficients will take place in different entries of each column in association with coefficients associated with the frequency bands. Conditioning learnable layer(s) <NUM>, <NUM>, <NUM>, conditioning feature parameter(s) <NUM>, <NUM> (β and γ) are present. The abscissa of β and γ is associated to different samples, while the ordinate is associated to different channels. In parallel, the first data provisioner <NUM> may provide the first data <NUM>. A first data <NUM> may be generated for each sample and may have many channels. At the styling element <NUM> (and more in general, at the first conditioning block <NUM>) the conditioning feature parameters β and γ (<NUM>, <NUM>) may be applied to the first data <NUM>. For example, an element-by-element multiplication may be performed between a column of the styling conditions <NUM>, <NUM> (conditioning feature parameters) and the first data <NUM> or an evolution thereof. It will be shown that this process may be reiterated many times. Summarizing, the preconditioning learnable layer(s) <NUM> may obtain a filter data 3a associated to a frame and output a spectrogram for that frame.

As clear from above, the first output data <NUM> generated by the first processing block <NUM> may be obtained as a <NUM>-dimensional matrix (or even a matrix with more than two dimensions) with samples in abscissa and channels in ordinate. At the second processing block <NUM>, the audio signal <NUM> is generated having one single channel and multiple samples. More in general, at the second processing block <NUM>, the number of samples per frame (bitrate) of the first output data <NUM> may evolve from the second number of samples per frame (second bitrate) to a third number of samples per frame (third bitrate), higher than the second number of samples per frame (second bitrate). On the other side, the number of channels of the first output data <NUM> may evolve from a second number of channels to a third number of channels, which is less than the second number of channels. Said in other terms, the bitrate (third bitrate) of the output audio signal <NUM> is higher than the bitrate of the first data <NUM> (first bitrate) and of the bitrate (second bitrate) of the first output data <NUM>, while the number of channels of the output audio signal <NUM> is lower than the number of channels of the first data <NUM> (first number of channels) and of the number of channels (second number of channels) of the first output data <NUM>.

It will be shown that many of the present techniques may make use of convolutional neural networks (or other learnable layers) which are adapted for streaming techniques. <FIG> shows an example of a convolution operation using a 3x3 convolutional kernel which, for example, may slide from left to right different sets of coded parameters according to a "sliding convolution". Each set of coded parameters of the bitstream <NUM> may be associated to one single frame. The input sequence may, for example, include multiple parameters juxtaposed with each other in the ordinate direction. The convolutional kernel, therefore, may slide from left to right to obtain a convolutional output. From that, a dense neural network layer may be obtained. We may imagine that in the input sequence, there may be either excitation data 3b or filter data 3a, or both.

<FIG> shows a set of coded parameters of the bitstream <NUM> which are obtained in different times for different frames (frame bt-<NUM> obtained before frame bt, which is obtain before frame it+<NUM>). At time t+<NUM>, a new set of coded parameters it+<NUM> (e.g., LPC coefficients for one frame) is juxtaposed to a set of coded parameters including bt-<NUM> and bt previously juxtaposed to each other. A convolution may be directly obtained from the newly arrived set of coded parameters and the two proceeding sets of coded parameters, so as to obtain, in state 2i, a convolutional value valid for the last sets of coded parameters (e.g., the last three frames). After that, a dense convolutional layer may be obtained. The buffer may be updated with the frames bt-<NUM> and bt.

The models processing the of coded parameters frame-by-frame by juxtaposing the current frame to the previous frames already in the state are also called streaming or stream-wise models and may be used as convolution maps for convolutions for real-time and streamwise applications like speech coding.

Examples of convolutions are discussed here below and it can be understood that they may be used at any of the preconditional learnable layers <NUM>, at least one conditional learnable layers <NUM>, <NUM>, <NUM>, and more in general, in the first processing block <NUM> (<NUM>). In general terms, the arriving set of conditional parameters (e.g., for one frame) is stored in a queue (not shown) to be subsequently processed by the first or second processing block while the first or second processing block, respectively, processes a previous frame.

<FIG> shows a schematization <NUM> that can be used for implementing the present techniques. The so-called "spectrogram enforcement" technique can be used (see also below, in section <NUM> below). The preconditional learnable layer(s) <NUM> may include at least a concatenation step <NUM> in which LPC parameters (or more in general, parameters) of the bitstream <NUM> are concatenated through each other. The LPC parameters may include both filter data 3a (such as mfcc, Mel-frequency cepstrum, coefficients) and other data data 3b (such as pich data, e.g. normalized correlation and pitch). For example, the operations shown in <FIG> may be used by juxtaposing several parameters associated to different frames (e.g., consecutive frames). The output <NUM> of the concatenation <NUM> may be input to a preconditional convolution <NUM>, which may provide an output <NUM> which a preconditional output. The preconditional output <NUM> may be input to a feature convolution <NUM>, which may provide a preconditional output <NUM>. Subsequently, a dense layer <NUM> may be provided, to obtain a spectrogram (e.g., a mel-spectrogram), which is target data <NUM>. Other strategies may be used.

The generator network <NUM> of <FIG> may play the role of the conditional learnable layer(s) <NUM>, <NUM>, <NUM> and the denormalization (styling element) block <NUM>. The output of the generator network <NUM> may be the first output data <NUM> and may be constituted by multiple waveforms. After that, the second processing block <NUM> may perform a synthesis (e.g., a PQMF synthesis) <NUM> from which an input audio signal <NUM> is obtained.

<FIG> shows another schematization <NUM> (a code being disclosed below in section <NUM>). The schematization <NUM> may include at least one preconditional learnable layer(s) <NUM> may include a first preconditional layer <NUM>, which may output a preconditional output <NUM>. The preconditional output <NUM> may be input to a second preconditional layer <NUM> which may output a preconditional output <NUM>. A preconditional dense layer <NUM> may be input with the preconditional output <NUM>. The output of the preconditional dense layer <NUM> may be the target data <NUM> discussed above (e.g. a spectrogram, mel-spectrogram, etc.). A generator network layer <NUM> may play the role of layer(s) <NUM>, <NUM>, <NUM> and <NUM> (see above and below). A first output data <NUM> may be generated, e.g., in the form of multiple waveforms. After that, a second processing block <NUM> may be used. The output audio signal <NUM> (which in an output frame) is therefore generated and can be playback.

<FIG> shows a third schematization <NUM> in which the first data <NUM> (which may be considered to be the same as the input data <NUM>) is obtained from the bitstream <NUM>. Here, filter data 3a are MFCCs, which are inputted into a pre-conditional convolution layer <NUM>. The output <NUM> may be the target data <NUM>. The target data <NUM> (<NUM>) may be inputted to the generator network <NUM> which may be embodied by layers <NUM>, <NUM>, <NUM> and <NUM>. The target data <NUM> (<NUM>) may set the conditions onto the generator network <NUM>. The generator network <NUM> may also be inputted by input data <NUM> (first data <NUM>) which are shown here as a multiplication <NUM> (scaling layer) between the pitch_embedding and the pitch_core, to obtain a multiplied value <NUM> which is used as first data <NUM> (latent, prior) for the generator network <NUM>. The pitch_embedding and the pitch_core may be understood as part of the pitch data 3b (other data). Both the filter data 3a (MFCC), the pitch_embedding the pitch_core (pitch data 3b) may be obtained from the bitstream <NUM>. By operating as a conditional network, the generator network <NUM> may provide a first output data <NUM> (multiband_waveforms) which may be inputted into the second processing block <NUM>. The second processing block <NUM> may perform, for example, a PQMF synthesis <NUM>. The output out_frame of the PQMF synthesis <NUM> may be the output audio signal <NUM>. The schematization <NUM> of <FIG> is also discussed below in section <NUM>.

In the examples of <FIG> and <FIG> and <FIG>, at least one of the layers <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> may be or imply a learnable layer. At least one of the layers <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> may be or imply a non-conditional learnable layer. At least one of the layers <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> may be a convolutional layer (e.g. using a kernel learned by training). At least one layer (e.g. at least one of layers <NUM>, <NUM>, <NUM>, <NUM>) may be activated by a ReLu activation function (e.g. a leaky ReLu). At least one layer (e.g. at least one of layers <NUM>, <NUM>, <NUM>) may have no activation function. Different combinations may however be possible.

At least one layer (and in particular the layer <NUM> may be subjected to weight normalization). Alternatively, any normalization can be employed as batch, instance or layer normalization. The weight normalization may be characterized by the fact that it separates the weight vector from its direction. This has a similar effect as in batch normalization with variance. The only difference is in variation instead of direction. Any of the learnable blocks <NUM>, <NUM>, <NUM>, <NUM>, <NUM> (or any combination thereof) may be or constitute the preconditional learnable layers <NUM>.

At least one of signals <NUM>, <NUM>, <NUM>, <NUM> for schematization <NUM> and <NUM>, <NUM>, <NUM>, <NUM> for schematization <NUM>, or <NUM> and <NUM> for schematization <NUM> may be understood as target data or target data predecessor predecessors taken from the bitstream <NUM>, and may be used to obtain the conditions <NUM>, <NUM> (conditioning feature parameters) to be applied to the first data <NUM> or any processed version thereof. The first data <NUM> are here shown as Gaussian noise <NUM> in <FIG> and <FIG>, but they could also be obtained from excitation data 3b (see for example in <FIG> and in section <NUM> below).

In some examples, (which may be embodied for example by the code in subsection <NUM> below) the noise (used in the "second option") may be substituted by pitch data 3b such as the pitch multiplied by the correlation value of the pitch, or by other data: the so-called "first option" of <FIG> is obtained. Between schematizations <NUM> and <NUM>, we see that schematization <NUM> does not have a concatenation layer <NUM>, but we have a scaling layer <NUM> in which a pitch data <NUM> is obtained by multiplying a pitch embedding value and a pitch correlation. In this case, there is no necessity of the concatenation, since the two relevant information are already combined by a multiplication operation at <NUM>. Notably, in the schematization <NUM> of <FIG>, the multiplication <NUM> is performed to obtain a value <NUM> which is the first data <NUM> (or input data <NUM>) to the generator network <NUM>, and is used as latent (or prior); contrary thereto, in the schematization <NUM> of <FIG>, the multiplication <NUM> obtains a multiplied value which is provided to the preconditional layer(s) <NUM>.

In the following there is a discussion on the operations mainly performed in blocks downstream to the preconditioning learnable layer(s) <NUM>. Therefore, most of the subsequent discussion takes into account the target data as already obtained from the preconditioning learnable layer(s) <NUM>, and which are applied to the conditioning learnable layer(s) <NUM>-<NUM> (the conditioning learnable layer(s) <NUM>-<NUM> being, in turn, applied to the stylistic element <NUM>). It is reminded that blocks <NUM>-<NUM> and <NUM> may be embodied by the generator network layer <NUM>. The generator network layer may include a plurality of learnable layers (e.g. a plurality of blocks 50a-<NUM>, see below).

<FIG> (and its embodiment in <FIG>) shows an example of an audio decoder (generator) <NUM> which decodes (generates, synthesizes) an audio signal (output signal) <NUM> from the bitstream <NUM>, e.g. according to the present techniques (also called StyleMelGAN). The output audio signal <NUM> is generated based on the input signal <NUM> (also called latent signal and which may be noise, e.g. white noise ("first option"), or which can be obtained from parameters of the bitstream <NUM>, such as filter data 3a, such as spectral envelope data) and target data <NUM> (also called "input sequence"), which may be obtained from the bitstream <NUM> (for example, from other parameters 3b such as least excitation data or a pitch data or harmonicity data or periodicity data or long-term prediction data encoded in the bitstream). The target data <NUM> may, as explained above, comprise (e.g. be) a spectrogram (e.g., a mel-spectrogram), the mel-spectrogram providing mapping, for example, of a sequence of time samples onto mel scale (e.g. obtained from the preconditioning learnable layer(s) <NUM> and/or by at least one of the layers <NUM>, <NUM>, <NUM>, <NUM> and/or by at least one of the layers <NUM>, <NUM>, <NUM> or <NUM>, as discussed above). The target data <NUM> is in general to be processed, in order to obtain a speech sound recognizable as natural by a human listener. The input signal <NUM> may be or be obtained from the bitstream <NUM> (e.g. from other information 3b) ("first option"), or may be ("second option") noise (which as such carries no useful information) (other options are possible). In the decoder <NUM>, the first data <NUM> obtained from the input (obtained from the bitstream <NUM> or from noise or from an external source) is styled (e.g. at block <NUM>) to have a vector with the acoustic features conditioned by the target data <NUM>. At the end, the output audio signal <NUM> will be understood as speech by a human listener. The input vector <NUM> (obtained from the bitstream <NUM> or from noise or from an external source) may be, like in <FIG>, a 128x1 vector (one single sample, e.g. time domain samples or frequency domain samples, and <NUM> channels). A different length of the input vector <NUM> could be used in other examples. The input vector <NUM> is processed (under the conditioning of the target data <NUM> obtained from the bitstream <NUM> through the preconditioning layers <NUM>) in the first processing block <NUM>. The first processing block <NUM> includes at least one, e.g. a plurality, of processing blocks <NUM> (e.g. 50a. In <FIG> there are shown eight blocks 50a. <NUM> (each of them is also identified as "TADEResBlock"), even though a different number may be chosen in other examples. The processing blocks 50a, 50b, etc. provide a gradual upsampling of the signal which evolves from the input signal <NUM> to the final audio signal <NUM> (e.g., at least some processing blocks 50a, 50b, 50c, 50d, 50e increases the sampling rate).

The blocks 50a-<NUM> may form one single block <NUM> (e.g. the one shown in <FIG>). In the first processing block <NUM>, a conditioning set of learnable layers (e.g., <NUM>, <NUM>, <NUM>, but different numbers are possible) may be used to process the target data <NUM> and the input signal <NUM> (e.g., first data <NUM>). Accordingly, conditioning feature parameters <NUM>, <NUM> (also referred to as gamma, γ, and beta, β) are obtained, e.g. by convolution, during training. The learnable layers <NUM>-<NUM> may therefore be part of a weight layer of a learning network. As explained above, the first processing block <NUM>, <NUM> may include at least one styling element <NUM> (normalization block <NUM>). The at least one styling element <NUM> outputs the first output data <NUM> (when there are a plurality of processing blocks <NUM>, a plurality of styling elements <NUM> may generate a plurality of components, which may be added to each other to obtain the final version of the first output data <NUM>). The at least one styling element <NUM> may apply the conditioning feature parameters <NUM>, <NUM> to the input signal <NUM> (latent) or the first data <NUM> obtained from the input signal <NUM>.

The first output data <NUM> has a plurality of channels.

The audio decoder <NUM> includes a second processing block <NUM> (in <FIG> shown as including the blocks <NUM>, <NUM>, <NUM>, <NUM>). The second processing block <NUM> is configured to combine the plurality of channels <NUM> of the first output data <NUM> (inputted as second input data or second data), to obtain the output audio signal <NUM> in one single channel, but in a sequence of samples.

The "channels" are not to be understood in the context of stereo sound, but in the context of neural networks (e.g. convolutional neural networks). For example, the input signal (e.g. latent noise) <NUM> may be in <NUM> channels (in the representation in the time domain), since a sequence of channels are provided. For example, when the signal has <NUM> samples and <NUM> channels, it may be understood as a matrix of <NUM> columns and <NUM> rows, while when the signal has <NUM> samples and <NUM> channels, it may be understood as a matrix of <NUM> columns and <NUM> rows (other schematizations are possible). Therefore, the generated audio signal <NUM> may be understood as a mono signal. In case stereo signals are to be generated, then the disclosed technique is simply to be repeated for each stereo channel, so as to obtain multiple audio signals <NUM> which are subsequently mixed.

At least the original input signal <NUM> and/or the generated speech <NUM> may be a vector. To the contrary, the output of each the blocks <NUM> and 50a-<NUM>, <NUM>, <NUM> may have in general a different dimensionality. The first data <NUM> may have a first dimension (e.g. the samples dimension) or at least one dimension lower than that of the audio signal <NUM>. The first data <NUM> may have a total number of samples across all dimensions lower than the audio signal <NUM>. The first data <NUM> may have one dimension (e.g. the samples dimension) lower than the audio signal <NUM> but a number of channels greater than the audio signal <NUM>. In at least some of the blocks <NUM> and 50a-50e, <NUM>, <NUM>, the signal (<NUM>, <NUM>, <NUM>, <NUM>), evolving from the input <NUM> (e.g. noise or pitch) towards becoming speech <NUM>, may be upsampled. For example, at the first block 50a among the blocks 50a-<NUM>, a <NUM>-times upsampling may be performed. An example of upsampling may include, for example, the following sequence: <NUM>) repetition of same value, <NUM>) insert zeros, <NUM>) another repeat or insert zero + linear filtering, etc..

The generated audio signal <NUM> may generally be a single-channel signal. In case multiple audio channels are necessary (e.g., for a stereo sound playback) then the claimed procedure shall be in principle iterated multiple times.

Analogously, also the target data <NUM> has multiple channels (e.g. in spectrograms), as generated by the preconditioning learnable layer(s) <NUM>. In any case, the target data <NUM> may be upsampled (e.g. by a factor of two, a power of <NUM>, a multiple of <NUM>, or a value greater than <NUM>) to adapt to the dimensions of the signal (59a, <NUM>, <NUM>) evolving along the subsequent layers (50a-<NUM>, <NUM>), e.g. to obtain the conditioning feature parameters <NUM>, <NUM> in dimensions adapted to the dimensions of the signal.

If the first processing block <NUM> is instantiated in multiple blocks 50a-<NUM>, the number of channels may, for example, remain at least some of the multiple blocks (e.g., from 50e to <NUM> and in block <NUM> the number of channels does not change). The first data <NUM> may have a first dimension or at least one dimension lower than that of the audio signal <NUM>. The first data <NUM> may have a total number of samples across all dimensions lower than the audio signal <NUM>. The first data <NUM> may have one dimension lower than the audio signal <NUM> but a number of channels greater than the audio signal <NUM>.

Examples may be performed according to the paradigms of generative adversarial networks (GANs). A GAN includes a GAN generator <NUM> (<FIG>) and a GAN discriminator <NUM> (<FIG>). The GAN generator <NUM> tries to generate an audio signal <NUM>, which is as close as possible to a real audio signal. The GAN discriminator <NUM> shall recognize whether the generated audio signal <NUM> is real or fake. Both the GAN generator <NUM> and the GAN discriminator <NUM> may be obtained as neural networks (or other by other learnable techniques). The GAN generator <NUM> shall minimize the losses (e.g., through the method of the gradients or other methods), and update the conditioning features parameters <NUM>, <NUM> by taking into account the results at the GAN discriminator <NUM>. The GAN discriminator <NUM> shall reduce its own discriminatory loss (e.g., through the method of gradients or other methods) and update its own internal parameters. Accordingly, the GAN generator <NUM> is trained to generate better and better audio signals <NUM>, while the GAN discriminator <NUM> is trained to recognize real signals <NUM> from the fake audio signals generated by the GAN generator <NUM>. The GAN generator <NUM> may include the functionalities of the decoder <NUM>, without at least the functionalities of the GAN discriminator <NUM>. Therefore, in most of the foregoing, the GAN generator <NUM> and the audio decoder <NUM> may have more or less the same features, apart from those of the discriminator <NUM>. The audio decoder <NUM> may include the discriminator <NUM> as an internal component. Therefore, the GAN generator <NUM> and the GAN discriminator <NUM> may concur in constituting the audio decoder <NUM>. In examples where the GAN discriminator <NUM> is not present, the audio decoder <NUM> can be constituted uniquely by the GAN generator <NUM>.

As explained by the wording "conditioning set of learnable layers", the audio decoder <NUM> may be obtained according to the paradigms of conditional neural networks (e.g. conditional GANs), e.g. based on conditional information. For example, conditional information may be constituted by target data (or upsampled version thereof) <NUM> from which the conditioning set of layers <NUM>-<NUM> (weight layer) are trained and the conditioning feature parameters <NUM>, <NUM> are obtained. Therefore, the styling element <NUM> is conditioned by the learnable layers <NUM>-<NUM>. The same may apply to the preconditional layers <NUM>.

The examples may be based on convolutional neural networks. For example, a little matrix (e.g., filter or kernel), which could be a 3x3 matrix (or a 4x4 matrix, etc.), is convolved (convoluted) along a bigger matrix (e.g., the channel x samples latent or input signal and/or the spectrogram and/or the spectrogram or upsampled spectrogram or more in general the target data <NUM>), e.g. implying a combination (e.g., multiplication and sum of the products; dot product, etc.) between the elements of the filter (kernel) and the elements of the bigger matrix (activation map, or activation signal). During training, the elements of the filter (kernel) are obtained (learnt) which are those that minimize the losses. During inference, the elements of the filter (kernel) are used which have been obtained during training. Examples of convolutions may be used at at least one of blocks <NUM>-<NUM>, 61b, 62b (see below), <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>. Where a convolution is conditional, then the convolution is not necessarily applied to the signal evolving from the input signal <NUM> towards the audio signal <NUM> through the intermediate signals 59a (<NUM>), <NUM>, etc., but may be applied to the target signal <NUM> (e.g. for generating the conditioning feature parameters <NUM> and <NUM> to be subsequently applied to the first data <NUM>, or latent, or prior, or the signal evolving form the input signal towards the speech <NUM>). In other cases (e.g. at blocks 61b, 62b) the convolution may be non-conditional, and may for example be directly applied to the signal 59a (<NUM>), <NUM>, etc., evolving from the input signal <NUM> towards the audio signal <NUM>. Both conditional and non-conditional convolutions may be performed.

It is possible to have, in some examples, activation functions downstream to the convolution (ReLu, TanH, softmax, etc.), which may be different in accordance to the intended effect.

ReLu may map the maximum between <NUM> and the value obtained at the convolution (in practice, it maintains the same value if it is positive, and outputs <NUM> in case of negative value). Leaky ReLu may output x if x><NUM>, and <NUM>*x if x≤<NUM>, x being the value obtained by convolution (instead of <NUM> another value, such as a predetermined value within <NUM> ± <NUM>, may be used in some examples). TanH (which may be implemented, for example, at block 63a and/or 63b) may provide the hyperbolic tangent of the value obtained at the convolution, e.g. <MAT> with x being the value obtained at the convolution (e.g. at block 61b). Softmax (e.g. applied, for example, at block 64b) may apply the exponential to each element of the elements of the result of the convolution, and normalize it by dividing by the sum of the exponentials. Softmax may provide a probability distribution for the entries which are in the matrix which results from the convolution (e.g. as provided at 62b). After the application of the activation function, a pooling step may be performed (not shown in the figures) in some examples, but in other examples it may be avoided.

It is also possible to have a softmax-gated TanH function, e.g. by multiplying (e.g. at 65b) the result of the TanH function (e.g. obtained at 63b) with the result of the softmax function (e.g. obtained at 64b).

Multiple layers of convolutions (e.g. a conditioning set of learnable layers) may, in some examples, be one downstream to another one and/or in parallel to each other, so as to increase the efficiency. If the application of the activation function and/or the pooling are provided, they may also be repeated in different layers (or maybe different activation functions may be applied to different layers, for example).

The input signal <NUM> (e.g. noise or a signal obtained from the bitstream, e.g. excitation data 3a, such as pitch information) is processed, at different steps, to become the generated audio signal <NUM> (e.g. under the conditions set by the conditioning sets of learnable layers <NUM>-<NUM>, and on the parameters <NUM>, <NUM> learnt by the conditioning sets of learnable layers <NUM>-<NUM>). Therefore, the input signal <NUM> (first data <NUM>) is to be understood as evolving in a direction of processing (from <NUM> to <NUM> in <FIG>) towards becoming the generated audio signal <NUM> (e.g. speech). The conditions will be substantially generated based on the target signal <NUM> and/or on the preconditions in the bitstream <NUM>, and on the training (so as to arrive at the most preferable set of parameters <NUM>, <NUM>).

It is also noted that the multiple channels of the input signal <NUM> (or any of its evolutions) may be considered to have a set of learnable layers and a styling element <NUM> associated thereto.

For example, each row of the matrixes <NUM> and <NUM> may be associated to a particular channel of the input signal (or one of its evolutions), e.g. obtained from a particular learnable layer associated to the particular channel. Analogously, the styling element <NUM> may be considered to be formed by a multiplicity of styling elements (each for each row of the input signal x, c, <NUM>, <NUM>, <NUM>', <NUM>, 59a, 59b, etc.).

<FIG> shows an example of the audio decoder (generator) <NUM> (which may embody the audio decoder <NUM> of <FIG>), and which may also comprise (e.g. be) a GAN generator <NUM>. It is noted that <FIG> does now show the preconditioning learnable layer <NUM> (shown in <FIG>), even though the target data <NUM> are obtained from the bitstream <NUM> through the preconditioning layer(s) <NUM> (see above and <FIG>). The target data <NUM> may be a mel-spectrogram obtain from the preconditioning learnable layer <NUM>; the input signal <NUM> may be a latent noise or a signal obtained from the excitation data (e.g. pitch information) 3b from the bitstream <NUM>, and the output <NUM> may be speech. The input signal <NUM> has only one sample and multiple channels (indicated as "x", because they can vary, for example the number of channels can be <NUM> or something else). The input vector <NUM> (noise or a signal obtain from the excitation data 3b, like pitch information) may be obtained in a vector with <NUM> channels (but other numbers are possible). In case the input signal <NUM> is noise ("first option"), it may have a zero-mean normal distribution, and follow the formula z ~ <IMG>(<NUM>, I<NUM>); it may be a random noise of dimension <NUM> with mean <NUM>, and with an autocorrelation matrix (square 128x128) equal to the identity I (different choice may be made). Hence, in examples in which the noise is used as input signal <NUM>, it can be completely decorrelated between the channels and of variance <NUM> (energy). <IMG>(<NUM>, I<NUM>) may be realized at every <NUM> generated samples (or other numbers may be chosen for different examples); the dimension may therefore be <NUM> in the time axis and <NUM> in the channel axis.

It will be shown that the input vector <NUM> (whether noise or obtained from the bitstream <NUM>) is step-by-step processed (e.g., at blocks 50a-<NUM>, <NUM>, <NUM>, <NUM>, etc.), so as to evolve to speech <NUM> (the evolving signal will be indicated with different signals <NUM>, 59a, x, c, <NUM>', <NUM>, 79a, 59b, 79b, <NUM>, etc.).

At block <NUM>, a channel mapping may be performed. It may consist of or comprise a simple convolution layer to change the number channels, for example in this case from <NUM> to <NUM>.

As can be seen, at least some of the processing blocks 50a, 50b, 50c, 50d, 50e, 50f, <NUM>, <NUM> (altogether embodying the first processing block <NUM> of <FIG>) increases the number of samples by performing an upsampling (e.g., maximum <NUM>-upsampling), e.g. for each frame. The number of channels may remain the same (e.g., <NUM>) along blocks 50a, 50b, 50c, 50d, 50e, 50f, <NUM>, <NUM>. The samples may be, for example, the number of samples per second (or other time unit): we may obtain, at the output of block <NUM>, sound at <NUM> or more (e.g. <NUM>). As explained above, a sequence of multiple samples may constitute one frame.

Each of the blocks 50a-<NUM> (<NUM>) can also be a TADEResBlock (residual block in the context of TADE, Temporal Adaptive DEnormalization). Notably, each block 50a-<NUM> (<NUM>) is conditioned by the target data (e.g., mel-spectrogram) <NUM> and/or by the bitstream <NUM> (e.g. by the filter data 3a, such as spectral envelope data, and/or by the other data 3b, such as pitch data).

At a second processing block <NUM> (<FIG> and <FIG>), only one single channel may be obtained, and multiple samples are obtained in one single dimension (see also <FIG>). As can be seen, another TADEResBlock <NUM> (further to blocks 50a-<NUM>) may be used (which reduces the dimensions to four single channels). Then, a convolution layer <NUM> and an activation function (which may be TanH <NUM>, for example) may be performed. A (Pseudo Quadrature Mirror Filter)-bank) <NUM> may also be applied, so as to obtain the final <NUM> (and, possibly, stored, rendered, etc.).

At least one of the blocks 50a-<NUM> (or each of them, in particular examples) and <NUM> may be, for example, a residual block. A residual block operates a prediction only to a residual component of the signal evolving from the input signal <NUM> (e.g. noise) to the output audio signal <NUM>. The residual signal is only a part (residual component) of the main signal evolving form the input signal <NUM> towards the output signal <NUM>. For example, multiple residual signals may be added to each other, to obtain the final output audio signal <NUM>.

<FIG> shows an example of one of the blocks 50a-<NUM> (<NUM>). As can be seen, each block <NUM> is inputted with a first data 59a, which is either the first data <NUM>, (or the upsampled version thereof, such as that output by the upsampling block <NUM>) or the output from a preceding block. For example, the block 50b may be inputted with the output of block 50a; the block 50c may be inputted with the output of block 50b, and so on. In examples, different blocks may operate in parallel to each other, and there results are added together.

From <FIG> it is possible to see that the first data 59a provided to the block <NUM> (50a-<NUM>) or <NUM> is processed and its output is the output data <NUM> (which will be provided as input to the subsequent block). As indicated by the line 59a', a main component of the first data 59a actually bypasses most of the processing of the first processing block 50a-<NUM> (<NUM>). For example, blocks 60a, <NUM>, 60b and <NUM> and 65b are bypassed by the main component 59a'. The residual component 59a of the first data <NUM> (<NUM>) will be processed to obtain a residual portion 65b' to be added to the main component 59a' at an adder 65c (which is indicated in <FIG>, but not shown). The bypassing main component 59a' and the addition at the adder 65c may be understood as instantiating the fact that each block <NUM> (50a-<NUM>) processes operations to residual signals, which are then added to the main portion of the signal. Therefore, each of the blocks 50a-<NUM> can be considered a residual block.

Notably, the addition at adder 65c does not necessarily need to be performed within the residual block <NUM> (50a-<NUM>). A single addition of a plurality of residual signals 65b' (each outputted by each of residual blocks 50a-<NUM>) can be performed (e.g., at one single adder block in the second processing block <NUM>, for example). Accordingly, the different residual blocks 50a-<NUM> may operate in parallel with each other.

In the example of <FIG>, each block <NUM> may repeat its convolution layers twice. A first denormalization block 60a and a second denormalization block 60b may be used in cascade. The first denormalization block 60a may include an instance of the stylistic element <NUM>, to apply the conditioning feature parameters <NUM> and <NUM> to the first data <NUM> (<NUM>) (or its residual version 59a). The first denormalization block 60a may include a normalization block <NUM>. The normalization block <NUM> may perform a normalization along the channels of the first data <NUM> (<NUM>) (or its residual version 59a). The normalized version c (<NUM>') of the first data <NUM> (<NUM>) (or its residual version 59a) may therefore be obtained. The stylistic element <NUM> may therefore be applied to the normalized version c (<NUM>'), to obtain a denormalized (conditioned) version of the first data <NUM> (<NUM>) (or its residual version 59a). The denormalization at element <NUM> may be obtained, for example, through an element-by-element multiplication of the elements of the matrix γ (which embodies the condition <NUM>) and the signal <NUM>' (or another version of the signal between the input signal and the speech), and/or through an element-by-element addition of the elements of the matrix β (which embodies the condition <NUM>) and the signal <NUM>' (or another version of the signal between the input signal and the speech). A denormalized version 59b (conditioned by the conditioning feature parameters <NUM> and <NUM>) of the first data <NUM> (<NUM>) (or its residual version 59a) may therefore be obtained.

Then, a gated activation <NUM> is performed on the denormalized version 59b of the first data <NUM> (or its residual version 59a). In particular, two convolutions 61b and 62b may be performed (e.g., each with 3x3 kernel and with dilation factor <NUM>). Different activation functions 63b and 64b may be applied respectively to the results of the convolutions 61b and 62b. The activation 63b may be TanH. The activation 64b may be softmax. The outputs of the two activations 63b and 64b may be multiplied by each other, to obtain a gated version 59c of the denormalized version 59b of the first data <NUM> (or its residual version 59a).

Subsequently, a second denormalization 60b may be performed on the gated version 59c of the denormalized version 59b of the first data <NUM> (or its residual version 59a). The second denormalization 60b may be like the first denormalization and is therefore here not described.

Subsequently, a second activation <NUM> is performed. Here, the kernel may be 3x3, but the dilation factor may be <NUM>. In any case, the dilation factor of the second gated activation <NUM> may be greater than the dilation factor of the first gated activation <NUM>.

The conditioning set of learnable layers <NUM>-<NUM> (e.g. as obtained from the preconditioning learnable layer(s)) and the styling element <NUM> may be applied (e.g. twice for each block 50a, 50b. ) to the signal 59a. An upsampling of the target data <NUM> may be performed at upsampling block <NUM>, to obtain an upsampled version <NUM>' of the target data <NUM>. The upsampling may be obtained through non-linear interpolation, and may use e.g. a factor of <NUM>, a power of <NUM>, a multiple of two, or another value greater than <NUM>. Accordingly, in some examples it is possible to have that the spectrogram (e.g. mel-spectrogram) <NUM>' has the same dimensions (e.g. conform to) the signal (<NUM>, <NUM>', c, <NUM>, 59a, 59b, etc.) to be conditioned by the spectrogram.

In examples, the first and second convolutions at 61b and 62b, respectively downstream to the TADE block 60a or 60b, may be performed at the same number of elements in the kernel (e.g., <NUM>, e.g., 3x3). However, the second convolutions in block <NUM> may have a dilation factor of <NUM>. In examples, the maximum dilation factor for the convolutions may be <NUM> (two).

As explained above, the target data <NUM> may be upsampled, e.g. so as to conform to the input signal (or a signal evolving therefrom, such as <NUM>, 59a, <NUM>', also called latent signal or activation signal). Here, convolutions <NUM>, <NUM>, <NUM> may be performed (an intermediate value of the target data <NUM> is indicated with <NUM>'), to obtain the parameters γ (gamma, <NUM>) and β (beta, <NUM>). The convolution at any of <NUM>, <NUM>, <NUM> may also require a rectified linear unit, ReLu, or a leaky rectified linear unit, leaky ReLu. The parameters γ and β may have the same dimension of the activation signal (the signal being processed to evolve from the input signal <NUM> to the generated audio signal <NUM>, which is here represented as x, <NUM>, 59a, or <NUM>' when in normalized form). Therefore, when the activation signal (x, <NUM>, 59a, <NUM>') has two dimensions, also γ and β (<NUM> and <NUM>) have two dimensions, and each of them is superimposable to the activation signal (the length and the width of γ and β may be the same of the length and the width of the activation signal). At the stylistic element <NUM>, the conditioning feature parameters <NUM> and <NUM> are applied to the activation signal (which may be the first data 59a or the 59b output by the multiplier 65a). It is to be noted, however, that the activation signal <NUM>' may be a normalized version (at instance norm block <NUM>) of the first data <NUM>, 59a, 59b (<NUM>), the normalization being in the channel dimension. It is also to be noted that the formula shown in stylistic element <NUM> (γc+β, also indicated with γ ⊙ c + β in <FIG>) may be an element-by-element product, and in some examples is not a convolutional product or a dot product.

The convolutions <NUM> and <NUM> have not necessarily activation function downstream of them. The parameter γ (<NUM>) may be understood as having variance values and β (<NUM>) as having bias values.

It is noted that for each block 50a-<NUM>, <NUM>, the learnable layers <NUM>-<NUM> (e.g. together with he styling element <NUM>) may be understood as embodying weight layers.

Also, block <NUM> of <FIG> may be instantiated as block <NUM> of <FIG>. Then, for example, a convolutional layer <NUM> will reduce the number of channels to <NUM> and, after that, a TanH <NUM> is performed to obtain speech <NUM>. The output <NUM>' of the blocks <NUM> and <NUM> may have a reduced number of channels (e.g. <NUM> channels instead of <NUM>), and/or may have the same number of channels (e.g., <NUM>) of the previous block <NUM> or <NUM>.

A PQMF synthesis (see also below) <NUM> is performed on the signal <NUM>', so as to obtain the audio signal <NUM> in one channel.

Therefore, an example of decoding and generation of the speech <NUM> is here provided (in some examples, some of the following steps may be omitted or submitted by other ones):.

The GAN discriminator <NUM> of <FIG> may be used during training for obtaining, for example, the parameters <NUM> and <NUM> to be applied to the input signal <NUM> (or a processed and/or normalized version thereof). The training may be performed before inference, and the parameters <NUM> and <NUM> may be, for example, stored in a non-transitory memory and used subsequently (however, in some examples it is also possible that the parameters <NUM> or <NUM> are calculated on line).

The GAN discriminator <NUM> has the role of learning how to recognize the generated audio signals (e.g., audio signal <NUM> synthesized as discussed above) from real input signals (e.g. real speech) <NUM>. Therefore, the role of the GAN discriminator <NUM> is mainly exerted during training (e.g. for learning parameters <NUM> and <NUM>) and is seen in counter position of the role of the GAN generator <NUM> (which may be seen as the audio decoder <NUM> without the GAN discriminator <NUM>).

In general terms, the GAN discriminator <NUM> may be input by both audio signal <NUM> synthesized generated by the GAN decoder <NUM>, and real audio signal (e.g., real speech) <NUM> acquired e.g., through a microphone, and process the signals to obtain a metric (e.g., loss) which is to be minimized. The real audio signal <NUM> can also be considered a reference audio signal. During training, operations like those explained above for synthesizing speech <NUM> may be repeated, e.g. multiple times, so as to obtain the parameters <NUM> and <NUM>, for example.

In examples, instead of analyzing the whole reference audio signal <NUM> and/or the whole generated audio signal <NUM>, it is possible to only analyze a part thereof (e.g. a portion, a slice, a window, etc.). Signal portions generated in random windows (105a-105d) sampled from the generated audio signal <NUM> and from the reference audio signal <NUM> are obtained. For example random window functions can be used, so that it is not a priori pre-defined which window 105a, 105b, 105c, 105d will be used. Also the number of windows is not necessarily four, at may vary.

Within the windows (105a-105d), a PQMF (Pseudo Quadrature Mirror Filter)-bank) <NUM> may be applied. Hence, subbands <NUM> are obtained. Accordingly, a decomposition (<NUM>) of the representation of the generated audio signal (<NUM>) or the representation of the reference audio signal (<NUM>) is obtained.

An evaluation block <NUM> may be used to perform the evaluations. Multiple evaluators 132a, 132b, 132c, 132d (complexively indicated with <NUM>) may be used (different number may be used). In general, each window 105a, 105b, 105c, 105d may be input to a respective evaluator 132a, 132b, 132c, 132d. Sampling of the random window (105a-105d) may be repeated multiple times for each evaluator (132a-132d). In examples, the number of times the random window (105a-105d) is sampled for each evaluator (132a-132d) may be proportional to the length of the representation of the generated audio signal or the representation of the reference audio signal (<NUM>). Accordingly, each of the evaluators (132a-132d) may receive as input one or several portions (105a-105d) of the representation of the generated audio signal (<NUM>) or the representation of the reference audio signal (<NUM>).

Each evaluator 132a-132d may be a neural network itself. Each evaluator 132a-132d may, in particular, follow the paradigms of convolutional neutral networks. Each evaluator 132a-132d may be a residual evaluator. Each evaluator 132a-132d may have parameters (e.g. weights) which are adapted during training (e.g., in a manner similar to one of those explained above).

As shown in <FIG>, each evaluator 132a-132d also performs a downsampling (e.g., by <NUM> or by another downsampling ratio). The number of channels may increase for each evaluator 132a-132d (e.g., by <NUM>, or in some examples by a number which is the same of the downsampling ratio).

Upstream and/or downstream to the evaluators, convolutional layers <NUM> and/or <NUM> may be provided. An upstream convolutional layer <NUM> may have, for example, a kernel with dimension <NUM> (e.g., 5x3 or 3x5). A downstream convolutional layer <NUM> may have, for example, a kernel with dimension <NUM> (e.g., 3x3).

During training, a loss function (adversarial loss) <NUM> may be optimized. The loss function <NUM> may include a fixed metric (e.g. obtained during a pretraining step) between a generated audio signal (<NUM>) and a reference audio signal (<NUM>). The fixed metric may be obtained by calculating one or several spectral distortions between the generated audio signal (<NUM>) and the reference audio signal (<NUM>). The distortion may be measured by keeping into account:.

In examples, the adversarial loss may be obtained by randomly supplying and evaluating a representation of the generated audio signal (<NUM>) or a representation of the reference audio signal (<NUM>) by one or more evaluators (<NUM>). The evaluation may comprise classifying the supplied audio signal (<NUM>, <NUM>) into a predetermined number of classes indicating a pretrained classification level of naturalness of the audio signal (<NUM>, <NUM>). The predetermined number of classes may be, for example, "REAL" vs "FAKE".

Examples of losses may be obtained as <MAT> where:.

The spectral reconstruction loss <IMG> is still used for regularization to prevent the emergence of adversarial artifacts. The final loss is can be, for example: <MAT> where each i is the contribution at each evaluator 132a-132d (e.g.. each evaluator 132a-132d providing a different Di) and <IMG> is the pretrained (fixed) loss.

During training, there is a search foddr the minimum value of <IMG>, which may be expressed for example as <MAT>.

Other kinds of minimizations may be performed.

In general terms, the minimum adversarial losses <NUM> are associated to the best parameters (e.g., <NUM>, <NUM>) to be applied to the stylistic element <NUM>.

A discussion on the inventive examples is provided here below. For more clarity, the discussion is divided in sections.

Recently, it has been noted that that Generative Adversarial Network (GAN)-based vocoders outperform autoregressive and flow-based approaches in terms of quality of the synthesized speech while being orders of magnitude faster. Meanwhile, neural vocoders have also been successfully applied to speech coding at very low bit-rates. The neural vocoder used at the decoder is able to generate natural speech from a highly compressed representation. However, coding schemes proposed so far rely on autoregressive models exploiting the causal nature of the speech waveform, but limited to a sample-by-sample signal generation. An aim of this work is twofold: make the GAN-based Neural Vocoder suitable for streaming and coding applications. Starting from the StyleMelGan model, the convolutions of the model were made causal, and conditioning and prior produced with a limited look ahead. It was ensured that the speech can be generated with a constrained delay and generated continuously frame-by-frame. The quality was found to be very close to that of a batch processing. Further, the coded parameters were fed in an efficient way to the neural vocoder by two ways. Spectral envelope information is used to conditioned the TADERes blocks of smGAN, while the pitch information is exploited to build the low-dimensionality prior. Experimental results show that the obtained coding scheme outperforms the previously published solutions to date in coding clean speech at very low bit-rate coding.

Combining parametric speech coding and neural Vocoder leads to new coding paradigms enabling a compression factor for speech much higher than with conventional approaches. It was demonstrated that conditioning a neural vocoder with coded parameters could produce natural wideband speech e.g. at bit-rates <NUM> kbps or even lower. Until now neural-vocoder speech coders relied on an auto-regressive generative Network, engendering low delay by it-self, but complex by nature since generation happens by design sample-by-sample.

Most GAN vocoders offer very fast generation on GPUs, but at the cost of compromising the quality of the synthesized speech. GAN-based neural vocoder were recently shown [<NUM>] to be competitive and viable alternatives to autoregressive approaches for speech synthesis, and in for Text-To-Speech applications (TTS). However, they are by design not suited for streaming or for real-time speech communication, since they generate by heavy parallelization of the processing a large block of samples at once. Moreover, it was still not demonstrated that GAN-based neural Vocoder could be conditioned with something else than a relatively continuous representation of the speech like Mel-Spectrogram. For speech coding applications using a highly discrete representation of the speech (also called parametric representation), one still needs to demonstrate the feasibility using a GAN approach.

The aim of the present invention is twofold:.

In this section we describe the main architecture of speech coding schemes based on a neural-vocoder. The principle was first exposed and demonstrated in [<NUM>], and adopted in subsequent works [<NUM>, <NUM>, <NUM>]. As illustrated in <FIG>, the encoder analysis the input speech signal at a sampling rate of <NUM> in case of wideband speech.

Eventually and since the very low bit-rate scheme is usually de-signed and trained for a given source, the input signal could been pre-processed beforehand by for example noise suppression module as proposed in [<NUM>] or any kind of source separation or speech enhancement processing to get speech signal as clean as possible.

The encoder first analyzes the signal for extracting a set of acoustic features, which will be quantized, coded and then trans-mitted. In our case, and for comparison purposes, we stick to the features employed in LPCNet <NUM> kbps coder as described in [<NUM>]. The coded parameters and their respective bit-rates are summarized in Table <NUM>. We find the usual acoustic features conveyed in parametric speech coding, i.e. the spectral envelope, the pitch information, and the energy.

The decoder mainly consists of a neural vocoder which is conditioned by the coded parameters. For example, LCPNet <NUM> kbps adopts a recursive architecture based on WaveRNN relying on linear prediction to reduce further complexity, generating the signal in the residual linear prediction domain. The decoding is then divided into two parts: a frame-rate network that computes from the coded parameters the conditioning at every <NUM> frame, and a sample-rate network that computes the conditional sampling probabilities. In addition to using the previously generated speech sample, LPCNet also uses the 16th-order prediction coefficients (LPC) and the previously generated excitation sample to predict the current excitation sample. WaveRNN is also adopted in more recent work [<NUM>], com-pressing speech at 3kbps by directly coding stacked Mel-Spectra.

In the present work, we propose to replace the auto-regressive neural Vocoder with a GAN-based approach in order to benefit from its feed-forward architecture and a potential gain in quality.

In the current work we have modified StyleMelGAN introduced in [<NUM>], a lightweight neural vocoder allowing synthesis of high-fidelity speech with low computational complexity. StyleMelGAN employs temporal adaptive normalization to style a low-dimensional noise vector with the acoustic features of the target speech. For efficient training, multiple random-window discriminators adversarially evaluate the speech signal analyzed by a filter bank, with regularization provided by a multi-scale spectral reconstruction loss. The highly parallelizable speech generation is several times faster than real-time on CPUs and GPUs.

The computational advantage of GAN-based vocoder relies mainly on the high parallelization of the speech synthesis, which allows the neural vocoder to generate a relatively large audio sequence in one shot. This is only possible if the model doesn't rely on any feedback loop but on a feedforward architecture. However in the case of audio coding, the algorithmic delay must be limited, and consequently the generation must be done frame-by-frame, in a streaming fashion and with a limited frame size. To ensure a total algorthmetic delay allowing comfortable communication, not only the frame size but also the access to future information during the generation must be limited. Deriving a streaming model from an original model is a relatively common optimization for achieving more efficient inference, which involves replacing non-causal convolutions with causal ones [<NUM>]. <FIG> shows an example of convolutions and their memory management in such a case.

Several modifications were therefore made to the original smGAN model. First, the convolutions were made causal and the synthesis lookahead was removed in the conditioning. It turned out that the model is robust enough for such modifications and through informal evaluations, the quality was found unaffected, or only very minimally, by such modifications. Coupled with the LPCNet features, streamwise smGAn generates frames of <NUM> and does not exploit any additional synthesis lookahead, leading to a total delay of <NUM>, which is uniquely coming from the original extraction of the LPCNet features and the size of the encoded packets. Total de-lay is then <NUM> lower than the original <NUM> of LPCNet <NUM>.

Moreover, the TADE residual blocks are slightly modified from the original model, as shown in <FIG>. It was observed that the complexity can be reduced by using a single TADE layer and applying the same β and γ twice rather than having two separate TADE layers. In addition, instance normalization was advantageously re-placed by channel normalization.

Finally, the whole architecture has been revisited to speed up the generation. In particular, smGAN has been made faster by adapting a multiband synthesis as introduced in [<NUM>, <NUM>]. Rather than synthesizing the speech signal in time domain at the output sample rate fs, the generator outputs simultaneously N different frequency bands samples at fs/N Hz. By design, smGan generates the frequency bands frame-wise, which are then fed to a Pseudo QMF synthesis filter-bank to obtain a frame of synthesized speech. <FIG> shows the dimensions for a <NUM> framing for generating a wideband speech sampled at <NUM>.

The next sections will introduce different ways of conditioning the stream-wise smGAN with the LPCNet coded parameters. For this we will introduce different pre-conditioning networks and different strategies.

In this section we introduce a way to couple the coded parameters to the GAN-based neural-vocoder by enforcing a homogenous speech representation for conditioning the stream-wise smGAN. As pre-conditional network is introduced to map coded parameters, well suited for an efficient coding, to the Mel-Spectrogram representation, well suited for neural vocoder conditioning. Mel-Spectrogram has been shown to convey sufficient acoustic features to generate a high-fidelity speech signal. Its two-dimensional representation is particularly well suited for convolutional kernels. Changing the nature of either the neural vocoder conditioning or the coded parameters will impact the speech synthesis quality or the coding efficiency respectively. Therefore, we propose in a pragmatic way to decouple the problem by recovering first a realistic Mel-Spectrogram from the coded parameters before the waveform generation. Both the neural vocoder and the pre-conditional network are jointly trained, and adversarial learning can make the Mel-Spectrogram and hence the generated speech more natural and realistic by generating a fine structured signal even in high frequencies.

The contribution allows mapping a non-homogeneous parametric representation of the speech, efficient for speech transmission, to a homogeneous representation, like the Mel-Spectrogram, well suited for conditioning neural generative networks.

The pre-conditioning net consists of two convolutional layers followed by two fully-connected layers. All layers are activated by Leaky RE-LU which was found better than the than activation used in the LPCNet frame-rate network. The pre-conditional net is trained using the L1-norm as a regression loss. pitch_embedding
= nn. Embedding (<NUM>, <NUM>)
precond_conv1
= ConvLayer (<NUM>, <NUM>)
precond_conv2
= ConvLayer (<NUM> , <NUM>)
precond_dense = Dense (
weigth_norm ( Linear (<NUM> , <NUM>)) ,
LeakyReLU ,
weigth_norm ( Linear (<NUM>, <NUM>)) ,
LeakyReLU)
precond_input
= concatenate (
mfcc ,
norm_corr , pitch_embedding ( pitch ) )
precond_output
= precond_conv1 (precond_input)
precond_output
= precond_conv2 (precond_output)
mel_spec
= precond_dense (precond_output)
multiband_waveforms
= generator_network (
conditioning = mel. spec ,
prior = gaussian_noise (<NUM> , <NUM>))
out_frame = pqmf. synthesis (multiband_waveforms).

A Mushra test was conducted comparing the speech coder with pre-cond Net amd smGan, and compared to LPCNet <NUM> kbps and conventional coding schemes. Results presented in <FIG> show the superiority of the present approach over prior arts.

Although the previously introduced approach delivers a quality superior to the original LPCNet <NUM>. 6kbps using exact same in-formation, problems are still observed especially in the prosody of certain generated speech items. The pitch information seems not be well preserved in the generated Mel-Spectrogram. Moreover this two steps approach is obviously suboptimal by design and by optimizing sequentially two networks.

Although the previously introduced approach delivers a higher quality than the original <NUM>. 6kbps LPCNet, and that using exactly the same information, some problems are still observed, especially in the prosody of some of the generated speech items. The pitch information does not seem to be not always well preserved in the generated Mel-Spectrogram. Furthermore, this two-step approach is clearly suboptimal by design and by sequentially optimizing two separated networks.

In the previous section, we observed that the enforcement of the Mel-Spectrogram as an intermediate representation works but still has some typical problems with the prosody, which is assumed to come from the pitch information. To circumvent the effect of coarse quantization on the pitch lag, we propose another pre-conditional network, this time enforcing the original pitch information to be recovered. For this, the pre-conditioning network was trained to minimize the error between a predicted pitch and an estimate of it performed on the original signal and found close to the ground truth. pitch_embedding
= Embedding (<NUM>, <NUM>)
precond_conv1
= ConvLayer (<NUM>
, <NUM> ,
LeakyReLU ,
kernel_size=<NUM>)
precond_conv2
= ConvLayer (<NUM>
, <NUM> ,
LeakyReLU ,
kernel_size =<NUM>)
precond_dense=
p_embed
= pitch_embedding (pitch) * pitch_corr
precond_out = precond_conv1 ([mfcc , p_embed])
precond_out = precond_conv2 (precond_out)
precond_out = precond_dense (precond_out)
multiband_waveforms
= generator_network (
conditioning = precond_out , prior = noise )
out_frame
= pqmf. synthesis (multiband_waveforms).

In this section, we introduce a new way of conditioning smGan by the coded parameters. Instead of enforcing the Mel-Spectrogram as an intermediate representation, the coded pa- rameters are introduced more directly into the generative network, thus reducing the size of the preconditioning network but also being more efficient in the generation. Since the pitch information was shown to be critical, it is processed separately from the spectral envelope information. In-deed, the decoded MFCCs and energy are used for conditioning the generating and styling the prior signal. The latter is in this case not created from random noise but rather from an embedded representation of the pitch information and the pitch correla- tion. pitch_embedding
= Embedding (<NUM>,<NUM>)
precond_conv
= ConvLayer (
<NUM>, <NUM> ,
LeakyReLU ,
kernel_size =<NUM>)
pitch_embed
= pitch_embedding (pitch)
* pitch_corr
multiband_waveforms
= generator_network (
conditioning = precond_conv (mfcc) ,
prior = pitch_embed)
out_frame
= pqmf. synthesis (multiband_waveforms).

We report the computational complexity estimate in Tab. It is worth mentioning that since smGAN is capable of high parallelization during the generation, unlike the other listed autoregressive models, it could benefit from a much faster generation on a dedicated platforms based on for example on GPUs.

We conducted a MUSHRA listening test assessing the stream-wise smGan conditioned with LPCnet parameters coded at <NUM> kbps using technique described in section <NUM>. The test involved <NUM> expert listeners and results are shown in <FIG>.

The anchor is generated using the Speex speech coder employed at a bit rate of <NUM> kbps. The quality is expected to be very low at this bit rate and it provides only narrowband speech. Three neural vocoder coders were considered: LPCnet at <NUM> kbps, Lyra at <NUM> kbps and our solution at <NUM> kbps. As benchmarks, two classical but still modern coders were added: AMR-WB at <NUM>. 6kbps and 3GPP EVS at <NUM>. It is worth mentioning that EVS at <NUM>. 9kbps works with a variable bit rate (VBR) and that <NUM> kbps reflects the average bit rate on active frames. During a long inactive phase, EVS goes into a non-transmission mode (DTX), transmitting only periodically packets at a bit-rate as low as <NUM>. The test items were composited with limited pauses, and DTX mode plays a very minor role in this test.

LPCNet was trained on the same database used to train our model, i.e. VCTK. Another difference from the original work is that we do not apply a domain adaptation by first training on unquantified and then quantified features. As VCTK is a noisier and much more diverse database than the NTT database used in the original work, we have removed the data augmentation which was found to penalize the final quality.

In this paper, we have proposed to adopt a GAN-based neural vocoder, using a feed-forward generator for the speech synthesis. Achieved quality was shown to be superior to existing solutions using auto-regressive models and this by using the exact same coded parameters. For this purpose, we introduce two main contributions: the streaming of the feed-forward generation, required for real-time applications with low algorithmic delay, and also a proper conditioning of the model using much a much more compact representation than the conventional Mel-Spectrogram acoustic feature. We demonstrate that a GAN-based neural vocoder can be very competitive for a coding application and bring a new mark to obtain peach at very low bitrates.

Potential applications and benefits from the invention for speech coding:.

According to an aspect, examples above are directed to an audio decoder, configured to generate an frame of an audio signal from an input signal and target data, the target data representing an audio signal windowed around the target frame to reproduce, comprising:.

According to an aspect, examples above are directed to an audio decoder, configured to generate a frame of an audio signal from an input signal and target data, the target data representing an audio signal windowed around the target frame to reproduce, comprising:.

According to an aspect, examples above are directed to an audio decoder, to generate audio signal from an input signal and coded data, the coded data representing information of an original audio signal to be generated comprising:.

In the present document we have provided examples of a so-called Streamwise StyleMelGAN (SStyleMelGAN).

StyleMelGAN [<NUM>] is a lightweight neural vocoder allowing synthesis of high-fidelity speech with low computational complexity. It employs Temporal Adaptive DE-normalization (TADE) that styles the upsampling of a low-dimensional noise vector with the acoustic features of the target speech (e.g., mel-spectrogram) via in-stance normalization and elementwise modulation. More precisely it learns adaptively the modulation parameters y (gamma) and □ (beta) from the acoustic features, and then applies the transformation <MAT> where c is the normalized content of the input activation. For efficient training, multiple random-window discriminators adversarially evaluate the speech signal analyzed by a set of PQMF filter banks, with the generator regularized by a multi-resolution STFT loss.

Convolutions in StyleMelGAN of [<NUM>] were non-causal and run as a moving-average on sliding windows of the input tensors. This results in significant amount of algorithmic delay due to the deep hierarchical structure of the model. In the present document, we describe major modifications to this baseline model that enable the generation at very low delay with different acoustic features for conditioning.

Here, we describe a new way of conditioning StyleMelGAN by the coded parameters. Instead of enforcing the mel-spectrogram as an inter-mediate representation, the coded parameters may be introduced more directly into the generative network (e.g. <NUM>, <NUM>, <NUM>), thus reducing the size of the preconditioning network (e.g. <NUM>) but also being more efficient in the generation. The pitch information (e.g. pitch data) 3b was shown to be critical for high-quality synthesis, and hence it may be processed separately from the spectral envelope information (or filter data) 3a. More precisely, the decoded MFCCs and energy may be used for conditioning the generating and styling the prior signal (e.g. <NUM>, <NUM>), which (in the "first option") is not necessarily created from random noise (as in the "second option"), but rather from a learned embedded representation of the pitch information and/or their memory management in such a case.

<FIG> may be understood as picturing an example of forward pass for a <NUM> sec framing for generating a wideband speech sampled at <NUM>.

The training procedure and hyperparameters are similar to the ones described in [<NUM>]. We train SSMGAN using one NVIDIA Tesla V100 GPU on a subset of the VCTK corpus [<NUM>] at <NUM> kbit/s. The conditioning features are calculated as in [<NUM>] as described in Section <NUM>. The generator is pretrained for <NUM> steps using the Adam optimizer [<NUM>] with learning rate Irg = <NUM>-<NUM>, β = {<NUM>, <NUM>}. When starting the adversarial training, we set Irg use the multi-scale discriminator described in [<NUM>] trained via the Adam optimizer with Ird = <NUM> * <NUM>-<NUM>, and same β. The batch size is <NUM> and for each sample in the batch we extract a segments of length <NUM>. The adversarial training lasts for about <NUM> steps.

The anchor is generated using the Speex speech coder em-ployed at a bit rate of <NUM> kbit/s. The quality is expected to be very low at this bit rate and it provides only narrowband speech. Two state-of-the-art neural vocoder coders were considered, LPC-net at <NUM> kbit/s, Lyra at <NUM> kbit/s, as well as two classical but still modern classical coders AMR-WB at <NUM> kbit/s and 3GPP EVS at <NUM> kbit/s. It is worth mentioning that EVS at <NUM> kbit/s works with a variable bit rate (VBR) and that <NUM> kbit/s reflects the average bit rate on active frames. During a long inactive phase, EVS switches to a non-transmission mode (DTX), transmitting only pe-riodically packets at a bit rate as low as <NUM> kbit/s. The test items were composited with limited pauses, and DTX mode plays a very minor role in this test.

LPCNet was trained on the same database used to train our model, i.e. VCTK. Another difference from the original work is that we do not apply a domain adaptation by first training on un-quantized and then fine-tuning quantized features. As VCTK is a noisier and much more diverse database than the NTT database used in the original work, we have removed the data augmentation which was found to penalize the final quality.

An important contribution to SSMGAN's computational complexity stems from the convolutions in the TADEResBlocks <NUM> (e.g. 50a-<NUM>, <NUM>) and the upsampling layers (see above). If Λ denotes the latent dimension (e.g. the number of channels of the target data or the input signal <NUM>)), K the lengths of the convolutional kernels, and Φ the dimension of the preconditioning input features (e.g. the filter data taken from the bitstream), then (ignoring activations and lower order terms) the evaluation of a TADEResBlock takes (Φ + <NUM>Λ) Λ K multiply accumulate operations (MACs) per output sample. Furthermore, an upsampling layer with kernel size K and latent dimension Λ takes Λ <NUM>K MACs. With Λ = <NUM>, K = <NUM>, Φ = <NUM> and TADEResBlock output sampling rates of <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> this accumulates to <MAT>.

On platforms providing fused multiply accumulate (FMA) instructions this translates into <NUM> GFLOPS/s. A comparison with other neural vocoders used for neural speech coding is given in Table <NUM>.

It should be noted, that the convolutional structure of the inventive SSMGAN allows for efficient parallel execution, which gives it a decisive advantage over autoregressive models on GPUs. Furthermore, SSMGAN runs in X times real-time on a desktop CPU. It should also run in real time on common smart-phone CPUs and will only consume a small fraction of the compute capabilities of modern Al accelerators for edge devices.

For some applications (e.g. TTS applications) some assumptions above may be relaxed, and some generalizations may be made.

The preconditioning layer <NUM> (e.g. for TTS applications, see also below, but also for other applications) is learnable.

In some examples (e.g. in TTS applications, see below), the target data <NUM> (which are often referred to as acoustic features such as mel-spectrograms, log-spectgrams, MFCCs), may also be text features and/or linguistic features.

The same applies to the bitstream <NUM>, which may be derived from text as explained above, and may include text features, acoustic features, or linguistic features (see also below).

Therefore, the learnable layer converts the bitstream <NUM> from a bitstream format onto the target data in target data format.

At first, we may consider the decoder <NUM> (generator) as being subdivided onto:.

In some cases, the block <NUM> may not be part of the decoder <NUM> (generator) (the block <NUM> may not exist or may be in an external device). With reference to <FIG> and <FIG>, the elements <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> may be considered to be part of waveform block <NUM>, while the at least one preconditioning layer <NUM> may be considered to be part of text analysis block <NUM>. With reference to <FIG>, the elements <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and 710c may be considered as being part of the block <NUM>, while text analysis block <NUM> may be external to the generator <NUM> (or it may be internal).

When referring to TTS applications, the bitstream <NUM> of <FIG>, <FIG>, <FIG> may be at least one of:.

The output (target data <NUM>) of the at least one preconditioning learnable layer <NUM> may be at least one of:.

<FIG> shows a synoptic table on the several possibilities for instantiating the at least one learnable layer <NUM> e.g. in a TTS application:.

Alternatively to the possibilities listed in <FIG> for instantiating the at least one learnable layer <NUM> e.g. in a TTS application, the target data nature may be learned and be a hidden representation of the speech or text and cannot be easily categorized or characterized. The target data <NUM> is then a form of a latent feature and is a called a latent representation, well suited for conditioning the generator network <NUM>. In this case, whatever the nature of the bitstream (acoustic, linguistic, or textual features), the output of the pre-cond network may be a latent representation made of latent features, i.e., a hidden/learned representation of the speech/text.

<FIG> and <FIG> show an example in which the bitstream <NUM> (which includes linguistic features) is inputted to the waveform synthesis block <NUM> from an external text analysis block <NUM>, and the bitstream <NUM> is processed by the at least one learnable layer 710c. Notably, the text analysis block <NUM> may be part of the generator <NUM> or may be external to the generator <NUM>. In some examples, the text analysis block <NUM> may be deterministic and the at least one preconditioning layer 710c may be learnable, but different configurations may be provided. Since the bitstream <NUM> includes linguistic features and the target data <NUM> include acoustic features, <FIG> and <FIG> pertain to case E of <FIG>.

<FIG> and <FIG> shows an example in which the text analysis block <NUM> is internal to the generator <NUM> and is embodied by the at least one preconditioning learnable layer 710b. In this case, the at least one preconditioning learnable layer 710b may only perform text analysis and provide target data <NUM> in the form of acoustic features from the bitstream <NUM> (which is in form of text <NUM>). Therefore, in <FIG> and <FIG>, we are in case C of <FIG>.

In general, the at least one learnable preconditioning layer 710b operates to elaborate the bitstream <NUM> in form of text (or other input sequence obtained from text) more and more, in a processing towards a target data <NUM> which is more elaborated than the bitstream <NUM> inputted to the at least one preconditioning learnable layer 710b. The at least one learnable preconditioning layer 710b may also use constraints (e.g. attention function, voice of man/woman, accent, emotional characterization, etc.) which may be absent in the original text. These constraints may be in general provided by the user.

It is noted that, in the cases above and below, the block <NUM> and/or the at least one learnable preconditioning layer <NUM>, 710c, 710b may use a statistical model, e.g. performing text analysis and/or using an acoustic model. In addition or in alternative, the block <NUM> and/or the at least one learnable preconditioning layer <NUM>, 710c, 710b may use a learnable model, e.g. performing text analysis and/or using an acoustic model. The learnable model may be based, for example, on neural networks, Marckv chains, ect. In further addition or in further alternative, the block <NUM> and/or the at least one learnable preconditioning layer <NUM>, 710c, 710b may make use of a rules-based algorithm performing text analysis and/or based on an acoustic model.

The block <NUM> and/or the at least one preconditioning layer <NUM>, 710c, 710b may be referred to as "text analysis block" (e.g. when converting text onto at least one linguistic feature) or "audio synthesis block" (e.g. when converting text or at least one linguistic feature onto at least one acoustic features, such as a spectrogram). Anyway, it is maintained that the target data <NUM> may be in the form of text, linguistic feature, or acoustic feature according to the embodiments.

Notably, <FIG> shows that some combinations of conversions are in general not expected. This because conversions from an elaborated feature towards a simple feature (e.g., from a linguistic feature to text or from an acoustic feature to text or a linguistic feature) is not imagined.

<FIG> shows an example of the generator 10c (decoder) for a TTS application, in which the bitstream <NUM> is obtained from a text <NUM>, e.g. received in encoded form from a remote transmitter or stored in a storage means, through a text analysis (audio synthesis) <NUM> and/or the at least one preconditioning layer <NUM>, 710c, 710b. All the subsequent operations may be understood as being performed by a waveform synthesis block <NUM> which permits to obtain the output audio signal <NUM>. Basically, all the operations performed above (e.g. by the implementation shown in <FIG>) can embody the waveform synthesis block <NUM>.

Hence, the generator 10c (decoder) may include:.

Text to speech (TTS) aims to synthesize intelligible and natural sounded speech <NUM> given a text <NUM>. It could have broad applications in the industry, especially for machine-to-human communication.

The inventive audio generator <NUM> include different components, among of them the vocoder <NUM>, in the last stage and includes mainly block(s) for converting acoustic features in audio waveform.

In particular, at block <NUM> the text <NUM> may be analyzed and linguistic features may be extracted from the text <NUM>, e.g. by a text analysis module. Text analysis may include, e.g., multiple tasks like text normalization, word segmentation, prosody prediction and graphene to phoneme. These linguistic features may be, or be comprised in, the bitstream <NUM>.

These linguistics features (bitstream <NUM>) may then be converted, e.g. through an acoustic model, to acoustics features, like MFCCs, fundamental frequency, mel-spectrogram for example, or a combinations of those. This operation may be performed by a preconditioning layer 710c, which is learnable. If the preconditioning layer 710c is a learnable layer, in some examples it may play the role of the preconditioning layer <NUM> of <FIG>.

It is worth noting that this classical pipeline can be replaced by end-to-end processing with the introduction of DNNs. For example, it is possible to condition a neural Vocoder directly from linguistic features, or an acoustic model could directly process characters bypassing the test analysis stage. For example, some end-to-end models like Tacotron <NUM> and <NUM> were proposed to simplify text analysis modules and directly take character/phoneme sequences as input, while outputting as acoustic features mel-spectrograms.

The current solution can be employed as a TTS system (i.e. including both blocks <NUM> and <NUM>), wherein the target data <NUM> may include, in some examples, a stream of information or speech representation derived from the text <NUM>. The representation could be for example characters or phonemes derived from a text <NUM>, that means usual inputs of the text analysis block <NUM>. In this case, a pre-conditioned learnable layer may be used for block <NUM> for extracting acoustics features or conditioning features appropriate (bitstream <NUM>) for the neural vocoder (e.g. block <NUM>). This pre-conditioning layer <NUM> usually leverage deep neural networks (DNNs) like an encoder-attention-decoder architecture to map characters or phonemes directly to acoustic features. Alternatively, the representation (target data) <NUM> could be or include linguistics features, that means phonemes associated with information like prosody, intonation, pauses, etc. In this case, the pre-conditioned learnable layer <NUM> can be an acoustic model mapping the linguistics features to acoustics features based on statistical models such as Hidden Markov model (HMM), deep neural network (DNN) or recurrent neural network (RNN). Finally, the target data <NUM> could include directly acoustics features derived from the text <NUM>, which may be used as conditioning features after a learnable pre-conditioning layer <NUM>. In an extreme case, the acoustic features in the target data <NUM> can be used directly as the conditioning features and the pre-conditioning layer bypassed.

The audio decoder (e.g. 10c) may obtain the input signal <NUM> from at least a time envelope information or time-domain structure derived from the bitstream <NUM>, which in turn has been obtained from the text <NUM> by the text analysis block <NUM>.

The audio synthesis block <NUM> (text analysis block) of <FIG> and <FIG> is obtained through at least one learnable layer.

In general terms, the bitstream <NUM> may include acoustic features like log-spectrogram, or a spectrogram, or MFCCs or a mel-spectrogram obtained from a text <NUM>.

In alternative, the bitstream <NUM> may include linguistics features like phonemes, words prosody, intonation, phrase breaks, or filled pauses obtained from a text.

The bitstream <NUM> may be derived from a text using at least one of statistical models, learnable models or rules-based algorithm, which may include a text analysis and/or an acoustic model.

In general terms, therefore, the audio synthesis block <NUM> which outputs the target data <NUM> from the text <NUM> (so that the target data <NUM> are derived from the text <NUM>) is a learnable block.

In general terms, the target data <NUM> may have multiple channels, while the text <NUM> (from which the target data <NUM> derive) may have one single channel.

<FIG> shows an example of generator 10c (which can be an example of the generator of <FIG>) in which the target data <NUM> comprise at least one of the acoustic features like log-spectrogram, or a spectrogram, or MFCCs or a mel-spectrogram obtained from the text <NUM>. Here, the block <NUM> may include a text analysis block <NUM> (which may be deterministic or learnable) which may provide the bitstream <NUM>. The bitstream <NUM> may include at least one of linguistic features like phonemes, words prosody, intonation, phrase breaks, or filled pauses obtained from the text <NUM>. A preconditioning layer 710c (e.g. using an acoustic model) may generate the target data <NUM> e.g. as at least one of acoustic features like log-spectrum, or a spectrogram, or MFCC(s) or mel-spectrogram(s) obtained from the text <NUM> through block <NUM>. After that, the waveform synthesis block <NUM> (which can be any of the waveform synthesis blocks discussed above, and may include at least one of blocks <NUM>, <NUM>-<NUM>, <NUM>) may be used to generate an output audio signal <NUM>. Bock <NUM> may, together with blocks <NUM>-<NUM> and <NUM>, be part of block <NUM> (e.g. <NUM>) as discussed above. The target data <NUM> may therefore be inputted in the conditional set of learnable layers <NUM>-<NUM> to obtain γ (gamma) and β (beta) (<NUM>, <NUM>), e.g. to be used for the stylistic element <NUM>.

<FIG> shows an example of a generator 10b (which may be an example of the generator 10b of <FIG>) in which the bitstream <NUM> is text <NUM> or part of text. The target data may be, for example, acoustic features (e.g. spectrograms, such as mel-spectrograms). The waveform synthesis block <NUM> (e.g. vocoder <NUM>) can output an audio signal <NUM>. The waveform synthesis block <NUM> can be any of those described in the <FIG> discussed above, but in particular like in <FIG>. In this case, for example, the target data <NUM> can be directly inputted in the conditional set of learnable layers <NUM>-<NUM> to obtain γ (gamma) and β (beta) (<NUM>, <NUM>), e.g. to be used for the stylistic element <NUM>.

In general terms, any of the audio generators above (e.g. <NUM>, 10b, 10c), the particular any of the text analysis blocks <NUM> (e.g. any of <FIG> or 9a-9c) may derive the target data <NUM> from a text using at least one of statistical models, learnable models or rules-based algorithm, e.g. comprising a text analysis and/or an acoustic model.

The target data <NUM> is obtained non-deterministically by the at least one preconditioning layer <NUM>, 1710b, 710c; block <NUM> and/or <NUM> is a learnable layer or a plurality of learnable layers.

In some examples, at least one of blocks <NUM> and <NUM> may comprise both one learnable layer and one deterministic layer.

It is noted that the examples of <FIG> may be used for a TTS application. The pitch data 3b (e.g. at least one parameter 3b indicating the pitch lag of the audio signal) or other data 3b and the filter data 3a (e.g. MFCCs, spectral envelope data) may be obtained from the bitstream <NUM>, e.g. when the bitstream <NUM> is in the form of acoustic features (e.g. case F in <FIG>). This also applies to the "first option" of <FIG>, <FIG>, <FIG>, so that the input signal <NUM> inputted to the first data provisioner <NUM> (e.g. to obtain first data, latent, prior <NUM>) may be obtained from the pitch data 3b, while the filter data (spectral envelope) 3a may be used (e.g. in the frame-by-frame branch) to be inputted onto the preconditioning learnable layer <NUM>.

Generally, examples may be implemented as a computer program product with program instructions, the program instructions being operative for performing one of the methods when the computer program product runs on a computer. The program instructions may for example be stored on a machine readable medium.

Other examples comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier. In other words, an example of method is, therefore, a computer program having a program instructions for performing one of the methods described herein, when the computer program runs on a computer.

A further example of the methods is, therefore, a data carrier medium (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier medium, the digital storage medium or the recorded medium are tangible and/or non-transitionary, rather than signals which are intangible and transitory.

A further example of the method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be transferred via a data communication connection, for example via the Internet.

A further example comprises a processing means, for example a computer, or a programmable logic device performing one of the methods described herein.

A further example comprises a computer having installed thereon the computer program for performing one of the methods described herein.

A further example comprises an apparatus or a system transferring (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver.

In some examples, a programmable logic device (for example, a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some examples, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods may be performed by any appropriate hardware apparatus.

The above described examples are merely illustrative for the principles discussed above. It is understood that modifications and variations of the arrangements and the details described herein will be apparent. It is the intent, therefore, to be limited by the scope of the claims and not by the specific details presented by way of description and explanation of the examples herein.

Claim 1:
Audio decoder (<NUM>), configured to generate an audio signal (<NUM>) from a bitstream (<NUM>), the bitstream (<NUM>) representing the audio signal (<NUM>), the audio signal being subdivided in a sequence of frames, the audio decoder (<NUM>) comprising:
a first data provisioner (<NUM>) configured to provide, for a given frame, first data (<NUM>) derived from an input signal (<NUM>) from an external or internal source or from the bitstream (<NUM>), wherein the first data (<NUM>) has multiple channels;
a first processing block (<NUM>, <NUM>, 50a-<NUM>), configured, for the given frame, to receive the first data (<NUM>) and to output first output data (<NUM>) in the given frame, wherein the first output data (<NUM>) comprises a plurality of channels (<NUM>), and
a second processing block (<NUM>), configured, for the given frame, to receive, as second data, the first output data (<NUM>) or data derived from the first output data (<NUM>),
wherein the first processing block (<NUM>) comprises:
at least one preconditioning learnable layer (<NUM>) configured to receive the bitstream (<NUM>) and, for the given frame, output target data (<NUM>) representing the audio signal (<NUM>) in the given frame with multiple channels and multiple samples for the given frame;
at least one conditioning learnable layer (<NUM>, <NUM>, <NUM>) configured, for the given frame, to process the target data (<NUM>) to obtain conditioning feature parameters (<NUM>, <NUM>) for the given frame; and
a styling element (<NUM>), configured to apply the conditioning feature parameters (<NUM>, <NUM>) to the first data (<NUM>, 59a) or normalized first data (<NUM>, <NUM>'); and
wherein the second processing block (<NUM>) is configured to combine the plurality of channels (<NUM>) of the second data (<NUM>) to obtain the audio signal (<NUM>),
characterized in that the first processing block (<NUM>) is configured to up-sample the first data (<NUM>) from a first number of samples for the given frame to a second number of samples for the given frame greater than the first number of samples.