Patent Description:
The present disclosure relates to a method for performing packet loss concealment using neural network system, a method for training a neural network system for packet loss concealment and a computer implemented neural network system implementing said method.

Most implementations within the field of communication technology operate under constrained real-time conditions to ensure that users do not experience any delay or interruptions in the communication. The Voice over Internet Protocol (VoIP) is one example of a communication protocol that operates under strict real-time conditions to enable users to have a natural conversation. To fulfil the strict conditions VoIP and similar communication protocols rely on, a steady stream of packets, each carrying a portion of the communication signal, that are continuously and without interruptions transmitted from a sending entity to a receiving entity. However, in practice packets are often delayed, delivered to the receiving entity in a wrong order, or even lost entirely, introducing distortions and interruptions in the communication signal that are noticeable and that degrade the communication quality experienced by users.

To this end there is a need for an improved method of performing Packet Loss Concealment (PLC).

In <NPL>a generative adversarial network designed to restore missing musical audio data is introduced. In<NPL> there is a brief review of how the problem of packet-loss in a realistic setting is modelled, and how to evaluate Packet Loss Concealment techniques. Finally, in <NPL> a simple yet powerful generative model that learns discrete representations is proposed.

Previous solutions for performing packet loss concealment involve replicating the structure of the most recent packet and employing audio processing, causing the signal energy to decay to naturally extend the duration in time of the latest packet in lieu of a next packet. However, while the previous solutions decrease the noticeability of lost packets to some extent, the interruptions of the communication signal still impedes on the communication signal quality and, especially for long interruptions, users can still perceive distortions in the communication signal even after PLC processing.

To this end, it is an object of the present disclosure to provide an improved method and system for performing packet loss concealment.

According to a first aspect of the disclosure, there is provided a method for packet loss concealment of an incomplete audio signal where the incomplete audio signal includes a substitute signal portion replacing an original signal portion of a complete audio signal. The method includes obtaining a representation of the incomplete audio signal and inputting the representation of the incomplete audio signal to an encoder neural network trained to predict a latent representation of a complete audio signal given a representation of an incomplete audio signal. The encoder neural network outputs a latent representation of a predicted complete audio signal, and the latent representation is input to a decoder neural network trained to predict a representation of a complete audio signal given a latent representation of a complete audio signal, wherein the decoder neural network outputs a representation of the predicted complete audio signal comprising a reconstruction of the original portion of the complete audio signal, wherein the encoder neural network and the decoder neural network have been trained with an adversarial neural network.

With a substitute signal portion it is meant a signal portion which replaces (is a substitute of) a corresponding (real) portion of a complete audio signal. For example, the substitute signal portion may be a silent (zero) signal portion which indicates that a portion of the signal has been lost, is corrupted or is missing. While a zero signal is commonly used to indicate a portion of a signal which has not been received or is missing other substitute signal portions may be employed, e.g. sawtooth signal of a certain frequency or any other predetermined type of signal which has been established to represent a substitute signal portion in lieu of the actual signal portion of the complete audio signal. In some implementations, the missing signal portion is indicated as metadata to enable distinguishing of the substitute signal portion from e.g., an actual completely silent (zero) signal portion.

The method further comprises quantizing the latent representation of the complete audio signal to obtain a quantized latent representation, wherein the quantized latent representation is formed by selecting a set of tokens out of a predetermined vocabulary set of tokens. At least one token of the quantized latent representation is used to condition a generative neural network, wherein the generative neural network is trained to predict a token of the set of tokens provided at least one different token of the set of tokens wherein the generative latent model outputs a predicted token of the latent representation and a confidence metric associated with the predicted token. Based on the confidence of the predicted token, a corresponding token of the quantized latent representation is replaced with the predicted token of the generative model to form a corrected set of tokens (corrected quantized representation) which is provided to the decoder neural network.

The above described neural network system comprises a deep causal adversarial auto-encoder formed by the encoder neural network and the decoder neural network which have learned together to generate a representation of a reconstructed complete audio signal provided an incomplete audio signal. The causal auto-encoder is a non-autoregressive model that may predict an arbitrarily long signal portion (e.g. spanning several packets) with a single inference step. In some implementations the decoder outputs a waveform representation directly and, due to the adversarial training, the outputted waveform may be a very accurate reconstruction of the complete audio signal. The causal auto-encoder may generate reconstructed complete audio signals wherein the substitute signal portion is beyond <NUM> milliseconds on the contrary to most existing models that make the generation frame-by-frame using an autoregressive loop.

There is no time dependency in the causal auto-encoder meaning that the model may output any length of reconstructed audio signal (at any sample rate) in one single feed-forward step, contrarily to the majority of state of the art packet loss concealment solutions that employ some form of autoregressive loop from output to input. With the optional generative latent model the packet loss concealment performance for long duration losses may be enhanced. Additionally, the causal auto-encoder is deterministic meaning that a same input will yield a same output. Nevertheless, the causal auto-encoder may be referred to as a generator in a generator-adversarial training setup as the causal auto-encoder generates data which emulates real data. This is in contrast to other generators which rely on a random variable to perform the generation of new data which would not make for a deterministic process.

The causal auto-encoder is assisted by a generative latent model which operates on a quantized latent representation of the causal auto-encoder. The generative latent model especially enables signal reconstruction on longer terms e.g., beyond <NUM> milliseconds, but also enables facilitated reconstruction for reconstructions of any length.

According to a second aspect of the disclosure there is provided a computer implemented neural network system for packet loss concealment acoording to claim <NUM>.

According to a third aspect of the disclosure, there is provided a method for training a neural network system for packet loss concealment accoring to claim <NUM>.

The disclosure according to the second and third aspects features the same or equivalent embodiments and benefits as the disclosure according to the first aspect. For example, the encoder and decoder of the neural network system may have been trained using a discriminator with at least two discriminator branches operating at different sample rates. Further, any functions described in relation to a method, may have corresponding structural features in a system or code for performing such functions in a computer program product.

Exemplifying embodiments will be described below with reference to the accompanying drawings, in which:.

<FIG> illustrates schematically the causal adversarial auto-encoder (causal auto-encoder). An incomplete audio signal <NUM> comprising a substitute signal portion <NUM> is input to the causal adversarial auto-encoder optionally via a transform unit <NUM>. The transform unit <NUM> transforms the incomplete audio signal <NUM> into a representation of the incomplete audio signal <NUM>. For example, the transform unit <NUM> may transform a time domain (waveform) representation of the incomplete audio signal <NUM> into a representation format chosen from a group comprising: a frequency domain representation, a time representation, a filter bank representation and a feature domain representation. Examples of such representations include a Mel-frequency cepstral coefficient representation or a short-time Fourier transform representation. The encoder neural network <NUM> may obtain the representation of the incomplete audio signal <NUM> from the transform unit <NUM>. Alternatively, the incomplete audio signal <NUM> is obtained directly, e.g. in a waveform time domain representation, by the encoder neural network <NUM>.

The incomplete audio signal <NUM> may be subdivided into one or more frames with the encoder neural network <NUM> accepting one or more frames for each inference step. For example, the encoder neural network <NUM> may have a receptive field of <NUM> milliseconds wherein a short-time Fourier transform (STFT) spectral frame is generated from the incomplete audio signal <NUM> with an interval of <NUM> milliseconds with some overlap (resulting in a signal sampled at <NUM>), meaning that the encoder neural network <NUM> accepts <NUM> frames for each inference step. The incomplete audio signal <NUM> may further be divided into a set of packets wherein each packet comprises one or more frames or a representation of a portion of the complete audio signal. If one or more packets from the set of packets is omitted (which occurs during a packet loss) a signal portion and/or one or more frames which were present in the complete audio signal is unavailable, and the incomplete audio signal <NUM> is thereby a representation of the available information with a substitute signal portion <NUM> replacing the signal portion(s) of the complete audio signal that is/are unavailable.

The encoder neural network <NUM> is trained to predict and output a latent representation of the reconstructed complete audio signal <NUM>. That is, the latent representation of the reconstructed complete audio signal <NUM> is a prediction of the original complete audio signal given the incomplete audio signal <NUM> (or a representation thereof), wherein the reconstructed complete audio signal <NUM> comprises a reconstructed signal portion <NUM> replacing the substitute signal portion <NUM> of the incomplete audio signal <NUM>. The latent representation is quantized and processed using a generative model <NUM> which will be described in detail in relation to <FIG>. The latent representation is provided as an input to the decoder neural network <NUM> which may be trained to predict the time representation (e.g. the waveform) of the reconstructed complete audio signal <NUM> directly irrespectively of if the encoder neural network <NUM> received a waveform representation or any other representation of the incomplete audio signal <NUM>. Alternatively, the decoder neural network <NUM> may output a representation of the reconstructed complete audio signal <NUM> which is converted to a waveform representation by an optional inverse transform unit <NUM>.

The receptive field of the encoder neural network <NUM> is preferably wide enough to capture a long context so as to be resilient to recent signal losses that may occur in proximity to a current signal portion which is to be reconstructed. The receptive field may be approximately <NUM> milliseconds and reducing the receptive field (e.g. to <NUM> milliseconds) may reduce the reconstruction quality.

<FIG> illustrates the quantization and generative model <NUM> of the causal auto-encoder. The generative latent model (GLM) 160a 160b operates autoregressively on a quantized representation 22a, 22b of the latent representation. As apparent in <FIG> the GLM 160a, 160b operates in the latent representation domain. The latent representation may be referred to as the context domain in which the audio signal e.g. is represented with context vectors.

The latent representation output by the encoder neural network <NUM> is fed to a quantization block <NUM>. The quantization block <NUM> performs at least one transformation of the latent representation to form at least one quantized latent representation 22a, 22b. In some implementations the quantization block <NUM> performs at least one linear transformation of the latent representation to form at least one quantized latent representation 22a, 22b. In some implementations, the quantization block <NUM> performs quantization of the latent representation by selecting a predetermined number of tokens to represent the latent representation, wherein the tokens are selected from a from a predetermined vocabulary of possible tokens. For instance, the quantization may be a vector quantization wherein a predetermined number of quantization vectors are selected from a predetermined codebook of quantization vectors to describe the latent representation as a quantized latent representation 22a, 22b.

In some implementations, the quantization of quantization block <NUM> comprises selecting a first set of tokens from a first vocabulary forming a first quantized representation 22a and selecting a second set of tokens from a second (different) vocabulary to form a second quantized representation 22b. The number of tokens in each of the first and second sets may be the same. The first set of tokens and the second set of tokens are alternative representations of the same latent representation. Accordingly, the quantization block <NUM> may provide as an output one, two, three or more quantized latent representations 22a, 22b. For example, the quantization block <NUM> may be a multi-head vector (VQ) quantization block.

The quantized latent representation 22a, 22b is provided to the decoder neural network <NUM>. The quantized latent representation may be of an arbitrary dimension e.g. using <NUM>, <NUM>, <NUM>, <NUM> tokens or quantization vectors.

In some implementations each of the at least one quantized latent representation 22a, 22b is associated with a respective GLM 160a, 160b that operates autoregressively on the associated set of tokens 22a, 22b. The GLM 160a is trained to predict the likelihood of at least one token given at least one other token of the set of tokens forming the quantized representation 22a. For example, the GLM 160a may be trained to predict at least one future token (selected from the vocabulary of tokens) given at least one previous token. The GLM 160a may be trained to predict a likelihood associated with each token in the set of tokens from the quantized latent representation 22a wherein the likelihood indicates the likelihood that the token should be at least one particular token from the associated vocabulary of tokens. That is, the GLM 160a may continuously predict new tokens given past tokens or predict a correction of a current set of tokens wherein the new or corrected predicted tokens are either the same or different from the tokens outputted by the quantization block <NUM>. In the comparing block <NUM>, the predicted token sequence predicted 23a is compared to the token sequence 22a output by the quantization block <NUM>. If there is a difference for at least one predicted token in the GLM predicted set of tokens 23a and the set of tokens 22a output from the quantization block <NUM>, a selection is made to use one of the tokens predicted in either set of tokens 22a, 23a. For example, the selection of token is based on the likelihood of the token predicted by the GLM 160a and/or encoder <NUM>. For example, if the GLM prediction likelihood is below a predetermined likelihood threshold, the token of the quantization block <NUM> is block is used. By means of a further example, the token selection is based on the likelihood of each token predicted by the GLM 160a and/or the Argmin distance with respect to the non-quantized latent representation.

Analogously, and in parallel, a second GLM 160b may predict at least one future/corrected token provided at least one token of a second latent representation 22b so as to form a predicted second set of tokens 23b. Similarly, based on the likelihood of the tokens predicted by the second GLM 160b the second tokens output by the quantization block <NUM> is compared to the second predicted tokens output by the second GLM 160b. If a difference is detected the selection of tokens is made based on the likelihood of the second tokens predicted by the second GLM 160b and/or the Argmin distance to the non-quantized latent representation.

Analogously, three or more quantized latent representations may be obtained each with an associated GLM performing predictions on likely continuations/corrections of the token sequence which may differ from the actual sequence as output by the quantization block <NUM>.

If a single quantized latent representation 22a and GLM 160a is used the most likely token sequence may be forwarded to the decoder neural network <NUM> to make the waveform prediction based on the quantized latent representation selected by the comparison block <NUM>.

If more than one quantized latent representation 22a, 22b and GLM 160a, 160b are used, the respective quantized latent representation 22a, 23b selected by the comparison block <NUM> may be concatenated or added in the concatenation block <NUM> to form an aggregate representation <NUM>. Accordingly, the bitrate may be increased by concatenating additional quantized representations 22a, 23b and forwarding the aggregated representation <NUM> to the decoder neural network <NUM>.

The GLM 160a, 160b is a discrete autoregressive model trained with a maximum likelihood criterion. Each GLM 160a, 160b may be configured to operate similarily to a language model in the natural language processing domain. Hence several neural architectures may be used to perform the task of the GLM 160a, 160b such as one or more causal convolutional networks, recurrent neural networks or self-attention models. The GLM 160a, 160b may add a capability of performing longer-term predictions in addition to the causal adversarial auto-encoder due to its generative nature. Arbitrarily large continuations of a latent representation may be predicted by the GLM(s) 160a, 160b. It is noted that the quantization and GLM(s) 160a, 160b are added to the causal adversarial auto-encoder so as to enable facilitated longer term predictions. For instance, the GLM(s) 160a, 160b may be operated adaptively and only activated in response to substitute signal portions exceeding a threshold duration.

<FIG> depicts a detailed exemplary implementation of the causal adversarial auto-encoder. The encoder neural network <NUM> may comprise one or more encoder blocks <NUM>' each comprising one or more neural network layers <NUM>, <NUM>, <NUM>. Multiple encoder blocks <NUM>' may be cascaded with each encoder block <NUM>', each block having the same structure. In one implementation one encoder block <NUM>' may comprise a normalization layer <NUM>. Layer normalizations may help to accelerate the model's convergence and contribute to yielding a better signal reconstruction quality. The normalization layer <NUM> may be followed by a causal convolutional layer <NUM> configured to perform causal convolution. The causal convolutional layer <NUM> may use weight normalization as it is a simple stabilization mechanism for adversarial models that generate waveforms. The causal convolutional layer <NUM> may be a dilated causal convolutional layer with a dilation factor d. Accordingly, the encoder <NUM> and encoder block <NUM>' is easily adaptable regarding receptive field modifications thanks to the dilation factor d per encoder block <NUM>'. In some implementations, the receptive field may be wider so as to increase the prediction quality. For example, the receptive field may be longer than <NUM> second, longer than <NUM> seconds or longer than <NUM> seconds. Each encoder block <NUM>' may comprise a skip connection to facilitate training.

The invention comprises quantization of the latent representation output by the encoder block(s) <NUM>'. The quantization may comprise one or more linear transformations of the output of the encoder block(s) <NUM>'. The linear transformation may be performed by a linear transformation block <NUM>' which outputs at least one quantized latent representation <NUM> which represents a reconstructed complete audio signal.

The quantized latent representation(s) <NUM> is provided to the decoder neural network <NUM>. The decoder neural network may comprise one or more cascaded decoder block(s) <NUM>' wherein each decoder block <NUM>' comprises one or more neural network layers. Optionally the decoder block(s) <NUM>' is preceded by a causal convolutional layer <NUM> which performs an initial upsampling of the latent representation or quantized latent representation <NUM>.

In one implementation a decoder block <NUM>' comprises a leaky ReLU (Rectified Linear Unit) layer <NUM>. Using leaky ReLU layers <NUM> as non-linear activations may reduce gradient flow issues. The leaky ReLU layer <NUM> may be followed by a transposed convolutional layer <NUM> which in turn is followed by one or more residual causal convolutional blocks 143a, 143b, 143c with different dilation factors D. In one implementation the dilation factors D of the residual causal convolutional layers 143a, 143b, 143c increases, for instance the first dilation factor is <NUM>, the second dilation factor is D and the third dilation factor is D<NUM> wherein D is an integer. An example of the residual causal convolutional block 143a, 143b, 143c is illustrated in detail in <FIG>. Accordingly, the decoder block(s) <NUM>' converts the latent representation or quantized latent representation <NUM> back to a waveform representation of a reconstructed complete audio signal <NUM> through a series of transposed convolutions (i.e. learnable up-sampling filters) and residual causal convolutional blocks 143a, 143b, 143c with different dilation factors. Alternatively, the decoder block(s) outputs a representation of a reconstructed complete audio signal <NUM>, wherein the waveform representation of the reconstructed complete audio signal is obtained by an inverse transform unit (not shown).

In some implementations the output of the decoder block(s) <NUM>' is provided to one or more a post processing layers <NUM>, <NUM>. In one exemplary implementation the post processing layers <NUM>, <NUM> comprise a linear transformation layer <NUM> with non-linear (e.g. Tanh) activation <NUM>.

The final sampling rate of the reconstructed complete audio signal <NUM> is determined by the number of transposed convolutions <NUM> (i.e. the number of cascaded decoder blocks <NUM>') and their striding factors. In one implementation the decoder <NUM> is comprised of one or more decoder blocks <NUM>' such as four decoder blocks <NUM>' with different upsampling factors. For example, the upsampling factors may be <NUM>, <NUM>, <NUM>, and <NUM> for each of the decoder blocks <NUM>'. However, other factors may be employed and fewer or more decoder blocks <NUM>' may be stacked to obtain any arbitrary sampling rate in the output reconstructed audio signal <NUM>. The transposed convolutions may be restricted to be non-overlapped so that causality is not broken while upsampling (i.e., there is no overlap among transposed convolution outputs from future data).

In some implementations, the causal adversarial auto-encoder may comprise an optional cross-fading post-filtering module configured to receive at least one reconstructed signal portion <NUM> and a subsequent signal portion (e.g. a signal portion which is indicated to be a representation of complete audio signal portion and not a substitute signal portion) and apply a cross-fading filter (e.g. a window function) to ensure a smooth transition between the reconstructed audio signal portion <NUM> and the complete audio signal portion. The cross-fading filter may then be applied to the reconstructed audio signal <NUM>. In some implementations the optional cross-fading post-filtering module comprises one or more neural networks trained to predict a cross fading filter provided at least a reconstructed signal portion and a subsequent and/or preceding portion of the complete audio signal. A benefit with using a neural network is that the neural network may be trained to adapt the predicted cross-fading filter after different acoustic conditions (e.g. noise, codec artifacts, reverberation effects) that are present in the training data.

<FIG> illustrates an exemplary residual causal convolutional block <NUM> which comprises a 1x1 causal convolutional block <NUM> with dilation factor D and a skip connection. As depicted in <FIG> the residual causal convolutional block <NUM> may comprise Leaky ReLU activation via a Leaky ReLU layer <NUM> which precedes the causal convolutional layer <NUM>. Optionally, the causal convolutional layer <NUM> may be followed by a second Leaky ReLU layer <NUM> which in turn is followed by a linear transformation layer <NUM>. Each of the one or more causal convolutional block(s) <NUM> may comprise a respective skip connection to facilitate model training.

<FIG> depicts a discriminator <NUM> according to some implementations of the present disclosure. The discriminator <NUM> comprises a discriminator neural network, wherein the discriminator neural network may comprise a plurality of neural network layers. The discriminator <NUM> is trained to predict an indicator <NUM> indicating whether the input complete audio signal <NUM> represents a (fake) reconstructed audio signal or a (real) complete audio signal. The discriminator <NUM> may be configured to output an indicator <NUM> indicating whether the input data <NUM> represents a complete audio signal or an audio signal comprising at least a reconstructed portion. The indicator <NUM> may be a Boolean variable or a value, wherein the value is between <NUM> and <NUM> where <NUM> indicates that the input audio signal is fake and <NUM> indicates that input audio signal is real.

In some implementations, the discriminator <NUM> comprises two, three, or more discriminator branches 210a, 210b, 210c, each trained to predict a respective individual indicator 25a, 25b, 25c indicating whether the input data represents a complete audio signal or a reconstructed audio signal. In one example, a first discriminator branch 210a obtains a representation of the input audio signal <NUM> whereas a second discriminator 210b branch obtains a downsampled representation of the same input audio signal <NUM>. Additionally, a third discriminator 210c branch may obtain a further downsampled representation of the same input audio signal <NUM>. To this end, the second discriminator branch 210b may be preceded by a downsampling stage <NUM>, whereas the third discriminator branch 210c is preceded by two downsampling stages <NUM>. Each downsampling stage may perform downsampling using a same factor S or individual factors S1, S2 wherein S2 ≠ S1.

Accordingly, each discriminator branch 210a, 210b, 210c predicts an individual indicator 25a, 25b, 25c indicating whether the input audio signal <NUM> appears to be a complete audio signal or a reconstructed audio signal at different sampling rates. Each indicator 25a, 25b, 25c is aggregated at an indicator aggregation stage <NUM> to form a total indicator <NUM> which indicates whether the input audio signal <NUM> is a complete audio signal or a reconstructed audio signal. The indicator aggregation stage <NUM> may determine the total indicator <NUM> based on the number of discriminator branches 210a, 210b, 210c indicating that the input audio signal is real or fake. The indicator aggregation stage <NUM> may determine the total indicator <NUM> based on a weighted sum of the individual indicators 25a, 25b, 25c of each respective discriminator branch 210a, 210b, 210c. The weighted sum may be weighted with a likelihood associated with each individual indicator 25a, 25b, 25c. Other aggregation or pooling strategies may be employed to generate the total indicator <NUM> from the individual indicators 25a, 25b, 25c. For instance, the most confident individual indicator of the individual indicators 25a, 25b, 25c may be taken as the total indicator <NUM> (max-pooling). That is, the total indicator may e.g. be an average, weighted average, or maximum of the individual indicators 25a, 25b, 25c.

<FIG> illustrates a detailed view of a discriminator <NUM> or discriminator branch <NUM>. A reconstructed complete audio signal or a complete audio signal is provided as the input the discriminator <NUM> which is configured to predict an indicator <NUM> indicating the likelihood that the input is a reconstructed complete audio signal <NUM> or a complete audio signal which has not been reconstructed using the neural system described in the above. The discriminator may comprise multiple neural network layers 211a, 212a, 211b, 212b, 211c, 212c, 211d, 212d. For example, the multiple neural network layers 211a, 212a, 211b, 212b, 211c, 212c, 211d, 212d may be a stack of subblocks wherein each subblock comprises a convolutional layer 211a and a Leaky ReLU activation layer 212a. Each convolutional layer 211a, 211b, 211c, 211d may be configured to downsample by a factor S. The convolutional layers 211a, 211b, 211c, 211d of the discriminator may have comparatively large kernels, e.g. the kernels may be <NUM> times the stride + <NUM>. In the discriminator <NUM>, the resolution in time of each feature map is reduced, and the number of features grows to compensate the time-dimension reduction. An output convolutional layer <NUM> may be configured to perform the final prediction of the indicator <NUM> for the discriminator <NUM>. All convolutional layers in the discriminator may use weight normalization to facilitate training stabilization.

With reference to <FIG> there is illustrated an exemplary Generative Adversarial Network (GAN) comprising two neural networks, a generator neural network <NUM> and a discriminator neural network <NUM>, that are trained together with opposite objectives. The generator neural network <NUM>, referred to as the generator <NUM>, learns to imitate the structure embedded in the training data 310b provided to the generator <NUM> such that the generator <NUM> can generate more samples that would be a plausible alternative to the ones existing in the training data 310b.

In GAN training, the training data 310b input to the generator <NUM> may be a vector of random noise samples z with some distribution Z. The distribution may be a uniform or Gaussian distribution while other distributions are possible. This would make the generator <NUM> a generator of random samples that resemble the samples of the training data 310b. Additionally or alternatively, training data 310a which comprises an example of audio data (e.g. recorded speech or general audio) may be used as the training data during GAN training. For instance, incomplete training data may be used wherein the generator <NUM> is tasked with predicting the continuation of the training signal as realistically as possible or fill-in a substitute (missing) signal portion of the training data. For example, if the training data 310a, 310b comprises a current melspectrogram, the generator <NUM> may generate future melspectrograms that fits as a realistic continuation of the current melspectrogram.

On the other hand, the discriminator neural network <NUM>, referred to as the discriminator <NUM>, is trained to detect whether the generated data output by the generator <NUM> is a reconstructed (fake) version of the original (real) data. The discriminator <NUM> may be simply seen as a classifier that identifies whether an input signal is real or fake. The discriminator <NUM> may be seen as a learnable (non-linear) loss function as it replaces and/or compliments a specified loss function for use during training.

The training process is summarized in <FIG>. In a first training mode 300b, the discriminator <NUM> learns to classify whether the input comes from the training dataset 310a, 310b or the generator <NUM>. In the first training mode, the internal weights of the generator <NUM> are not modified (i.e. the generator <NUM> is not learning and is in a frozen state) while the internal weights of the discriminator are updated so as to recognize the output from the generator <NUM> as fake. In the second training mode 300c, the discriminator <NUM> is frozen while the internal weights of the generator <NUM> are updated so as to make the output of the discriminator <NUM> misclassify the predicted output from the generator <NUM> as a real signal. That is, if the generator <NUM> is successful in the second training mode 300c the generator <NUM> is able to reproduce a reconstructed signal well enough to succeed in making the discriminator <NUM> misclassify the input as an original (real) signal. By alternating between the first training mode 300b and the second training mode 300c and/or using different training data the generator <NUM> and discriminator <NUM> are trained together.

Optionally, the discriminator <NUM> may be trained in a third training mode 300a using training data 310a representing a complete audio signal. In the third training mode the internal weights of the discriminator <NUM> are updated so as to classify the training data 310a which represents a complete signal as real.

The training data 310a, 310b may comprise at least one example of an audio signal comprising speech. For example, the training data 310a, 310b may comprise a variant of or a mixture of publicly available datasets for speech synthesis, such as VCTK and LibriTTS. Both of these may be resampled at <NUM>, but the model may be adaptable to work at higher and lower sampling rates as well, e.g. by adjusting the decoder strides. The training data 310a, 310b may comprise clean speech, but additional training data 310a, 310b may be obtained by augmenting the clean speech to introduce codec artifacts which may emulate the artifacts that might be present in real communication scenarios. For instance, for each utterance in the training data 310a, 310b one of the following codecs may be applied randomly with a random bitrate amongst the possible ones:.

The above listed codecs are only exemplary and additional or other codecs may be used as an alternative to the above. For example, a codec with possible bitrates of <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> kbps may be used.

Additionally, the training data 310a, 310b may be further augmented by the addition of noise, reverberation, and other acoustic variabilities such as number of speakers, accents, or languages coming from other dataset sources.

The training data 310a, 310b may be augmented by randomly replacing portions of the training data audio signal with a substitute signal portion of random length. The portions which are replaced with a substitute signal portion may correspond to the audio signal portions of one or more packets and/or frames. For example, the training data 310a, 310b may be augmented by omitting one or more packets and/or frames of a packetized or frame representation of the training data audio signal wherein each omitted packet and/or frame is replaced with a substitute signal portion of corresponding length. Additionally or alternatively, two portions of a training data audio signal may be swapped or an audio signal of a second training data audio signal may be added as a substitute audio signal portion of a first training data audio signal. That is, the training data 310a, 310b may comprise a concatenation of two chunks that belong to different utterances, therefore provoking a sudden linguistic mismatch. Accordingly, the generator and discriminator may be trained with another loss that enforces linguistic content continuity. Preferably, the two mismatched audio signal portions are real signal portions such that the discriminator learns to detect incoherent contents and the generator learns to generate realistic (in signal quality) and coherent (linguistically).

The training data 310a, 310b may define a past audio signal wherein the generator <NUM> is trained to construct a plausible future continuation of given the past audio signal so as to succeed in making the discriminator <NUM> misclassify the constructed future signal as an a real audio signal. For example, if operating with melspectrogram representations, the generator <NUM> should generate future trajectories that look like real melspectrograms and that fit as continuations of a past or current melspectrogram. Additionally or alternatively, the training data 310a, 310b may be random noise samples drawn from a predetermined noise distribution (e.g. a Gaussian or uniform distribution) wherein the generator <NUM> is tasked with reconstructing substitute signal portion or constructing a future signal given the noise training data so as to succeed in making the discriminator <NUM> misclassify the reconstructed noise data.

<FIG> depicts an exemplary training setup for the causal adversarial auto-encoder. The causal adversarial auto-encoder comprises the encoder neural network <NUM>, optionally the generative model <NUM>, and the decoder neural network <NUM>. The generative model <NUM> may comprise a quantization block so as to operate with quantized latent representations and comprise at least one generative latent model (GLM) which is trained together with the rest of the causal adversarial auto-encoder. During training, the causal adversarial auto-encoder learns to reconstruct an incomplete audio signal <NUM> which comprises a substitute signal portion <NUM> to form a reconstructed complete audio signal <NUM> comprising a reconstructed signal portion <NUM> which replaces the substitute signal portion <NUM>. Ideally, the reconstructed audio signal <NUM> is similar and/or identical to the complete audio signal <NUM>. These reconstructions may be learned through one of, or a combination of, two losses that operate upon the reconstructed audio signal <NUM>, namely the multi-STFT loss <NUM> and the adversarial (discriminator) loss <NUM>. The multi-STFT minimizes the difference of magnitudes between the predicted reconstructed audio signal <NUM> and the complete audio signal <NUM>. Preferably, this difference is computed at different time-frequency resolutions. The multi-STFT loss <NUM> helps the model converge faster and achieve better quality than if only the adversarial loss obtained with the discriminator <NUM> is used. However, the multi-STFT loss <NUM> may not be sufficient to generate good quality speech due to it lacking a phase reconstruction objective. Hence a model trained only with the multi-STFT may sound robotic and with artifacts in speaker identity and prosody although the reconstructed signal may be completely intelligible.

In some implementations the adversarial loss may be obtained for the reconstructed audio signal <NUM>. Moreover, the training may be conducted based on a conjunction of the multi-STFT <NUM> loss and adversarial loss obtained with the discriminator <NUM>. Other regression losses like L1-loss may be used as an alternative to, or in addition to, the adversarial loss and/or the multi-STFT loss <NUM>. Other deep feature losses may also be utilized, such as regressions of features extracted through deep neural speech encoders such as PASE or contrastive losses that enrich the knowledge on context such as contrastive predictive coding.

For example, the discriminator <NUM> may comprise a plurality of branches wherein an indicator of each branch is aggregated to form a final aggregation loss. The mean squared error may be used to aggregate the losses and provide value of <NUM> when its input is fake and <NUM> when its input is real. The model may then be a least-squares GAN formulation, while other formulations like Hinge or Wasserstein are applicable as well.

Once the causal adversarial auto-encoder is trained, it can be used in inference mode with or without the quantization and/or the generative latent model. The substitute signal portion <NUM> is indicated in the inference mode with the same pattern as in the training mode (e.g. by zeros that cover the length of the gap). However, by training the causal adversarial auto-encoder with another type of substitute of signal portion <NUM>, any other type of substitute signal portion pattern may be used.

An additional substitute indicator may be received together with the substitute signal portion <NUM> itself. The additional substitute signal indicator may be included as metadata, and may indicate that the substitute signal portion <NUM> is an actual substitute portion and not a part of the complete audio signal <NUM>. If melspectrograms are used to represent the audio signal, the additional substitute indicator may be concatenated as an additional frequency channel in the melspectrogram.

Using the additional substitute indicator, the causal adversarial auto-encoder may select to replace only the substitute signal portion with a reconstructed signal portion while keeping the other portions of the incomplete audio signal intact. In doing so, the causal adversarial auto-encoder may perform a cross-fading operation by applying a window function to smoothly transition between the incomplete audio signal and the reconstructed signal portion of the incomplete audio signal to form the reconstructed audio signal <NUM>. In some implementations, the cross-fading operation is performed when a future portion of the complete audio signal <NUM> is available, meanwhile zeros (or any other substitute signal pattern) is appended to the incomplete audio signal <NUM>. Additionally or alternatively, the causal adversarial auto-encoder reconstructs several frames of the audio signal in one inference step and keeps the frames buffered until a next packet (comprising one or more frames) is received. During this time a window function may be slid over the buffered frames to save the number of interference steps per unit of time for the audio signal.

<FIG> illustrates a flowchart describing a method according to embodiments of the present disclosure. At S10 an incomplete audio signal, or a representation of an incomplete audio signal, is received wherein the incomplete audio signal comprises a substitute signal portion replacing an original signal portion. The incomplete audio signal (or representation thereof) is provided to an encoder neural network at S11 which outputs a latent representation of a reconstructed audio signal at S12. The reconstructed complete audio signal may be quantized at optional step S13 to form a quantized representation of the reconstructed audio signal which comprises a set of tokens. The at least one quantized latent representation is provided to a respective GLM at S14 which predicts an updated set of (more likely) tokens given at least one token of the quantized representation. At S15, the tokens of the quantized (original) representation are compared to the tokens predicted by the GLM at S14. If there is a difference, a selection of the most likely tokens is made to form a final predicted set of tokens. The final predicted set of tokens is inputted to the decoder neural network at S16, and at S17 the decoder neural network outputs a reconstructed audio signal in the time domain (a waveform) or a representation of the reconstructed audio signal. Optionally, S17 is followed by post processing which may involve performing an inverse transform or cross-fading-filtering as described in the above.

<FIG> illustrates a flowchart which describes a method for training the causal auto-encoder according to embodiments of the present disclosure. At S20 a neural network system (e.g. an implementation of the causal auto-encoder) is provided alongside a discriminator neural network and training data. At S21 the training data is provided to the neural network system, and at S22 a reconstructed audio signal (or representation thereof) is output by the neural network system. For example, the neural network system has replaced a substitute signal portion of the training data signal with a reconstructed signal portion or has predicted a reconstructed future continuation of the training data audio signal. At S24a, the reconstructed audio signal is provided to a discriminator which outputs a first likelihood (indicator) indicating whether the reconstructed audio signal is a complete audio signal or not at S25a. Optionally, a second discriminator branch receives the reconstructed audio signal output at S22 and downsamples the reconstructed audio signal (or representation thereof) at S23 prior to providing the downsampled reconstructed audio signal (or representation thereof) to a second discriminator (discriminator branch) at S24b. At S25a the second discriminator outputs a second likelihood indicating whether the downsampled audio signal (or representation thereof) is a complete audio signal or a reconstructed audio signal. At S26 the likelihoods of each discriminator branch are aggregated to form a final likelihood indicating whether the reconstructed audio signal is a complete audio signal or a reconstructed audio signal. At S27 the internal weights of either the discriminator or the neural network system is updated in accordance with generator-adversarial training as described in the above. As seen, the method may repeat after S27 and comprise updating the internal weights of the other one of the discriminator and the neural network system in the next pass and/or provide and use new training data in the next pass.

In the above, possible methods of training and operating a deep-learning-based system for determining an indication of an audio quality of an input audio sample, as well as possible implementations of such system have been described. Additionally, the present disclosure also relates to an apparatus for carrying out these methods. An example of such apparatus may comprise a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these) and a memory coupled to the processor. The processor may be adapted to carry out some or all of the steps of the methods described throughout the disclosure.

The apparatus may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that apparatus. Further, the present disclosure shall relate to any collection of apparatus that individually or jointly execute instructions to perform any one or more of the methodologies discussed herein.

The present disclosure further relates to a program (e.g., computer program) comprising instructions that, when executed by a processor, cause the processor to carry out some or all of the steps of the methods described herein.

Yet further, the present disclosure relates to a computer-readable (or machine-readable) storage medium storing the aforementioned program. Here, the term "computer-readable storage medium" includes, but is not limited to, data repositories in the form of solid-state memories, optical media, and magnetic media, for example.

Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the disclosure discussions utilizing terms such as "processing", "computing", "calculating", "determining", "analyzing" or the like, refer to the action and/or processes of a computer or computing system, or similar electronic computing devices, that manipulate and/or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

The methodologies described herein are, in one example embodiment, performable by one or more processors that accept computer-readable (also called machine-readable) code containing a set of instructions that when executed by one or more of the processors carry out at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken are included. Thus, one example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system further may include a memory subsystem including main RAM and/or a static RAM, and/or ROM. A bus subsystem may be included for communicating between the components. The processing system further may be a distributed processing system with processors coupled by a network. If the processing system requires a display, such a display may be included, e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes an input device such as one or more of an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, and so forth. The processing system may also encompass a storage system such as a disk drive unit. The processing system in some configurations may include a sound output device, and a network interface device. The memory subsystem thus includes a computer-readable carrier medium that carries computer-readable code (e.g., software) including a set of instructions to cause performing, when executed by one or more processors, one or more of the methods described herein. Note that when the method includes several elements, e.g., several steps, no ordering of such elements is implied, unless specifically stated. The software may reside in the hard disk, or may also reside, completely or at least partially, within the RAM and/or within the processor during execution thereof by the computer system. Thus, the memory and the processor also constitute computer-readable carrier medium carrying computer-readable code. Furthermore, a computer-readable carrier medium may form, or be included in a computer program product.

In alternative example embodiments, the one or more processors operate as a standalone device or may be connected, e.g., networked to other processor(s), in a networked deployment, the one or more processors may operate in the capacity of a server or a user machine in server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. The one or more processors may form a personal computer (PC), a tablet PC, a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.

Note that the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

Thus, one example embodiment of each of the methods described herein is in the form of a computer-readable carrier medium carrying a set of instructions, e.g., a computer program that is for execution on one or more processors, e.g., one or more processors that are part of web server arrangement. Thus, as will be appreciated by those skilled in the art, example embodiments of the present disclosure may be embodied as a method, an apparatus such as a special purpose apparatus, an apparatus such as a data processing system, or a computer-readable carrier medium, e.g., a computer program product. The computer-readable carrier medium carries computer readable code including a set of instructions that when executed on one or more processors cause the processor or processors to implement a method. Accordingly, aspects of the present disclosure may take the form of a method, an entirely hardware example embodiment, an entirely software example embodiment or an example embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.

The software may further be transmitted or received over a network via a network interface device. While the carrier medium is in an example embodiment a single medium, the term "carrier medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term "carrier medium" shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by one or more of the processors and that cause the one or more processors to perform any one or more of the methodologies of the present disclosure. A carrier medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical, magnetic disks, and magneto-optical disks. Volatile media includes dynamic memory, such as main memory. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise a bus subsystem. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications. For example, the term "carrier medium" shall accordingly be taken to include, but not be limited to, solid-state memories, a computer product embodied in optical and magnetic media; a medium bearing a propagated signal detectable by at least one processor or one or more processors and representing a set of instructions that, when executed, implement a method; and a transmission medium in a network bearing a propagated signal detectable by at least one processor of the one or more processors and representing the set of instructions.

It will be understood that the steps of methods discussed are performed in one example embodiment by an appropriate processor (or processors) of a processing (e.g., computer) system executing instructions (computer-readable code) stored in storage. It will also be understood that the disclosure is not limited to any particular implementation or programming technique and that the disclosure may be implemented using any appropriate techniques for implementing the functionality described herein. The disclosure is not limited to any particular programming language or operating system.

Reference throughout this disclosure to "one example embodiment", "some example embodiments" or "an example embodiment" means that a particular feature, structure or characteristic described in connection with the example embodiment is included in at least one example embodiment of the present disclosure. Thus, appearances of the phrases "in one example embodiment", "in some example embodiments" or "in an example embodiment" in various places throughout this disclosure are not necessarily all referring to the same example embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more example embodiments.

It should be appreciated that in the above description of example embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single example embodiment, Fig., or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed example embodiment. Thus, the claims following the Description are hereby expressly incorporated into this Description, with each claim standing on its own as a separate example embodiment of this disclosure.

Furthermore, while some example embodiments described herein include some but not other features included in other example embodiments, combinations of features of different example embodiments are meant to be within the scope of the disclosure, and form different example embodiments, as would be understood by those skilled in the art.

However, it is understood that example embodiments of the disclosure may be practiced without these specific details.

Thus, while there has been described what are believed to be the best modes of the disclosure, those skilled in the art will recognize that other and further modifications may be made thereto.

For example, any formulas given above are merely representative of procedures that may be used.

Claim 1:
A method for packet loss concealment of an incomplete audio signal, the incomplete audio signal comprising a substitute signal portion replacing an original signal portion of a complete audio signal, the method comprising:
obtaining a representation of the incomplete audio signal;
inputting the representation of the incomplete audio signal to an encoder neural network trained to predict a latent representation of a complete audio signal given a representation of an incomplete audio signal;
outputting, by the encoder neural network, a latent representation of a predicted complete audio signal;
quantizing the latent representation of the complete audio signal to obtain a quantized latent representation, wherein the quantized latent representation is formed by selecting a set of tokens out of a predetermined vocabulary set of tokens;
conditioning, with at least one token of the quantized latent representation, a generative neural network, wherein the generative neural network is trained to predict a token of the set of tokens provided at least one different token of the set of tokens;
outputting by the generative neural network a predicted token of the latent representation and a confidence metric associated with the predicted token;
based on the confidence metric of the predicted token, replacing a corresponding token of the quantized latent representation with the predicted token,
inputting the quantized latent representation of the predicted complete audio signal to a decoder neural network trained to predict a representation of a complete audio signal given a latent representation of a complete audio signal; and
outputting, by the decoder neural network, a representation of the predicted complete audio signal comprising a reconstruction of the original portion of the complete audio signal, wherein said encoder neural network and said decoder neural network have been trained with an adversarial neural network.