Patent ID: 12223970

DETAILED DESCRIPTION

Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings. However, various alterations and modifications may be made to the example embodiments. Here, the example embodiments are not construed as limited to the disclosure. The example embodiments should be understood to include all changes, equivalents, and replacements within the idea and the technical scope of the disclosure.

The terminology used herein is for the purpose of describing particular example embodiments only and is not to be limiting of the example embodiments. The singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises/comprising” and/or “includes/including” when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groups thereof.

Unless otherwise defined, all terms including technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which example embodiments belong. It will be further understood that terms, such as those defined in commonly-used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

When describing the example embodiments with reference to the accompanying drawings, like reference numerals refer to like components and a repeated description related thereto will be omitted. In the description of example embodiments, detailed description of well-known related structures or functions will be omitted when it is deemed that such description will cause ambiguous interpretation of the present disclosure.

FIG.1is a block diagram illustrating an encoder100and a decoder200according to an example embodiment.

Referring toFIG.1, the encoder100may include an LP analysis module160, a quantization module170, and a first neural network module180. The decoder200may include an inverse quantization module260, a second neural network module270, a residual signal synthesis module280, or a linear prediction synthesis filter290.

Referring toFIG.1, the encoder100may output a first bitstream and a second bitstream obtained by encoding a residual signal of an audio signal or speech signal, which is an input signal. The encoder100may also output LP coefficients bitstream obtained by quantizing LP coefficients, and a weight bitstream obtained by quantizing a weight vector.

The decoder200may output an output signal obtained by reconstructing an input signal, using the first bitstream, the second bitstream, the LP coefficients bitstream, and the weight bitstream that are received from the encoder100.

For example, a processor of the encoder100may output LP coefficients bitstream and a residual signal by performing an LP analysis on the input signal using the LP analysis module160.

In an example, the LP analysis module160may include LP coefficients calculator105, LP coefficients quantizer110, LP coefficients de-quantizer115, or an LP analysis filter120.

For example, the processor of the encoder100may calculate LP coefficients for each frame corresponding to an analysis unit of the input signal, using the LP coefficients calculator105.

For example, the processor of the encoder100may input the LP coefficients to the LP coefficients quantizer110and may allow the LP coefficients quantizer110to output LP coefficients bitstream.

For example, the processor of the encoder100may calculate quantized LP coefficients by de-quantizing the LP coefficients bitstream using the LP coefficients de-quantizer115.

For example, the processor of the encoder100may calculate a residual signal from the input signal using the LP analysis filter120with the quantized LP coefficients.

In an example, the processor of the encoder100may output a first latent signal, a second latent signal, and a weight vector for each of the first latent signal and the second latent signal from the residual signal, using the first neural network module180.

For example, the first neural network module180may include a first neural network125, a second neural network130, or a third neural network135. For example, the processor of the encoder100may input the residual signal to the first neural network125or the second neural network130, and may allow the first neural network125or the second neural network130to output the first latent signal or the second latent signal. The first latent signal or the second latent signal may refer to an encoded code vector or bottleneck.

For example, the processor of the encoder100may input the residual signal to the third neural network135, and may allow the third neural network135to output the weight vector.

In an example, the processor of the encoder100may output a first bitstream obtained by quantizing the first latent signal, a second bitstream obtained by quantizing the second latent signal, and a weight bitstream obtained by quantizing the weight vector, using the quantization module170.

In an example, the quantization module170may include a first quantization layer140, a second quantization layer145, or a third quantization layer150.

For example, the processor of the encoder100may quantize the first latent signal output from the first neural network125and output the first bitstream, using the first quantization layer140.

For example, the processor of the encoder100may quantize the second latent signal output from the second neural network130and output the second bitstream, using the second quantization layer145.

For example, the processor of the encoder100may quantize the weight vector output from the third neural network135and output the weight bitstream, using the third quantization layer150.

In an example, a processor of the decoder200may de-quantize the LP coefficients bitstream, the first bitstream, the second bitstream, and the weight bitstream and output quantized LP coefficients, a first quantized latent signal, a second quantized latent signal, and a quantized weight vector, using the de-quantization module260.

In an example, the de-quantization module260may include LP coefficients de-quantizer215, a first de-quantization layer240, a second de-quantization layer245, or a third de-quantization layer250.

For example, the processor of the decoder200may output quantized LP coefficients by de-quantizing an LP coefficients bitstream using the LP coefficients de-quantizer215.

For example, the processor of the decoder200may output a first quantized latent signal by de-quantizing an first bitstream using the first de-quantization layer240.

For example, the processor of the decoder200may output a second quantized latent signal by de-quantizing an second bitstream using the second de-quantization layer245.

For example, the processor of the decoder200may output a quantized weight vector by de-quantizing a weight bitstream using the third de-quantization layer250.

In an example, the processor of the decoder200may output a first decoded residual signal obtained by decoding the first quantized latent signal and a second decoded residual signal obtained by decoding the second quantized latent signal, using the second neural network module270.

For example, the second neural network module270may include a fourth neural network225or a fifth neural network230. For example, the processor of the decoder200may input the first quantized latent signal to the fourth neural network225, and may allow the fourth neural network225to output the first decoded residual signal obtained by decoding the first quantized latent signal. For example, the processor of the decoder200may input the second quantized latent signal to the fifth neural network230and may allow the fifth neural network230to output the second decoded residual signal obtained by decoding the second quantized latent signal.

The first neural network125and the fourth neural network225may refer to an encoder and a decoder of an autoencoder having a recurrent structure suitable for modeling a periodic component of a speech signal or an audio signal. For example, the first neural network125may allow an input layer to output a code vector, i.e., a first latent signal, using an input signal. The code vector may generally refer to a dimensionality-reduced representation of a input signal under the constraint that an input signal and an output signal of the autoencoder may be the same. The fourth neural network225may output a reconstructed signal, using the code vector output from the first neural network125. A signal output from the fourth neural network225may refer to a reconstructed signal of the input signal to the first neural network125. The principles of the autoencoder of the first neural network125and the fourth neural network225may apply equally to an autoencoder of the second neural network130and the fifth neural network230. However, an autoencoder with a pair of the second neural network130and the fifth neural network230may have a feedforward structure suitable for modeling non-periodic components of speech or audio signals.

In an example, the processor of the decoder200may synthesize the residual signal based on the first decoded residual signal, the second decoded residual signal, and the quantized weight vector, using the residual signal synthesis module280. The residual signal synthesized by the residual signal synthesis module280may refer to a signal obtained by reconstructing the residual signal output from the LP analysis filter120of the encoder100.

For example, the processor of the decoder200may synthesize an output signal based on the reconstructed residual signal and the quantized LP coefficients, using the LP synthesis filter290. The reconstructed residual signal synthesized by the residual signal synthesis module280and the quantized LP coefficients from the de-quantization module260may be fed into the LP synthesis filter290. The output signal synthesized by the LP synthesis filter290may refer to a signal obtained by reconstructing the input signal of the encoder100.

Example embodiments provide an encoding method and a decoding method for enhancing an encoding quality in an encoding process of sequential signals such as audio signals or speech signals and for preventing overfitting of a neural network model that encodes or decodes a residual signal. According to an example embodiment, the encoder100may perform modeling of the residual signal through a dual-path neural network. The first neural network125may include a recurrent neural network (RNN) configured to perform modeling of a periodic component using the input residual signal. The second neural network130may include an FNN configured to perform modeling of a non-periodic component using the input residual signal. The third neural network135may output a weight vector dependent on signal characteristics to reconstruct a residual signal as a weighted sum of the first decoded residual signal and the second decoded residual signal output from the fourth neural network225and the fifth neural network230, respectively.

The block diagram of the encoder100and the decoder200is shown inFIG.1for convenience of description, and components of the encoder100and the decoder200shown inFIG.1may refer to software or programs executable by the processor.

FIG.2is a diagram illustrating operations of the encoder100and the decoder200according to an example embodiment.

The processor of the encoder100may calculate LP coefficients {ai} based on an input signal x(n), using the LP coefficients calculator105. A linear prediction may refer to predicting a current sample as a linear combination of past samples, and the LP coefficients calculator105may calculate LP coefficients based on samples in an LP analysis frame. The processor of the encoder100may calculate p-th order LP coefficients {ai}i=1, . . . , pto minimize the prediction error E as shown in Equation 2 below using the LP coefficients calculator105. Typically, the LP coefficients are calculated using autocorrelation method and Durbin's recursive algorithm to solve the minimization problem efficiently.
{tilde over (x)}(n)=Σi=1paix(n−i),n=0, . . . , (NLP−1)  [Equation 1]

In Equation 1, {tilde over (x)}(n) denotes the predicted signal, and NLPdenotes a number of samples in an LP analysis frame.
E=Σn=0NLP−1{e(n)}2=Σn=0NLP−1{x(n)−{tilde over (x)}(n)}2[Equation 2]

In Equation 2, x(n) denotes an input signal, and {tilde over (x)}(n) denotes the predicted input signal of Equation 1.

The processor of the encoder100may quantize the LP coefficients and output LP coefficients bitstream Ia, using the LP coefficients quantizer110. If the LP coefficients is directly quantized, the LP synthesis filter290of the decoder200for synthesizing an output signal may become unstable due to a quantization error. To prevent the LP synthesis filter290from being unstable, the processor of the encoder100may convert the LP coefficients into, for example, a line spectral frequency (LSF) or an immittance spectral frequency (ISF), etc., to quantize the LP coefficients, using the LP coefficients quantizer110.

The processor of the encoder100may de-quantize LP coefficients bitstream and may output quantized LP coefficients {âi}, using the LP coefficients de-quantizer115.

The processor of the encoder100may calculate a residual signal r(n) based on the quantized LP coefficients {âi} and the input signal x(n), using the LP analysis filter120. The residual signal r(n) may be calculated using the LP analysis filter120as shown in Equation 3 below.
r(n)=x(n)+Σi=1pâix(n−i),n=0, . . . , (N−1)  [Equation 3]

In Equation 3, N denotes a number of samples in an analysis frame.

The encoder100may reduce a dynamic range of an input signal and may obtain a spectrally-flattened residual signal through an LP analysis.

The LP analysis may be applied to an audio signal as well, and may refer to a process of extracting a residual signal and LP coefficients from an audio signal. A scheme of extracting LP coefficients is not limited to a specific example, and it is apparent to one of ordinary skill in the art that various schemes of extracting LP coefficients may be applied without departing from the spirit of the present disclosure.

The processor of the encoder100may input the residual signal r(n) to the first neural network125and may allow the first neural network125to output a first latent signal zp(n) of the residual signal. The first latent signal may refer to a code vector which is a dimensionality-reduced representation under the constraint that an input signal of the first neural network125and an output signal of the fourth neural network225are the same. For example, the first neural network125may output the first latent signal that is a code vector obtained by encoding the residual signal.

The processor of the encoder100may input the residual signal r(n) to the second neural network130and may allow the second neural network130to output a second latent signal zn(n) of the residual signal. The second latent signal may refer to a code vector which is a dimensionality-representation under the constraint that an input signal of the second neural network130and an output signal of the fifth neural network230are the same. For example, the second neural network130may output the second latent signal that is a code vector obtained by encoding the residual signal.

The first neural network125may be a neural network model configured to perform modeling of a periodic component of the residual signal, and the second neural network130may be a neural network model configured to perform modeling of a non-periodic component of the residual signal.

A training model may be a neural network model that includes one or more layers and one or more model parameters based on deep learning. However, there is no limitation to the type of neural network models used herein, a size of input/output data.

The processor of the encoder100may input the residual signal r(n) to the third neural network135, and may allow the third neural network135to output a weight vector w(n) calculated from the residual signal. The weight vector may refer to weighting values used for calculating a reconstructed residual signal {circumflex over (r)}(n) as a weighted sum of two decoded outputs, for example, a first decoded residual signal {circumflex over (r)}p(n) and a second decoded residual signal {circumflex over (r)}n(n), output from the fourth neural network225and the fifth neural network230of the decoder200, respectively.
w=gate(r;gate)  [Equation 4]

For example, the third neural network135may output a weight vector w as shown in Equation 4. In Equation 4,gatedenotes a model parameter of the third neural network135, and r denotes a residual signal in vector form inputted to the third neural network135.

The processor of the encoder100may output a first bitstream Ip, a second bitstream In, and a weight bitstream Iwobtained by quantizing a first latent signal zp(n), a second latent signal zn(n), and a weight vector w(n), using the first quantization layer140, the second quantization layer145, and the third quantization layer150, respectively.

The encoder100may multiplex the first bitstream Ip, the second bitstream In, the weight bitstream Iwand LP coefficients bitstream Ia, and transmit the multiplexed bitstreams to the decoder200.

To transmit the first latent signal zp(n), the second latent signal zn(n), and the weight vector w(n) to the decoder200to the decoder200, the encoder100may perform a quantization process in the first quantization layer140, the second quantization layer145, and the third quantization layer150. Since quantization process may be generally non-differentiable or may have discontinuous derivative values, the general quantization process may not be suitable for training a neural network model by updating model parameters based on a loss function. In the training phase of neural network models (e.g., the first neural network125through the fifth neural network230), a quantization process may be replaced with a continuous approximated quantization which can be differentiable. In the test phase of trained neural network models (e.g., the first neural network125through the fifth neural network230), the encoder100and the decoder200may perform a typical quantization and dequantization process. For example, a softmax quantization scheme, a uniform noise addition scheme, and the like may be used to approximate a quantization process to be differentiable, however, the example embodiments are not limited thereto.

The decoder200may receive the multiplexed bitstreams from the encoder100, may demultiplex each bitstream, and may output the first bitstream Ip, the second bitstream In, the weight bitstream Iw, and the LP coefficients bitstream Ia.

The processor of the decoder200may output a first quantized latent signal {circumflex over (z)}p(n), a second quantized latent signal {circumflex over (z)}n(n), a quantized weight vector ŵ(n), and quantized LP coefficients {âi} obtained by de-quantizing the first bitstream Ip, the second bitstream In, the weight bitstream Iw, and the LP coefficients bitstream Ia, using the first de-quantization layer240, the second de-quantization layer245, the third de-quantization layer250, and the LP coefficients de-quantizer215, respectively. The weight vector ŵ(n) may be split into a first quantized weight vector ŵp(n) for the first quantized latent signal {circumflex over (z)}p(n) and a second quantized weight vector ŵn(n) for the second quantized latent signal {circumflex over (z)}n(n).

The processor of the decoder200may input the first quantized latent signal {circumflex over (z)}p(n) to the fourth neural network225and may allow the fourth neural network225to output the first decoded residual signal {circumflex over (r)}p(n) by decoding the first quantized latent signal {circumflex over (z)}p(n).

The processor of the decoder200may input the second quantized latent signal {circumflex over (z)}n(n) to the fifth neural network230and may allow the fifth neural network230to output the second decoded residual signal {circumflex over (r)}n(n) by decoding the second quantized latent signal {circumflex over (z)}n(n).

A encoding and decoding pair of the first neural network125and the fourth neural network225may have an recurrent autoencoder structure that may effectively encode and decode a periodic component of a residual signal, and a encoding and decoding pair of the second neural network130and the fifth neural network230may have an feedforward autoencoder structure that may effectively encode and decode a non-periodic component of the residual signal.

For example, the fourth neural network225and the fifth neural network230may have symmetrical structures with the first neural network125and the second neural network130, respectively, and may share model parameters between symmetrical layers. For example, the first neural network125may output a code vector by encoding an input signal using a trained model parameter, and the fourth neural network225may output a signal by decoding the code vector using a symmetrical structure with the first neural network125and a model parameter shared between symmetrical layers.

The processor of the decoder200may reconstruct the residual signal {circumflex over (r)}(n) based on the quantized weight vectors ŵp(n) and ŵn(n), the first decoded residual signal {circumflex over (r)}p(n) and the second decoded residual signal {circumflex over (r)}n(n), using the residual signal synthesis module280. For example, the processor of the decoder200may reconstruct the residual signal {circumflex over (r)}(n) by a weighted sum of the first decoded residual signal {circumflex over (r)}p(n) and the second decoded residual signal {circumflex over (r)}n(n), based on the quantized weight vectors ŵp(n) and ŵn(n), using the residual signal synthesis module280.
{circumflex over (r)}(n)=ŵp(n){circumflex over (r)}p(n)+ŵn(n){circumflex over (r)}n(n),n=0, . . . , (N−1)  [Equation 5]

For example, each of the quantized weight vectors ŵp(n) or ŵn(n) may have the same dimension with the corresponding decoded residual signal {circumflex over (r)}p(n) or {circumflex over (r)}n(n), may have a different dimension with the corresponding decoded residual signal, or may have a single dimension to apply the common weight for each sample of the corresponding decoded residual signal as a simplest example. In case that the quantized weight vector may have a different dimension with the corresponding decoded residual signal, each element of the quantized weight vector may apply to multiple samples of the decoded residual signal in a block-wise fashion.

In Equation 5, ŵp(n) or ŵn(n) denote quantized weight vectors output by de-quantizing the weight bitstream Iwin the third de-quantization layer250. The processor of the encoder100may output the weight bitstream Iwby quantizing the weight vectors wn(n) using the third quantization layer150.

Although the weight vector, w(n) output by the third neural network135may include two weight vectors, wp(n) and wn(n), as shown inFIG.2, the residual signal {circumflex over (r)}(n) may be reconstructed using a single quantized weight vector ŵ(n) even when w(n) is a single weight vector output from the third neural network135, as shown in Equation 6 below. In Equation 6, ŵ(n) may be assumed as a weight vector for a first decoded residual signal.
{circumflex over (r)}(n)=ŵ(n){circumflex over (r)}p(n)+(1−ŵ(n)){circumflex over (r)}n(n)  [Equation 6]

In Equation 6, ŵ(n) denotes a weight vector output by de-quantizing the weight bitstream Iwin the third de-quantization layer250. The processor of the encoder100may output the weight bitstream Iwby quantizing the weight vector w(n) using the third quantization layer150.

In an example, the processor of the decoder200may synthesize an output signal {circumflex over (x)}(n) based on the reconstructed residual signal {circumflex over (r)}(n) and the quantized LP coefficients {âi}, using the LP synthesis filter290as shown in Equation 7 below.

xˆ(n)=rˆ(n)-∑i=1pa^i⁢xˆ(n-i),n=0,…,(N-1)[Equation⁢7]

An LP synthesis may be a process of generating an signal from a residual signal using LP coefficients. A scheme of LP synthesis is not limited to a specific example, and it is apparent to one of ordinary skill in the art that various schemes of LP synthesis may be applied without departing from the spirit of the present disclosure.

In an example, a training device (not shown) for training a neural network model may train the first neural network125through the fifth neural network230. For example, the first neural network125through the fifth neural networks230shown inFIGS.1and2may refer to neural networks trained by the training device.

For example, the training device may include at least one of an LP analysis module (e.g., the LP analysis module160ofFIG.1), a quantization module (e.g., the quantization module170ofFIG.1), a first neural network module (e.g., the first neural network module180ofFIG.1), an de-quantization module (e.g., the de-quantization module260ofFIG.1), a second neural network module (e.g., the second neural network module270ofFIG.1), a residual signal synthesis module (e.g., the residual signal synthesis module280ofFIG.1), or a linear prediction synthesis filter (e.g., the LP synthesis filter290ofFIG.1).

In an example, the description of the encoder100and/or the decoder200ofFIG.2may be substantially equally applied to the LP analysis module, the quantization module, the first neural network module, the de-quantization module, the second neural network module, the residual signal synthesis module or the LP synthesis filter of the training device. The process in the quantization module and de-quantization module of the training device may be replaced with its approximated process to be differentiable.

In an example, the training device may calculate a loss function based on at least one of reconstruction loss, D between the residual signal r(n) output from the linear prediction analysis filter120, the reconstructed residual signal {circumflex over (r)}(n) output from the residual signal synthesis module280, or a bit rate loss, R indicating a quantization entropy obtained by the quantization module170, in a neural network training operation. The training device may train the first neural network125through the fifth neural network230so that a value of the loss function may be minimized in the neural network training operation.

In an example, the training device may calculate the reconstruction loss D in terms of an error of the reconstructed residual signal {circumflex over (r)}(n) with respect to the original residual signal r(n), as shown in Equation 8 below. In Equation 8, Dmsedenotes a mean squared error (MSE), and Dmaedenotes a mean absolute error (MAE). The signal distortion D may be calculated as an MSE and an MAE, but is not limited thereto.

Dm⁢s⁢e=1N⁢∑n=0N-1⁢{r⁡(n)-rˆ(n)}2⁢Dm⁢a⁢e=1N⁢∑n=0N-1⁢❘"\[LeftBracketingBar]"r⁡(n)-rˆ(n)❘"\[RightBracketingBar]"[Equation⁢8]

The training device may calculate an overall loss functionas shown in Equation 9 below. In Equation 9, R denotes a bit rate loss as sum of each entropy computed using probability distribution of the first quantized latent signal, the second quantized latent signal, and the quantized weight vector, and λrateand λmsedenote hyperparameters as weights for the bit rate loss of R and the reconstruction loss of Dmseor Dmae.
=λrateR+λmseDmse[Equation 9]

The training device may train the first neural network125, the second neural network130, the third neural network135, the fourth neural network225, and the fifth neural network230to minimize an overall loss function calculated using Equation 9. The training device may include a quantization layer and an de-quantization layer, which are approximated to be differentiable according to a design of a neural network, in a training process. For example, the training device may train the first neural network125through the fifth neural network230by backpropagating an error calculated as the overall loss function, however, the example embodiments are not limited thereto. For example, when the fourth neural network225and the fifth neural network230may be designed to have symmetric structure with the first neural network125and the second neural network130, respectively, the training device may perform training by constraining model parameters to be shared between symmetrical layers.

In an example, the encoder100or the decoder200shown inFIGS.1and2may encode or decode an input signal using the first neural network125through the fifth neural network230trained by the training device.

Referring toFIG.2, the encoder100according to various example embodiments may normalize, in advance, intrinsic features of an input signal, such as speech and music, through a spectrally flattening effect resulted from the LP analysis and may output a residual signal. A neural network model, for example, the first neural network125through the fifth neural network230, for encoding and decoding the residual signal may be less sensitive to a change in characteristics of an input signal, and a reconstruction quality of the input signal may be enhanced. For example, the encoder100and the decoder200according to an example embodiment may resolve a quality degradation problem caused usually by an mismatch between training dataset and testing dataset.

InFIG.2, a configuration including the first neural network125, the first quantization layer140, the first de-quantization layer240, and the fourth neural network225may be referred to as an adaptive codebook neural network for modeling a periodic component of a residual signal. In addition, a configuration including the second neural network130, the second quantization layer145, the second de-quantization layer245, and the fifth neural network230may be referred to as a fixed codebook neural network for modeling a non-periodic component of a residual signal.

In an example, the adaptive codebook neural network may perform modeling of a periodic component of a residual signal having a periodic characteristic. The fixed codebook neural network may perform modeling of a non-periodic component of a residual signal having a noisy characteristic.

As shown inFIG.2, the adaptive codebook neural network (e.g., the configuration including the first neural network125, the first quantization layer140, the first de-quantization layer240, and the fourth neural network225) and the fixed codebook neural network (e.g., the configuration including the second neural network130, the second quantization layer145, the second de-quantization layer245, and the fifth neural network230) may have neural network structures with different attributes in an LP analysis framework. For example, the first neural network125and the fourth neural network225of the adaptive codebook neural network may each include an RNN, and the second neural network130and the fifth neural network230of the fixed codebook neural network may each include an FNN. Each of the first neural network125, the second neural network130, the fourth neural network225, and the fifth neural network230may include a neural network suitable for modeling a desired component of an input signal, to enhance a reconstruction quality of the input signal.

For example, the encoder100and the decoder200may perform modeling of a residual signal that is output from the LP analysis filter120through a dual-path neural network. A dual path may refer to a path for processing a residual signal through the first neural network125and the fourth neural network225, and a path for processing the residual signal through the second neural network130and the fifth neural network230. The encoder100and the decoder200may reconstruct a residual signal by weighted summing two residual signals (e.g., the first decoded residual signal and the second decoded residual signal) output respectively from the adaptive codebook neural network and the fixed codebook neural network using the quantized weight vector output from the third de-quantization layer250depending on signal characteristics.

FIG.3is a diagram illustrating an example of an operation of an encoding method according to an example embodiment.

Referring toFIG.3, in operation305, an encoder100according to various example embodiments may output LP coefficients bitstream and a residual signal by performing an LP analysis on an input signal.

In operation310, the encoder100may output a first latent signal, a second latent signal, and a weight vector, using a first neural network module180. For example, a processor of the encoder100may input the residual signal to the first neural network module180. For example, the first latent signal may refer to a code vector obtained by modeling a periodic component of the residual signal, or a code vector obtained by encoding the periodic component of the residual signal. For example, the second latent signal may refer to a code vector obtained by modeling a non-periodic component of the residual signal, or a code vector obtained by encoding the non-periodic component of the residual signal. For example, the weight vector may refer to a set of weights for reconstructing the residual signal in the decoder200.

In operation315, the encoder100may output a first bitstream, a second bitstream, and a weight bitstream, using a quantization module170. For example, the quantization module170may include a first quantization layer140, a second quantization layer145, or a third quantization layer150.

For example, the encoder100may quantize the first latent signal and output the first bitstream, using the first quantization layer140. For example, the encoder100may quantize the second latent signal and output the second bitstream, using the second quantization layer145. For example, the encoder100may quantize the weight vector and output the weight bitstream, using the third quantization layer150. For example, the encoder100may transmit the LP coefficients bitstream output in operation305, and the first bitstream, the second bitstream, and the weight bitstream that are output in operation315to a decoder200. For example, the encoder100may multiplex the LP coefficients bitstream, the first bitstream, the second bitstream, and the weight bitstream, and may transmit the multiplexed bitstream to the decoder200.

FIG.4is a diagram illustrating another example of an operation of an encoding method according to an example embodiment.

Referring toFIG.4, in operation405, an encoder100according to various example embodiments may calculate LP coefficients using an input signal. For example, a processor of the encoder100may calculate LP coefficients for each frame corresponding to an analysis unit of the input signal, using LP coefficients calculator105.

In operation410, the encoder100may output LP coefficients bitstream by quantizing the LP coefficients. For example, the processor of the encoder100may input the LP coefficients to LP coefficients quantizer110and may allow the LP coefficients quantizer110to output the LP coefficients bitstream.

In operation415, the encoder100may calculate quantized LP coefficients by de-quantizing the LP coefficients bitstream. For example, the processor of the encoder100may calculate the quantized LP coefficients by de-quantizing the LP coefficients bitstream using LP coefficients de-quantizer115.

In operation420, the encoder100may calculate a residual signal using the input signal and the quantized LP coefficients.

In operation425, the encoder100may output a first latent signal by inputting the residual signal to a first neural network125. For example, the first neural network125may include an RNN configured to encode a periodic component of the residual signal.

In operation430, the encoder100may output a second latent signal by inputting the residual signal to a second neural network130. For example, the second neural network130may include an FNN configured to encode a non-periodic component of the residual signal.

The first neural network125used in operation425may refer to an encoder part of an autoencoder having a recurrent structure suitable for modeling a periodic component of a speech signal or an audio signal. The second neural network130used in operation430may refer to a decoder part of an autoencoder having a feedforward structure suitable for modeling a non-periodic component of a speech signal or an audio signal.

For example, the first latent signal or the second latent signal may be an encoded code vector or bottleneck.

In operation435, the encoder100may output a weight vector by inputting the residual signal to a third neural network135. For example, the third neural network135may include a neural network configured to output a weight vector depending on characteristics of the residual signal. In an example, the weight vector may be associated with weights of the first latent signal and the second latent signal to reconstruct a residual signal.

In operation440, the encoder100may output a first bitstream by quantizing the first latent signal. In operation445, the encoder100may output a second bitstream by quantizing the second latent signal. In operation450, the encoder100may output a weight bitstream by quantizing the weight vector.

For example, the encoder100may quantize the first latent signal, the second latent signal, and the weight vector to the first bitstream, the second bitstream, and the weight bitstream, using the first quantization layer140, the second quantization layer145, and the third quantization layer150of the quantization module170, respectively.

In operation455, the encoder100may multiplex the LP coefficients bitstream, the first bitstream, the second bitstream, and the weight bitstream and transmit the multiplexed bitstream to a decoder200.

FIG.5is a diagram illustrating an example of an operation of a decoding method according to an example embodiment.

Referring toFIG.5, in operation505, a decoder200according to various example embodiments may output quantized LP coefficients, a first quantized latent signal, a second quantized latent signal, and a quantized weight vector by de-quantizing LP coefficients bitstream, a first bitstream, a second bitstream, and a weight bitstream.

In operation510, the decoder200may output a first decoded residual signal and a second decoded residual signal using a second neural network module270. For example, the second neural network module270may include a fourth neural network225, and a fifth neural network230. For example, the decoder200may input the first quantized latent signal to the fourth neural network225to output the first decoded residual signal. For example, the decoder200may input the second quantized latent signal to the fifth neural network230to output the second decoded residual signal.

In operation515, the decoder200may reconstruct a residual signal using the first decoded residual signal, the second decoded residual signal, and the quantized weight vector. For example, the decoder200may reconstruct the residual signal as a weighted sum of the first decoded residual signal and the second decoded residual signal, using the quantized weight vector.

In operation520, the decoder200may synthesize an output signal using the reconstructed residual signal and the quantized LP coefficients. For example, the decoder200may generate an audio signal from the reconstructed residual signal using an LP synthesis filter290constructed with the quantized LP coefficients. The audio signal generated by the decoder200may be an output signal.

FIG.6is a diagram illustrating another example of an operation of a decoding method according to an example embodiment.

Referring toFIG.6, in operation605, a decoder200according to various example embodiments may output LP coefficients bitstream, a first bitstream, a second bitstream, and a weight bitstream by demultiplexing multiplexed bitstreams.

In operation610, the decoder200may output quantized LP coefficients by de-quantizing the LP coefficients bitstream. For example, the decoder200may output the quantized LP coefficients obtained by de-quantizing the LP coefficients bitstream using LP coefficients de-quantizer215.

In operation615, the decoder200may output a first quantized latent signal by de-quantizing the first bitstream. For example, the decoder200may output the first quantized latent signal obtained by de-quantizing the first bitstream using a first de-quantization layer240.

In operation620, the decoder200may output a first decoded residual signal by inputting the first quantized latent signal to a fourth neural network225.

In operation625, the decoder200may output a second quantized latent signal by de-quantizing the second bitstream. For example, the decoder200may output the second quantized latent signal obtained by de-quantizing the second bitstream using a second de-quantization layer245.

In operation630, the decoder200may output a second decoded residual signal by inputting the second quantized latent signal to a fifth neural network230.

In operation635, the decoder200may output a quantized weight vector by de-quantizing the weight bitstream. For example, the decoder200may output the quantized weight vector obtained by de-quantizing the weight bitstream using a third de-quantization layer250.

In operation640, the decoder200may reconstruct a residual signal as a weighted sum of the first decoded residual signal and the second decoded residual signal, using the quantized weight vector.

In operation645, the decoder200may synthesize an output signal using the reconstructed residual signal and the quantized LP coefficients.

For example, the decoder200may synthesize the reconstructed residual signal based on the first decoded residual signal, the second decoded residual signal, and the quantized weight vector, using a residual signal synthesis module280. For example, the decoder200may synthesize the output signal based on the reconstructed residual signal and the LP coefficients, using a linear prediction synthesis filter290.

FIG.7is a diagram illustrating first neural networks125-1and125-2and fourth neural networks225-1and225-2, each including an RNN, according to an example embodiment.

Referring toFIG.7, the first neural network125-1,125-2according to various example embodiments may include an input layer126-1,126-2, an RNN127-1,127-2, or a code layer128-1,128-2. The fourth neural network225-1,225-2according to various example embodiments may include a code layer228-1,228-2, an RNN227-1,227-2, and an output layer226-1,226-2.

FIG.7illustrates the first neural networks125-1and125-2and the fourth neural networks225-1and225-2at the current time steps t and the next time step (t+1). The first neural networks125-1and125-2and the fourth neural networks225-1and225-2may include the RNNs127-1,127-2,227-1, and227-2, respectively. Each hidden state of the RNN127-1,227-1at the current time step t may be input to the RNN127-2,227-2at the next time step (t+1), respectively.

For example, Each hidden state at the previous time step (t−1), although not shown inFIG.7, may be input to the RNN127-1of the first neural network125-1and the RNN227-1of the fourth neural network225-1, respectively. At the current time step t, a residual signal obtained from the LP analysis filter120may be the input layer126-1of the first neural network125-1to output a code vector. The code layer128-1may be a code vector, for example, a first latent signal, which is a signal output from the RNN127-1of the first neural network125-1. A first quantization layer140may transmit a first bitstream obtained by quantizing the first latent signal to a first de-quantization layer240. The first de-quantization layer240may de-quantize the first bitstream and output the first quantized latent signal corresponding to the code layer228-1of the fourth neural network225-1. The RNN227-1of the fourth neural network225-1may output a first decoded residual signal corresponding to the output layer226-1.

In substantially the same manner as operations of the first neural network125-1and the fourth neural network225-1at the current time step t, hidden states of the RNNs127-1and227-1at the time step t may be input to the RNN127-2of the first neural network125-2and the RNN227-2of the fourth neural network225-2at the next time step (t+1). At the next time step (t+1), the residual signal may be the input layer126-2of the first neural network125-2. The code layer128-2may be a code vector, for example, a first latent signal, according to a signal output from the RNN127-2of the first neural network125-2. A first quantization layer140may transmit a first bitstream obtained by quantizing the first latent signal to a first de-quantization layer240. The first de-quantization layer240may de-quantize the first bitstream and output the first quantized latent signal corresponding to the code layer228-2of the fourth neural network225-2. The RNN227-2of the fourth neural network225-2may output a first decoded residual signal corresponding to the output layer226-2.

As described above, the first neural network125and the fourth neural network225may include the RNNs127and227, respectively, and the RNNs127and227may pass each hidden state information at a current time step to RNNs127and227at a next time step. Since the first neural network125and the fourth neural network225include the RNNs127and227, respectively, an encoder and a decoder according to an example embodiment may efficiently model a periodic component of a residual signal, for example, a long-term redundancy.

For example, the first neural network125, the first quantization layer140, the first de-quantization layer240, and the fourth neural network225may be trained using an end-to-end scheme.

FIG.8is a diagram illustrating the second neural network130and the fifth neural network230that include FNNs132and232, respectively, according to an example embodiment.

Referring toFIG.8, the second neural network130according to various example embodiments may include an input layer131, the FNN132, and a code layer133. The fifth neural network230according to various example embodiments may include a code layer233, the FNN232, and an output layer231.

For example, a residual signal may be the input layer131of the second neural network130at a current time step t. The code layer133may be a code vector, for example, a second latent signal, according to a signal output from the FNN132of the second neural network130. A second quantization layer145may transmit a second bitstream obtained by quantizing the second latent signal to a second de-quantization layer245. The second de-quantization layer245may de-quantize the second bitstream and output the second quantized latent signal corresponding to the code layer233of the fifth neural network230. The FNN232of the fifth neural network230may output a second decoded residual signal corresponding to the output layer231.

Since the second neural network130and the fifth neural network230include the FNNs132and232, respectively, an encoder and a decoder according to an example embodiment may efficiently model a non-periodic component of a residual signal, for example, a short-term redundancy.

In an example, the second neural network130, the second quantization layer145, the second de-quantization layer245, and the fifth neural network230may be trained using an end-to-end scheme.

As shown inFIGS.7and8, the first neural network125and the fourth neural network225may include the RNNs127and227, respectively, and the second neural network130and the fifth neural network230may include the FNNs132and232, respectively. A periodic component of an input signal, for example, a speech signal or an audio signal, may be processed using the first neural network125and the fourth neural network225that include the RNNs127and227, respectively. A non-periodic component of the input signal may be processed by using the second neural network130and the fifth neural network230that include the FNNs132and232, respectively. Two decoded residual signals with different attributes may be combined through a gating neural network, for example, including the third neural network135to reconstruct a residual signal, and thus an reconstruction quality of the input signal may be enhanced as well as a coding efficiency may be improved.

The components described in the example embodiments may be implemented by hardware components including, for example, at least one digital signal processor (DSP), a processor, a controller, an application-specific integrated circuit (ASIC), a programmable logic element, such as a field programmable gate array (FPGA), other electronic devices, or combinations thereof. At least some of the functions or the processes described in the example embodiments may be implemented by software, and the software may be recorded on a recording medium. The components, the functions, and the processes described in the example embodiments may be implemented by a combination of hardware and software.

The method according to example embodiments may be written in a computer-executable program and may be implemented as various recording media such as magnetic storage media, optical reading media, or digital storage media.

Implementations of the various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof. Implementations may be implemented as a computer program product, i.e., a computer program tangibly embodied in an information carrier, e.g., in a machine-readable storage device (computer-readable medium), for processing by, or to control an operation of, a data processing apparatus, e.g., a programmable processor, a computer, or multiple computers. A computer program, such as the computer program(s) described above, may be written in any form of a programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, a component, a subroutine, or other units suitable for use in a computing environment. A computer program may be deployed to be processed on one computer or multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

Processors suitable for processing of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory (ROM) or a random access memory (RAM), or both. Elements of a computer may include at least one processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also may include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. Examples of information carriers suitable for embodying computer program instructions and data include semiconductor memory devices, for example, magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as compact disc ROMs (CD-ROMs) or digital versatile discs (DVDs), magneto-optical media such as floptical disks, ROMs, RAMs, flash memories, erasable programmable ROMs (EPROMs), or electrically erasable programmable ROMs (EEPROMs). The processor and the memory may be supplemented by, or incorporated in special purpose logic circuitry.

In addition, non-transitory computer-readable media may be any available media that may be accessed by a computer and may include both computer storage media and transmission media.

While the present specification contains many specific implementation details, these should not be construed as limitations on the scope of any disclosure or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular disclosures. Specific features described in the present specification in the context of individual example embodiments may also be combined and implemented in a single embodiment. On the contrary, various features described in the context of a single embodiment may be implemented in a plurality of example embodiments individually or in any appropriate sub-combination. Moreover, although features may be described above as acting in specific combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be changed to a sub-combination or a modification of a sub-combination

Likewise, although operations are depicted in a predetermined order in the drawings, it should not be construed that the operations need to be performed sequentially or in the predetermined order, which is illustrated to obtain a desirable result, or that all of the shown operations need to be performed. In some cases, multi-tasking and parallel processing may be advantageous. In addition, it should not be construed that the division of various device components of the aforementioned example embodiments is required in all types of embodiments, and it should be understood that the described program components and devices are generally integrated as a single software product or packaged into a multiple-software product.

The example embodiments disclosed in the present specification and the drawings are intended merely to present specific examples in order to aid in understanding of the present disclosure, but are not intended to limit the scope of the present disclosure. It will be apparent to one of one of ordinary skill in the art that various modifications based on the technical spirit of the present disclosure, as well as the disclosed example embodiments, can be made.