METHOD AND DEVICE FOR ENCODING/DECODING AUDIO SIGNAL BASED ON DEQUANTIZATION THROUGH POTENTIAL DIFFUSION

A method and device for encoding/decoding an audio signal based on dequantization through potential diffusion are provided. The method of decoding an audio signal includes obtaining a discrete latent vector in which a speech signal is quantized and based on the discrete latent vector, outputting a continuous latent vector in which the discrete latent vector is dequantized.

BACKGROUND

1. Field of the Invention

One or more embodiments relate to a method and device for encoding/decoding an audio signal based on dequantization through potential diffusion.

2. Description of the Related Art

A neural speech codec (NSC) has been developed to more effectively capture complex patterns in a speech signal. The NSC may be mainly classified into two types, for example, an end-to-end codec and a neural vocoder.

A generative model may be a model for generating data in the field of artificial intelligence. The generative model may generate new data based on previously provided data or may generate new samples by training distribution of provided data. The generative model is utilized in various fields and may be applied in image generation, natural language generation, and voice generation.

A diffusion model, an example of the generative model, is one of the recently emerged generative models and may use a diffusion process to approximate distribution of data. The diffusion model may gradually transform provided initial data to generate desired data.

The above description has been possessed or acquired by the inventor(s) in the course of conceiving the present disclosure and is not necessarily an art publicly known before the present application is filed.

SUMMARY

Embodiments may provide technology of estimating a high bit rate continuous latent vector from a low bit rate discrete latent vector using a generative model to decode a compressed speech signal.

However, the technical aspects are not limited to the aforementioned aspects, and other technical aspects may be present.

According to an aspect, there is provided a method of processing a speech signal, the method including obtaining a discrete latent vector in which the speech signal is quantized and based on the discrete latent vector, outputting a continuous latent vector in which the discrete latent vector is dequantized.

The outputting of the continuous latent vector may include gradually up-sampling the discrete latent vector according to a plurality of layers included in a neural network.

The up-sampling of the discrete latent vector may include, based on the discrete latent vector and a first continuous latent vector corresponding to a first layer among the plurality of layers, estimating a second continuous latent vector corresponding to a second layer among the plurality of levels.

The estimating of the second continuous latent vector may include, based on the discrete latent vector and the first continuous latent vector, estimating noise in the first continuous latent vector and calculating the second continuous latent vector by removing the noise from the first continuous latent vector.

The method may further include generating a restored speech signal based on a continuous latent vector output through a layer of a highest level among the plurality of layers.

According to another aspect, there is provided an electronic device for processing a speech signal, the electronic device including a processor and a memory configured to store instructions, wherein the instructions, when executed by the processor, may cause the electronic device to obtain a discrete latent vector in which the speech signal is quantized and based on the discrete latent vector, output a continuous latent vector in which the discrete latent vector is dequantized.

The instructions, when executed by the processor, may cause the electronic device to gradually up-sample the discrete latent vector according to a plurality of layers included in a neural network.

The instructions, when executed by the processor, may cause the electronic device to, based on the discrete latent vector and a first discrete latent vector corresponding to a first level among the plurality of levels, estimate a second continuous latent vector corresponding to a second level among the plurality of levels.

The instructions, when executed by the processor, may cause the electronic device to, based on the discrete latent vector and the first continuous latent vector, estimate noise included in the first continuous latent vector and calculate the second continuous latent vector by removing the noise from the first continuous latent vector.

The instructions, when executed by the processor, may cause the electronic device to generate a restored speech signal based on a continuous latent vector output through a layer of a highest level among the plurality of layers.

DETAILED DESCRIPTION

The following detailed structural or functional description is provided as an example only and various alterations and modifications may be made to the embodiments.

Accordingly, the embodiments are not construed as limited to the disclosure and should be understood to include all changes, equivalents, and replacements within the idea and the technical scope of the disclosure.

Although terms, such as first, second, and the like are used to describe various components, the components are not limited to the terms. These terms should be used only to distinguish one component from another component. For example, a first component may be referred to as a second component, and similarly the second component may also be referred to as the first component.

It should be noted that if one component is described as being “connected”, “coupled”, or “joined” to another component, a third component may be “connected”, “coupled”, and “joined” between the first and second components, although the first component may be directly connected, coupled, or joined to the second component.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure pertains. Terms, such as those defined in commonly used dictionaries, should be construed to have meanings matching with contextual meanings in the relevant art, and are not to be construed to have an ideal or excessively formal meaning unless otherwise defined herein.

Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. When describing the embodiments with reference to the accompanying drawings, like reference numerals refer to like components and a repeated description related thereto will be omitted.

FIG.1illustrates an example of a speech signal processing system according to an embodiment.

Referring toFIG.1, a speech signal processing device may include an encoder110and a decoder130. However,FIG.1is an example for describing the present disclosure, and the scope of the present disclosure should not be construed as being limited thereto. For example, the speech signal processing device100may include only one of the encoder110and the decoder130.

Processing a speech signal may include compressing a speech signal and/or restoring a compressed speech signal to a signal before compression.

The encoder110may encode an input speech signal to generate a bit stream and may transmit (or output) the bit stream to the decoder130.

The decoder130may decode the bitstream obtained (or received) from the encoder110to generate a restored speech signal.

The specific configuration and operation of the encoder110and the decoder130are described in detail below with reference toFIG.2.

FIG.2is a diagram illustrating an encoder and a decoder shown inFIG.1.

Referring toFIG.2, the encoder110may include a discrete encoder290.

The decoder130may include a continuous decoder210and a generative model250.

The encoder110may generate a bit stream in which a speech signal (e.g., the input speech signal ofFIG.1) is encrypted. The discrete encoder290may output a discrete latent vector based on the speech signal. The discrete latent vector is a vector in which the speech signal is quantized and may have a low bit rate. The discrete encoder290may be an auto encoder for training a discrete latent space. The discrete encoder290may reduce a time-domain speech signal into the discrete latent spaceto generate the discrete latent vector.

The discrete latent vector described above may include a latent vector of the discrete encoder290and/or a discrete decoder (e.g., a discrete decoder230ofFIG.3A) configured through end-to-end training in the field of neural speech codec (NSC) technology.

The decoder130may obtain (e.g., receive) a bit stream (e.g., the discrete latent vector) from the encoder110.

The decoder130may generate a high-quality restored speech signal by using a low bit rate discrete latent vector. For example, the decoder130may generate a continuous latent vector from the discrete latent vector through the generative model250and may restore the continuous latent vector by the continuous decoder210to generate the high-quality restored speech signal.

Hereinafter, a training method of the encoder110and/or the decoder130is described in detail with reference toFIGS.3A and3B.

FIGS.3A and3Bare diagrams illustrating a training method of the encoder and the decoder shown inFIG.2.

Referring toFIG.3A, training of a continuous encoder270and the continuous decoder210, training of the discrete encoder290and the discrete decoder230, and training of the generative model250may be performed independently.

The discrete encoder290may be trained to output a discrete latent vector based on a speech signal (e.g., the input speech signal ofFIG.1). A detailed description of the discrete latent vector is provided with reference toFIG.2, and the description thereof is omitted herein.

The discrete decoder230may be trained to generate a first restored speech signal based on the discrete latent vector. The first restored speech signal is generated based on a low bit rate discrete latent vector and may have low quality.

The continuous encoder270may be trained to output a continuous latent vector based on a speech signal (e.g., the input speech signal ofFIG.1). The continuous latent vector may be a latent vector in which the discrete latent vector is dequantized. In addition, the continuous latent vector is a vector in which the speech signal is encrypted but not quantized and may have a high bit rate. The continuous encoder270may be an auto encoder for training a continuous latent space. The continuous encoder270may reduce a time-domain speech signal into the continuous latent spaceto generate the continuous latent vector. The continuous decoder210may obtain the continuous latent vector from the continuous encoder270.

The continuous decoder210may be trained to generate a second restored speech signal based on the continuous latent vector. The second restored speech signal is generated based on a high bit rate continuous latent vector and may have high quality. The reason for the difference in quality between the first restored speech signal and the second restored speech signal may be that the discrete latent vector, in which an encrypted speech signal is quantized, has information (e.g., including information about speech features of the input speech signal inFIG.1) less than the continuous latent vector.

The continuous latent vector described above may include a latent vector of the continuous encoder270and/or the continuous decoder210configured through end-to-end training in the field of NSC technology.

When training of the continuous encoder270and the continuous decoder210and training of the discrete encoder290and the discrete decoder230are completed, the continuous decoder210may be mounted on the encoder110and the discrete encoder290may be mounted on the decoder130, as shown inFIG.2. However, embodiments are not limited thereto.

Hereinafter, a training method of the generative model250is described.

Referring toFIG.3B, the decoder130may use a discrete latent vector and a continuous latent vector for training the generative model250. When the training of the generative model250is completed, the generative model250may generate the continuous latent vector based on the discrete latent vector.

The generative model250may refer to a neural network for producing a specific output (e.g., the continuous latent vector) based on an input condition (e.g., the discrete latent vector). The neural network may include a plurality of layers (e.g., layers255-1to255-N).

The neural network (or an artificial neural network) may include a statistical training algorithm that mimics biological neurons in machine learning and cognitive science. The neural network may generally refer to a model having a problem-solving ability implemented through artificial neurons or nodes forming a network through synaptic connections where the strength of the synaptic connections is changed through learning.

A neuron of the neural network may include a combination of weights or biases. The neural network may include one or more layers, each including one or more neurons or nodes. The neural network may infer a result from a predetermined input by changing weights of the neurons through training.

The neural network may include a deep neural network (DNN). The neural network may include a convolutional neural network (CNN), a recurrent neural network (RNN), a perceptron, a multilayer perceptron, a feed forward (FF), a radial basis function (RBF) network, a deep feed forward (DFF), a long short-term memory (LSTM), a gated recurrent unit (GRU), an auto encoder (AE), a variational auto encoder (VAE), a denoising auto encoder (DAE), a sparse auto encoder (SAE), a Markov chain (MC), a Hopfield network (HN), a Boltzmann machine (BM), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a deep convolutional network (DCN), a deconvolutional network (DN), a deep convolutional inverse graphics network (DCIGN), a generative adversarial network (GAN), a liquid state machine (LSM), an extreme learning machine (ELM), an echo state network (ESN), a deep residual network (DRN), a differentiable neural computer (DNC), a neural turning machine (NTM), a capsule network (CN), a Kohonen network (KN), a visual geometry group (VGG) network, and an attention network (AN).

Hereinafter, a method of training the generative model250is described in detail.

The generative model250may be trained to generate the continuous latent vector based on the discrete latent vector. For example, the generative model250may be implemented as a diffusion model to generate the continuous latent vector. The diffusion model may be a model trained to generate output based on an input condition through a Markov process. The Markov process may include a forward process and a reverse process. Hereinafter, a method of training the generative model250through the forward process and the reverse process is described.

For ease of description, terms are defined as follows. It is assumed that the layer255-1is the lowest level layer and levels increase as next operations are performed. For example, the layer255-2may be a layer one level higher than the layer255-1. In addition, the continuous latent vector corresponding to a layer (any one of the layers255-1to255-N) is assumed to be the continuous latent vector input to the layer based on the reverse process. This may be identical to the continuous latent vector output through a layer in the forward process. For example, a continuous latent vectors zTmay correspond to the layer255-1.

In the forward process, a first continuous latent vector z0 may be progressively down-sampled according to the plurality of layers (e.g., the layers255-1to255-N) included in the neural network (e.g., the generative model250). For example, the first the continuous latent vector z0 may be input to the layer255-N and may be added to noise ∈0(e.g. Gaussian noise) to become a second continuous latent vector z1. The second continuous latent vector z1 may also be input to a next layer (e.g., a layer one level lower than the layer255-N) to become a third continuous latent vector z2 in substantially the same method as the first continuous latent vector z0 is down-sampled. When this method is performed repeatedly, an N-th continuous latent vector zTmay be generated. The N-th continuous latent vector zTmay be a latent vector converted into noise by adding noise N times (e.g., the number of times input to a layer (e.g., at least one of the layers255-1to255-N)) to the first continuous latent vector z0. That is, in the forward process, noise may be added for each operation when the continuous latent vector is advanced to a next operation.

In the reverse process, a discrete latent vector h may be progressively up-sampled according to the plurality of layers (e.g., the layers255-1to255-N) included in the neural network (e.g., the generative model250). The discrete latent vector h may be up-sampled according to the plurality of layers to become the continuous latent vector z0. For example, the layer255-1may receive the discrete latent vector h and the continuous latent vector zTas input and may estimate noise {circumflex over (∈)}T-1(e.g. Gaussian noise) included in (or added to) the continuous latent vector zT. The layer255-1may generate a continuous latent vector zT-1by removing noise {circumflex over (∈)}T-1estimated in the continuous latent vector zT. When this method is performed repeatedly on the continuous latent vector zTcorresponding to the layer255-1as the level of layers is increased, the continuous latent vector z0 corresponding to the layer255-N having the highest level may be calculated (or estimated). That is, when the discrete latent vector is advanced to a next operation in the reverse process, noise may be removed for each operation.

A parameter (e.g., ∈θ(zT,T,h)) included in the generative model250may be trained through the forward process and/or the reverse process. A parameter may be for estimating noise in the continuous latent vector input for each layer. For example, parameters may be trained so that the difference between noise ∈T-1to which the continuous latent vector zT-1input to the layer255-1in the forward process is added and the noise {circumflex over (∈)}T-1estimated in the layer255-1in the reverse process is minimized. The parameters may be trained through Equation 1 below.

In Equation 1,z0,t,h denotes a loss function about noise, ∈tdenotes noise that is added to a t−1-th continuous latent vector input to a t-th layer in the forward process, and ∈θ(zt,t,h) denotes a parameter for estimating noise in a t-th continuous latent vector input to the t-th layer in the reverse process.

When training of the generative model250is completed, the generative model250may generate the continuous latent vector z0 based on the discrete latent vector h. Hereinafter, the description thereof is provided in detail below.

The generative model250may gradually up-sample the discrete latent vector according to the plurality of layers included in the neural network. The generative model250may estimate a second continuous latent vector corresponding to a second layer among the plurality of layers (the layers255-1to255-N), based on the discrete latent vector and a first continuous latent vector corresponding to a first layer among the plurality of layers (the layers255-1to255-N). The first layer may be a layer with a level lower than the second layer. However, the first layer is not necessarily the layer255-1but may be any one of the plurality of layers (the layers255-1to255-N). For example, the first layer may be a layer255-n. Here, n may be an integer between 1 and N.

The generative model250may estimate noise in the first continuous latent vector based on the discrete latent vector and the first continuous latent vector. The generative model250may remove noise from the first continuous latent vector and may calculate the second continuous latent vector. This is described in detail in the reverse process above, so the description thereof is omitted hereinafter.

The generative model250may output a finally-generated continuous latent vector (e.g., a continuous latent vector output through the highest level layer among the plurality of layers (the layers255-1to255-N)) to the continuous decoder210. The continuous decoder210may generate a restored speech signal based on the finally-generated continuous latent vector.

The discrete encoder290, the continuous decoder210, and the generative model250may be mounted on an encoder (e.g., the encoder110ofFIG.2) and/or a decoder (e.g., the decoder130ofFIG.2) when training is completed. This is the same as that shown inFIG.2, so the detailed description thereof is omitted herein.

FIG.4illustrates an example of a flowchart of a speech signal processing method according to an embodiment.

Referring toFIG.4, operations410and430may be performed sequentially but are not limited thereto. For example, two or more operations may be performed in parallel. Operations410and430may be substantially the same as the operation of the speech signal processing device (e.g., the speech signal processing device100ofFIG.1) described with reference toFIGS.1and3B. Accordingly, further description thereof is not repeated herein.

In operation410, the speech signal processing device100may obtain (e.g., receive) a discrete latent vector in which a speech signal is quantized.

In operation430, the speech signal processing device100may output a continuous latent vector in which the discrete latent vector is dequantized based on the discrete latent vector.

FIG.5illustrates an example of an electronic device according to an embodiment.

Referring toFIG.5, an electronic device500may include a memory510and a processor530. The electronic device500may include the speech signal processing device100ofFIG.1. For example, the electronic device500may be a device including the decoder130ofFIG.1.

The memory510may store instructions (or programs) executable by the processor530. For example, the instructions include instructions for performing an operation of the processor530and/or an operation of each component of the processor530.

The memory510may be implemented as a volatile memory device or a non-volatile memory device.

The volatile memory device may be implemented as dynamic random-access memory (DRAM), static random-access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM).

The non-volatile memory device may be implemented as electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic RAM (MRAM), spin-transfer torque (STT)-MRAM, conductive bridging RAM (CBRAM), ferroelectric RAM (FeRAM), phase-change RAM (PRAM), resistive RAM (RRAM), nanotube RRAM, polymer RAM (PoRAM), nano floating gate memory (NFGM), holographic memory, a molecular electronic memory device, or insulator resistance change memory.

The processor530may process data stored in the memory510. The processor530may execute computer-readable code (e.g., software) stored in the memory510, and instructions triggered by the processor530.

The processor530may be a hardware-implemented data processing device having a circuit that is physically structured to execute desired operations. For example, the desired operations may include code or instructions in a program.

The hardware-implemented data processing device may include, for example, a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), and a field-programmable gate array (FPGA).

The processor510may cause the electronic device500to perform one or more operations by executing the instructions and/or code stored in the memory530. The operations performed by the electronic device500may be substantially the same as the operations performed by the speech signal processing device100described with reference toFIGS.1to3(e.g., a voice signal processing method performed by the speech signal processing device100and/or a training method of a neural network (e.g., the generative model250ofFIG.2) performed by the speech signal processing device100). Accordingly, a repeated description thereof is omitted.

The components described in the embodiments may be implemented by hardware components including, for example, at least one digital signal processor (DSP), a processor, a controller, an ASIC, a programmable logic element, such as an FPGA, other electronic devices, or combinations thereof. At least some of the functions or the processes described in the embodiments may be implemented by software, and the software may be recorded on a recording medium. The components, the functions, and the processes described in the embodiments may be implemented by a combination of hardware and software.

The above-described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described embodiments, or vice versa.