Apparatus and method for speech processing using a densely connected hybrid neural network

Disclosed is a speech processing apparatus and method using a densely connected hybrid neural network. The speech processing method includes inputting a time domain sample of N*1 dimension for an input speech into a densely connected hybrid network; passing the time domain sample through a plurality of dense blocks in a densely connected hybrid network; reshaping the time domain samples into M subframes by passing the time domain samples through the plurality of dense blocks; inputting the M subframes into gated recurrent unit (GRU) components of N/M-dimension; outputting clean speech from which noise is removed from the input speech by passing the M subframes through GRU components.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Korean Patent Application No. 10-2020-0054733, filed on May 7, 2020, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference.

BACKGROUND

1. Field of the Invention

The present invention relates to a speech processing apparatus and method using a densely connected hybrid neural network.

2. Description of the Related Art

Monaural speech enhancement can be described as a process of extracting a target speech signal by suppressing background interference in a speech mixture in a single microphone setup.

There have been various classic methods, such as spectral subtraction, Wiener-filtering and non-negative matrix factorization, to remove the noise without leading to objectionable distortion or adding too much artifacts, such that the denoised speech is of decent quality and intelligibility.

Recently, a deep neural network (DNN), a data-based computing paradigm, has been widely used due to its strong parameter estimation capacity and its promising performance. DNN formulates monaural speech enhancement by mask estimation or end-to-end mapping. For mask estimation, DNN usually takes acoustic features in time-frequency (T-F) domain to estimate a T-F mask, such as ideal binary mask (IBM), ideal ratio mask (IRM) and phase-sensitive mask (PSM). Recently, end-to-end speech enhancement DNN gains attention as it takes noisy speech and outputs the denoised signal in time-domain directly, without any feature engineering.

In both mask estimation and end-to-end mapping DNNs, dilated convolution serves a critical role to aggregate contextual information with the enlarged receptive field. Gated residual network (GRN) employs dilated convolutions to accumulate context in temporal and frequency domains, leading to a better performance than a long short-term memory (LSTM) cell-based model. In end-to-end setting, WaveNet and its variations also run-on dilated convolution to give competitive results in speech enhancement and speech coding.

For real-time systems deployed in resource-constrained environment, however, oversized receptive fields from dilated convolution can cause a severe delay issue. Although causal convolution can enable real-time speech denoising, it performs less well comparing to the dilated counterpart. Besides, when the receptive field is too large, the amount of padded zeroes in the beginning of the sequence and a large buffer size for online processing can be a burdensome spatial complexity for a small device, too.

Meanwhile, recurrent neural networks (RNN) can also aggregate context by using recurrent model architectures, which enables a frame-by-frame processing without relying on the large receptive field. However, the responsiveness of a practical RNN system, such as LSTM, comes at the cost of increased model parameters, which are neither as easy to train nor resource-efficient.

SUMMARY

An aspect provides a sound event detection method comprising inputting a time domain sample of N*1 dimension for an input speech into a densely connected hybrid network; passing the time domain sample through a plurality of dense blocks in a densely connected hybrid network; reshaping the time domain samples into M subframes by passing the time domain samples through the plurality of dense blocks; inputting the M subframes into gated recurrent unit (GRU) components of N/M-dimension; outputting clean speech from which noise is removed from the input speech by passing the M subframes through GRU components, wherein the densely connected hybrid network is combination a convolutional neural network (CNN) and a recurrent neural network (RNN).

DETAILED DESCRIPTION

Hereinafter, example embodiments will be described in detail with reference to the accompanying drawings. The scope of the right, however, should not be construed as limited to the example embodiments set forth herein. Like reference numerals in the drawings refer to like elements throughout the present disclosure.

Various modifications may be made to the example embodiments. Here, the examples are not construed as limited to the disclosure and should be understood to include all changes, equivalents, and replacements within the idea and the technical scope of the disclosure.

Although terms of “first,” “second,” and the like are used to explain various components, the components are not limited to such terms. These terms are used only to distinguish one component from another component. For example, a first component may be referred to as a second component, or similarly, the second component may be referred to as the first component within the scope of the present disclosure.

Unless otherwise defined herein, all terms used herein including technical or scientific terms have the same meanings as those generally understood by one of ordinary skill in the art. Terms defined in dictionaries generally used should be construed to have meanings matching contextual meanings in the related art and are not to be construed as an ideal or excessively formal meaning unless otherwise defined herein.

Hereinafter, the example embodiments will be described described in detail with reference to the accompanying drawings.

FIG.1is a flowchart showing a speech processing method according to an embodiment of the present invention.

To leverage the benefit from temporal contextual aggregation, but with a low delay and complexity in the end-to-end setting, a densely connected convolutional and recurrent network (DCCRN) is suggested, which conducts dual-level context aggregation.

The first level of context aggregation in DCCRN is done by a dilated 1D convolutional network component, with a DenseNet architecture [20], to extract the target speech from the noisy mixture in the time domain. It is followed by a compact gated recurrent unit (GRU) component [21] to further leverage the contextual information in the “many-to-one” fashion.

Note that the present application employ a cross-component identical shortcut linking the output of DenseNet component to the output of GRU component to reduce the complexity of the GRU cells. To better tune our heterogeneous neural network components, a component wise training scheme specially designed for DCCRN is proposed. The proposed speech processing method represents that the hybrid architecture of dilated CNN and GRU in DCCRN consistently helps outperform the CNN variations with only one level of context aggregation. In terms of model complexity, the proposed speech processing method is computationally efficient even with the additional GRU layers. The delay is down to 0.016 second for the potential in real-time scenarios, with a potential “cold start” period of 0.048 second (i.e., the maximum lookback size).

Referring toFIG.1, a process of removing noise of an input speech using DCCRN will be described.

In step101, the speech processing apparatus may input a time domain sample of an N*1 dimensional input speech to the densely connected hybrid network. Here, the densely connected hybrid network may be a network in which a convolution neural network (CNN) and a recurrent neural network (RNN) are combined.

The densely connected hybrid network may include a plurality of dense blocks. And, each of the dense blocks may be composed of a plurality of convolutional layers.

And, the time domain sample of the N*1 dimension is extended to convolutional layers of the N*D dimension to the N*MD dimension. In this case, each of the convolutional layers of the N*D dimension to the N*MD dimension may be connected. D denotes an expansion rate, and M may be the number of convolutional layers constituting the dense block.

In step102, the speech processing apparatus may pass the time domain samples through a plurality of dense blocks constituting a densely connected hybrid network.

Each of the plurality of dense blocks included in the densely connected hybrid network may be represented as a repetitive neural network due to having the same convolutional layers with each other. In addition, each of the dense blocks may output a data tensor in the form of a combination of features and channels by performing a 1D convolution operation. Meanwhile, each of the dense blocks may perform a 1D convolution operation or an extended convolution operation according to the expansion rate.

In step103, the speech processing apparatus may reshape the time domain samples into M subframes by passing the time domain samples through a plurality of dense blocks.

In step104, the speech processing apparatus may input M subframes into N/M-dimensional gated recurrent unit (GRU) components. The output of the last dense block of the densely connected hybrid network may be reshaped into M subframes. In addition, the reshaped M subframes may be input to N/M-dimensional GRU components.

The GRU components may enhance speech by mixing the first hidden state and the second hidden state using the update gate. Here, the second hidden state may be determined using a linear combination of an input of the GRU component and a first hidden state that is a gated previous hidden state.

In step105, the speech processing apparatus may output a clean speech from which noise is removed from the input speech by passing M subframes through the GRU components.

FIG.2is a diagram illustrating a DCCRN including Dilated DenseNet according to an embodiment of the present invention.FIG.3is a diagram illustrating a DCCRN connecting a fourth DenseNet ofFIG.2and a GRU component.

When tuning a deep convolutional neural network (CNN), residual learning is an important technique for solving the gradient vanishing problem. So, deep CNN achieves high performance with low model complexity. ResNet enables residual learning by adding the same short cut to the bottle-neck structure.

Although the bottleneck structure includes a direct path to feedforward information from an early layer to a later layer, it does not extend the full capacity of the information flow. Therefore, ResNet is sometimes accompanied by gating mechanisms, a technique heavily used in the RNNs, such as LSTM or GRU, to further facilitate the gradient propagation in convolutional network.

In comparison, DenseNet resolves the above problem by relying on the same shortcuts. The dense block differs from the bottleneck structure in that each layer concatenates the output from all preceeding layers as its input, while it feeds its own output to all subsequent layers. Due to the density skip connection. DenseNet tends to require fewer model parameters to achieve more competitive performance.

In a fully convolutional architecture, a dilated convolution is a technique that expands the receptive field to cover long sequences and shows excellent performance in speech enhancement as well. As the CNN's model complexity is lower, extended convolution is considered an inexpensive alternative to recurrence operations. The speech processing method is method to adopt with a receptive field size that does not exceed the frame size.

In the present invention,(l)denotes a convolution operation between an input X(l)and a filter H(l)in the l-th layer, which γ(l)is a dilation rate. When there is no expansion, the default value of the dilation rate γ(l)is 1.

In Equation 1, n, τ, d, k denotes an input feature, an output feature, channels, and filter coefficients, respectively. k is an integer that satisfies k≤└K/2┘, and K is the 1D kernel size. In the speech processing method, two kernel sizes of Ks≅5 and Kl=55 are proposed, for example.

The DCCRN proposed in the present invention is based on 1D convolution, and a tensor is in the form of a product of a feature and a channel ((features)×(channels)).

The speech processing method uses zero padding to maintain the number of features in the same cross layers. A specific convolution, such as the second equation in Equation 1, activates an dilated convolution with a selection of γ(l)>1.

In DCCRN, a dense block combines five convolutional layers as shown inFIG.2. The input of the l-th layer in each dense block is a channel-wise concatenated tensor of all previous feature maps in the same dense block.
Xl+1←(l)([X(l),X(l-1), . . . ,X(lb)],H(l),γ(l))  <Equation 2>

Here, X(lb)denotes the first input feature mapped to the b-th block. DensNet architecture H(l)grows according to H(l)∈K×(l-lb+1)Dhaving a growth rate D and depth X(lb).

In the last layer of the dense block, the concatenated input channels collapse down to D, which forms the input to the next block. The first dense block inFIG.2represents this process. According to an embodiment of the present invention, the speech processing method stacks four dense blocks with the dilation rate of the last layer in each block to be 1, 2, 4 and 8, respectively. Unlike the original DenseNet architecture, there is no transition between dense blocks in the present invention. In the present invention, one dense block for the first layer is expanded to a channel N×D using time domain samples N×1, and the other dense block for the first layer is reduced to one channel again after the stacked dense blocks. This forms the final output of the complete convolutional DenseNet baseline. In all convolutional layers, the present invention uses ReLU as the activation function.

The LSTM and GRU as RNN variants are used for speech enhancement due to their sequence modeling capacity facilitated by the invention of memory cells and various gates. The DCCRN selects GRU components with reduced computational complexity compared to LSTM. The information flow in each GRU component is represented as in Equation 3.
h(t)=(1−z(t))⊙h(t−1)+z(t)⊙{tilde over (h)}(t)
{tilde over (h)}(t)=tanh(Whx(t)+Uh(r(t)⊙h(t−1)))
z(t)=σ(Wzx(t)+Uzh(t−1))
r(t)=σ(Wrx(t)+Urh(t−1)),  <Equation 3>

Where t is the index of the sequence. h and {tilde over (h)} are the hidden state and the newly proposed state. h and {tilde over (h)} are mixed by the update gate Z in a complementary manner as in the first equation of equation 3. The GRU cell calculates the tan h unit {tilde over (h)} using a linear combination of the input X and the gated previous hidden state h according to the second equation of Equation 3.

Similarly, gates are estimated using different sigmoid units, such as the third and fourth equations of Equation 3. In all linear operations, GRU uses the corresponding weight matrices Wh, Wz, Wr, Uh, Uz, Ur, but the bias term is omitted.

The GRU component follows a “many-to-one” mapping style for an additional level of context collection. During training, it looks back the Mth time step and produces an output corresponding to the last time step. To this end. DCCRN reshapes the N×1 vector, which is the output of the CNN part, into M subframes. Each of the M subframes is an N/M dimensional input vector for a GRU cell. The DCCRN described in the present invention has two GRU layers. One GRU layer has 32 hidden units, and the other has N/M units, which are the output dimensions of the system.

In addition, in order to facilitate optimization and limit the model complexity of the GRU layer, the present invention passes the output of the last N/M subframe of the DenseNet component to the output of the GRU component through a skipping connection. The noise-removed speech is the sum of the outputs of the DenseNet component and the GRU component. With a well-tuned DenseNet, output close to clear speech may be obtained, and the work of optimizing GRU components can be reduced.

FIG.4is a process for explaining a feed forward procedure in DCCRN according to an embodiment of the present invention.

During training, the noise signal is first fed to the DenseNet component D (x;CNN) as shown inFIG.2(third row inFIG.4). It means that L consecutive convolutional layers may be grouped into 4 dense blocks.

If the dense blocks are not related to GRU layers, the DenseNet component may take N samples of the noise signal and predict the last N/M clean speech sample for which the loss is calculated. Since the first N−N/M output samples or future samples are not considered, this process may be viewed as a causal convolution process. Instead, to accommodate the GRU layers, the DCCRN training procedure generates an N/M dimensional vector (X(L)−∈N/M×M) using all N samples from the CNN part.

As a result, final GRU hidden states, which is a matching the N/M dimensional vector, may generate an output cleanup signal Ŝ. During the test time, the GRU layers may select an N/M-dimensional subframe at a given time and predict an output cleanup signal Ŝ. Each of input sub-frames for the GRU layer is the result of the stacked dense blocks with a receptive field of up to N past samples.

FIG.5is a diagram illustrating an output shape derived through each component according to an embodiment of the present invention.

The objective function is based on the mean squared error (MSE), but the present invention may improve the objective function using an additional perceptual term. MSE alone cannot measure perceptual quality and intelligibility. To this end, the MSE may either accept a psychoacoustic weighting scheme or be replaced by a clearly salient measure such as short-time objective intelligibility (STOI), but an differentiable objective function. Does not exist.

In the present invention, the objective function is based on the MSE having normalization comparing the Mel-Spectra of the target and the output signal.
ε(s∥ŝ)=MSE(s∥ŝ)+λMSE(Mel(s)∥Mel(ŝ))  <Equation 4>

FIG.5shows the architecture of DCCRN for CNN layers. Size of data tensor are expressed as the size in samples and channels, while the shape of CNN kernels is expressed as the site of samples, input channels and output channels. For GRU layers, the additional dimension of the data tensors defines the length of the sequence (M=4). On the other hand, kernel sizes define a linear operation (input features, output features). The middle layer of each dense block, denoted as a dagger, has a large kernel size K1=55, with optional dilations 1, 2, 4, and 8 for the 4 dense blocks.

According to the present invention, after training CNN components and RNN components, respectively, the combined network may be finely tuned.

CNN training: First, the speech processing method trains a CNN (DenseNet) component that minimizes errors

RNN training: the speech processing method trains the RNN part in many-to-one by minimizing the following

Integrative finetuning: Once the CNN and RNN are trained in advance, the speech processing method fixes the CNN parameter and performs fine tuning using the CNN component and the RNN component in order to minimize the final error

argmin𝕎CNN,𝕎RNN⁢ℰ(s⁢⁢(𝒟⁡(x;𝕎CNN);𝕎RNN).
The learning rate for integrated fine tuning may be reduced.

The speech processing method provides a dual-staged temporal context aggregation network having both a CNN component and an RNN component for modeling a time domain signal in an end-to-end form.

The speech processing method provides a dual-staged temporal context aggregation network having both a CNN component and an RNN component for end-to-end monaural speech enhancement.

The speech processing method provides an efficient heterogeneous model architecture that is dense and enhanced by cross-component residual connectivity.