# Integrating Dilated Convolution into DenseLSTM for Audio Source Separation
## 3 Proposed Architecture
### Multi-Scale Dilated Time-Frequency DenseLSTM  
MDilDenseLSTM is an architecture consisting of the dilated dense block (DDB), compression block (\"Compr.\"), LSTM block, down-sampling, and up-sampling, as shown in Figure 2. MDilDenseLSTM has four down-sampling (DS) and four up-sampling (US). The down-sampling used \(2\times 2\) average pooling and the up-sampling used \(2\times 2\) transposed convolution. The LSTM block is placed before the up-sampling, which makes scale 1 and the up-sampling located at the smallest scale. The advantage of the information and error transmission is further enhanced through an inter-block skip connection that connects outputs of the same size to each other during encoding and decoding [25]. The compression block compresses the information of many feature map into a small number of feature maps. The down-sampling and up-sampling reduces or increases the time-frequency size of the compressed feature map. The LSTM block, which is located between the compression block and up-sampling, outputs the target sequence information along the time axis. MDilDenseLSTM is a CRNN architecture by combining a CNN-based dilated dense block and an RNN-based LSTM block and is suitable when the input is an image and time series data such as a spectrogram.  
The compression block [36] consists of the BN, ReLU, and \(1\times 1\) convolution, and is placed behind the dilated dense block to appropriately limit the number of output feature maps from the dilated dense block and compress information at the same time. In the dilated dense block, which is a DenseNet-based architecture, since the output feature maps of each layer are concatenated, the number of output feature maps increases by the product of the number of layers. The compression rate \(\theta\) has a value of \(0<\theta\leq 1\). The number of output feature maps by placing a compression block behind the dilated dense block can be expressed as \((m_{0}+(3+L)k)\times\theta\), and when \(\theta=1\), the number of feature maps is maintained without compression.  
As shown in Figure 3, the LSTM block consists of a sequence layer of a \(1\times 1\) convolution, BLSTM, and FCNN, and has an architecture in which the input feature map is concatenated to the output feature map. The LSTM block makes the CNN-based DenseNet architecture into a CRNN architecture. In the CRNN architecture, a feature is firstly extracted with the CNN, and then the RNN using time series information extracts features and classifies the input into the target class [28]. Since the feature maps obtained from the spectrogram retain the time series information as it is, adding the RNN architecture  
Figure 2: The architecture of multi-scale dilated time-frequency DenseNet with LSTM (MDilDenseLSTM). This architecture is in the full band of multi-band MDilDenseLSTM (MMDilDenseLSTM). The red dashed box represents the recurrent neural network (RNN)-based architecture. DDB: dilated dense block, LSTM: long short-term memory.  
can reflect the time series information for all frames on the time axis. In addition, the receptive field can be widened on all frames of the time axis. The \(1\times 1\) convolution makes the number of input feature maps to 1 and puts them into the BLSTM layer. Since the frequency dimension of the output feature map varies according to the number of nodes in the BLSTM layer, the FCNN adjusts the number of frequency dimensions in the output feature map to be the same as the number of frequency dimensions in the input feature map. Since the input feature maps and the output feature map in LSTM block are concatenated, the number of output feature maps in the dilated dense block, compression block, and LSTM block can be expressed as \((m_{0}+(3+L)k)\times\theta+1\).