# The Application and Improvement of Deep Neural Networks in Environmental Sound Recognition
## 2 Related Works
### Feed-Forward Neural Network  
The simplest feed-forward neural networks would be Single-layer perceptrons, which can be built up to do a regression. Assume we would like to project a \(\mathcal{X}\in\mathbb{R}^{n}\) to \(\mathcal{Y}\in\mathbb{R}^{m}\), the two variables could be rewrite as two vectors \(\underline{X}=\ \left[\mathbb{S}_{1}\ \cdots\ \mathbb{S}_{n}\right]^{T}\) and \(\underline{\mathcal{Y}}=\ \left[\mathbb{t}_{1\ \cdots\ }\ \mathbb{t}_{m}\right]^{T}\), so we could simply try the formula below:  
\[\underline{\mathcal{Y}}=\mathsf{W}\underline{X}+\mathsf{b}. \tag{1}\]  
In (1), \(\mathsf{W}\ e\ \mathbb{R}^{m\times n}\) and \(\mathsf{b}\ e\ \mathbb{R}^{m\times 1}\), therefore, the main purpose to solve the equation is to find the suitable \(\mathsf{w}\) and \(\mathsf{b}\). If we already had a certain sample \(\mathcal{S}_{i}=(\mathcal{X}_{i},\mathcal{Y}_{i})\), obviously, we make the result of input \(\mathcal{X}_{i}\) could be as close to \(\mathcal{Y}_{i}\) as possible. There are several methods that can be used to retrieve the correct value of \(\mathsf{w}\) and \(\mathsf{b}\), such as the stochastic gradient descent (SGD) or [PERSON]'s method. No matter which method is applied, the equation will have a good result when \(\underline{X}\) and \(\underline{\mathcal{Y}}\) among all the samples are linearly dependent. Inspired by the animal neuron system, the activation function \(\varphi\) was added to improve Equation (1), and the new equations are listed as (2) and (3):  
\[\underline{\mathcal{Y}}=\mathsf{W}\underline{X}+\mathsf{b} \tag{2}\]  
\[\underline{\mathcal{Y}}=\varphi(\underline{\mathcal{Y}}). \tag{3}\]  
The activation function provides a non-linear transform to filter out the weaker signal. For example, the classic activation function sigmoid is:  
\[\text{sigmoid}(\mathsf{x})=\frac{1}{1+e^{-x}}. \tag{4}\]  
After passing Equation (4), every output value is straightly normalized to a range between 0 and 1, which is a superior non-linear transform. Equation (3) can now have the ability to make a regression to the non-linear equation.  
From Equation (3), it can be clearly seen that each element in \(\underline{\mathcal{Y}}\) is actually composed of every element of \(\underline{X}\) in different weights. Here, \(\underline{X}\) form an input layer, and each node create an element of \(\underline{\mathcal{Y}}\) is called a neuron in the network.  
To enhance the network structure, a hidden layer can be added to improve the network performance. The neuron numbers in the hidden layer needs to be decided by users, it usually setup to the value bigger than both input and output dimensions. The hidden layer is used to project the input vector into another dimension, resulting in a greater chance of finding a linear way to transform from a higher dimension to the output layers. Several non-linear transform also make the network hold greater power to complete complex regression.  
To obtain the correct weights of the feed-forward neural networks, the backpropagation method [24] (BP) is widely used. By calculating the gradient from the loss function, the gradients can be backward propagated to each of the weights.  
It seems that the network would better be design deeper, more layers, or wider, more neurons per layer, but actually both of the two methods all get some issues need to be deal with. The weight number grows exponentially with the width of the network, which also leads to a large growth of the computation times and also causes the network to face a serious overfitting condition. This also means that the network might easily fit the training data but still result in poor performance while testing. Deeper networks need to solve the gradient decent problem. When performing BP, the gradient travels from the end of the network and gets thinner and thinner while arriving at the front, and it can even vanish directly. A number of methods have been proposed to improve the vanishing gradient problem, of which a deeper network is recommended to be built as a solution.