# Road Extraction Based on Improved Convolutional Neural Networks with Satellite Images
## 2 Methods
### Review of CNNs  
The whole convolutional network can be constructed by stacking many layers, such as the convolutional layer, batch normalization (BN) [22], rectified linear units (ReLUs) [23], pooling [24], and dropout [25]. Set \(x_{l}\) as the output of the \(l\)th layer. For CNNs, \(x_{l}\) is taken as a non-linear transformation, \(H_{l}\), to the output of the former layer, \(x_{l-1}\)[26]  
\[x_{l}=H_{l}(x_{l-1}) \tag{1}\]  
where \(H_{l}\) is an integrated function including BN, ReLUs, pooling, dropout, or convolution (Conv). The degradation problem of CNNs would become more serious if the network depth is deeper because of a vanishing gradient, which has been documented by normalized initialization [27, 28, 29] and intermediate normalization layers [22].  
In terms of the above problems and training suitable deeper networks, many researchers introduced various structure networks, such as highway networks [30], stochastic depth [31], and FractalNets [32]. The advantage of these networks is to solve the problem by using a shortcut connection between the input and the output without more non-linear transformations. The authors of [33] introduce a residual block (Figure 1) and plug a skip connection to summarize the input and the output. \(x_{l}\) can be expressed as:  
\[x_{l}=H_{l}(x_{l-1})+x_{l-1} \tag{2}\]  
where \(H_{l}\) is the repetition (2 or 3 times) of a block combined with BN, followed by ReLUs and a Conv. With the shortcut connection, the gradient can flow from the deep layer to the shallow layer in the backpropagation, which can avoid a vanishing gradient. So, a network with more than one thousand layers with residual blocks can achieve the convergence aim.  
In terms of promoting the flow of information and gradients through the whole network, DenseNets [34] introduces a new connection pattern among layers (Figure 2). The dense block connects each layer in feed-forward fashion instead of the residual block. The feature maps of all preceding layers can be taken as inputs, and the inputs are put into the subsequent layers by concatenation. Different from ResNets, the network combined all feature maps by concatenation instead of summation, to avoid impeding information flow. Consequently, the \(l\)th layer obtains the outputs of the total preceding layers, \(x_{0},\dots,x_{l-1}\), which are taken as inputs, and the output \(x_{l}\) can be expressed as:  
\[x_{l}=H_{l}([x_{0},x_{1},\cdots,x_{l-1}]) \tag{3}\]  
where \([x_{0},x_{1},\cdots,x_{l-1}]\) indicates the concatenation of the feature maps in layers \(0,\dots,l-1\). So, \(H_{l}\) is a composite function with BN, followed by ReLUs and a Conv. Because of its dense connectivity, DenseNets can exploit the network potential through feature reuse. Then, the output dimension for each layer, \(l\), has \(k\) feature maps, and \(k\) refers to the growth rate parameter, which can be set to a small value (e.g., \(k=12\)) [26].  
Figure 2 shows that an input, \(x_{0}\), may be the input image at the initial position of the network or the end of the last layer, and an output feature map, \(x_{1}\), with \(k\) channels is computed by applying the transformation function, \(H_{1}(x_{0})\). Further, \(x_{1}\) is stacked to the previous feature  
Figure 1: The structure of the residual block [33].  
Figure 2: The structure of the dense block.  
map, \(x_{0}\), by concatenation (i.e., \([x_{1},x_{0}]\)) and taken as input to the next layer. The similar operation is repeated \(n\) times and presents a dense block with \(n\times k\) feature maps (Figure 3).  
The concatenation operation can bring more inputs, because \(1\times 1\) convolution has been introduced in the network as a bottleneck layer before each \(3\times 3\) convolution for decreasing the number of input feature maps [20; 29], which can develop computational efficiency. So, the dense block adopts a bottleneck design and produces \(4k\) feature maps, and \(H_{l}\) can be referred to as BN-ReLU-Conv(\(1\times 1\))-BN-ReLU-Conv(\(3\times 3\)).