# Multi-Branch Deep Residual Network for Single Image Super-Resolution
## 3 Proposed Methods
### Training  
As described in Section 3.2, the feature extraction network takes the low-resolution image \(I^{LR}\) as input. Assuming that this part of network is model \(F\), we can get the output \(F(I^{LR})\), which is the input to the mapping network \(M\). Determined by the multi-branch structure, the output of each branch network \(M_{n},n=1,2,\cdots,N\) can be described as:  
\[M_{n}=g_{n}(g_{n-1}\cdots(g_{1}(F(I^{LR}),I^{LR}))) \tag{1}\]  
where the operator \(g\) denotes the function which represents each branch network. Since every branch network has different components, each \(g_{n},n=1,2\cdots,N\) represents different function expression. The reconstruction network \(R\) takes the \(M_{n}\) as input and complete the reconstruction of super-resolution images. Under the branch-supervision, the predictions \(R_{n},n=1,2,\cdots,N\) from each reconstruction network are:  
\[R_{n}=h_{n}(I^{LR},M_{n}) \tag{2}\]  
Same as the mapping network, the function \(h\) completes the reconstruction task of each recursive network. This process can be considered as the inverse operation of feature extraction network in a sense. Following our model, the final output \(I^{SR}\) is the weighted average of all intermediate predictions and original input \(I^{LR}\):  
\[I^{SR}=\sum_{n=1}^{N}w_{n}\cdot R_{n}+w_{0}\cdot I^{LR} \tag{3}\]  
Obviously, the \(w_{n}\) represents the weight of every intermediate prediction in reconstruction. Learning from training, these weights will directly determine the final reconstruction quality. In order to achieve superior performance, besides the ingenious network architecture, a loss function, which is as accurate as possible, is also necessary. Here, the training loss function, which can find optimal parameters for a proposed network, will be introduced. First, the difference between super-resolution image and high-resolution is the most intuitive expression. Combined with L2 loss, which is generally preferred since it minimizes the MSE, the part of loss function is defined as:  
Figure 2: The architecture of our proposed SR network (MRSR), which consists of three parts: Feature extraction, mapping, and reconstruction. The feature extraction is composed of multiple filters and residual blocks. The non-linear mapping between LR and SR adopt the multi-branch network structure and each branch is made up of residual blocks. In the reconstruction, the final output is restored from every branch output and the LR input with different weights. \[\mathcal{\epsilon}_{1}^{SR}=\frac{1}{WH}\sum_{i=1}^{W}\sum_{j=1}^{H}\left(I_{(i,j)}^ {SR}-I_{(i,j)}^{HR}\right)^{2} \tag{4}\]  
where \(W\) and \(H\) are the image size. The \(I^{SR}\) and \(I^{HR}\) represent the reconstructed image and reference image respectively. According to proposed model, every branch network needs to be supervised:  
\[\mathcal{\epsilon}_{2}^{SR}=\frac{1}{NWH}\sum_{n=1}^{N}\sum_{i=1}^{W}\sum_{j=1 }^{H}\left(R_{n_{(ij)}}-I_{(i,j)}^{HR}\right)^{2} \tag{5}\]  
Furthermore, based on the ideas of [PERSON] et al. [34] and [PERSON] et al. [35], a pre-trained 16 layer VGG network is adopted to compute the VGG loss \(I_{VGG}^{SR}\), which is closer to perceptual similarity. The \(I_{VGG}^{SR}\) is based on the ReLU activation layers of the VGG network, described as Zisserman [36]. Each activation layer in the VGG network will get different feature maps for the two different inputs of the reconstructed image \(I^{SR}\) and the truth image \(I^{HR}\). Then, we define the \(I_{VGG}^{SR}\) as the Euclidean distance between these feature maps:  
\[\mathcal{\epsilon}_{VGG}^{SR}=\frac{1}{XY}\sum_{i=1}^{X}\sum_{j=1}^{Y}\Bigl{|} \Bigl{|}\psi_{(i,j)}(I^{SR})-\psi_{(i,j)}(I^{HR})\Bigr{|}\Bigr{|}^{2} \tag{6}\]  
where \(\psi_{(i,j)}\) denotes the \(i\)-th feature map of \(j\)-th activation layer of the VGG network. \(X\) and \(Y\) are the dimensions of the feature maps.