# Deep Residual Network with Sparse Feedback for Image Restoration
## 3 Deep Residual Network with Sparse Feedback Loops  
In the design and application of neural network, the researchers are only required to focus on the input and output, the number of hidden layers, and the initial parameters. As the network depth gradually increases, the parameters of neural network are also difficult to tune. There are also no relevant theories to be presented on how to tune these parameters. Moreover, the updating of neuron parameters depends on the gradient; the more far away from the output layer, the more difficult for updating of neuron parameters. It will be invariant, or will change dramatically, which is called gradient disappearance or gradient explosion problem. Although the dropout strategy, or batch normalization, was adopted to reduce explosion problems, it still often happened. The more layers of neural network, the more obvious the gradient disappearance or explosion problem is.  
This question reminds me of the amplifier cascade problem in electronics. When connecting the circuit, the output signal is usually unstable. Single negative feedback or inter-stage negative feedback will generally be added to stabilize the output signal. In the amplifier circuit, the negative feedback is added to enhance the performance of the anti-noise and stability of the circuit, but the feedback will also reduce the amplification factor of the circuit. Usually, sparse feedback with a longer span is adopted to keep a tradeoff between the robustness and amplification factor. We believe that this situation is very similar to the shortcut in residual neural network. We try to adjust the shortcut in ResNet to a sparse longer connection. However, the idea of sparse longer feedback comes from the concept of negative feedback in the circuit; hence it is represented by \"feedback.\"  
However, the denoising results of ResNet are not better than that of the convolution neural network without feedback. For example, denoising convolutional neural network (DnCNN) [25] is good at removing Gaussian noise. Therefore, we combine the network structure of ResNet and DnCNN, and propose the deep residual network with sparse feedback for image restoration, which is called FbResNet. The proposed network structure is shown in Figure 1.  
There are only two feedbacks in Figure 1; one is a short feedback; the other is a long feedback. The short feedback is connected from layer 1 to layer 5. Because the output dimension of layer 1 and layer 5 is equal, the outputs of layer 1 and layer 5 can be directly added. The dimension of the input layer is the same as that of the last convolutional layer. Hence, the long feedback adds the input image to the output of the last convolutional layer, which can add a constraint to FbResNet to keep the most similarity between the input noisy image and output clean image.  
The mean squared error between the clean images and the degraded images can be defined as the loss function to learn the trainable parameters \(\Xi\) of FbResNet as follows:  
\[f(\Xi)=\frac{1}{N}\underset{i=1}{\overset{N}{\underset{i=1}{\sum}}}\|R(x_{i}, \Xi)-y_{i}\|^{2} \tag{1}\]  
where the input is a noisy image \(x_{i}\), and FbResNet aims to learn a mapping function \(R(x_{i})=y_{i}\), to predict the clean image. \(\{(x_{i},y_{i})\}\) represents \(N\) degraded-clean training image (patch) pairs; \(R\) represents the network structure, of which parameters \(\Xi\) require to be trained.