Compound Multi-branch Feature Fusion for Real Image Restoration
Authors: Chi-Mao Fan
Tsung-Jung Liu
Kuan-Hsien Liu
Venue: ICLR 2022 Submitted (2022)

Abstract: Image restoration is a challenging and ill-posed problem which also has been a long-standing issue. However, most of learning based restoration methods are proposed to target one degradation type which means they are lack of generalization. In this paper, we proposed a multi-branch restoration model inspired from the Human Visual System (i.e., Retinal Ganglion Cells) which can achieve multiple restoration tasks in a general framework. The experiments show that the proposed multi-branch architecture, called CMFNet, has competitive performance results on four datasets, including image dehazing, deraindrop, and deblurring, which are very common applications for autonomous cars. The source code and pretrained models of three restoration tasks are available at https://github.com/publish_after_accepting/CMFNet.

Under review as a conference paper at ICLR 2022
COMPOUND MULTI-BRANCH FEATURE FUSION FOR REAL IMAGE RESTORATION
Anonymous authors Paper under double-blind review
ABSTRACT
Image restoration is challenging and ill-posed problem which also has been long-standing issue. However, most of learning based restoration methods are proposed target one degradation type which means they are lack of gener- alization. In this paper, we proposed a from the Human Visual System (i.e., Retinal Ganglion Cells) which can achieve multiple restoration tasks in a general framework. The experiments show that the proposed multi-branch architecture, called CMFNet, has competitive perfor- mance results on four datasets, including image dehazing, deraindrop, and de- blurring, which are very common applications for autonomous cars. The source code and pretrained models of three restoration tasks are available at https: //github. cwtp after httpssaafaeaccccotngggooooo accepting/CMPNe.
restoration model inspired
INTRODUCTION
Image restoration is low-level vision task, and is usually the preprocessing step improve the performance of the high-level vision task, such as image classification, object detection or image segmentation. As its name implies, Image Restoration is meant restore an image free from degra- dation. Common degraded types include additive noise, blur, and JPEG blocking effect. However, with the rapid development of computer vision in recent years, the types of degradation that im- age restoration can handle are becoming more and more diverse, like super-resolution (Ledig et al.] 2017 Zhang et al.] (0188), single image dehazing (Zhang Patel, 2018: Qin et al. 2020), image deraining (Ren et al.
2019 Zhang et al. 2019a) even demoireing (uuan et al. 2019 2020
For decades, traditional image restoration was mostly based on external image prior (He et al. 2010 Shi et al., 2013) or dictionary learning (Hu et al.) 2010 [ong et al. (011)) which are generally called masel-based algorithms. Although these methods performed acceptably on the ill-posed problem as above mentioned, there are some shortages for model-based optimization methods, like time- consuming, computationally expensive and difficult to restore complex image textures. With the success of learning- -basd algorithms on both high-level (Krizhevsky al., 2012) and low-level vision tasks, convolution neural network (CNN) is dominant in the computer Vision field nowadays.
CNN-based methods not only outperform the model-based algorithms, but also have the state-of-the- art results in lots of image restoration tasks. Numerous networks and techniques of shaming-bassed algorithms, including encoder-decoders (Mao al. 20166. generative models (Ledig et al. 2017). residual learning (Zhang et al. 2017). and attention mechanisms (Chen et al. 2016) have been proposed enhance the performance of image restoration tasks. In addition different network architectures, the complex blocks (e.g.. residual block, residual dense block (RDB) (Zhang et al., 2018ch) or modules (Convolution Block Attention Module (CBAM) (Woo al.] 2018) and Dual Attention Units (DAU) (Zamir et al. 2020a b)) are presented to replace the naive convolutions.
We observed that most of restoration models only focus on one degradation type to restore Nah et al. 2017b] Zhang et al.) 2018c, Ren et al., 2019 Chen et al. 2020 Qin et al. 2020). The perfor- mances of these restoration models are impressive, but are not able to restore other degradation tasks. To put it another way, the generalization ability of these models has to be improved. Fortunately, all of the aforementioned restoration tasks have common final objective (i.e., converting an image from illegible and low-que low-quality to satisfactory and high-quality). Thus, proposed general framework inspired from the Human Visual System (HVS) which could conduct the restoration tasks, including deblurring, dehazing and deraining. The reason why focus these three restoration tasks is

---

Under review as a conference paper at ICLR 2022
because they usually oceur at the environment of autonomous cars. Hence, the proposed framework is practical and prospective in the future applications.
The proposed compound multi-branch feature fusion image restoration architecture, called CMFNet has several important characteristics. First, inspired from the early path of the HVS, we use three types of existing attention blocks, including channel attention, spatial attention, and pixel attention blocks to simulate the Retina Ganglion Cells (RGCs) independently. The main idea is to separate different attention features from stacking multiple complicated blocks multiple branches with simple block architecture. Second, we use the Supervised Attention Module (SAM) proposed by Waqas Zamir al. (202T) improve the performance. We also remove the supervised loss between the output images from SAM and ground truth images, because we think it would limit the learning of network. Thirdly, we propose mixed skip connection (MSC) to replace traditional residual connection with a learnable constant, which makes the residual learning more flexible on different restoration tasks. Finally, we optimize our CMFNet end-to-end by novel loss function, which provides more selections about the loss function rather than the common losses. All of these key components about our architecture will be described in Section 3 in detail.
Overall, our main contributions of this paper can be summarized as the following:
We propose restoration model CMFNet inspired by the RGCs which can handle multiple image restoration tasks, including image deblurring, dehazing and deraindrop with single framework.
We propose the skip connection with a learnable parameter named mixed residual mod- ule to integrate the branch information. The experiments proved that it is better than the monotonous residual pathway aggregation.
A new loss function which comprises two widely used evaluation metrics (i.e., Peak Signal- to-Noise Ratio (PSNR) and Structure Similarity (SSIM) Index) in restoration task is pro- posed.
We demonstrate the competitive results of our CMFNet in both synthetic and real-world datasets for three types of restoration tasks, especially the improvements of SSIM values. In other words, the performances of our model are closer the human sense.
RELATED WORK
With the rapid development of hardware (e.g., Graphics Processing Unit (GPU)), the deep learning methods become more and more mature, which make the computer vision prosperously grow in recent years. However, as aforementioned, most of methods perform well in specific restoration task. In this paper, we propose general architecture to solve deblurring, dehazing and deraining. Also, we are going to introduce the related works about RGCs and attention mechanism. Then, we will describe the representative methods for each task in detail.
Retina Ganglion Cells (RGCs). Human the final receiver of the images. The pictures we saw were passed from the complicated visual neural network, and the early receivers from the real world are RGCs, which is mainly composed of 3 types of cells: Parvocellular cells (P-cells), Koniocellular cells (K-cells) and Magnocellular cells (M-cells). They are sensitive to different external stimuli. First, P-cells are sensitive to the shape and color of images, and account for 80% of the RGCs. The second one is the K-cells, the smallest RGCs, as small as dust, which mainly respond to changes in color. The third one is M-cells, the biggest RGCs, and it only transmits the light/dark signal, and more sensitive at low spatial frequencies than high spatial frequencies. All of these cells have independent pathways toward the Lateral Geniculate Nucleus (LGN), Some computer vision researches such as Image Quality Assessment (IQA) (Chang et al. 2020) and Image Reconstruction (Zhang et al. 2020) are based on RGCs, too.
Attention Mechanism. Due to the success of the application on the high-level vision tasks, like object detection and image classification, attention mechanism also be used in low-level vision tasks (Anwar& & Barnes] (019).. Well known attention modules or blocks are mainly distinguished by the dimension of the generated mask, such as channel attention (Zhang al. 2018b), spatial
2

---

Under review as a conference paper at ICLR 2022
attention, pixel attention (Chen al. 2016) and combined attention blocks (Qin et al. 2020 Zamir let al. 2020b). Although the attention could enhance the performance in general, some researchers consider that attention will increase the inference and training time of the model (Liu et al. 2020),
and the benefit of using it is very small.
Image Dehazing, Image dehazing has the conventional model-based methods. For example, some researchers (Narasimhan & Nayar 2000; 2002) provided an important model to approximate the haze effect which is shown as: x 3 + A o (1 m)) where and x mean the degradative images and restored images, respectively. A and m are the global atmospheric light and medium transmission map, and @ represented as pixel-wing multiplication. In addition. He et al. (2010) pro- posed an image-prior method depends on the statistical results called Deep Channel Prior (DCP) and has the promising results. As for learning- methods, the techniques such as attention, feature fusion (Qin et al., 2020) and contrastive learning (Wu et al., (021) are widely used to ameliorate the single image dehaze performance. Moreover, they outperform the traditional prior-based image dehazing.
Image Deraining. For raining dataset, it is almost impossible to get the real-world rainy images until now, so most of experiments are based on synthetic images. And previous tradition deraining methods (Barnum et al., 2010 Bossu et al. 2011) almost all use linear-mapping transformations to restore the rainy images. However, these methods are not robust when the rainy imput has variations. Thus, the learning-basse -hasid model is here again Zhang et al.) 2019a Chen et al. 202T)! But we want to emphasize all of these methods are trained artificial rainy images. On the other hand, the de- raindrop restoration task which is the branch of image deraining can acquire the real-world dataset. Qian et al. (2018) used two pieces of exactly the same glasses to obtain the real-world raindrop dataset. Compared with the synthesized rain streaks dataset, the Raindrop dataset is closer to the images captured from cameras.
Image Deblurring. Because the real-world supervised blur data is difficult to acquire, most of traditional deblur methods, like deconvolution (Krahmer al. 2006 Carasso 200T). image prior (Joshi et al. 2010; Shi et al. 2013) are generally tested on synthesized images from J to x. which can be represented as: is the blur kernel or called convolution kernel, @ denotes the convolution operator and is additive noise. However, the model-based methods are not good at dealing with real-world blur such as hand- shake motion blur or defocus blur. On the contrary, CNN- based methods can handle real- world blur very well if we have the dataset of pair images (Nah et al., 20176). [Tao et al. (2018) proposed a multi-scale method based encoder-decoder recurrent network (SRN) which the first method integrating the Recursive Neural Networks (RNN) into deblur model. Some methods (Kupyn et al.) 2018 2019) based on Generative Adversarial Network (GAN) also get the competitive evaluation scores on real-world deblur results. Recently, multi-stage architecture network (Waqas Zamir et al. Chen et al. 202T) have state-of-the-arr results in the deblur restoration task.
& k + n. where è is the blurred image generated from clean image x,
2021
3
PROPOSED METHOD
In this section, we mainly introduce the multi-branch network CMFNet, and provide more explana- tions about the components we proposed in the following subsections.
3.1 CMFNET
The architecture of the proposed compound multi-branch feature fusion network (CMFNet) is illus- trated in Fig. [ CMFNet consists of three branches which simulate the P-cells, M-cells and K-cells after receiving signals from cone and rod cells. Each branch is based on the U-Net (Ronneberger al., 2015), and we use different attention blocks to replace naive convolutions in each branch. The output features of encoder- branch networks will enter the Residual Attention Module (RAM) which is obtained by removing the loss calculation from (Waqas Zamir et al., 2021). And we replace 1 convolutions in the original SAM with 3 x 3 convolutions (Chen et al.) 1021). Then, each RAM will generate two outputs (e.g., Fe and Ig in the upper branch as shown in Fig. D.. where Fc IR "xxWxX C denotes the feature map generated by the mask (M) obtained by passing the image Ic through sigmoid, and Iç E I # denotes the output image obtained by passing the degraded image through 3 x 3 convolution. The Ic, Ip and Ig from three branches will be
3

---

Under review as a conference paper at ICLR 2022
Channel attention
U-Net
Residual Attention Module
Fee
RAM
Mixed skip Connection
Concatenation
Aggregation
Pixel attention
U-Net
Prell
RAM
Dagraded (x')
Restored (x)
Spatial attention
10.TV
U-Net
RAM
1R
MSC
Figure 1: Proposed compound multi-brance feature fusion network (CMFNet) architecture. Each branch is based on 3 layers of U-Net with attention blocks in each scale.
fed into the Mixed Skip Connection (MSC) to obtain the residual image 7R. Finally, L and the concatenated fusion features (Fc, Fp, and Fs) will both go through 3 convolution to aggregate the restored image.
3.2 BRANCH NETWORK
The CMFNet comprises three branch networks which are based on the U-Nets, and each of branch network extracts shallow features by two
convolutions and one Parametric ReLU (PReLU), All of the branch networks have 3 layers of U-Ner architecture with 3 blocks at each scale as shown Fig. 2 The main difference among the three branch networks is the type of attention blocks used in the Netet We will introduce each branch network in detail as below:
P-cells network. We use the pixel attention block (PAB) (Zhao et al. 2020 to simulate the P-cells, which are sensitive to the color and shape (edge) of images. The PAB is shown in Fig. 4(b where generates the 3-D attention mask (M E IR xxXXxC without any pooling or sampling, which means the output feature map has local information. However, because the dimension of the mask is the largest among the three attention blocks, the computation time is the most. On the contrary, the PAB acquire the most accurate attention features. We observed that the PAB and P-cells have to-one correspondence: 1) the transmission speed of P-cells is the slowest since the pixel attention map has the most parameters, and the spatial resolution of P-cells is the biggest which related to the largest dimension of the pixel attention mask.
M-cells network. As for M-cells, is almost exactly the opposite of the P-cells. M-cells have the fastest transmission speed and the smallest spatial resolution, which can be explained by the channel attention block (CAB). The CAB (Fig. [Cc) is proposed by (Zhang al. 2018b) and is widely used on lot of restoration tasks (7amir et al.) 2020b] Zhao et al. 2020 Waqas Zamir et al. 2021). It uses global average pooling (GAP) to squeeze the input features from 3-dimension
I-dimension, and then generates the I-D attention mask (M RC). Because of the GAP, the output features have non-local (global) information, corresponding to the response to the luminance of images for the M-cells.
K-cells network. In either transmission speed or spatial resolution, K-cells are in the medium among the three. We simulate it the spatial attention block (SAB) illustrated in Fig, 4(d) which can generate the 2-D attention mask (M E HxW), The same as channel attention, spatial attention is also operated by the GAP
average pooling (MAP) to squeeze features to 2-dimension.

---

Under review as a conference paper at ICLR 2022
Because of the pooling, the output features have non-local (global) information. In other words, the K-cells mainly responds to changes in color which also belongs to the global information.
a(
Frarbrasch
ansemie
f
Sedkelleeet catout
Figure 2: Branch network architecture
Figure 3: Mixed Skip Connection (MSC).
3.3 MIXED SKIP CONNECTION
Residual connection first proposed by (He et al. 2016) in image recognition tasks. There countless deep learning models using it nowadays. In our CMFNet, we proposed a mixed skip connection (MSC) which integrates the output images from RAM of each branch. Fig. 3 is the framework of MSC, which has three input pathways and uses learnable constant a()) to optimize the whole architecture by the backward propagation. The main pathway (output image of Pixel branch network) multiplies by the parameter a@). and the others (output images of both Channel and Spatial branch networks) multiply by (1 a(0))/2. As result, the output residual image In. from MSC can be represented as:
IR MMCMMS,,,, (p, = ( ( aa(( lc + a()) 1 +
1 - a(()) 1s.
(1)
where Tc, Tp and Is are the output images from channel, pixel and spatial branch networks, respec- tively. a()) is learnable parameter from sigmoid activation function which is bounded from 0 to by parameter We can also think of it weighted average skip connection, and will prove the proposed MSC is effective in enhancing the performance by the ablation study in Section 4.6.
(a) Attention Blocks
(b) Pixel attention
(c) Channel attention
(d) Spatial attention
Figure 4: Attention blocks framework (a) and different attentions modules (b), (c) and (d).
3.4 LOSS FUNCTION
In the image restoration task, we usually used the Mean Absolute Error (MAE) Mean Square Error (MSE) as loss function to optimize the network. However, both L1 and L2 losses belong to pixel loss which did not consider the global information, so it is usually suffered from the over- smoothing problem. There were many loss functions proposed, such as perceptual loss or even compound loss that combines several loss functions together to solve the problem. To focus the tradeoff between human sense and metric scores, we propose the new loss function to optimize our CMFNet end-to-end shown below:
Liotal (X, aLedge X,Y),
where e XX denotes the degradation images, B is the batch size of training data, C is the number of feature channels, H and W are the size of images. And Y represents the ground-truth
5

---

Under review as a conference paper at ICLR 2022
data. The total loss function is composed of two terms, Lps and Ledge. The Lps is the loss which is comprised of Peak Signal-to-Ner Ratio (PSNR) and Structure Similarity (SSIM) Index as follows:
(-SSIM(X,Y) Cps PSNR(X,
(3)
where the parameter w is a small constant which is empirically set to 0.005. It is used to stabilize the value which can prevent occurrence of Not a Number (NaN) or infinity when PSNR extremely small. Our proposed PS loss has two advantages. First, it is suitable for different image restoration tasks because it adopts two standard metrics as the loss function. The other advantage is PS loss does not need additional parameters compared simply combining different loss functions in several terms with weighting parameters. In other words, it will save time in finding the optimal parameter values by performing many experiments. The second term Ledge is the edge loss which refers to (Jiang et al. 2020) and can be represented as:
Ledge VIIx(X) A( e2,
(4)
where means the Laplacian operator. The constants a in Eq. (2) and Eq. (4) are set 0.05 and 10-3 the same as (Waqas Zamir et al. 2021).
EXPERIMENTS
4.1 EXPERIMENT SETUP
Implementation Details. Our CMFNet an end-to-end trainable model without any pretrained
networks and implemented by PyTorch 1.8.0 with one NVIDIA GTX 1080Ti GPU. We train separate models for three different tasks and the feature channel numbers are all set to 96. The models are trained using Adam optimizer with initial learning rate of 2 10-4, and decreased to 10" by cosine annealing strategy. Because of the limitation of the hardware equipment, we train our network 256 x 256 patches with batch size of only for 4 10 Random flip and rotation are applied as data augmentation.
Evaluation Metrics. As mentioned in Section 3.4, PSNR is widely used in evaluating the quality of restored images, However, it is sometimes not close to real human perception when PSNR metric is high. Therefore, we also consider the SSIM index in our experiments. which can represent human perceptual feelings more accurately. Note that both PSNR and SSIM values are all the higher the better, and the unit of PSNR is decibel (dB). The training datasets for each restoration task are described as follows.
4.2 EXPERIMENT DATASETS
Image Dehazing. Since we want our proposed CMFNet could handle the real-world haze. we use the I-Haze (Ancuti al., 2018a) and O-Haze (Ancuti al. 2018b) which contain 30 and 45 real-world high definition hazy pair images for training. respectively. And we test the proposed CMFNet on real-world haze dataset D-haze (Ancuti et al. 2019) that contains 55 pair images with the resolution of 1600 x 1200.
Image Deraindrop. We use the DeRainDrop dataset (Qian al. 2018) for training and testing. provides 861 image pairs for training and has two testing datasets (i.e., testA and testB). TestA is subset of testB, which contains 58 pairs of good aligned images. TestB has 249 image pairs with a small portion of images which are not perfectly aligned.
Image Deblurring. For image deblurring, we train the images on the synthesized GoPro dataset Nah et al. 2017a) most deblur methods Waqas Zamir et al. (2021); Chen et al. (2021) did. GoPro dataset has 3.124 pair blurred images with the size of 1280 x 720, including 2,013 images for training and 1,111 blurred/sharp images for testing. As in Waqas Zamir et al. (2021) also use the model pretrained on GoPro to test the HIDE (Shen et al. 2019) dataset which contains 2,025
test images by human-aware motion blur.

---

Under review as a conference paper at ICLR 2022
4.3
IMAGE DEHAZING PERFORMANCE
For image dehazing task, we compare the proposed method with the prior- based method (He et al.
2010), learning-basse methods (Cai al. 2016 Li al. 2017) Liu et al. 2019a; Hong al. 2020 Qin et al. 2020; Dong et al. 2020), and the contrastive learning method (Wu et al.) 2021 Table [| shows that our method achieves the best SSIM score (0.533) on D-Haze dataset (Ancuti et al.
2019), which means restored images are closer to human perception. It should be noted that
other proposed dehazing methods could not restore the real-world hazy images very well as shown in Fig.
Table 1: Image Dehazing results on Dense- Haze (D-Haze) dataset (Ancuti al. (2019). Best and second best scores are highlighted and underline, respectively.
Table 2:
Image Deraindrop results on DeRain-
Drop test dataset (Qian et al., 2018). Best and second best scores are highlighted and underlined, respectively.
D D-Haze Ancuti et al.
2019
DeRainDrop (Qian et al.
2018
Method
PSNR
SSIM
Method
PSNR
SSIM
DCP (He et al.
2010
10.06
0.386
Eigen13 Eigen et al
2013
23.74
0.799
Net-Net (Li et al. (2017)
13.14
0.414
Pix2Pix (Isola ct al. 2017)
27.73
0.876
GridDehazeNet (Luu et
2019a
13.31
0.368
DeRaindrop (w/o GAN)
29.25
0.785
DehazeNet (Cai
2016
13.84
0,425
D-DAM (Zhang et al.
2021
30.63
0.927
KDDN (Hong ct al.
202(1)
14.28
0.407
A"Net (Lin et al. 2020
30.79
0,926
FFA-Net (Qin et al.
2020)
14.39
0.452
BPP (Michelin el al
2021
30.85
0,918
MSBDN (Dong
2020
15.37
0.486
DuRN (Liu 2019b
31.24
0.926
AECR-NeT Wu
2021
15.80
0.466
DeRaindrop QQan
2018
31.57
0.902
CMFNet (Ours)
14.46
0,533
CMFNet (Ours)
31.49
0.933
11.673/0.5477 Hazy
10.074/0.5088 DCP
12.819/0.5749 DehazeNet
13.024/0.5708 GridDehazeNet
12.207/0.5565
FFA-Net
15.494/0.5841 AOD-Net
16.477/0.5827 MSBDN
18.846/0.5873
18.721/0.6949
PSNR/SSIM GT
AECR-Net
CMENet (Ours)
Figure 5: Visual comparisons for image dehazing on the D-Haze dataset (Ancuti et al
2019
4.4 IMAGE DERAINING PERFORMANCE
In T we report the PSNR/SSIM scores of deraindrop task compared with several deraindrop methods on the DeRainDrop testA dataset. We follow QQan et al. (2018)'s quantitative evaluation which transforms the testing images from RGB YCbCr color space to calculate PSNR and SSIM metrics. Our CMFNet achieves the best SSIM score (0.933) and second best PSNR score (31.49 dB). Fig. 6 illlustrates visual results on DeRainDrop testB images. We are not able generate the deraindrop images of both BPP(Miccelimi et al. (02TI) and D-DAM (Zhang et al. 2021) methods because the source codes not available yet when submit paper. However, it still shows

---

Under review as a conference paper at ICLR 2022
method effectively removes raindrops and restored images visually closer the ground-truth than the other models.
18. 13/0.99999 Raindrop
17.05/0.580 Eigen
15.2440.831 DeRainDrop
13/0.699 Raindrop Image
25.61/0.846 DuRN
26.85/0.875 CMFNet (Ours)
PSNR/SSIM GT
21.54/0.788 Raindrop
18.686/0.686 Eigen
27.89/0.874 DeRainDrop
28.14 14/0.888
28.83/0.895 CMFNet (Ours)
PSNR/SSIM
DuRN
GT
Figure 6: Visual comparisons for image deraindrop on the DeRainDrop dataset (Qian al. 2018
4.5 IMAGE DEBLURRING PERFORMANCE
As for deblurring, Table shows the results with both synthetic and real-world dataset. In Table 3 although our CMFNet does not achieve the best performance, the evaluation scores are still accept- able (ranked in the middle). means the proposed model could handle the degradations occurred on the images taken by cameras of autonomous cars. Due to page limit, visual comparison of deblurring results are provided in the appendix.
Table 3: Image Deblurring results on GoPro Nah et al.
2017 HIDE Shen et al. 2019 datasets.
GoPro Nah et al. PSNR
2017a
(hen et al. PSNR
HIDE
2019
Methods
SSIM
SSIM
Xu et al.
2013
21.00
0.741
DeblurG Kupyn et al.
2018
28.70
0.858
24.51
0.871
Nah al.
2017aa
29.08
0.914
25.73
0.874
DeblurGA N-v2 (Kupyn
2019
29.55
0.934
26.61
0.875
Zhang et al.
(2018a)
29.19
0.931
SRN (Tao al. 2018
30.26
0.934
28.36
0,915
DMPHIN Zhang
2019b 9b
31.20
0.940
29.09
0.924
MPRNet Waqas Zamir al. 2021
32.66
0.959
30.96
0.939
CMFNet (Ours)
30,54
0.913
28.23
0.882
4.6 ABLATION STUDY
To demonstrate the contribution of each component of the proposed CMFNet, we present ablation studies to analyze different elements, including multi- branch, proposed PS loss and MSC. The abla- tion studies are experimented on the GoPro dataset (Nah et al.) 2017a) with training image patches of size 128 128 for x 10a iterations. The whole ablation study results are shown in Table
8

---

Under review as a conference paper at ICLR 2022
Number of branches. We set model from single branch to triple branch framework, which proves that the multi-branch architecture is effective in improving the performance (from 28.00 dB to 28.88 dB).
Attention blocks of branch networks. Because each U-Net branch network is comprised of differ- attention blocks to simulate the RGCs, we present the experiments of different attention blocks, and validate the pixel attention block has better performance (28.63 dB) than channel and spatial attention blocks (28.00 dB and 28.26 dB, respectively).
Choices of loss functions. As for the choice of the loss functions, we provide the results of widely used MAE (L. loss) and proposed PS loss. We could observe that the result with PS loss has a maximum growth in SSIM metrics (+0.008) without any additional parameters. It shows the proposed PS loss is good restoring perceptually
images.
Skip connection. The last component is the skip connection. We can observe the CMFNet with proposed MSC yields the best performance (29.16 dB), which also validates the effectiveness of our MSC design. In the process of our experiments, the gradient exploding problem occurs, so we decrease the initial learning rate from x 10-4 10-4. However, the network with skip connection in the last two rows can successfully be trained with the initial learning rate of 2 x 10-1
Table 4: Ablation study results of CMFNet. The last two columns are the average PSNR and SSIM of GoPro testing images.
# of Branch
Attention block
Loss
SC
PSNR
SSIM
Channel attention (CA)
loss
X
28.00
0.858
Spatial attention (SA)
L1 loss
X
28.26
0.859
Pixel attention (PA)
LI loss
X
28.63
0.868
CA+ SA
LI loss
X
28.58
0.873
CA+ PA
LI loss
X
28.76
0,870
SA+ PA
loss
X
28.81
0.872
3
CA+ PA+SA
LI loss
X
28.88
0.873
3
CA+ PA+SA
PS loss
X
28.90
0.881
CA+ PA+SA
PS loss
(ASC)
29.08
0.886
CA+ PA+SA
PS loss
(MSC)
29.16
0.887
5 CONCLUSION
In this paper, we proposed general framework for image restoration which mimics the multi-path RGCs, called CMFNet. It could restore the degradation types, including blur, haze and rain, which usually occur on the images taking by the camera of self-driving. Besides, we proposed a novel loss function for general restoration tasks and the MSC to replace the traditional skip connection. Both of them are proved that they are effective in improving the restoration performance. Furthermore, our model achieves significant SSIM gain for four different datasets with different degradation types, which demonstrates the perceptually-faithe more different restoration tasks, such low-light image enhancement, streak rain removal, and the defocus blur. Additionally, we will also focus on balancing the trade off between the accuracy and computational efficiency.
restored images. In the future, we are going to attempt
REFERENCES
Codruta O. Ancuti, Cosmin Ancuti, Radu Timofte, and Christophe De Vleeschouwer. I-haze: dehazing benchmark with real hazy and haze-free indoor images. arXiv:/8000 000..844.00000 80440/0///// 2018a.
Codruta O. Ancuti, Cosmin Ancuti, Radu Timofte, and Christophe De Vleeschouwer. O-haze: dehazing benchmark with real hazy and haze-free outdoor images. In IEEE Conference on Computer Vision and Pattern Recognition, NTIRE Workshop, NTIRE CVPR 18, 2018b

---

Under review as a conference paper at ICLR 2022
Codruta O Ancuti, Cosmin Ancuti, Mateu Sbert, and Radu Timofte. Dense-haze: A benchmark for image dehazing with dense-haze and haze-free images. In 2019 IEEE international conference image processing (ICIP), pp. 1014-1018. IEEE, 2019.
Saeed Anwar and Nick Barnes. Real image denoising with feature attention. In Proceedings of the IEEE/CVF International Conference Computer Viston. pp. 3155-3164, 2019.
Peter C Barnum, Srinivasa Narasimhan, and Takeo Kanade. Analysis rain and snow in frequency space. International journal of computer vision, 86(2):25-2274, 2010.
Jérémie Bossu, Nicolas Hautiere, and Jean- -ean-Philippe Tarel. Rain or snow detection in image se- quences through use of histogram of orientation of streaks. International journal of computer vision, 3363)3348-367, 2011.
Bolun Cai, Xiangmin Xu, Kui Jia, Chunmei Qing, and Dacheng Tao. Dehazenet: An end-to-end system for single image haze removal. IEEE Transactions on Image Processing, 25(11):5187, 5198, 2016.
Alfred Carasso. Direct blind deconvolution, SIAM Journal on Applied Mathematics, 61(6): 2007, 2001.
Yongli Chang, Sumei Li, Li Ma, and Jie Jin. Stereo image quality assessment considering the asym- metry statistical information in early visual pathway. In 2020 IEEE International Conference on Visual Communications and Image Processing (VCIP), pp. 342-345, IEEE, 2020.
Staag-Chich Chen, Yi Yang. Jiang Wang. Wei Xu, and Alan Yuille. Attention to scale: Scale- aware semantic image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3640-3649, 2016.
Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. Hinet: Half instance normal- ization network for image restoration. Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 182-192, 2021.
Zailiang Chen, Ziyang Zeng, Hailan Shen, Xianxian Zheng. Peishan Dai, and Pingbo Ouyang. Dn- gan: Denoising generative adversarial networks for speckle noise reduction in optical coherence tomography images. Biomedical Signal Processing and Control, 55:101632, 2020.
Hang Dong, Jinshan Pan, Lei Xiang, Zhe Hu, Xinyi Zhang, Fei Wang, and Ming- Hsuan Yang. Multi-scale boosted dehazing network with dense feature fusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 2157-2167, 2020.
Weisheng Dong, Xin Li, Lei Zhang, and Guangming Shi. Sparsity-based image denoising via dic- tionary learning and structural clustering. In CVPR 2011, pp. 457-464. IEEE, 2011.
David Eigen, Dilip Krishnan, and Rob Fergus. Restoring an image taken through a window covered with dirt or rain. In Proceedings of the IEEE international conference on computer vision, pp. 633-640, 2013.
Kaiming He, Jian Sun, and Xiaoou Tang, Single image haze removal using dark channel prior. IEEE transactions on pattern analysis and machine intelligence, 33(12):2341-2333 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ming Hong, Yuan Xie, Cuihua Li, and Yanyun Qu. Distilling image dehazing with heterogeneous task imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Zhe Hu, Jia-Bin Huang, and Ming Hsuan Yang. Single image deblurring with adaptive dictionary learning. In 2010 IEEE International Conference on Image Processing, pp. 1179-1172. IEEÉ, 2010.
10

---

Under review as a conference paper at ICLR 2022
Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-ime -mmge translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Kui Jiang, Zhongyuan Wang. Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, and Junjun Jiang. Multi-scale progressive fusion network for single image deraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8346-8355 2020.
Neel Joshi, Sing Bing Kang, C Lawrence Zitnick, and Richard Szeliski. Image deblurring using inertial measurement sensors. ACM Transactions on Graphics (TOG), 29(4): 2010.
Felix Krahmer, Youzuo Lin, Bonnie McAdoo, Katharine Ott, Jiakou Wang, David Widemann, and Brendt Wohlberg. Blind image deconvolution: Motion blur estimation. 2006.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiñí Matas. Deblur- gan: Blind motion deblurring using conditional adversarial networks. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 8183-8192, 2018.
Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. Deblurgan-v2: Deblurring (orders-of-magnin) faster and better. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 8878-8887, 2019.
Christian Ledig Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681 16900, 2017.
Boyi Li, Xiulian Peng, Zhangyang Wang, Jizheng Xu, and Dan Feng. Aod-net: All-in-one dehazing network. Proceedings of the IEEE international conference on computer vision, pp. 4770- 4778, 2017.
Huangxing Lin, Changxing Jing, Yue Huang, and Xinghao Ding. net: Adjacent aggregation networks for image raindrop removal. IEEE Access, 8:60769-60779, 2020.
Shuai Liu, Chenghua Li, Nan Nan, Ziyao Zong, and Ruixia Song. Mmdm: Multi-frame and multi- scale for image demoireing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 434-435, 2020.
Xiaohong Liu, Yongrui Ma, Zhihao Shi, and Jun Chen. Griddehazenet: Attention-batt multi- scale network for image dehazing. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7314-7323, 2019a.
Xing Liu, Masanori Suganuma, Zhun Sun, and Takayuki Okatani. Dual residual networks lever- aging the potential of paired operations for image restoration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7007-7016, 2019b.
Xiao-Jiao Mao, Chunhua Shen, and Yu-Bin Yang. Image restoration using convolutional auto- encoders with symmetric skip connections. arXiv preprint arXiv:1606.08921, 2016.
Pablo Navarrete Michelini, Hanwen Liu, Yunhua Lu, and Xingqun Jiang. Back-projectton pipeline. In 2021 IEEE International Conference on Image Processing (ICIP),pp. 1949-1953. IEEE, 2021.
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3883-3891, 2017a.
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural net- work for dynamic scene deblurring. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 201 7b.
11

---

Under review as a conference paper at ICLR 2022
Srinivasa G Narasimhan and Shree K Nayar. Chromatic framework for vision in bad weather. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PRO0662), volume 1, pp. 598-605. IEEE, 2000.
Srinivasa Narasimhan and Shree K Nayar. Vision and the atmosphere. International journal of computer vision, 48(3):233-255 2002.
Rui Qian, Robby T Tan, Wenhan Yang. Jiajun Su, and Jiaying Liu, Attentive generative adversarial network for raindrop removal from single image. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2482-2491, 2018.
Xu Qin, Zhilin Wang, Yuanchao Bai, Xiaodong Xie, and Huizhu Jia. Ffa-net: Feature fusion at- tention network for single image dehazing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11908-11915, 2020.
Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3937-3946, 2019.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Ziyi Shen, Wenguan Wang. Xiankai Lu, Jianbing Shen, Haibin Ling, Tingfa Xu, and Ling Shao. Human-aware motion deblurring. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 5572-5581, 2019.
Ming-zhu shin--zhuBoe Shi, Ting-fa Xu, Liang Feng, Jiong Liang, and Kun Zhang. Single image deblurring using novel image prior constraints. Optik, 124(20):4429-4333 2013.
Xin Tao, Hongyun Gao, Xiaoyong Shen, Jue Wang. and Jiaya Jia. Scale-recurrent network for deep image deblurring. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8174-8182, 2018
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. arXiv e-prints, pp.
arXiv-2102, 2021.
Sanghyun Woo, Jongchan Park, Joon- Yoon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pp. 3- 2018
Haiyan Wu, Yanyun Qu, Shaohui Lin, Jian Zhou, Ruizhi Qiao, Zhizhong Zhang, Yuan Xie, and Lizhuang Ma. Contrastive learning for compact single image dehazing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10551-10560, 2021.
Li Xu, Shicheng Zheng, and Jiaya Jia. Unnatural 10 sparse representation for natural image deblur- ring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1107-1114, 2013.
Shanxin Yuan, Radu Timofte, Gregory Slabaugh, and Aleš Leonardis. Aim 2019 challenge on image demoireing: Dataset and study. In 2019 IEEE/CVF International Conference Computer Vision Workshop (ICCVW), pp.
IEEE, 2019.
Shanxin Yuan, Radu Timofte, Ales Leonardis, and Gregory Slabaugh. Ntire 2020 challenge im- age demoireing: Methods and results. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 460-461, 2020.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Cycleisp: Real image restoration via improved data synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2696-2705, 2020a.
12

---

Under review as a conference paper at ICLR 2022
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Learning enriched features for real image restoration and enhance- ment. In Computer Vision-ECC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pp. 492-511. Springer, 2020b.
He Zhang and Vishal M Patel. Densely connected pyramid dehazing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3194-3203, 2018.
He Zhang, Vishwanath Sindagi, and Vishal M Patel. Image de-raining using conditional generative adversarial network. IEEE transactions circuits and systems for video technology, 30(11); I)): 3943-3956, 2019a.
Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz. Deep stacked hierarchical multi- patch network for image deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5978-5986, 2019b.
Jiawei Zhang, Jinshan Pan, Jimmy Ren, Yibing Song, Linchao Bao, Rynson WH Lau, and Ming- Hsuan Yang. Dynamic scene deblurring using spatially variant recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2521-
2529, 2018a.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep enn for image denoising. IEEE transactions image processing, 26 (7):3142-3155) 2017.
Kaihao Zhang, Dongxu Li, Wenhan Luo, Wenqi Ren, Lin Ma, and Hongdong Li. Dual attention- in-attention model for joint rain streak and raindrop removal. arXiv preprint arXiv:2103.0700 2021.
Yichen Zhang, Shanshan Jia, Yajing Zheng, Zhaofei Yu, Yonghong Tian, Siwei Ma, Tiejun Huang, and Jian K Liu. Reconstruction of natural visual scenes from neural spikes with deep neural networks. Neural Networks, 125:19-30, 90000000000000 2020.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. Proceedings of the European conference on computer vision (ECCV), pp. 286-301, 2018b.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 2472-2481, 2018c.
Hengyuan Zhao, Xiangtao Kong, Jingwen He, Yu Qiao, and Chao Dong. Efficient image super- resolution using pixel attention. In European Conference on Computer Vision, pp. 56-72. Springer, 2020.
A APPENDIX
A.1
DEBLUR VISUAL PERFORMANCE
show the visual deblur performances on the GoPro, HIDE datasets by our proposed CMFNet. Although our average PSNR and SSIM are not the best, still has the competitive perfor- mance in deblurring.
Figs.
7 8
A.2 DEBLUR DISCUSSION
Fig. reportt our training loss and validation performance for deblur training process. We can see that the loss does not converge yet, and the validation deblurring model for 10 days by one 1080Ti GPU, and just stop at 94 epochs. In the future, we will keep training till the loss is convergent in order obtain the best performance for the deblurring lask.
(PSNR) is also growing. We train
13

---

Under review as a conference paper at ICLR 2022
24.050/0.7170 Blurry Image
31.278/0.9109 CMFNet (Ours)
PSNR/SSIM Ground Truth
24.193/0 Blurry Image
26.778/0.8979 CMFNet (Ours)
PSNR/SSIM Ground Truth
19.586/0.5793 Blurry Image
25.277/0.8297 CMFNet (Ours)
PSNR/SSIM Ground Truth
Figure 7: Visual performances for image deblurring the GoPro dataset.
19.629/0.6179 Blurry Image
24,89 88869.88896
PSNR/SSIM Ground Truth
CMFNet (Ours)
18.244/0.5428 Blurry Image
22.693/0.8088 CMFNet (Ours)
PSNR/SSIM Ground Truth
Figure 8: Visual performances for image deblurring on the HIDE dataset.
14

---

Under review as a conference paper at ICLR 2022
loss
PSNR
tag:
tag: val/PSNR
30.6
11.5
30.2
9.5
8.5
29.4
7.5
20
40
60
80
10
20
30
40
50
60
80
90
Training loss (PS loss)
Validation score (PSNR)
Figure 9: Training loss and validation curve for deblurring.
15


Reviews:
[{"review_id": "YvLxqeDuv55", "reviewer": "Reviewer_MkTj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\n\nWhile most image restoration methods employ one or two attention types, this method combines 3 of them. The proposed loss function ($\\mathcal{L}_{ps}$) seems to be more effective than L1 loss.\n\n[Weaknesses]\n\n1. Motivation and the contributions\n\nThe authors claim that the proposed method is inspired by the human visual system, the composition of retinal ganglion cells, however, I don’t agree.\nThe used attention modules are already present and have been widely studied in previous works and no such works claim the biological connection.\nWhile the authors may claim that combining them is novel, I don’t find the connection between the proposed MSC module for aggregation and the biological systems.\n\nThe authors claim that the proposed M-cells network and the K-cells network each handle the luminance and the color information but there is no supporting proof or analysis. As the attention module operates on feature space, such an interpretation is hard to build. I believe this is a jump of logic without sufficient grounds.\n\n2. Mixed Skip Connection\n\nMSC module produces a weight $a$ and puts the weight on $I_C$, $I_P$, $I_S$ with $(1-a)/2$, $a$, $(1-a)/2$.\nI can understand that the weights are designed to satisfy a sum-to-one constraint. However, I don’t understand why the weights for $I_C$ and$ I_S$ are set equal and why only the weight for $I_P$ differs from them.\n\n\n3. Related Works\n\nFor image dehazing, He et al. (2010) proposed **dark channel prior**, not deep channel prior.\n\nIn image deraining section, I suggest the authors maintain a formal tone rather than using colloquial language.\n\n4. References\n\nOn page 1 3rd paragraph, (Chen et al., 2016) is not an image restoration paper. It is an image semantic segmentation paper. \n\nNah et al., 2017a and 2017b are duplicated.\n\nPlease unify the bibliography format \n(Waqas Zamir et al., 2021) -> (Zamir et al., 2021)\n\n(Zhao et al., ECCV 2020) is not correct. It was published in ECCV 2020 **Workshop**.\nhttps://link.springer.com/chapter/10.1007/978-3-030-67070-2_3\n\n5. Task Generalization\n\nIn the abstract, the authors claim that the previous method lack generalization to multiple degradation types. However, the proposed method does not lead to task generalization. It seems that the authors trained the model independently to each task which could have been done by other methods, too. This is a misleading claim.\n\n6. Experimental results\n\nThe ablation study shown in Table 4 is not fair.\nBy adding blocks, the models would have increased numbers of parameters. The results are not beyond expectation.", "summary_of_the_paper": "This paper proposes an attention-based image restoration neural network architecture. The authors employ 3 types of attention methods in parallel: pixel attention, channel attention, and spatial attention. The outputs from the 3 attention-based UNet modules are aggregated by another module, mixed skip connection (MSC). The proposed method is applied to image dehazing, deraining, and deblurring. To train the model, the authors proposed a loss function using PSNR and SSIM.\n\n", "summary_of_the_review": "Considering the strengths and weaknesses, the contributions are minor and the authors’ interpretation of the proposed method seems void. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YeRwvZI3oKJ", "reviewer": "Reviewer_WqhR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well structured, written succinctly and presents an interesting idea vaguely inspired by the human visual system. The drawn comparison between P-, M- and K- cells of retina ganglion and attention mechanisms is intriguing. The paper also includes insightful observations and thorough prior work overview. The proposed model shows impressive qualitative results of image restoration, significantly outperforming related works visually (Figure 5 and 6). \n\nHowever, there are also a few concerns:\n1. Quantitative evaluation of the image restoration performance is based only on two metrics, PSNR and SSIM. While on it’s own those metrics are valid and representative, the problem arises from the fact that PSNR and SSIM are also directly used in a loss function for the model (PS loss term, section 3.4). Essentially, during training the model is optimized on metrics that are used to evaluate the model.\nThat does not invalidate the proposed approach or loss function, but highlights that evaluation requires more thought. I strongly suggest adding additional independent image quality assessment metrics, for example MAE, MSE or at least modified SSIM metrics: MSSIM, FSIM, IW-SSIM.\n\n2. Paper does not include any mention of important performance details, such as number of parameters, memory usage or training stability. For example, Table 4 (ablation study results) would strongly benefit from the first two, while section 3.4, describing novel loss function, would benefit from information about the training process.\n\n3. Few basic model design decisions in the paper are not backed by anything other than intuition. One of the most noticeable examples is in the description of novel Residual Attention Module block, adapted from Supervised Attention Module by removing loss function: \n\n    >  “We also remove the supervised loss between the output images from SAM and ground truth images, because we think it would limit the learning of network.”\n\n    I believe additional argumentation or at least some empirical backing can improve the clarity and explain motivation behind such design decisions.\n", "summary_of_the_paper": "The paper “Compound multi-branch feature fusion for real image restoration” proposes CMFNet, an image restoration model inspired by retina ganglion cells (RGC) of the human visual system. Namely, authors draw comparisons between parts of RGC and attention mechanisms: P-, M-, K-cells and pixel, channel, spatial attention respectively. CMFNet infers the attentions in parallel branches and fuses them by using proposed Residual Attention Modules and Mixed Skip Connection blocks to obtain a restored image. The model is trained by a novel loss function that integrates PSNR and SSIM image quality metrics. The experiments described in the paper show competitive results.", "summary_of_the_review": "The paper proposes an intriguing model inspired by the human visual system and shows impressive qualitative results. At the same time, quantitative evaluation definitely requires more work. The paper would benefit from the addition of practical details about the model and more explanation about motivation behind some design decisions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XZ2FJHtni9K", "reviewer": "Reviewer_eNZ2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The idea to handle multiple restoration tasks in one model is meaningful and reasonable.\n2. The way of using different kind of attention module in parallel branches is interesting and new.\n3. The paper writing is clear and easy to follow.\n\n\nWeaknesses\n1. The idea of using parallel branches for each kind of attention module to mimic HVS seems interesting. \n    However, the connection between P/M/K-cell and the 3 kinds of attention module is not very convincing.\n    The explanation in Sec 3.2 gives abstract analysis. However, no proof and visualizations are given.\n    It would be better to show and visualize the actual feature maps and the effect of different branches.\n\n2. The ablation study part is not very fair.\n     In Sec 4.6, the experiment 'number of branch' compares model with different trainable parameters. \n     The three branch model is more complex than one branch model, which may be the reason for performance gain.\n\n3. My main concern is about the novelty part. The whole network seems a combination of existing modules, without a clear clue why they are suitable for restoration tasks?\n    What are the specific characteristics for restoration tasks? Why the designed network is suitable? And how to visualize the benefits of intermediate results?\n\n4. In Sec 4.2 and 4.3, I wonder does the model train on 3 datasets simoutanously? Or the authors train 3 model for different tasks each?\n    Actually, there already exists method to handle real degradation simoutanously. Please at least cite or compare:\n[1] Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data, ICCVW 2021.\n[2] ESRGAN: Enhanced super-resolution generative adversarial networks, ECCVW 2021.\n\n5. The authors mentioned many times that they can handle real images.\n    However, they only use existing dataset for training and testing. \n    Even if the proposed method can really handle real degradation, the contribution comes from previous works.\n    The authors do not explain why their model can do this.\n\n6. Several typos and mistakes. For example, in Eq (2), the authors say X denotes degradation images. Is X denoted for output images?\n\n7. Some of the statements about previous methods are not accurate. \n    a. In Sec 2 'Image Dehazing', He et al.'s work is a very famous and foundamental work in that area. And it should be 'dark channel prior'.\n\n    b. In Sec 2 'Image Deblurring', they said those methods are `generally tested on synthesized images`. However, that is not correct.\n    To my knowledge, \n    [1] Two-Phase Kernel Estimation for Robust Motion Deblurring, ECCV 2010\n    [2] Removing camera shake from a single image, SIGGRAPH 2006\n    [3] Fast Motion Deblurring, SIGGRAPH 2009\n    [4] Non-uniform Deblurring for Shaken Images, CVPR 2010\n   They are all very famous early works and they all test on real captured photos.\n\n   On the contrary, this paper only show results on testing dataset, and do not compare photos that from a different source from training dataset.\n\n\n", "summary_of_the_paper": "This paper proposes a new model CMFNet to handle multiple image restoration tasks. More specifically, they introduce a new skip connection scheme, a new loss function and demonstrate comperitive results on 3 tasks: deblurring, deraining and dehazing.", "summary_of_the_review": "This paper proposes a new model CMFNet to handle multiple image restoration tasks. The results seem promising on several existing datasets. However, the proposed framework and modules seems similar to widely used techniques. And there is no clear and solid explanations for the design choice.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LUoJMxwyFs9", "reviewer": "Reviewer_sRLq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. A compound multi-branch feature fusion architecture is proposed to simulate the complex human visual system for better restoration.\n2. This paper tends to improve the generalization ability of a single restoration model for different degradation types.\n\nWeaknesses:\n1. Although the motivation to improve the model generalization ability is good, this paper does not clearly explain why the proposed architecture could actually do this. The authors state that they propose a multi-branch feature fusion network to simulate the human visual system, but the proposed architecture is by nature a simple combination of three existing attention blocks. And the authors relate these blocks with some types of cells in the human visual system. However, the so-called generalization of the proposed architecture is mainly derived from different training data for various tasks.\n2. In Table 4, what does ASC denote? Please explain it in both the table caption and main text. To better understand the effect of the mixed skip connection, it would be better to show the values of a(theta) in the training process.\n3. Since this paper states that their performance is closer to the human sense and they already include both PSNR and SSIM metrics in their loss function, it would be more convincing to use some other image quality assessment methods, e.g., NIQE [Mittal et al., Making a completely blind image quality analyzer. IEEE Signal Processing Letters 2012], LPIPS [Zhang et al., The unreasonable effectiveness of deep features as a perceptual metric. CVPR 2018], etc.\n4. According to Section 3.1, each RAM in Figure 1 should have another input of the degraded image.\n5. To verify the effectiveness of the proposed method, it would be better to compare with [Chen et al., Pre-Trained Image Processing Transformer, CVPR 2021], which presents an architecture that can handle various image processing tasks. It also evaluates on image deraining.\n6. The statement that the supervised loss in SAM will limit the learning of network lacks justification.\n7. Overall, the experimental results are not impressive and even not good. \nFor image dehazing, although the SSIM result is the best, the visual result by the proposed method is clearly not good compared to that by AECR-Net in Figure 5. There are distinct haze residuals in the image recovered by this method. \nFor image deraindrop, it is hard to see that the proposed method performs better than other state-of-the-art methods.\nFor image deblurring, the quantitative performance of the proposed approach is much worse than other methods. In Figures 7-8, it is necessary to include the corresponding results by other methods. Or it is not convincing to demonstrate the effectiveness of the proposed method.\n", "summary_of_the_paper": "This paper proposes a multi-branch restoration model to handle multiple restoration tasks in a general framework, including image dehazing, deraindrop, and deblurring. Specifically, the authors develop a multi-branch architecture with three types of existing attention blocks to simulate the Retina Ganglion Cells in the Human Visual System. The mixed skip connection with a learnable parameter is proposed to integrate the information from difference branches.", "summary_of_the_review": "My biggest concerns are the first and last comments as listed above in the weaknesses. This paper does not clearly explain why the proposed architecture can help improve the generalization ability for various tasks. In addition, the experimental results are not good enough to demonstrate the effectiveness. Therefore, I recommend this paper with rejection. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LGbOoknPytP", "reviewer": "Reviewer_ttvV", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Main strengths:\n1) The authors propose a network that can handle multiple image restoration tasks with comparable or better performance than some of recent methods that are directly designed for each of the considered tasks.\n2) They also justify their design choices with inspirations from human visual system.\n\n## Main Weaknesses:\n3) The authors have discussed how different branches of the proposed architecture resemble RGCs. However, the discussions are in high level and do not quite make the link as strongly as they claim, for example \"PAB and P-cells have one-to-one correspondence\". There needs to be a more detailed justification of how each of the paired paths resemble each other. This might have been shown in previous experimental studies on the RGCs, providing some estimation of the function that each path calculates and how the chosen architectures for each path also to some extent calculate a similar function. The main thing that I can understand from the text is that among the three considered choices each one is correctly matched to the related paths in RGCs. But is there a reason or an empirical study that shows these are good design choices and many other commonly used encoder-decoder networks are not?\n4) Using skip connections with learnable components to gather information from different parts of the network in general, cannot be claimed as a novel contribution. For example it is a common component of gradient-based neural architecture search methods. Refer to \"Neural Architecture Search: A Survey\" by Elsken et al. for a few examples.\n5) The ablation study on \"Number of branches\" claims in the text that going from one to three branches improves the PSNR \"from 28.00 dB\nto 28.88 dB\". However, table 4 shows that the best result obtained by only one branch is 28.63. So, the statement should be corrected. Overall, the improved performance added by using additional channels seems marginal to me. The ablation also does not consider the effect of added parameters when using more than one branch. It could be that a single branch of Pixel attention with equal number of parameters as the three branches together perform better than the proposed combination of branches. Another possible configuration that is missing is repeating the PA branch instead of using the other choices (with considerations to equalize the number of parameters). Unless such ablations are provided, I cannot be sure the proposed combination has any empirical advantage.\n6) The authors have chosen to use a two years old version of the dehazing task (ntire'19) rather than the latest version. There are also missing SOTA papers from this year's CVPR and ICCV conferences with reported results on the chosen dataset such as \"A Two-branch Neural Network for Non-homogeneous Dehazing via Ensemble Learning\" by Yu et al. and \"DW-GAN: A Discrete Wavelet Transform GAN for Non Homogeneous Dehazing\" by Fu et al. which seem to be much better than the proposed method in terms of PSNR and SSIM. The authors should include those in their comparisons.\n7) There are also missing SOTA papers from this year's CVPR and ICCV conferences with reported results on the deraindrop dataset such as \"Spatially-Adaptive Image Restoration using Distortion-Guided Networks\" by Purohit et al. and \"Removing Raindrops and Rain Streaks in One Go\" by Quan et al. which seem to be much better than the proposed method in terms of either SSIM or both PSNR and SSIM. The authors should include those in their comparisons.\n8) For the task of deblurring, there are many published papers that have much better performance. A few examples from this year's CVPR and ICCV conferences are: \"SDWNet: A Straight Dilated Network with Wavelet Transformation for image Deblurring\" by Zou et al., \"Perceptual Variousness Motion Deblurring with Light Global Context Refinement\" by Li et al., \"Training Weakly Supervised Video Frame Interpolation with Events\" by Yu et al., \"Bringing Events into Video Deblurring with Non-consecutively Blurry Frames\" by Shang et al., and \"Pyramid Architecture Search for Real-Time Image Deblurring\" by Hu et al. The authors should include some of those in their comparisons.\n\n#### Minor points:\n9) The phrase in the abstract \"which means they are lack of generalization\" could be modified to \"which points to their lack of generalization\"\n10) \"I_C ∈ R^{H×W×3} denotes the output image obtained by passing the degraded image through 3 × 3 convolution.\" What I understand from this statement is that there should be a connection from the input degraded image to a 3x3conv module and then the output would be I_C. However, Fig.1 shows that I_C is output of the RAM module on the top which itself is getting its input from the previous encoder-decoder module. It seems like either the statement or the figure should be corrected.\n11) Regarding the L_PS portion of the loss, the authors claim that the formulation results in avoiding to consider a ratio is not necessarily true. Just because avoiding to consider a ratio has worked, does not mean considering one and experimenting with different values of it would not impact the results to be better or worse. such a scale ratio could be multiplied by the PSNR as an example. Note that the loss has another component and this ratio should be empirically studied in parallel to the (\\epsilon).", "summary_of_the_paper": "The paper addresses the issue of multi-task image restoration. Inspired by the human visual system, the authors propose a multi-branch encoder-decoder architecture to address three tasks, dehazing, deraindrop, and deblurring, which coincide together, for example in autonomous driving scenes. They also propose to fuse the information using skip connections with a learnable parameter controlling the mixture ratios. Their results are comparable in all of the three tasks to some recent methods targeted for each individual task in terms of SSIM.", "summary_of_the_review": "The paper provides a nice intuition on how to simulate the human visual system in terms of designing a generic architecture that can work for multiple tasks. However, the mentioned points of weakness substantially impact the quality of the work. These include the proposed design (3), claimed contributions (4), empirical approach to ablations (5), and the fact that results are far worse than the state of the art (6,7,8). I have provided the details above with feedback on how to address each point). To address the mentioned concerns and achieve at least somewhat comparable performance to the SOTA, the paper needs both much further experiments and major revisions. Therefore, I cannot recommend the current version for the conference. However, this should not stop the authors from improving on their current work and hopefully submit to a future conference when it is ready.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers have substantial concerns regarding this work including novelty and experimental validation. The authors do not provide a rebuttal for the raised concerns. As such, the area chair agrees with the reviewers and does not recommend it be accepted at this conference.
Decision: Reject