Title: Training Your Image Restoration Network Better with  Random Weight Network as Optimization Function
Venue: NeurIPS 2023
Accepted: yes
Keywords: Image restoration

low-light image enhancement

image de-noising
Abstract: The blooming progress made in deep learning-based image restoration has been largely attributed to the availability of high-quality, large-scale datasets and advanced network structures. However, optimization functions such as L_1 and L_2 are still de facto. In this study, we propose to  investigate new optimization functions to improve image restoration performance. Our key insight is  that ``random weight network can be acted as a constraint for training better image restoration networks''. However, not all random weight networks are suitable as constraints. We draw inspiration from Functional theory and show that alternative random weight networks should be represented in the form of a strict mathematical manifold. We explore the potential of our random weight network prototypes that satisfy this requirement:  Taylor's unfolding network, invertible neural network, central difference convolution, and zero-order filtering. We investigate these prototypes from four aspects: 1)   random weight strategies, 2)  network architectures, 3)   network depths, and 4) combinations of random weight networks. Furthermore, we devise the random weight in two variants:  the weights are randomly initialized only once during the entire training procedure, and  the weights are randomly initialized in each training epoch. Our approach can be directly integrated into existing networks without incurring additional training and testing computational costs. We perform extensive experiments across multiple image restoration tasks, including image denoising, low-light image enhancement, and guided image super-resolution to demonstrate the consistent performance gains achieved by our method.  Upon acceptance of this paper, we will release the code.

Training Your Image Restoration Network Better with Random Weight Network as Optimization Function

Abstract

The blooming progress made in deep learning-based image restoration has been largely attributed to the availability of high-quality, large-scale datasets and advanced network structures. However, optimization functions such as L 1 and L 2 are still de facto. In this study, we propose to investigate new optimization functions to improve image restoration performance. Our key insight is that 'random weight network can be acted as a constraint for training better image restoration networks'. However, not all random weight networks are suitable as constraints. We draw inspiration from Functional theory and show that alternative random weight networks should be represented in the form of a strict mathematical manifold. We explore the potential of our random weight network prototypes that satisfy this requirement: Taylor's unfolding network, invertible neural network, central difference convolution, and zero-order filtering. We investigate these prototypes from four aspects: 1) random weight strategies, 2) network architectures, 3) network depths, and 4) combinations of random weight networks. Furthermore, we devise the random weight in two variants: the weights are randomly initialized only once during the entire training procedure, and the weights are randomly initialized in each training epoch. Our approach can be directly integrated into existing networks without incurring additional training and testing computational costs. We perform extensive experiments across multiple image restoration tasks, including image denoising, low-light image enhancement, and guided image super-resolution to demonstrate the consistent performance gains achieved by our method.

1 Introduction

Image restoration is a challenging task that involves recovering a latent clear image from a given degraded observation. This task is highly ill-posed as there are infinite feasible results for a single degraded image [1, 2]. Researchers have tackled this problem through two main approaches: traditional optimization methods [3, 4, 5] and deep learning-based methods [2, 6, 7].

Traditional methods formulate image restoration as an optimization problem and use various image priors to regularize the solution space of the latent clear image, such as low-rank prior [4, 5], dark channel prior [8, 9, 10], graph-based prior [11, 12], total variation regularization [13, 14], and sparse image priors [15, 16]. However, these priors require carefully designed priors and involve iteration optimization, making them computationally expensive.

In recent years, deep neural networks (DNNs) have shown promising results in image restoration tasks [17, 18, 19]. These methods have three key components: data, model, and optimization function.

Random

Initialization

Task-specific

F(x)

fout

Model

Low-quality Image y

Restored Image x'

?

[F(x),x]

→F'（x)（x-xo)

fout+gout

Loss

C

[x]

→F"(x)（x-x0)²

fout+gout

GroundTruth Image x

C

[9ou.x]

+F(n)(x)(x-xo)(n)

172C

Loss

PriorNetwork

akF(y)

x=F(y)+

aky

Taylor'sManifolds

Invertible Manifolds

While significant effort has been devoted to collecting high-quality and large-scale datasets and designing advanced network structures, the L 1 and L 2 losses remain the de facto optimization functions. In this study, we explore the potential of using random weight networks as a constraint for training better image restoration networks. By incorporating a random weight network as a constraint during the training process, we aim to encourage the network to learn more robust features and produce better results. This approach may help to overcome some of the limitations of traditional and deep learning-based methods for image restoration.

Related Work. In terms of the prior loss function studies, the representative one [20] customized a dual regression scheme for image super-resolution task in the form of loss regularization term. This method introduces an additional constraint on low-resolution data to reduce the space of the possible image super-resolution solutions. Similar to the loss regularization function, CycleGAN framework [21, 22, 23] exploited two sets of parallel generative adversarial networks to formulate the image restoration function and the image degradation mechanism respectively where the corresponding cycle mechanism is modeled in the loss function by the form of regularization term. In addition, this work [24] explored the Range-Null space decomposition to enable the relationship between realness and data consistency, and the consistency constraint is transferred into loss function. Despite the remarkable progress, the first two have been trained in a delicate manner while the remaining one only works on the particular forms of known degradation matrix. These issues leave room to further study the potential of loss function for image restoration.

In Figure 1, we show the flowchart of a random weight network as optimization prior to train an image restoration network, where the random weight network is treated as an optimization function without incurring additional training and testing computational costs. Despite the succinct idea, there still exists a question 'whether any network architecture with random weights can be used as optimzation function?". With this question, we propose our analysis and solution. Specifically, we stand on the Functional theory and show that alternative random weight networks should be represented in the form of a strict mathematical manifold. We explore the potential of our random weight network prototypes that satisfy this requirement: Taylor's unfolding network, invertible neural network, central difference11 convolution, and zero-order filtering. We investigate these prototypes from four aspects: 1) random weight strategies, 2) network architectures, 3) network depths, and 4) combinations of random weight networks. Based on the above settings, we employ the random weight networks as optimization functions to better optimize the task model in the following two variants: 1) the weights are randomly initialized only once during the entire training procedure, and 2) the weights are randomly initialized in each training iteration epoch. The illustration of random

Epoch n

Epochn

Sample

Sample

Model, 01

Model, 01

Training Set

BatchImages

Output

Training Set

BatchImages

Output

Initialization

Random

Loss, 02

Initialization

Random

Loss, 02

(a)randomlyinitializingonly once

(b) randomly initializing at each training epoch

weights strategies is shown in Figure 2. Our approach is verified over the representative baselines across multiple image restoration tasks including image denoising, low-light image enhancement, and guided image super-resolution and the extensive experimental results demonstrate its effectiveness.

To summarize, we make the following key contributions. 1) It is the first attempt to propose the insight that 'random weight network can be acted as a constraint for training better image restoration networks'. Our insight would spark the studies of optimization functions for image restoration. 2) Orthogonal to the existing data and model studies conducted in the field of deep learning-based image restoration, our proposed approach is a plug-and-play, thus improving image restoration performance without changing the original model and data configurations. 3) Our approach can improve the performance of multiple image restoration tasks without incurring additional training and testing computational costs.

2 Methodology

In this section, we will first introduce the prior image restoration optimization flowchart and discuss the limitation of current optimization functions. We then detail the alternative random weight networks as optimization functions and show their feasibility. At last, we present the random weights initialization strategies.

2.1 Image Restoration Flowchart

Suppose that the restoration task model as f ( X ) that transforms the input image x to the output Y , the restoration process can be written as

Suppose that the ground truth as GT , the commonly used optimization function e.g., L 1 or L 2 can be written as

where || . || 1 , 2 is the image-level L 1 or L 2 loss.

Discussion. From the Bayesian perspective, it is well known that minimizing L 1 or L 2 can be equivalent to maximum likelihood estimation in regression. The prediction of a regressor can be treated as the mean of a noisy prediction distribution, which is modeled as a Gaussian or Laplace distribution in the classic probabilistic interpretation:

where σ noise is the scale of an i.i.d. error term ϵ ∼ N (0 , σ 2 noise I) and b = √ σ noise 2 . However, the real distribution is more complex, making the model optimization difficult and pushing the model prediction biased. To solve this issue, we propose to customize the additional network as loss prior regularization term to better constrain the output close to the ground truth distribution.

2.2 Random Weight Network Manifold

In this study, we found that only random weight networks adhering to a specific mathematical manifold are suitable as optimization functions. To meet this requirement, we propose alternative random weight networks: Taylor's Unfolding Network, Invertible Neural Network, Central Difference Convolution, and Zero-order Filtering.

Taylor's Unfolding Network Manifold in Figure 3. Drawing inspiration from image decomposition, we leverage Taylor's unfolding [25] to formulate the manifold that is oriented towards decomposition

where we denote the observation, latent clear image, transformation matrix, and error term as Y , X , T , and N , respectively.

Let

Y

0

=

T

X =

Y -

N

, we learn

X

by the function

F

When only regarding n order Taylor's approximations, it can be simplified as

Recalling Equation (7), for the k order derivative part, it can be written as

To achieve this objective, we employ a Derivative function sub-network denoted as G , which operates in the aforementioned process. The k th order output of network G is represented as F k ( Y ) ϵ k and conveniently recorded as g k out

By combining the aforementioned two operational steps, we can obtain the final output of the n th order deep Taylor's approximations framework as follows:

Invertible Neural Network Manifold in Figure 3. Drawing inspiration from image transformation techniques such as Fourier transform and wavelet transform, we can designate the aforementioned invertible transformation as:

where T is the wavelet function for wavelet transform while representing the Trigonometric basis for Fourier transform.

To capture the general invertible manifold, the fundamental units divide the input into x 1 and x 2 . Reversible blocks receive (X 1 , X 2 ) and generate outputs ( Y 1 , Y 2 ) using additive coupling rules inspired by NICE's transformation [26, 27, 28], as demonstrated in Figure 3(right):

Central Difference Convolution Manifold in Figure 4. In deep networks, the vanilla 2D convolution is a fundamental operator with two steps: 1) sampling , selecting a local neighbor region R from input feature map x ; and 2) aggregating , combining sampled values with learnable weights w . This process formulates the output feature map Y

Here, p 0 represents the current location on both input and output feature maps, while p n iterates over locations in R . For example, in a convolution operator with a 3 × 3 kernel and dilation 1, the local receptive field region is defined as R = { ( -1 , -1) , ( -1 , 0) , · · · , (0 , 1) , (1 , 1) } .

y2

MappingFunctionPart

DerivativeFunctionPart

?

[F(y),y]

C

[gout,y]

?

[guty]]

(a)Forward Process

×1

xn

田

①

F(y)

F'(y)(y-yo)

F"(y)(y-yo)2

F(n)(y)(y -yo)n

1

1

1

1

X

2!

n!

F

G

X1

y1

fout

fout+gout

1

fout + Zk=1k9ut

(b)ReverseProcess

InvertibleNetwork'smanifold

Contrary to the vanilla convolution, the central difference convolution incorporates central gradient features to augment the representation and generalization capabilities. This can be formulated as:

Reverse Filtering Network Manifold inspired by [29] in Figure 4. Definition. Let ( H , d ) be a metric space, and T : H → H be a mapping function. For all x, y ∈ H , if there exists a constant c ∈ [0 , 1) such that the following formula holds:

mapping T : H → H is called Contraction Mapping.

Theorem-1. Given a function Φ : H → H , a variable x ∗ is considered a fixed point if Φ( x ∗ ) = x ∗ . If Φ is a contraction mapping, it guarantees the existence of a unique fixed point x ∗ in H . Moreover, the fixed point x ∗ can be determined using the following iterative process. Let the initial guess be x 0 and define a sequence { x n } such that x n = Φ( x n -1 ) . As the iterative process converges, we have lim n →∞ x n = x ∗ .

Reverse Filtering. The function F ( · ) can be viewed as a set of versatile filters used for image smoothing. This filtering process can be described as Y = F (X) , where X represents the input image and Y represents the filtering result. By performing reverse filtering, we can estimate X without explicitly computing F -1 ( · ) and update the restored image based on the filtering effect, resulting in:

Here, X k represents the current estimation of X in the k -th iteration. The iteration begins with X 0 = Y , and as the iteration count k increases, X k gradually approaches the true X . To facilitate this process, we introduce the auxiliary function φ ( · ) as: φ (X) = X + Y - F (X) . Therefore, the above iterative process can be regarded as a fixed point iteration

In Figure 4(right), the filtering process F ( · ) is realized using the Multi-scale Gaussian Convolution Module, which satisfies the necessary condition outlined in Theorem-1. Specifically, taking φ 1 ( · ) as an example, the sufficient condition for Theorem 1 to hold is that φ 1 ( H ) constitutes a contraction mapping

For linear filters, the condition is further simplified as

Sampling

Feature Map

Gaussian kernel

Aggregation

f()

f()

f()

Sampling

f(x)

f(x1)

f(x²)

CentralDifference'smanifold

Reverse Filtering's manifold

2.3 Random Weight Strategies

Based on the above settings, we employ random weight networks as optimization functions to better optimize the image restoration models. In Figure 2, we present the following two variants for the random weights initialization:

(1) the weights are randomly initialized only once during the entire training procedure,

(2) the weights are randomly initialized in each training iteration epoch, denoted as 'epochR';

We also discuss their effects in the experiment part.

2.4 Optimization Pipeline

Suppose that the randomly initialized manifold model as f random ( . ) , it is employed as the complementary loss function to the original image-level loss function. The total loss function is remarked as

where λ indicates the weighted factor, || . || 1 , 2 is the L 1 or L 2 loss, and GT denotes the ground truth.

3 Experiments

To demonstrate the efficacy of our proposed approach, we conduct extensive experiments on multiple image restoration tasks, including image denoising, low-light image enhancement, and guided image super-resolution. We provide more experimental results in the Appendix.

3.1 Experimental Settings

Image Enhancement. We verify our approach on the image enhancement benchmarks, LOL [30]. Further, we adopt representative SID [31] and DRBN [32] as two baselines.

Image Denoising. Following [33], we employ the widely-used SIDD dataset [34] as the training benchmark. Further, the corresponding performance evaluation is conducted on the remaining validation samples from the SIDD dataset. Two representative image denoising algorithms DnCNN [35] and MPRnet [33] are selected as the baselines.

Guided Image Super-resolution. Following [7, 36], we adopt the pan-sharpening, the representative task of guided image super-resolution, for evaluations. The WorldView II and GaoFen2 datasets [7, 36] are used for experiments. We employ INNformer [7] and SFINet [36] as the baselines.

Several widely-used image quality assessment (IQA) metrics are employed to evaluate the performance, including the relative dimensionless global error in synthesis (ERGAS) [37], PSNR, the spectral angle mapper (SAM) [38], SSIM.

3.2 Experimental Settings

Image Enhancement. We verify our approach on the image enhancement benchmarks, LOL [30]. Further, we adopt representative SID [31] and DRBN [32] as two baselines.

Image Denoising. Following [33], we employ the widely-used SIDD dataset [34] as the training benchmark. Further, the corresponding performance evaluation is conducted on the remaining validation samples from the SIDD dataset. Two representative image denoising algorithms DnCNN [35] and MPRnet [33] are selected as the baselines.

Guided Image Super-resolution. Following [39, 40], we adopt the pan-sharpening, the representative task of guided image super-resolution, for evaluations. The WorldView II and GaoFen2 datasets [7, 36] are used for experiments. We employ the representative INNformer [7] and SFINet [36] as the baselines.

3.3 Implementation Details

For concision, we denote some annotations of the proposed alternative solutions in strict mathematical manifolds before our presentation, and the implementation variants of the baselines are organized as the five configurations:

Original : the baseline with the basic loss ( L 1 or L 2 );

+Taylor : complementing the basic loss with the Taylor's unfolding network manifold;

+CDC : complementing the basic loss with the central difference convolution manifold;

+INN : complementing the basic loss with the invertible neural network manifold;

+Reverse : complementing the basic loss with the reverse filtering network manifold.

3.4 Comparison and Analysis

For quantitative Comparison, we perform the model performance comparison over different configurations. The quantitative results of image denoising, low-light image enhancement, and guided image super-resolution are respectively presented in Table 1, Table 2, and 3. From the results, we can observe performance gain against the baselines across all the datasets in the corresponding tasks, suggesting the effectiveness of our approach. For example, in terms of image enhancement, the baseline DRBN with '+Taylor', '+CDC', 'INN' and 'Reverse' has obtained the 0.4dB, 0.3dB, 0.2dB, 0.2dB PSNR gains over LoL dataset respectively.

4 Ablation Studies

Initialization

Xavier

Strategy

Initialization

Transformer

Architecture

V

000

0000000

Depth

+

Baseline

川

Kaiming

Initialization

000

Parallel

000

+

To verify the stability of our approach, we conduct the following analysis from four aspects: 1) initialization strategy, 2) model architecture, 3) model depth, and 4) model numbers.

Model architecture. In the previous experiments, all of the random weight networks are implemented by convolution networks as default. To explore the impact of network structures, we replace the default CNN with Transformer. The results in Table 4 and Table 5 demonstrate that replacing the network rarely affects performance.

Model depth. For model depth, we change the model depth of the random weight network by adding more layers. To ensure a fair comparison, other factors keep the same. The results in Table 6 and Table 7 further demonstrate the stable and robust performance gains by introducing our designs.

Initialization strategy. In our work, the default initialization strategy is Kaiming initialization. To ex- plore the impact of the initialization mode, we replace the default Kaiming initialization with Xavier initialization. Table 8 and Table 9 show that replacing the default initialization mode almost has little impact on the performance.

Model numbers. In our experiment, we use the single loss network as default and employ multiple parallel loss networks to verify the impact of model numbers. The results in Table 10 and Table 11 indicate that increasing the number of models will improve the performance. It attributes to the advantages of model ensemble.

5 Conclusion and Limitation

In this paper, we explore the potential of optimization functions and present our insight that "random weight networks can serve as a constraint for training improved image restoration networks." Drawing inspiration from Functional theory, we offer several alternative solutions within strict mathematical manifolds, known as "random weights network prototypes." Our approach seamlessly integrates into existing image restoration networks, and extensive experiments across multiple tasks validate its effectiveness. While our experiments are extensive, limitations in space prevent us from conducting more comprehensive evaluations, such as including experiments on image de-blurring and additional representative baselines.

Broader Impact

Our work demonstrates the potential of random weight networks as optimization functions for lowlevel image restoration tasks, with applications in mobile photography, healthcare, entertainment, and other fields that rely on high-quality images. We observe no apparent negative societal consequences. While our experiments show significant performance improvements, additional real-world testing and validation are necessary to ensure robustness and generalizability.

References

Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(12):10045-10067, 2022.

man zhou, Xueyang Fu, Zeyu Xiao, Gang Yang, Aiping Liu, and Zhiwei Xiong. Unfolding taylor's approximations for image restoration. In Advances in Neural Information Processing Systems , volume 34, pages 18997-19009. Curran Associates, Inc., 2021.

Dong Gong, Mingkui Tan, Yanning Zhang, Anton van den Hengel, and Qinfeng Shi. Blind image deconvolution by automatic gradient activation. In IEEE Conference on Computer Vision and Pattern Recognition , 2016.

Wenqi Ren, Xiaochun Cao, Jinshan Pan, Xiaojie Guo, Wangmeng Zuo, and Ming-Hsuan Yang. Image deblurring via enhanced low-rank prior. IEEE Transactions on Image Processing , 25(7):3426-3437, 2016.

Feng Shi, Jian Cheng, Li Wang, Pew-Thian Yap, and Dinggang Shen. Lrtv: Mr image superresolution with low-rank and total variation regularizations. IEEE transactions on medical imaging , 34(12):2459-2466, 2015.

Man Zhou, Keyu Yan, Jie Huang, Zihe Yang, Xueyang Fu, and Feng Zhao. Mutual informationdriven pan-sharpening. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1788-1798, 2022.

Man Zhou, Xueyang Fu, Jie Huang, Feng Zhao, Aiping Liu, and Rujing Wang. Effective pansharpening with transformer and invertible neural network. IEEE Transactions on Geoscience and Remote Sensing , 60:1-15, 2022.

Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. Blind image deblurring using dark channel prior. In IEEE Conference on Computer Vision and Pattern Recognition , 2016.

Jinshan Pan, Deqing Sun, Hanspeter Pfister, and Ming-Hsuan Yang. Deblurring images via dark channel prior. IEEE transactions on pattern analysis and machine intelligence , 40(10):23152328, 2017.

Yanyang Yan, Wenqi Ren, Yuanfang Guo, Rui Wang, and Xiaochun Cao. Image deblurring via extreme channels prior. In IEEE Conference on Computer Vision and Pattern Recognition , 2016.

Siyuan Li, Iago Breno Araujo, Wenqi Ren, Zhangyang Wang, Eric K Tokuda, Roberto Hirata Junior, Roberto Cesar-Junior, Jiawan Zhang, Xiaojie Guo, and Xiaochun Cao. Single image deraining: A comprehensive benchmark analysis. In IEEE Conference on Computer Vision and Pattern Recognition , 2019.

Yuanchao Bai, Gene Cheung, Xianming Liu, and Wen Gao. Graph-based blind image deblurring from a single photograph. IEEE Transactions on Image Processing , 28(3):1404-1418, 2019.

T.F. Chan and Chiu-Kwong Wong. Total variation blind deconvolution. IEEE Transactions on Image Processing , 7(3):370-375, 1998.

Shuangli Du, Yiguang Liu, Mao Ye, Zhenyu Xu, Jie Li, and Jianguo Liu. Single image deraining via decorrelating the rain streaks and background scene in gradient domain. Pattern Recognition , 79:303-317, 2018.

Siyuan Li, Wenqi Ren, Feng Wang, Iago Breno Araujo, Eric K Tokuda, Roberto Hirata Junior, Roberto M Cesar-Jr, Zhangyang Wang, and Xiaochun Cao. A comprehensive benchmark analysis of single image deraining: Current challenges and future perspectives. International Journal of Computer Vision , 129(4):1301-1322, 2021.

Wenhan Yang, Robby T Tan, Shiqi Wang, Yuming Fang, and Jiaying Liu. Single image deraining: From model-based to data-driven and beyond. IEEE Transactions on pattern analysis and machine intelligence , 2020.

Li Xu, Jimmy S. J. Ren, Ce Liu, and Jiaya Jia. Deep convolutional neural network for image deconvolution. In Neural Information Processing Systems , 2014.

Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. DeblurGAN-v2: Deblurring (orders-of-magnitude) faster and better. In IEEE International Conference on Computer Vision , 2019.

Jie Huang, Feng Zhao, Man Zhou, Jie Xiao, Naishan Zheng, Kaiwen Zheng, and Zhiwei Xiong. Learning sample relationship for exposure correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9904-9913, June 2023.

Yong Guo, Jian Chen, Jingdong Wang, Qi Chen, Jiezhang Cao, Zeshuai Deng, Yanwu Xu, and Mingkui Tan. Closed-loop matters: Dual regression networks for single image super-resolution. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5406-5415, 2020.

Yanyan Wei, Zhao Zhang, Yang Wang, Mingliang Xu, Yi Yang, Shuicheng Yan, and Meng Wang. Deraincyclegan: Rain attentive cyclegan for single image deraining and rainmaking. IEEE Transactions on Image Processing , 30:4788-4801, 2021.

Jawook Gu and Jong Chul Ye. Adain-based tunable cyclegan for efficient unsupervised low-dose ct denoising. IEEE Transactions on Computational Imaging , 7:73-85, 2021.

Zhiyuan Huang, Zixiang Chen, Qiyang Zhang, Guotao Quan, Min Ji, Chengjin Zhang, Yongfeng Yang, Xin Liu, Dong Liang, Hairong Zheng, and Zhanli Hu. Cagan: A cycle-consistent generative adversarial network with attention for low-dose ct imaging. IEEE Transactions on Computational Imaging , 6:1203-1218, 2020.

Jiwen Yu Jian Zhang Yinhuai Wang, Yujie Hu. Gan prior based null-space learning for consistent super-resolution. In AAAI Conference on Artificial Intelligence (AAAI) , 2023.

Xueyang Fu, Zeyu Xiao, Gang Yang, Aiping Liu, Zhiwei Xiong, et al. Unfolding taylor's approximations for image restoration. Advances in Neural Information Processing Systems , 34:18997-19009, 2021.

Ricky TQ Chen David Duvenaud Jens Behrmann, Will Grathwohl and Jorn-Henrik Jacobsen. Invertible residual networks. ICML , 2019.

Jascha Sohl-Dickstein Laurent Dinh and Samy Bengio. Density estimation using real nvp. ICLR , 2017.

L. Dinh, D. Krueger, and Y. Bengio. Nice: Non-linear independent components estimation. International Conference on Learning Representations , 2015.

Keyu Yan, Man Zhou, Jie Huang, Feng Zhao, Chengjun Xie, Chongyi Li, and Danfeng Hong. Panchromatic and multispectral image fusion via alternating reverse filtering network. Advances in Neural Information Processing Systems , 35:21988-22002, 2022.

Wenhan Yang Jiaying Liu Chen Wei, Wenjing Wang. Deep retinex decomposition for low-light enhancement. In British Machine Vision Conference . British Machine Vision Association, 2018.

Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning to see in the dark. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 32913300, 2018.

Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3063-3072, 2020.

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, MingHsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14816-14826, 2021.

Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1692-1700, 2018.

Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing , 26(7):3142-3155, 2017.

Zhou Man, Huang Jie, Yan Keyu, Yu Hu, Fu Xueyang, Liu Aiping, Wei Xian, and Zhao Feng. Spatial frequency domain information integration for pan-sharpening. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 234-250, 2022.

L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M. Bruce. Comparison of pansharpening algorithms: Outcome of the 2006 grs-s data fusion contest. IEEE Transactions on Geoscience and Remote Sensing , 45(10):3012-3021, 2007.

A. F. Goetz J. R. H. Yuhas and J. M. Boardman. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. Proc. Summaries Annu. JPL Airborne Geosci. Workshop , pages 147-149, 1992.

Man Zhou, Keyu Yan, Jie Huang, Zihe Yang, Xueyang Fu, and Feng Zhao. Mutual informationdriven pan-sharpening. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1798-1808, June 2022.

Man Zhou, Keyu Yan, Jinshan Pan, Wenqi Ren, Qi Xie, and Xiangyong Cao. Memoryaugmented deep unfolding network for guided image super-resolution, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a novel and orthogonal approach by exploring the potential of using random weights network as a loss function. The authors have carefully designed the random weights network with theoretical constraints based on mathematical manifolds. To validate the proposed solutions, extensive experiments have been conducted on mainstream image restoration tasks. The results consistently demonstrate the effectiveness of the approach.\n\nSTRENGTHS:\nThere are several strengths here:\n1. This paper presents a pioneering exploration of the potential of using random weights network as a loss function. With a clear motivation and a series of interesting experiments, this study offers valuable insights into the applicability of the proposed approach, potentially shaping the direction of the loss function community.\n2. The proposed designs seamlessly integrate into existing methods, resulting in performance improvements. A thorough set of ablation studies provides strong evidence to validate these findings.\n3. The paper is well-written and maintains a high level of readability, ensuring that it is easily understandable and accessible to readers.\n\nWEAKNESSES:\nThere are several weakness here:\n1. To ensure clarity, it is recommended to provide detailed information regarding the experimental settings, including the specific methodologies and procedures employed. This will allow readers to have a clear understanding of how the experiments were conducted.\n2. The figures and tables in the paper lack consistent style, indicating the need for a thorough review by the author to identify and correct any errors. Additionally, it would be beneficial to address potential limitations and investigate the extent to which the random weights network can be applied to various tasks or datasets.\n\nQUESTIONS:\nThe authors are recommended to provide a detailed response addressing each concern raised in the weaknesses section.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper seeks to explore the untapped capabilities of random weights networks as a loss function. Inspired by mathematical manifolds, the authors propose innovative and straightforward solutions for random weights networks based on rigorous mathematical properties. Extensive experimental results across various image restoration tasks validate the efficacy of these solutions, showcasing their plug-and-play nature and ability to enhance model performance while preserving the original model and data configuration as the baseline. The novelty and interest of the idea are noteworthy.\n\nSTRENGTHS:\n1. Innovative approach: The authors propose a novel concept of utilizing a well-designed random weights network as a loss function, offering a plug-and-play solution that leads to remarkable performance improvements when integrated into existing baselines. This approach avoids the need for complex network architecture designs, making it highly appealing in the field of efficiency.\n2. Theoretical foundation: The design of the random weights network is derived from rigorous mathematical manifolds, ensuring a solid theoretical basis. Furthermore, the authors have tailored the random sampling strategies to enrich the manifold representation, adding depth to the approach.\n3. Comprehensive experiments: The paper provides extensive comparison experiments in both the main paper and the appendix, showcasing the advantages of the proposed flowchart. The inclusion of ablation studies and motivation analysis further strengthens the findings, ensuring convincing evidence of the method's effectiveness.\n\nWEAKNESSES:\n1. In all the tables, the authors have suggested to highlight the best results for a clear illustration. In addition, the more visual comparison is required to show the main body.\n2. The authors have performed sufficient ablation studies. However, the corresponding experimental configuration like convolution kernel sizes needs to be detailed.\n3. It would be better if the authors have presented more experimental analysis.\n\nQUESTIONS:\nSee weaknesses part.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the idea that random weight networks can be used as loss functions for training image restoration networks. The paper proposes to use Taylor’s Unfolding Network, Invertible Neural Network, Central Difference Convolution, and Zero-order Filtering as random weight networks. The analysis and ablation studies show the effects of initialization strategy, model architecture, model depth, and model numbers. Experiments on image enhancement, image denoising, and guided image super-resolution validate that the proposed loss functions improve the performance of several existing image restoration methods.\n\nSTRENGTHS:\n+ The idea of using random weight networks as image restoration loss functions is interesting.\n+ The quantitative evaluation is performed on several image restoration tasks.\n\nWEAKNESSES:\n- The paper has a major technical flaw. The abstract states that the proposed loss functions do not incur additional training computational cost. This is unreasonable because the gradients of these loss functions require additional computation cost. To be effective, the proposed loss functions must be used in conjunction with a pixel loss and are more complex than the pixel loss. It consumes more GPU memory and time during training. The paper should report the additional training cost or the extra training time.\n- The quantitative improvements are not significant. From Table 1 to Table 9, the proposed loss functions have limited impact on the PSNR and SSIM results. For example, MPRNet is a representative denoising method, but its PSNR gain is less than 0.1 dB. In addition, the paper reports MPRNet achieves 39.24 dB on the SIDD dataset, which is far behind the PSNR result of 39.71 dB in the original paper of MPRNet. I suspect that this paper does not train MPRNet to convergence, and it is unreasonable to compare different loss functions without full convergence. As far as I know, NAFNet [a] achieves state-of-the-art 40.30 dB on the SIDD dataset. Are the proposed loss functions applicable to NAFNet?\n- Lack of visual results in the main paper. As a paper on image restoration, it is unreasonable that the main paper does not contain any visual results. Moreover, the visual results in the supplementary materials have negligible differences, which suggests the proposed loss functions are ineffective.\n- Lack of evaluation on more general image restoration tasks. The paper selects image enhancement, image denoising, and pan-sharpening as the image restoration tasks. Are the proposed loss functions applicable to more general image restoration tasks such as super-resolution? \n\n[a] Chen et al. “Simple Baselines for Image Restoration”, ECCV, 2022.\n\nQUESTIONS:\nPlease see the weaknesses section.\n\nLIMITATIONS:\nThe paper does not mention any limitations of the proposed method. I believe some discussions on the perceptual quality are necessary since the quantitative improvements are limited and the visual results in the supplementary materials have negligible difference.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the notion of using random weight networks as a constraint during the training process for image restoration. This approach aims to encourage the network to learn more robust features and produce better results, addressing the limitations of traditional optimisation methods and deep learning-based methods. By incorporating the random weight network as a constraint, the authors validate the approach towards improving image restoration performance.\n\nSTRENGTHS:\n1. The authors provide sufficient theoretical insights behind the formulation of using a randomly initialised network as an auxiliary loss function during the optimisation process. \n\n2. The ablation studies are elaborate and cover a wide variety of initialisation configurations and examine its effect on final restoration performance.\n\nWEAKNESSES:\n1. Experimental Setting section is repeated \n\n2. The proposed approach is similar to [1, 2, 3] and without discussion on differences, the contribution of the proposed work is weak. Specifically identification of different distributions and its impact on final restoration performance. \n\n3. The authors should discuss the impact of utilising multiple network architectures on the overall training period as well as memory requirements. \n\n4. The impact of initialisation distribution should be discussed, which is missing. Furthermore in the qualitative results the authors should also provide corresponding input and ground truth images for easier evaluation. \n\n5. While the authors evaluated the impact of network structures by replacing the CNN with transformer architectures in ablation. Other configurations such as using transformer based restoration networks and implications of using a lightweight optimisation network aren't considered. These ablations are necessary to identify the overall implication of using different strategies during optimisation.\n\n\n[1] Gallicchio, Claudio, and Simone Scardapane. \"Deep randomized neural networks.\" Recent Trends in Learning From Data: Tutorials from the INNS Big Data and Deep Learning Conference (INNSBDDL2019). Springer International Publishing, 2020.\n\n[2] Herrera, Calypso, et al. \"Optimal stopping via randomized neural networks.\" arXiv preprint arXiv:2104.13669 (2021).\n\n[3] Tarvainen, Antti, and Harri Valpola. \"Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.\" Advances in neural information processing systems 30 (2017).\n\nQUESTIONS:\nKindly address the comments raised in weakness section\n\nLIMITATIONS:\nThe authors have addressed the limitations arising from space but not the limitation of their methodology, which was the original objective.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}