Decomposition Ascribed Synergistic Learning for Unified Image Restoration

## Abstract

Abstract Learning to restore multiple image degradations within a single model is quite beneficial for real-world applications.
Nevertheless, existing works typically concentrate on regarding each degradation independently, while their relationship has been less exploited to ensure the synergistic learning.
To this end, we revisit the diverse degradations through the lens of singular value decomposition, with the observation that the decomposed singular vectors and singular values naturally undertake the different types of degradation information, dividing various restoration tasks into two groups, i.e. , singular vector dominated and singular value dominated.
The above analysis renders a more unified perspective to ascribe the diverse degradations, compared to previous task-level independent learning.
The dedicated optimization of degraded singular vectors and singular values inherently utilizes the potential relationship among diverse restoration tasks, attributing to the Decomposition Ascribed Synergistic Learning (DASL).
Specifically, DASL comprises two effective operators, namely, Singular VEctor Operator (SVEO) and Singular VAlue Operator (SVAO), to favor the decomposed optimization, which can be lightly integrated into existing image restoration backbone.
Moreover, the congruous decomposition loss has been devised for auxiliary.
Extensive experiments on blended five image restoration tasks demonstrate the effectiveness of our method.

## 1 Introduction

Image restoration aims to recover the latent clean images from their degraded observations, and has been widely applied to a series of real-world scenarios, such as photo processing, autopilot, and surveillance.
Compared to single-degradation removal (Zhou et al., ; Xiao et al., ; Qin et al., ; Song et al., ; Lehtinen et al., ; Lee et al., ; Pan et al., ; Nah et al., ; Li et al., ; Zhang et al., ), the recent flourished multi-degradation learning methods have gathered considerable attention, due to their convenient deployment.
However, every rose has its thorn.
How to ensure the synergy among multiple restoration tasks demands a dedicated investigation, and it is imperative to include their implicit relationship into consideration.

Figure: Figure 1: An illustration of the decomposition ascribed analysis on various image restoration tasks through the lens of the singular value decomposition. The decomposed singular vectors and singular values undertake the different types of degradation information as we recompose the degraded image with portions of the clean counterpart, ascribing diverse restoration tasks into two groups, *i.e.*, singular vector dominated rain, noise, blur, and singular value dominated low-light, haze. Dedicated to the decomposed optimization of the degraded singular vectors and singular values rendering a more unified perspective for synergistic learning, compared to previous task-level independent learning.
Refer to caption: extracted/5465152/pic/tease.png

Generally, existing multi-degradation learning methods concentrated on regarding each degradation independently.
For instance, (Chen et al., ; Li et al., ; Valanarasu et al., ) propose to deal with different restoration tasks through separate subnetworks or distinct transformer queries. (Li et al., ; Chen et al., ) propose to distinguish diverse degradation representations via contrastive learning.
Remarkably, there are also few attempts devoted to duality degradation removal with synergistic learning.
() proposes to leverage the blurry and noisy pairs for joint restoration as their inherent complementarity during digital imaging.
(Zhou et al., ) proposes a unified network with low-light enhancement encoder and deblurring decoder to address hybrid distortion.
(Wang et al., ) proposes to quantify the relationship between arbitrary two restoration tasks, and improve the performance of the anchor task with the aid of another task.
However, few efforts have been made toward synergistic learning among more restoration tasks,
and there is a desperate lack of perspective to revisit the diverse degradations for combing their implicit relationship, which set up the stage for this paper.

To solve the above problem, we propose to revisit diverse degradations through the lens of singular value decomposition, and conduct experiments on five common image restoration tasks, including image deraining, dehazing, denoising, deblurring, and low-light enhancement.
As shown in Figure 1, it can be observed that the decomposed singular vectors and singular values naturally undertake the different types of degradation information,
in that the corruptions fade away when we recompose the degraded image with portions of the clean counterpart.
Thus, various restoration tasks can be ascribed into two groups, *i.e.*, singular vector dominated degradations and singular value dominated deagradations.
The statistic results in Figure 2 further validate this phenomenon, where the quantified recomposed image quality and the singular distribution discrepancy have been presented.
Therefore, the potential relationship emerged among diverse restoration tasks could be inherently utilized through the decomposed optimization of singular vectors and singular values, considering their ascribed common properties and significant discrepancies.

In this way, we decently convert the previous task-level independent learning into more unified singular vectors and singular values learning, and form our method, Decomposition Ascribed Synergistic Learning (DASL).
Basically, one straightforward way to implement our idea is to directly perform the decomposition on latent high-dimensional tensors, and conduct the optimization for decomposed singular vectors and singular values, respectively.
However, the huge computational overhead is non-negligible.
To this end, two effective operators have been developed to favor the decomposed optimization, namely, Singular VEctor Operator (SVEO) and Singular VAlue Operator (SVAO).
Specifically, SVEO takes advantage of the fact that the orthogonal matrices multiplication makes no effect on singular values and only impacts singular vectors, which can be realized through simple regularized convolution layer.
SVAO resorts to the signal formation homogeneity between Singular Value Decomposition and the Inverse Discrete Fourier Transform, which can both be regarded as a weighted sum on a set of basis.
While the decomposed singular values and the transformed fourier coefficients inherently undertake the same role for linear combination.
And the respective base components share similar principle, *i.e.*from outline to details.
Therefore, with approximate derivation, the unattainable singular values optimization can be translated to accessible spectrum maps.
We show that the fast fourier transform is substantially faster than the singular value decomposition.
Furthermore, the congruous singular decomposition loss has been devised for auxiliary.
The proposed DASL can be lightly integrated into existing image restoration backbone for decomposed optimization.

Figure: Figure 2: The statistic validation that the decomposed singular vectors and singular values undertake the different types of degradation information. (a) The reconstruction error between the recomposed image and paired clean image on five common image restoration tasks. Low error denotes the degradation primarily distributed in the replaced portion of the image. (b) The boxplot comparison of singular value distribution between the degraded image and corresponding clean image, where the singular value dominated low-light and haze exhibit extraordinary difference. (c) The singular vector difference on separate orders of the component between the degraded image and clean image, where the singular vector dominated rain, noise, and blur present more disparity. The results are obtained under calculation on 100 images for each restoration task.
Refer to caption: extracted/5465152/pic/stas.png

The contributions of this work are summarized below:

- •
We take a step forward to revisit the diverse degradations through the lens of singular value decomposition, and observe that the decomposed singular vectors and singular values naturally undertake the different types of degradation information, ascribing various restoration tasks into two groups, *i.e.*singular vector dominated and singular value dominated.
- •
We propose the Decomposition Ascribed Synergistic Learning (DASL) to dedicate the decomposed optimization of degraded singular vectors and singular values respectively, which inherently utilizes the potential relationship among diverse restoration tasks.
- •
Two effective operators have been developed to favor the decomposed optimization, along with a congruous decomposition loss, which can be lightly integrated into existing image restoration backbone. Extensive experiments on five image restoration tasks demonstrate the effectiveness of our method.

## 2 Related work

Image Restoration. Image restoration aims to recover the latent clean images from degraded observations, which has been a long-term problem. Traditional image restoration methods typically concentrated on incorporating various natural image priors along with hand-crafted features for specific degradation removal (Babacan et al., ; He et al., ; Kundur & Hatzinakos, ).
Recently, learning-based methods have made compelling progress on various image restoration tasks, including image denoising (Lehtinen et al., ; Lee et al., ), image deraining (Zhou et al., ; Xiao et al., ), image deblurring (Pan et al., ; Nah et al., ), image dehazing (Zheng et al., ; Song et al., ), and low-light image enhancement (Li et al., ; Guo et al., ), *etc*.
Moreover, numerous general image restoration methods have also been proposed. (Zamir et al., , ; Fu et al., ) propose the balance between contextual information and spatial details.
(Mou et al., ) formulates the image restoration via proximal mapping for iterative optimization.
(Zhou et al., , ) proposes to exploit the frequency characteristics to handle diverse degradations.
Additionally, various transformer-based methods (Zamir et al., ; Liu et al., ; Liang et al., ; Wang et al., ) have also been investigated, due to their impressive performance in modeling global dependencies and superior adaptability to input contents.

Recently, recovering multiple image degradations within a single model has been coming to the fore, as they are more in line with real-world applications.
() proposes to leverage the short-exposure noisy image and the long-exposure blurry image for joint restoration as their inherent complementarity during digital imaging.
(Zhou et al., ) proposes a unified network to address low-light image enhancement and image deblurring.
Furthermore, numerous all-in-one fashion methods (Chen et al., ; Li et al., , ; Valanarasu et al., ; Chen et al., ) have been proposed to deal with multiple degradations.
(Zhang et al., ) proposes to correlate various degradations through underlying degradation ingredients. While (Park et al., ) advocates to separate the diverse degradations propcessing with specific attributed discriminative filters. Besides, most of existing methods concentrated on the network architecture design and few attempts have been made toward exploring the synergy among diverse image restoration tasks.

Figure: Figure 3: An illustration of the proposed Singular Vector Operator (SVEO), which is dedicated on the optimization of the singular vector dominated degradations, *i.e.*rain, noise, blur. Theorem 3.1 supports the feasibility and the orthogonal regularization $\mathcal{L}_{orth}$ refers to Eq. 1.
Refer to caption: extracted/5465152/pic/SVEO.png

Tensor Decomposition. Tensor decomposition has been widely applied to a series of fields, such as model compression (Jie & Deng, ; Obukhov et al., ), neural rendering (Obukhov et al., ), multi-task learning (Kanakis et al., ), and reinforcement learning ().
In terms of image restoration, a large number of decomposition-based methods have been proposed for hyperspectral and multispectral image restoration (Peng et al., ; Wang et al., , ), in that establishing the spatial-spectral correlation with low-rank approximation.

Alternatively, a surge of filter decomposition methods toward networks have also been developed.
(Zhang et al., ; Li et al., ; Jaderberg et al., ) propose to approximate the original filters with efficient representations to reduce the network parameters and inference time.
(Kanakis et al., ) proposes to reparameterize the convolution operators into a non-trainable shared part and several task-specific parts for multi-task learning.
(Sun et al., ) proposes to decompose the backbone network and only finetune the singular values to preserve the pre-trained semantic clues for few-shot segmentation.

## 3 Method

In this section, we start with introducing the overall framework of Decomposition Ascribed Synergistic Learning in Section 3.1, and then elaborate the singular vector operator and singular value operator in Section 3.2 and Section 3.3, respectively, which forming our core components.
The optimization objective is briefly presented in Section 3.4.

### 3.1 Overview

The intention of the proposed Decomposition Ascribed Synergistic Learning (DASL) is to dedicate the decomposed optimization of degraded singular vectors and singular values respectively, since they naturally undertake the different types of degradation information as observed in Figures 1 and 2.
And the decomposed optimization renders a more unified perspective to revisit diverse degradations for ascribed synergistic learning.
Through examining the singular vector dominated degradations which containing rain, noise, blur, and singular value dominated degradations including hazy, low-light, we make the following assumptions: (i) The singular vectors responsible for the content information and spatial details. (ii) The singular values represent the global statistical properties of the image.
Therefore, the optimization of the degraded singular vectors could be performed throughout the backbone network.
And the optimization for the degraded singular values can be condensed to a few of pivotal positions.
Specifically, we substitute half of the convolution layers with SVEO, which are uniformly distributed across the entire network.
While the SVAOs are only performed at the bottleneck layers of the backbone network.
We ensure the compatibility between the optimized singular values and singular vectors through remaining regular layers, and the proposed DASL can be lightly integrated into existing image restoration backbone for decomposed optimization.

Figure: Figure 4: An illustration of the core idea of the proposed Singular Value Operator (SVAO), which is dedicated on the optimization of the singular value dominated degradations, *i.e.*haze and low-light. Two-dimensional signal formations are provided for simplicity.
Refer to caption: extracted/5465152/pic/SVAO.png

### 3.2 Singular Vector Operator

The singular vector operator is proposed to optimize the degraded singular vectors of the latent representation, and supposed to be decoupled with the optimization of singular values.
Explicitly performing the singular value decomposition on high-dimensional tensors solves this problem naturally with little effort, however, the huge computational overhead is non-negligible.
Whether can we modify the singular vectors with less computation burden.
The answer is affirmative and lies in the orthogonal matrices multiplication.

###### Theorem 3.1 .

For an arbitrary matrix $X\in\mathbb{R}^{h\times w}$ and random orthogonal matrices $P\in\mathbb{R}^{h\times h},Q\in\mathbb{R}^{w\times w}$, the products of the $PX$, $XQ$, $PXQ$ have the same singular values with the matrix $X$.

We provide the proof of theorem 3.1 in the Appendix A.1.
In order to construct the orthogonal regularized operator to process the latent representation, the form of the convolution operation is much eligible than matrix multiplication, which is agnostic to the input resolution.
Hence the distinction between these two forms of operation ought to be taken into consideration.

Prior works (Sedghi et al., ; Jain, ) have shown that the convolution operation $y=conv(x)$ with kernel size $k\times k$ can be transformed to linear matrix multiplication $vec(y)=A\,vec(x)$.
Supposing the processed tensors $y,x\in\mathbb{R}^{1\times n\times n}$ for simplicity, the size of the projection matrix $A$ will come to be $n^{2}\times n^{2}$ with doubly block circulant, which is intolerable to enforce the orthogonal regularization, especially for high-resolution inputs.
Another simple way is to employ the $1\times 1$ convolution with regularized orthogonality, however,
the singular vectors of the latent representation along the channel dimension will be changed rather than spatial dimension.

Inspired by this point, SVEO proposes to transpose spatial information of the latent representation $X\in\mathbb{R}^{c\times h\times w}$ to channel dimension with the ordinary unpixelshuffle operation (Shi et al., ), resulting in $X^{{}^{\prime}}\in\mathbb{R}^{cr^{2}\times h/r\times w/r}$. And then applying the orthogonal regularized $1\times 1$ convolution $\mathcal{K}\in\mathbb{R}^{cr^{2}\times cr^{2}}$ in this domain, as shown in Figure 3.
Thereby, the degraded singular vectors can be revised pertinently, and the common properties among various singular vector dominated degradations can be implicitly exploited.
We note that the differences between SVEO and conventional convolution lie in the following:
(i) The SVEO is more consistent with the matrix multiplication as it eliminates the overlap operation attached to the convolution.
(ii) The weights of SVEO are reduced to matrix instead of tensor, where the orthogonal regularization can be enforced comfortably.
Besides, compared to the matrix multiplication, SVEO further utilizes the channel redundancy and spatial adaptivity within a local $r\times r$ region for conducive information utilization.
The orthogonal regularization is formulated as

$$ $\mathcal{L}_{orth}=\|WW^{T}\odot(\textbf{1}-I)\|_{F}^{2},$ (1) $$

where $W$ represents the weight matrix, 1 denotes a matrix with all elements set to 1, and $I$ denotes the identity matrix.

**Table 1: Time comparison (ms) between SVD and FFT formation for signal representation on high-dimensional tensor, with supposed size $64\times 128\times 128$, where the Decom. and Comp. represent the decomposition and composition.**
| Formation | Decom. time | Comp.time | Total time |
| --- | --- | --- | --- |
| SVD | 180.243 | 0.143 | 180.386 |
| FFT | 0.159 | 0.190 | 0.349 |

###### Theorem 3.1 .

### 3.3 Singular Value Operator

The singular value operator endeavors to optimize the degraded singular values of the latent representation while supposed to be less entangled with the optimization of singular vectors.
However, considering the inherent inaccessibility of the singular values, it is hard to perform the similar operation as SVEO in the same vein.
To this end, we instead resort to reconnoitering the essence of singular values and found that it is eminently associated with inverse discrete fourier transform.
We provide the formation of a two-dimensional signal represented by singular value decomposition (SVD) and inverse discrete fourier transform (IDFT) in Eq. 2 and Eq. 3.3 as follows.

$$ $\textstyle X=U\Sigma V^{T}=\sum_{i=1}^{k}\sigma_{i}u_{i}v_{i}^{T}=\sum_{i=1}^{ k}\sigma_{i}X_{i},$ (2) $$

where $X\in\mathbb{R}^{h\times w}$ represents the latent representation and $U\in\mathbb{R}^{h\times h}$, $V\in\mathbb{R}^{w\times w}$ represent the decomposed singular vectors with columns $u_{i}$, $v_{i}$, $k=min(h,w)$ denotes the rank of $X$. $\Sigma$ represents the singular values with diagonal elements $\sigma_{i}$.

$$ $\displaystyle X$ $\displaystyle=\frac{1}{hw}\sum_{u=0}^{h-1}\sum_{v=0}^{w-1}G(u,v)e^{j2\pi(\frac {um}{h}+\frac{vn}{w})}$ $\displaystyle=\frac{1}{hw}\sum_{u=0}^{h-1}\sum_{v=0}^{w-1}G(u,v)\phi(u,v),\;$ (3) $$

where $G(u,v)$ denotes the coefficients of the fourier transform of $X$, and $\phi(u,v)$ denotes the corresponding two-dimensional wave component. $m\in\mathbb{R}^{h-1}$, $n\in\mathbb{R}^{w-1}$.
Observing that both SVD and IDFT formation can be regarded as a weighted sum on a set of basis, *i.e.*$u_{i}v_{i}^{T}$ and $e^{j2\pi(\frac{um}{h}+\frac{vn}{w})}$, while the decomposed singular values $\sigma_{i}$ and the transformed fourier coefficients $G(u,v)$ inherently undertake the same role for the linear combination of various bases.
In Figure 5, we present the visualized comparison of the reconstruction results using partial components of SVD and IDFT progressively, while both formations conform to the principle from outline to details.
Therefore, we presume that the SVD and IDFT operate in a similar way in terms of signal formation, and the combined coefficients $\sigma_{i}$ and $G(u,v)$ can be approximated to each other.

Figure: Figure 5: Visual comparison of the progressive reconstruction results with SVD and IDFT components, respectively. First row, IDFT reconstruction result. Second row, SVD reconstruction result. Both conform to the principle from outline to details.
Refer to caption: extracted/5465152/pic/SVDDFT.png

In this way, we successfully translate the unattainable singular values optimization to the accessible fourier coefficients optimization, as shown in Figure 4.
Considering the decomposed singular values typically characterize the global statistics of the signal, SVAO thus concentrates on the optimization of the norm of $G(u,v)$ for consistency, *i.e.*the amplitude map, since the phase of $G(u,v)$ implicitly represents the structural content (Stark, ; Oppenheim & Lim, ) and more in line with the singular vectors.
The above two-dimensional signal formation can be easily extended to the three-dimensional tensor to perform the $1\times 1$ convolution.
Since we adopt the SVAO merely in the bottleneck layers of
the backbone network with low resolution inputs, and the fast fourier transform is substantially faster than the singular value decomposition; see Table 1. The consequent overhead of SVAO can be greatly compressed.
Note that the formation of Section 3.3 is a bit different from the definitive IDFT, and we provide the equivalence proof in the Appendix A.2.

### 3.4 Optimization objective

The decomposition loss $\mathcal{L}_{dec}$ is developed to favor the decomposed optimization congruously, formulated as

$$ $\mathcal{L}_{dec}=\sum_{i=1}^{3}\beta\|U_{rec}^{(i)}V_{rec}^{(i)T}-U_{cle}^{(i )}V_{cle}^{(i)T}\|_{1}+\|\Sigma_{rec}^{(i)}-\Sigma_{cle}^{(i)}\|_{1},$ (4) $$

where $U_{cle}$, $V_{cle}$, and $\Sigma_{cle}$ represent the decomposed singular vectors and singular values of the clean image, $U_{rec}$, $V_{rec}$, and $\Sigma_{rec}$ represent the decomposed singular vectors and singular values of the recovered image. For simplicity, we omit the pseudo-identity matrix between $UV^{T}$ for dimension transformation. $\beta$ denotes the weight.

The overall optimization objective of DASL comprises the orthogonal regularization loss $\mathcal{L}_{orth}$ and the decomposition loss $\mathcal{L}_{dec}$, together with the original loss functions of the integrated backbone network $\mathcal{L}_{ori}$, formulated as

$$ $\mathcal{L}_{total}=\mathcal{L}_{ori}+\lambda_{orth}\mathcal{L}_{orth}+\lambda _{dec}\mathcal{L}_{dec},$ (5) $$

where $\lambda_{orth}$ and $\lambda_{dec}$ denote the balanced weights.

## 4 Experiments

In this section, we first clarify the experimental settings, and then present the qualitative and quantitative comparison results with eleven baseline methods for unified image restoration. Moreover, extensive ablation experiments are conducted to verify the effectiveness of our method.

**Table 2: Quantitative results on five common image restoration datasets with state-of-the-art general image restoration and all-in-one methods. The baseline results are in grey.**
|  | Rain100L | BSD68 | GoPro | SOTS | LOL | Average | Params |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ |  |
| NAFNet | 35.56 | 0.967 | 31.02 | 0.883 | 26.53 | 0.808 | 25.23 | 0.939 | 20.49 | 0.809 | 27.76 | 0.881 | 17.11M |
| HINet | 35.67 | 0.969 | 31.00 | 0.881 | 26.12 | 0.788 | 24.74 | 0.937 | 19.47 | 0.800 | 27.40 | 0.875 | 88.67M |
| MIRNetV2 | 33.89 | 0.954 | 30.97 | 0.881 | 26.30 | 0.799 | 24.03 | 0.927 | 21.52 | 0.815 | 27.34 | 0.875 | 5.86M |
| SwinIR | 30.78 | 0.923 | 30.59 | 0.868 | 24.52 | 0.773 | 21.50 | 0.891 | 17.81 | 0.723 | 25.04 | 0.835 | 0.91M |
| Restormer | 34.81 | 0.962 | 31.49 | 0.884 | 27.22 | 0.829 | 24.09 | 0.927 | 20.41 | 0.806 | 27.60 | 0.881 | 26.13M |
| MPRNet | 38.16 | 0.981 | 31.35 | 0.889 | 26.87 | 0.823 | 24.27 | 0.937 | 20.84 | 0.824 | 28.27 | 0.890 | 15.74M |
| DGUNet | 36.62 | 0.971 | 31.10 | 0.883 | 27.25 | 0.837 | 24.78 | 0.940 | 21.87 | 0.823 | 28.32 | 0.891 | 17.33M |
| DL | 21.96 | 0.762 | 23.09 | 0.745 | 19.86 | 0.672 | 20.54 | 0.826 | 19.83 | 0.712 | 21.05 | 0.743 | 2.09M |
| Transweather | 29.43 | 0.905 | 29.00 | 0.841 | 25.12 | 0.757 | 21.32 | 0.885 | 21.21 | 0.792 | 25.22 | 0.836 | 37.93M |
| TAPE | 29.67 | 0.904 | 30.18 | 0.855 | 24.47 | 0.763 | 22.16 | 0.861 | 18.97 | 0.621 | 25.09 | 0.801 | 1.07M |
| IDR | 35.63 | 0.965 | 31.60 | 0.887 | 27.87 | 0.846 | 25.24 | 0.943 | 21.34 | 0.826 | 28.34 | 0.893 | 15.34M |
| AirNet | 32.98 | 0.951 | 30.91 | 0.882 | 24.35 | 0.781 | 21.04 | 0.884 | 18.18 | 0.735 | 25.49 | 0.846 | 8.93M |
| DASL+MPRNet | 38.02 | 0.980 | 31.57 | 0.890 | 26.91 | 0.823 | 25.82 | 0.947 | 20.96 | 0.826 | 28.66 | 0.893 | 15.15M |
| DASL+DGUNet | 36.96 | 0.972 | 31.23 | 0.885 | 27.23 | 0.836 | 25.33 | 0.943 | 21.78 | 0.824 | 28.51 | 0.892 | 16.92M |
| DASL+AirNet | 34.93 | 0.961 | 30.99 | 0.883 | 26.04 | 0.788 | 23.64 | 0.924 | 20.06 | 0.805 | 27.13 | 0.872 | 5.41M |

### 4.1 Implementation Details

Tasks and Metrics.
We train our method on five image restoration tasks synchronously. The corresponding training set includes Rain200L (Yang et al., ) for image deraining, RESIDE-OTS (Li et al., ) for image dehazing, BSD400 (Martin et al., ), WED (Ma et al., ) for image denoising, GoPro (Nah et al., ) for image deblurring, and LOL (Chen et al., ) for low-light image enhancement.
For evaluation, 100 image pairs in Rain100L (Yang et al., ), 500 image pairs in SOTS-Outdoor (Li et al., ), total 192 images in BSD68 (Martin et al., ), Urban100 (Huang et al., ) and Kodak24 (Franzen, ), 1111 image pairs in GoPro (Nah et al., ), 15 image pairs in LOL (Chen et al., ) are utilized as the test set.
We report the Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) as numerical metrics in our experiments.

Training. We implement our method on single NVIDIA Geforce RTX 3090 GPU. For fair comparison, all comparison methods have been retrained in the new mixed dataset with their default hyper parameter settings.
We adopt the MPRNet (Zamir et al., ), DGUNet (Mou et al., ), and AirNet (Li et al., ) as our baseline to validate the proposed Decomposition Ascribed Synergistic Learning.
The entire network is trained with Adam optimizer for 1200 epochs.
We set the batch size as 8 and random crop 128x128 patch from the original image as network input after data augmentation.
We set the $\beta$ in $\mathcal{L}_{dec}$ as 0.01, and the $\lambda_{orth}$, $\lambda_{dec}$ are set to be 1e-4 and 0.1, respectively.
We perform evaluations every 20 epochs with the highest average PSNR scores as the final parameters result.
More model details and training protocols are presented in the Appendix B.

**Table 3: Quantitative results of image denoising on BSD68, Urban100 and Kodak24 datasets (PSNR$\uparrow$).**
|  | BSD68 | Urban100 | Kodak24 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $\sigma$=15 | $\sigma$=25 | $\sigma$=50 | $\sigma$=15 | $\sigma$=25 | $\sigma$=50 | $\sigma$=15 | $\sigma$=25 | $\sigma$=50 |
| NAFNet | 33.67 | 31.02 | 27.73 | 33.14 | 30.64 | 27.20 | 34.27 | 31.80 | 28.62 |
| HINet | 33.72 | 31.00 | 27.63 | 33.49 | 30.94 | 27.32 | 34.38 | 31.84 | 28.52 |
| MIRNetV2 | 33.66 | 30.97 | 27.66 | 33.30 | 30.75 | 27.22 | 34.29 | 31.81 | 28.55 |
| SwinIR | 33.31 | 30.59 | 27.13 | 32.79 | 30.18 | 26.52 | 33.89 | 31.32 | 27.93 |
| Restormer | 34.03 | 31.49 | 28.11 | 33.72 | 31.26 | 28.03 | 34.78 | 32.37 | 29.08 |
| MPRNet | 34.01 | 31.35 | 28.08 | 34.13 | 31.75 | 28.41 | 34.77 | 32.31 | 29.11 |
| DGUNet | 33.85 | 31.10 | 27.92 | 33.67 | 31.27 | 27.94 | 34.56 | 32.10 | 28.91 |
| DL | 23.16 | 23.09 | 22.09 | 21.10 | 21.28 | 20.42 | 22.63 | 22.66 | 21.95 |
| Transweather | 31.16 | 29.00 | 26.08 | 29.64 | 27.97 | 26.08 | 31.67 | 29.64 | 26.74 |
| TAPE | 32.86 | 30.18 | 26.63 | 32.19 | 29.65 | 25.87 | 33.24 | 30.70 | 27.19 |
| IDR | 34.11 | 31.60 | 28.14 | 33.82 | 31.29 | 28.07 | 34.78 | 32.42 | 29.13 |
| AirNet | 33.49 | 30.91 | 27.66 | 33.16 | 30.83 | 27.45 | 34.14 | 31.74 | 28.59 |
| DASL+MPRNet | 34.16 | 31.57 | 28.18 | 34.21 | 31.82 | 28.47 | 34.91 | 32.46 | 29.18 |
| DASL+DGUNet | 33.94 | 31.23 | 27.94 | 33.74 | 31.31 | 27.96 | 34.69 | 32.16 | 28.93 |
| DASL+AirNet | 33.69 | 30.99 | 27.68 | 33.35 | 30.89 | 27.46 | 34.32 | 31.79 | 28.61 |

**Table 4: Comparison of the model size and computation complexity between baseline / DASL.**
| Method | Params (M) | FLOPs (B) | Inference Time (s) |
| --- | --- | --- | --- |
| MPRNet | 15.74 / 15.15 | 5575.32 / 2905.14 | 0.241 / 0.210 |
| DGUNet | 17.33 / 16.92 | 3463.66 / 3020.22 | 0.397 / 0.391 |
| AirNet | 8.93 / 5.41 | 1205.09 / 767.89 | 0.459 / 0.190 |

### 4.2 Comparison with state-of-the-art methods

We compare our DASL with eleven state-of-the-art methods, including general image restoration methods: NAFNet (Chen et al., ), HINet (Chen et al., ), MPRNet (Zamir et al., ), DGUNet (Mou et al., ), MIRNetV2 (Zamir et al., ), SwinIR (Liang et al., ), Restormer (Zamir et al., ), and all-in-one fashion methods: DL (Fan et al., ), Transweather (Valanarasu et al., ), TAPE (Liu et al., ), AirNet (Li et al., ) and IDR (Zhang et al., ) on five common image restoration tasks.

Table 2 reports the quantitative comparison results.
It can be observed that the performance of the general image restoration methods is systematically superior to the professional all-in-one fashion methods when more degradations are involved, attributed to the large model size.
While our DASL further advances the backbone network capability with fewer parameters, owing to the implicit synergistic learning.
Consistent with existing unified image restoration methods (Zamir et al., ; Li et al., ), we report the detailed denoising results at different noise ratio in Table 3, where the performance gain are consistent.

In  Table 4, we present the computation overhead involved in DASL, where the FLOPs and inference time are calculated over 100 testing images with the size of 512×512.
It can be observed that our DASL substantially reduces the computation complexity of the baseline methods with considerable inference acceleration, *e.g.*12.86% accelerated on MPRNet and 58.61% accelerated on AirNet.
We present the bountiful visual comparison results in the Appendix G, while our DASL exhibits superior visual recovery quality, *i.e.*more precise details in singular vector dominated degradations and more stable global recovery in singular value dominated degradations.

**Table 5: Ablation experiments on the components design.**
|  |  |  |  |  | Rain100L | BSD68 | GoPro | SOTS | LOL | Avg. |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | SVEO | SVAO | $\mathcal{L}_{orth}$ | $\mathcal{L}_{dec}$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ |
| Baseline |  |  |  |  | 38.16 | 0.981 | 31.35 | 0.889 | 26.87 | 0.823 | 24.27 | 0.937 | 20.84 | 0.824 | 28.27 | 0.890 |
| With no orth. SVEO | ✓ |  |  |  | 37.73 | 0.981 | 31.31 | 0.889 | 26.79 | 0.819 | 24.63 | 0.939 | 20.83 | 0.824 | 28.26 | 0.890 |
| With SVAO |  | ✓ |  |  | 37.92 | 0.980 | 31.41 | 0.889 | 26.85 | 0.821 | 25.58 | 0.943 | 21.05 | 0.828 | 28.56 | 0.892 |
| With SVEO | ✓ |  | ✓ |  | 38.04 | 0.981 | 31.46 | 0.890 | 26.97 | 0.826 | 25.53 | 0.945 | 20.76 | 0.822 | 28.55 | 0.893 |
| With SVEO and SVAO | ✓ | ✓ | ✓ |  | 38.01 | 0.980 | 31.53 | 0.890 | 26.94 | 0.825 | 25.63 | 0.948 | 20.92 | 0.826 | 28.61 | 0.893 |
| With $\mathcal{L}_{dec}$ |  |  |  | ✓ | 38.10 | 0.982 | 31.39 | 0.889 | 26.78 | 0.819 | 24.70 | 0.942 | 20.98 | 0.827 | 28.39 | 0.892 |
| DASL+MPRNet | ✓ | ✓ | ✓ | ✓ | 38.02 | 0.980 | 31.57 | 0.890 | 26.91 | 0.823 | 25.82 | 0.947 | 20.96 | 0.826 | 28.66 | 0.893 |

### 4.3 Abalation Studies

We present the ablation experiments on the combined degradation dataset with MPRNet as the backbone to verify the effectiveness of our method.
In Table 5, we quantitatively evaluate the two developed operators SVEO and SVAO, and the decomposition loss. The metrics are reported on the each of degradations in detail, from which we can make the following observations:
a) Both SVEO and SVAO are beneficial for advancing the unified degradation restoration performance, attributing to the ascribed synergistic learning.
b) The congruous decomposition loss is capable to work alone, and well collaborated with developed operators for decomposed optimization.
c) The orthogonal regularization is crucial to the reliable optimization of SVEO for preventing the performance drop.

To further verify the scalability of the decomposed optimization, Table 6 evaluates the performance with partially trained on singular vector dominated degradations (vec.) and singular value dominated degradations (val.).
While some properties have been observed:
a) Basically, the baseline methods concentrate on the trainable degradations, while our DASL further contemplates the untrainable ones in virtue of its slight task dependency.
b) The performance of MPRNet on val. is unattainable due to the non-convergence, however, our DASL successfully circumvents this drawback owing to the more unified decomposed optimization on singular values rather than task-level learning.
c) The vec. seems to be supportive to the restoration performance of val., see the comparison of Tables 2 and 6, indicating the potential relationship among decomposed two types of degradations.

We present the comparison of the training trajectory between baseline and DASL on singular vector dominated and singular value dominated degradations in Figures 6 and 7.
It can be observed that our DASL significantly suppresses the drastic optimization process, retaining the overall steady to better convergence point with even fewer parameters, attributing to the ascribed synergistic learning.

Figure: Figure 6: Evaluating the synergy effect through training trajectory between baseline and DASL on vec. dominated degradations.
Refer to caption: extracted/5465152/pic/DASL_SVEtraing.png

Figure: Figure 7: Evaluating the synergy effect through training trajectory between baseline and DASL on val. dominated degradations.
Refer to caption: extracted/5465152/pic/DASL_SVAtraing.png

### 4.4 Limitation and Future works

Despite the great progress that DASL has been made in ascribing the implicit relationship among diverse degradations for synergistic learning, the more sophisticated correlations demand further investigation.
We notice that beyond the decomposed singular vectors and singular values, the distribution discrepancy of various degradations on separate orders of decomposed components exhibits the splendid potential, which can be glimpsed from Figure 2.
Additionally, how to leverage and represent their sophisticated relationship in a unified framework remains an open problem.

**Table 6: Evaluating the scalability of decomposed optimization on the full test set with partially trained on singular vector dominated degradations (vec.) and singular value dominated degradations (val.) (PSNR$\uparrow$).**
| Method | Rain100L | BSD68 | GoPro | SOTS | LOL | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| MPRNet (vec.) | 39.47 | 31.50 | 27.61 | 15.91 | 7.77 | 24.45 |
| MPRNet (val.) | - | - | - | - | - | - |
| DGUNet (vec.) | 39.04 | 31.46 | 28.22 | 15.92 | 7.76 | 24.48 |
| DGUNet (val.) | 23.10 | 20.39 | 21.84 | 24.59 | 20.45 | 22.07 |
| AirNet (vec.) | 36.62 | 31.33 | 26.35 | 15.90 | 7.75 | 23.59 |
| AirNet (val.) | 19.52 | 19.17 | 14.47 | 20.63 | 16.01 | 17.96 |
| DASL+MPRNet (vec.) | 39.39 | 31.63 | 27.57 | 17.21 | 11.23 | 25.41 |
| DASL+MPRNet (val.) | 21.87 | 19.96 | 21.35 | 25.13 | 20.33 | 21.73 |
| DASL+DGUNet (vec.) | 39.11 | 31.55 | 28.16 | 16.87 | 10.21 | 25.18 |
| DASL+DGUNet (val.) | 23.19 | 20.28 | 22.69 | 25.05 | 20.87 | 22.42 |
| DASL+AirNet (vec.) | 39.11 | 31.55 | 28.16 | 16.87 | 10.21 | 25.18 |
| DASL+AirNet (val.) | 23.19 | 20.28 | 22.69 | 25.05 | 20.87 | 22.42 |

## 5 Conclusion

In this paper, we revisited the diverse degradations through the lens of singular value decomposition and observed that the decomposed singular vectors and singular values naturally undertake the different types of degradation information, ascribing various restoration tasks into two groups, *i.e.*singular vector dominated degradations and singular value dominated degradations.
The proposed Decomposition Ascribed Synergistic Learning dedicates the decomposed optimization of degraded singular vectors and singular values respectively, rendering a more unified perspective to inherently utilize the potential relationship among diverse restoration tasks for ascribed synergistic learning.
Furthermore, two effective operators SVEO and SVAO have been developed to favor the decomposed optimization, along with a congruous decomposition loss, which can be lightly integrated into existing image restoration backbone.
Extensive experiments on bunch of image restoration tasks validated the effectiveness of the proposed method and the generality of the SVD-based degradation analysis.

Broader impact.
This work potentially release the redundant model deployments in real world scenarios, and sincerely benefits a lot of edge applications with limited resources, such as mobile photography and 24/7 surveillance.
The privacy of our method may raise potential concerns when considering the removal of some important occlusions in the original images, resulting in the disclosure of private information. Therefore, how to ensure the user-agnostic security of our method needs further investment.

## Appendix A Proof of the Propositions

### A.1 Proof of Theorem 1

###### Theorem A.1 .

For an arbitrary matrix $X\in\mathbb{R}^{h\times w}$ and random orthogonal matrices $P\in\mathbb{R}^{h\times h},Q\in\mathbb{R}^{w\times w}$, the products of the $PX$, $XQ$, $PXQ$ have the same singular values with the matrix $X$.

Proof. According to the definition of Singular Value Decomposition (SVD), we can decompose matrix $X\in\mathbb{R}^{h\times w}$ into $USV^{T}$, where $U\in\mathbb{R}^{h\times h}$ and $V\in\mathbb{R}^{w\times w}$ indicate the orthogonal singular vector matrices, $S\in\mathbb{R}^{h\times w}$ indicates the diagonal singular value matrix. Thus $X^{{}^{\prime}}$=$PXQ$=$PUSV^{T}Q$. Denotes $U^{{}^{\prime}}$=$PU$ and $V^{{}^{\prime}T}$=$V^{T}Q$, then $X^{{}^{\prime}}$ can be decomposed into $U^{{}^{\prime}}SV^{{}^{\prime}T}$ if $U^{{}^{\prime}}$ and $V^{{}^{\prime}T}$ are orthogonal matrices.

$$ $\displaystyle U^{{}^{\prime}-1}$ $\displaystyle=(PU)^{-1}=U^{-1}P^{-1}=U^{T}P^{T}=(PU)^{T}=U^{{}^{\prime}T}$ (6) $\displaystyle(V^{{}^{\prime}T})^{-1}$ $\displaystyle=(V^{T}Q)^{-1}=Q^{-1}(V^{T})^{-1}=Q^{T}V=(V^{T}Q)^{T}=V^{{}^{ \prime}}$ (7) $$

Therefore, $U^{{}^{\prime}}U^{{}^{\prime}T}=I$ and $V^{{}^{\prime}T}V^{{}^{\prime}}=I$, where $I$ denotes the identity matrix, and $U^{{}^{\prime}}$, $V^{{}^{\prime}T}$ are orthogonal. $X^{{}^{\prime}}$ and $X$ have the same singular values $S$, and the singular vectors of $X$ can be orthogonally transformed to $PU$, $Q^{T}V$. Correspondingly, it can be easily extended to the case of $PX$ and $XQ$.
$\square$

###### Theorem A.1 .

### A.2 Equivalence Proof of Equation 3 and IDFT

Proposition. The signal formation principle in Equation 3 is equivalence to the definitive Inverse Discrete Fourier Transform (IDFT), where we restate the Equation 3 as following:

$$ $X=\frac{1}{hw}\sum_{u=0}^{h-1}\sum_{v=0}^{w-1}G(u,v)e^{j2\pi(\frac{um}{h}+ \frac{vn}{w})},\ m\in\mathbb{R}^{h-1},n\in\mathbb{R}^{w-1}.$ (8) $$

Proof. For the two-dimensional signal $X\in\mathbb{R}^{h\times w}$, we can represent any point on it through IDFT. Supposing ($m$,$n$) and ($m^{{}^{\prime}}$, $n^{{}^{\prime}}$) are two random points on $X$, where $m$, $m^{{}^{\prime}}$ $\in$ [0 0, $h$-$1$], $n$, $n^{{}^{\prime}}$ $\in$ [0 0, $w$-$1$], and ($m$,$n$) $\neq$ ($m^{{}^{\prime}}$, $n^{{}^{\prime}}$), we have

$$ $\displaystyle X(m,n)$ $\displaystyle=\frac{1}{hw}\sum_{u=0}^{h-1}\sum_{v=0}^{w-1}G(u,v)e^{j2\pi(\frac {um}{h}+\frac{vn}{w})},$ (9) $$

$$ $\displaystyle X(m^{{}^{\prime}},n^{{}^{\prime}})$ $\displaystyle=\frac{1}{hw}\sum_{u=0}^{h-1}\sum_{v=0}^{w-1}G(u,v)e^{j2\pi(\frac {um^{{}^{\prime}}}{h}+\frac{vn^{{}^{\prime}}}{w})}.$ (10) $$

$X(m,n)$ represents the signal value at ($m$,$n$) position on $X$, and the same as $X(m^{{}^{\prime}},n^{{}^{\prime}})$. Thus, we can rewrite $X$ as

$$ $\displaystyle X=$ $\displaystyle\begin{bmatrix}X(0,0)&\cdots&X(0,w-1)\\ \vdots&\ddots&\vdots\\ X(h-1,0)&\cdots&X(h-1,w-1)\end{bmatrix}$ (14) $\displaystyle=$ $\displaystyle\frac{1}{hw}\sum_{u=0}^{h-1}\sum_{v=0}^{w-1}G(u,v)\;\mathchoice{ \mathbin{\vbox{\hbox{\scalebox{0.5}{$\displaystyle\bullet$}}}}}{\mathbin{\vbox {\hbox{\scalebox{0.5}{$\textstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{ 0.5}{$\scriptstyle\bullet$}}}}}{\mathbin{\vbox{\hbox{\scalebox{0.5}{$ \scriptscriptstyle\bullet$}}}}}\begin{bmatrix}e^{j2\pi(\frac{u0}{h}+\frac{v0}{ w})}&\cdots&e^{j2\pi(\frac{u0}{h}+\frac{v(w-1)}{w})}\\ \vdots&\ddots&\vdots\\ e^{j2\pi(\frac{u(h-1)}{h}+\frac{v0}{w})}&\cdots&e^{j2\pi(\frac{u(h-1)}{h}+ \frac{v(w-1)}{w})}\end{bmatrix}$ (18) $\displaystyle=$ $\displaystyle\frac{1}{hw}\sum_{u=0}^{h-1}\sum_{v=0}^{w-1}G(u,v)e^{j2\pi(\frac{ um}{h}+\frac{vn}{w})},$ (19) $$

where $m$ $\in\mathbb{R}^{h-1},n\in\mathbb{R}^{w-1}$. And the two-dimensional wave $e^{j2\pi(\frac{um}{h}+\frac{vn}{w})}\in\mathbb{R}^{h-1\times w-1}$ denotes the base component. Therefore, the formation principle of Section 3.3 is equivalent to the definitive IDFT, *i.e.*Equations 9 and 10.
$\square$

## Appendix B Model details and Training Protocols

We implement our DASL with integrated MPRNet (Zamir et al., ), DGUNet (Mou et al., ), and AirNet (Li et al., ) backbone to validate the effectiveness of the decomposed optimization.
All experiments are conducted using PyTorch, with model details and training protocols provided in Table 7.
Figure 8 (a) presents the compound working flow of our operator. Note that the SVAO is only adopted in the bottleneck layer, as described in Section 3.1.
We introduce how we embed our operator into the backbone network from a microscopic perspective. Sincerely, the most convenient way is to directly reform the basic block of the backbone network.
We present two fashions of the basic block of baseline in Figure 8 (b) and (c), where the MPRNet fashion is composed of two basic units, *e.g.*, channel attention block (CAB) (Zhang et al., ), and DGUNet is constructed by two vanilla activated convolutions.
We simply replace one of them (dashed line) with our operator to realize the DASL integration.
Note that AirNet shares the similar fashion as MPRNet.

**Table 7: Model details and training protocols for DASL integrated baselines.**
| Configurations | MPRNet | DGUNet | AirNet |
| --- | --- | --- | --- |
| optimizer | Adam | Adam | Adam |
| base learning rate | 2e-4 | 1e-4 | 1e-3 |
| learning rate schedular | Cosine decay | Cosine decay | Linear decay |
| momentum of Adam | $\beta_{1}=0.9,\beta_{2}=0.999$ | $\beta_{1}=0.9,\beta_{2}=0.999$ | $\beta_{1}=0.9,\beta_{2}=0.99$ |
| channel dimension | 80 | 80 | 256 |
| augmentation | RandomCropFlip | RandomCropFlip | RandomCropFlip |
| num. of replaced operator | 18 | 14 | 50 |
| basic block | channel attention block | activated convolution | degradation guided module |
| optimization objective | CharbonnierLoss + EdgeLoss | CharbonnierLoss + EdgeLoss | L1Loss |

Figure: Figure 8: The strategy of model integration with DASL. (a) The working flow of our operator. (b) The basic building block of MPRNet fashion. (c) The basic building block of DGUNet fashion.
Refer to caption: extracted/5465152/pic/model_embedding.png

## Appendix C Trivial Ablations on operator design

The ablation experiment on the choice of scale factor r in SVEO is provided in Table 9. Note that the larger r will incur larger model size. We empirically set the scale ratio r in SVEO as 2. The working flow ablation of combined operator is provided in Table 9, and the compound fashion is preferred.

**Table 8: Ablation experiments on the scale ratio $r$ in the SVEO (PSNR$\uparrow$).**
| Scale ratio $r$ | Rain100L | SOTS | GoPro | BSD68 | LOL | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | 38.01 | 31.55 | 26.88 | 25.84 | 20.93 | 28.64 |
| 2 | 38.02 | 31.57 | 26.91 | 25.82 | 20.96 | 28.66 |
| 4 | 38.07 | 31.58 | 26.92 | 25.81 | 20.98 | 28.67 |

## Appendix D Extension experiments for property validation

In Table 11, we provide the performance of DASL on real-world image restoration tasks, *i.e.*, under-display camera (UDC) image enhancement. Typically, images captured under UDC system suffer from both blurring due to the spread point spread function, and lower light transmission rate. Compared to vanilla baseline models, DASL is capable of boosting the performance consistently. Note that the above experiments are performed on real-world UDC dataset (Zhou et al., ) without any fine-tuning, validating the capability of the model for processing undesirable degradations.
Table 11 evaluates the potential of DASL integration on transformer-based image restoration backbone. Albeit the convolutional form of the developed decomposed operators, the supposed architecture incompatibility problem is not come to be an obstacle. Note that we replace the projection layer at the end of the attention mechanism with developed operators for transformer-based methods.

**Table 10: Quantitative results of real-world image restoration tasks (under-display camera image enhancement) on TOLED and POLED datasets.**
|  | TOLED | POLED |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |
| MPRNet | 24.69 | 0.707 | 0.347 | 8.34 | 0.365 | 0.798 |
| DGUNet | 19.67 | 0.627 | 0.384 | 8.88 | 0.391 | 0.810 |
| AirNet | 14.58 | 0.609 | 0.445 | 7.53 | 0.350 | 0.820 |
| DASL+MPRNet | 25.65 | 0.733 | 0.326 | 8.95 | 0.392 | 0.788 |
| DASL+DGUNet | 25.25 | 0.727 | 0.329 | 9.80 | 0.410 | 0.783 |
| DASL+AirNet | 18.83 | 0.637 | 0.426 | 9.13 | 0.398 | 0.784 |

## Appendix E Cultivating the SVD potential for image restoration.

In fact, Singular Value Decomposition (SVD) has been widely applied for a range of image restoration tasks, such as image denoising, image compression, etc., attributing to the attractive rank properties (Sadek, ) including truncated energy maximization and orthogonal subspaces projection.
The former takes the fact that SVD provides the optima low rank approximation of the signal in terms of dominant energy preservation, which could greatly benefit the signal compression. The latter exploits the fact that the separate order of SVD-decomposed components are orthogonal, which inherently partition the signal into independent rank space, e.g., signal and noise space or range and null space for further manipulation, supporting the application of image denoising or even prevailing inverse problem solvers (Wang et al., ).
Moreover, the SVD-based degradation analysis proposed in this work excavates another promising property of SVD from the vector-value perspective, which is essentially different from previous rank-based method.
Encouragingly, the above two perspectives have the potential to collaborate well and the separate order property is supposed to be incorporated into the DASL for sophisticated degradation relationship investigation in future works.
We note that the above two SVD perspectives have the opportunity to collaborate well and the separate order potential is supposed to be incorporated into the DASL for sophisticated degradation relationship investigation in future works.

## Appendix F More decomposition ascribed analysis for degradations

We provide more visual results of decomposition ascribed analysis for diverse degradations in Figures 9 and 10, to further verify our observation that the decomposed singular vectors and singular values naturally undertake the different types of degradation information.
In Figures 11 and 12, we provide more degradation analysis to validate the generality of the proposed decomposition ascribed analysis, including downsampling, compression, color shifting, underwater enhancement, and sandstorm enhancement. The former three types are ascribed into singular vector dominated degradations and the latter two types are ascribed into singular value dominated degradations.
Note that the aforementioned analysis also holds for real scene degradation.

Figure: Figure 9: An illustration of the decomposition ascribed degradation analysis on various image restoration tasks through the lens of the singular value decomposition.
Refer to caption: extracted/5465152/pic/supp1.png

Figure: Figure 10: An illustration of the decomposition ascribed degradation analysis on various image restoration tasks through the lens of the singular value decomposition.
Refer to caption: extracted/5465152/pic/supp2.png

Figure: Figure 11: An illustration of the decomposition ascribed degradation analysis on various image restoration tasks through the lens of the singular value decomposition.
Refer to caption: extracted/5465152/pic/supp3.png

Figure: Figure 12: An illustration of the decomposition ascribed degradation analysis on various image restoration tasks through the lens of the singular value decomposition.
Refer to caption: extracted/5465152/pic/supp4.png

## Appendix G Visual comparison results

We present the visual comparison results of the aforementioned image restoration tasks in Figures 13, 14, 15, 16 and 17, including singular vector dominated degradations rain, noise, blur, and singular value dominated degradations low-light, haze. It can be observed that our DASL exhibits superior visual recovery quality in both types of degradation, *i.e.*, more precise content details in singular vector dominated degradations and more stable global recovery in singular value dominated degradations, compared to the integrated baseline method.

Figure: (a) Rainy Image
Refer to caption: extracted/5465152/baseline/rainy/Bip.png

Figure: (a) Noisy Image
Refer to caption: extracted/5465152/baseline/noisy/Bip.png

Figure: (a) Blurry Image
Refer to caption: extracted/5465152/baseline/blurry/Bip.png

Figure: (a) Hazy Image
Refer to caption: extracted/5465152/baseline/hazy/ip.jpg

Figure: (a) Low-light Image
Refer to caption: extracted/5465152/baseline/low-light/ip.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the above Weaknesses.
soundness: 3
strengths: 1. The paper uncovers an observation that the decomposed singular vectors and values naturally undertake the different types of degradation information, ascribing various restoration tasks into two groups, i.e., singular vector dominated and singular value dominated.
2. Two operators are developed to favor the decomposed optimization of degraded singular vectors and values for various restoration tasks.
summary: The paper introduces Decomposition Ascribed Synergistic Learning (DASL), a method for unified image restoration that optimizes singular vectors and values for handling multiple degradations within a single model, and integrates seamlessly into existing restoration architectures.
weaknesses: 1. There is a limited evaluation with recent AIR methods like PromptIR, GenLV, and the diffusion-based AIR techniques. Such comparisons ensure the research is aligned with the latest advancements in the field.
2. Do the observations regarding the decomposition of singular vectors and singular values hold true in terms of other degradations (low-resolution, jepg compression, etc.) or compound degradations (noise+blur, rain+haze, etc.)? Whether the proposed method is available in multiple degradation combinations?
3. How well do the decomposed singular vectors and singular values represent degradation information, and are they discriminative enough to distinguish between different degradations?
4. Some typos like ", ,".

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Are there specific references supporting the findings presented in Figures 1 and 2? Additionally, clarification is needed on the type of Singular Value Decomposition (SVD) employed in the analysis and whether any post-processing techniques were applied to produce the displayed images. It is noted that the deblurring case exhibits a relatively large area of damaged pixels, while the hazy and low-light image enhancement cases only show minor pixel degradation. What factors contribute to this discrepancy?

The authors conducted tests on underwater enhancement and sandstorm enhancement but did not include these results in Figure 1 & 2. Are the observations in Figure 1 commonly encountered across different datasets, including underwater and sandstorm? Moreover, Figure 2 is based on 100 images; Additional images should be tested to validate the findings. It is unclear whether only synthetic images were used to generate Figures 1 and 2, such as in the rainy case.

The assumptions regarding singular vectors capturing content information and spatial details, and singular values representing global statistical properties, are questionable. The authors should provide more supporting references for these claims. Furthermore, given that degradations are categorized into two groups, does the proposed method require prior knowledge of which group the degradation falls into before reconstruction?

A clear diagram illustrating the entire blueprint of the proposed methods should be included to demonstrate how the SVEO and SVAO are integrated into the algorithm. The current description lacks clarity, particularly the statement about substituting convolution layers with SVEO.

The experiments primarily report PSNR results; however, additional metrics should also be considered to provide a more comprehensive evaluation of the proposed method's performance.

The rationale for using the L1 norm in L_dec should be clarified.

Figure 5 does not effectively illustrate progressive component reconstruction as intended.

Typos in the abstract: 'tasks into two groups, , '
soundness: 3
strengths: This paper addresses a significant gap in the field by proposing a unified approach to handling multiple image degradations, moving beyond the conventional practice of treating each degradation in isolation.

The use of singular value decomposition to categorize restoration tasks into singular vector dominated and singular value dominated groups provides a fresh perspective on the relationships between different types of degradations.
summary: The manuscript explored the restoration of multiple image degradations through a unified model. By employing singular value decomposition, the authors classified restoration tasks into singular vector dominated and singular value dominated categories. They introduced the Decomposition Ascribed Synergistic Learning (DASL) framework, which optimized the decomposed components to exploit relationships among various restoration tasks. The framework included two operators, Singular VEctor Operator (SVEO) and Singular VAlue Operator (SVAO), along with a supporting decomposition loss.
weaknesses: The manuscript raises several important questions regarding its methodology and results that require clarification.

The current description lacks clarity.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: [Q1] As per my understanding, the orthogonal regularization loss and decomposition loss in formula (5) represent the reconstruction loss of singular vectors and singular values respectively, and different dominant factors are employed for various types of degradation. During the training process, the balanced weights are set as a fixed ratio. Will this setting negatively harm the mixed results?
﻿
[Q2] In Figure 2, which baseline method did you use? It seems that the exchange reconstruction error for dehazing is very close. Thus, this figure can't convince me of the effectiveness of the proposed idea in dehazing task.
﻿
[Q3] This work aims to use a single model to handle multiple kinds of degraded images with one degradation type. How does it perform on images that contain multi-types of degradation? What is the difference in performance between your model and a model that trains only with a single type of degradation?
soundness: 2
strengths: [S1] Lightweight networks have great application prospects at present.
﻿
[S2] DASL reduces the computations of the baseline model and improves the effect.
summary: This paper mainly proposes Decomposition Ascribed Synergistic Learning (DASL) to restore multiple image degradations by SVD and FFT. Specifically, diverse degradations are ascribed into two groups, singular vector-dominated degradations and singular value-dominated degradations. The proposed DASL dedicates the decomposed optimization of them respectively, rendering a more unified perspective to inherently utilize the potential partnership among diverse restoration tasks for ascribed synergistic learning. Two effective operators SVEO and SVAO have been developed to favor the decomposed optimization. Experimental results demonstrate that DASL improves restoration quality while reducing model parameters and accelerating inference speed.
weaknesses: [W1] The paper writing is very poor, containing many typos. Many sentences, paragraphs, and sections are difficult to understand. 

[W2] The proposed method applies the SVD in the FFT space and validates its effectiveness to some extent. However, it does not clearly explain the insight and why it works.

[W3] More visualization results of DASL+baseline should be presented rather than only MPRNet. In addition, in Fig. 14-16, the superiority of the proposed method is hard to distinguish. 
﻿
[W4] The article presumes that the SVD and IDFT operate similarly in terms of signal formation, but does not show the Ablation Study of the restore results.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: Same as described in the weaknesses.
soundness: 2
strengths: 1. The paper is easy to understand and has a clear overall structure.
2. Exchange of the singular value and the singular vector analysis is performed on the image restoration tasks of different degradation types, showing the role of singular values ​​and singular vectors in image restoration.
3. The singular value operator and singular vector operator are proposed to make the existing methods lighter, with some improvement in computational complexity.
summary: The goal of this paper is to solve the image restoration task of multiple image degradation types. The paper explores the relationship between singular values and singular vectors between degraded images and clear images in different degradation types. Singular value-dominated and singular vector-dominated image restoration are ascribed. Singular vector operators and singular value operators are proposed and lightly integrated into the existing image restoration backbone. Experiments on multiple degradation types are conducted to verify the effectiveness of the method.
weaknesses: 1. In Figure 2, the authors slightly show that the decomposed singular values ​​and singular vectors between different degradation types undertake different degradation information. Specifically, the exchange reconstruction error of Figure 2 (a) in the dehazing task is close to 50% for both singular value and singular vector, and the low-light and haze degradation in Figure 2 (b) are similar to other degradations such as rain and noise, while the blur and low-light in Figure 2 (c) are also very close. Therefore, I think that the authors don't explain the relationship between singular values ​​and singular vectors and degradation information well. The assumptions proposed are also untenable.
The authors could consider conducting an in-depth analysis in high-dimensional space or frequency space to analyze the impact of singular values ​​and singular vectors in terms of signal-to-noise ratio, frequency changes, etc. Even ablation analysis without using singular values ​​or singular vectors can be used instead of just statistical analysis of reconstruction errors.

2. The experiments only include results on a single degradation task dataset, not on a mixed dataset. This does not meet the requirement of using a single model for a unified image restoration task. The authors could refer to the similar experimental setup in the work of Li et al. [1] to conduct experiments on mixed degradation datasets to demonstrate the effectiveness and robustness of the proposed DASL. 

3. The stability improvements brought by the loss curves of the training trajectories in Figure 6 and  Figure 7 are also not clearly demonstrated for tasks such as denoise and derain, etc. The authors should include the corresponding intermediate feature map changes or any other visualization statistics that can assist in demonstrating the effectiveness of singular values ​​and singular vectors.

[1] Li, Boyun, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. "All-in-one image restoration for unknown corruption." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17452-17462. 2022.

Meta-review: This paper proposes a decomposition ascribed synergistic learning approach for unified image restoration. It is mainly based on the observation that the decomposed singular vectors and singular values are related to the different types of degradation. Experimental results show the effectiveness of the proposed method. 

This paper received reviews with significant divergent ratings. The major concerns include the motivation, presentation, and evaluations.  

The authors provide the rebuttal to solve the concerns of reviewers. The rebuttal solves some concerns well. For example, the authors provide more comparisons with the suggested methods by Reviewer 8UnW. However, the motivation and technique details are not explained well. For the provided comparisons (single v.s. mixed), the model on the mixed degradations does perform well compared to the one on the single degradation (about 2dB lower based on the results in the rebuttal). 

Based on the recommendations of reviewers, the paper is not accepted.

Decision: Reject