Towards Ultra-High-Definition Image Deraining: A Benchmark and An Efficient Method

## Abstract

Abstract Despite significant progress has been made in image deraining, existing approaches are mostly carried out on low-resolution images. The effectiveness of these methods on high-resolution images is still unknown, especially for ultra-high-definition (UHD) images, given the continuous advancement of imaging devices.
In this paper, we focus on the task of UHD image deraining, and contribute the first large-scale UHD image deraining dataset, 4K-Rain13k, that contains 13,000 image pairs at 4K resolution. Based on this dataset, we conduct a benchmark study on existing methods for processing UHD images.
Furthermore, we develop an effective and efficient vision MLP-based architecture (UDR-Mixer) to better solve this task. Specifically, our method contains two building components: a spatial feature rearrangement layer that captures long-range information of UHD images, and a frequency feature modulation layer that facilitates high-quality UHD image reconstruction.
Extensive experimental results demonstrate that our method performs favorably against the state-of-the-art approaches while maintaining a lower model complexity.
The code and dataset will be available at https://github.com/cschenxiang/UDR-Mixer .

## 1 Introduction

Single image deraining aims to remove the undesired degradation induced by rain streaks from input images, enhancing its visual quality and improving the accuracy of perception system [].
In image deraininig, deep learning-based methods become predominate ones as the formation of image deraining is quite simplified compared to the conventional prior-based methods []. One can choose deep models based on convolutional neural network (CNN) [, , ] or Transformer architectures [, , , ] to directly estimate clear image from rainy one.

Among these approaches, most of them are trained and evaluated on low-resolution datasets []. These commonly used benchmark datasets consist of 1K or even lower resolution images, such as Rain200L/H [] and Rain13k [], as illustrated in Figure [1](#S1.F1)(a).
Based on the existing empirical studies [, , ], existing image deraining approaches trained on these low-resolution datasets are not likely to generalize well on high-resolution images.
However, few effort has been made in ultra-high definition (UHD) image deraining due to the absence of UHD deraining dataset.
As UHD devices have been widely used, it is urgent and essential to build a high-resolution benchmark and pave the way for future research in this field.

Figure: (a) Statistics on the image resolution
Refer to caption: /html/2405.17074/assets/x1.png

To explore the performance of existing approaches on UHD images, in this paper, we first establish a large-scale dataset called 4K-Rain13k to benchmark existing methods. The proposed 4K-Rain13k contains 13,000 rainy/rain-free image pairs at 4K resolution ($3840\times 2160$), with 12,500 pairs allocated for training and 500 pairs for testing.
Unlike existing datasets [, , , , , ] that directly add rain streaks proportionally to clear images to synthesize rainy images, we observe that geometric inconsistencies in the lengths and thicknesses of rain streaks between low-resolution and high-resolution rainy images.
In high-resolution UHD images, rain streaks typically appear longer and slighter due to the increased pixel information, whereas in low-resolution images, the same length of rain streaks may appear blurred into shorter or thicker lines due to fewer pixels. To this end, by integrating geometric transformations (i.e., scaling) into the rain synthesis pipeline, we can enhance the harmony and consistency of the synthesized images, enabling them to better reflect the attributes of the original UHD image content. Based on this new dataset, we conduct extensive evaluations to analyze the performance of existing methods.

Furthermore, we find that when dealing with UHD images, most state-of-the-art (SOTA) methods often encounter memory overflow issues, making it difficult to perform full-resolution inference on consumer-grade GPUs.
This motivates us to develop an effective and efficient method tailored for UHD image deraining. In this paper, we develop an vision MLP-based architecture (called UDR-Mixer) to better solve this task, rather than relying on the self-attention mechanism that is computationally expensive in Transformers [].

The proposed UDR-Mixer consists of two parallel branches, each dedicated to exploring spatial and frequency representations to complement each other. In the main branch, we construct spatial feature mixing blocks (SFMB) as the core components, establishing global information perception through a simple yet effective feature rearrangement mechanism. Unlike strategies based on single-view spatial region rearrangement [, ], we recursively encode the entire image from different perspectives in multi-stage dimensional transformations and correlate multi-view features by permuting the tensor to better capture long-range pixel dependencies in UHD images. Simultaneously, we introduce an auxiliary branch composed of frequency feature mixing blocks (FFMB) to facilitate high-quality restoration of UHD images. Figure [1](#S1.F1)(b) illustrates that our method achieves favorable performance with a better trade-off between efficiency and performance.

The main contributions of this paper are summarized as follows:

- •
We propose the first high-quality UHD image deraining dataset (4K-Rain13k). Based on this dataset, we conduct a benchmark evaluation on existing methods for processing UHD images.
- •
We develop the spatial feature mixing block and the frequency feature mixing block to handle UHD images efficiently and formulate them into an end-to-end trainable MLP-based network (UDR-Mixer) based on a dual-branch architecture for UHD image deraining.
- •
We quantitatively and qualitatively evaluate the proposed method on the proposed 4K-Rain13k dataset as well as real-world UHD images. The results demonstrate that our approach achieves a favorable trade-off between performance and model complexity.

## 2 Related Work

Single image deraining.

When we revisit this field of single image deraining, numerous deraining approaches and benchmark datasets have been proposed in recent years with demonstrated success []. Several classic benchmark datasets are widely adopted to evaluate single image deraining performance, such as Rain200L/H [], DID-Data [], DDN-Data [] and Rain13k []. These early benchmark datasets consist of lower resolution images (1K or less). However, in this field, there is a lack of exploration specifically for higher resolution images, particularly UHD images. Furthermore, when dealing with UHD images, existing SOTA methods frequently encounter memory overflow issues, preventing them from conducting full-resolution inference on consumer-grade GPUs.

UHD image processing.

With the development of photography equipment, UHD image processing has emerged as a new research trend in recent years [, , ]. Zheng *et al.* [] formulated the UHD image dehazing network using multi-guide bilateral learning. Zhang *et al.* [] explored the task of image super-resolution for UHD resolutions, and further created two large-scale image datasets, UHDSR4K and UHDSR8K. Ren *et al.* [] developed a multi-scale separable network to address UHD deblurring problem. The UHD low-light image enhancement task has also received increasing attention from researchers, and representative datasets include UHD-LOL [] and UHD-LL []. Beyond that, other related tasks have focused on the application of UHD images, e.g., reflection removal and HDR reconstruction []. To the best of our knowledge, we first focus on the task of removing rain from UHD images, and we propose both a benchmark dataset and a baseline method.

Vision MLP.

Given the high computational cost of self-attention mechanism in vision Transformers (ViT), several researchers have designed efficient vision models comprising solely multi-layer perceptrons (MLPs). For example, MLP-Mixer [] utilizes a straightforward token-mixing MLP instead of self-attention in ViT, leading to an all-MLP network. It employs token-mixing MLP to capture token relationships and channel-mixing MLP to capture channel relationships. Afterwards, some studies further improve the performance of MLP-based models by designing other architectures, such as gMLP [] and Hire-MLP []. Recently, Tu *et al.* [] formulated a multi-axis MLP-based framework MAXIM for image processing tasks. Wu *et al.* [] developed an efficient MixNet for UHD low-light image enhancement by modeling global and local feature dependencies. Inspired by these works, we leverage the vision MLP architecture to flexibly handle UHD image deraining.

## 3 UHD Image Deraining Dataset Construction

To evaluate the performance of existing approaches on the UHD image deraining problem, we first create a large-scale benchmark dataset named 4K-Rain13k.
We note that existing low-resolution rain datasets [, , , , , ] simply add rain streaks into the clear backgrounds to obtain rainy images.
However, this copy-and-pasting approach is not suitable for synthesizing UHD rainy images due to the *geometric inconsistency* between the low-resolution and high-resolution image synthesis processes.
Thus, we develop an effective method for synthesizing rainy images tailored for UHD images, aiming to achieve more realistic visual effects. Our method involves background collection, rain streak generation and geometric transformation, which will be presented below.

Background collection.

We collect numerous clear backgrounds using a Python program based on Scrapy to download high-resolution images from the web and various devices. Our ground-truths includes a wide range of typical daytime and nighttime scenes in urban locations (e.g., buildings, streets, cityscapes) as well as natural landscapes (e.g., lakes, hills, and vegetations).

Rain streak generation.

The diversity and fidelity of rain play crucial roles in the synthesis of rain streaks. Instead of using Photoshop software to render rain streaks, we synthesize corresponding rainy images by modeling the generation of rain streaks as a motion blur process to ensure diversity. In addition, we take into account the transparency of rain layer to ensure fidelity through alpha blending.

Geometric transformation.

In fact, there is an easily overlooked problem of geometric inconsistency in the synthesis of low resolution and high-resolution rainy images, with noticeable discrepancies in the length and thickness of rain streaks.
Specifically, in high-resolution 4K images, with more pixel information available, rain streaks tend to appear longer and slighter as each rain-effected region can be accurately represented. In low-resolution images with fewer pixels, rain streaks of the same length may appear shorter or thicker due to blurring, resulting in a rougher and more ambiguous appearance.

To this end, we introduce geometric transformation operations to adjust the scale of rain streaks on the UHD images. By applying simple geometric transformations such as scaling, we aim to harmonize the proportions and sizes of rain streaks in the synthesized images with those observed in the high-resolution 4K images. This step helps alleviate the geometric disparities caused by varying image resolutions, ensuring that the rain streak patterns maintain their intended appearance and spatial relationships during the image synthesis process. We present sample images in Figure [2](#S3.F2).

Figure: Table 1: Comparison between existing image deraining datasets and our proposed 4K-Rain13k dataset. ‘Number’: the number of paired images. ‘Resolution’: the average resolution of the dataset.
Refer to caption: /html/2405.17074/assets/x3.png

Benchmark statistics.

Our proposed 4K-Rain13k dataset contains 12,500 synthetic training pairs and 500 test images at 4K resolution ($3840\times 2160$). The training and test partitions are distinct in terms of their scenes and data, with no overlap. Table [2](#S3.F2) presents a comparison between our dataset and existing public datasets. Following [], we utilize the Kullback-Leibler divergence (KLD), also known as relative entropy, to quantify the difference between the distribution of synthetic images and real images. Due to limited space, we provide the analysis results in the *supplementary material*. The results show that our 4K-Rain13k dataset is closer to the distribution of real-world rainy scenes.

## 4 Proposed UDR-Mixer

In this section, we develop an effective and efficient MLP-based method (UDR-Mixer) for UHD image deraining. We first describe the overall pipeline, and then present the details of two main components, i.e., spatial feature mixing blocks (SFMB) and frequency feature mixing blocks (FFMB).

### 4.1 Overall pipeline

As shown in Figure [3](#S4.F3), UDR-Mixer is an end-to-end dual-branch parallel network architecture that models UHD images by exploring both spatial and frequency domain information. Specifically, we first embed an input rainy image $I_{rain}\in\mathbb{R}^{H\times W\times 3}$ into the feature space $F_{0}\in\mathbb{R}^{H\times W\times C}$ through a $3\times 3$ convolution layer, where $H$, $W$ and $C$ represents the height, width, and channel, respectively.
To reduce computational complexity in high-resolution images, following previous studies [, ], we employ a PixelUnshuffle operation to downsample the features to $1/4$ of the original resolution. Then, the low-level features are processed by an encoder-decoder network consisting of $N_{i}$ SFMBs with a $2\times$ downsample operation and a $2\times$ upsample operation to produce output features.

To alleviate the issue of losing image details caused by straightforward downsampling operations, we further introduce an auxiliary branch to help UHD image reconstruction. Specifically, we stack $N_{i}$ FFMBs to excavate the frequency information of the full-resolution UHD image. Then, the learned deep features are fed to the decoder network of the main branch for guiding latent clear image restoration. Finally, the output features are obtained to estimate the derained image using a $3\times 3$ convolutional layer followed by a PixelShuffle operation. We employ the $L_{1}$ loss as the objective function:

$$ $\mathcal{L}=\left\|I_{derain}-I_{gt}\right\|_{1},$ (1) $$

where $I_{gt}$ denotes the ground-truth image, and $\|\cdot\|_{1}$ denotes the $L_{1}$-norm.

### 4.2 Spatial feature mixing block

Inspired by MLP-Mixer [], we introduce the MLP-based backbone module to encode feature information. Firstly, we develop a simple yet effective SFMB to aggregate global spatial information. Given the input feature $\mathbf{X}_{l-1}\in\mathbb{R}^{H\times W\times C}$, the proposed SFMB can be formulated as:

$$ $\displaystyle\mathbf{X}_{l}^{\prime}=\mathbf{X}_{l-1}+\text{SFRL}\left(\text{LN}\left(\mathbf{X}_{l-1}\right)\right),$ (2) $\displaystyle\mathbf{X}_{l}=\mathbf{X}_{l}^{\prime}+\text{FFL}\left(\text{LN}\left(\mathbf{X}_{l}^{\prime}\right)\right),$ $$

where LN refers to the layer normalization. $\mathbf{X}_{l}^{\prime}$ and $\mathbf{X}_{l}$ are output feature from the spatial feature rearrangement layer (SFRL) and feed-forward layer (FFL).

Figure: Figure 3: The overall architecture of the proposed UDR-Mixer for UHD image deraining, which mainly contains pixel unshuffle/shuffle operations and MLP-based components, i.e., spatial feature mixing blocks (SFMB) and frequency feature mixing blocks (FFMB).
Refer to caption: /html/2405.17074/assets/x4.png

Spatial feature rearrangement layer.

To model global spatial information with lower computational costs, we develop a SFRL based on continuous rearrangements and channel-mixing MLPs. Different from Hire-MLP [] that utilizes region rearrangement strategy, we introduce a more flexible mechanism via dimension transformation operations, which directly considers the spatial properties between pixels in 3D feature maps. This enables the model to progressively capture global features across the entire image by scrolling, thereby establishing a gradual perception of long-range information from UHD images (see Figure [3](#S4.F3)).
Specifically, we first normalize the input features, and then perform multi-stage dimension transformations to rotate the spatial perspective of the tensor across three dimensions of $H$, $W$ and $C$. Here, the 3D feature map undergoes recursive encoding from $(C,H,W)$ to $(H,W,C)$ and then to $(W,C,H)$, enabling the capture of global spatial information through multi-view dimensions.
Finally, we adjust the feature map to the original resolution, and interact with the input features to activate useful features. Mathematically, given an input feature $\mathbf{F}_{0}$, the feature propagation process of SFRL can be expressed as:

$$ $\displaystyle\mathbf{F}_{l}=\text{Interpolate}\left(\mathbf{F}_{0}\right),$ (3) $\displaystyle\mathbf{F}_{l}^{\prime}=[\mathcal{P}\left(\text{GELU}\left(\text{MLP}\left(\mathbf{F}_{l}\right)\right)\right)]_{\times 2}+\mathcal{P}\left(\text{Sigmoid}\left(\text{MLP}\left(\mathbf{F}_{l}\right)\right)\right),$ $\displaystyle\hat{\mathbf{F}_{l}}=\text{Interpolate}\left(\mathbf{F}_{l}^{\prime}\right)\odot\mathbf{F}_{0},$ $$

where $\text{Interpolate}(\cdot)$ and $\mathcal{P}(\cdot)$ denote the interpolation and permute functions. $\text{GELU}(\cdot)$ and $\text{Sigmoid}(\cdot)$ represents GELU and Sigmoid functions. MLP is a $1\times 1$ convolution layer. $\odot$ represents element-wise multiplication.

Feed-forward layer.

Similar to vision Transformer (ViT) [], FFL performs dimension reduction and non-linear transformations. Here, we adopt a FFL to transform features into compact representations, which is defined as follows:

$$ $\mathbf{F}_{l}=\text{Conv}_{1\times 1}\left(\text{GELU}\left(\text{Conv}_{3\times 3}\left(\left(\mathbf{F}_{0}\right)\right)\right)\right),$ (4) $$

where $\text{Conv}_{3\times 3}$ is a $3\times 3$ convolution layer. The initial $3\times 3$ convolution is used to enhance locality and increase the number of channels for channel mixing. The later $1\times 1$ convolution is adopted for reducing the channels back to the original input dimension.

### 4.3 Frequency feature mixing block

We note that existing methods [, , , ] mostly employ direct downsampling of UHD images to create low-resolution versions, aiming to reduce computational burden. In such cases, the full-resolution restoration process is predominantly governed by information learned solely from the low-resolution images, resulting in suboptimal performance and a tendency to lose high-frequency details, which are abundant in UHD images. To this end, we develop the FFMB as a unit for the auxiliary branch. It leverages the frequency domain information of the full-resolution images and guides the decoding restoration process of the main branch composed of SFMB. Given the input feature $\mathbf{Y}_{l-1}\in\mathbb{R}^{H\times W\times C}$, the proposed FFMB can be formulated as:

$$ $\displaystyle\mathbf{Y}_{l}^{\prime}=\mathbf{Y}_{l-1}+\text{FFML}\left(\text{LN}\left(\mathbf{Y}_{l-1}\right)\right),$ (5) $\displaystyle\mathbf{Y}_{l}=\mathbf{Y}_{l}^{\prime}+\text{FFL}\left(\text{LN}\left(\mathbf{Y}_{l}^{\prime}\right)\right),$ $$

where $\mathbf{Y}_{l}^{\prime}$ and $\mathbf{Y}_{l}$ are output feature from the frequency feature modulation layer (FFML) and FFL. The structure of FFL remains the same as that in SFMB.

Frequency feature modulation layer.

According to the convolution theorem, convolution in one domain is mathematically equivalent to the Hadamard product in its corresponding Fourier domain []. This also motivates us to introduce a FFML for implementing frequency-space manipulation. Given an input feature $\mathbf{F}_{0}$, we employ Fast Fourier Transform (FFT) to obtain the corresponding frequency representations. Then, we adopt two stacks of MLP layers with a ReLU layer in between. Finally, we perform an inverse FFT and interact with the input features to obtain updated feature representations in the original latent space. This process can be formulated as:

$$ $\displaystyle\mathbf{F}_{l}$ $\displaystyle=\mathcal{F}^{-1}\left(\text{MLP}\left(\text{ReLU}\left(\text{MLP}\left(\mathcal{F}\left(\mathbf{F}_{0}\right)\right)\right)\right)\right),$ (6) $\displaystyle\hat{\mathbf{F}_{l}}$ $\displaystyle=\mathbf{F}_{l}\odot\mathbf{F}_{0},$ $$

where $\mathcal{F}(\cdot)$ denotes the FFT and $\mathcal{F}^{-1}(\cdot)$ denotes the inverse FFT.

## 5 Experiments

In this section, we first present the experimental settings of our proposed UDR-Mixer. Then we conduct a benchmark study on our method and other comparative methods. More results can be found in the supplementary material.

### 5.1 Experimental settings

Implementation details.

In our model, $\left\{N_{1},N_{2},N_{3}\right\}$ are set to $\{2,2,4\}$. The initial number of feature channels for the main and auxiliary branches is set to 48 and 64, respectively. We conduct model training on four NVIDIA GeForce RTX 3090 GPUs with 24GB memory. In total, we perform 500 epochs of training. During the training, we adopt the Adam optimizer with a learning rate of $2\times 10^{-4}$. The patch size is set to be $768\times 768$ pixels and the batch size is set to be 8. To augment the training data, we apply random horizontal and vertical flips. For testing UHD images, we use one NVIDIA TESLA V100 with 32GB memory.

Compared methods.

We compare our MLP-based approach with prior-based algorithms (i.e., DSC []), CNN-based networks (i.e., LPNet [], JORDER-E [], RCDNet [], SPDNet []), and Transformer-based models (i.e., IDT [], Restormer []
, DRSformer [], and UDR-S2Former []). For fair comparison, we utilize the official released code of these approaches. All deep learning-based methods are retrained on the proposed 4K-Rain13k dataset. We uniformly select the weights from their final training epoch for testing purposes. Note that for some approaches (JORDER-E [], RCDNet [], SPDNet [], Restormer [], and DRSformer []), we are unable to infer full-resolution results on UHD images. Following previous UHD studies [, ], we adjust the input to the largest size that the model can handle and then resize the image to the original resolution.

Evaluation metrics.

For the 4K-Rain13k benchmark with ground truth images, we employ full-reference metrics PSNR [], SSIM [] and MSE to evaluate the image quality of each restored results. For the real-world scenes without ground truth images, we adopt the non-reference metrics NIQE [], PIQE [], and BRISQUE []. Higher PSNR and SSIM values signify better restoration quality, while lower MSE, NIQE, PIQE and BRISQUE scores indicate better perceptual quality. We also test the trainable parameters and FLOPs to analyze the computational complexity of the model.

**Table 2: Quantitative evaluations on the proposed 4K-Rain13k dataset. “Params” and “FLOPs” represent the number of trainable model parameters (in M) and FLOPs (in G), respectively. The results of FLOPs are tested on the images with $1024\times 1024$ pixels.**
| Methods | Venue | 4K-Rain13k | Complexity |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | SSIM $\uparrow$ | MSE $\downarrow$ | Params | FLOPs |  |  |  |
| Rainy input | 21.14 | 0.7594 | 812.98 | - | - |  |  |
| Prior-based methods | DSC [] | ICCV’15 | 22.93 | 0.6299 | 498.63 | - | - |
| CNN-based methods | LPNet [] | TNNLS’19 | 27.86 | 0.8924 | 171.33 | 0.03 | 57.1 |
| JORDER-E [] | TPAMI’19 | 30.46 | 0.9117 | 103.93 | 4.21 | 4363.3 |  |
| RCDNet [] | CVPR’20 | 30.83 | 0.9212 | 95.40 | 3.17 | 3400.3 |  |
| SPDNet [] | ICCV’21 | 31.81 | 0.9223 | 78.17 | 3.04 | 1428.8 |  |
| Transformer-based methods | IDT [] | TPAMI’22 | 32.91 | 0.9479 | 57.04 | 16.41 | - |
| Restormer [] | CVPR’22 | 33.02 | 0.9335 | 60.43 | 26.12 | 2478.1 |  |
| DRSformer [] | CVPR’23 | 32.96 | 0.9334 | 62.95 | 33.65 | 3887.8 |  |
| UDR-S2Former [] | ICCV’23 | 33.36 | 0.9458 | 50.69 | 8.53 | 395.8 |  |
| MLP-based method | UDR-Mixer | Ours | 34.30 | 0.9505 | 42.03 | 4.90 | 200.1 |

Figure: (a) Rainy input
Refer to caption: /html/2405.17074/assets/x5.png

### 5.2 Comparisons with the state of the art

Evaluations on the proposed 4K-Rain13k.

Table [2](#S5.T2) presents the quantitative results of different approaches on the proposed 4K-Rain13k dataset. It can be observed that our proposed UDR-Mixer achieves the highest PSNR and SSIM values while maintaining the lowest MSE value, indicating the superiority of our method in rain removal from UHD images. Specifically, our method outperforms the state-of-the-art UDR-S2Former [] by 0.94dB in terms of PSNR, while utilizing fewer network parameters and lower FLOPs.
In Figure [4](#S5.F4), we further present the visual results of different methods. We observe that CNN-based methods struggle to recover texture details, such as those in building areas, under the influence of densely packed rain streaks. Additionally, despite its ability to model global information, UDR-S2Former [], as a Transformer-based method, still exhibits sensitivity to spatially-long rain streaks present in UHD images, resulting in residual rain artifacts. In contrast, our approach produces clearer images while preserving high-frequency information.

Evaluations on real 4K rainy images.

To further evaluate the generalization capability of various deraining methods in real rainy scenes, we collect 320 real 4K rainy images from the Internet and real-world sources, referred to as 4K-RealRain. These scenes mostly originate from high-definition captures using smartphones. The quantitative results for different methods are reported in Table [3](#S5.T3). Clearly, our method achieves the lowest values across three metrics: NIQE, PIQE, and BRISQUE. This indicates that, compared to other models, the output results from our UDR-Mixer exhibit clearer content and better perceptual quality in real rainy scenes. Figure [5](#S5.F5) displays a comparison of visual results. Our method effectively removes most rain streaks and exhibits visually pleasing restoration effects, indicating its capability to generalize well to unseen real-world data types.

**Table 3: Quantitative evaluations on real rainy images. For all methods, we use the pre-trained models from the 4K-Rain13k dataset to evaluate the image deraining capabilities in real-world scenarios.**
| Methods | Input | RCDNet | SPDNet | IDT | Restormer | DRSformer | UDR-S2Former | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NIQE $\downarrow$ | 8.208 | 9.997 | 9.917 | 9.067 | 8.636 | 8.493 | 8.104 | 7.509 |
| PIQE $\downarrow$ | 54.863 | 63.816 | 64.774 | 55.049 | 60.335 | 60.441 | 55.204 | 53.104 |
| BRISQUE $\downarrow$ | 67.855 | 71.967 | 67.461 | 67.100 | 65.102 | 63.823 | 65.177 | 53.192 |

**Table 4: Quantitative evaluations on the RainDS-RS dataset, which contains Syn-RS and Real-RS.**
| Methods | JORDER-E | MSPFN | MPRNet | Uformer | Restormer | IDT | UDR-S2Former | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | 30.11 | 32.53 | 34.05 | 33.79 | 34.41 | 34.56 | 35.15 | 35.72 |
| SSIM $\uparrow$ | 0.819 | 0.851 | 0.859 | 0.851 | 0.861 | 0.863 | 0.867 | 0.868 |

Figure: (a) Rainy input
Refer to caption: /html/2405.17074/assets/x15.png

Evaluations on low-resolution benchmarks.

We further validate the scalability of our method on low-resolution benchmarks. Following [], we conduct experiments on the RainDS-RS dataset [], which contains Syn-RS and Real-RS subnets. Here, we adjust our UDR-Mixer model for a fair comparison. Specifically, we remove the pixel unshuffle/shuffle operations used for UHD images in the main branch, while keeping other components consistent. For distinction, we name it UDR-Mixer-L. According to the quantitative results in Table [4](#S5.T4), our proposed method not only demonstrates satisfactory deraining effects on 4K images but also proves effective in low-resolution scenes.

### 5.3 Ablation study

Effectiveness of rearrange strategy in SFMB.

We first replace the proposed SFMBs with residual blocks
that have comparable parameters as the baseline model (i). Table [5](#S5.T5) shows that our method improves restoration performance better compared to the baseline model by introducing SFMB. The feature rearrangement strategy is a critical component of our proposed SFMB. Here, we compare with recent MLP-based feature rearrangement methods, including spatial shift [], height-direction region rearrange and width-direction region rearrange []. Compared to methods (ii-iv), our dimension rearrangement approach yields superior quantitative results. The visual results in Figure [7](#S5.F7) (b-d) and (f) also demonstrate that our method not only effectively removes complex rain streaks but also better preserves the fine details of the image. The reason behind this lies in our method implicitly enhancing the capture of multi-view features through dimension transformation, making it more suitable for modeling long-range spatial relationships in UHD images.

Effect of the number of permute stages.

We further analyze the effect of the number of permute stages in the SFMB. Note that we utilize Permute operations to rotate rotate 3D feature maps between adjacent stages. When stage=1, the model (v) can only capture the single-view features. We find that through multiple stages of recursive encoding, features learned from three perspectives are effectively correlated, thus aiding in further boosting the image restoration performance.

Effectiveness of FFMB.

The FFMB in the auxiliary branch is used to better explore frequency information in our UDR-Mixer for high-quality UHD image restoration. To demonstrate the effectiveness of this branch, we remove this component and investigate its influence in Table [5](#S5.T5). In comparison to our approach, the restoration performance of model (vii) is suboptimal. In addition, Figures [7](#S5.F7) (e) and (f) also show that our method generates much clearer details.

**Table 5: Ablation comparison on different variants of our UDR-Mixer on the 4K-Rain13k dataset.**
| Methods | SFMB | FFMB | PSNR | SSIM |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Spatial<br>Shift | H-Region<br>Rearrange | W-Region<br>Rearrange | Dimension Rearrange |  |  |  |  |  |  |
| Stage-1 | Stage-2 | Stage-3 |  |  |  |  |  |  |  |
| (i) |  |  |  |  |  |  |  | 33.41 | 0.9417 |
| (ii) | ✔ |  |  |  |  |  |  | 33.65 | 0.9433 |
| (iii) |  | ✔ |  |  |  |  |  | 34.09 | 0.9437 |
| (iv) |  |  | ✔ |  |  |  |  | 33.74 | 0.9413 |
| (v) |  |  |  | ✔ |  |  |  | 32.41 | 0.9394 |
| (vi) |  |  |  | ✔ | ✔ |  |  | 32.50 | 0.9409 |
| (vii) |  |  |  | ✔ | ✔ | ✔ |  | 34.15 | 0.9465 |
| (viii, Ours) |  |  |  | ✔ | ✔ | ✔ | ✔ | 34.30 | 0.9505 |

Figure: Table 6: Comparisons of model complexity with MAXIM []. The size of the test image is $256\times 256$.
Refer to caption: /html/2405.17074/assets/x21.png

Figure: (a) Rainy input
Refer to caption: /html/2405.17074/assets/x22.png

### 5.4 Discussions with the closely-related method

We note that the recent method MAXIM [] proposes a multi-axis MLP based architecture to solve imgae deraining. Different from MAXIM that employs the multi-axis gated strategy, our method utilizes a simple yet effective dimension rearrange mechanism to capture spatial information. First, we report the model complexity in Table [6](#S5.F6). Compared to MAXIM, our UDR-Mixer-L achieves a 81.6% reduction in model parameters while decreasing FLOPs by 69.9%. Note that as the training code of MAXIM is not available, we do not benchmark this approach on our proposed 4K-Rain13k. Since the testing code of MAXIM and the pre-trained model on the Rain13k dataset [] are available, we compare their generalization ability of MAXIM and our method in real rainy scenes. As shown in Figure [6](#S5.F6) (b) and (c), our method successfully removes rain streaks and generates a clearer image.

## 6 Concluding Remarks

This paper explores the task of UHD image deraining for the first time and proposes a high-quality dataset 4K-Rain13k to facilitate the performance comparison. Furthermore, we develop an efficient MLP-based method UDR-Mixer for UHD image deraining. Our approach utilizes a dimension rearrange mechanism to establish the global spatial context of UHD images and combines it with the original frequency representation of UHD images to help image restoration. The benchmark results show that our model achieves a favorable trade-off between performance and model complexity.

Limitations.

Although our method achieves favorable performance, it fails to handle the presence of fog-like rain accumulation in real rainy scenes. Future work will consider expanding 4K data with veiling effect and introducing physical models to guide enhancing the quality of image reconstruction.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see my comments in the weakness and answer the questions I raised in the weakness.
soundness: 3
strengths: - The paper introduces the first large-scale UHD image deraining dataset, 4K-Rain13k, filling an the gap in high-resolution deraining research
- The method is evaluated on both synthetic and real dataset, which verifies its practical use.
summary: This paper addresses the task of deraining UHD images, which is not yet well-explored despite advances in low-resolution image deraining research. The authors introduce the first large-scale UHD image deraining dataset, containing 13000 pairs of 4K resolution images. Using this dataset, they benchmark existing methods and identify performance limitations for UHD images. Additionally, the paper presents a model called UDR-Mixer for a balance of both effectiveness and efficiency. Experimental results show that the proposed method outperforms state-of-the-art methods with lower model complexity, making it suitable for practical use.
weaknesses: - I do not really understand how oversized models are evaluated from the expressions in Line 361- 363. If I do not misunderstand, the authors first resize the image to the largest size that can be processed by a single GPU, and then perform these deraining models, finally enlarge the size to the original size and evaluate the metrics. If so, I concern it would be unfair for these compared methods. Changing the resolution or dpi during the inference would greatly reduce the performance, since the model is not trained on this resolution, and resize operation would also lead to blurry artifacts. A more rigorous way is to divide the input image to several overlapped patches, evaluate on each patch, and then combine them.
- I am not sure whether it is necessary to have a training dataset with 4K resolution. We cannot send the full resolution images during training, just like this paper crop $768 \times 768$ patches for training. So from my view, it is equivalant to have a training dataset with $768 \times 768$  resolution. Please explain the rationale behind using 4K resolution for the training dataset given the $768 \times 768$ patch size used in training.
- I concern whether the rain streak generation approach is authentic enough to generate 4K images. With higher resolution, the details of real rain streak (like the texture, position, perspective relationship) should be clearer than that in a low-resolution image. It requires more realistic generation method for high resolution. However, in Fig. 2, the generated rain streak seems like a separate layer in front of the scene, and the discrepencies are enlarged in high resolution images.
- The dataset is regarded as an important contribution to this paper. Therefore, the authors should also conduct ablation studies on dataset, to verify why 4K training dataset instead of low-resolution training dataset is important for 4K deraining testing under synthetic/real scenarios. Specifically, conduct ablation studies comparing models trained on 4K data vs. lower resolution data, but tested on 4K images.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see 'Weaknesses'.
soundness: 3
strengths: 1. The paper constructs the first high-quality UHD image deraining dataset (4K-Rain13k).
2. The paper develops a dual-branch architecture UDR-Mixer, where the spatial feature mixing block and the frequency feature mixing block are proposed.
3. Experimental results demonstrate that UDR-Mixer achieves a favorable trade-off between performance and model complexity.
summary: The paper contributes the first large-scale UHD image deraining dataset and proposes MLP-based architecture (UDR-Mixer) to achieve this task. Extensive experiments demonstrate that UDR-Mixer performs favorably against the state-of-the-art approaches.
weaknesses: 1. It is more appropriate to calculate #FLOPs on 4K images.
2. The testing time on 4K images can also be given.
3. For Lines 362-363, why not evaluate by cropping the image into multiple patches?
4. In Table 3, the authors can use some more advanced non-reference IQA metrics, e.g., MANIQA, MUSIQ, and CLIPIQA.
5. It is better to give some 4K deraining results of UDR-Mixer trained on low-resolution datasets.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: * The comparison to MAXIM is only performed in terms of one example image. How does the proposed model compare in terms of PSNR on the Rain13k dataset?
* How was the selection of previous methods done? Specifically, how did you determine that a model is not possible to use on the UHD dataset? Is it due to architectural constraints, memory consumption, or computational complexity? If memory or computations is a deciding factor, what is the threshold for deciding if it is not applicable to UHD?
* In relation to training of the compared methods, it is explained that "We uniformly select the weights from their final training epoch for testing purposes". Does this mean that all models, including the proposed, are trained for an equal number of epochs?
* How was other hyper-parameters tuned for the different models? Are all methods trained using their respective default settings? Wouldn't it potentially be other optimal hyper-parameters when training on the UHD dataset?
* Some limitations are mentioned in the end of the paper. It would be interesting to see some examples of typical failure cases. Are there some other notable artifacts in specific situations?
soundness: 3
strengths: The focus on UHD image de-raining is a natural extension of previous work, and there is good value in providing a dataset for this purpose. The proposed design focusing on efficient processing at the UHD resolution level is demonstrated to be both efficient and generating competitive performance in comparison to previous methods. Although this is an adaptation of previous work, there seem to be some novel aspects in terms of promoting detail reconstruction at high resolution. Overall, the combination of UHD focus, providing a dataset for this purpose, and competitive results, makes for relatively high significance.
summary: The paper introduces a 4K dataset with synthetically generated rain streaks, for the purpose of data-driven de-raining image enhancement. The dataset contains 13K images and is produced by taking some of the specific challenges in synthesizing rain at this resolution level into account. An MLP-Mixer-based network is formulated, with the aim to allow for de-raining of UHD images with less computational demands compared to previous state-of-the-art methods. The architecture is designed using separate branches for spatial and frequency domain processing. Low-level image features are processed in an autoencoder network composed of spatial feature mixing blocks, while an auxiliary branch performs frequency domain mixing to promote quality at the UHD resolution. Experiments compare against previous methods that are applicable in the UHD domain, on the synthetic images with reference-based metrics and on real rainy images using non-reference quality metrics. Additional experiments also compare on datasets of lower resolution, where the proposed method is adapted to work better at this resolution. An ablation study explore the different components of the network, e.g., different strategies for mixing.
weaknesses: One of the formulated contributions is the dataset. However, in its current form the paper falls short of demonstrating this as a strong contribution. There is very little detail regarding the dataset. Online sources are not specified, and the simulation of rain is under-explained. The motivation for why the simulation is different in UHD images is unclear and speculative, and there is very little information on the transformations used to account for this in the simulation. It would not be possible to reproduce the simulation without a significant amount of additional details. 

While the paper has an ok structure overall, the writing could be improved in terms of formulations and grammar. Also, the reference format needs to be revised (differentiating between \citet and \citep depending on usage).

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: The authors should address the issues mentioned in the weaknesses, particularly those related to the technical design and implementation details.
soundness: 2
strengths: 1. The proposed dataset could benefit the community and inspire further research.
2. The experiments and ablation studies are comprehensive, demonstrating the effectiveness of the proposed method compared to existing state-of-the-art approaches.
summary: This paper focuses on ultra-high-definition (UHD) image deraining and introduces the first UHD image deraining dataset, 4K-Rain13k. The authors also propose a dual-branch network architecture, UDR-Mixer, to better address this task. Specifically, a spatial feature rearrangement layer is employed to capture long-range information in UHD images, while a frequency feature modulation layer complements the reconstruction of UHD image details. Qualitative and quantitative experimental results demonstrate that the proposed method outperforms existing state-of-the-art approaches on both the proposed dataset and several public datasets.
weaknesses: 1. The technical contribution of the proposed method is limited. The proposed Spatial Feature Rearrangement Layer (SFRL) is quite similar to the Global Feature Modulation Layer (GFML) in MixNet [1].  The author should explicitly compare and contrast the two modules. Adding a paragraph to discuss the key similarities and differences, and highlighting the novel design of the proposed SFRL, is essential.
2. Many implementation details are unclear and not described in the paper. For instance, the authors mention that raindrop generation is modeled as a motion blur process to synthesize corresponding raindrop images. However, how exactly are these images generated? Additionally, the authors claim that alpha blending is used to ensure fidelity, but how are the blending weights determined, and how do different weights affect fidelity?  The authors should include a more detailed subsection on the dataset generation process, covering the exact motion blur modeling and alpha blending techniques used. To provide a clearer understanding, examples showing how different blending weights affect the fidelity of the final synthesized images should also be provided.
3. To better demonstrate the efficiency of the proposed method, the comparison of runtime should be reported. The author should include a table or figure comparing the runtime of the proposed approach to other SOTA methods on a specific hardware setup.

[1] Wu et al., MixNet: Efficient Global Modeling for Ultra-High-Definition Image Restoration. arXiv2024.

Meta-review: This paper presents the first dataset for UHD-resolution image draining and a computationally efficient deraining model for such high-resolution images. The dataset outstands the existing lower-resolution datasets as the rain streak shape patterns differ in higher resolutions. The existing methods and the proposed method are compared in both the high- and low-resolution datasets. 

* However, there is a concern about the technical novelty in the proposed method.

The proposed SFRL is quite similar to GFML; multi-scale integration compared with single-scale GFML is not very significant.
Another proposed component, FFML, is without technical justification for why FFT is employed.
Lack of significant differences and the lack of design justification make it hard to say the contribution is novel and technically effective even if the experimental results are better.

* The reviewers asked for the details how the compared methods were trained and how were the hyper-parameters were chosen. The authors said that all methods were with the default settings, however, the revised paper does not appear so.

* As the dataset is the major contribution of this work, there was a request for grounds why 4K training dataset is necessary compared with low-resolution training data validated on 4K test environment with real/synthetic scenarios. The authors responded by suggesting KL divergence between the proposed dataset and real rainy images. The expectation is not fulfilled.

Considering the reviews and the author responses, the authors are suggested to improve the work.

Decision: Reject