RainyScape: Unsupervised Rainy Scene Reconstruction using Decoupled Neural Rendering

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 12:02:45

Abstract: We propose RainyScape, an unsupervised framework for reconstructing clean scenes from a collection of multi-view rainy images. RainyScape consists of two main modules: a neural rendering module and a rain-prediction module that incorporates a predictor network and a learnable latent embedding that captures the rain characteristics of the scene. Specifically, based on the spectral bias property of neural networks, we first optimize the neural rendering pipeline to obtain a low-frequency scene representation. Subsequently, we jointly optimize the two modules, driven by the proposed adaptive direction-sensitive gradient-based reconstruction loss, which encourages the network to distinguish between scene details and rain streaks, facilitating the propagation of gradients to the relevant components. Extensive experiments on both the classic neural radiance field and the recently proposed 3D Gaussian splatting demonstrate the superiority of our method in effectively eliminating rain streaks and rendering clean images, achieving state-of-the-art performance. The constructed high-quality dataset and source code will be publicly available.

Main paper: 
 introduction: 
 Neural Radiance Field (NeRF) [29] has emerged as a groundbreaking technique for novel view synthesis by learning a continuous, volumetric representation of the scene through differentiable volume rendering. NeRF's ability to create highly realistic and consistent novel views with fine details has led to its widespread adoption in various applications, such as 3D reconstruction [40] , surface reconstruction [3] , 3D object editing [36] , and large-scale scene reconstruction [53] .However, when the input images are degraded by various factors such as blur, noise, or rain, the rendering results inevitably exhibit obvious artifacts. To address this issue, recent works have proposed a range of task-specific solutions. For instance, Ma et al. [27] achieved clear scene reconstruction from images affected by camera motion blur or defocus blur by learning ray fusion within the blur kernel. Huang et al. [16] tackled the problem of input images with different dynamic ranges by adjusting the ray intensity using a camera response function and obtaining the final color with a learnable tone mapper. Chen et al. [6] extended the atmospheric scattering model into the volume rendering process to fit hazy images using two neural rendering processes. In contrast to these methods, our approach specifically targets the rainy scene reconstruction task, which can be seamlessly integrated with our observation of the neural rendering prior, as illustrated in Fig. 2 . Moreover, the sparse and intermittent nature of rain precipitation in 3D space makes it challenging to represent through an additional neural rendering field. Furthermore, our proposed framework can be readily adapted to work with various rendering techniques, demonstrating its versatility and flexibility.In this paper, we propose RainyScape, a decoupled neural rendering framework that is capable of reconstructing a rain-free scene from rainy images in an unsupervised fashion. First, we obtain a low-frequency representation of the scene through a selected radiance field rendering process (e.g., NeRF [29] or 3D Gaussian Splatting [19] ), where the remaining high-frequency scene details and rain information are coupled together. We then characterize the rain in the scene using learnable rain embeddings composed of rainy scene state vectors, viewpoint state vectors, and camera parameters, and predict rain maps through a multilayer perceptron (MLP) and a CNN predictor. To decouple the high-frequency scene details from the rain streak, we propose an adaptive angle estimate strategy that significantly improves the distinguishability of rain by considering the directions along and perpendicular to the rain streak. We use the obtained angle information to design a gradient rotation loss. Combining our proposed network framework and unsupervised loss, we employ an alternating optimization strategy to update network parameters and rain embeddings. Additionally, to address the lack of multi-view rainy scene datasets, we render 10 sets of scenes using Maya [1] , resulting in more consistent and realistic rain trails compared to data simulated by simple methods.In summary, the main contributions of this paper lie in:• we explore priors in neural rendering processes and propose a general rainy scene reconstruction framework; • we represent the rain in the scene using rain embeddings and use a predictor to predict rain streaks; • we introduce an adaptive scene rain streak angle estimate strategy and a corresponding gradient rotation loss for decoupling scene high-frequency details and rain streaks; and • we construct a multi-view rainy scene dataset for more realistic and consistent rain streaks. The combined rain embedding is processed through an MLP to obtain latent space representations, which are then fed into a CNN predictor to produce a rain map. The framework is trained using unsupervised losses that facilitate the decoupling of high-frequency scene details and rain streaks, yielding a rain-free neural radiance field. 
related work 2.1 neural rendering: 
 NeRF [29] revolutionized viewpoint generation by introducing differentiable volume rendering and learnable radiance fields, enabling joint optimization of geometry and appearance for static 3D scenes from posed RGB images. Subsequent works explore various designs to improve NeRF's training or inference speed, such as MLP capacity [34] , space discretization [10] . Recently, Kerbl et al. [19] extended the NeRF to explicit GPU-friendly 3D Gaussians Splatting (3DGS) and replaced the neural rendering, achieving real-time rendering of radiance fields. Other works focus on enhancing rendering quality and addressing challenging input scenarios. For instance, PixelNeRF [48] enables few-shot reconstruction, Mip-NeRF [2] improves rendering under multi-scale inputs, Deblur-NeRF [27] handles blurry images using canonical kernels, and NaN [31] leverages inter-view and spatial information to deal with noisy data. HDR-NeRF and RawNeRF [16, 28] utilize camera response functions and raw images to process low dynamic range inputs. In contrast, our work tackles the novel task of rainy scene reconstruction using decoupled neural rendering. We discover that the neural rendering prior is well-suited for rainy scene reconstruction and propose a general framework based on this insight, setting our approach apart from existing NeRF-based methods.Single-image rain removal is an ill-posed problem that aims to decompose a rainy image into a clean background and a rain streak layer. Traditional methods rely on prior characteristics of rain, such as photometric properties [13] , morphological component analysis [18] , non-local means filtering [20] , and layer priors [24] to tackle this challenge. Other approaches employ optimization techniques, including sparse coding [26] , Gaussian mixture models [46] , and low-rank models [8] , to separate rain streaks from the background. With the advent of deep learning, numerous learning-based methods [5, 7, 11, 15, 23, 30, 32, 38, 39, 41, 49, 51] have emerged, significantly advancing the state-of-the-art in single image rain removal performance. Video provides additional temporal information that can be exploited for rain removal. Garg et al. [12] pioneered the task using photometric properties. Chen et al. [4] employed superpixel segmentation and a robust deep CNN for restoration. Li et al. [22] proposed a multi-scale convolutional sparse coding model to capture repetitive local patterns and rain streaks of different scales. Yang et al. introduced a two-stage recurrent framework [44] and extended it to a self-learned approach [45] leveraging temporal correlation and consistency. Yue et al [50] proposed a semi-supervised framework with a dynamical rain generator. Yan et al. developed SLDNet+ [43] , an augmented self-learned deraining network utilizing temporal information and rain-related priors, and combined single-image and multi-frame modules for raindrop removal. Zhang et al. [52] proposed ESTINet, an efficient end-to-end framework based on deep residual networks and convolutional LSTMs, to capture spatial features and temporal correlations among successive frames. Xiao et al. [25] introduced a teacher-student framework for adaptive nighttime video deraining.In addition to video-based methods, some works tackle structured multi-view image rain removal. Ding et al. [9] proposed a GAN-based architecture utilizing depth information to remove rain streaks from 3D EPIs of rainy light field images (LFIs). Yan et al. [42] employed 4D convolutions and multi-scale Gaussian processes for LFI rain removal.In this work, we employ the classic Neural Radiance Fields (NeRF) [29] as the neural rendering method to describe our work more concisely and elegantly. NeRF is a novel view synthesis method that learns a continuous, volumetric representation of a scene through differentiable volume rendering. NeRF employs a multilayer perceptron (MLP) network 𝐹 Θ to parameterize a 5D input (3D position x = (𝑥, 𝑦, 𝑧) and 2D viewing direction d = (𝜃, 𝜙)) into a density 𝜎 EQUATIONTo render a novel view, NeRF casts a ray r(𝑡) = o + 𝑡d for each pixel, where o is the camera origin and d is the ray direction. The expected color 𝐶 (r) of the ray is computed by integrating the radiance along the ray:EQUATIONwhere𝑇 (𝑡) = exp(- ∫ 𝑡 𝑡 𝑛 𝜎 (r(𝑠))𝑑𝑠)is the accumulated transmittance along the ray, and [𝑡 𝑛 , 𝑡 𝑓 ] is the near and far bounds of the ray. In practice, the continuous integral is approximated using numerical quadrature:EQUATIONwhere 𝑇 𝑖 = exp(-𝑖 -1 𝑗=1 𝜎 𝑗 𝛿 𝑗 ), 𝛿 𝑖 = 𝑡 𝑖+1 -𝑡 𝑖 is the distance between adjacent samples, and 𝑁 is the number of samples along the ray. 
proposed method: 
 As aforementioned, given a set of multi-view rainy images, we aim to reconstruct a rain-free scene in an unsupervised manner. As shown in Fig. 1 , the proposed framework, named RainyScape, consists of a neural rendering module based on the NeRF architecture and a rain prediction module built upon MLP and CNN layers. To be specific, the neural rendering module takes ray positions and view directions as input to estimate color and density values. The rain prediction module explicitly models the rain characteristics, which consumes a learnable rain embedding composed of scene state vectors and viewpoint state vectors, as well as provided camera parameters, to predict a rain map. During training, we alternately optimize the entire framework and use unsupervised losses to encourage the decoupling of high-frequency scene details and rain streaks. This enables the reconstruction of a rain-free radiance field, which can be directly rendered during inference to obtain sharp, rain-free views. Moreover, our framework can be adapted to other rendering pipelines, such as the recent 3D Gaussian Splatting (3DGS), as experimentally demonstrated in Section 5.Remark. Our RainyScape differs from existing image/videobased deraining methods in several key aspects. First, those methods rely on image space operations and do not fully exploit the scene's 3D geometry. Second, most of them require ground-truth rainfree data as supervision. Third, those methods often struggle to handle multi-view inputs with large baselines. Furthermore, by reconstructing a rain-free radiance field, our RainyScape enables the rendering of novel views, which is not applicable to existing ones.The task of deraining aims to decompose a rainy image I ∈ R ℎ×𝑤×3 into a rain-free background scene B ∈ R ℎ×𝑤×3 and a rain layer R ∈ R ℎ×𝑤×3 [35, 37] , which can be formulated asEQUATIONInitially, we leverage the neural rendering prior, which is fundamentally introduced by the spectral bias property of neural networks [33] , to obtain a low-frequency scene representation B 𝑙 ≈ B during the warm-up stage, as illustrated in Fig. 2 . Specifically, at this stage, the rainy image I can be decomposed into the low-frequency scene content B 𝑙 and the high-frequency information I ℎ , as shown in the residual map in Fig. 3 , which includes both rain streaks and useful scene details:EQUATIONTo obtain a more detailed background B from the initial lowfrequency representation B 𝑙 , we need to further decouple the highfrequency information I ℎ into rain streaks and detailed scene components. To achieve this goal, we next introduce a rain prediction module that receives and reflects the rain characteristics from the rainy scene.To effectively model the rain characteristics and decouple the highfrequency information I ℎ into rain streaks R and detailed scene components B ℎ , we introduce a rain prediction module 𝐸 Φ . This module consists of a three-layer MLP to extract a latent space representation of the rain characteristics and a six-layer CNN to process and upsample feature maps to obtain a rain map. It takes learnable rain embeddings as input and predicts a rain map R 𝑒 ∈ R ℎ×𝑤×3 .The rain embeddings comprise three parts: learnable scene state vectors s ∈ R 128 , learnable viewpoint state vectors v ∈ R 64×𝑛 , and fixed camera parameters p ∈ R 16×𝑛 , where 𝑛 is the number of input views. For a specific view 𝑖, the predicted rain streaks can be expressed as These parameters are designed to capture and represent essential information about the rain in the scene that influences the appearance of rain streaks. The rainy scene state vectors s represent global rainy characteristics, such as the scale and direction of the rain. The viewpoint state vectors v record view-specific characteristics, allowing for subtle changes in the state of rain between viewpoints. The camera parameters p represent the viewing direction and position, affecting the rain streaks' orientation and perspective.EQUATIONIn addition to the proposed rain prediction model, we propose a novel unsupervised loss function to help the network effectively decouple high-frequency details and rain streaks. A key component of our loss function is the adaptive gradient rotation loss, which plays a crucial role in distinguishing rain streaks from background textures.Adaptive Gradient Rotation Loss. As illustrated in Fig. 3 (a), angle information can be used to differentiate rain streaks from background images. Unlike existing works, such as [17] that shifts the image through several preset angles and [49] that requires pretraining an angle estimation network, we propose an adaptive rain streak angle estimation strategy based on several experimental observations:• The residual map I -B 𝑙 contains high-frequency details and rain, with the angles of high-frequency details in the scene being relatively evenly distributed.• The gradient direction of the rain is dominant after suppressing the minimum gradient. • For local image patches, there is a high probability of rain occurring in only one direction. Based on these observations, we calculate the orientation of the residual image gradients and construct a histogram to identify the dominant rain streak directions. Considering that the discrimination does not change significantly within a small angle range, we discretize the orientation range into 60 bins, each spanning an angle of 3 degrees for robustness. Additionally, for the special case of rain with multiple angles, we can obtain multiple angles by adjusting the number of top angles considered.After adaptively obtaining the rain angle, for each dominant direction 𝜃 , the adaptive gradient rotation loss is given byEQUATION)where 𝐾 is the number of dominant directions, typically set to 1, and ∇𝜃 denotes the gradient operator along the direction 𝜃 . Likelihood Loss. The likelihood loss measures the discrepancy between the reconstructed image B + R and the input rainy image, promoting stable network performance:EQUATIONwhere 𝜎 is the standard deviation of the residual, and 𝜖 is a small constant to prevent division by zero. Reconstruction Loss. The reconstruction loss further emphasizes the consistency between the reconstructed image and the input rainy image:EQUATIONTotal Variation Loss. The total variation loss encourages the rendered background image to be smooth, reducing artifacts and noise:EQUATIONwhere ∇ 𝑥 and ∇ 𝑦 denote the horizontal and vertical gradient operators, respectively.The overall loss function is written asEQUATIONwhere 𝜆 1 , 𝜆 2 , 𝜆 3 , and 𝜆 4 are the weight coefficients for the corresponding loss terms.We propose an alternating optimization approach to train our framework effectively, as outlined in Algorithm 1. The optimization process consists of two main stages: network update and latent variable update. During the network update stage, we predict rain maps using the predictor 𝐸 Φ , render background rays using the NeRF network 𝐹 Θ , and update the network parameters Θ and Φ based on the defined loss functions Eq. ( 11) . In the latent variable update stage, we freeze the networks and update the latent variables s and v using Langevin Monte Carlo [14] , which introduces random perturbations to explore the latent space and discover better representations for fitting the input rainy image I. The optimization process iterates until convergence, alternating between the two stages, with Network Update: Predict rain maps using 𝐸 Φ , Render background rays using 𝐹 Θ , and Update network parameter Θ and Φ via Eq. 11.Latent Variable Update: Freeze 𝐹 Θ and 𝐸 Φ . Update s, and v using Langevin Monte Carlo to fit I via Eq. 11. a warm-up stage to initialize the NeRF network before introducing the rain prediction module. By leveraging this approach, our framework learns to separately represent the background scene and the rain layer, progressively refining the deraining results without relying on ground-truth supervision. 
experiments 4.1 proposed dataset: 
 We construct a comprehensive and diverse multi-view rainy image dataset using Maya [1] . As illustrated in Fig. 4 , our dataset consists of 10 scenes, each captured from 50 viewpoints with 1024×1024 resolution. To ensure the view consistency of raindrops, we construct 3D models of raindrops, distribute them randomly within the scene, and render them together with the scene objects. This approach enables the generation of interaction effects between raindrops and light rays, generating more realistic rainy scenes compared to simulation methods that directly add rain streaks to 2D images. The dataset encompasses a wide range of rain densities, directions, and streak orientations to accurately represent real-world variations.The scenes are composed of outdoor objects such as cars, bridges, roads, buildings, and boats to facilitate generalization to unseen rainy environments. The 50 viewpoints are strategically distributed on a spherical surface to maximize the overlap rate of scene content captured by different cameras, providing diverse information for rendering. The dataset includes rainy and ground truth images, depth maps, and camera parameters, making it suitable for a wide range of viewpoint synthesis tasks. We will make the dataset publicly available.Our rain prediction module consists of a 3-layer MLP with a feature dimension of 128 channels, which generates a 1024-dimensional latent space representation. This representation is then reshaped into an image and processed using a 6-layer convolutional network, followed by super-resolution to obtain the required 64 × 64 random patches. For the neural rendering component, we utilize the Py-Torch re-implementation NeRF [47] with a batch size of 4096 rays, each ray undergoing 64 coarse samplings and 64 fine samplings.To further validate our approach, we also conduct experiments using the official implementation of 3D Gaussian Splatting (3DGS) [19] with its default configuration. During model optimization, we employ the Adam optimizer [21] with its default settings. For NeRF, we use the default learning rate update strategy, with an initial learning rate of 5 × 10 -4 . After the warm-up period, the learning rate is set to 1 × 10 -6 . The learning rates for the MLP and CNN part in the rain prediction module are set to 1 × 10 -3 and 1 × 10 -4 , respectively. For latent variable updates, we perform 5 updates per epoch, with the first two updates using Langevin Monte Carlo to introduce random perturbations. The loss coefficients 𝜆 1 , 𝜆 2 , 𝜆 3 , and 𝜆 4 are set to 0.1, 500, 0.5, and 1, respectively. We optimize a single model for 18K iterations on a single NVIDIA A6000 GPU.We set up two comparison baselines that leverage supervised information for rain removal. First, we fine-tune two state-of-the-art rain removal methods, namely the single-image rain removal method DRSformer [7] and the video-based rain removal method ESTINet [52] , using six randomly selected scenes (Ruins, Sidewalk, Factory, Bridge, Corner, and Alley). These fine-tuned models are then employed as pre-processing steps for the input images. Subsequently, we train NeRF on the pre-processed images and refer to these two approaches as DRSf+NeRF and ESTI+NeRF, respectively. Additionally, we include a baseline that directly applies the original NeRF [29] to the rainy data without any pre-processing or modifications. We employ widely-used metrics to quantitatively evaluate the performance of our method and the baselines. Peak Signal-to-Noise Ratio (PSNR) measures the image quality and rainy scene reconstruction performance, with higher values indicating better results. Structural Similarity Index Measure (SSIM) assesses quality via structural information, luminance, and contrast, with higher values indicating better results. Perceptual similarity (LPIPS) measures the similarity in feature space, with lower values indicating better results.Quantitative results. Table 1 showcases the performance comparison of our proposed method against three baselines (DRSF+NeRF, EIST+NeRF, and NeRF) on four scenes (Crossroad, Square, Sailboat, and Yard). The "Supervised" column in the table indicates whether the method utilizes supervision information. Our method achieves the best performance on the majority of the scenes and metrics (highlighted in bold), outperforming the baselines in most cases. On average, our approach obtains a PSNR of 29.88 dB, surpassing the second-best method, DRSF+NeRF, by 1.47 dB and the original NeRF by 2.95 dB. We also achieve the highest average SSIM and second-best average LPIPS scores, indicating superior structural preservation and perceptual quality. It is worth noting that our method achieves state-of-the-art performance without relying on any supervision information. These quantitative results validate the effectiveness of our novel contributions and highlight its potential to render high-quality clear images across diverse rainy scenes.Qualitative results. Fig. 5 presents a visual comparison of the rendering results obtained by our method and the baseline approaches. Our approach effectively renders rain-free images while accurately capturing and separating the rain streaks from the scene content. The error maps demonstrate our method's superiority in recovering high-frequency details and preserving textures and edges compared to the baselines. By explicitly modeling and decoupling the rain streaks from the high-frequency content, our framework can effectively transfer useful scene details to the radiance field, resulting in improved restoration quality. In contrast, the baseline methods often struggle to recover fine details, introducing artifacts or losing important texture information. Since the rain streaks are relatively small compared to the scene, please zoom in for better observation of the details in all figures. Post-processing with predicted rain. The accuracy of the predicted rain streak images, as demonstrated in Fig. 5 and Fig. 7 , enables further enhancement of the deraining performance through post-processing. When the rainy image is provided, we can apply a simple thresholding method to convert the rain streak image into a binary mask, which allows us to identify and selectively fuse regions unaffected by rain with our rendered image. This selective fusion approach leads to an improvement in the overall deraining performance. Fig. 6 illustrates the performance progression of different methods, including our post-processing step, during training. By combining the information from the rain-free regions with our RainyScape rendering results, we achieve a further performance improvement of approximately 1 dB.Rain Prediction. To demonstrate the effectiveness of our rain prediction module, we first visualize the evolution of the predicted rain streaks during training in Fig. 7 . As the training progresses, the predicted rain streaks gradually resemble the real rain streaks more closely, indicating that our model effectively learns to capture the characteristics of scene rain. Furthermore, we investigate the impact of the rain embedding on the predicted rain streaks. In Fig. 8 , we present the results of adding random Gaussian noise with a variance of 0.5 to the scene state vector s and viewpoint state vector v, denoted as Δ 𝑠 and Δ 𝑣 , respectively. The results demonstrate that the added noise influences the intensity and density distribution of the predicted rain streaks, suggesting that the rain embedding plays a crucial role in controlling the appearance of the predicted rain. In addition, we show the effect of changing the camera parameters, denoted as Δ 𝑝 . The predicted rain adapts to the changes in camera parameters, demonstrating the ability to predict rain streaks that are consistent with the scene information and camera setup.Loss configuration. To investigate the contribution of each term in our loss function (Eq. ( 11)), we conduct an ablation study by removing the reconstruction loss L 𝑟𝑒𝑐 , total variation loss L 𝑡 𝑣 , and adaptive gradient rotation loss L 𝑎𝑑𝑔 terms individually. The likelihood loss L 𝑙𝑙 is kept in all configurations to ensure stable training. Additionally, we explore the impact of varying the number of bins used in the adaptive gradient rotation loss, considering 30, 60 (default), and 90 bins. All the results are shown in Table 2 . Notably, when the L adg is not used, the network fails to predict visually realistic rain streaks, emphasizing its critical role in capturing the directional characteristics of rain.Framework Effectiveness. We compare our framework with two alternative approaches to demonstrate its effectiveness. The first approach, Two-NeRF, uses separate neural rendering fields for rain and background, similar to Dehaze-NeRF [6] . However, as shown in Fig. 10 , Two-NeRF fails to effectively decouple rainRainyScape-3DGSPredicted rain E-3DGS E-RainyScape-3DGS from the background, while our method accurately distinguishes between the two components. We also evaluate Deblur-NeRF [27] , which adjusts and fuses rays to handle motion blur. As highlighted in Fig. 10 , Deblur-NeRF severely damages the scene structure, erroneously removing objects like utility poles. 
extension to 3d gaussian splatting: 
 3D Gaussian splatting (3DGS) [19] has emerged as a promising technique in the field of radiance field rendering, offering high efficiency and impressive visual quality. In this section, we explore the extension of our RainyScape framework to incorporate 3DGS as the rendering module. By leveraging the capabilities of 3DGS, we aim to enhance the deraining performance while maintaining the computational efficiency and visual fidelity of the rendered results.To extend our framework to 3DGS, we replace the NeRF rendering component in the pipeline with the 3DGS architecture, getting RainyScape-3DGS, while the rain prediction modules remain unchanged. To ensure optimal performance, the warm-up stage is set to 4K iterations. Subsequently, joint training of the entire framework is performed using the default configurations and loss functions. We evaluate the effectiveness of RainyScape-3DGS using the same datasets. Table 3 presents the quantitative results, proving that our RainyScape framework can significantly improve the performance of 3DGS in rainy scenes. Fig. 9 provides visual comparisons, showcasing the ability of our framework to suppress the rendered rain in 3DGS and more visually appealing rain-free images. 
conclusion: 
 In this paper, we introduced RainyScape, a novel unsupervised framework for reconstructing scenes from multi-view rainy images using decoupled neural rendering. Our approach tackles the challenges of rain streak removal in the rendering process by integrating a low-frequency scene representation, a rain prediction module with a learnable rain embedding, and an unsupervised rainy scene reconstruction loss that incorporates an adaptive gradient rotation loss. Extensive experiments on both classic NeRF and stateof-the-art 3DGS demonstrate the effectiveness and versatility of RainyScape in generating clean, visually appealing images with sharp details and accurate scene geometry.  
 
 Appendix:  


Decision: This paper presented a method for rainy scene reconstruction from multi-view images. A neural rendering module and a rain prediction module were proposed to achieve the goal. A multi-view rainy dataset was also contributed. The paper received comments from four reviewers. The strengths of the paper include: the idea of rainy scene reconstruction is interesting, the contributed dataset, and the good performance. Limitations include:  unclear descriptions for some claims/statements, potential applications, insufficient comparison with other related methods, and some writing issues. The paper received mixed recommendations (both negative and positive) before the rebuttal, but after the rebuttal, most concerns were well addressed and the recommendation scores were increased.

Considering the above, the AC would recommend Accept, but suggests the authors carefully revise their paper according to the comments from all the reviewers.

Reviews: review: paper_summary: paper_summary: This paper introduced an unsupervised framework for reconstructing clean scenes from multi-view rainy images. RainyScape includes a neural rendering module and a rain-prediction module with a predictor network and a learnable latent embedding for rain characteristics. By leveraging the spectral bias property of neural networks, the neural rendering  is optimized to obtain a low-frequency scene representation. Extensive experiments show that it eliminated rain streaks and rendering clean images.
strength_weakness: strength_weakness: It provided a clear picture to show the proposed RainyScape framework.
It provided diverse applications.
limitations: limitations: Sections 3.1 and 3.2 have little novelty, with the main contributions coming from the unsupervised loss functions. However, the paper lacks clear motivations, significantly reducing its overall quality.

Why do the PSNR-SSIM-LPIPS results of the proposed method only start to show improvement around epoch 9000?

PSNR, SSIM, and LPIPS are typically used to evaluate image noise quality. Are they appropriate metrics for measuring deraining performance?

It is difficult to discern details in Figures 7 and 8. What is the specific purpose of Figure 7?

Many deraining papers are published annually, and this paper lacks compelling reasons for acceptance. Additionally, it is only marginally related to MM, and  suitable for CV.
paper_summary: paper_summary: This work introduce an adaptive scene rain streak angle estimate strategy and a corresponding gradient rotation loss for rainy scene reconstruction. In addition, it provides a datasets.
strength_weakness: strength_weakness: The visualization results seems good, and the proposed Adaptive Gradient Rotation Loss is reasonable and is detailed.
limitations: limitations: 1. What is the application environment of rainy scene reconstruction task? In the introduction, you only show the importance of your task by listing previous other tasks. I am interested in understanding the real-world scenarios where this task would provide substantial benefits or improvements over existing solutions.

2. I do not have a clear understanding about how to get the low-frequency scene content B_l and the high-frequency information I_h. Does this work manually choose a image in one epoch to obtain them? What exact the epoch? Is the epoch selected for each scene the same? Can it be selected automatically?

3. Adaptive Gradient Rotation Loss is proposed based on the assumption that the gradient direction of the rain is dominant after suppressing the minimum gradient. If the rain line is smaller than it in the image of the article, will the gradient direction still be dominated by rain? Does this method still work?

4. Is there any comparison with the single-image rain removal method alone? In Figure 5, you provide the result combining single-image rain removal method and Nerf. I do not know whether the image is novel view or input view. Therefore, I want to see the compare result of input view, if we only use rain removal methods.

5. Since one of the contribution is that representing the rain in the scene using rain embeddings, can this work provide quantitative ablation study to learnable scene state vector s, learnable viewpoint state vectors v , and fixed camera parameters p, separately?
paper_summary: paper_summary: The paper introduces the RainyScape framework, which combines neural rendering and rain-prediction modules to achieve unsupervised clear scene reconstruction from multi-view rainy images. It utilizes neural rendering priors to obtain a low-frequency scene representation by optimizing the neural rendering pipeline, effectively removing rain streaks and generating clear images. 
The paper introduces an adaptive gradient-based reconstruction loss to drive the joint optimization of the two modules. 
The paper constructs a multi-view rainy scene dataset for more realistic and consistent rain streaks.
Experimental results demonstrate the efficacy of the proposed method.
strength_weakness: strength_weakness: 1. The authors generate a diverse and high-quality multi-view video derain dataset for training and evaluation purposes in the context of rainy scene reconstruction.
2. The Adaptive Gradient Rotation Loss enhances the network's ability to effectively decouple high-frequency details and rain streaks. The proposed strategy in the paper is adaptive and based on experimental observations.
limitations: limitations: 1. The reviewer recommend the authors to provide more detailed explanations about the operations of rain embedding to enhance the understanding of readers.
2. There are some typos in this article. For example, there is an extra 'and' on line 111. Please further refine.
3. It is recommended to include additional references from recent works in video deraining to enhance the comprehensiveness of paper. Here are some suggested references:
(1) Sun S, et al. Event-aware video deraining via multi-patch progressive learning[J]. IEEE Transactions on Image Processing, 2023.
(2) Wang J, et al. Unsupervised Video Deraining with An Event Camera[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
(3) Wu H, et al. Mask-Guided Progressive Network for Joint Raindrop and Rain Streak Removal in Videos[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023.
4. Additionally, it is better to compare your work with more representative video deraining methods, such as SLDNet, and other NeRF-based video restoration methods, if available.
paper_summary: paper_summary: This work proposes to reconstruct rainy scenes under Neural Rendering in an unsupervised spirit. They design a framework RainyScape, which introduces rain embedding to predict rain streaks and an adaptive gradient rotation loss to decouple high-frequency details and rain streaks. A new multi-view rainy scene dataset is also constructed for better benchmarking.
strength_weakness: strength_weakness: It is interesting to investigate rainy scene reconstruction under NeRF and 3DGS.
The dataset is a great contribution to the community of image deraining.
It is practically useful to develop an unsupervised method for image deraining, which does not need paired data.
Superior results are achieved in diverse scenarios.
limitations: limitations: 1. The contribution (ii) should be described in detail. Its motivation and novelty are unclear. Why do we need rain embedding? Could you provide more description on this point?
2. The proposed gradient rotation loss, the author states the characteristic of 'adaptive', differing from existing work. More details and motivation should be given to explain 'adaptive'.
3. More details and analysis of Fig. 5 are needed. It is hard to observe that the proposed method is superior in recovering high-frequency details and preserving textures and edges from the displayed error maps. How the error maps come from?
4. Analysis on the sensitivity of balancing hyper-parameters is missing. It would be great to investigate how the proposed Adaptive Gradient Rotation Loss contributes to the results.