Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a CVAE(conditional variational auto-encoder) based method of single image rain removal. The key idea is to use the generative network VAE to learn the probability distribution of the pixel-wise rain status P(R|F,Z) and the derainning kernels P(K|F, R, Z), where R, F is the status map and the kernels, respectively. F are the extracted features of the input image, and Z is the latent variables generated by the normal distribution P(Z). For the inference stage, they use the rainy image I to compute F, and randonly generate m Zs. By using F, m Zs, they compute m Rs and consequently m Ks. The final derain kenerl map K is the mean value map of m Ks. The experimental results demostrate the effictiveness of the proposed method on the deraining benchmarks of Rain100H, Rain100L, Rain1400, SPA, the snow removal task dataset of Snow100k, and the haze removal task dataset of ITS&OTS. 

strengths_and_weaknesses: The proposed method is an extension of [Du et.al. TIP 2020]. Instead of using CVAE to directly estimate P(Y|F,Z) as Du et. al., they formulate the problem as P(R, K| F, Z) and factorize the distribution P(R, K| F, Z) = P(K|R,F,Z) P(R|F, Z), so that they can decouple the latent factors to learn P(R|F,Z) and P(K|R, F, Z), respectively. They experimental results show the effictiveness of the proposed formulation. 

However, the paper presentation is lack of clarity. There are many places in the method(both formulation and architecture) which are not clear. Especially about information decoupling, the authors should give a more explaination about why they use the corrent formulation of K (EQ6~9), and how it can get correct K as ground truth.   
questions: Please provide the clear defination of the function "conv" in EQ3, EQ6. If it is a convolution, it should have two varibles but not one. if not, please give the clear explaination. 
Please explain the "encoder", "condition", and "decoder" operations in the paper. If they are CNN blocks, please decribe them clearly, especially where the mean and standard deviation are computed from by the "condition" block ( condition(F) in EQ4,  condition(F, R) in EQ12). 
Please explain the equations of the information decoupling. Please explain the conv function in Eq6 and fig. 2. Is the conv a fixed numarical fountion? How do you promise the current formulation get correct K of  the input pair?      

Please provide more explaination about the discriminative networks they used in the comparison experiments. No information is provided in the current draft. 
limitations: The authors have not provide discussion about the limitations or failure cases of their work. The authors should include such discussions. In ablation study, more discussions and analysis are needed about the comparison of the proposed CVAE with GAN or other discriminative methods.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Paper proposes an image deraining method where they compute pixel status in status space that indicates whether the pixel is belonging to rain or an object.  this information is used in the decoupling layer to construct kernel space for the pixel to derain and recover a clean background.
strengths_and_weaknesses: Weaknesses:
- Can the proposed method handle multiple degradations like images with rain and fog or images with fog and snow.
- comparison with a few recent papers is missing
[a] Tu et al., Maxim: Multi-axis mlp for image processing, cvpy 2022
[b] Valanarasu et al., Transweather: Transformer-based restoration of images degraded by adverse weather conditions, cvpr 2022.

- can authors elaborate or provide pseudo-code for training and testing
- Proposed methods seem complex to implement, will the code be made public

questions: Please refer weaknesses
limitations: Please refer weaknesses
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper discusses a method for image rain removal based on two CVAE models. The first CVAE learns the distribution of rain condition (status), conditioned on the image features. The second CVAE learns the distribution of kernels that are used to remove rain effects, conditioned on the rainy status and the image features. During testing, multiple status maps and kernels are repeatedly sampled from the distribution to remove rainy effects from the input image. 
strengths_and_weaknesses: Strengths:
The technical details are clear. The idea of using CVAE to learn the distribution of status estimation and the corresponding restoration kernels is novel. The results are strong and the framework seems to work well for not just rain removal but also other image restoration tasks including snow, haze, and shadow removal. 

Weaknesses:

My main concern is that it is unclear to me how the method works. Conceptually, I think we aim to correctly estimate the status maps and the kernels, given the image feature? With the correct status map and kernels then we can remove rainy effects properly. However, the two CVAEs sample several status maps as well as multiple kernels, conditioned on the image features. I don’t understand how these randomly sampled status maps approximate rainy status in the input image. Should an input rainy image be associated with exactly one status map? What does it mean (conceptually) to randomly sample multiple status maps given a single input image? Moreover, I also don’t understand why taking the average of randomly sampled kernels is better than one kernel. While I am aware that the authors have provided quantitative evidence in the s.m. that using these CVAEs is better than using regression models directly, I don’t think the authors have provided proper explanations for the effectiveness of their method.

The motivation also doesn’t make much sense to me since I can’t connect the motivation to the proposed method. Why the “confusing information of the pixels” can be resolved by sampling multiple pseudo rainy status maps? 

The shadow removal experiment should be conducted on the Adjusted ISTD dataset (shadow image decomposition for shadow removal - ICCV19) since the original ISTD dataset exhibit a major color shift between input and GT images.

questions: Please see the weaknesses. 
limitations: Some limitations have been discussed in the s.m. It's unclear to me when the method doesn't work well in comparison to other approaches and why.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: A CNN structure based on the combination of two Conditionally Variational Auto-Encoders (CVAE) is proposed for image rain removal. Two CVAE are used, one for generating hypothesis on rain presence at a pixel and on for the scale of the rain steak. The proposed structure is tested on several database and compared to several others methods. It is also applied separately to snow, haze and shadow removal.  
strengths_and_weaknesses: The proposed structure is interestingly introduced and seems original. The evaluation seems substantial and show improvements over 5 others methods. It is interesting that the proposed structure can work on other applications such as snow, haze and shadow removal.  
questions: In the supplementary material, shown maps of rainy pixels have numerous errors on bright objects. May it be possible de comment on this point ? 
One may notice that the size of the test database is small or very small compared to the size of the learning databases in several cases in the experiments. This may induce a lack of confidence on the proposed results that should be interesting to discuss ?
 
limitations: No societal impact.
review_title: null

Meta-review: metareview: The paper has received positive reviews. There was substantial discussion, and the authors are strongly encouraged to include the clarifications they made rot the final copy, as well as a more extensive discussion of the limitations and possible danger of overfitting.

confidence: Certain
Decision: Accept