Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method to learn neural radiance fields that represent the underlying scene free of reflective components in the scene, i.e. explicitly represented the transmitted regions of the scene. Prior work in representing transmitted radiance field relies on reflection removal from the input image sequence, however this is a challenging problem and typically results in photometric inconsistencies. The proposed method uses a novel formulation leveraged on the observation that reflective components in the radiance field are sparser than the transmitted components. A patch-based rendering scheme is used to handle the local characteristics of reflective/transmissive components. 
strengths_and_weaknesses: Strengths:
The paper is well written and the exposition is clear. The paper provides a through introduction and a motivation for the solution, before properly explaining the proposed solution. As such I find the paper to be a useful contribution to the community and beneficial for the reader. 
The use of the transmission encoder with pyramid-scale features is interesting and the choice of Wg and Wl is properly motivated. 
The recurring edge constraints are the core strength of the paper and the description provided in section 4.2 is succinct. 
The qualitative and quantitative results in the paper and supplemental material clearly demonstrates that the transmitted radiance field is captured free form noise due to reflection. 

Weaknesses:
The authors rightly point out that weighting coefficients are dependent on several factors. The viewing direction (wrt lights in the scene) and the camera position are correlated and more discussion is warranted on whether an MLP that encodes the weighting coefficients is sufficient in general. 
questions: It would be an interesting study to see the performance of this method as the sparsity assumption is slowly relaxed, where the reflective components in the scene change from transmissive glass plates on one end of the spectrum to fully silvered mirrors on the other. Have the authors performed any experiments on such scenes? 

While occlusions are one of the main limiting factors, occlusions in the reflected part of the scene however are not likely to cause a problem in my opinion, since the edge constraint assumption and the sparsity assumption are still valid? 
limitations: Yes, the authors discuss the limitations of the work
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper targets to solve the novel-view synthesis problem with reflection removal, that is, novel-view synthesis of a transmitted object from images corrupted by reflections. A naive baseline, that applies reflection removal techniques to each input image before NeRF, does not solve the problem as the resultant image would not be multi-view consistent; This is because most reflection-removal techniques cannot take advantage of multiple viewpoints. This paper solves this problem by introducing 1) transmission feature integration and 2) recurring edge constraints. First, Transmission feature integration is based on the idea of pixel-NeRF that the feature from other viewpoints can assist the training, and the paper used “transmission feature” instead of the vanilla pixel feature in pixel-NeRF. Second, recurring edge constraints are based on the assumption that a reflected component is sparse in its presence in the aligned image. The paper also collected a new dataset for real multi-view images corrupted by reflections, and the proposed method shows promising results. 
strengths_and_weaknesses: ### Strengths
- Promising results. The proposed method shows promising results on real multi-view images corrupted by reflections. The comparison with other methods such as NeRF, NeRF-W, and RR + NeRF, also shows that the proposed method performs superior both qualitatively and quantitatively, especially when the number of input images is limited.
- New dataset of multi-view images with and without reflections. The paper shows the newly collected multi-view images, which can facilitate further research on multi-view reconstruction and reflection removal.

### Weaknesses
While the paper proposes an interesting method with promising results, there are some weaknesses that can be improved:
- The presentation of the manuscript can be improved. There are some ambiguous definitions or explanations:
    - [Line 123] What is transmission and reflection entanglement? If it means transmission and reflection have an inherent ambiguity, then the proposed method cannot disambiguate either. “Due to the absorption, reflection, and refractive effect ~” should be further clarified.
    - [Motion inconsistency] The terminology “motion inconsistency” (used frequently all around the paper including the abstract) used for recurring edge constraints is somewhat misleading. The key idea used for recurring edge constraints is that the reflected component may not exist in some viewpoints and thereby have a sparse presence. The reason for this phenomenon is the size of the reflector is limited, which causes the reflected object to be outside the reflector and disappear in some viewpoints. It has nothing to do with motion and thus the term “motion inconsistency” is not the appropriate term to describe the method. Maybe the reflected object is at a different depth from the transmitted object and moves differently in the image (e.g., larger disparity when it is further), but it is not the information that the proposed method directly uses. The description in the main paragraph (line 187-) is already clear, so just choosing a better terminology would improve the clarity of the proposed method.
    - [Line 210] What is \Psi? The notation seems to be not defined.
- Some important details about the transmission feature are missing. What network is used for feature W? From Line 155, I assume the network is based on ERRNet but it is difficult to see which part of the ERRNet is used as there are many components in the ERRNet. Line 162 is not enough for understanding the exact structure. Also, Line 232 explains the pretraining of the transmission encoder briefly and it is somewhat confusing if the method is different from the original ERRNet. The network structure and the training detail needs to be added to the supplemental material.
- Missing baseline. A baseline (that might be interesting) is missing, that is RR + pixel-NeRF (without transmission feature). One of the main contributions of this paper is using the transmission feature, which is the combination of 1) reflection removal and 2) pixel-NeRF (assist the training of NeRF). If these two parts are divided into the reflection removal part and the pixel-NeRF part, it can be another baseline of RR + pixel-NeRF, which will be a more fair and interesting baseline.
- REC has a limited performance (at least quantitatively). The second main contribution of this paper is using recurring edge constraints (REC), but the effect of REC seems to be marginal quantitatively as shown in the ablation study (Table 2). The PSNR  without REC is 22.48, which is almost the same as that of the complete model (22.75). It would be interesting to see how REC works in more challenging data.
questions: Additional experiments that may further demonstrate the robustness of the proposed method:

- Non-planar reflector. How would the proposed method work when the reflector is not planar? It seems that all the experiments are done with a planar reflector. Does any data include a non-planar reflector? When the reflector is not planar, the behavior of reflections in multi-view images will be very different as the reflected parts will not be aligned after warping. To be specific, when the reflector is planar, the reflected object is equivalent to the virtual object that is behind the reflector and thereby the reflected parts from different viewpoints will be located at the same pixels after warping. However, we cannot expect this alignment in the non-planar reflector as it will be projected differently depending on the viewpoint. This will affect the performance of REC that relies on the aligned edge after warping, and it would be interesting to see how the method works in the case of a non-planar reflector.
- Large reflector. What if the reflector is large enough and thereby the reflections exist in every viewpoint? (e.g., large window as in [9]). This case breaks the assumption used for REC that the reflection is sparse across the different viewpoints, and only “motion” inconsistency can disambiguate reflections and transmissions (though it is still ambiguous to determine which one is transmission).
- Ablation study of feature pyramid W_g and W_l. What if the transmission feature W is used without a pyramid?
- How is the threshold 0.6 in (12) determined?
limitations: The questions in the above section (Questions) include some limitations that are not handled in the paper: non-planar reflector and large reflector. The proposed method may not work for those cases of reflectors.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel neural radiance field rendering method that is dealing with specular reflection on the object’s surface. The proposed method aims at recovering only the transmission radiance behind the reflection. To that end, this paper proposes to prepare two dedicated networks, i.e. T-MLP and R-MLP, to learn the transmission features and reflection features. This is achieved by applying a single image reflection removal method to the training data to separate the background and the reflection. The learned transmission and reflection color radiance are then combined in a convex combination. In addition, in order to guide the learning of background high-frequency details, this method also applies recurring edge constraints which utilize the observation that background edges appear consistently in multiple different views. 
strengths_and_weaknesses: Strengths
1. This paper is generally well-written with clear motivation in the introduction section. It clearly defines the current problem and challenge left by existing NeRF-based methods, which is the reconstruction of scenes behind the transparent surfaces with specular reflection. 
2. The comprehensive experiments show that the proposed method consistently outperforms the state-of-the-art methods by a considerable margin, in both qualitative and quantitative evaluations. 
3. This paper proposes a new NeRF purpose dataset, which is particularly focusing on the scenes behind the specular reflection. The proposed dataset may impose a strong impact on future research in this area. 

Weakness
1. I found the performance comparison with respect to baseline method MVSNeRF is a bit unfair because the selected baseline methods are not designed to deal with reflection, and hence it tends to predict the reflected scene as is. Therefore, the quantitative PSNR results are much worse than the proposed method as expected. Especially, in Figure 3, MVSNeRF almost reconstructs the exact appearance of the target view. 
2. For a NeRF method, it is also important to know the performance of the proposed method applied to normal (non-reflective) scenes. Otherwise, the usage of the proposed method is just limited to reflective scenes.  In the submitted paper and supplementary material, all the examples and benchmark data are performed on the scenes with reflection. The authors are suggested to provide more comparison (quantitative) and real normal scene examples in the rebuttal period. 
3. What is the processing speed and the network complexity of the proposed method compared to baseline methods? In order to prove the effectiveness of the proposed method, it is crucial to verify that the performance gain is not coming from the extra number of parameters in the network as well as the pre-processed edge map and reflection purged features.  
4. From the ablation study, the recurring edge constraints (REC) only bring in very little improvement, but it is considered as one of the two contributions in the method section. It seems that the proposed method is not very effective. 
5. It is true that the proposed method outperforms other baselines on the reflective NeRF dataset by a large number. However, the method itself is quite straightforward with limited novelty. It is critical to understand the effectiveness of the proposed method by providing the performance comparison on normal datasets, and hence prove the validity of the proposed method. 


questions: 1. The transmission features W_g and W_i are in different size. How do they feed into the T-MLP network?

limitations: The limitation of the proposed method is to apply it to any normal scenes or NeRF datasets. If it cannot perform well on non-reflective scenes, the generalizability of the method will be the biggest limitation.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a novel view synthesis network specially designed for see-through scenarios. This paper introduces a transmission encoder, which separately estimates the transmission amount against the specular highlight's reflection. In addition, this paper introduces a recurring edge constraint to account for the frequency of edges. 
strengths_and_weaknesses: [Strengths]
+ The application and approach of the transmissive scenario sound interesting to me. The specular reflection on glass in the see-through scenario has been rarely discussed in the neural rendering field yet. I found that this new research problem is interesting. Existing solutions such as vanilla NeRF seem to fail when there is a specular reflection in input images, while the proposed method works properly. 

[Weaknesses]
- Even though the motivation of the proposed method sounds interesting, I'm not fully sure if this paper is completely developed and evaluated to solve the technical challenges. Specular reflection works very differently from transmission. For instance, when the camera motion occurs, the specular reflection and transmitted image move in opposite directions about the depth position of glass surfaces. The proposed model doesn't seem to account for the physical phenomenon. Instead, it just tries to separate the transmission and reflection along the given view vector, which is not physically plausible. This observation should be valid from a specific view angle. If the method accumulates multiple observations in a voxel grid, the accurate separation cannot be achievable by increasing the number of observations. I would like to hear more in the rebuttal.

- The evaluation of this paper is one of the weakest points. Except for the main results shown in the teaser, most results do not include strong specular reflection. According to the proposed formulation of the recurring edge constraint, the proposed method may work properly when there are strong contrast edges in the transmitted image. The main result of the picture frame is the case. In other cases, the results do not include any strong specular reflection. I think the results look very cherry-picking with a very small number of examples. I would like to see more results to validate the performance of the proposed method. 
questions: - As mentioned earlier, specular reflection moves in the opposite direction of the transmitted image when the camera moves. It causes structure-from-motion failures that are supposed to give the right camera pose information as input. This paper doesn't clearly mention how the camera pose is estimated from the input image. I might miss it. If then, please let me know where the information is. I'm worried about the impact of the camera pose when producing other methods' results. I would like to hear more in the rebuttal.
limitations: Limitations are clearly mentioned in the main paper.
review_title: null

Meta-review: metareview: This paper proposes a novel neural radiance field rendering method that is dealing with specular reflection on the object’s surface. The authors present a novel method to solve the limitation of the existing NeRF-based methods for the scenes behind the transparent surfaces with specular reflection. The review results are two A(7) and two BA(5). After carefully checking out the rebuttals and discussions, I recommend the paper to be accepted for this NeurIPS.
confidence: Certain
Decision: Accept