Neural Transmitted Radiance Fields
Authors: Chengxuan Zhu
Renjie Wan
Boxin Shi
Venue: NeurIPS 2022 Accept (2022)

Abstract: Neural radiance fields (NeRF) have brought tremendous progress to novel view synthesis. Though NeRF enables the rendering of subtle details in a scene by learning from a dense set of images, it also reconstructs the undesired reflections when we capture images through glass. As a commonly observed interference, the reflection would undermine the visibility of the desired transmitted scene behind glass by occluding the transmitted light rays. In this paper, we aim at addressing the problem of rendering novel transmitted views given a set of reflection-corrupted images. By introducing the transmission encoder and recurring edge constraints as guidance, our neural transmitted radiance fields can resist such reflection interference during rendering and reconstruct high-fidelity results even under sparse views. The proposed method achieves superior performance from the experiments on a newly collected dataset compared with state-of-the-art methods. 

Neural Transmitted Radiance Fields
Chengxuan Zhu
Renjie Wan
Nat'l Eng. Research Center of Visual Technology School of Computer Science
Department of Computer Science
Hong Kong Baptist University renjiewante renanewnn hkbu edu.
Peking University peterzhuêpku. edu.cn
Boxin Shi
Nat'l Eng. Research Center of Visual Technology
School of Computer Science
Peking University sh prriboii@ppkk.. www.:ekkk.uuuueccmmmmommo
Abstract
Neural radiance fields (NeRF) have brought tremendous progress novel view synthesis. Though NeRF enables the rendering of subtle details in a scene by learning from dense set of images, it also reconstructs the undesired reflections when we capture images through glass. As a commonly observed interference, the reflection would undermine the visibility of the desired transmitted scene behind glass by occluding the transmitted light rays. In this paper, we aim at addressing the problem of rendering novel transmitted views given set of reflection-corrupted images. By introducing the transmission encoder and recurring edge constraints as guidance, our neural transmitted radiance fields can resist such reflection interfer- ence during rendering and reconstruct high-fich high-fidelity results even under sparse views. The proposed method achieves superior performance from the experiments on a newly collected dataset compared with state-of-the-an methods. Our code and data is available hhttp://gittuubcomm// http:///itcuubcccooeeeBttttttt https////eeeeeut///e/////////e/
Introduction
Novel view synthesis plays vital role in various computer vision applications. Tremendous progress has been witnessed recently with the introduction of neural radiance fields (NeRF) D. By modeling the emitted radiance and density of scene from a set of images with multi-layer perceptron (MLP), demonstrates an unprecedented level of fidelity on a range challenging scenes. Despite its success in diverse scenarios, such framework still faces challenges when working with glass, a medium which is frequently observed photography and daily life 2 to provide see-through protection for desired scenes. However, glass's reflectivity may reflect undesired scenes images captured through it. People always feel interested in the transmitted scenes behind the glass and want to remove the undesired reflection. This traps neural radiance fields into dilemma: Its well-designed framework is capable modeling every detail a scene, including those undesired ones.
One plausible way to resolve this dilemma is opting for veepprcessiing tool to remove the undesired components before the captured images are fed into NeRF []. For example, existing reflection removal methods 3 4 [ can be adopted to suppress the undesired components and provide better visibility for transmitted scenes. However, the processed images from different views may violate
Equal Contribution. Cooresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Vhrp
NeRF, 12 views
NeRF-W, 12 views
NeRF, 18 views
recunra edpr
NeRF, views
NeRF-W, views Our result, views
Examples obtained by NeRF, NeRF-W, and our method
Recurring edge estimation
Figure 1: Left: The results obtained by NeRF [T] and NeRF-W 7 with 6 and 12 training views, the result obtained by NeRF M with 18 training views, and result. The rendered results are with low reconstruction fidelity for NeRF []] and NeRF-W ( only with 6 and 12 training views. For NeRF [ with 18 training views, the result shows higher fidelity, but the undesired reflection is also finally rendered (labeled by green box). Right: The recurring edge estimation process. By warping images from neighboring views into a predefined view, the recurring edges belonging the transmission can be extracted to guide the novel transmitted view synthesis. The blue and red boxes denote the repeated transmission and sparse reflection pattern, respectively.
the assumption of NeRF that the scene is photometrically static. Besides, the reflection cannot be completely removed most cases by existing single-image- basee reflection removal approaches [6]. and the remaining reflection may be modeled by NeRF as part of the scenes, undermining the quality of rendered results.
An alternative solution is to render with undesired components like NeRF-W 7 by separating the whole scene into static and transient elements with exclusive properties. However, the reflection and transmission usually present highly "twisted" relationship since they may influence each other during light transportation and reflect such ill-posed essence in captured images. From results shown in Figure Ileft). it may challenge the applicability of a purely unsupervised strategy in NeRF-W [7]. thus showing degraded performance. Besides, NeRF-W 7 also partially relies on a dense set of multi-vies images to conduct the separation. As an impractical demand inherited from NeRF []] contradicts the casual photographing experience, where only limited inputs with sparse views are available. The results in Figure([left) [eeft) illustrate that NeRF-W [7] cannot perform high-fidelity synthesis with limited views, let alone effective separation between the transmitted scene and the reflected scene. Thus, how to render novel transmitted scenes with sparse views poses unique challenges.
In contrast to the exclusive properties required by NeRF-W 77). we employ the recurring edges, phenomenon verified by pionecring reflection removal methods g 91, link the above issues by warping images with different viewpoints into a predefined target view. As shown in Figure []] (right), given an aligned image sequence, the transmitted scenes appear in different views with large overlapping area while the reflected scenes only have a sparse presence [8. Besides, the aligned image features from the neighboring views can also be used for a more high-fidelity reconstruction of the target view. We, therefore, design a method supported by two pillars: 1) Edges of the repeated patterns, denoted as the recurring edges shown Figure rightt, that could estimated as a pilot to guide the differentiation during rendering: 2) features from reference views, which are incorporated into rendering for high-fidelity reconstruction under sparse views.
Several modifications have to be conducted to accommodate the above designs. Since the trans- mission/reflection edges only show their significance in certain area, the classical "pixel-wise" rendering framework cannot capture such internal physical properties. We employ a bittch-baedd rendering framework, which aggregates the pixels from certain area into a patch as the training batch. Besides, due to the interference caused by the reflection, the features extracted by network pre-trained for general tasks [00 may introduce interference from feature level, which deteriorates the transmission/reflection separation described before. We, therefore, propose transmission encoder to penetrate the feature-level interference caused by the reflection and render the transmission structure more accurately. At last, since the strength of reflection tends to rely on the viewing direction,

---

Aägnad imagis
feature integration
See-through scenarios
Image projection
Transmission
feeture encoder
T-MLP
R-MLP
Image wcdges
rendering
Pellection
Edtrated
GO
Vittaal imagus
Captured images
Cj
Glass
Recurning edge
Transmission features
csilmation
Reflecton embedding
Viewing direction
Figure 2: The proposed framework. Given images captured through glass from several different views, our method first projects the images neighboring views to their target view (image projection). Next, a transmission feature encoder extracts the transmission features from the reference images (transmission feature integration). The extracted transmission features are then passed into the transmission MLP (T-MLP) for transmission rendering (CT). For the reflection rendering (CR). we feed the viewing direction and corresponding reflection embeddings into the reflection MLP (R-MLP). The recurring edges repeatedly appearing in the aligned image sequence are extracted to further differentiate the transmission and reflection based on the edge loss Ledge (recurring edges estimation).
the weighting coefficients for the two layers are obtained from the viewing direction to combine transmission and reflection layers.
In summary, the above considerations lead to our Neural Transmitted Radiance Fields as depicted in Figure 2 and its main contribution can be concluded follows:
An approach to render novel transmitted views, which is capable of penetrating the reflection interference.
A unified consideration derived from recurring edges for removing reflection/cransmisson ambiguity and increasing rendering fidelity of transmission.
A dataset collected from the real world for evaluation.
2
Related work
Novel view synthesis with NeRF. Novel view synthesis aims at generating images from another view- point using set of established views. The classical approaches rely on the explicit representations like multi-plane images [T[22) meshes [13], point cloud (14) to render novel views. Despite their rendering efficiency at test time. they usually suffer from the limited expressiveness within their representation. In contrast to the explicit representations, Neural Radiance Fields (NeRF) [I utilize the implicit representation to represent 3D scenes using neural networks. For example, NeRF M approximates continuous 3D MLP by mapping an input 5D coordinate to corresponding scene properties. Given its promising representation ability, NeRF [I has been extended to solving prob- lems like relighting and scene editing. Recently, NeRF-W 7 proposes to adapt the original NeRF [I] to wild scenes by introducing grants the original NeRF [0 extended ability to model the scenes with photometric variations and transient components. Since NeRF requires large number of images for training, several meth- ods 15) are proposed to make it feasible with sparse views by utilizing the feature information from neighboring views. In this work, we aim to extend NeRF to see-through scenarios by eliminating negative interference caused by reflection.
additional MIP to model the undesired transient element, which
Transmission recovery from reflection-corrupted images. How to recover the transmitted scene from eefection-coorrppee images has been discussed for decades. Early methods [16. 17 make
use of the edge difference to reduce the ambiguity between the transmission and reflection through optimization. Recent methods mainly adopt the deep learning framework solve this problem. For
example, Fan et al. [18) proposed a two-stage deep learning approach to learn the mapping between
3

---

the mixture images and the estimated clean images. Wan et al. [9 20 proposed cooperative model better preserve the background details. Zhang et al. [2T proposed perceptual reflection removal method based on the generative adversarial network. Wei et al. S proposed another method to solve the reflection removal problem with non-aligned image pairs. Besides, some recently proposed [ methods also try to focus on the reflection-corrupter regions by localizing the reflection first. The recently proposed method [22] also removed reflections under the guidance of an additional polarizer. Instead of solely relying single images, plenty of methods [8] 23 24 25] also leverage advantages from the priors provided by multiple images to remove reflections based on recurring edges belonging to the transmission. Such inconsistency observed from different views also allows our framework to differentiate transmission and reflection during rendering.
Preliminaries about Neural Radiance Fields
We first briefly review the settings in NeRF D and NeRF-W 77. A NeRF represents scene continuous volumetric radiance field f. which maps the position of any given point x = (r, y: 2) and viewing direction d = (0, の) into color = (r, 9, b) and density as f(x, (c, o). Then, the camera ray r(t) projection of camera at position o can be obtained as
the expected color Ć corresponding
id emitted from the center of
C(r) C(r)= = J.M ((t)a(t)c(t)dtt
where in and Ły denote the near and far bounds of rendering, and T(t) = exp(- fe (r(()) ds)
denotes the accumulated transmittance. NeRF-W 7 extends NeRF [I with two MLPS to model the desired static (f,) and undesired transient (ft) components as
fx d, La) (ca, and ft(x, d, Ot).
Due the introduction of the additional MLP for transient elements, as well as the embeddings é, and & adapting to the various appearances and transient objects, NeRF-W W [ is capable of decomposing the desired static components during rendering.
For an image I captured in see- theethr scenarios, the relationship between the reflection R and = 3T + aR. (6), where 3 and a denotes the weighting coefficients. Applying NeRF-W 7 to achieve novel transmitted view synthesis is not directly applicable due to the following issues: 1) Dense view requirement: Both NeRF-W (7) and NeRF ( inherit the substantial demand for dense multi-view images, and their performances degenerate under real-world scenarios where only limited input with sparse views are available. 2) Undesired reflection interference: For the objects on both sides of the glass, their emitted light rays may be absorbed, refracted, and reflected, leading the light energy attenuation or highly mixed phenomenon in the captured image [6). This makes their separation present a ill-posed essence, which cannot be simply resolved by using an unsupervised model. We propose construct Neural Transmitted Radiance Fields to address these issues.
transmission T can be formulated
4 Neural Transmitted Radiance Fields
Our goal is learn radiance fields representation for the transmitted scene behind glass. To avoid the challenging separation ambiguity for transmission/reflecin and low reconstruction fidelity under sparse views, we build a unified approach by warping pixels to the desired target views. By extracting the recurring edges after warping, our approach can get rid of the interference from the reflection. Meanwhile, a transmission feature integration scheme is introduced to merge the transmission features from the warped images, which enables higher reconstruction fidelity even under sparse views.
The pixel-wise rendering scheme in NeRF 1 and NeRF-W M lacks the ability reflect the transmission/reellciion correlation, since some statistical properties can only be observed in a certain area. To better consider the spatial correlation among neighboring pixels, we employ the patch-based rendering scheme. Specifically, during training we sample a K. x IC patch P(u, s) in a given image following the way proposed in (26] as follows:
{
K
K
-11)
P(u, s) = (sz + u, sy + v) 1. E
(3)
2 "...
2

---

where = (u, u) denotes the center position of the patch, z and y range from K to K 2
to create a sampling grid. and denotes the scale of the image patch P. our setting. we apply a combined sampling of grid sampling with 8 = 1 over the entire image and random sampling with s > 1, allowing image patches to capture spatial information of different scales. The aggregated pixels from the image patch are then used for training, during which the model can not only learn the colors and densities through one ray, but also get a thorough understanding towards the context.
4.1 Transmission feature integration
In contrast to NeRF D and NeRF-W 77 which solely rely on point coordinates and viewing direction as the input, we propose to address the fidelity issue by means of the additional guidance from reference images. Given a scene only with sparse views, the feature guidance from neighboring views can help to reconstruct novel target views with higher fidelity. In general, such image features extracted by using 2D CNN network [0]] pretrained for general tasks. However, since the images under neighboring views are also captured through glass, general CNN network for view synthesis like previous methods [15] may lead to feature-level interference during rendering.
From Figure 2 we first project the neighboring views to their target view. To resist such interference during feature integration, we propose to obtain the transmission features by means of a transmission encoder 1) obtained by pre-training a reflection removal network, e.g., ERRNet [5] As feature encoder pretrained for reflection removal task, 1) aims at extracting the transmission features as W=n'L), =n(L), where W denotes the extracted feature volume. Instead of solely relying on the single- scale feature like pixel-NeRF (27]. which suffers from view misalignment in our setting, we propose to build a feature pyramid as W { g, Wi). where W, 9 and W, denote the global and local features with different scales, respectively. In general, CNNs naturally store local information in shallow layers and encode global information in deeper layers with a large receptive field, we extract W, g and W from the bottleneck layer and second last layer of 17, respectively. Prior works also suggest combining features from different scales can lead to better convergence and geometric details [28). The two feature maps are upsampled using Nearest- Interpolation the size of the image.
The extracted global and local transmission features for the i-th target view are then passed into the MLP network for the transmission rendering (T-MLP) fr, along with the position and viewing direction, as
fr(x, d. ( 9 ((") (c.") (
where 4 and of are the estimated color and density of the transmitted scene for the i-th view. As for reflection, learnable reflection embedding ( is introduced for each view i, and the color (8) er and density (f of the reflection scene are estimated using an additional MLP network for the reflection rendering (R-MLP) as
dx - (c{
(5)
where the reflection embedding is optimized along with the weights of fR. Similar existing NeRFs [ 77, we construct self-consistent loss as follows:
L.c
Σ ||(()) - C(r)]l,. P(X)EP
where p(-) denotes the mapping from ray to its corresponding pixel position, and C(r) denotes the observed color and C denotes the composite of CT and CR physically defined as C = 3CT+ + a &CR. Here denotes the weighting coefficients to balance CT and CR. For simplicity, we directly set a like previous reflection removal methods [29].
Several factors during light transportation can influence the weighting coefficients a. In our setting, since the strength of reflection largely depends on viewing direction [30] rather than the exact camera position, we propose to model such influence by simple MLP that outputs the weighting coefficients given corresponding viewing direction encoding as [a d) = a. Specifically, as d and x refer to the viewing direction and the position of any given points, respectively, the weighting map of a given view rendered similarly to Equation (T) by considering the two factors. enables the network to increase its robustness for general cases.
5

---

4.2 Recurring edge constraints
The settings described above ensure a high-fidelity novel view reconstruction even under sparse views. However, since no reflection removal model can reach completely reflection-fie free status, the reflection residuals left by the transmission feature encoder may further deteriorate the ambiguity issue. We then utilize the benefits provided by the aligned image sequence to further disentangle their twisted relationship.
As pointed by pioneer methods 9 29 when multiple images with different viewpoints are captured through glass, the transmission appears frequently across all samples, while the reflection may only have sparse presence. This property makes the recurring transmission edges become a "pilot" to relieve ambiguity between the transmission and reflection. In our method, first transform aligned images into their corresponding edge domain and measure the frequency of edge occurrence as
((z) Ex G.(z)2 DE-.G(Z))
where (z) denotes the gradient magnitude pixel location Z of view and & denotes the total number of pixels. measures the sparsity of the vector which achieves its maximum value of I (when only one non-zero item exists) and achieves its minimum value of 1/k (when all items non-zero and have identical values). Using this measurement, each edge pixel is assigned with two probabilities regarding whether it belongs the transmitted scene not.
Recognizing the transmission and reflection poses a great challenge for a vanilla NeRF network aiming at multi-view consistency, we provide our network with a likelihood estimation of the transmission and reflection, following the strategy proposed (9):
(z) = s(-(0(z) })
(8)
In, (2) = s(((2) 7
where s is a sigmoid function to facilitate the separation. Then, the final likelihood for transmission edges can be defined follows:
ET(Z) = { 1, LT./R, (z) 0.6
0, otherwise.
From the results shown in Figure D rright), the likelihood of target transmitted scene effectively labels the transmission edges, which provides clear guidance during rendering. We search from 0 to with a step of 0.1 and fix the threshold in Equation (9) as 0.6 in our experiments.
Based the binarized edge map obtained in Equation simple edge loss as follows:
propose to constrain rendering using
Ledge
Σ [[ET(((r)) @ (Ò(r) C( p(r)EP
(10)
which means we constrain the rendered image more on wherever the transmission edges are present, encouraging more faithful reconstruction towards the mixture image.
Besides, the transmission edges and those of the reflection tend to different, as prior works pointed out (21). We inherit the exclusion assumption between the gradient of transmission and reflection to differentiate the two parts the gradient domain as follows:
Lexel (0) V/ (CT, CR)ll
(1I)
(Cr, CR) = tanh (CT))(CT)) @ tanh (AR
where \T
/ and AR G
G |GCr The downsampling operation in its original
setting is discarded since our patch has been small enough. (f.,)) defines a pixel-wise correlation between the transmission and reflection, which helps separate them in the gradient domain.
By considering the above settings and constraints, our loss functions for the whole optimization can be concluded as L = 7Lec + &Lodge + wheaeet where 7 = 2, 8 = 0.002, and w = are the weighting coefficients to balance the influence of each term. All the calculations and optimization processes based on the patch sampled from the image.

---

4.3 Implementation details
We implement our framework using PyTorch. In the training and testing phase, two eight-layer MLPs with 256 channels are used to predict colors с and densities a corresponding to the transmitted and reflection scenes. We train a "coarse" network along with "fine" network network for importance sampling. We sample 64 points along each ray in the coarse model and 64 points in the fine model. A batch contains an image patch of 32x32 pixels, equivalent to 1024 rays. Similar to the settings in NeRF (T. positional encoding is applied to input location before they are passed into the MLPs. We use the Adam optimizer with defaults values 31 = 0.999, 3n = 0.9, = 10-8, and a learning rate 10-4 that decays following the cosine scheduler during the optimization. We optimize single model for about 100K iterations on two NVIDIA V100 GPUs.
Our transmission encoder can extract transmission features of multiple resolutions by leveraging advantages from ERRNet [5] and U-Net (31. Based on our experiments, even a simple U-Net [31 can also help to resist the reflection interference during rendering. The transmission encoder is first integrated with a feature decoder to compress the information
lower dimension, and the output fed into the transmission MLP. For the pre-training of the transmission feature encoder, we follow the strategy proposed in [5] with its release training data. Thus, the model is approximately equivalent to its release model.
5 Experiments
Dataset. Our experiments based real-world dataset we collect. This dataset contains different real-world scenes, each consisting of 20 to 30 mixture images with different poses. Specifically, 4 scenes are with the ground truth for quantitative evaluations in the experiments. We follow the setup proposed in [6] by first capturing the mixture image through transparent glass and then its corresponding transmitted scene by removing the glass. We follow the previous settings in NeRF I and its variants [00] 32) to estimate the pose via COLMAP 33 [.
Like most reflection removal methods (6). we assume piece of planar glass when capturing images. Though some glasses in the real world are slightly curved, it does not obviously affect the robustness of our method.
We also our network on the LLFF dataset 335 and the RFFR dataset [32]
Baselines. We compare our method with five NeRF-based methods: 1) NeRF [1] the original NeRF method; 2) NeRF-W [7] an unofficial implementation of NeRF in the Wild 3) MVSNeRF [00: a NeRF-based multi- stereo model for novel view synthesis (NVS) which also relies on reference views; 4) Reflection removal (RR)= + NeRF: the most direct way for novel transmitted view synthesis, namely training a NeRF network based on the results of reflection removal method (without losing generality, we use ERRNet [5), a state-of-the-an method inspiring our design); 5) NeRFReN [3]: a NeRF-based method designed for see-through scenarios released recently; 6) RR MVSNeRF: NVS using M [10] with reflection removal applied as a pre-processing. All the results in Section obtained using six views for training. Due the page size limitation, the results of other methods trained under more views can be found in the supplementary material.
Metrics. We report quantitative performance using PSNR, SSIM and LPIPS [3). For PSNR and SSIM, higher value indicates better performance. For LPIPS [36, lower value indicates better performance. By measuring image fidelity using high-level features, LPIPS (6) can better match human judgements of image similarity.
5.1 Qualitative results
The qualitative results of the rendered novel transmitted views are presented in Figure Bnd Figure 4 Due to the effectiveness of the proposed transmission feature integration scheme, our method can reliably reconstruct novel transmitted views under only six views. Besides, the feature integration scheme and its cooperative recurring edge constraints further reduce the ambiguity between reflection and transmission during rendering. For other methods, though NeRF [[] can partially reconstruct the desired transmitted views, still suffers from the interference caused by the reflection; NeRF-W [ and NeRFReN (32 output degenerated results as they cannot separate reflection from transmission
hhttss//gthbbb.co///wwe1///////

---

Target view
Ours
NeRF
NeRF-W
MVSNeRF
RR+NeRF
RR+MVSNeRF
NeRFReN
Target
Ours
NeRF
NeRF-W
MVSNeRF
RR+NeRF
RVSSER
NeRFReN
Figure 3: From left right: the target view, results, the results obtained by NeRF 1. NeRF- W (7). MVSNeRF (10), RR+NeRF, RR+MVSNeRF (TO), and NeRFReN [32]. The target views in the scenarios (scenes with non-detachable glass, e.g., painting with fixed frames) are captured through glass for reference only (no ground truth transmission). Please zoom in for more details.
with as few inputs as our method; MVSNeRF 110 can keep high fidelity of the rendering results like methods, but obvious reflection residuals still remain in their results since their feature extractor cannot resist the reflection interference.
Table 1: Quantitative evaluation compared with NeRF M.. NeRF-W 77, MVSNeRF [0)
RR+NeRF,
NeRFReN (32), and RR+MVSNeRF [10). Higher PSNR and SSIM values denote better results (1), while lower LPIPS values denote better results (L).
PSNR t
SSIM t
LPIPS
Ours
22.75
0.841
0.205
NeRF
11
16.72
0.644
0.510
N
7
15.44
0.624
0.529
MVSNeRF
10
17.83
0.685
0.400
RR+NeRF I
18.42
0.691
0.539
NeRFReN [32]
18.63
0.716
0.449
RR+MVSNcRF 10]
18.16
0.679
0.447
5.2 Quantitative results
The quantitative results of the rendered novel transmitted view in Table also validate the observation in Figure 4 Higher PSNR values show that our method can render novel transmitted views and recover the color information with higher accuracy. Higher SSIM values indicate that our method can preserve the structural information with high-frequency details. Lower LPIPS values show that recovered images by our method better aligns with human perception. Specifically, for the results obtained by other methods, their higher LPIPS values indicate that they cannot reconstruct realistic results with high fidelity when only six views are provided. Specifically, MVSNeRF (10) achieves the second best LPIPS results among all methods, which shows that the feature integration scheme

---

Target view
Ours
NeRF
NeRF-W
PSNR:23.95 RR+NeRF
PSNR: 16.69 RR+MVSNeRF
PSNR:13.88 NeRFReN
MVSNeRF
PSNR:21.30 Target view
PSNR: 17.82
PSNR:21.19 NeRF
PSNR:20.91
Ours
NeRF-W
PSNR:21.56 RR+NeRF
PSNR:
PSNR:17.08
MVSNeRF
RR+MVSNeRF
NeRFReN
PSNR:14.35
PSNR:19.03
PSNR:15.12
PSNR:16.35
Figure 4: From left right: the target view, results, the results obtained by NeRF T. NeRF- W 17]. MVSNeRF [10]. RR+NeRF (1]. RR+MVSNeRF (TO), and NeRFReN [32). The target views in the scenarios (scenes with detachable glass, captured with the similar setup as (6) are obtained
by removing the glass, and are compared with the results as ground truth for PSNR values shown below each image. Please zoom for more details.
Table 2: Ablation study on the model without the transmission encoder (1), the recurring edge constraints (REC), the patch-based rendering scheme (P), and either the transmission encoder or patch-based rendering scheme. Higher PSNR and SSIM values denote better results (f), while lower LPIPS values denote better results (L).
PSNR t
SSIM
LPIPS
Complete model
22.75
0.841
0.205
(wfo)
15.60
0.512
0.565
(w/o) REC
22.48
0.836
0.265
(wfo) P
18.17
0.769
0.271
(w/o) 1) P
15.65
0.588
0.594
NeRF
16.72
0.644
0.510
can indeed improve the rendering fidelity. On the other hand, RR+NeRF setting achieves the second best PSNR and SSIM result, which shows the effectiveness of this simple strategy to some degrees.
5.3 Ablation study
Our network consists of two parts: the transmission encoder to disentangle the transmitted and reflected scenes, and the patch-rendering scheme to utilize the physical constraints. We conduct several experiments to evaluate the benefits of these two parts. We first remove the transmission encoder and directly feed the MLP with the position and viewing direction. From the results shown in Figure 5 without the support of reference views, position and viewing directions cannot effectively reconstruct the desired transmitted views. The errors shown in Table also become closer

---

Targel view
Full model
Wio REC
W/o P
W/o
Wio y
Figure 5: From left right: the target view (with reflection), the results of complete model, without the recurring edge constraints (REC), without the transmission encoder (7), without patch- based rendering scheme (P), and with neither transmission encoder patch-based rendering scheme.
(a) MVSNeRF, 6 training views, mean PSNR: 18.09 (b) Ours, 6 training views, mean PSNR: 21.69
Figure 6: Non-reflective scenes from LLFF dataset 35 Please use Adobe Acrobat to see the animated results.
to those of NeRF in T We then remove the recurring edge constraints, and more reflection is observed in the rendered target view since the network attaches equal importance to every pixel in the patch. We further remove the patch-based rendering scheme entirely. In this situation, reflection evidently remains on the generated target view due to a lack of physical constraint, as we discussed before. At last, both the patch-based rendering scheme and the transmission encoder removed, and we achieve a degenerated result similar to that of NeRF-V W (7) shown in Table Without loss of generality, we specifically compare with MVSNeRF [00 on non-reflective common examples to show that our method can also adapt to more general scenarios in Figure
Conclusions
We solve the problem of novel view synthesis for see-through scenarios in this paper. We introduce a transmission encoder to address the fidelity issue caused by sparse views and ambiguity issues led by the reflection interference in a unified framework. Specifically, to further disentangle the twisted rela- tionship between transmission and reflection during the rendering process, edge constraint by counting the frequency of edge occurrence among the aligned image sequence. Experimental evaluation newly collected dataset demonstrates the promising performance for novel transmitted view synthesis our method could achieve.
introduce recurring
Limitations and future work. Our method still faces the challenge led by the occlusion issue, since we rely on features from neighboring views complement the missing information given sparse set of input views. However, when features are not consistent with certain view, occlusions may negatively undermine the performance of our proposed method, due to the inaccurate feature information for synthesizing novel views in that area. Besides, method relies on COLMAP [33 34 for the rendering, while it may fail on low-transmitted reflections covering large areas or the transmission distortion caused by irregular glass. In this situation, COLMAP 33 34 cannot correctly extract the transmission features. Investigating NeRF without the requirement for established poses like (37 may help to alleviate this issue. We will also consider the irregular glass for its influence the transmitted objects' light in our future work.
Acknowledgement: This work is supported by National Natural Science Foundation of China under Grant No. 62136001, 61872012. Renjie Wan is supported by the Blue Sky Research Fund under the Research Committee of Hong Kong Baptist University under the Project Number BSRF/21-22/16
10

---

References
[1] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
[2] Teju Cole. The history of photography is history of shattered glass. New York Times, 2017.
[3] Renjie Wan, Boxin Shi. Ling- Yu Duan, Ah-Hwee Tan, Wen Gao, and Alex C Kot. Region- aware reflection removal with unified content and gradient priors. IEEE Transactions on Image Processing. 2018
[4] Zheng Dong, Ke Xu, Yin Yang, Hujun Bao, Weiwei Xu, and Rynson WH Lau. Location-aware single image reflection removal. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[5] Kaixuan Wei, Jiaolong Yang. Ying Fu, David Wipf. and Hua Huang. Single image reflection removal exploiting misaligned training data and network enhancements. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019.
[6] Renjie Wan, Boxin Shi, Haoliang Li, Yuchen Hong, Lingyu Duan, and Alex Kot Chichung. Benchmarking single-image reflection removal algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[7] Ricardo Martin Barilla,, Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Doso- vitskiy, and Daniel Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[8] Yu Li and Michael S Brown. Exploiting reflection change for automatic reflection removal, In Proceedings of the IEEE international conference on computer vision, 2013.
[9] Tianfan Xue, Michael Rubinstein, Ce Liu, and William Freeman. computational approach for obstruction-free photography. ACM Transactions on Graphics (TOG), 34(4):1-11, 2015.
[10] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[11] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. arXiv preprint 10XXvv1800.0090 2018.
[12] John Flynn, Michael Broxton, Paul Debevec, Matthew Dulall, Graham Fyffe, Ryan Styles Overbeck, Noah Snavely, and Richard Tucker. Deep High-quality view synthesis by learned gradient descent. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019.
[13] Meng-Li Shih, Shih-Yar Yang Su, Johannes Kopf, and Jia-Bin Huang. 3D photography using context-aware layered depth inpainting. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[14] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. SynSin: End-to-end view synthesis from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[15] Julian Chibane, Aayush Bansal, Verica Lazova, and Gerard Pons-Moll. Stereo radiance fields (SRF): Learning view synthesis for sparse views of novel scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[16) Anat Levin and Yair Weiss. User assisted separation of reflections from single image using sparsity prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(9): 1647- 1654, 2007.
11

---

[17] Renjie Wan, Boxin Shi, Tan Ah Hwee, and Alex Kot. Depth of field guided reflection removal. In IEEE International Conference on Image Processing (ICIP), 2016.
[18] Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, and David Wipf. A generic deep architecture for single image reflection removal and image smoothing. In Proc. International Conference on Computer Vision (ICCV), 2017.
[19] Renjie Wan, Boxin Shi, Ling-Ya Yu Duan, Ah-Hwee Tan, and Alex C. Kot. CRRN: Concurrent multi-scale guided reflection removal network. Proc. Computer Vision and Pattern Recognition (CVPR), 2018.
[20] Renjie Wan, Boxin Shi, Haoliang Li, Ling-Yu Yu Duan, Ah-Hwee Tan, and Alex C Kot. CORRN: Cooperative reflection removal network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(12))2999-2982, 2019.
[21] Xuaner Zhang, Ren Ng. and Qifeng Chen. Single image reflection separation with perceptual losses. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
[22] Youwei Lyu, Zhaopeng Cui, Si Li, Marc Pollefeys, and Boxin Shi. Reflection separation using a pair of unpolarized and polarized images. Advances in neural information processing systems, 2019.
[23] Byeong-Ju Han and Jae-Young Sim. Reflection removal using low-rank matrix completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[24] Abhijith Punnappurath and Michael S Brown. Reflection removal using a dual-pixel sensor. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[25] Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang, Yung-Yo Yu Chuang, and iia-Bin Huang, Learning to see through obstructions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[26] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. GRAF: Generative radiance fields for 3D-aware image synthesis. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
[27] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, PixeINeRF: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[28] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Mare Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12796-12796, 2022.
[29] Yu Li and Michael Brown. Single image layer separation using relative smoothness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014.
[30] H.-C. Lee, F.J. Breneman, and C.P. Schulte. Modeling light reflection for computer color vision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(4):402-409 1990.
[31] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 2015.
[32] Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Sang-Hai Zhang. NeRFReN: Neural radiance fields with reflections. arXiv preprint arXiv:2111.15234 2021.
[33] Johannes Lutz Schönberger and Jan-Michael Frahm. Conference Computer Vision and Pattern Recognition (CVPR), 2016.
revisited. In
[34] Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan Frahm. Pixelwise view selection for unstructured multi-vien stereo. In European Conference on Computer Vision (ECCV), 2016.
12

---

[35] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz- Cayio, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 38(4):1-14, 2019.
[36] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2018.
[37] Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. Nerf-: Neural radiance fields without known camera parameters. arXiv preprint urXiv:2102.07064 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section O
(c) Did you discuss any potential negative societal impacts of your work? [No] We do not think this work will have any potential negative socictal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See Section
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as a URL)? [Yes] The code and data will be released on the website mentioned in the abstract.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] We think error bars unnecessary in our work.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] The data will be released on the website mentioned in the abstract.
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "UxXchnQsb83", "reviewer": "Reviewer_9GAV", "summary": "The paper proposes a method to learn neural radiance fields that represent the underlying scene free of reflective components in the scene, i.e. explicitly represented the transmitted regions of the scene. Prior work in representing transmitted radiance field relies on reflection removal from the input image sequence, however this is a challenging problem and typically results in photometric inconsistencies. The proposed method uses a novel formulation leveraged on the observation that reflective components in the radiance field are sparser than the transmitted components. A patch-based rendering scheme is used to handle the local characteristics of reflective/transmissive components. ", "questions": "It would be an interesting study to see the performance of this method as the sparsity assumption is slowly relaxed, where the reflective components in the scene change from transmissive glass plates on one end of the spectrum to fully silvered mirrors on the other. Have the authors performed any experiments on such scenes? \n\nWhile occlusions are one of the main limiting factors, occlusions in the reflected part of the scene however are not likely to cause a problem in my opinion, since the edge constraint assumption and the sparsity assumption are still valid? ", "limitations": "Yes, the authors discuss the limitations of the work", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is well written and the exposition is clear. The paper provides a through introduction and a motivation for the solution, before properly explaining the proposed solution. As such I find the paper to be a useful contribution to the community and beneficial for the reader. \nThe use of the transmission encoder with pyramid-scale features is interesting and the choice of Wg and Wl is properly motivated. \nThe recurring edge constraints are the core strength of the paper and the description provided in section 4.2 is succinct. \nThe qualitative and quantitative results in the paper and supplemental material clearly demonstrates that the transmitted radiance field is captured free form noise due to reflection. \n\nWeaknesses:\nThe authors rightly point out that weighting coefficients are dependent on several factors. The viewing direction (wrt lights in the scene) and the camera position are correlated and more discussion is warranted on whether an MLP that encodes the weighting coefficients is sufficient in general. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QLkPGH-LYkF", "reviewer": "Reviewer_K1pa", "summary": "This paper targets to solve the novel-view synthesis problem with reflection removal, that is, novel-view synthesis of a transmitted object from images corrupted by reflections. A naive baseline, that applies reflection removal techniques to each input image before NeRF, does not solve the problem as the resultant image would not be multi-view consistent; This is because most reflection-removal techniques cannot take advantage of multiple viewpoints. This paper solves this problem by introducing 1) transmission feature integration and 2) recurring edge constraints. First, Transmission feature integration is based on the idea of pixel-NeRF that the feature from other viewpoints can assist the training, and the paper used “transmission feature” instead of the vanilla pixel feature in pixel-NeRF. Second, recurring edge constraints are based on the assumption that a reflected component is sparse in its presence in the aligned image. The paper also collected a new dataset for real multi-view images corrupted by reflections, and the proposed method shows promising results. ", "questions": "Additional experiments that may further demonstrate the robustness of the proposed method:\n\n- Non-planar reflector. How would the proposed method work when the reflector is not planar? It seems that all the experiments are done with a planar reflector. Does any data include a non-planar reflector? When the reflector is not planar, the behavior of reflections in multi-view images will be very different as the reflected parts will not be aligned after warping. To be specific, when the reflector is planar, the reflected object is equivalent to the virtual object that is behind the reflector and thereby the reflected parts from different viewpoints will be located at the same pixels after warping. However, we cannot expect this alignment in the non-planar reflector as it will be projected differently depending on the viewpoint. This will affect the performance of REC that relies on the aligned edge after warping, and it would be interesting to see how the method works in the case of a non-planar reflector.\n- Large reflector. What if the reflector is large enough and thereby the reflections exist in every viewpoint? (e.g., large window as in [9]). This case breaks the assumption used for REC that the reflection is sparse across the different viewpoints, and only “motion” inconsistency can disambiguate reflections and transmissions (though it is still ambiguous to determine which one is transmission).\n- Ablation study of feature pyramid W_g and W_l. What if the transmission feature W is used without a pyramid?\n- How is the threshold 0.6 in (12) determined?", "limitations": "The questions in the above section (Questions) include some limitations that are not handled in the paper: non-planar reflector and large reflector. The proposed method may not work for those cases of reflectors.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Promising results. The proposed method shows promising results on real multi-view images corrupted by reflections. The comparison with other methods such as NeRF, NeRF-W, and RR + NeRF, also shows that the proposed method performs superior both qualitatively and quantitatively, especially when the number of input images is limited.\n- New dataset of multi-view images with and without reflections. The paper shows the newly collected multi-view images, which can facilitate further research on multi-view reconstruction and reflection removal.\n\n### Weaknesses\nWhile the paper proposes an interesting method with promising results, there are some weaknesses that can be improved:\n- The presentation of the manuscript can be improved. There are some ambiguous definitions or explanations:\n    - [Line 123] What is transmission and reflection entanglement? If it means transmission and reflection have an inherent ambiguity, then the proposed method cannot disambiguate either. “Due to the absorption, reflection, and refractive effect ~” should be further clarified.\n    - [Motion inconsistency] The terminology “motion inconsistency” (used frequently all around the paper including the abstract) used for recurring edge constraints is somewhat misleading. The key idea used for recurring edge constraints is that the reflected component may not exist in some viewpoints and thereby have a sparse presence. The reason for this phenomenon is the size of the reflector is limited, which causes the reflected object to be outside the reflector and disappear in some viewpoints. It has nothing to do with motion and thus the term “motion inconsistency” is not the appropriate term to describe the method. Maybe the reflected object is at a different depth from the transmitted object and moves differently in the image (e.g., larger disparity when it is further), but it is not the information that the proposed method directly uses. The description in the main paragraph (line 187-) is already clear, so just choosing a better terminology would improve the clarity of the proposed method.\n    - [Line 210] What is \\Psi? The notation seems to be not defined.\n- Some important details about the transmission feature are missing. What network is used for feature W? From Line 155, I assume the network is based on ERRNet but it is difficult to see which part of the ERRNet is used as there are many components in the ERRNet. Line 162 is not enough for understanding the exact structure. Also, Line 232 explains the pretraining of the transmission encoder briefly and it is somewhat confusing if the method is different from the original ERRNet. The network structure and the training detail needs to be added to the supplemental material.\n- Missing baseline. A baseline (that might be interesting) is missing, that is RR + pixel-NeRF (without transmission feature). One of the main contributions of this paper is using the transmission feature, which is the combination of 1) reflection removal and 2) pixel-NeRF (assist the training of NeRF). If these two parts are divided into the reflection removal part and the pixel-NeRF part, it can be another baseline of RR + pixel-NeRF, which will be a more fair and interesting baseline.\n- REC has a limited performance (at least quantitatively). The second main contribution of this paper is using recurring edge constraints (REC), but the effect of REC seems to be marginal quantitatively as shown in the ablation study (Table 2). The PSNR  without REC is 22.48, which is almost the same as that of the complete model (22.75). It would be interesting to see how REC works in more challenging data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6hKE4Voh6mo", "reviewer": "Reviewer_2Mxb", "summary": "This paper proposes a novel view synthesis network specially designed for see-through scenarios. This paper introduces a transmission encoder, which separately estimates the transmission amount against the specular highlight's reflection. In addition, this paper introduces a recurring edge constraint to account for the frequency of edges. ", "questions": "- As mentioned earlier, specular reflection moves in the opposite direction of the transmitted image when the camera moves. It causes structure-from-motion failures that are supposed to give the right camera pose information as input. This paper doesn't clearly mention how the camera pose is estimated from the input image. I might miss it. If then, please let me know where the information is. I'm worried about the impact of the camera pose when producing other methods' results. I would like to hear more in the rebuttal.", "limitations": "Limitations are clearly mentioned in the main paper. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n+ The application and approach of the transmissive scenario sound interesting to me. The specular reflection on glass in the see-through scenario has been rarely discussed in the neural rendering field yet. I found that this new research problem is interesting. Existing solutions such as vanilla NeRF seem to fail when there is a specular reflection in input images, while the proposed method works properly. \n\n[Weaknesses]\n- Even though the motivation of the proposed method sounds interesting, I'm not fully sure if this paper is completely developed and evaluated to solve the technical challenges. Specular reflection works very differently from transmission. For instance, when the camera motion occurs, the specular reflection and transmitted image move in opposite directions about the depth position of glass surfaces. The proposed model doesn't seem to account for the physical phenomenon. Instead, it just tries to separate the transmission and reflection along the given view vector, which is not physically plausible. This observation should be valid from a specific view angle. If the method accumulates multiple observations in a voxel grid, the accurate separation cannot be achievable by increasing the number of observations. I would like to hear more in the rebuttal.\n\n- The evaluation of this paper is one of the weakest points. Except for the main results shown in the teaser, most results do not include strong specular reflection. According to the proposed formulation of the recurring edge constraint, the proposed method may work properly when there are strong contrast edges in the transmitted image. The main result of the picture frame is the case. In other cases, the results do not include any strong specular reflection. I think the results look very cherry-picking with a very small number of examples. I would like to see more results to validate the performance of the proposed method. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "6IjNF0Eyqf3", "reviewer": "Reviewer_Pr2K", "summary": "This paper proposes a novel neural radiance field rendering method that is dealing with specular reflection on the object’s surface. The proposed method aims at recovering only the transmission radiance behind the reflection. To that end, this paper proposes to prepare two dedicated networks, i.e. T-MLP and R-MLP, to learn the transmission features and reflection features. This is achieved by applying a single image reflection removal method to the training data to separate the background and the reflection. The learned transmission and reflection color radiance are then combined in a convex combination. In addition, in order to guide the learning of background high-frequency details, this method also applies recurring edge constraints which utilize the observation that background edges appear consistently in multiple different views. ", "questions": "1. The transmission features W_g and W_i are in different size. How do they feed into the T-MLP network?\n", "limitations": "The limitation of the proposed method is to apply it to any normal scenes or NeRF datasets. If it cannot perform well on non-reflective scenes, the generalizability of the method will be the biggest limitation. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. This paper is generally well-written with clear motivation in the introduction section. It clearly defines the current problem and challenge left by existing NeRF-based methods, which is the reconstruction of scenes behind the transparent surfaces with specular reflection. \n2. The comprehensive experiments show that the proposed method consistently outperforms the state-of-the-art methods by a considerable margin, in both qualitative and quantitative evaluations. \n3. This paper proposes a new NeRF purpose dataset, which is particularly focusing on the scenes behind the specular reflection. The proposed dataset may impose a strong impact on future research in this area. \n\nWeakness\n1. I found the performance comparison with respect to baseline method MVSNeRF is a bit unfair because the selected baseline methods are not designed to deal with reflection, and hence it tends to predict the reflected scene as is. Therefore, the quantitative PSNR results are much worse than the proposed method as expected. Especially, in Figure 3, MVSNeRF almost reconstructs the exact appearance of the target view. \n2. For a NeRF method, it is also important to know the performance of the proposed method applied to normal (non-reflective) scenes. Otherwise, the usage of the proposed method is just limited to reflective scenes.  In the submitted paper and supplementary material, all the examples and benchmark data are performed on the scenes with reflection. The authors are suggested to provide more comparison (quantitative) and real normal scene examples in the rebuttal period. \n3. What is the processing speed and the network complexity of the proposed method compared to baseline methods? In order to prove the effectiveness of the proposed method, it is crucial to verify that the performance gain is not coming from the extra number of parameters in the network as well as the pre-processed edge map and reflection purged features.  \n4. From the ablation study, the recurring edge constraints (REC) only bring in very little improvement, but it is considered as one of the two contributions in the method section. It seems that the proposed method is not very effective. \n5. It is true that the proposed method outperforms other baselines on the reflective NeRF dataset by a large number. However, the method itself is quite straightforward with limited novelty. It is critical to understand the effectiveness of the proposed method by providing the performance comparison on normal datasets, and hence prove the validity of the proposed method. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept