Generalizable Non-Line-of-Sight Imaging with Learnable Physical Priors

## Abstract

Abstract Non-line-of-sight (NLOS) imaging, recovering the hidden volume from indirect reflections, has attracted increasing attention due to its potential applications. Despite promising results, existing NLOS reconstruction approaches are constrained by the reliance on empirical physical priors, e.g., single fixed path compensation. Moreover, these approaches still possess limited generalization ability, particularly when dealing with scenes at a low signal-to-noise ratio (SNR). To overcome the above problems, we introduce a novel learning-based solution, comprising two key designs: Learnable Path Compensation (LPC) and Adaptive Phasor Field (APF). The LPC applies tailored path compensation coefficients to adapt to different objects in the scene, effectively reducing light wave attenuation, especially in distant regions. Meanwhile, the APF learns the precise Gaussian window of the illumination function for the phasor field, dynamically selecting the relevant spectrum band of the transient measurement. Experimental validations demonstrate that our proposed approach, only trained on synthetic data, exhibits the capability to seamlessly generalize across various real-world datasets captured by different imaging systems and characterized by low SNRs.

## 1 Introduction

Non-line-of-sight (NLOS) imaging represents a groundbreaking advancement in visual perception, enabling the visualization of hidden objects with significant implications in diverse fields, including autonomous navigation, remote sensing, disaster recovery, and medical diagnostics (Bauer et al., ; Lindell et al., ; Scheiner et al., ; Laurenzis et al., ; Wu et al., ; Maeda et al., ).
By harnessing sophisticated time-of-flight (ToF) configuration, NLOS imaging systems can effectively capture light signals bounced off hidden objects, even when direct line-of-sight visibility is obstructed, as illustrated in Fig. 1(a). The core components of such systems typically include pulse lasers, which emit short bursts of light, and time-resolved detection sensors like Single Photon Avalanche Diode (SPAD) and Time-Correlated Single Photon Counting, which precisely capture the flight of the time that takes for photons to travel from the light source to the hidden object and back to the SPAD. The captured signals, known as transient measurement, undergo reconstruction using various algorithms, including traditional approaches (Velten et al., ; Arellano et al., ; Liu et al., ) and learning-based approaches (Chen et al., ; Grau Chopite et al., ; Mu et al., ; Yu et al., ; Li et al., ; ).

For the traditional approaches, the back projection algorithms (Laurenzis & Velten, ; Velten et al., ) and the light path transport algorithms (Heide et al., ; O’Toole et al., )
typically assume isotropically scattering, no inter-reflection, and no occlusions within the hidden scenes. However, these approaches always yield noisy results and lack details. Conversely, the wave propagation approaches (Lindell et al., ; Liu et al., ) require no special assumptions and tend to produce better results while being sensitive to scenes with large depth variations.
Learning-based approaches (Chen et al., ; Mu et al., ; Li et al., ) leverage the powerful representation capabilities of neural networks, and push NLOS reconstruction to a higher level.

Despite promising results, current NLOS reconstruction algorithms are constrained by the reliance on empirical physical priors and are still confronted with challenges.
The primary challenge is Radiometric Intensity Fall-off (RIF), i.e., the intensity of the reflected photons attenuates and the degree of attenuation is related to the surface material of the hidden object.

To address this phenomenon, quadratic and quartic operations are commonly applied to the light propagation path for retro-reflective and diffuse surfaces, respectively (O’Toole et al., ), to compensate for intensity attenuation.
However, since various surface materials coexist within the same scene, applying path compensation based on a single material type across the entire scene, as performed in physical priors or previous work, may not effectively counteract the effects of attenuation.
Additionally, the problem is exacerbated by the low quantum efficiency of the imaging system, particularly over long distances.
As shown in Fig. 1(b), using a single coefficient to compensate for the entire scene can enhance the reconstruction of objects with corresponding material properties, but it will significantly reduce the SNR for other objects in the same scene.
Another challenge is the limited generalization ability mainly caused by various noises. In this study, we concentrate on two specific noise sources: the dark count of the SPAD and the ambient light (Hernandez et al., ). As the data acquisition time decreases, the signal-to-noise ratio (SNR) decreases, resulting in higher noise levels.
The Poisson-distributed noise photons degrade the quality of transient measurements especially at low SNR, manifesting as high-frequency aliasing. This phenomenon poses grave challenges to existing approaches, with traditional ones yielding a plethora of artifacts, and learning-based ones experiencing a breakdown in their ability to generalize.

Figure: Figure 1: (a) An overview of the NLOS imaging system, including objects with distinct surface materials. (b) Reconstructed images from our method and RSD (Liu et al., ) with different compensation coefficients. Near to Far: Dragon, Bookshelf, Statue.
Refer to caption: x1.png

To address the above two challenges, we propose a novel learning-based approach by leveraging the virtual wave phasor field (Liu et al., ). Our approach incorporates two key designs: the Learnable Path Compensation (LPC) and the Adaptive Phasor Field (APF).
Given that reflected light with different degrees of RIF may be captured simultaneously, the LPC utilizes three physics-based predefined compensation weights to initialize the features of transient measurements for path compensation.
Subsequently, a convolutional neural network is trained to implicitly learn and assign distinct compensation coefficients to each scanning point in the transient measurements. By utilizing these learnable compensation coefficients, the LPC adaptively mitigates light wave attenuation in the same scene, as shown in Fig. 1(b), particularly for distant regions.
Meanwhile, the APF learns an applicable standard deviation for the Gaussian window of the illumination function, allowing it to dynamically choose the relevant spectrum band for each transient measurement. The emphasis on the effective spectrum enables the discrimination of useful information from noise under distinct SNR conditions.

To demonstrate the efficacy of our proposed approach, we train the approach on a synthetic dataset and subsequently test them on unseen data, including both synthetic and real-world datasets captured from different imaging systems. The exceptional performance on unseen synthetic data and the diverse real-world data highlight the robust generalization capabilities of our approach. Even under challenging conditions, i.e., fast acquisition time and low SNR, our method consistently outperforms competitors. To further increase the diversity of NLOS data, we provide three real-world data captured by our own NLOS imaging system to conduct more comprehensive experiments.

In summary, the contributions of this paper can be listed as follows:

- •
We propose a novel learning-based solution for NLOS reconstruction, breaking the reliance on empirical physical priors and boosting the generalization capability.
- •
We design the LPC to adaptively mitigate the light attenuation in the same scene. The embedded learnable physical prior greatly improves the generalization capability across different object materials, especially for long-distance regions.
- •
We design the APF to prioritize the relevant information from the frequency domain, which improves the generalization capability across transient measurements under distinct SNR conditions.
- •
Our proposed approach, trained on synthetic data, achieves the best generalization performance on both synthetic and publicly real-world datasets with diverse SNRs. Additional real-world data captured by our own imaging system further showcases the capability of our approach.

## 2 Related Work

### 2.1 Traditional Approaches

In the rapidly advancing field of NLOS imaging, significant progress has been made towards unveiling hidden objects. The groundwork was established by Kirmani et al. (), who pioneered the use of time-resolved imaging to navigate photons around obstructions, despite facing computational challenges due to complex multi-path light transport. Efforts to streamline the complex inverse problem have led to the development of back projection approaches, notable for their ability to approximate the geometry of obscured objects through ultrafast time-of-flight information capturing and light geometric relationship (Velten et al., ; Arellano et al., ). The Light-cone Transform (LCT), marked by the introduction of simple assumptions for light propagation, further facilitated the NLOS reconstruction with unprecedented detail by solving inverse problems in the linear space (O’Toole et al., ). The wave propagation approaches like frequency-wavenumber migration (FK) (Lindell et al., ) and Rayleigh Sommerfeld Diffraction (RSD) (Liu et al., ; ) provided enhanced accuracy for NLOS imaging by considering the interaction between the light wave and multiple hidden object surfaces. Despite considerable progress, traditional algorithms are still limited with challenges in noise effects and complicated scenes.

### 2.2 Learning-based Approaches

Recently, learning-based approaches have been gradually introduced into NLOS imaging. Grau Chopite et al. () proposed the first end-to-end learnable network for NLOS reconstruction. The UNet (Ronneberger et al., ) based network regressed the depth from transient measurements directly. However, it is an unstable solution that transforms the non-linear spatial-temporal domain into the linear spatial domain solely by convolution layers. The instability is particularly evident in real-world scenarios, resulting in poor reconstructions. To solve this problem, Chen et al. () developed the physics-based feature propagation module (LFE, Learned Feature Embeddings) to transform different domains, narrowing the domain gap between the synthetic and real-world data. Building on the insights from NeRF (Mildenhall et al., ), recent solutions (Mildenhall et al., ; Mu et al., ) can render the albedo of hidden objects through the radiance field in the unsupervised manner, which consumes large computation time for each inference. Through analysis of transients histogram, Li et al. () produced the first transformer-based framework (NLOST) for capturing local and global correlations, while entailing a substantial computational burden. Yu et al. () introduced a learnable Inverse Kernel (I-K) with attention mechanisms. However, I-K is actually tailored for the point spread function of the imaging system rather than the transient measurements. While the above physics-based approaches (Chen et al., ; Li et al., ; Yu et al., ) consistently improve NLOS reconstruction performance, they still encounter challenges when reconstructing real-world scenes with diverse object materials. Additionally, these approaches overlook the generalization of the real-world transient measurements with low SNRs. In this paper, we present specific solutions tailored to these two challenges.

## 3 Methodology

### 3.1 Imaging Formulation

Figure: Figure 2: An overview of our proposed approach. Given the transient measurements as input, the approach generates the albedo volume, intensity image, and depth map.
Refer to caption: x2.png

We begin with an impulse response captured from the relay wall, noted as $H(x_{p}\to x_{s},t)$. With the virtual illumination source wavefront $\mathcal{P}(x_{p},t)$, the phasor field at the virtual aperture $\mathcal{P}(x_{s},t)$ can be formulated (Liu et al., ; ) as:

$$ $\displaystyle\mathcal{P}(x_{s},t)=\int_{P}\mathcal{P}(x_{p},t)*\left(\frac{1}{ r^{z}}\cdot H(x_{p}\to x_{s},t)\right)dx_{p},$ (1) $$

where $*$ denotes the convolution operator, and $x_{p}$ and $x_{s}$ represent the illumination point and the scanning point, respectively. The term $1/r^{z}$ represents the RIF, where $r$ is the distance between the scanning point and the target point. The parameter $z$, which indicates the attenuation coefficient associated with different surface materials, is the parameter the LPC module is designed to learn.

The $\mathcal{P}(x_{p},t)$, referred to as the illumination function, is defined as a Gaussian-shaped function modulated with the virtual wave $e^{j\Omega_{C}t}$, which can be represented as illumination phasor field $\mathcal{P_{\mathcal{F}}}(x_{p},\Omega)$ in the Fourier domain as Liu et al. ():

$$ $\displaystyle\mathcal{P_{\mathcal{F}}}(x_{p},\Omega)=\delta(x_{p}-x_{vp})\cdot \bigg{(}2\pi\delta(\Omega-\Omega_{C})\underset{\mathcal{F}}{*}\sigma\sqrt{2\pi }\exp\left(-\frac{\sigma^{2}\Omega^{2}}{2}\right)\bigg{)},$ (2) $$

where $\mathcal{F}$ represents the Fourier domain, $x_{vp}$ denotes the position at the virtual light source, $\delta$ is the Dirac function, $\Omega_{C}$ denotes the central frequency of the wave, and $\sigma$ represents the standard deviation. The standard deviation of a Gaussian is inversely proportional to its pass-band width in the frequency domain, which can be learned and adjusted automatically by our APF module.

The point $I(x,y)$ of the hidden object can be reconstructed from $\mathcal{P}(x_{s},t)$ with the wave propagation function $\Phi(\cdot)$, which is modeled by the Rayleigh-Sommerfeld Diffraction integral:

$$ $\displaystyle I(x,y)=\Phi\left(\mathcal{P}(x_{s},t)\right).$ (3) $$

Without loss of generality, considering Poisson noise resulting from ambient light and background noise, the computational model of SPAD sensor (Saunders et al., ; Grau Chopite et al., ) can be written as:

$$ $\displaystyle H^{\prime}(x_{p}\to x_{s},t)\sim\text{Poisson}(H(x_{p}\to x_{s}, t)+B),$ (4) $$

where $B$ represents detected photons from background noise and dark counts (Bronzi et al., ) of SPAD sensors. Poisson($\cdot$) represents the Poisson distribution (Snyder & Miller, ).

### 3.2 Overview

Figure: Figure 3: The pipeline of the LPC.
Refer to caption: x3.png

To address the problems mentioned in Section 1, we integrate the proposed LPC and APF modules into the LFE (Chen et al., ) framework, which comprises a feature extraction module, a wave propagation module, and a rendering module. An overview of the network is shown in Fig. 2. Given transient measurements as input, similar to those described in the literature (Chen et al., ; Li et al., )), the feature extraction module downsamples the transient measurements in both spatial and temporal dimensions and extracts feature embeddings $F_{E}$.

Instead of directly applying the wave propagation module to convert transient measurements to the spatial domain, we first employ the LPC to learn different attenuation coefficients for each scanning position at the aperture. This allows us to compute the corresponding feature compensation amplitudes, resulting in the compensated feature $F_{C}$. Subsequently, the APF module predicts the optimal frequency domain window width for the illumination function, which illuminates $F_{C}$ and generates $F_{A}$. Finally, the wave propagation and rendering module converts $F_{A}$ from the spatial-temporal domain to the spatial domain and renders intensity and depth images. We provide details of the network in the Supplementary Material.

### 3.3 Learning to Compensate Radiometric Intensity Fall-off

To alleviate the aforementioned RIF, we design the LPC module, which can predict the clean transient measurements before attenuation. An overview of the LPC is shown in Fig. 3. Given the features $F_{E}$ from the previous feature extraction module, the LPC first enhances the features using a convolutional layer with normalization, yielding $F_{E}^{{}^{\prime}}$. Let $G_{Z}$ denote the grid representing the distance from the hidden volume to the relay wall, we predefine three path compensation weights $\{(G_{Z})^{r},r=1,2,4\}$, which correspond to different attenuation amplitudes of surface materials, as referenced in O’Toole et al. () and Liu et al. (). The weights and enhanced features are multiplied to obtain initially compensated features $F_{C}^{ini}$, which can be expressed as:

$$ $\displaystyle F_{C}^{ini}=\left\{(G_{Z})^{1},(G_{Z})^{2},(G_{Z})^{4}\right\} \otimes F_{E}^{{}^{\prime}},$ (5) $$

where $\otimes$ denotes the Hadamard product.

After that, the initial compensated features are down-sampled across the spatial dimensions using an average pooling layer. Instead of predicting the RIF term directly, we design the LPC to predict probabilities of initial compensation features first, and then the weights and features are combined through a weighted sum. In such a way, the LPC is capable of explicitly selecting appropriate compensation amplitudes based on physical constraints. The downsampled features thus undergo a series of operations including convolution layers, interpolation, and the Softmax operation, which outputs probabilities. The probabilities and the initial compensated features are then multiplied using the Hadamard product, resulting in compensated features. Subsequently, the compensated features and the input features are added together, outputting the final compensated features.

As demonstrated in Section 4.5, our carefully designed LPC module effectively mitigates the RIF issue, enhancing the reconstruction performance for challenging real-world scenes, especially in complex and distant regions.

### 3.4 Denoising with Adaptive Phasor Field

Figure: Figure 4: The pipeline of the APF. The module predicts the illumination function with an appropriate bandwidth to compensate for the noisy transient features, outputting clean, denoised features.
Refer to caption: x4.png

As described in the imaging formulation in Section 3.1, the transient measurement is illuminated by the virtual illumination function. In the frequency domain, the illumination phasor field $\mathcal{P_{\mathcal{F}}}(x_{p},\Omega)$ acts as a Gaussian filter on the feature of transient measurements, modulated to the central frequency $\Omega_{C}$. It should be noted that not all frequency components contribute positively to the final scene reconstruction, including frequency components associated with noise (Liu et al., ; Hernandez et al., ). Applying an illumination function to the feature of transient measurements can be understood as a process of selecting a certain effective frequency spectrum band $\bigtriangleup\Omega$. The bandwidth of the Gaussian function for the illumination function is decided by the standard deviation, which can be expressed as

$$ $\bigtriangleup\Omega=\frac{1}{2\pi\sigma}.$ (6) $$

For convenience, the $\bigtriangleup\Omega$ is defined as the 3 dB bandwidth. Selecting an appropriate standard deviation is crucial for obtaining clean measurements. However, past works have relied on a single empirical standard deviation, which is not conducive to selecting the correct frequency components for the reconstruction of complicated scenarios.

To address this problem, we devise the APF module to adaptively learn the standard deviation, as illustrated in Fig. 4.
Given the feature $F_{C}$, the first step is to transform the feature into the frequency domain along the temporal dimension. This allows the module to intuitively learn to distinguish between useful information and noise directly in the frequency domain. Subsequently, the Fourier features are convolved across the spatial and the spectrum parts successively to further enhance features. We then employ additional fully connected layers to predict the standard deviations $\sigma_{pred}$ from frequency feature representation, generating the adaptive Gaussian function $K_{G}(\sigma)$ in the frequency domain. As such, the illumination phasor field can be formulated by the adaptive Gaussian function and the virtual waves $e^{j\Omega_{C}t}$, as

$$ $\vspace{-0.2cm}\mathcal{P_{\mathcal{F}}}(x_{p},\Omega)=\delta(x_{p}-x_{vp}) \cdot\bigg{(}\mathcal{F}\left(e^{j\Omega_{C}t}\right)\underset{\mathcal{F}}{*} K_{G}(\sigma_{pred})\bigg{)},$ (7) $$

where

$$ $K_{G}(\sigma)=\sigma\sqrt{2\pi}\exp\left(-\frac{\sigma^{2}\Omega^{2}}{2}\right).$ (8) $$

Finally, the input features $F_{C}$ and the illumination phasor field are convolved across the temporal dimension, as

$$ $F_{A}=F_{C}*\mathcal{P_{\mathcal{F}}}(x_{p},\Omega)=\mathcal{F}^{-1}\bigg{(} \mathcal{F}(F_{C})\cdot\mathcal{F}\left(\mathcal{P}(x_{p},t)\right)\bigg{)},$ (9) $$

where $F_{A}$ is the output feature in the temporal domain at the scanning point, and $*$ means the convolution operator.

As demonstrated in Section 4.4 and Section 4.5, the APF module selectively emphasizes useful information and attenuates noise across various SNR conditions within the transient measurements, thereby boosting the generalization capability and improving the reconstruction quality.

### 3.5 Loss Function

The approach is trained in an end-to-end manner. The total loss consists of the intensity loss and the depth loss, balanced by a regularization weight $\lambda$:

$$ $\displaystyle\mathcal{L}=\mathcal{L_{I}}(I,\hat{I})+\lambda\mathcal{L_{D}}(D, \hat{D}),$ (10) $$

and

$$ $\displaystyle\mathcal{L_{I}}(I,\hat{I})=\frac{1}{N}\sum_{i}^{N}(I_{i}-\hat{I}_ {i})^{2},\mathcal{L_{D}}(D,\hat{D})=\frac{1}{N}\sum_{i}^{N}(D_{i}-\hat{D}_{i}) ^{2},$ (11) $$

where $\hat{I}$ and $I$ denote the reconstructed intensity image and the ground truth, respectively. $\hat{D}$ and $D$ denote the recovered depth map and corresponding ground truth. $N$ denotes the total number of pixels of the intensity image and depth map.

## 4 Experimental Results

**Table 1: Quantitative comparisons of different approaches upon the Seen test set. The best in bold, the second in underline.**
| Method | Backbone | Memory | Time | aIntensity | Depth |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR$\uparrow$ | SSIM$\uparrow$ | RMSE$\downarrow$ | MAD$\downarrow$ |  |  |  |  |
| LCT (O’Toole et al., ) | Physics | 18 GB | 0.11 s | 19.51 | 0.3615 | 0.4886 | 0.4639 |
| FK (Lindell et al., ) | Physics | 26 GB | 0.16 s | 21.69 | 0.6283 | 0.6072 | 0.5801 |
| RSD (Liu et al., ) | Physics | 33 GB | 0.23 s | 21.74 | 0.1817 | 0.5677 | 0.5320 |
| LFE (Chen et al., ) | CNN | 13 GB | 0.05 s | 23.27 | 0.8118 | 0.1037 | 0.0488 |
| I-K (Yu et al., ) | CNN | 14 GB | 0.08 s | 23.44 | 0.8514 | 0.1041 | 0.0476 |
| NLOST (Li et al., ) | Transformer | 38 GB | 0.38 s | 23.74 | 0.8398 | 0.0902 | 0.0342 |
| Ours | CNN | 17 GB | 0.24 s | 23.99 | 0.8703 | 0.0874 | 0.0312 |

### 4.1 Baselines and Datasets

Baselines selection. To assess the efficacy of our proposed approach, we undertake thorough validations by comparing it against several baseline approaches on the synthetic and real-world datasets. These baselines encompass three traditional approaches commonly used in the field: LCT (O’Toole et al., ), FK (Lindell et al., ), and RSD (Liu et al., ), as well as three learning-based approaches: LFE (Chen et al., ), I-K (Yu et al., ), and NLOST (Li et al., ).

Public data. For the synthetic dataset, we utilize a publicly available dataset generated from LFE (Chen et al., ). A total of 2704 samples are used for training and 297 samples for testing, denoted as Seen test set. Each transient measurement possesses a resolution of 256$\times$256$\times$512, with a bin width of 33 ps and a scanning area of 2m$\times$2m. To assess the generalization capabilities, we rendered 500 transient measurements from the objects not included in the Seen test set, denoted as Unseen test set. For qualitative validation, particularly in complicated scenarios, we employ publicly available real-world data from FK (Lindell et al., ) and also the data from NLOST (Li et al., ) with low SNR conditions. For example, instead of the commonly used measurements with 180 minutes acquisition time, we utilize the measurements with 10 minutes acquisition time in FK (Lindell et al., ). We preprocess the real-world data for testing, and the real-world data has a spatial resolution of 256$\times$256 and a bin width of 32ps.

Self-captured data. To further increase the diversity of NLOS data, we also capture additional real-world measurements using our own active confocal imaging system. The system utilizes a 532 nm VisUV-532 laser that generates pulses with an 85 picosecond width and a 20 MHz repetition rate, delivering an average power output of 750 mW. The laser pulses are directed onto the relay wall using a two-axis raster-scanning Galvo mirror (Thorlabs GVS212). Both the directly reflected and diffusely scattered photons are then collected by another two-axis Galvo mirror, which funnels them into a multimode optical fiber. This fiber channels the photons into a SPAD detector (PD-100-CTE-FC) with approximately 45% detection efficiency. The motion of both Galvo mirrors is synchronized and controlled via a National Instruments acquisition device (NI-DAQ USB-6343). The TCSPC (Time Tagger Ultra) records the pixel trigger signals from the DAQ, synchronization signals from the laser, and photon detection signals from the SPAD. The overall system achieves a temporal resolution of around 95 ps. During data acquisition, the illuminated and sampling points remain aligned in the same direction but are intentionally offset to prevent interference from directly reflected photons. As such setting, we capture three transient measurements from customized scenes, each containing different types of surface materials. All measurements are captured over a duration of 10 minutes.

### 4.2 Implementation Details and Metrics

We implement our approach using the PyTorch framework (Paszke et al., ). For optimization, we employ the Adam optimizer (Kingma & Ba, ) with a learning rate of $6\times 10^{-5}$ and a weight decay of 0.95. The $\lambda$ is set to 1. Baseline approaches are implemented using their respective public code repositories. The batch size is uniformly set to 1 for all approaches. Training is conducted for 50 epochs using a single NVIDIA RTX 3090 GPU, except for NLOST, which is trained on Tesla A100 GPUs. Due to memory consumption, NLOST is trained on transient measurements with the shape of $128\times 128\times 512$, and the results are interpolated to $256\times 256$ for comparison.

For quantitative evaluation in intensity reconstruction, we adopt peak signal-to-noise ratio (PSNR) and structural similarity metrics (SSIM) averaged on the test set. For depth reconstruction, we compute the root mean square error (RMSE) and mean absolute distance (MAD) for test samples. Following Li et al. (), we crop the central region for a more reliable evaluation.

### 4.3 Comparison on Synthetic Data

Figure: Figure 5: Intensity results recovered by different approaches on the Seen test set. GT means ground truth of the intensity images.
Refer to caption: x5.png

Quantitative evaluation. The quantitative evaluations presented in Table 1 demonstrate that our approach achieves decent advancements in NLOS reconstruction. For the synthetic results, our approach outperforms all competitors in terms of all evaluation metrics. Specifically, our approach exhibits a substantial enhancement over traditional approaches, achieving a 2.25 dB increase in PSNR compared to the leading approach RSD. Furthermore, when compared with the recent state-of-the-art (SOTA) learning-based approaches I-K and NLOST, our approach still achieves a 0.55 dB and 0.25 dB improvement in PSNR, respectively. The merits of our approach are further substantiated by the highest SSIM for intensity, which underscores the superior capability of our network in preserving the structural integrity of hidden scenes. Additionally, for the depth estimation, our approach reduces the RMSE and MAD metrics by 3.10% and 8.77%, respectively, over the strongest competitor NLOST.

Notably, the existing Transformer-based SOTA approach NLOST requires approximately 38 GB of GPU memory and a substantial amount of inference time. In contrast, our approach achieves higher performance while using only half the memory and requiring less inference time.

Figure: Figure 6: Depth error maps from different approaches on the Seen test set. The first column denotes the ground-truth depth map, and the other columns indicate the depth error maps. The color bars show the value of depth and error maps, respectively.
Refer to caption: x6.png

Qualitative evaluation. We present the qualitative results of intensity images and depth error maps for visualization comparisons, depicted in Fig. 5 and Fig. 6. Regarding the intensity visualization comparisons, LCT reconstructs the main content yet sacrifices details, FK fails to recover most of the structural information, and RSD introduces significant noise in the background. The LFE and I-K perform better than traditional approaches but still lack details. Compared to the SOTA approach NLOST, our approach generates content with greater fidelity and high-frequency details (e.g., the texture of the scene in the first row). In terms of the depth error map, the blue regions dominate the scene in the error map corresponding to our approach, indicating the smallest magnitude of the error. In contrast, traditional approaches as well as LFE demonstrate a greater tendency for errors, as shown by the increased presence of red parts, especially in distant regions (e.g., the right part of the motorcycles in the second row). These areas are challenging due to the complex geometrical features and distinct RIF degrees with different kinds of materials. While I-K and NLOST show improvement over the former approaches, they still fail to precisely estimate the depth in the wheel area, where our approach succeeds.

Figure: Figure 7: Visualization comparison on the public real-world data (Lindell et al., ; Li et al., ). The left annotation indicates the shortest acquisition time in total. Zoom in for details.
Refer to caption: x7.png

Figure: Figure 8: Visualization comparison on our self-captured real-world data. The left annotation indicates the total acquisition time. Zoom in for details.
Refer to caption: x8.png

Generalization evaluation. To further validate the network’s generalization performance, we conduct quantitative tests under varying SNR conditions. Specifically, we test different approaches with the Unseen test set under varying SNR levels (10 dB, 5 dB, and 3 dB) of the Poisson noise. Extreme SNR conditions make separating background noise from the limited number of collected photons more challenging, while the new scenes in the Unseen test set validate the performance when transferred to unknown domains. As can be seen in Table 2, in most cases, our approach achieves the best results compared to other approaches. These outstanding results demonstrate the superior generalization performance of our approach in dealing with test data that is distinct from the training data. This superiority is then further verified on various real-world data that has no ground truth below.

### 4.4 Comparison on Real-world Data

Public data. Results on two public NLOS datasets are presented in Fig. 7. When utilizing measurements with reduced acquisition time, nearly all approaches, except for NLOST and our approach, produce reconstructions with significant noise. The traditional approaches, while reconstructing main content, produces blurred results. LFE and I-K manage to reconstruct more objects but struggle to capture high-frequency details. NLOST excels in reducing background noise, but it still misses certain details such as the legs of the deer and the intricate patterns of the tablecloth. Our approach shows remarkable resilience to variation in different acquisition times, consistently delivering detailed reconstructions comparable to those of the same objects captured at high acquisition time. The exceptional robustness of our approach demonstrates the superior generalization ability over the existing approaches.

Self-captured data. Apart from the public data, we also capture several new scenes with our own NLOS system for further assessment. We present results from three distinct scenes: one depicting retro-reflective letters arranged on a ladder (referred to as ‘ladder’), another featuring a panel composed of multiple A4 sheets inscribed with ‘123XYZ’ (referred to as ‘resolution’), and the third containing multiple objects with varying surface materials (referred to as ‘composite’). As shown in Fig. 8, it can be observed that learning-based approaches still exhibit less reconstruction noise compared to traditional approaches. In the low SNR scenario of the ‘ladder’, other approaches either fail to reconstruct or produce poor-quality reconstructions. However, our reconstruction exhibits notably high quality, with the ladder legs even discernible. In the heavily attenuated diffuse reflection scenario ‘resolution’, our approach still manages to reconstruct relatively clear details. In the ‘composite’ scene, which includes depth variations and multiple surface materials, our approach produces reconstruction with the least noise and the most complete structural information (e.g., the lower edge of the bookshelf and the letter ‘S’ in the upper right of the scene). The promising outcomes achieved by our approach underscore its superiority over existing approaches.

**Table 2: Quantitative results on the Unseen test set under different SNRs. The best in bold, the second in underline.**
| Method | Intensity (PSNR$\uparrow$ / SSIM$\uparrow$) | Depth (RMSE$\downarrow$ / MAD$\downarrow$) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| 10 dB | 5 dB | 3 dB | 10 dB | 5 dB | 3 dB |  |
| LCT | 18.92 / 0.1708 | 18.38 / 0.1195 | 18.06 / 0.1007 | 0.6992 / 0.6499 | 0.7490 / 0.1195 | 0.7666 / 0.7197 |
| FK | 21.62 / 0.6496 | 21.62 / 0.6471 | 21.62 / 0.6452 | 0.5813 / 0.5562 | 0.5672 / 0.5427 | 0.5598 / 0.5351 |
| RSD | 22.77 / 0.2045 | 22.48 / 0.1510 | 22.24 / 0.1280 | 0.4198 / 0.3934 | 0.3679 / 0.3358 | 0.3496 / 0.3160 |
| LFE | 23.22 / 0.8122 | 23.15 / 0.7951 | 23.10 / 0.7805 | 0.1036 / 0.0484 | 0.1041 / 0.0491 | 0.1044 / 0.0496 |
| I-K | 23.45 / 0.8386 | 23.38 / 0.8020 | 23.32 / 0.7689 | 0.1045 / 0.0500 | 0.1071 / 0.0571 | 0.1099 / 0.0636 |
| NLOST | 23.63 / 0.7747 | 23.74 / 0.8294 | 23.71 / 0.8135 | 0.0939 / 0.0409 | 0.0909 / 0.0351 | 0.0918 / 0.0368 |
| Ours | 23.91 / 0.8577 | 23.83 / 0.8387 | 23.80 / 0.8645 | 0.0893 / 0.0333 | 0.0914 / 0.0365 | 0.0902 / 0.0332 |

### 4.5 Ablation Studies

In this section, we ablate the contribution of the modules. As shown in the qualitative results in Fig. 9, the LPC and the APF modules each contribute to improving the performance of the approach in distinct ways, with their combination yielding the best results. Specifically, it can be seen that the network without the proposed modules loses image details and contains significant noise in the reconstruction. In contrast, introducing the LPC module enhances object details (e.g., the deer’s legs), and introducing the APF module suppresses background artifacts. When both the APF and the LPC modules are integrated, the network produces images with complete details and clear boundaries.

Figure: Figure 9: Ablation results on public real-world data. Baseline denotes w/o LPC and APF modules. The total acquisition time of the left and right scenes is 10 min and 0.3 min, respectively.
Refer to caption: x9.png

## 5 Discussion and Conclusion

In this paper, we propose a novel learning-based approach for NLOS reconstruction including two elaborate designs: learnable path compensation and adaptive phasor field. Experimental results demonstrate that our proposed solution effectively mitigates RIF and improves the generalization capability. Additionally, we contribute three real-world scenes captured by our NLOS imaging system.
The future work of our study is twofold. We conduct experiments on the confocal imaging system, with the extension to the non-confocal imaging system being one direction of our future research. The modeling of the SPAD acquisition process still exhibits a certain gap from the real-world sensor, and considering additional factors remains a focus for future research.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: How much does the initial compensated feature in Eq. 5 contribute to the method?

What is the motivation behind average pool as opposed to max pool which would pick the prominent feature?

For the LPC pipeline, how much is the 3D CNN contributing as opposed to the initially compensated feature?

What is the network architecture of spectrum convolution in Figure 4? 

How many different materials were used in the captured scene?
soundness: 3
strengths: Typically training on synthetic data does not generalize to real data due to synthetic and real gap. However, the proposed method learns to compensate for radiometric falloff and suppress background noise using synthetic training data and is able to generalize to real-world data.
summary: The paper proposes to learn the physical priors for  non-line-of-sight imaging. More specifically two components are proposed, learnable path compensation to attenuate the features to compensate for radiometric falloff and an adaptive bandpass filter to suppress background noise. Training and quantitative evaluations are done on synthetic data. Qualitative evaluations are done on real-world data including a three-scene dataset captured by the authors. The method shows better performance in terms of preserving sharp features and reducing background noise.
weaknesses: The paper lacks discussion on the choice of G_z for initial compensation. It’s also unclear how much the 3D CNN block is contributing compared to the initially compensated features.

The paper shows results on a custom dataset with the “composite” scene consisting of multiple surface materials. However, there is no quantitative evaluation on how different surface materials at different distances are compensated by LPC.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Major:
Reiterating some points from weaknesses:
- I would recommend the authors include a image plot of the compensation coefficients predicted by the learnable path compression, and would expect that regions per object would have their own coefficient each, because I wonder if the model can actually infer that different materials should have different compensation coefficients like the authors claim. 
- I recommend that the authors provide some visualization of the standard deviations for the gaussian of the illumination phasor field predicted by their adaptive phasor field to better help the reader understand what the method is actually doing, because I have questions about if the adaptive standard deviations used in the wave propagation are actually reducing the noise in low SNR environments like the authors claim. Specifically, I would expect for low SNR environments the standard deviation to be lower and vice versa, because low SNR implies higher noise and thus the model should use a tighter gaussian during wave propagation to filter out more noise. 

I would be willing to upgrade my score if the authors could provide more evidence that the method works the way they claim. 

Minor:
- I don't quite understand in equation (8) why the inverse Fourier transform is taken, because according to Figure 2, $F_A$ goes through a Fourier transform before getting input into the wave propagation model. It seems to me to be more concise and efficient to keep the result of convolving the illumination phasor field with the compensated features in the frequency domain for wave propagation rather than transforming it back to the spatial/time domain.
- For the real data at varying SNR levels, 10 dB, 5 dB, and 3 dB, how did the authors capture this data? I would expect that the exposure time is different between captures, but I couldn't see any details about this in the paper.
soundness: 4
strengths: The paper is well motivated and distinguished clearly from previous works, as the method is structured around leveraging flexibility in the virtual phasor field formulation that was not used by prior methods to improve NLOS reconstruction performance. The improvement in reconstruction performance is clearly demonstrated with extensive experiments on existing and newly gathered datasets across a large number of NLOS reconstruction methods.
summary: This paper proposes a learning-based Non-line-of-sight (NLOS) reconstruction method based on the standard virtual wave phasor field formulation that learns radiometric intensity falloff coefficients to compensate for falloff due to varying materials present in the scene and frequency cutoff values to compensate for varying SNR conditions.

EDIT: upgraded score from a 5 to a 6 given authors response.
weaknesses: Though the method shows strong performance compared to baselines, I'm not sure the method is better for the reasons that the authors say, rather than just because the method has more parameters and is trained on more data than previous methods. In particular, the authors make several claims about why their method works better that I am not sure are precisely substantiated in the paper. 

First, the authors claim that the using adaptive coefficients for the radiometric intensity falloff is better because different materials will have different coefficients and their design can adapt to the varying materials. For example, diffuse objects should have a falloff coefficient of 4 and retro-reflective 2. While the authors do visualize in Figure 1 that their method using adaptive coefficients can recover both diffuse and retroreflective objects simultaneously, I could not find in the paper any visualizations of the compensation coefficients. I would recommend the authors include a image plot of the compensation coefficients, and would expect that regions per object would have their own coefficient each. 

Second, the authors claim that the adaptive phasor field improves performance across a variety of SNR conditions. However, in Table 2, which contains the quantitative results across a variety of SNR conditions, the decrease in performance as SNR decreases for the author's proposed method is similar to the other methods. Thus, given the data presented in the paper it seems that it is not the case the author's proposed method is more robust in lower SNR conditions than other methods, despite having uniformly better performance across SNR settings. I'm not sure if it's because the SNR conditions in the testing sets are not varied enough, or if the adaptive phasor field doesn't actually help in diverse SNR settings. Regardless, I recommend that the authors provide some visualization of the standard deviations for the gaussian of the illumination phasor field predicted by their adaptive phasor field to better help the reader understand what the method is actually doing.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1.	$P(x_p,t)$ is the virtual illumination function in the Phasor-field methods (Liu et al.,2019), but it is not necessary a Gaussian-shaped function. It can be any type of the light source function, such as ambient light. Please refer to Table S.2 of Liu’s paper. Therefore, the physical explanation of learning a Gaussian window is not correct in my opinion. 
2.	Even though some light source functions are Gaussian-shaped, such as pulsed point light function, the frequency spectrum band should be a certain value. It shouldn’t be a changeable value for different scenarios. So, from this point of view, the explanation of learning a Gaussian window is also not correct.
3.	The $I(x,y)$ in Eq.(3) should be the image of the virtual imaging system, it is not the hidden object as claimed in the paper, please refer to the explanation of Eq.(6) in Liu’s paper.
4.	Could you explain in what dimensions of measurements you did the FFT to get the spectrum of the light source function? I think you can only get the spatial and temporal sampling frequency if you operate FFT on raw ToF measurement, rather than illumination spectrum.
5.	Lines 78-79 claims that the proposed method can solve the dark counts noise of SPAD and ambient light. But I didn’t find the detailed solution in the main paper. There is just a simple description of the definition in Eq.(4).
6.	Why did you choose 3 path compensation weights? I didn’t find the related content in O’Toole et al., 2018. Could you clarify it for me?
7.	Which method is chosen as your baseline in Fig.9. I suppose to be LFE according to Fig.2. But the baseline results shown in Fig.9 are different with the LFE results in Fig.7. So, it’s hard to demonstrate the effectiveness of this ablations studies.

## Reference
- Chen et al, Learned feature embeddings for non-line-of-sight imaging and recognition, TOG 2020
- Liu et al, Non-line-of-sight imaging using phasor-field virtual wave optics, Nature, 2019
- O’Toole et al, Confocal non-line-of-sight imaging based on the light-cone transform, Nature 2018

------
**post-rebuttal:**

My questions are properly answered. However, it also reveals the authors' original motivation (the physical intuition and explanation) is somewhat far-fetched. As a result, I raise my score to borderline reject.
soundness: 2
strengths: The originality of this paper arises from the improvement of an existing NLOS reconstruction network LFE (Chen et al., 2020). The proposed method can achieve global intensity consistency and avoid manual adjustment of empirical parameters.
summary: This paper aims at solving the problem that existing NLOS methods rely on some empirical physical prior during their implement. To address this problem, the authors introduce a Learnable Path Compensation (LPC) module and an Adaptive Phasor Field (APF) module to optimize 3 distance-related weights and a Gaussian window separately. The results on both synthetic data and real data demonstrate the effectiveness of the proposed method.
weaknesses: IMO, the physical explanation of this paper is inaccurate and misleading, especially the explanation of the learning of the Gaussian window. And the improvement of the results is limited quantitatively and qualitatively, especially compared with NLOST.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: - why is the rendering model called 'rendering'? Is not clear to me why the relatively generic 2D CNN architecture should be a renderer.
soundness: 3
strengths: - the proposed architecture incorporates insights from virtual wave phasor fields.
- the method achieves state-of-the-art performance for NLOS reconstruction.
summary: The paper presents a new end-to-end network architecture for reconstructing non-line-of-sight (NLOS) scenes from low signal-to-noise ratio (SNR) single-photon avalanche diode measurements. The network architecture is inspired by the virtual wave phasor field, and introduces two new modules, ie the learnable path compensation module and the adaptive phasor field module. 
The proposed method is shown to perform better than previous NLOS approaches (both learned and physics-based) on a public dataset and a new dataset introduced in the paper.
weaknesses: - I found the presentation of the method hard to follow, especially for someone who is not an expert in virtual phasor fields, and in particular the paper "Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems". I believe that an ICLR submission should be accessible to a larger audience. 
    - It was unclear to me how equation (4) and (1) are related, since $H$ is inside an integral in (1)
    - the notation is confusing in equation 3: how are the coordinates $x,y$ in $I$ related to $x_s$? Is $x_s$ referring to a 2D coordinate whereas $x$ and $y$ are 1-D coordinates respectively?
    - there should be at least some intuitive explanation of the function $\Phi$ in equation 3.

- Despite the state-of-the-art performance, the gap with the previous best performing model is relatively mild (often less than 0.5 dB).

Meta-review: This paper aims to enhance non-line-of-sight (NLOS) imaging performance through active time-transient imaging with SPAD. The authors argue that traditional methods rely on empirical physical priors, which lack generalization abilities. They demonstrated improved results with both "seen" and "unseen" data compared to existing methods. However, two reviewers (M5tk and iqtw) raised concerns regarding the physical interpretation and the actual reasons behind the performance improvements. After extensive discussion, the authors acknowledged that some aspects of their presentation were improper, while the reviewers were partially persuaded. Reviewer iqtw also highlighted the marginal improvement over NLOST. In response, the authors explained the differences in performance based on the network structure and parameters (UNet vs. Transformer).

The AC conducted a detailed review of the performance gains over NLOST. It was observed that for all "seen" cases, the improvement is marginal. In the "unseen" cases with high SNR, the performance gains are similarly minor. These findings undermine the authors' claims regarding the superiority of their method in terms of physical priors and generalization ability. Furthermore, the authors' feedback to Reviewer iqtw about performance gains over NLOST heightened the AC’s concerns, as the paper's motivation did not initially include any discussion on network structure or parameter differences (UNet vs. Transformer). It would have been straightforward for the authors to isolate the effect of the network used by employing the same architecture. Additionally, the learned coefficient maps shown in Fig. 12 do not align well with the scene structure, which weakens the authors' claims to some extent.

Given the concerns regarding the physical interpretation, the explanation of performance improvements, and the lack of sufficient justification for the paper's motivation, the AC finds the foundation of the work insufficiently supported. The final recommendations from the four reviewers are split, with 2 marginal accepts and 2 marginal rejects. Considering the concerns outlined above, the paper’s key motivation and the principled interpretation of the underlying physics need to be revisited and validated with experimental results. Thus, the AC recommends rejecting this paper in its current form.

Decision: Reject