Virtual Scanning: Unsupervised Non-line-of-sight Imaging from Irregularly Undersampled Transients

Authors: xingyu  cui
huanjing  yue
song  li
xiangjun  yin
yusen  hou
yun  meng
kai  zou
xiaolong  hu
jingyu  yang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-05 14:07:08

Abstract: Non-line-of-sight (NLOS) imaging allows for seeing hidden scenes around corners through active sensing.
Most previous algorithms for NLOS reconstruction require dense transients acquired through regular scans over a large relay surface, which limits their applicability in realistic scenarios with irregular relay surfaces.
In this paper, we propose an unsupervised learning-based framework for NLOS imaging from irregularly undersampled transients~(IUT).
Our method learns implicit priors from noisy irregularly undersampled transients without requiring paired data, which is difficult and expensive to acquire and align. 
To overcome the ambiguity of the measurement consistency constraint in inferring the albedo volume, we design a virtual scanning process that enables the network to learn within both range and null spaces for high-quality reconstruction.
We devise a physics-guided SURE-based denoiser to enhance robustness to ubiquitous noise in low-photon imaging conditions. 
Extensive experiments on both simulated and real-world data validate the performance and generalization of our method.
Compared with the state-of-the-art (SOTA) method, our method achieves higher fidelity, greater robustness, and remarkably faster inference times by orders of magnitude.
The code and model are available at https://github.com/XingyuCuii/Virtual-Scanning-NLOS.

Main paper: 
 introduction: 
 Non-line-of-sight (NLOS) imaging aims to reconstruct hidden scenes beyond the direct line of sight of the detector, garnering interest across various fields such as robot vision, autonomous driving, rescue operations, remote sensing, and medical imaging [1] [2] [3] [4] [5] . In a typical active confocal NLOS imaging system, as depicted in Fig. 1 , a laser source and a detector are both focused on the same point on a relay surface. Pulses emitted by the laser reflect off the surface to illuminate the hidden scene. The detector captures photons bouncing back from the scene toward the relay surface, referred to as transients, from which the hidden scene can be recovered using elaborately designed algorithms.While existing works have achieved remarkable breakthroughs, they also face significant limitations that hinder their practical applicability. These methods assume dense and regular scanning of a large relay surface, which may not be feasible in realistic scenarios with irregular relay surfaces such as latticed windows or fences. Transients obtained through irregular undersampling can result in severe ill-posedness, leading to artifacts or reconstruction failure. This raises the challenging task of NLOS imaging from irregularly undersampled transients (IUT). To address this, Liu et al. [6] proposed introducing manually designed strong regularization terms under a functional optimization framework. However, this approach is hindered by the need for lengthy numerical iterative computations. Recent learning-based NLOS imaging methods [7] [8] [9] [10] enable quick inference, but they require supervision from a large amount of paired data, including well-aligned ground-truth albedos, which are difficult and expensive to acquire. Therefore, exploring NLOS imaging under unsupervised learning is worthwhile to eliminate the heavy reliance on paired data.In this paper, we propose a novel unsupervised framework capable of learning implicit priors from noisy IUT. This framework comprises two components: a virtual scanning reconstruction network (VSRnet) that learns high-quality measurement-to-albedo mapping beyond the range space induced by the measurement consistency, and a SURE-based denoiser that enhances our method's robustness to measurement noise by incorporating the physics model of the low-photon time-resolved detector. We conduct extensive experiments on simulated data, publicly available real data, and data acquired from our self-built NLOS system. Our method outperforms existing algorithms, particularly in providing robustness for real data and diverse irregular relay surfaces.Our main contributions are summarized as follows:• We propose an unsupervised NLOS imaging framework capable of learning implicit priors from noisy IUT, effectively overcoming the dependency on paired data that is difficult to acquire and align.• We introduce a virtual scanning process that enables the network to learn within both range and null spaces for high-quality reconstruction from IUT, extending NLOS imaging to realistic scenarios with irregular relay surfaces.• We propose a SURE-based denoiser, an unsupervised physics-guided module that incorporates the low-photon time-resolved detector's physics model, to enhance our method's robustness to noise.• We evaluate our method on simulated, publicly available, and self-captured real data, demonstrating its superior reconstruction quality and significantly faster inference compared to the SOTA method.2 Related work 
model-based nlos reconstruction: 
 Model-based NLOS algorithms have achieved significant advances in recent years. Direct reconstruction algorithms [11] [12] [13] [14] [15] offer rapid implementations for NLOS imaging, while iterative algorithms [16] [17] [18] 5 ] leverage more accurate physical models for higher reconstruction quality. Recent efforts aim to extend NLOS imaging to challenging real-world scenarios. For instance, Manna et al. [19] and Gu et al. [20] addressed NLOS imaging with dynamic and non-planar relay surfaces, respectively. However, these methods still rely on large relay surfaces and dense measurements. Another set of algorithms [21] [22] [23] [24] [25] aims to achieve high spatial resolution reconstruction using sparse sampling measurements to significantly reduce acquisition time. Yet, they are constrained to specific scanning patterns, such as regular or Hadamard patterns. To address NLOS imaging from irregularly undersampled transients, Liu et al. [6] introduced a reconstruction model using Confocal Complemented Signal-Object Collaborative Regularization (CC-SOCR). Despite its effectiveness, CC-SOCR suffers from long inference times due to its iterative nature.3 Problem formulation and motivation 3.1 NLOS Imaging from IUT Fig. 1 illustrates confocal NLOS imaging with time-resolved systems. By scanning a set of points P = {(x i , y i , 0) | i = 1, 2, . . . , s, x i , y i ∈ R} on the relay surface, the forward model of NLOS imaging can be modeled asEQUATION)where τ is the spatial-temporal measurement, p = (x, y, 0) denotes the scanning point, κ(q) denotes the albedo value at point q in the 3D hidden scene Q, c is the speed of light. The distance ∥p -q∥ is related to the time of flight t through the Dirac delta function δ(•). For compact presentation and analysis, we employed a discretized version of the above forward model. Let u ∈ R st and ρ ∈ R l 2 z represent the vectorized measurements and albedos of the hidden object, respectively, where l denotes the size of the vertical and horizontal dimensions, and z denotes the depth dimension. We denote the forward operator, also known as the light transport matrix, by H ∈ R st×l 2 z . The forward processing can be described by the following linear model:EQUATIONNotably, the forward operator H is related not only to the optical-electronic characteristics of the NLOS system but also to the scannable region on the relay surface. 
motivation: 
 Deep learning-based algorithms have demonstrated significant potential to enhance NLOS imaging performance compared to model-based approaches. Most prior work has adopted a supervised paradigm, which requires substantial amounts of high-quality paired data. However, it is prohibitively expensive or even infeasible to acquire ground-truth 3D albedo volumes precisely aligned with the spatio-temporal measurements. This limitation motivates us to develop an unsupervised NLOS reconstruction framework that avoids this dependency and further enhances the generalization of learning-based methods to real-world data.The standard approach in unsupervised learning is to train the reconstruction mapping f θ by minimizing the measurement consistency (MC) loss:EQUATIONNevertheless, solely enforcing the MC loss without ground truth supervision does not guarantee highquality reconstruction. This can be analyzed from the perspective of the range-null decomposition [29] .Let N H = {v ∈ R l 2 z | Hv = 0} be the null space of the operator H. Its complementary space is the range space of H ⊤ , denoted byR H = {H ⊤ u, u ∈ R st }, such that R l 2 z = R H ⊕ N H .Any albedo volume ρ can be decomposed into a range-space component and a null-space component. The virtual scanning process, which involves virtually observing ρ (1) with a relay surface M k that is distinct from M g and enforcing consistency between ρ (1) and ρ (2) .where This necessitates an unsupervised framework capable of learning beyond the range space. We note that model-based algorithms [18, 6, 25, 23] manually design regularization terms to learn beyond range space, but suffer from long inference times. In other computational imaging tasks, supervised methods [30, 31, 29, 32] and unsupervised methods [33] [34] [35] [36] [37] have been proposed to recover nullspace components of the reconstructions. Along this avenue, we propose an effective unsupervised framework capable of learning in both range-null spaces for NLOS imaging from IUT.H † ∈ R l 2 z×stQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: We provide the analysis of the motivation in Sec. 3.2 and the virtual scanning strategy in Sec. 4.2. And we provide the derivation of the proposed SURE-based loss in Sec. C.2.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
unsupervised framework: 
 Fig. 2 shows the proposed unsupervised framework via virtual scanning for NLOS imaging from IUT. The framework consists of two components: 1) a virtual scanning reconstruction network (VSRnet) to recover the 3D albedo volume from both range and null space, and 2) a SURE-based denoiser to enhance the robustness to ubiquitous noise in transients. Given a set of G noisy irregularly undersampled transients U = {ũ g ∈ R st | g = 1, . . . , G} and the set of their corresponding forward operators Sec. 3.1, due to the one-to-one correspondence between the forward operator and the relay surface, we will loosely use H g to refer to different relay surfaces without ambiguity in the following sections.H = {H g ∈ R st×l 2 z | g = 1, . . . ,We first briefly sketch the workflow of the inference stage, as shown in Fig. 2 (a) and 2(b). The noisy IUT ũg , observed through an irregular relay surface H g , is first zero-padded to ũgr on a full scanning grid. This is then passed through the SURE-based denoiser F ϕ to remove noise. Subsequently, the albedo volume ρ is reconstructed using VSRnet f θ , which incorporates the physical prior of NLOS imaging (LCT [11] ) and a learned reconstruction mapping F θ . During the training stage, the denoiser is regularized by the SURE loss, while the reconstruction network is regularized by the MC loss. Additionally, we introduce the virtual scanning process (Fig. 2(d) ) to capture the measurement details within the null space of observation operators. In this process, the reconstructed ρ (1) is projected into the measurement space as u kr by virtually scanning with a different relay surface H k ∈ H, which is distinct from H g . The virtual scanned measurement u kr is then projected back into the reconstruction space as ρ (2) by the reconstruction network. We impose a virtual scanning (VS) loss between the two reconstructed volumes, ρ (1) and ρ (2) , to promote null-space learning (Sec. 4.2). The modules of our framework and the loss functions utilized for training are detailed in the following subsections. The structures of F ϕ and F θ are detailed in the Supplementary Material (SM).Strategy The training strategy of VSRnet is depicted in Fig. 2(b ) and 2(d). Specifically, at the reconstruction module (Fig. 2(b) ), the denoised sample ûgr is initially transformed to the albedo domain using the inverse operator of LCT H † r . The resulting H † r ûgr is then mapped to the albedo volume ρ (1) by the reconstruction network F θ with a global residual connection. At the virtual scanning module, the reconstructed albedo volume ρ (1) is projected into a virtual undersampled measurement u kr using another forward operator H k ∈ H (H k ̸ = H g ). To achieve efficient implementation of forward operators, we decouple H k into M k ⊙ H r , which can be efficiently computed using Hadamard product and the fast Fourier transform. In practice, M k ∈ R l×l×z is the 3D binary mask associated with the relay surface, constructed by repeating the 2D sampling pattern t times along the time dimension. H r represents the forward operator of LCT. Following the same reconstruction pipeline, an albedo volume, denoted by ρ (2) , is obtained from the virtual measurement u kr . The processes of obtaining ρ (1) and ρ (2) can be formalized as:EQUATIONThe two reconstructed albedos, ρ (1) and ρ (2) , should be identical if the learned mapping F θ enables perfect reconstruction. This motivates us to impose a proximity constraint between ρ (1) and ρ (2) in the albedo domain, named the virtual scanning loss. The virtual scanning process in the training pipeline facilitates the learning of the null-space component, thereby providing a promising prior that complements the range space, as analyzed below.Analysis Fig. 3 provides a more intuitive understanding of how the proposed virtual scanning facilitates the network in learning the null-space component. Let H 1 and H 2 be two observation operators associated with two different relay surfaces. We observe ρ using the forward operator H 1 , obtaining the measurement u 1 . As illustrated in Fig. 3 (a), ρ is projected into the range-space R H1 of H 1 . When we only impose the MC constraint, the resulting output ρ (1) will belong to the following set X H1 :EQUATION) As depicted by the blue dashed line in Fig. 3(b) , there exist multiple outputs that satisfy the measurement consistency. If, for instance, we obtain inaccurate estimated results denoted as ρ (1) through ρ (1) = D r (ρ) + F θ (D r (ρ)), we then utilize H 2 to virtually scan ρ (1) and project it into the range space R H2 . Subject to the constraint ρ (1) = ρ (2) , F θ (D r (ρ (1) )) converges to D n (ρ (1) ) due to the relationship ρ (1) = D r (ρ (1) ) + D n (ρ (1) ). Following this iteration, the network can learn within R H1 and N H2 . The entire process is illustrated in Fig. 3(c ). Similarly, by altering the order of operators H 1 and H 2 , F θ will also observe within R H2 and N H1 . In practice, a set of operators H is provided to enhance the network's generalization across various operators with different relay surfaces.NLOS imaging inherently operates under photon-limited conditions, and noise in transient measurements can lead to severe background artifacts in the albedo space. This hinders the network's ability to learn implicit priors from noisy IUT, especially in unsupervised learning. Inspired by previous studies [38] [39] [40] [41] [42] , we introduce a physics-guided unsupervised denoiser to suppress measurement noise. Specifically, we leverage Stein's Unbiased Risk Estimator (SURE) [43] to derive an unsupervised learning loss function that considers the physical model of the time-resolved detector in NLOS imaging systems, which can be modeled asEQUATIONwhere b represents the dark counts of the detector along with the background photons. The parameterized denoiser F ϕ learns to map the noisy measurement ũ to its clean version u via minimization of the SURE loss function given in Eq. ( 8).Given a measurement set {ũ i,g , i = 1, 2, . . . , I, g = 1, 2, . . . , G} collected by observing I hidden scenes, each with G relay surfaces, the training of our framework involves three loss functions: the SURE loss L SURE , the MC loss L MC , and the VS loss L VS . The SURE loss is an unbiased estimation of the mean squared error (MSE) under the Poisson noise model taking dark count consideration:EQUATIONwhere ε is a positive number, e i,g ∈ {-1, 1} st is a binary vector, whose elements follow a Bernoulli distribution with equal probability [40] , and ⊙ denotes element-wise multiplication. Detailed derivation of the SURE loss is given in supp. material.We adopted the mean squared error for L MC and L VS :EQUATION)We first trained the denoiser F ϕ using the SURE loss. Next, we froze the SURE-based denoiser and trained the network F θ with a combined loss function,L(θ) = L M C (θ) + βL V S (θ), where β is a trade-off parameter (see SM for training details).Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We describe proposed framework in Sec. 4. We present proposed network architecture in Sec. A and provide training details in Sec. B. 
experiment setup: 
 Dataset We generated 8,000 transients using the transient rasterizer from [8] dataset contains 111 objects from the alphanumerics dataset [9] , covering lowercase and uppercase letters from the English and Greek alphabets, and numerals from 0 to 9. We also rendered a sample "bunny" for quantitative comparison. To assess our method's generalization, we tested it on real-world data acquired by three different systems [10] [11] [12] and a self-built system (see SM for system details).The hidden scenes feature various reflective materials, depth ranges, and geometric shapes, and were acquired under different conditions including scanned areas, spatial resolutions, bin widths, and integration times. For a fair comparison with CC-SOCR [6] within a manageable time frame, we resized the full-sampled transients [10, 12] to 128 × 128 × 256 and the full-sampled transients [11] to 64 × 64 × 256. We used these resized transients for all methods.Relay surface To simulate the irregularly undersampled process, we extracted signals from the full-sampled transients according to various irregular relay surfaces. For training, we sampled five horizontal shutter patterns with intervals of [4, 8, 12, 16, 20] and 40 uniform rotations from 0 to 180 degrees, resulting in 200 sampling patterns (see Fig. 2(c )). For testing, we included more realistic irregular relay surfaces to evaluate our method's generalization capability in real-world scenarios.Compared methods We compare our method with three traditional direct reconstruction algorithms (LCT [11] , FK [12] , RSD [13] ), two learning-based algorithms (Unsupervised NeTF [26] , Supervised USM [28] ), and one iterative algorithm (CC-SOCR [6] ). Since LCT, FK, RSD, and USM are designed for raster scanning, we use zero-padded versions of the IUT as input, similar to our method. NeTF and CC-SOCR accept irregularly undersampled transients. For a fair comparison, we also applied our SURE-based denoiser to pre-denoise the transients for all compared methods. However, we did not apply the pre-denoising step for CC-SOCR, as the results reconstructed by CC-SOCR did not show significant improvements. This is because the strong CC-SOCR regularization inherently performs some level of denoising. We compute the peak signal-to-noise ratio (PSNR) to quantitatively evaluate the reconstruction results on the simulated dataset for all methods.Simulated data Fig. 4 shows comparison results on the "bunny". LCT, FK, NeTF, CC-SOCR, and our method can recover the main body. However, FK and CC-SOCR lose structures around the ear, while LCT and RSD exhibit aliasing artifacts due to irregular undersampling. Among the learning-based methods, NeTF produces a blurry object with diffused artifacts and loss of structures around the ear. USM struggles to adapt to irregularly undersampled transients and fails to reconstruct the hidden object. In contrast, our method successfully recovers most geometric structures of the bunny without aliasing artifacts, achieving the best quantitative results.We first tested our method on publicly available real datasets [10] [11] [12] (Fig. 5 ), and then on self-captured real-world data (Fig. 6 ). Without proper regularization, the direct reconstruction methods, i. e., LCT and RSD, exhibit severe aliasing artifacts due to undersampling. FK is depthsensitive and fails to recover far-end structures in the hidden scene. NeTF tends to produce blurry shapes due to its struggle with utilizing limited information in IUT. USM produces results that are nearly overwhelmed by aliasing artifacts in irregularly undersampled cases, but can achieve acceptable results in regularly undersampled cases (Fig. 6 ). CC-SOCR can recover main objects for all the test cases, but loses fine structures at distant depth layers ("Man Deer", "Letter", "Deer" and "Teaser") and underperforms on glossy ("Dragon") or retroreflective ("SU") objects. Our method achieves the best quality for various objects and sampling patterns. Our method generates stable results on real-world data with diverse attributes and relay surfaces despite being trained only on a simple alphanumeric dataset and shutter-like relay surfaces. It notably outperforms the range-space solver LCT, successfully removing aliasing artifacts while preserving structure. The promising generalization highlights the effectiveness of our method in learning beyond the range space. We compare the inference time of various methods on an Intel(R) Xeon(R) Platinum 8369B 2.90GHz CPU with 32 cores and an NVIDIA 3090 GPU, respectively. The inference times of NeTF and CC-SOCR vary with different IUT. Therefore, we average the inference times across 16 IUT shown in Fig. 5 and Fig. 6 each with a size of 128 × 128 × 256. As shown in Tab. 1, the direct reconstruction methods, LCT, FK, and RSD, are faster than the other methods. Unlike these one-step approaches, CC-SOCR iteratively solves the functional model by a series of alternative sub-problems, requiring nearly eight hours to reconstruct an albedo. NeTF stands on the per-scene rendering framework and thus requires significantly longer inference times than the other two learning-based methods, USM, and our method. Note that both our method and CC-SOCR achieve more accurate reconstructions than the other methods. However, our method is 12,000× faster than CC-SOCR in CPU mode.We validate the effectiveness of the two core components of our method: the virtual scanning process (VS) and the SURE-based denoiser. For quantitative evaluation, we simulated a dataset of 1,000 transients by rendering objects with complex geometries, such as chairs, clocks, guitars, sofas, and motorcycles. Our method and its two variants were tested on the simulated transients sampling with 15 different relay surfaces. For qualitative evaluation, we assess the two components using realworld datasets, "Teaser" and "Dragon". "Teaser" features complex structures, which helps evaluate VS's ability to recover details. "Dragon", with its glossy material, exhibits a lower signal-to-noise ratio in its acquired transient, which helps validate the SURE-based denoiser's effectiveness. Virtual Scanning As shown in Tab. 2, the virtual scanning process resulted in an average improvement of 0.89 dB. Fig. 7 demonstrates that without the VS component, our method can only recover a limited portion of the scene structure, which is impaired by aliasing artifacts. In contrast, our full model reconstructs more detailed and cleaner structures, confirming the effectiveness of VS component in recovering null-space components.As evident from the quantitative results, the SURE-denoiser achieved an average performance improvement of 1.83 dB, which is more significant than that of VS. This result aligns with expectations because background noise affects the entire reconstruction volume, while aliasing artifacts caused by irregular undersampling disrupt the main structure. For both LCT and our method, as illustrated in Fig. 7 (b), the denoiser effectively suppresses noise artifacts. In Sec. 5.2, we apply the denoiser to competing methods to enhance their robustness to noise for a fair comparison. This demonstrates the versatility of the denoiser as a plug-and-play module in other NLOS algorithms.Additional ablation studies on the hyperparameters within the loss functions and the relay surfaces used for training are provided in the supplementary materials.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: The data and code will be released after publication.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion and limitations: 
 Conclusion In this paper, we propose an unsupervised learning-based framework for NLOS imaging from irregularly undersampled transients (IUT). By introducing a virtual scanning process and a SURE-based denoiser, our framework achieves high-quality and fast NLOS reconstructions from IUT. Furthermore, it can be trained solely from noisy IUT, enabling future work on direct learning from real-world datasets to bridge the gap between simulated and real datasets. Our method outperforms the state-of-the-art method on both simulated and real-world data with various relay surfaces. In future work, we will extend our method to non-confocal imaging systems for more practical applications.Limitations Our method faces two main limitations. Firstly, it is constrained to the confocal system due to the lack of a high-speed, low-memory non-confocal forward operator for deep learning training. However, the framework is theoretically extendable to general forward operators, indicating future research on new forward operators in NLOS imaging may not require special consideration for irregular undersampling. Secondly, while our method is entirely unsupervised, we only present the results of our method which is trained using simulated transients due to the time-consuming nature of collecting real transient datasets. Nonetheless, this approach partially mitigates the data gap introduced by simulated ground truth, and the results demonstrate excellent performance. Transitioning to deep learning that exclusively uses real transients is one of our future goals.A Network architecture We begin by presenting the architecture of the network F ϕ . Since the vanilla convolution treats all voxels of the zero-padded IUT as valid, it leads to the distortion of information in undersampled transients during feature propagation. As illustrated in Fig. 8 , we present an encoder-decoder network (P-Unet) based on 3D partial convolution [44] . Partial convolution utilizes a sampling matrix to differentiate between valid and invalid voxels, effectively capturing spatial information from the transients. Additionally, we incorporate the instance normalization (IN) layer [45] , which is insensitive to input distribution. This layer enhances the network's generalization capability to diverse zero-padded IUT with varying distributions from different sampling rates.Given that the performance of F θ in VSRnet mainly depends on acquiring null space information from the training forward operators, we adopt a 3D residual network featuring an attention-gated network [46] . While attention mechanisms might not significantly improve reconstruction quality, they help speed up the network's training process. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Our method is implemented using PyTorch [47] , and we employ the Adam optimizer [48] with a weight decay of 10 -8 . In the first stage, the SURE-based denoiser model F ϕ is trained with a batch size of 4 for 40 epochs. We set the initial learning rate to 1 × 10 - We applied the proposed SURE-based denoiser to the compared methods and selected relatively better results for comparative experiments. As depicted in Fig. 9 , the results of LCT, FK, and RSD exhibit reduced diffuse background noise when the denoiser is applied. For learning-based algorithms, our denoiser effectively removes cluster artifacts caused by measurement noise, thereby enhancing the clarity of primary structures. However, in the case of CC-SOCR, which already incorporates strong regularization, the improvement of our denoiser is limited. This is likely due to the algorithm's sub-problem already including a denoising effect.Stein's Unbiased Risk Estimation (SURE) framework has been developed and used in prior signal denoising works [38] [39] [40] [41] [42] 49] . These works did not consider the dark current of sensor outputs that might be removed in some image signal pipelines. Therefore, we extend the SURE denoising model to NLOS imaging by incorporating more practical noise characteristics of the detector. In this section, we provide the derivation of the SURE loss mentioned in the main text, following the notations in [40, 41] .For compact presentation, we simply denote the training set of transients by {ũ j ∈ R st , j = 1, 2, . . . , I × G} instead of the two-subscript version {ũ i,g , i = 1, 2, . . . , I, g = 1, 2, . . . , G}. I is the number of observed hidden scenes, and G is the number of forward operators associated with relay surfaces. s and t are the number of scanning points and the number of time bins in each histogram of the transient, respectively.In the supervised learning, given a paired dataset of noisy transients {ũ j } and its clean version {u j }, the model F ϕ can be trained by minimizing the mean squared error (MSE) loss function:EQUATION)However, the clean transient set {u j } is not available. To achieve unsupervised learning only with noisy transients, our goal is to obtain an unbiased estimator of the mean squared error (MSE) standing on the SURE framework. To this end, we can further decompose the inner expectation in Eq. 10 asEQUATIONThe unbiased estimator of the second term in Eq. 11 is ∥F ϕ (ũ j )∥ 2 , which can be obtained without the clean transient u j . The first term in Eq. 11 can be rewritten as E ũ|u u ⊤ j ũj because E ũ|u ũj = u j . The first and third terms depend on u. Thus, we need to obtain unbiased estimators for them by considering the characteristics of the noise model .In the case of NLOS imaging, the noisy transients can be modeled as ũ ∼ Poisson(u + b), where b denotes the dark counts. The unsupervised expressions of E ũ|u u ⊤ j ũj and E ũ|u u ⊤ j F ϕ (ũ j ) can be obtained using the following lemma.Lemma 1 (Lemma 1.2 in [40] ) Let v ∈ R st such that v ∼ P oisson(u) be an independent random variables and let Φ : R st → R st be a function such that E v|u |Φ m (v)| < +∞ for all m.EQUATIONLet Φ(v) = F ϕ (ũ), then E ũ|u u ⊤ j F ϕ (ũ j ) can be expressed as:EQUATIONHere, we use the first-order Taylor approximation FEQUATION)Then, the unbiased estimator of Eq. 14 isEQUATION) Note that it is difficult to obtain an analytic form of ∂F ϕ (ũ j ) when F ϕ (•) is a neural network. We adopt the Monte-Carlo approach [49] to obtain an estimate of the divergence of F ϕ (ũ) with the following approximation:EQUATION)where ε is a small positive number, and e ∈ {-1, 1} st is a binary vector whose entities follow a Bernoulli distribution with equal probability [40] .Similarly, let Φ(v) be an identity function such that Φ(v) = v, and v = ũ. Using the first-order Taylor approximation, E ũ|u u ⊤ j ũj can be expressed asEQUATION)We obtain the unbiased estimator of Eq. 17:EQUATIONwhere 1 is a vector consisting of st ones. With the above derivation, the total unbiased estimator of the MSE is given by:EQUATION)After rearrangement, Eq. 19 can be expressed as:EQUATIONThus, we get the SURE loss function as described in the main text:EQUATION)During the training process, we use zero-padded irregularly undersampled transients as input, but only consider valid values in the calculation of the SURE loss.We used a femtosecond fiber laser as a light source with a central wavelength of 1560 nm and a pulse repetition rate of 82 MHz. The light was split by a 99 : 1 fiber coupler. The channel with 1% of the light was attenuated and sent into a fast photodetector, functioning as the start signal for the time-to-amplitude convertor (TAC). The other channel, with 99% of the light, was connected to a collimator by a piece of single mode fiber (SMF) and then travelled through a hole in the mirror. The beam was diffusively reflected by the relay surface, and the raster scanning was controlled by the beam-steering mirror. After diffusive reflections by the object and again by the relay surface, the echo light travelled back along the same optical path and was reflected to a second collimator. Finally, the echo light was detected by a fractal superconducting nanowire single-photon detector (SNSPD) coupled with SMF. The output voltage pulses from the fractal SNSPD were amplified by the RF amplifiers and input to the TAC as a stop signal for the coincidence counting. The transients were captured over a 0.8 × 0.8 m 2 scanning area, with a size of 128 × 128 × 512 and a bin width of 8 ps. In this section, we discuss how different relay surfaces affect reconstruction quality. Generally, a relay surface with a large missing area leads to result in poor reconstruction quality due to information loss. To verify this, we removed signals from the fully-sampled transient according to the relay surfaces and reconstructed them using our method. It is clear that parts of the hidden scene facing large missing areas are nearly impossible to reconstruct, as illustrated in the second, third, and fourth columns of Fig 10 (b) . However, even with the same sampling rate, a more uniform distribution of scanning points produces better results (e.g., comparing the fifth column to the second).Therefore, in the challenging task of NLOS imaging from IUT, the sampling rate is not the sole determinant of reconstruction quality. When the detector focuses a point on the relay surface, only the parts of the hidden scene that are close to that point can be observed. As shown in Fig 10 (a) , the light attenuation term 1/r 4 causes that a histogram (1 × 1 × T ) captured from scanning point p primarily contains information from hidden areas close to that point, as indicated by the green oval area. This results in poor reconstruction when large areas of the relay surface are invalid for scanning. However, determining the acceptable proportion of the invalid area relative to the entire surface for successful reconstruction is theoretically challenging, as it depends on various factors, including the relative depth between the hidden scene and the relay surface, the scene's reflectivity and normals, detector efficiency, and laser power. We consider this an important area for future work.In this section, we extend our ablation study further to explore the optimal values and sensitivity of the hyperparameters within the loss functions. Additionally, we examine the effect of relay surfaces used for training. To accommodate variations in testing data, we evaluated our method on different IUT of the "bunny", sampled according to 15 distinct relay surfaces mentioned in the main text. Effect of the positive number ε In our experiments, the transient is normalized to the range [0, 100]. Following the recommendation in [50] , where the value of ε is suggested to be set around max(u)/1000, we set ε to 0.1. As shown in Tab. 3, our SURE-based denoiser exhibits optimal performance when ε = 0.1. Effect of the hyperparameter β The hyperparameter β acts as a weight to balance the MC loss and the VS loss. As shown in Tab. 4, our method achieves optimal performance when β = 0.001. The effectiveness of our method decreases notably as β decreases. When the VS loss is completely disabled (β = 0), the model fails to learn beyond the range space, highlighting the significance of virtual scanning in our approach.It's widely recognized that the robustness of neural networks depends heavily on the diversity of the training dataset. As discussed in [51] , some neural networks struggle to adapt to changes in the sampling rate. They perform best when the sampling rate of input matches that of training dataset. Any deviation, whether a decrease or increase in the testing data's sampling rate, can lead to a decline in performance. To address this challenge, a common strategy is to augment the diversity of the training data's sampling rate. Interval In the context of NLOS imaging from irregularly undersampled transients, we can modify the shape of the relay surface to adjust the sampling rate of transients. In theory, light reflecting from the hidden scene spreads across the entire relay surface. However, due to the radiometric fall-off ∥p -q∥ 4 , only a limited portion of the reflected light, concentrated in a small area of the relay surface, can be effectively captured by the time-resolved detector. As a result, we manipulate the interval of the shutter-like surfaces to modify the local sampling rate of undersampled transients.As shown in Tab. 5, the optimal combination of interval values for relay surfaces is [4, 8, 12, 16, 20] . Our method's performance shows a decline with a reduction in the variety of interval types. This suggests that using undersampled transients with a more diverse range of sampling rates for training can enhance the generalization capability of our method. Rotation Similarly, we can modulate the diversity of relay surfaces by varying the number of rotations. As shown in Tab. 6, the optimal number of rotations is 40 when the count of transients using for training is 8,000 in total. When the number of rotations falls below 40, the diversity of relay surfaces becomes inadequate. Conversely, when it exceeds 40, the allocated transients for each relay surfaces becomes too limited.We present additional results on publicly available real data [10, 12] with more irregular relay surfaces, as shown in • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: The paper received positive reviews overall that became stronger after the authors’ rebuttal. Reviewers highlighted the extensive evaluations (wGeV, 8wFw, WCZG), the novel analysis for unsupervised methods (8wFw), and the derivation of SURE loss (8wFw, WCZG). Although there were initial concerns regarding irregularity of scanning patterns, unclear presentations, missing experiments, and motivations, these were effectively addressed in the rebuttal, leading to stronger overall reviews.

After carefully reviewing the paper, the reviews, and the rebuttal, the AC agrees with the reviewers’ positive consensus, and hence recommends acceptance of the paper.

For the camera-ready revision, the authors should ensure to incorporate all discussions from the rebuttal into the main paper. Specific changes to be implemented are:

**1. Additional experiments**: Include the quantitative evaluation results for VSRnet and SURE-denoiser (8wFw), and the experiments and discussion for the effects of different relay patterns and more irregularly varied masks (WCZG, 8wFw, wGeV)

**2. Improved presentation**: Clarify all symbols and schematics (8wFw, cypv), update citation format (cypv) and hyperlinks (WCZG), and improve figure 2 (WCZG)

Reviews: review: paper_summary: paper_summary: The authors attempt the problem of NLOS imaging in the irregularly undersampled transients data case i.e. where the scan pattern on the relay wall isn’t dense or regular. To tackle the problem the authors introduce two main components (both trained unsupervised):

1) A SURE-based denoiser, which denoises the input transients 

2) A VSRNet, a “virtual scanning” module, where the estimated albedo is then transformed back into an undersampled transient, and the albedo is estimated again from this sample. VSRNet is trained with an MC loss, enforcing consistency between the two albedo estimates. 

The authors justify their choices by arguing about the null space of the light-transport matrix. 
The method is benchmarked against a suite of datasets, one of which is also self-collected.
strength_weakness: strength_weakness: Originality/Significance: I’m not sure the authors do a great job motivating the problem of irregularly undersampled transients itself. There is a slight motivation in the introduction of scanning through fences etc, but I’m not sure the problem is convincing on its own. I think this also bleeds into a discussion about significance, I’m not sure if the contributions of the paper will be valued enough without a stronger motivation in the introduction/abstract for the problem itself.


Clarity: I appreciate the authors trying to introduce a more theoretically grounded framework, but I’m not sure how important the discussion of the null space actually is. I might have misunderstood some parts, but if I understand correctly the problem is just fundamentally ill-defined, and we could have done without the discussion between lines 100-114.

I think it’s usually good practice to have the method understandable through the figure itself, which is why I would suggest that the authors have a better description for Figure 2. I think it would be good if the authors described the separate components shortly there itself. 


Typos:

L61: Citation for Manna et al. not hyperlinked.
questions: questions: 1) Did the authors try to see the effects of different patterns on the reconstruction (not just in training as in the supplement)? I can see a bit in Fig 4 and 6, but are there any limits we can expect i.e. how many scan points do we need, are some patterns worse than others etc?
limitations: limitations: Limitations of the method are not really discussed in the paper, but there is an adequate section in the supplement. Societal impact N/A.
paper_summary: paper_summary: This paper proposes a non-line-of-sight (NLOS) imaging method for scenarios where transients are irregularly undersampled on the relay surface. The proposed method includes a SURE-based denoising technique to handle noisy transient data, specifically addressing Poisson noise. Additionally, a novel unsupervised learning network called VSRnet is introduced, enabling consistent reconstruction from different irregularly undersampled transients (IUT) of the same 3D albedo volume through a process termed virtual scanning. Extensive comparisons with recent NLOS methods using both simulated and real IUT data demonstrate the superiority of the proposed approach.
strength_weakness: strength_weakness: 1. While the SURE-based denoiser may contribute to handling noisy transient data, its originality is limited, and its connection to IUT is unclear.
1. The assumptions regarding the relay surface are not practical. In a setting like Figure 1, the BRDF of the relay surface would vary among the transient samples. However, in the experimental setting, "we extracted signals from the complete transients according to various irregular relay surfaces" (207), which overly simplifies conditions.
1. The justification for zero padding is not clear and it might be unfair for other methods. For instance, instead of treating regions with no samples as zero, using simple interpolation methods like bilinear interpolation might improve the quality of the competing methods.
1. The validation of the proposed virtual scanning is insufficient. While it is claimed that virtual scanning complements the null space (159), this explanation is intuitive rather than theoretically substantiated. There would be no guarantee that "a set of operators $\mathcal{H}$" (170) will fully complement the null space. For example, while it is mentioned that the sampling pattern for virtual scanning is random (138), there is no discussion or experimental validation on the necessary sampling ratio. Although the proposed method conducts the virtual scanning only once for $\rho^{(2)}$, it can be conducted iteratively. However, the iteration might also lead to deviations from observations. This aspect is not adequately discussed.
- Minor comments:
  - The meanings of notations such as $\hat{}$, $\tilde{}$, and various subscripts are not clear enough.
  - Variables like $l$ (86), $G$ (131), and $k\neq g$ (149) are not explained.
  - The process of obtaining $\rho^{(1)}$ and $\rho^{(2)}$ (154) is described in text but not well defined in equations, making it difficult to understand correctly.
  - Referring to the 3D albedo volume $\rho$ as the latent 3D volume (133) seems inappropriate.
  - The citation format might be wrong.
questions: questions: If there are any misunderstandings in the weaknesses pointed out, please clarify them.
limitations: limitations: The limitations are mentioned only in the supplementary material and not referenced in the main text.
paper_summary: paper_summary: To address the challenges of slow inference speed and poor generalization to irregularly relay surfaces 
in non-line-of-sight (NLOS) imaging, this paper proposes a learnable unsupervised training framework
with the excellent novelty, as well as a Virtual Scanning Reconstruction Network (VSRnet).
Furthermore, to mitigate the noise present in transient data, the authors introduce a denoiser based on
Stein's Unbiased Risk Estimate (SURE) loss. Extensive comparative experiments and ablation studies
demonstrate that the proposed method significantly enhances reconstruction quality compared to
existing approaches.
strength_weakness: strength_weakness: 1. The lack of explanations for some key symbols (e.g., the explanation for M_g in Figure 2(a)
and the construction method of M_k in Fig. 2(d)) and schematics (e.g., Fig. 3(c) is difficult for
readers to understand from the brief descriptions in the paper) might hinder readers from
reproducing the proposed algorithm. Providing further explanations could significantly enhance
the readability of the paper.
2. There is a lack of quantitative evaluation results for VSRnet and SSIM-denoiser. Section 5.4 of
the paper only provides qualitative comparisons of the two modules, neglecting quantitative
experimental assessments. For readers, quantitative results are more effective than qualitative
results in intuitively understanding the model's effectiveness.
3. The title of the paper includes the keywords "unsupervised" and "irregularly undersampled transients," which seem to represent two independent issues. Specifically, is the choice of an unsupervised framework a necessary outcome for addressing the reconstruction problem of irregularly undersampled transients? If the authors could clarify this point, I believe it would constructively enhance the paper.
questions: questions: 1. I am confused about the content in M_k and M_g. Could you please provide further definitions or
explanations for M_k and M_g?
2. The masks used during training exhibit regular patterns (e.g., evenly spaced stripes). Would
introducing more irregularly varied masks further enhance the robustness of the network?
3. It is unclear how the authors constructed different forward propagation operators H based on the
varying irregularly relay surfaces?
limitations: limitations: Discussed.
paper_summary: paper_summary: The paper introduces a unique unsupervised learning framework for non-line-of-sight imaging from irregularly undersampled transients. The proposed method overcomes the dependency on paired data and achieves higher fidelity, greater robustness, and remarkably faster inference times.
strength_weakness: strength_weakness: 1. The 'irregular windows' in the test scenarios composed of regular, repetitive, or symmetrical simple geometric shapes, which are more close to the relay surfaces simulated during training. Whether this method can be extended to more complex real-world scenarios remains a concern.
questions: questions: 1. Have you considered other irregular relay surfaces that is not composed of simple repetitive or symmetrical shapes?
2. What is the motivation of using different shutter pattens as the simulated relay surfaces? Have you tried other patterns?

I will reconsider my rating if your answers address my confusion.
limitations: limitations: The authors presented the limitations and proposed some future direction to addressed them.

Comments: title: Appreciation for the Increased Rating and Continued Feedback
Official Comment by Authors
Official Comment by Authors
Reply for the rebuttal
Official Comment by Authors
Response Acknowledgement and Appreciation
My concern is resolved
comment: Thank you very much for your valuable feedback throughout the author-reviewer discussion, and especially for raising your rating. We greatly appreciate your time and effort in helping us improve our work.

In our previous responses, we added extensive quantitative and qualitative experiments (e.g., Figures 2 and 4 in the global rebuttal and the table in the comment) to address your concerns, as well as many detailed explanations of our contributions.

To ensure that we have fully addressed all of your concerns, could you please let us know if there are any remaining aspects of our response that require further clarification or improvement?
If you have any additional feedback or suggestions regarding the revisions we've made, we would be very grateful to receive them and make any necessary adjustments in the camera-ready version. Our goal is to address all of your concerns as thoroughly as possible and to earn your further approval, just as we have with the other reviewers. 

Thank you once again for your review and feedback. We look forward to your further input.
Thank you very much for your continued engagement and constructive feedback.
We can feel your strong sense of responsibility and rigorous academic standards from your comments.

We're pleased that you found our general rebuttal and the insights from Figure 1 valuable. Per your suggestion, we will include a detailed discussion of these results in the camera-ready version of the paper, highlighting the impact of irregular sampling.

Regarding your concerns about the motivation of our work, we understand that our initial explanation may not have fully addressed your questions.
Looking back, when NLOS imaging was first introduced, it faced far more limitations than now, including restricted scene conditions, limited relay surfaces, low resolution and reconstruction quality, and slow acquisition and processing times. Over the past decade, improvements (incremental some times) and breakthroughs have been built on the collective efforts of the NLOS research community addressing each subproblem/aspect.
However, it is important to note that NLOS imaging is still primarily a laboratory-scale problem, with many challenges yet to be addressed for practical deployment.
Our work focusing on NLOS imaging from IUT is significant, as it liberates NLOS research from the reliance on large and continuous relay surfaces (such as walls and floors).

As recognized by reviewers, our work stands out for its well-conducted experiments, thorough ablations, novel analysis, and superior performance compared to existing methods. 
Meanwhile, the main technical contributions have been recognized by the reviewers.
Notably, the proposed unsupervised paradigm has been praised for potentially enhancing the generalization capabilities of existing deep learning methods, as noted by Reviewer 8wFw.
We will open-source our code to enable other NLOS researchers to build on our work and advance practical NLOS applications.

We are committed to improving the paper in our camera-ready version based on your and other reviewers' feedback, and we welcome any additional suggestions you may have. 
Thanks once again for your valuable input.
Thank you very much for your detailed feedback and for recognizing the value of the additional experiments we provided in the rebuttal. We are glad that our response addressed some of your concerns and we appreciate the opportunity to address your remaining concerns in more details.

**Q1: Connection and novelty of the SURE-based denoiser.**

We are glad to know that the informative results in Figure 2 of the global rebuttal clarify the effectiveness of the proposed SURE-based denoiser.
It is important to emphasize that the extensive real-world experimental results demonstrate that our method consistently achieves superior reconstruction quality with different sampling rates compared to other algorithms, strongly validating the generalization of our approach to real scenes. The sampling rates for the test cases in Fig. 3 and 4 in the global rebuttal, and Fig. 4$\sim$7 in the manuscript are within the range of 6.25\% $\sim$ 50\%. 
We commit to following your suggestion by providing the corresponding sampling rates for the qualitative results in the camera-ready version of the paper. 


**Q2: The assumption of relay surfaces.**

We understand your concern regarding the practical relevance of this assumption and appreciate the opportunity to address it.
We agree that exploring NLOS imaging with non-uniform BRDFs is a valuable problem, but it is a parallel challenging problem to our current work, and deserves dedicated efforts from the entire NLOS community. 

Looking back, when NLOS imaging was first introduced, it faced far more limitations than now, including restricted scene conditions, limited relay surfaces, low resolution and reconstruction quality, and slow acquisition and processing times. Over the past decade, improvements (incremental some times) and breakthroughs have been built on the collective efforts of the NLOS research community addressing each subproblem/aspect.
However, it is important to note that NLOS imaging is still primarily a laboratory-scale problem, with many challenges yet to be addressed for practical deployment.
Our work focusing on NLOS imaging from IUT is significant, as it liberates NLOS research from the reliance on large and continuous relay surfaces (such as walls and floors).

As recognized by reviewers, our work stands out for its well-conducted experiments, thorough ablations, novel analysis, and superior performance compared to existing methods. 
Meanwhile, the main technical contributions have been recognized by the reviewers.
Notably, the proposed unsupervised paradigm has been praised for potentially enhancing the generalization capabilities of existing deep learning methods, as noted by Reviewer 8wFw.
In addition, our response has been acknowledged by the reviewers and addressed most concerns from them, including the applicability to more complex irregular relay patterns (Reviewer wGeV, you also mentioned).

We are confident in our paper and are prepared to open-source our code to enable other NLOS researchers to build on our work and advance practical NLOS applications.

**Q3: The padding (interpolation) mode for unobserved points.**

**Q4: Null-space learning of the virtual scanning.**

Following your suggestions on providing quantitative results for variants of competing methods combined with the SURE denoiser and other two interpolation methods. For  testing, we simulated a test dataset of 1,000 transients by rendering objects (chairs, clocks, guitars, sofas, motorcycles) with random scales and positions. The quantitative results are as follows:
| Method          | LCT (neighbor) | LCT (bilinear) | FK (neighbor) | FK (bilinear) |
|-----------------|----------------|----------------|---------------|---------------|
| PSNR (dB)   | 17.63          | 18.19          | 15.69         | 15.95         |

| Method          | RSD (neighbor) | RSD (bilinear) | USM (neighbor) | USM (bilinear) | Ours  |
|-----------------|----------------|----------------|----------------|----------------|-------|
| PSNR (dB)   | 15.58          | 15.81          | 14.32          | 14.74          | 20.52 |

As evident from the quantitative results in the table, our method still offers a significant quantitative improvement over other methods, which confirmed the importance and contribution of virtual scanning.

Many thanks once again for your valuable feedback.
Thank you very much to the authors for their responses. 


I'm not sure my comments on the motivation for the paper have been adequately addressed. But I'm not sure this can be addressed. 
Nonetheless, I appreciate the general rebuttal, especially Figure 1, I think it's quite interesting to see the effects of the reconstructions with the patterns the authors show, and I would suggest the authors add this discussion to the paper. 


I'm not yet sure I will increase my score, I will be following the author's discussions with the other reviewers to set my final score, thanks!
The additional experiments presented in the global rebuttal are informative and address some of my concerns.

### Q1: Connection and novelty of the SURE-based denoiser.

I appreciate the additional experiment shown in Fig. 2 of the global rebuttal. It clearly demonstrates the effectiveness of the SURE-based denoiser. It shows that a 1.1% sampling rate in the red plot (with the SURE-denoiser) is approximately equivalent to the 100% sampling rate in the blue plot (without the SURE-denoiser) . While using the Poisson distribution for modeling the SPAD sensor is quite common, and I initially considered this work a straightforward extension of existing methods, the effectiveness demonstrated in Fig. 2 raises my rating on it, even though it may seem like an incremental contribution.

However, my remaining concern is that the sampling rates are unclear in the reconstruction results (in Figs. 4-7 of the original submission and in Figs. 3 and 4 of the global rebuttal). The visual quality could vary significantly at sampling rates between 1% and 4%, which the authors emphasize. If the authors also show the sampling rates used for each of the results, Fig.2 would be more informative.

### Q2: The assumption of relay surfaces.

As other reviewers also raised similar concerns under "Originality/Significance" (WCZG) and as "a concern" (wGeV), the focus on irregular sampling on the relay surface seems far from real-world situations. The focus should be more on non-uniform diffuse reflectance.

### Q3: The padding (interpolation) mode for unobserved points.
### Q4: Null-space learning of the virtual scanning.

I appreciate the authors providing the additional experiment shown in Fig. 4 of the rebuttal. However, I would like to see quantitative results for other interpolation methods as well.

My main concern with "zero padding" is that it might misinform the existing network that "the space is empty" rather than "the measurement is missing." This seems unfair to the existing methods. Based on the effectiveness of the SURE-denoiser shown in Fig. 2 of the global rebuttal, I still have doubts about the contribution of "virtual scanning," which the authors claim is the main contribution of this paper.

Did the authors quantitatively clarify whether the proposed virtual scanning still offers a significant advantage over the combination of "SURE denoiser" and "other interpolation methods" on "LCT, FK, RSD, USM"?
Thank you for your positive feedback and for raising your rating. We are pleased that our response effectively addressed your concerns and that you recognize the value of our method in applying unsupervised learning to NLOS imaging.

We will carefully update Q1, Q2, and Q4 in the revised manuscript as per your suggestion. We appreciate your thoughtful review and the time you've taken to help improve our work.
Thanks to the authors for the detailed response, which effectively addressed my concerns. I'll raise my rating accordingly.

In light of the new paradigm for NLOS—specifically, unsupervised learning for NLOS reconstruction—I believe this approach could significantly enhance the generalization capabilities of existing deep learning methods. The application of this approach to reconstruction from irregular transients strongly supports this view.

Please update Q1, Q2, and Q4 in the revised manuscript accordingly.
Thank you for your positive feedback and for raising your rating. We appreciate your thoughtful review and are glad that our response addressed your concern.
Thanks for your reply. I would like to raise my rating.