Virtual Scanning: Unsupervised Non-line-of-sight Imaging from Irregularly Undersampled Transients
Authors: Xingyu Cui
Huanjing Yue
Song Li
Xiangjun Yin
Yusen Hou
Yun Meng
Kai Zou
Xiaolong Hu
Jingyu Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Non-line-of-sight imaging
Machine Vision
Computational Imaging
Primary area: machine_vision
TL;DR: We propose an unsupervised learning-based framework for NLOS imaging from irregularly undersampled transients for for high-quality and fast inference.

Abstract: Non-line-of-sight (NLOS) imaging allows for seeing hidden scenes around corners through active sensing.
Most previous algorithms for NLOS reconstruction require dense transients acquired through regular scans over a large relay surface, which limits their applicability in realistic scenarios with irregular relay surfaces.
In this paper, we propose an unsupervised learning-based framework for NLOS imaging from irregularly undersampled transients~(IUT).
Our method learns implicit priors from noisy irregularly undersampled transients without requiring paired data, which is difficult and expensive to acquire and align. 
To overcome the ambiguity of the measurement consistency constraint in inferring the albedo volume, we design a virtual scanning process that enables the network to learn within both range and null spaces for high-quality reconstruction.
We devise a physics-guided SURE-based denoiser to enhance robustness to ubiquitous noise in low-photon imaging conditions. 
Extensive experiments on both simulated and real-world data validate the performance and generalization of our method.
Compared with the state-of-the-art (SOTA) method, our method achieves higher fidelity, greater robustness, and remarkably faster inference times by orders of magnitude.
The code and model are available at https://github.com/XingyuCuii/Virtual-Scanning-NLOS.

Virtual Scanning: Unsupervised Non-line-of-sight
Imaging from Irregularly Undersampled Transients
Xingyu Cui Huanjing Yue' Song Li Xiangjun Yin'
Yusen Hou Yun Meng.. Kai Zou Xiaolong Hu Jingyu Yang!.
School of Electrical and Information Engineering, Tianjin University, China
SScooo of Precision Instrument and Optoelectronic Engineering, Tianjin University, China 3 Key Lab. of Optoelectronic Information Science and Technology, Ministry of Education, China
Abstract
Non-line-of-sight (NLOS) imaging allows for seeing hidden scenes around corners through active sensing. Most previous algorithms for NL.OS reconstruction require dense transients acquired through regular scans limits their applicability in realistic scenarios with irregular relay surfaces. In this paper, we propose an unsupervised learning-baa based framework for NLOS imaging from irregularly undersampled transients (IUT). Our method learns implicit priors from noisy irregularly undersampled transients without requiring paired data, which is difficult and expensive to acquire and align. To overcome the ambiguity of the measurement consistency constraint in inferring the albedo volume, we design a virtual scanning process that enables the network to learn within both range space and null space for high-quality reconstruction. We devise physics-guided SURE- based denoiser to enhance robustness to ubiquitous noise in low-photon imaging conditions. Extensive experiments on both simulated and real-world data validate the performance and generalization of our method. Compared with the state-of-the the- art (SOTA) method, our method achieves higher fidelity, greater robustness, and remarkably faster inference times by orders of magnitude. The code and model are available at https://gthub..om/Xigggguu Virtiiia-aaaaanniiinn Scanning-NLOSS
large relay surface, which
Introduction
Non-line-of-sight (NLOS) imaging aims to reconstruct hidden scenes beyond the direct line of sight of the detector, garnering interest across various fields such robot vision, autonomous driving, rescue operations, remote sensing, and medical imaging [1:5]. In a typical active confocal NLOS imaging system, depicted in Fig. [ laser source and detector are both focused on the same point on a relay surface. Pulses emitted by the laser reflect off the surface illuminate the hidden scene. The detector captures photons bouncing back from the scene toward the relay surface, referred to as transients, from which the hidden scene can be recovered using elaborately designed algorithms.
While existing works have achieved remarkable breakthroughs, they also face significant limitations that hinder their practical applicability. These methods assume dense and regular scanning of large relay surface, which may not be feasible in realistic scenarios with irregular relay surfaces such as latticed windows or fences. Transients obtained through irregular undersampling can result in severe ill- ill-posedness, leading to artifacts reconstruction failure. This raises the challenging task of NLOS imaging from irregularly undersampled transients (IUT). To address this, Liu et al. [6] proposed introducing manually designed strong regularization terms under a functional optimization framework. However, this approach is hindered by the need for lengthy numerical iterative computations. Recent
"Corresponding author: yjy @ @tw...eeu.u.n....
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Laser & Detector
Relay Surface
Hidden Scene
Zero Padded Undersampled Transient
Scannable P Points
Figure 1: Illustration of the active confocal NLOS imaging with an irregular relay surface. Yellow points indicate scannable points, while blue points indicate non- scannable points.
enable quick inference, they require supervision from large amount of paired data, including well-aligned ground-truth albedos, which are difficult and expensive to acquire. Therefore, exploring NLOS imaging under unsupervised learning is worthwhile to eliminate the heavy reliance paired data.
learning-based NLOS imaging methods 7-10
In this paper, we propose novel unsupervised framework capable of learning implicit priors from noisy IUT. This framework comprises two components: a virtual scanning reconstruction network to-aleddo
(VSRnet) that learns high-quality measurement to albedo mapping beyond the range space induced by the measurement consistency, and SURE-based denoiser that enhances our method's robustness measurement noise by incorporating the physies model of the low-photon time-resolved detector. We conduct extensive experiments on simulated data, publicly available real data, and data acquired from our self-built NLOS system. Our method outperforms existing algorithms, particularly in providing robustness for real data and diverse irregular relay surfaces.
Our main contributions are summarized follows:
We propose an unsupervised NLOS imaging framework capable of learning implicit priors from noisy IUT, effectively overcoming the dependency paired data that is difficult to acquire and align.
We introduce a virtual scanning process that enables the network to learn within both range and null spaces for high-quality reconstruction from IUT, extending NL.OS imaging to realistic scenarios with irregular relay surfaces.
We propose SURE-based denoiser, an unsupervised physics-guided module that incorporates the low-photon time-resolved detector's physies model, to enhance our method's robustness to noise.
We evaluate our method on simulated, publicly available, and self-captured real data, demonstrating its superior reconstruction quality and significantly faster inference compared to the SOTA method.
2
Related work
2.1 Model-based NLOS reconstruction
Model-based NLOS algorithms have achieved significant advances in recent years. Direct recon- struction algorithms 11,15) offer rapid implementations for NLOS imaging, while iterative algo- rithms 186.18 5 leverage more accurate physical models for higher reconstruction quality. Recent efforts aim to extend NLOS imaging to challenging real-world scenarios. For instance, Manna et al. [19] and Gu al. [20] addressed NLOS imaging with dynamic and non-planar relay surfaces, respectively. However, these methods still rely on large relay surfaces and dense measurements. Another set of algorithms 25 aims achieve high spatial resolution reconstruction using sparse sampling measurements to significantly reduce acquisition time. Yet. they are constrained to specific scanning patterns, such as regular or Hadamard patterns. To address NLOS imaging from irregularly undersampled transients, Liu et al. 6 introduced reconstruction model using Confocal Comple- mented Signal-Object Collaborative Regularization (CC-SOCR). Despite its effectiveness, CC-SOCR suffers from long inference times due to its iterative nature.

---

2.2 Learning-based NLOS reconstruction
Recently, deep learning has gained attention for NLOS imaging due to its learning capabilities and fast inference. Chopite et al. [7] first employed deep learning for NLOS reconstruction, but their method fell short compared to model-based approaches due to the lack of physical guidance. Physics-guid guided methods 88 26, 9 [0] have since emerged to enhance reconstruction quality, particularly for real- world data. Recent works focus on NLOS imaging from regularly undersampled transients (27,28) but they require large datasets of paired data and fail to reconstruct from IUT. Furthermore, these supervised algorithms still have significant room for improvement in robustness on real data due to the gap between simulated datasets used for training and real-world data.
Problem formulation and motivation
3.1 NLOS Imaging from IUT
Fig. iillustrates confocal NLOS imaging with time-resolved systems. By scanning a set of points P
(Ii, Vi, 0) 1,2,...,& 8, Ie, Yı E }R} on the relay surface, the forward model NLOS
imaging can be modeled as
(p. 1 g -OPP--1 -(21pp
x(4)
q11 lc)dq
(1)
where is the spatial-tempores measurement, p (ar, y, 0) denotes the scanning point, n((g) denotes the albedo value at point q in the 3D hidden scene Q. is the speed of light. The distance ||p 4l is related to the time of flight through the Dirae delta function 5(-). For compact presentation and analysis, we employed discretized version of the above forward model. Let u E R and p R!= represent the vectorized measurements and albedos of the hidden object, respectively. where denotes the size of the vertical and horizontal dimensions, and denotes the depth dimension. We denote the forward operator, also known as the light transport matrix, by H E RRtxx"= The forward processing can be described by the following linear model:
Hp.
Notably, the forward operator H is related not only the optical-electronic characteristics of the NLOS system but also to the scannable region on the relay surface.
3.2 Motivation
Deep learning-basee based algorithms have demonstrated significant potential to enhance NLOS imaging performance compared model-based approaches. Most prior work has adopted supervised paradigm, which requires substantial amounts of high-quality paired data. However, it is prohibitively expensive infeasible to acquire ground-truth 3D albedo volumes precisely aligned with the spatio-temporal measurements. This limitation motivates us to develop an unsupervised NLOS reconstruction framework that avoids this dependency and further enhances the generalization of learning-based methods to real-world data.
The standard approach in unsupervised learning is to train the reconstruction mapping fo by minimiz- ing the measurement consistency (MC) loss:
E,, [H(( ull.
(3)
Nevertheless, solely enforcing the MC loss without ground truth supervision does not guarantee high- quality reconstruction. This can be analyzed from the perspective of the range-null decomposition 29] Let Wet = {u E | He = 0} be the null space of the operator H. Its complementary space the range space of H T denoted by RB = {H T u, u E RR*), such that !²= R @ NH.
Any albedo volume p can be decomposed into range- component and null-space component.
H'HP rango-spacc component
+ I (I-HHH
4)
null-space component
3

---

(a)
Bg
Mg
Mg
(b)
Hadamard Product T Element-wise Addition
MC Loss
Pad
LCT
Hy 1
Bgr
Bgr
pt)
SURE Loss
VS Loss
Mx
LCT
F
p()
Figure The pipeline of our unsupervised framework: (a) The SURE-based denoiser, which consists of an encoder-decoder network designed for IUT and is trained by minimizing SURE loss in the first stage; (b) The virtual scanning reconstruction network (VSRnet), which consists of Unet-like network and is trained by MC loss and VS loss in the second stage; (c) Relay surfaces used for training (yellow indicates scanning areas, and their corresponding 3D binary masks are used in implementation)) (d) The virtual scanning process, which involves virtually observing p( with relay surface Mj that is distinet from Mg and enforcing consistency between p and p!
where H E xx==xxt is the pseudo-inverse of H satisfying HH = H. The operator H H projects the sample p into the range space Ry: Dp)p) H whereas its complementary operator H-HHH) projects p into the null-space Nn: (n) (T- Ht H)p. As long the trained network fo reconstructs from input u as fo(u) = (p) + on for On E NH,, the reconstructed volume fo(u) would fully meet the MC requirement: (p)
HH H HitHpp- HUn = u since we have H Hp = and Нu, = This suggests that the MC constraint only locates the albedo volume in broad subspace surrounding the range-space projection D, and the inference of the component N1 ad-hoe without guidance.
This necessitates an unsupervised framework capable of learning beyond the range space. We note that model-based algorithms [18
25 23 manually design regularization terms learn beyond range space, but suffer from long inference times. In other computational imaging tasks, supervised methods (30
20 32 and unsupervised methods 33 have been proposed recover null- space components of the reconstructions. Along this avenue, we propose an effective unsupervised framework capable of learning in both range-null spaces for NLOS imaging from IUT.
31
Method
4.1
Unsupervised framework
shows the proposed unsupervised framework via virtual scanning for NLOS imaging from Fig.
IUT. The framework consists of two components: 1) virtual scanning reconstruction network (VSRnet) to recover the 3D albedo volume from both range and null space, and 2) BURE-based denoiser to enhance the robustness to ubiquitous noise in transients. Given set G noisy irregularly undersampled transients U = {ū, E = g 1, .,... G} and the set of their corresponding forward operators H -H, Ratx!==| 1,..., G), our goal to train deep neural mapping without labeled supervision to reconstruct the 3D albedo volume p from the noisy IUT ū. As discussed in

---

In
Xir.
(0)
No..
(b)
No.
(0)
No. No.
(o))
pcv
pai
p
Pa(o)
p
Fo (D)(om)) )
R12
Pr(p
RHs
Ryt.
Rih
D+(0)
Dp(p)
Range-Null Space Decomposition
Without
With Virtual Scanning
Virtual Scanning
Figure 3: Toy visualization of NLOS reconstruction from the perspective of range-null space de- composition (RNSD). (a) illustrates the RNSD of observed by H1. (b) illustrates that p cannot be accurately recovered with only the measurement consistency loss. (c) shows that the proposed virtual scanning promotes the acquisition of null-space components.
Sec. 3.1 due to the one-to-one correspondence between the forward operator and the relay surface, we will loosely use Hg to refer different relay surfaces without ambiguity in the following sections.
We first briefly sketch the workflow of the inference stage, shown in Fig. a and bb). The noisy IUT Ng. observed through an irregular relay surface Hg. is first zero-padded ügr on full scanning grid. This is then passed through the SURE-based denoiser Fg to remove noise. Subsequently, the albedo volume p is reconstructed using VSRnet fe. which incorporates the physical prior of NLOS imaging (LCT (III) and learned reconstruction mapping Fg. During the training stage, the denoiser regularized by the SURE loss, while the reconstruction network is regularized by the MC loss. Additionally, we introduce the virtual scanning process (Fig. d)) to capture the measurement details within the null space of observation operators. In this process, the reconstructed p(4) is projected into the measurement space as Uky by virtually scanning with a different relay surface H3 H. which distinct from g- The virtual scanned measurement Mär is then projected back into the reconstruction space 2d2 by the reconstruction network. We impose virtual scanning (VS) loss between the two reconstructed volumes, ((3) and p(2), to promote null-space learning (Sec. (4.)). The modules of our framework and the loss functions utilized for training are detailed in the following subsections. The
structures of Fg and Fg are detailed in the Supplementary Material (SM).
4.2 Virtual Scanning
Strategy
The training strategy of VSRnet is depicted in Fig, 2b)) and d).). Specifically, the reconstruction module (Fig. b)),, the denoised sample ügr is initially transformed to the albedo domain using the inverse operator of LCT HJ. The resulting H. is then mapped to the albedo volume ((3) by the reconstruction network Fe with a global residual connection. At the virtual scanning module, the reconstructed albedo volume p(2) is projected into a virtual undersampled measurement okr using another forward operator Hg H (H& * Hg)- To achieve efficient implementation of forward operators, we decouple H₃. into M₁ @ H,, which can be efficiently computed using Hadamard product and the fast Fourier transform. 3D binary mask associated with the relay surface, constructed by repeating the 2D sampling pattern t times along the time dimension. H, represents the forward operator of LCT. Following the same reconstruction pipeline, albedo volume, denoted by ~ , obtained from the virtual measurement Ukr. The processes of obtaining (1)) and 2( can be formalized as:
practice, M2 E gixix= the
(1) = fo (Pad(Paa()))) (gg, My) @ My), (2) = So 3 M&).
(5)
The two reconstructed albedos, (11) and 2(2), should be identical the learned mapping Fo enables perfect reconstruction. This motivates us to impose a proximity constraint between p(1) and p in the albedo domain, named the virtual scanning loss. The virtual scanning process in the training pipeline facilitates the learning of the null-space component, thereby providing a promising prior that complements the range space, analyzed below.
Analysis Fig. provides more intuitive understanding of how the proposed virtual scanning facilitates the network in learning the null-space component. Let Hj and H2 be two observation
5

---

operators associated with two different relay surfaces. We observe using the forward operator H1. obtaining the measurement u As illustrated in Fig. Ba), p is projected into the range-space R1, of H1. When we only impose the MC constraint, the resulting output p()) will belong to the following set Xn,
(6) As depicted by the blue dashed line in Fig. Bb), there exist multiple outputs that satisfy the measure- ment consistency. If, for instance, we obtain inaccurate estimated results denoted as p(t) through p( = (p) + (p)) we then utilize H2 to virtually scan p(1) and project it into the range space RH,- Subject to the constraint p()) =p(²) Fo (D() converges to Dr (p(() due to the relationship - () + () Following this iteration. the network can learn within Ræ and N The entire process illustrated in Fig. B(c). Similarly, by altering the order of operators HI and H2, Fo will also observe within RH, and NHH, H In practice, set of operators H. is provided to enhance the network's generalization across various operators with different relay surfaces.
Xn, = {u H - D.))) E Rr, }.
4.3 SURE-based denoiser
NLOS imaging inherently operates under photon-limited conditions, and noise in transient measure- ments can lead severe background artifacts in the albedo space. This hinders the network's 's ability to learn implicit priors from noisy IUT, especially in unsupervised learning. Inspired by previous studies (38 introduce Etyysissgguide unsupervised denoiser suppress measurement noise. Specifically, we leverage Stein's 's Unbiased Risk Estimator (SURE) 43 to derive an unsupervised learning loss function that considers the physical model of the time-resolved detector in NLOS imaging systems, which can be modeled as
Poisson (u b),
U Hp.
where b represents the dark counts of the detector along with the background photons. The parame- terized denoiser Fg learns to map the noisy measurement to its clean version u via minimization of the SURE loss function given in Eq. (8).
4.4 Loss function
collected by observing I hidden scenes, each with G relay surfaces, the training of our framework involves three loss functions: the SURE loss LSURE, the MC loss LMC, and the VS loss Lvs. The SURE loss an unbiased estimation of the mean squared error (MSE) under the Poisson noise model taking dark count consideration:
Given a measurement set {ūr.g.. = 2,, 1.2.....
1 _ + []+ ūi.g
LSURE =(iig) st
-lūt.. Fo(ūrgg)
(8)
+ 2 b Fo(ue))) + stc (eig @ "r.g) (Folūr. + ces.g) - Fo(ūrgg))
st
where is a positive number, 6i.g E 1,1.1 1 is a binary vector, whose elements follow a Bernoulli distribution with equal probability 40). and @ denotes element-wise multiplication. Detailed derivation of the SURE loss is given in supp. material.
We adopted the mean squared error for Lvc and Lvs:
LMc B(igg) Hy (fo (fo
Lvs
(fo(
We first trained the denoiser Fs using the SURE loss. Next, we froze the SURE-based denoiser and trained the network Fo with combined loss function, L()) = (M) BLvs(O), where B is trade-off parameter (see SM for training details).
5
Experiment
5.1
Experiment setup
Dataset We generated 8,000 transients using the transient rasterizer from g with default parameters. The transients have spatial-temporal resolution of 128 x 128 x 512 with a bin width of 33 ps. The

---

Relay Surface
USM
CC-SOCR
Ground
Verical Shukes
15.57
15,72
17/8
11.77
1647
2MB
Houmg
18:75
Wirdow
Burry
1,00
Figure 4: Reconstruction results the "bunny" with different relay surfaces. The intensity images normalized to the range of O to I through maximum value normalization. PSNR (dB) for each intensity image is displayed in the top right corner.
dataset contains 111 objects from the alphanumerics dataset [9. covering lowercase and uppercase letters from the English and Greek alphabets, and numerals from We also rendered a sample "bunny" for quantitative comparison. To assess our method's generalization, we tested it on real-world data acquired by three different systems [0 and self-built system (see SM for system details). The hidden scenes feature various reflective materials, depth ranges, and geometric shapes, and were acquired under different conditions including scanned areas, spatial resolutions, bin widths, and integration times. For fair comparison with CC-SOCR [6] within manageable time frame, resized the full-sampled transients [T0 [2] to 128 x 128 x 256 and the full full-sampled transients [I] to 64 x 64 x 256. We used these resized transients for all methods.
Relay surface To simulate the irregularly undersampled process, we extracted signals from the full-sampled transients according to various irregular relay surfaces. For training, sampled five horizontal shutter patterns with intervals of [4, 8, 12, 16, 20] and 40 uniform rotations from to 180 degrees, resulting in 200 sampling patterns (see Fig. [c)) For testing, included more realistic irregular relay surfaces to evaluate our method's generalization capability in real-world scenarios.
Compared methods We compare method with three traditional direct reconstruction algorithms (LCT 11, FK [12), RSD [3),, two learning-based algorithms (Unsupervised NeTF (26). Supervised USM 28 ), and one iterative algorithm (CC-SOCR (6). Since LCT, FK, RSD,, and USM designed for raster scanning, we use sero-padded versions of the IUT as input, similar to our method. NeTF and CC-SOCR accept irregularly undersampled transients. For a fair comparison, we also applied our SURE-based denoiser to pre-denoise the transients for all compared methods. However, we did not apply the pre-denoising step for CC-SOCR, as the results reconstructed by CC-SOCR did not show significant improvements. This is because the strong CC-SOCR regularization inherently performs some level of denoising. We compute the peak signal-to-mg -o-ises ratio (PSNR) to quantitatively evaluate the reconstruction results on the simulated dataset for all methods.
5.2 Results
Simulated data Fig. 4 shows comparison results on the "bunny". LCT, FK, NeTF, CC-SOCR, and our method can recover the main body. However, FK and CC-SOCR lose structures around the ear, while LCT and RSD exhibit aliasing artifacts due to irregular undersampling. Among the learning-bai based methods, NeTF produces a blurry object with diffused artifacts and loss of structures around the ear. USM struggles to adapt to irregularly undersampled transients and fails reconstruct the hidden object. In contrast, our method successfully recovers most geometric structures of the bunny without aliasing artifacts, achieving the best quantitative results.
Real-world data We first tested our method on publicly available real datasets 10 12 then on self-captured real-world data (Fig/6). Without proper regularization, the direct reconstruction methods, i. e., LCT and RSD, exhibit severe aliasing artifacts due to undersampling. FK is depth- sensitive and fails to recover far-end structures the hidden scene. NeTF tends to produce blurry shapes due to its struggle with utilizing limited information in IUT. USM produces results that are nearly overwhelmed by aliasing artifacts in irregularly undersampled cases, but can achieve acceptable results in regularly undersampled cases (Fig 6). CC-SOCR can recover main objects for
(Fig and
7

---

Relay
Man Dweer (100)
Man Dener (100
Rardom
Window3
Lobber [101
Letbe [1t1
Deer (10)
Windoww
Deeer [10]
Droggnn
Deoggnn [22)
Gralfe
Vertcaa Shuiter
Statian [12]
Towner [X]]
Grafuu
Window
Tennee [12]
Wirdow
SU p3]
Window
SUU D-3
Figure 5: Reconstruction results of publicly available real-world dataset surfaces.
10
12 with different relay

---

Relay Surface
USM
CC-SOCR
Ours
GroundTruhh
Windon
Dovid
Rocuer 32d22 Harizotal Shu Shuter Requirr 32d2
Daud
Home Head
Hore Hoad
Figure 6: Reconstruction results of self self-captured real- real-world dataset with different relay surfaces.
all the test cases, but loses fine structures at distant depth layers ("Man Deer", "Letter", "Deer" and "Teaser") and underperforms on glossy ("Dragon") or retroreflective ("SU") objects. Our method achieves the best quality for various objects and sampling patterns. Our method generates stable results on real-world data with diverse attributes and relay surfaces despite being trained only on simple alphanumeric dataset and shutter-lin like relay surfaces. It notably outperforms the range-space solver LCT, successfully removing aliasing artifacts while preserving structure. The promising generalization highlights the effectiveness of our method in learning beyond the range space.
5.3 Inference time
Table I: Inference time of various methods. The values in the table represent the average inference time across 16 IUT with size of 128 128 256.
Method
LCT
FK
RSD
NeTF
USM
CC-SOCR
Ours
Runtime (CPU)
0.81 s
1.52 s
0.94 s
N/A
2.34
7.73 h
2.24
Runtime (GPU)
0.09 s
0.15
0.12 s
0.69
0.24
N/A
0.18
We compare the inference time of various methods on an Intel(R) Xeon(R) Platinum 8369B 2.90GHz CPU with cores and an NVIDIA 3090 GPU, respectively. The inference times of NeTF and average the inference times across 16 IUT shown in Fig. and Fig. Glach with size of 128 x 128 x 256. As shown Tab. T the direct reconstruction methods. LCT, FK, and RSD, are faster than the other methods. Unlike these one-step approaches, CC-SOCR iteratively solves the functional model by a series of alternative sub- tob-problems, requiring nearly eight hours to reconstruct thus requires significantly longer inference times than the other two learning-basee methods, USM, and our method. Note that both method and CC-SOCR achieve more accurate reconstructions than the other methods. However, our method is 12,000x faster than CC-SOCR CPU mode.
CC-SOCR vary with different IUT. Therefore,
albedo. NeTF stands the per-scene rendering framework and
5.4 Ablation study
We validate the effectiveness of the two core components of our method: the virtual scanning process (VS) and the SURE-based denoiser. For quantitative evaluation, we simulated dataset of 1,000 transients by rendering objects with complex geometries, such as chairs, clocks, guitars, sofas, and motorcycles. Our method and its two variants were tested on the simulated transients sampling with different relay surfaces. For qualitative evaluation, we assess the two components using real- world datasets, "Teaser" and "Dragon". "Teaser" features complex structures, which helps evaluate VS's ability to recover details. "Dragon",, with its glossy material, exhibits a lower signal-to-noise ratio in its acquired transient, which helps validate the SURE-based denoiser's effectivvness.

---

Table 2: Ablation study for the virtual scanning process and the SURE-based denoiser.
SURE-based denoiser
Virtual Scanning Process
PSNR (dB)
18.69
19.63
20.52
Relay Surface
With Denoiser
Without vs
With VS
Reby Surface
Without Denakeer
With Denoiser
Withour Denoiser
With Denaiser
Window
LCT
Ours
Ours
Window 4
LCT
LCT
Ours
Ours
(b)
Figure 7: Ablation studies on the virtual scanning process (VS) (a) and the SURE-based denoiser (b).
Virtual Scanning As shown in Tab. 2 the virtual scanning process resulted in an average im- provement of 0.89 dB. Fig. F demonstrates that without the VS component, method only recover a limited portion of the scene structure, which impaired by aliasing artifacts. contrast, full model reconstructs more detailed and cleaner structures, confirming the effectiveness of VS component in recovering null-space components.
SURE-based denoiser As evident from the quantitative results, the SURE-denoiser achieved an average performance improvement of 1.83 dB, which is more significant than that of VS. This result aligns with expectations because background noise affects the entire reconstruction volume, while aliasing artifacts caused by irregular undersampling disrupt the main structure. For both LCT and our method, as illustrated in Fig. b)), the denoiser effectively suppresses noise artifacts. In Sec. 5.2 we apply the denoiser to competing methods to enhance their robustness to noise for fair comparison. This demonstrates the versatility of the denoiser plug-and and aad---ayyyyyy play module other NLOS algorithms.
Additional ablation studies on the hyperparameters within the loss functions and the relay surfaces used for training provided in the supplementary materials.
6 Conclusion and limitations
Conclusion In this paper, propose an unsupervised learning-basse based framework for NLOS imaging from irregularly undersampled transients (IUT). By introducing virtual scanning process and SURE-based denoiser, our framework achieves high-quality and fast NLOS reconstructions from IUTT. Furthermore, it can be trained solely from noisy IUT, enabling future work on direct learning from real-world datasets to bridge the gap between simulated and real datasets. Our method outperforms the state-of-the-arr method on both simulated and real-world data with various relay surfaces. In future work, will extend method to non-confocal imaging systems for more practical applications.
Limitations Our method faces two main limitations. Firstly, is constrained the confocal system due to the lack of high-speed, cow-memory non-confocal forward operator for deep learning training. However, the framework theoretically extendable to general forward operators, indicating future re- search forward operators in NLOS imaging may require special consideration for irregular undersampling. Secondly, while our method entirely unsupervised, we only present the results of method which is trained using simulated transients due the time-consuming nature of collecting real transient datasets. Nonetheless, this approach partially mitigates the data gap introduced by simulated ground truth, and the results demonstrate excellent performance. Transitioning to deep learning that exclusively uses real transients is one of our future goals.
Acknowledgment
This work is supported by the National Natural Science Foundation of China under Grants 62231018. 6207 1322, 62072331.
10

---

References
[1] D. Faccio, A. Velten, and G. Wetzstein, "Non-line-of-sihtt imaging." Nature Reviews Physics, vol. no. 6. pp. 318-327, 2020.
[2] S. Chan, R. E. Warburton, Gariepy, Leach, and D. Faccio, "Non-line-of-sipe line-of-sight tracking of people at long range," Optics express, vol. 25, no. 9, pp. 10 109-10117, 2017.
[3] M. Isogawa, Y. Yuan, M. O'Toole, and K. M. Kitani, "Optical lon-line-of-sight physics- hased 3d human pose estimation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020, pp. 7013-7022.
[4] N. Scheiner. F. Kraus, F. Wei, B. Phan. F. Mannan, N. Appenrodt, W. Ritter, J. Dickmann, K. Dietmayer. B. Sick al., "Seeing around street comers: Non-line-on ofnee-o--sgee sight detection and tracking in an-th-wwll using doppler radar," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020, pp. 2068-2077.
[5] C. Wu, Liu, X. Huang, Z.-P. Li, C. Yu, J.-T. Ye, J. Zhang, Q. Zhang, X. Dou, K. Goyal et al., "Non-line-of-sigle imaging over 1.43 km," Proceedings of the National Academy of Sciences, vol. 118, no. 10, e20244681 18, 2021.
[6] X. Liu, Wang. L. Xiao, Z. Shi. X. Fu, and L. Qiu, "Non-line-ofe of- imaging with arbitrary illumination and detection pattern." Nature Communications, vol. 14, p. 3230, 2023.
[7] J. Grau Chopite, M. B. Hullin, M. Wand, and J. Iseringhausen, "Deep non- non-line-off lf reconstruction," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 960-969.
[8] Chen, F. Wei, K. N. Kutulakos, Rusinkiewicz, Heide, "Learned feature embeddings for non-line-of-sight imaging and recognition," ACM Transactions on Graphics (ToG), vol. 39, no. 6, pp. 1-18, 2020.
[9] F. Mu, S. Mo, J. Peng, X. Liu, J. H. Nam. S. Raghavan, A. Velten, and Y. Li, "Physics to the rescue: Deep -on-line-of-sight reconstruction for high-speed imaging," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[10] Y. J. Peng,, J. Ye, Y. Zhang, Xu, and Z. Xiong, "Nlost: Non-line-of-sight imaging with transformer." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 13 313-13322..
[11] M. O'Tole, D. B. Lindell, and G. Wetzstein, "Confocal non- line-of- imaging based on the light-cone transform," Nature, vol. 555, no. 7696, pp. 338-341, 2018.
[12] D. B. Lindell, G. Wetzstein, and M. O'Toole, "Wave-based non non-linc-of-sight imaging using fast fk migration," ACM Transactions Graphics (ToG), vol. 38, no. 4, pp. 1-13, 2019.
[13] X. Liu, I. Guillén, M. La Manna, J. H. Nam. S. A. Reza, T. Huu Le, A. Jarabo, D. Gutierrez, and A. Velten, "Non-line-of-sighe imaging using phasor- field virtual wave optics." Nature, vol. 572, 7771, pp. 620-623. 2019.
[14] S. Xin, Nousias, N. Kutulakos, A. Sankaranarayanan, G. Narasimhan, and Gkioulekas, "A theory of fermat paths for oon-line-of-sight shape reconstruction." " in Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, 2019, pp. 6800-6809.
[15] S. I. Young, D. B. Lindell, B. Girod, D. Taubman, and G. Wetzstein, "Non-line-of-sighe surface reconstruc- tion using the directional light-cone transform." in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2020, pp. 1407-1416.
[16] M. La Manna, F. Kine, E. Breitbach, J. Jackson, T. Sultan, and A. Velten, "Error backprojection algorithms non-line- sfggtt imaging." IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 7, pp. 1615-1626, 2018
[17] B. Ahn, A. Dave, A. Veeraraghavan, Gkioulekas, A. C. Sankaranarayanan, "Convolutional approxi- mations to the general non-line-offsight imaging operator," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 7889-7899.
[18] X. Liu, J. Wang. Z. Li, Z. Shi, X. Fu, and L. Qiu, "Non Moon-line-of of-e--ff-sghht reconstruction with signal-objer collaborative regularization," Light: Science & Applications, vol. 10, no. 1, 198, 2021.
11

---

[19] M. La Manna, J. Nam, S. A. Reza, and A. Velten, ofon-iin-of-sighh-imaanii using dynamic relay surfaces," Optics express, vol. 28, 4, pp. 5331-5339, 2020.
[20] C. Gu. T. Sultan, K. Masumnia-Bisheh, L. Waller, and A. Velten. "Fast non non-line-onn line--i----eeeee of- imaging with non-planar relay surfaces," in 2023 IEEE International Conference Computational Photography (ICCP).
JEEE, 2023, pp. 1-12.
[21] C. A. Metzler, D. B. Lindell, and G. Wetzstein, "Keyhole imaging: non-line- of -ight imaging and tracking of moving objects allong single optical path," IEEE Transactions on Computational Imaging, vol. pp.
1 12, 2020.
[22] M. Isogawa, D. Chan, Y. Yuan, K. Kitani, and M. O"Toole, "Efficient non-line-of-sight imaging from transient sinograms," in Computer Vision - ECCV 2020, A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, Eds. Springer International Publishing, 2020, pp. 193-208.
[23] J. Ye, Huang, P. Li, and Xu, "Compressed sensing for active non-line-of-sight imaging," Optics Express, vol. 29, no. 2, pp. 1749-1763, 2021.
[24] Yang, C. Zhang. W. Jiang, Zhang, and B. Sun, -Noos--ine-of-sightt imaging enhanced with spatial multiplexing," Optics Express, vol. 30, no. 4, pp. 5855-5867, 2022.
[25] X. Liu, J. Wang, L. Xiao, X. Fu, L. Qiu, and Z. Shi, "Fow-shot non-line-of-sight imaging with signal- surface collaborative regularization." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023, pp. 13 303-13312.
[26] S. Shen, Z. Wang, P. Liu, 7. Pan, R. Li, T. Gao, S. Li, and J. Yu, -Non-line-of-sight imaging via neural transient fields." IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 7, pp. 2257-2266 2021.
[27] J. Wang, X. Liu, L. Xiao, Z. Shi, L. Qiu, and X. Fu, "Non-line-of-sight imaging with signal superresolution network." in Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023, pp. 17 77422-177 429.
[28] Y. Li, Y. Zhang, J. Ye, F. Xu, and Z. Xiong. "Deep non-line-of-sight imaging from under-scanning mcasurements," Advances in Neural Information Processing Systems, vol. 36, 2024.
[29] J. Schwab, S. Antholzer, and M. Haltmeier, "Deep null space learning for inverse problems: convergence analysis and rates," Inverse Problems, vol. 35, no. 2, p. 025008, 2019.
[30] C. K. Sonderby, J. Caballero. L. Theis, W. Shi, and F. Huszár, "Amortised map inference for image super-resolution"" arXiv preprint 1r0.441610.044900 2016.
[31] M. Mardani, E. Gong, J. Y. Cheng, S. Vasanawala, G. Zaharchuk, M. Alley, N. Thakur, S. Han, W. Dally, M. Pauly er al., "Deep generative adversarial networks for compressed sensing automates mri," arXiv preprint /706 00051, 2017.
[32] D. Chen and M. F. Davies, "Deep decomposition learning for inverse imaging problems," in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16. Springer, 2020, pp. 510-526.
[33] A. Pajot, De Bézenac, and P. Gallinari, "Unsupervised adversarial image reconstruction," International conference on learning representations, 2018.
[34] Liu, Sun, C. Eldeniz. W. Gan, An, and U. Kamilov, "Rare: Image reconstruction using deep priors learned without groundtruth." IEEE Journal of Selected Topics in Signal Processing, vol. 14, no. 6, pp. 1088-1099, 2020.
[35] B. Yaman, S. A. H. Hosseini, S. Moeller, J. Ellermann, K. Ugurbil, and M. Akçakaya, "Self-supervised learning of thysics-guided reconstruction neural networks without fully sampled reference data," Magnetic resonance medicine, vol. 84, no. 6. pp. 3172-3191, 2020.
[36] J. Tachella, D. Chen, and M. Davies, "Unsupervised learning from incomplete measurements for inverse problems." Advances in Neural Information Processing Systems, vol. 35, pp. 4983 4995, 2022.
[37] A. Bora, E. Price, and A. G. Dimakis, "Ambientgan: Generative models from lossy measurements." in International conference on learning representations, 2018.
[38] Y. C. Eldar, "Generalized sure for exponential families: Applications to regularization," IEEE Transactions Signal Processing, vol. 57, no. 2, pp. 471 471-481, 2008
12

---

[39] F. Luisier, T. Blu, and M. Unser. "Image denoising in mixed poisson-gaussian noise," IEEE Transactions on Image Processing, p. 696-708, Mar 2011. [Online]. Available: httpl///e////.....o..o//////////
[40] Y. Le Montagner, E. D. Angelini, and J.-C. Olivo-Marin, "An unbiased risk estimator for image denoising in the presence of mixed poisson-gaussian noise." " IEEE Transactions on Image processing, vol. 23, no.
pp. 1255 1268, 2014.
[41] D. Chen, J. Tachella, and M. E. Davies, "Robust equivariant imaging: fully unsupervised framework for learning to image from noisy and partial measurements," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp.
[42] S. Soltanayev and Chun, "Training deep learning based denoisers without ground truth data." Advances in neural information processing systems, vol. 31, 2018.
[43] C. M. Stein, "Estimation of mean a multivariate normal distribution." " The annals of Statistics, pp. 1155-1151, 1981.
[44] G. Liu, A. Reda, J. Shih, T.-C. Wang, A. Tao, and Catanzaro, "Image inpainting irregular holes using partial convolutions," in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 85-100.
[45] D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Instance normalization: The missing ingredient for fast stylization." arXiv preprint arXiv: 08002,, 2016.
[46] J. Schlemper, O. Oktay, M. Schaap. M. Heinrich, B. Kainz, B. Glocker, and D. Rueckent, "Attention gated networks: Learning to leverage salient regions in medical images," Medical image analysis, vol. 53, pp. 197 207, 2019.
[47] A. Paszke, S. Gross, F. Massa, A. Lerer. J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga al., "Pytorch: An imperative style, pigh-performance deep learning library," Advances in neural information processing systems, vol. 32, 2019.
[48] D. P. Kingma and J. Ba. "Adam: A method for stochastic optimization." arXiv preprint a12.v:1412.6980. 2014.
[49] S. Ramani, T. Blu, and M. Unser, "Monte-cati carlo sure: A black-box optimization of regularization parameters general denoising algorithms," IEEE Transactions on image processing, vol. 17, no. 9, pp. 1540 1554, 2008.
[50] C. A. Metzler, Mousavi, Heckel, and G. Baraniuk, "Unsupervised learning with stein's unbiased risk estimator," arXiv preprint arXiv: 10531, 2018.
[51] V. Antun, F. Renna, C. Poon, B. Adcock, and A. C. Hansen, "On instabilities deep learning in image reconstruction and the potential costs of ai," Proceedings of the National Academy of Sciences, vol. 117, no. 48, pp. 30 08803000000 2020.
13

---

A Network architecture
128xt 220296
128xt 28x296
24
8
126x1 64x04x120
64xD4ix 28
04X64x120
Transient
546441128
32x32x64
32x32x64
32x32x64
Sampling Matrix
18
92
96
32
IN LeakyRalu
Tex:
16xt Bc32
16x1
3x3xá Conv IN LeakyReLu
Nearest
Concatenato
Figure 8: Overview of our P-Unet with 3D partial convolution for denoising irregularly undersampled transients. The spatial resolution of transients features in each layer is depicted below the blocks, and their respective channel counts are presented above the blocks. The resolutions and channels of the sampling matrices are identical to the transients features in the corresponding layer. The numbers on the blue arrows indicate the kernel size of the 3D convolution.
We begin by presenting the architecture of the network Fg- Since the vanilla convolution treats all voxels of the zero-padded IUT valid, it leads the distortion of information in undersampled transients during feature propagation. As illustrated in Fig. S we present an encoder-decoder network (P-Unet) based on 3D partial convolution 44]. Partial convolution utilizes sampling matrix to differentiate between valid and invalid voxels, effectively capturing spatial information from the transients. Additionally, is insensitive to input distribution. This layer enhances the network's generalization capability to diverse zero-padded IUT with varying distributions from different sampling rates.
incorporate the instance normalization (IN) layer [45], which
Given that the performance of Fg in VSRnet mainly depends on acquiring null space information from the training forward operators, we adopt 3D residual network featuring aitention-gated network [4).. While attention mechanisms might not significantly improve reconstruction quality, they help speed up the network's training process.
B Training details
Our method is implemented using PyTorch [47]. and we employ the Adam optimizer [48 with weight decay of 10 the first stage, the SURE-based denoiser model F& is trained with a batch size of for 40 epochs. We set the initial learning rate 1 10-3 and reduce it by a factor of 0.1 epoch 30. Subsequently, in the second stage, the VSRnet model Eg is trained with a batch size of for 20 epochs, utilizing an initial learning rate of 5 10-4 and reduction by a factor of 0.1 at epoch 10. In each epoch, we randomly select 40 complete simulated transients for each relay surface and extract signals from them to generate irregularly undersampled transients for training. All models were trained on 2 NVIDIA 3090 GPUs, taking nearly 40 hours in total. Regarding the loss function, the hyperparameters and are 0. 0.001 and respectively.
Details of proposed SURE-based denoiser
C.1
Results of compared methods using our denoiser
applied the proposed SURE-based denoiser to compared methods and selected relatively better results for comparative experiments. As depicted in Fig. 9 the results of L.TT, FK, and RSD exhibit reduced diffuse background noise when the denoiser applied. For learning-based, algorithms, our denoiser effectively removes cluster artifacts caused by measurement noise, thereby enhancing the
14

---

Relay Surface
LCT
FK
RSD
NcTF
USM
CC-SOCR
Ground Truth
With Denose Without Deroiser With Derolsee Withour Denoiser
Deer [10]
Deer [10]
Deer 110
Deer (10) [10]
Figure 9: Qualitative results of compared methods with and without the SURE-based denoiser.
clarity of primary structures. However, in the case of CC-SOCR, which already incorporates strong regularization, the improvement our denoiser limited. This likely due to the algorithm's sub-pre uub-problem already including denoising effect.
C.2 Derivation of the SURE loss
Stein's Unbiased Risk Estimation (SURE) framework has been developed and used in prior signal denoising works 42 49 These works did not consider the dark current of sensor outputs that might be removed in some image signal pipelines. Therefore,
extend the SURE denoising model to NLOS imaging by incorporating more practical noise characteristics of the detector. In this section, we provide the derivation of the SURE loss mentioned in the main text, following the notations in [40] 41
For compact presentation, simply denote the training set of transients by {ū, R 1, 2..... I G} instead of the two-subscript version fur.g = 1, 2.... I..,,,,g 1,2, G). I the
number of observed hidden scenes, and G is the number of forward operators associated with relay surfaces. and I are the number of scanning points and the number of time bins in each histogram of the transient, respectively.
In the supervised learning, given paired dataset of noisy transients {ū, and clean version {uy).
the model F's can be trained by minimizing the mean squared error (MSE) loss function:
E(
uu Fa(ū,)11 En
Enjullus - (())))1
(10)
st
However, the clean transient set {uy} is not available. To achieve unsupervised learning only with noisy transients, our goal is obtain an unbiased estimator of the mean squared error (MSE) standing on the SURE framework. To this end, we can further decompose the inner expectation in Eq, aO
Eujullu; Fo (2 Eau [My [12) + Eaju (1 Fa (ū,) - 2B ( (o())) (11)
The unbiased estimator of the second term Eq. 11 (ū, 112, which can be obtained without the clean transient uy. The first term Eq. [] can be rewritten as Ea { (u ū. because Eaju (ū,} uy. The first and third terms depend on u. Thus, we need to obtain unbiased estimators for them by considering the characteristies of the noise model.
In the case NLOS imaging, the noisy transients can be modeled as ~ Poisson(u + b), where denotes the dark counts. The unsupervised expressions of Taay {u] ú, and Eaju [u] Fū,)) can be obtained using the following lemma.
15

---

Lemma 1 (Lemma 1.2 in [40] Let v E Be such that ~ Poisson(ul (u) be an independent random
variables and let + Ret + Rhst be a function such that Euju 1 ( (w) < +00 for all m.
Buju [(())
Bujoi क1 (m)}
(12)
Let ((v) = Fo(ū), then Eaju [ [ can be expressed as:
Buju {u, F {b(ū)
Eu {(y))}
Eeju b) [(;;)) - Euju ( ((2))
b)
(13)
E, Φ (0;)} - Buju bb{))
Eaju 2 F (ū,)) (b Fo(ū,))
Here, we use the first-order Taylor approximation into:
(T,)) ~ Fo (ūjz - -FFo (ū²) to simplify Eq. 13
Euju (ū) (ū, Eqju (ūj 0F (ū)) Edju uu 1 F (ū,)}-
(14)
Then, the unbiased estimator of Eq. 14 is
-
(15)
Note that it is difficult obtain analytic form of ( when F.(.) is neural network. We adopt the Monte-Carlo approach [49 to obtain an estimate of the divergence of (o(ū) with the
following approximation:
dava (F)(
F (ū)).
(16)
5 where E is a small positive number, and e E {-1, 13tt is binary vector whose entities follow a Bernoulli distribution with equal probability [40].
Similarly, let (()) an identity function such that ((v) = u, and = Using the first-order
Taylor approximation, Eaju [u, ūt can be expressed
Enju u 4j}
Euju 11 o (1;)) Boju(b K(1;))
(17)
= E.ja 12 (uy - -du))} Ev (b (v,))
= Eaju [u,] (ūg -bū,)} - Eaju {b ū.).
We obtain the unbiased estimator of Eq. 17
(ūj 1) bb ūz.
(18)
where 1 is vector consisting of st ones. With the above derivation, the total unbiased estimator of the MSE given by:
| 1| 2ū, Fd(ūi) +1 -Tū bTū ūz
(19)
+ 2b Fo(ūt) + 2 (e, (Fs tee,) - Fo(ūg)))
After rearrangement, Eq.
19 can be expressed as:
{1اا Fo(ū;)11 -1
(20)
2 + 26 28 Fa + (c, (c, (o) (Fo((, +ee)) Fo(ū))))
Thus, we get the SURE loss function as described the main text:
1 2 + +b) üig
LSURE = -E(ig) { |Íūn (o st.
(21)
2 5T FTF 2 (cr,g " (Fo(uig Ees.g) (ūr.))))
2
sts
During the training process, we use zero-padded irregularly undersampled transients as input, but only consider valid values in the calculation of the SURE loss.
16

---

D Details of our NLOS system
We used a femtosecond fiber laser as a light source with a central wavelength of 1560 nm and a pulse repetition rate of 82 MHz. The light split by a 99 1 fiber coupler. The channel with 1 1% of the light was attenuated and sent into a fast photodetector, functioning as the start signal for the time-to-ame -o--mllitude convertor (TAC). The other channel, with 99% of the light, was connected to collimator by a piece of single mode fiber (SMF) and then travelled through a hole in the mirror. The beam was diffusively reflected by the relay surface, and the raster scanning beam-steering mirror. After diffusive reflections by the object and again by the relay surface, the echo light travelled back along the same optical path and was reflected to a second collimator. Finally, the echo light was detected by a fractal superconducting nanowire single-photon detector (SNSPD) coupled with SMF. The output voltage pulses from the fractal SNSPD were amplified by the RF amplifiers and input to the TAC as a stop signal for the coincidence counting. The transients were captured over 0.8 x 0.8 m2 scanning area, with a size of 128 128 512 and a bin width of ps.
controlled by the
E
Discussion
100%
50%
77.4%
62.4%
50%
Relay sufface
Scanning point
acteruattin
Laser & Detector
Hidden Scene
(a)
(b)
Figure 10: (a) The top view of the NLOS imaging system. The green oval region on the object indicates the area from which the scanning point can capture information. Light with different optical distances is represented by three colors. (b) The first row shows the maximum intensity projections of the transient along the time dimension, with the corresponding sampling rates indicated at the top.
In this section, discuss how different relay surfaces affect reconstruction quality. Generally, relay surface with a large missing area leads result in poor reconstruction quality due to information loss. To verify this, we removed signals from the fully-sampled transient according to the relay surfaces and reconstructed them using our method. is clear that parts the hidden scene facing large missing areas nearly impossible to reconstruct, illustrated in the second, third, and fourth columns of Fig (b). However, even with the same sampling rate, more uniform distribution of scanning points produces better results (c.g., comparing the fifth column to the second).
Therefore, in the challenging task of NLOS imaging from IUT, the sampling rate is not the sole determinant of reconstruction quality. When the detector focuses point on the relay surface, only parts of the hidden scene that are close to that point can be observed. As shown Fig 10 (a), the light attenuation term 1/r4 causes that histogram (1 x 1 x T) captured from scanning point primarily contains information from hidden areas close to that point, as indicated by the green oval area. This results in poor reconstruction when large areas of the relay surface are invalid for scanning. However, determining the acceptable proportion of the invalid area relative to the entire surface for successful reconstruction is thcoretically challenging, as it depends various factors, including the relative depth between the hidden scene and the relay surface, the scene's reflectivity and normals, detector efficiency, and laser power. We consider this an important for future work.
Additional ablation study
In this section, we extend our ablation study further to explore the optimal values and sensitivity of the hyperparameters within the loss functions. Additionally, we examine the effect of relay surfaces used for training. To accommodate variations testing data, we evaluated our method on different IUT of the "bunny", sampled according 15 distinct relay surfaces mentioned in the main text.
17

---

F1 Ablation study on hyperparameters
Table 3: Effect of the hyperparameters on the denoising performance in terms of PSNR (mean value and variance).
e
0.0001
0.001
0.01
PSNR
39.11=1.99
40.19
41.16#2.61
0.1
10
PSNR
41.53 + 1.25
36.97
34.92:10.17
Effect of the positive number In experiments, the transient is normalized the range [0, 100]. Following the recommendation in [50), where the value of is suggested to be set around max(u)/100g) set to 0.1. As shown Tab. our SURE-based denoiser exhibits optimal performance when 8 = 0.1.
Table 4: Effect of the hyperparameters 3 on the reconstruction performance in terms of PSNR (mean value and variance).
B
0
0.0001
0.0005
PSNR
16.19:1..21
17.36:10.45
18.25::0.77
0.001
0.005
0.01
PSNR
19.03 + 0.64
18.39-10.81
18.03++0.86
Effect of the hyperparameter 3 The hyperparameter / acts as a weight to balance the MC loss and the VS loss. As shown in Tab. our method achieves optimal performance when 0.001. The effectiveness of our method decreases notably as 3 decreases. When the VS loss is completely disabled (3 0), the model fails learn beyond the range space, highlighting the significance of virtual scanning in our approach.
F.2 Ablation study relay surfaces for training
It's widely recognized that the robustness of neural networks depends heavily on the diversity of the training dataset. As discussed in [51) some neural networks struggle to adapt to changes in the sampling rate. They perform best when the sampling rate of input matches that of training dataset.
Any deviation, whether decrease or increase in the testing data's sampling rate, can lead to decline in performance. To address this challenge, common strategy is to augment the diversity of the training data's s sampling rate.
Table 5: Effect of the intervals of relay surfaces for training on the reconstruction performance in terms of PSNR (mean value and variance).
Interval PSNR
[4]
[12]
[20]
07.63++0.64
17.77+0.95
18.16+00.76
Interval
[8.16]
[4,12.20]
[4,8,12,16620]
PSNR 18.36
18.85+00.8
19.03 + 0.64
Interval In the context of NLOS imaging from irregularly undersampled transients, we can modify the shape of the relay surface adjust the sampling rate of transients. In theory, light reflecting from the hidden scene spreads across the entire relay surface. However, due to the radiometric fall-off' ||p igl" only limited portion of the reflected light, concentrated in a small area of the relay surface, can be effectively captured by the time-resolved detector. As a result, we manipulate the interval of the like surfaces to modify the local sampling rate of undersampled transients.
As shown in Tab. 5 the optimal combination of interval values for relay surfaces is [4,8, 12, 16, 20). Our method's performance shows decline with a reduction in the variety of interval types. This suggests that using undersampled transients with more diverse range of sampling rates for training can enhance the generalization capability of our method.
18

---

Table 6: Effect of the rotations of relay surfaces for training on the reconstruction performance in terms of PSNR (mean value and variance).
Rotation PSNR
10 18.08=00.55
20
30
18.33=1.05
18.74+1.32
Rotation
40
50
60
PSNR
19.03 + 0.64
18.85:00.94
18.76:10.75
Rotation Similarly, we can modulate the diversity of relay surfaces by varying the number of rotations. As shown in Tab. [6] the optimal number of rotations is 40 when the count of transients using for training is 8,000 in total. When the number of rotations falls below 40, the diversity of relay surfaces becomes inadequate. Conversely, when it exceeds 40, the allocated transients for each relay surfaces becomes too limited.
G Additional results
We present additional results on publicly available real data TO
with more irregular relay surfaces, as shown in Fig 11 and Fig [2 The experiment setup for data processing and the compared methods
remains consistent with the description in the main text.
Reley Surface
LCT
FK
RSD
NeTF
USM
CC-SOCR
Our
Ground Trush
Man Mon [10)
Reqular 32x32
Man Letee 100
Window
Horzortal Shuter
Man Inter [10]
Windoww
Leioor 200
Lether [10]
Figure 11: Reconstruction results of publicly available real-world dataset [10 with different relay surfaces.
19

---

Relay Surfacee
LCT
CC-SOCR
Tooos 1zz)
Papercud Horiznral
Teaser 1121
Papercuas Papercut
Bkn 122
Bie
Dusbor (12)
Window
Vertica Shuter
Outtoor 1121
Figure 12: Reconstruction results of publicly available real-world dataset surfaces.
12
with different relay
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We make the main claims in introduction in Sec. [ The claims matches theoretical results in Sec, 3.2 and experimental results Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Our paper discusses the limitations Sec.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
20

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specinf well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide the analysis of the motivation Sec. 3.2 and the virtual scanning
strategy in Sec. 4.2 And we provide the derivation of the proposed SURE-baa -bsdd loss in
Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe proposed framework Sec. We present proposed network architecture in Sec. Aland provide training details in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
21

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
The data and code will be released after publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
http:///otteuubmsssooooooooo.ooo //nips.cc cc/ httpc//ppwwccccuiieeeeeeeeeeeeee hwwp:///wuudde///teeebbiiissssss
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Detalls
22

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We specify all the training and testing details in Sec. Sec. and Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropriate
information about the statistical significance of the experiments?
Answer: [No]
Justification:
The computational resources require for error bars are too high.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide information the computer resources in Sec. 5.3 and Sec.
B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
23

---

9. Code Of Ethics
Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://www.aaaa....ppppppaaaaaaa
Question:
Answer:
[Yes]
Justification:
We reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work on non-line-of-sight imaging does not have any apparent societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
24

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We cited the original paper that provided code package and dataset in refer- ences.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.ppaperswi paperswi thcode. com datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The new assets, including the dataset and code, will released with documen- tation after publication.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
25

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "JKANdCHBEQ", "reviewer": "Reviewer_WCZG", "summary": "The authors attempt the problem of NLOS imaging in the irregularly undersampled transients data case i.e. where the scan pattern on the relay wall isn’t dense or regular. To tackle the problem the authors introduce two main components (both trained unsupervised):\n\n1) A SURE-based denoiser, which denoises the input transients \n\n2) A VSRNet, a “virtual scanning” module, where the estimated albedo is then transformed back into an undersampled transient, and the albedo is estimated again from this sample. VSRNet is trained with an MC loss, enforcing consistency between the two albedo estimates. \n\nThe authors justify their choices by arguing about the null space of the light-transport matrix. \nThe method is benchmarked against a suite of datasets, one of which is also self-collected.", "questions": "1) Did the authors try to see the effects of different patterns on the reconstruction (not just in training as in the supplement)? I can see a bit in Fig 4 and 6, but are there any limits we can expect i.e. how many scan points do we need, are some patterns worse than others etc?", "limitations": "Limitations of the method are not really discussed in the paper, but there is an adequate section in the supplement. Societal impact N/A.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Quality: The experiments are well conducted, with great ablations. I especially like that the authors use the SURE denoiser for other methods as well, better justifying the VSRNet. I also like the idea behind VSRNet itself. I think its also great that the authors benchmark the reconstruction speed itself. \n\nClarity: For the most part I think the paper is well written.", "weaknesses": "Originality/Significance: I’m not sure the authors do a great job motivating the problem of irregularly undersampled transients itself. There is a slight motivation in the introduction of scanning through fences etc, but I’m not sure the problem is convincing on its own. I think this also bleeds into a discussion about significance, I’m not sure if the contributions of the paper will be valued enough without a stronger motivation in the introduction/abstract for the problem itself.\n\n\nClarity: I appreciate the authors trying to introduce a more theoretically grounded framework, but I’m not sure how important the discussion of the null space actually is. I might have misunderstood some parts, but if I understand correctly the problem is just fundamentally ill-defined, and we could have done without the discussion between lines 100-114.\n\nI think it’s usually good practice to have the method understandable through the figure itself, which is why I would suggest that the authors have a better description for Figure 2. I think it would be good if the authors described the separate components shortly there itself. \n\n\nTypos:\n\nL61: Citation for Manna et al. not hyperlinked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are highly encouraged by the positive recommendation and comments from the reviewer on our experiment, method and presentation. Furthermore, the raised questions are both central and valuable. We address the main questions as follows.\n\n**Q1: The significance of NLOS imaging from irregularly undersampled transients.**\n\n**Reply:** The problem of irregularly undersampled transients in NLOS imaging is critical for practical applications. \nIn real-world scenarios, relay surfaces are often discrete and irregularly shaped, such as fences, window shutter and window frame. \nMost current algorithms rely on dense and continuous transients, limiting their practical application. \nDeveloping algorithms for IUTs is essential to extend NLOS imaging to these common environments. \nOur method addresses this by efficiently using sparse transient information, significantly broadening the applicability of NLOS imaging.\n\n\n**Q2: The importance of the discussion of the null space.**\n\n**Reply:** The discussion in lines 100-114 is crucial for understanding the foundation of our proposed **unsupervised learning** algorithm, as it explains why our virtual scanning strategy is necessary and effective.\n\nSpecifically, our method is inspired by the range-null decomposition [1] and null space learning [2][3][4].\nWe observed that using only measurement consistency loss (MC loss) is insufficient for high-quality reconstruction. This is because only using MC loss for training cannot recover the null-space component of the target. To this end, we propose the virtual scanning strategy to recover the missing null-space component for high quality reconstructions, as shown in Figure 3(c) of the manuscript. \n\nIt establishes the theoretical foundation for our approach, elucidating the rationale behind our method and emphasizing how we address the limitations of prior techniques. Without this theoretical context, readers might question why the pipeline performs exceptionally well.\n\n**Q3: The effects of different patterns on the reconstruction and \nsome limits we can expect.**\n\n**Reply:** Thank you for this very insightful observation and suggestion. \nIn Figure 1(b) (see global rebuttal), we illustrate how the missing pattern affect the reconstruction quality. In general, a relay surface with a large missing area often results in poor reconstruction quality due to the loss of information. Even with the same sampling rate, more uniform scanning point distribution yields better results (e.g., the fifth column vs. the second).\n\nAs shown in Figure 1(a) (global rebuttal), light attenuation term $1/r^4$ causes that a histogram ($1\\times1\\times T$) captured from scanning point $p$ often only contain information from hidden areas close to the scanning point $p$, such as the green oval area shown.\nThis leads to poor reconstruction if large parts of the relay surface are missing. \nHowever, the acceptable size of the missing area for reconstruction is hard to derive theoretically. As it depends on multiple factors such as the relative depth between the scene and the relay surface, the scene's reflectivity, normals, detector efficiency, and laser power.\n\nNevertheless, we tried to obtain a general trend through quantitative experiments.\nWe simulated a dataset of 1,000 transients by rendering objects (chairs, clocks, guitars, sofas, motorcycles) with random scaling and positioning. \nThe relay surface consisted of random points with sampling rates ranging from 0.1\\% to 100\\%, forming 30 groups. \nAs shown by the red curve in Figure 2, the PSNR of the reconstructed intensity map starts to decline sharply below the sampling rate of 6\\%. Therefore, we recommend that the sampling rates for patterns should larger than 6\\%. We will add these discussion in the final version.\n\n**Q4: Presentations about Figure 2 and others.**\n\n**Reply:**\nThanks for your suggestions on the presentation. We will include additional descriptions of the components in Figure 2, such as the networks $F_\\theta$ and $F_\\phi$, in the camera-ready version. \nAdditionally, we will repair the hyperlinks in the citations and move the limitations section to the main text.\n\n[1] Schwab, Johannes, Stephan Antholzer, and Markus Haltmeier. \"Deep null space learning for inverse problems: convergence analysis and rates.\" Inverse Problems 35.2 (2019): 025008.\n\n[2] Sønderby, Casper Kaae, et al. \"Amortised MAP Inference for Image Super-resolution.\" International Conference on Learning Representations. 2022.\n\n[3] Wang, Yinhuai, et al. \"Gan prior based null space learning for consistent super-resolution.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 3. 2023.\n\n[4] Wang, Yinhuai, Jiwen Yu, and Jian Zhang. \"Zero-Shot Image Restoration Using Denoising Diffusion null space Model.\" The Eleventh International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ksa34l4Fj6", "reviewer": "Reviewer_cypv", "summary": "This paper proposes a non-line-of-sight (NLOS) imaging method for scenarios where transients are irregularly undersampled on the relay surface. The proposed method includes a SURE-based denoising technique to handle noisy transient data, specifically addressing Poisson noise. Additionally, a novel unsupervised learning network called VSRnet is introduced, enabling consistent reconstruction from different irregularly undersampled transients (IUT) of the same 3D albedo volume through a process termed virtual scanning. Extensive comparisons with recent NLOS methods using both simulated and real IUT data demonstrate the superiority of the proposed approach.", "questions": "If there are any misunderstandings in the weaknesses pointed out, please clarify them.", "limitations": "The limitations are mentioned only in the supplementary material and not referenced in the main text.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "+ The virtual scanning process makes a contribution by achieving robust reconstruction from incomplete observations in NLOS imaging through unsupervised learning.\n+ The SURE-based denoiser, which accounts for Poisson noise, is a contribution as it appears to be universally applicable for denoising transients.", "weaknesses": "1. While the SURE-based denoiser may contribute to handling noisy transient data, its originality is limited, and its connection to IUT is unclear.\n1. The assumptions regarding the relay surface are not practical. In a setting like Figure 1, the BRDF of the relay surface would vary among the transient samples. However, in the experimental setting, \"we extracted signals from the complete transients according to various irregular relay surfaces\" (207), which overly simplifies conditions.\n1. The justification for zero padding is not clear and it might be unfair for other methods. For instance, instead of treating regions with no samples as zero, using simple interpolation methods like bilinear interpolation might improve the quality of the competing methods.\n1. The validation of the proposed virtual scanning is insufficient. While it is claimed that virtual scanning complements the null space (159), this explanation is intuitive rather than theoretically substantiated. There would be no guarantee that \"a set of operators $\\mathcal{H}$\" (170) will fully complement the null space. For example, while it is mentioned that the sampling pattern for virtual scanning is random (138), there is no discussion or experimental validation on the necessary sampling ratio. Although the proposed method conducts the virtual scanning only once for $\\rho^{(2)}$, it can be conducted iteratively. However, the iteration might also lead to deviations from observations. This aspect is not adequately discussed.\n- Minor comments:\n  - The meanings of notations such as $\\hat{}$, $\\tilde{}$, and various subscripts are not clear enough.\n  - Variables like $l$ (86), $G$ (131), and $k\\neq g$ (149) are not explained.\n  - The process of obtaining $\\rho^{(1)}$ and $\\rho^{(2)}$ (154) is described in text but not well defined in equations, making it difficult to understand correctly.\n  - Referring to the 3D albedo volume $\\rho$ as the latent 3D volume (133) seems inappropriate.\n  - The citation format might be wrong.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for valuable comments. We address the main questions as follows.\n\n**Q1: Connection and novelty of our SURE-based denoiser.**\n\n**Reply:** In the irregularly undersampling, the quality and stability of reconstruction could be severely affected by noise, necessitating a robust denoiser to rescue. As shown in Fig. 2 (see experiment setup in the global rebuttal), the PSNR improvement of our method with SURE denoiser over the one without denoiser is up to more than 7 dB at sampling rates between 1.0\\% $\\sim$ 4\\%. With the SURE denoiser, the operating range of sampling rates is broadened to low rate end, which significantly extend the applicability of our method. \n\nRegarding the originality, the SURE denoiser is specifically tailored for NLOS imaging. First, we incorporated the noise model of time-resolved detectors (see Sec. 4.3), addressing the dark noise $b$ in transients, and derived the SURE loss, which is not a straightforward modification of an existing one. \nSecond, we proposed a neural network combining partial convolution and instance normalization (IN) layers (see Sec. A).\nPartial convolution contributes to suppress the loss of IUT information during feature propagation through the vanilla convolution and IN layers make network suitable to various distributions of IUTs with different sampling rates. \n\n**Q2: The assumption of relay surfaces.**\n\n**Reply:** Presently, NLOS imaging research community commonly assumes that relay surfaces exhibit uniform diffuse reflectance. However, to our knowledge, none previous methods have taken into account relay surfaces with varying BRDFs. We thank the reviewer poses a challenging task for achieving more practical NLOS imaging and warrants in-depth investigation in future studies.\n\n**Q3: The padding (interpolation) mode for unobserved points.**\n\n**Reply:** We further evaluate two interpolation methods, the bilinear method and nearest-neighbor method, as the pre-processing of reconstruction from IUTs with four baseline methods (LCT, FK, RSD and USM). \nAs shown in Figure 4, the performance of interpolators on the final reconstruction is really dependent on the quality of interpolated transient signals, and essentially on the scanning patterns. For scanning patterns with relatively uniform scanning points (small holes on the relay surface, such as Window 3 and 5), the bilinear interpolater can help significantly suppress background reconstruction artifacts. However, scanning patterns with skewed distributed scanning points (large missing areas on the relay surface, such as Window 7), the bilinear interpolater can even degrade the reconstruction quality due to the unreliable interpolated transients in large missing areas.\nFor all cases, our method still outperforms than other four baselines.\n\nSince there is not an all-conquering one, we felt that using the simple zero padding is a reasonable comparison setup. Another choice is to adaptively select interpolation methods according to the scanning pattern, e.g. bilinear interpolation for patterns with more uniform sampling and zero padding with skewed sampling. We will discuss more above the padding mode in the final version.\n\n**Q4: Null-space learning of the virtual scanning.**\n\n**Reply:** \n1. This could be a misunderstanding.\nOur goal is trying to recover the null space components of the reconstruction target (line 159). For this, We provide an intuitive illustration in Fig. 3 of the manuscript and some theoretical explanations in lines 159-171. \nFor the special task of NLOS imaging from IUT, some portions of information about the hidden scene may not be captured due to light attenuation. So there is no guarantee that the proposed virtual scanning could complement the entire null space. However, adding various operators for training can effectively improve the generalization of our method. We will discuss more about this in our final version.\n\n2. The phrase ''randomly sampled from the surface base'' (line 138) indicates that during training, we select one sampling pattern different from $M_g$ from the group of 200 sampling patterns during training to learn the null space components. \n\n3. Indeed, the sampling rate for each sampling pattern should not be too low for stable and acceptable reconstruction.  We conducted a quantitative experiment, which is detailed in the global rebuttal. Figure 2 illustrates a significant drop in reconstruction quality when the sampling rate falls below 6\\%. Notably, all sampling rates for patterns in the surface base exceed this lower bound.\n\n4. There are two main reasons for the design of virtual scanning: \nFirst, more iterations in virtual scanning would accumulates errors, causing severe fluctuations in losses and hindering network convergence, which affects final the performance of network. This would also increase required computation. \nSecond, in practice, we iterate through each pair of sampling patterns from the surface base to make network learn the null space of the associated operator. This training strategy successfully accelerates network convergence, stabilizes parameter updates, and improves reconstruction quality.\n\n**Q5: Some minor comments.**\n\n**Reply:** The notation $\\tilde{}$ refers to the noisy version of transients, and $\\hat{}$ refers to their denoised version. $l$ refers the spatial dimension of recovered volume and $G$ is the total number of the operators group. $k\\neq g$ means that $k$ is a different number from $g$ and hence their corresponding sampling patterns are also different.\n\nThank you for pointing out these issues. We will add an additional section to clarify all symbols and variables. We will add more descriptions on the obtaining process of $\\rho^{(1)}$ and $\\rho^{(2)}$, and updates other presentation issues in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c88uFVY2ng", "reviewer": "Reviewer_8wFw", "summary": "To address the challenges of slow inference speed and poor generalization to irregularly relay surfaces \nin non-line-of-sight (NLOS) imaging, this paper proposes a learnable unsupervised training framework\nwith the excellent novelty, as well as a Virtual Scanning Reconstruction Network (VSRnet).\nFurthermore, to mitigate the noise present in transient data, the authors introduce a denoiser based on\nStein's Unbiased Risk Estimate (SURE) loss. Extensive comparative experiments and ablation studies\ndemonstrate that the proposed method significantly enhances reconstruction quality compared to\nexisting approaches.", "questions": "1. I am confused about the content in M_k and M_g. Could you please provide further definitions or\nexplanations for M_k and M_g?\n2. The masks used during training exhibit regular patterns (e.g., evenly spaced stripes). Would\nintroducing more irregularly varied masks further enhance the robustness of the network?\n3. It is unclear how the authors constructed different forward propagation operators H based on the\nvarying irregularly relay surfaces?", "limitations": "Discussed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The authors provide a novel analysis of the challenges faced by unsupervised methods in NLOS\nimaging applications. They specifically incorporate different types of relay surfaces as prior information\nduring the training phase of the model, which effectively enhances the network's generalization ability to\ntransient data collected from irregular relay surfaces.\n2. Based on relevant mathematical theories, the authors provide detailed derivation steps for the\nproposed SURE Loss in the supplementary material. They offer a computable form of the unbiased\nestimation term using techniques such as Monte Carlo approximation. Although this loss function is\nprimarily designed based on existing theoretical work, its application in NLOS imaging remains a\nvaluable contribution.\n3. The authors provide extensive experimental results on real-world data, demonstrating that their\nproposed method can deliver high-quality reconstruction results", "weaknesses": "1. The lack of explanations for some key symbols (e.g., the explanation for M_g in Figure 2(a)\nand the construction method of M_k in Fig. 2(d)) and schematics (e.g., Fig. 3(c) is difficult for\nreaders to understand from the brief descriptions in the paper) might hinder readers from\nreproducing the proposed algorithm. Providing further explanations could significantly enhance\nthe readability of the paper.\n2. There is a lack of quantitative evaluation results for VSRnet and SSIM-denoiser. Section 5.4 of\nthe paper only provides qualitative comparisons of the two modules, neglecting quantitative\nexperimental assessments. For readers, quantitative results are more effective than qualitative\nresults in intuitively understanding the model's effectiveness.\n3. The title of the paper includes the keywords \"unsupervised\" and \"irregularly undersampled transients,\" which seem to represent two independent issues. Specifically, is the choice of an unsupervised framework a necessary outcome for addressing the reconstruction problem of irregularly undersampled transients? If the authors could clarify this point, I believe it would constructively enhance the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are highly encouraged by the positive recommendation and comments from the reviewer. We address the main questions as follows.\n\n**Q1: The lack of explanations for some key symbols and schematics.**\n\n**Reply:**\nExplanations: As stated in line 90-92, the forward operator $H$ is highly related to the scannable region on the relay surface. \nFor the relay surface numbered $g$, $H_g$ is the forward operator using this relay surface, $M_g$ is the 3D version mask of the sampling pattern, constructed by repeating the 2D sampling pattern $t$ times along the time-dimension. \nSimilarly, $M_k$ is another 3D version mask different from $M_g$. \n\nSchematics:  Figure 3(c) (main text) builds on Figure 3(b) (main text), which shows the training case with only the measurement consistency (MC) loss, corresponding to Figure 2(b) (main text).\nIf only the MC constraint is imposed, the network cannot uniquely recover $\\rho$ because multiple outputs can satisfy the MC constraint, as depicted by the blue dashed line in Figure 3(b) (main text).\nIn Figure 3(c) (main text), when using the virtual scanning, $\\rho^{(1)}$ is projected onto the $H_2$'s range-space $R_{H_2}$, and we can recover the range-space component $D_r(\\rho^{(1)})$ of $\\rho^{(1)}$ using the pseduo-inverse operator (LCT). \nGiven $\\rho^{(2)}=D_r(\\rho^{(1)})+F_\\theta(D_r(\\rho^{(1)}))$ and $\\rho^{(1)}=D_r(\\rho^{(1)})+D_n(\\rho^{(1)})$, the proposed VS loss $\\rho^{(1)}=\\rho^{(2)}$ is equivalent to $F_\\theta(D_r(\\rho^{(1)}))=D_n(\\rho^{(1)})$.\nTherefore, virtual scanning ensures that the network $F_\\theta$ learns a mapping from $D_r(\\rho)$ to $D_n(\\rho)$, ensuring the ability of our algorithm to achieve high-quality reconstruction.\n\n**Q2: The quantitative evaluation results for VSRnet and SURE-denoiser.**\n\n**Reply:**\nWe simulated a dataset of 1,000 transients by rendering objects (chairs, clocks, guitars, sofas, motorcycles) with random scaling and positioning. \nWe tested our method and its variants on the simualted dataset sampling with 16 various relay patterns.\nThe quantitative results for VSRnet and SURE-based denoiser are as follows:\n| SURE-denoiser | Virtual Scanning | PSNR (dB) |\n|----------|----------|----------|\n| &times; | &check; | 18.69 |\n| &check; | &times; | 19.63 |\n| &check; | &check; | 20.52 |\n\nAs evident from the results, the SURE-denoiser provided an average performance improvement of 1.83 dB and the virtual scanning provided an average improvement of 0.89 dB.\nThe SURE-denoiser's quantitative performance improvement is more significant because background noise affects the entire reconstruction volume, while aliasing artifacts caused by irregular undersampling mainly disrupt the primary structure. \n\n**Q3: Is the choice of an unsupervised framework a necessary outcome for addressing the reconstruction problem of irregularly undersampled transients?**\n\n**Reply:** This is a very insightful question. We believe that the unsupervised framework is not only an effective approach for NLOS imaging from IUT but also a promising paradigm for deep learning in NLOS imaging research.\n\nIn recent years, learning-based algorithms have gained wide attention in the NLOS field. Their powerful function fitting and prior learning capabilities allow them to achieve better performance upper bound compared to model-based algorithms. \nHowever, most of these methods rely on supervised learning with simulated data, which creates a gap between simulated and real-world data, limiting their real-world performance. \nFurthermore, supervised learning requires paired datasets, which makes the model be prone to overfitting to the training domain, leading to poor generalization.\n\nIn contrast, unsupervised learning does not rely on paired data, allowing models to generalize better to new and unseen scenes.\nThis approach is crucial for NLOS imaging tasks where recovering diverse hidden scenes is necessary. \nWe believe the robust generalization capability of the unsupervised paradigm has been demonstrated by our extensive experiments with various real data and irregular patterns.\nAdditionally, the unsupervised paradigm reduces the costs and time associated with acquiring labeled datasets, accelerating research progress. \nTherefore, the choice of an unsupervised framework is crucial for effectively addressing NLOS imaging. \nWe will also open-source our code to support further research in the NLOS community.\n\n**Q4: Would introducing more irregularly varied masks further enhance the robustness of the network?**\n\n**Reply:** In our experiment, we found that introducing irregular masks led to unstable training and further reduced network performance. \nDue to the light attenuation effect, only a portion of the transient typically contain useful scene information. Therefore, using irregular masks for training often miss a significant part of the information in fully-sampled transients, leading to anomalous irregularly undersampled transient. \nSince our simulated dataset includes objects of various sizes and poses, these anomalies frequently occur during training, making it difficult for the network to converge and reducing its robustness.\n\n**Q5: How the authors constructed different forward propagation operators H based on the varying irregularly relay surfaces.**\n\n**Reply:**\nWe constructed different forward operators $H$ using an operator decoupling method, as described in line 149-152.\n\nFor the full-sampled relay pattern, the forward operator is denoted as $H_r$, and the corresponding full-sampled transient is $u_r=H_r\\rho$, where $\\rho$ is the hidden volume.\nSimilarly, for a relay surface numbered $g$, the observed transient $u_g$ is given by $u_g=H_g\\rho$. \nBy replacing through expere $u_g=M_g\\odot u_r$, we can obtain expression that $M_g\\odot H_r\\rho = H_g\\rho$. \nSo we composition $M_g\\odot H_r$ to constructed forward operator $H_g$. In practice, the composition can be efficiently computed through Hadamard product and the fast Fourier transform.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9q4sSqkYRA", "reviewer": "Reviewer_wGeV", "summary": "The paper introduces a unique unsupervised learning framework for non-line-of-sight imaging from irregularly undersampled transients. The proposed method overcomes the dependency on paired data and achieves higher fidelity, greater robustness, and remarkably faster inference times.", "questions": "1. Have you considered other irregular relay surfaces that is not composed of simple repetitive or symmetrical shapes?\n2. What is the motivation of using different shutter pattens as the simulated relay surfaces? Have you tried other patterns?\n\nI will reconsider my rating if your answers address my confusion.", "limitations": "The authors presented the limitations and proposed some future direction to addressed them.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes an unsupervised learning framework that overcomes the dependence on dense scanning and paired data required by existing methods.\n2. Extensive experiments, including both simulated and real-world data, validate the effectiveness and generalization ability of the method.", "weaknesses": "1. The 'irregular windows' in the test scenarios composed of regular, repetitive, or symmetrical simple geometric shapes, which are more close to the relay surfaces simulated during training. Whether this method can be extended to more complex real-world scenarios remains a concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are highly encouraged by the positive recommendation and comments from the reviewer on our experiment, method and presentation. Furthermore, the comments and suggestions are inspiring, helpful, and valuable. We address the main issues as follows.\n\n**Q1: Irregularity of scanning patterns.**\n\n**Reply:**\nWe tested our method on publicly available real datasets with more irregular relay surface. These irregular relay surfaces do not contain simple repetitive or symmetrical shapes, such as the signage of NIPS 2024, window papercut with various shapes and even random graffiti. As shown in Figure 3, our method still achieves the best quality for various sampling patterns. In addition, as shown in Fig. 5 and 6 in the manuscript, the relay surface ''random points'' and ''regular 32$\\times$32'' are more different from our training patterns, but our method can also handle them well. This is because our method's generalization is guaranteed by the theoretically grounded framework in learning beyond the range space.\n\n**Q2: The motivation of using different shutter patterns for training.**\n\n**Reply:** In the early development of our pipeline, we have tried to use random irregular relay surfaces for training, hoping that it would enhance our algorithm's robustness. However, this setting resulted in dramatic loss fluctuations and unstable training. \nWe found that shutter-like patterns yield stable training. By adjusting the rotation angles and intervals of these patterns, the network generalizes well to a wide range of irregular sampling patterns (see Ablation studies in Sec. C.2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Figure 1:** As suggested by reviewers WCZG, 8wFw and wGeV who concern about the effects of different relay patterns, we address this issue by analysing the NLOS imaging model and corresponding experiments.

Figure 1(a) shows the top view of a typical confocal imaging system. 
Different colored lines indicate light rays illuminating to different positions in the hidden scene, and transparency of them represents light intensity.  The light attenuation term $1/r^4$ causes that a histogram ($1\times1\times T$) captured from scanning point $p$ often only contain information from hidden areas close to the scanning point $p$, such as the green oval area shown. Photons from farther areas attenuate due to diffuse reflection term $1/r^4$, making them almost undetectable.

In Figure 1(b), we illustrate how the missing pattern affect the reconstruction quality. In general, a relay surface with a large missing area often results in poor reconstruction quality due to the loss of information. Even with the same sampling rate, more uniform scanning point distribution yields better results (e.g., the fifth column vs. the second).

Similarly, using irregular relay patterns for training can miss a significant part of the information in fully-sampled transients, generating anomalous IUT. 
Since the simulated dataset is rendered from objects of various sizes and poses, these anomalous IUTs frequently occur during training, making the network difficult to converge and ultimately reducing its robustness.

**Figure 2:**
As suggested by reviewers WCZG, cypv and wGeV who concern about necessary sampling ratio, we address this issue by adding quantitative experiments on different sampling ratios.

Specially, We simulated a dataset of 1,000 transients by rendering objects (chairs, clocks, guitars, sofas, motorcycles) with random scaling and positioning. 
The relay surfaces for testing are consisted of random points with sampling rates ranging from 0.1\% to 100\%, forming 30 groups.

As shown by the red curve in Figure 2, the PSNR of the reconstructed intensity map starts to decline sharply below the sampling rate of 6\%. Therefore, we recommend that the sampling rates for patterns should larger than 6\%.

In addition, the PSNR improvement of our method with SURE denoiser over the one without denoiser is up to more than 7 dB at sampling rates between 1.0\% $\sim$ 4\%. With the SURE denoiser, the operating range of sampling rates is broadened to low rate end, which significantly extend the applicability of our method. 

**Figure 3:**
As suggested by reviewer wGeV who concerns about our method's generalization to irregular relay surfaces that is not composed of simple repetitive or symmetrical shapes, we address this issue by adding qualitative experiments.

We tested our method on publicly available real datasets with more irregular relay surface. These irregular relay surfaces do not contain simple repetitive or symmetrical shapes, such as the signage of NIPS 2024, window papercut with various shapes and even random graffiti. As shown in Figure 3, our method still achieves the best quality for various sampling patterns. In addition, as shown in Fig. 5 and 6 in the manuscript, the relay surface ''random points'' and ''regular 32$\times$32'' are more different from our training patterns, but our method can also handle them well. This is because our method's generalization is guaranteed by the theoretically grounded framework in learning beyond the range space.

**Figure 4:**
As suggested by reviewer cypv who concerns about the pre-processing methods of the baselines, we address this issue by adding comparison experiments.

We further evaluate two the interpolation methods, the bilinear method and nearest-neighbor method,as the pre-processing of reconstruction from IUTs with four baseline algorithms (LCT, FK, RSD and USM). 
As shown in Figure 4, the performance of interpolators on the final reconstruction is really dependent on the quality of interpolated transient signals, and essentially on the scanning patterns. For scanning patterns with relatively uniform scanning points (small holes on the relay surface, such as Window 3 and 5), the higher-order bilinear interpolater can help significantly suppress background reconstruction artifacts. However, scanning patterns with skewed distributed scanning points (large missing areas on the relay surface, such as Window 7), the higher-order bilinear interpolater can even degrade the reconstruction quality due to the unreliable interpolated transients in large missing areas.
For all cases, our method still outperforms than other four baselines.

Meta-review: The paper received positive reviews overall that became stronger after the authors’ rebuttal. Reviewers highlighted the extensive evaluations (wGeV, 8wFw, WCZG), the novel analysis for unsupervised methods (8wFw), and the derivation of SURE loss (8wFw, WCZG). Although there were initial concerns regarding irregularity of scanning patterns, unclear presentations, missing experiments, and motivations, these were effectively addressed in the rebuttal, leading to stronger overall reviews.

After carefully reviewing the paper, the reviews, and the rebuttal, the AC agrees with the reviewers’ positive consensus, and hence recommends acceptance of the paper.

For the camera-ready revision, the authors should ensure to incorporate all discussions from the rebuttal into the main paper. Specific changes to be implemented are:

**1. Additional experiments**: Include the quantitative evaluation results for VSRnet and SURE-denoiser (8wFw), and the experiments and discussion for the effects of different relay patterns and more irregularly varied masks (WCZG, 8wFw, wGeV)

**2. Improved presentation**: Clarify all symbols and schematics (8wFw, cypv), update citation format (cypv) and hyperlinks (WCZG), and improve figure 2 (WCZG)
Decision: Accept (poster)