Deep Non-line-of-sight Imaging from Under-scanning Measurements
Authors: Yue Li
Yueyi Zhang
Juntian Ye
Feihu Xu
Zhiwei Xiong
Venue: NeurIPS 2023 poster (2023)
Keywords: Non-line-of-sight imaging
Transient Recovery
Volume Reconstruction
TL;DR: We propose a new approach to NLOS reconstruction with fewer scanning points, utilizing deep learning for high-quality and rapid inference.

Abstract: Active confocal non-line-of-sight (NLOS) imaging has successfully enabled seeing around corners relying on high-quality transient measurements. However, acquiring spatial-dense transient measurement is time-consuming, raising the question of how to reconstruct satisfactory results from under-scanning measurements (USM). The existing solutions, involving the traditional algorithms, however, are hindered by unsatisfactory results or long computing times. To this end, we propose the first deep-learning-based approach to NLOS imaging from USM. Our proposed end-to-end network is composed of two main components: the transient recovery network (TRN) and the volume reconstruction network (VRN). Specifically, TRN takes the under-scanning measurements as input, utilizes a multiple kernel feature extraction module and a multiple feature fusion module, and outputs sufficient-scanning measurements at the high-spatial resolution. Afterwards, VRN incorporates the linear physics prior of the light-path transport model and reconstructs the hidden volume representation. Besides, we introduce regularized constraints that enhance the perception of more local details while suppressing smoothing effects. The proposed method achieves superior performance on both synthetic data and public real-world data, as demonstrated by extensive experimental results with different under-scanning grids. Moreover, the proposed method delivers impressive robustness at an extremely low scanning grid (i.e., 8$\times$8) and offers high-speed inference (i.e., 50 times faster than the existing iterative solution).

Deep Non-line-of-sight Imaging from Under-scanning Measurements
Yue Li
Yueyi Zhang" Juntian Ye Feihu Xu Zhiwei Xiong
University of Science and Technology of China jt1418,,,,1111114mmmmmm ust.. edu. en {zhyuey, feihuxu, zwxi www...........c.eeeooo....... cn
Abstract
Active confocal non-line-of-sight (NLOS) imaging has successfully enabled seeing around corners relying on high-quality transient measurements. However, acquir- ing spatial-dense transient measurement is time-consuming, raising the question of how to reconstruct satisfactory results from under-scanning measurements (USM). The existing solutions, involving the traditional algorithms, however, are hindered by unsatisfactory results or long computing times. To this end, we propose the first deep-learning-bet based approach to NLOS imaging from USM. Our proposed end-to- end network is composed two main components: the transient recovery network (TRN) and the volume reconstruction network (VRN). Specifically, TRN takes the under-scanning measurements as input, utilizes a multiple kernel feature extraction module and multiple feature fusion module, and outputs sufficient-scanning measurements at the high-spatial resolution. Afterward, VRN incorporates the linear physics prior of the light-path transport model and reconstructs the hidden volume representation. Besides, we introduce regularized constraints that enhance the perception of more local details while suppressing smoothing effects. The proposed method achieves superior performance on both synthetic data and public real-world data, as demonstrated by extensive experimental results with different under-scanning grids. Moreover, the proposed method delivers impressive robust- ness an extremely low scanning grid (i.e., 8x8) and offers high-speed inference (i.e., 50 times faster than the existing iterative solution). The code is available at https://githin httpau/.cc//pppi//ool///eeeeeee scanning_NLOS,
1 Introduction
off-line-of-sight (NLOS) imaging technology permits the detection of objects located behind occluders, and has numerous real-world applications, such autonomous driving, remote sensing, and disaster rescue [ 2 3
7 8 9]. The widely- known time-of-flight confocal imaging system, composed of pulse laser, Single Photon Avalanche Diode (SPAD), and a Time-Correlated Single Photon Counting sensor (TCSPC), is favored for its ability to capture high-quality transient measurements 10] As illustrated Fig. [ the picosecond pulse laser illuminates the relay wall at certain location. The light scatters at the target object behind the wall and then re-scatters to the relay wall, from which the photons are captured by the SPAD along with TCSPC, forming a histogram. The transient measurement records the histograms of all raster-scanned locations and can be used to derive the hidden object volume by the well-studied reconstruction algorithms such those developed in [5] 12 13 [4] 15). However, the total acquisition time, which is directly proportional to the number of scanning points and the exposure time of each scanning point, hinders the real-time NLOS imaging process.
4
5
6
"ooresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Hidden Object
Wall
NLOS
System
Visible Wall
Scanning Points
Histogram
Figure I: Overview of the active confocal NLOS imaging. The red circles denote a sufficient-scaniing grid the high-spatial resolution, and the blue triangles denote an under-scanning grid.
The consuminn dilemma raises the question of how to reconstruct the hidden volume from the under-scanning measurement (USM) without compromising the imaging quality. As shown in Fig. [ the spherical waves are scattered back by the illuminated object, which indicates that the temporal histogram from each scanning point contains all the geometry information of the hidden object. As a result, it is feasible to recover the high-sspatial-rsoouure
hidden volume from USM. Previous studies [16 [17] explored NLOS imaging from USM using circular scanning the compressed sensing algorithm [8]. These solutions either produce unsatisfactory outcomes or require lengthy numerical iterative computations, leaving large room for improvement.
In this paper, we propose a new approach to NLOS reconstruction from USM, pioneering the utilization of deep learning achieve superior quality and swift inference. Specifically, our proposed network consists of two main components: the transient recovery network (TRN) and the volume reconstruction network (VRN). Given the under-scanning transient measurement, multiple kernel feature extraction module in TRN generates the features at different kernel scales to obtain the pyramid receptive field. Then, the features concatenated and aggregated by multiple feature fusion module to produce the high-spattal-resolutiie subsequently. The VRN consists of feature extraction module, feature propagation module, volume refinement module, and a projection reconstruction module. Inspired by [19 5 6 we integrate the linear physics prior 212 into the feature propagation module. Additionally, we introduce the local similarity and total variation losses to regularize our network. The local similarity loss assists with perceiving more local details, and the total variation loss aids in suppressing smoothing effects. Our proposed network has an end-to-end architecture that necessitates only one forward pass for reconstruction. The high-fidelity reconstruction an extremely low scanning grid (i.e., 8x8) and the high-speed inference (i.e., 50 times faster than the existing iterative solution) demonstrate the superiority of our proposed method. The contributions are summarized as follows:
transient measurement which undergoes VRN
We study the challenging task of NLOS imaging from USM and develop and-to-end network for high-quality and fast inference, which the first its kind.
We exploit the multiple kernel mechanism the transient recovery network to adapt transient measurements with different under-scanning grids.
Our proposed method achieves superior performance both synthetic data and public real-world data, as demonstrated by extensive experimental results.
Related Work
2.1 NLOS Imaging Algorithms
In recent years, transient-based NLOS imaging has garnered significant attention in the research community, Despite the apparent progress made by direct reconstruction methods [2 B3] 14
15 20
23] these methods remain sensitive to exposure noise, data quality, and variations
regions. Concurrently, iterative methods 2 24 25 27 can effectively reduce imaging noise but suffer considerably from lengthy computation time. In contrast, deep-learning-ban techniques 28 5 offer a balance between the imaging quality and inference time, making them an attractive proposition for NLOS reconstruction. To this end, we approach NLOS reconstruction

---

from a deep-learning perspective, leveraging its advantages of efficient inference and robustness to noisy data.
2.2 NLOS Imaging From USM
The experiments conducted in 12 indicated that the exposure time per scanning point is more crucial than the number of scanning points for imaging quality. Subsampling the scanning points can be a viable trade-off for reducing the total acquisition time. CLNLO [16] advocated for circular scanning methodology which improved the transient capturing efficiency, but the corresponding reconstruction quality left much to be desired. CSA [7] investigated a compressed sensing algorithm called SPITAL [18] for NL.OS imaging from USM at a grid shape and demonstrated impressive results at low scanning ratio, however, its iterative computation required an extended time. In contrast, our proposed method leverages deep-learning technology for NLOS imaging from USM, and necessitates single forward pass for inference. As concurrent work, Wang et al. [30] utilized the learning-basee method and focused on recovering the high-spatial-resolle resolution transient measurement, rather than NLOS reconstruction from USM.
3 Problem Formulation
The travel time of the direct photons between the visible wall and the NLOS imaging system is disregarded for simplicity. The transient measurement, which solely describes the indirect photon bounces, is constructed via a 3D spatial-tempont
(2D spatial and 1D temporal) representation. Similar to the previous approaches [12 16 22,, we assume that light scatters just once at the hidden object in an isotropic manner, without any occlusions in the object. Given the illuminated point at = 0, the object o and the scanning point s which located at the same position as the illuminated point, the active confocal NLOS imaging observation model can be expressed as follows:
/|] p(o)
(2 s ol e.e. 1) da,
r(s, ()
o|1
where T denotes the transient measurement, Ω denotes the spherical waves of the scattered light, P represents the volume of the albedo of the hidden object surface, 8 models the light propagation function, |ss - olI is the quartic radiance falloff of light transport between the relay wall and the object, and is the speed of the light. As discussed in [6]. Eq. can be derived to a straightforward 3D> convolution form via variables changing. After discretized, the formation model can be regarded as linear model:
p P=A'TT
(2)
where A refers to the light transport matrix, the vectorized E 1h the vectorized E Rh, th h2 and Wy are the number of the scanning points along the x and axis, n, and n, are the length of the discretized histogram bins along time t axis and depth bins along depth axis. The determination of p represents an inverse problem, involving 3D deconvolution procedures.
For NLOS imaging from USM, define the under-scanning transient measurement Tu E "n and the corresponding hidden volume Pa E th wy,., the suffcient-scannnngg transient measurement
Ts E H. R.H.W and the corresponding hidden volume Ps E HH, where H₂ > h and W, > wy. The NLOS imaging from USM is transformed into the following optimization problem:
0r, By arg min (C(T,, 7s) + C(phy Px) + AΦ(00, 0v)),
07.0v
where -ςद and pt are the predicted high-spatial-resolunee the algorithms 0y and By respectively, L represents the loss function between the generated and ground-trum -ruth one, @ is the regularization term, and X is a parameter to weigh the regularization term.
transient data and hidden volume through
4 Proposed Method
In this section, we present an end-to-end deep network for NLOS imaging from USM. The network takes in under-scanning transient measurement and finally outputs the hidden volume and the intensity image at a high-spatial resolution. The overall network comprises two stages: the Transient Recovery Network (TRN) and the Volume Reconstruction Network (VRN). In the first stage, the TRN recovers
3

---

(a)
Transient Recovery
Volume Reconstruction
Network
Network
Under scanning
Sufficient scanning
Transient M Measurement
Transient Measurement
Intensity Image
(b)
(c)
K.17
Redure Conoil
X-9
K55
Fedtuee
Volumz
Refwenett
Projection Recarednutioo
K3
Feature
In
Transform
Out
Multiple Kemal
Multiple
Feature Extraction
Feature Fusion
Padding
Interpolation
XX:
size
(d)
(e)
Kernel
Padding points
Scanning points
Figure 2: (a) The pipeline of the proposed network. The network consists of transient recovery network (TRN) and volume reconstruction network (VRN). (b) An overview of TRN. The blue rectangular blocks denote 3D residual blocks with corresponding kernel sizes. The orange and yellow rectangular blocks represent the 3D convolutions and 3D residual blocks, respectively, (c) An overview of VRN. (d) A diagram for the multiple kernel feature extraction. (e) A diagram for the regularized loss functions. The points the objects always have similar values with the neighbors, while points on the edges have different values from the outside.
the under-scanning transient data to the desired high-spatial resolution. the second stage, the VRN reconstructs the hidden volume and then generates the intensity image, illustrated in Fig. (a).
4.1 Transient Recovery Network
Histograms captured from distinct scanning points contain the complete hidden geometry infor- mation. However, each histogram has unique characteristics due to the differences in traveling time. Consequently, the data from even the adjacent scanning points can not be identical, but similar. To explore this information and restore the transient measurement at the high-spatial resolution, design the transient recovery network (TRN) to recover suffiiient-scannigg transient measurement.
TRN consists of a Multiple Kernel Feature Extraction module (MKFE) and Multiple Feature Fusion module (MFF), as illustrated in Fig. b). The MKFE module comprises four residual blocks with different convolution kernels and interpolation operator. Each residual block consists of two 3D convolutions, two Leaky ReLU activation functions, and one residual connection. The kernel scales in the residual blocks of the MKFE are arranged from large to small to perceive the entire geometry, therefore providing a sizeable receptive field. As illustrated in Fig. d). the scanning points in the padded scanning grid are sparse and distant from each other. To exploit the fcature connection between the scanning points one convolution operation directly, the size of the convolution kernel set to cover the farthest scanning point distance. The large kernel perceives the distant regions, and the small kernel focuses on the local areas. Various under-scanning grids require different appropriate kernel sizes. For more general architecture, for TRN to adapt the different under- scaaning transient measurements. The MFF module alternately involves three 3D convolutions and three 3D residual blocks with fixed kernels.
further implement multiple kernel mechanism
Given the under-scanning transient measurement Tus we first pad the under-scanning transient measurement to reach the desired high-spatial resolution. All histograms, except the initial position,

---

are zeroed. Next, we pass the padded transient data through four residual blocks to obtain fine features. Then, we utilize the nearest interpolation for the imput under-scanning transient measurement to get coarse features at the desired resolution, which facilitates stable training and fast convergence. Afterward, the coarse and fine features are concatenated along the channel dimension. Then the combined features are supplied to the fusion recovery decoder to generate the sufficient-scanning transient measurement Ts.
4.2 Volume Reconstruction Network
For the hidden volume reconstruction network (VRN), we borrow the idea from the previous works [5] 3 which incorporated the physics prior feature propagation. Inspired by LFE (5), VRN is composed of a feature extraction module, a feature transform module, a volume refinement module, and a projection reconstruction module, as shown in Fig. ee.. Different from LFE (5), utilize the physical model-based approach called LCT 12 to build the feature transform module. LCT [12] design volume
adopts deconvolution operation and conducts efficient computation. Moreover, refinement module to enhance the 3D features for high fidelity. The module is constructed with three 3D convolutions and three interlaced residual layers. Each of these residual layers is composed of two 3D convolutions, followed by ReLU operation and a residual connection. This module has been purposefully designed to proficiently accentuate essential features while amplifying the 3D volume. The objective is to enhance fidelity, thereby avoiding direct projection of the 3D volume onto 2D planes which could potentially result in losing crucial inherent features.
Given the sufficient-ccanning transient measurement Ts which is recovered by TRN, we first extract the intrinsic spatial-temporal features by the feature extraction module. Then the extracted features are supplied to the feature transform module, where the 3D spatial-tempon into 3D spatial features. Afterward, the 3D spatial features proceed the volume refinement module to produce the reconstructed 3D volume p. For further optimizing the output intensity images, the 3D volume representations are mapped the 2D domain using maximum intensity projection along the depth axis, and then the projected 2D features are convolved to generate the final intensity images I.
features are transformed
4.3 Regularized Training
Our proposed network is trained in an end-to-end supervised manner. The overall loss function consists of four components: the measurement loss Lenas the intensity loss Lints the local similarity loss Lts and the total variation loss L.v. The total loss is formulated as follows:
Ltotai Lmea Å Lint X2 Lis As Ltv
where \1, Ag. and λ are the typer-parameters that weigh the corresponding loss function.
To achieve end-to-end training, the LI norms for the outputs from TRN and VRN are computed:
Lmea ||T, - Lint = -
(5)
where T, and Tt correspond to the predicted sufficient- sufficient-scanning and the ground-truth transient measurements, I and Ist refer to the predicted and ground-truth intensity images of the objects hidden from view. The efficacy of Emea is showcased in Sec 5.5
Under conditions of fewer scanning points, two problems are encountered in deep learning methodol- ogy. Firstly, the reduction of transient measurements causes difficulties in volume reconstruction, making small or distant objects mostly unrecoverable or partially recovered. Secondly, effective signal reduction or signal sparsity increase leads to network overfitting to the simulation data, thereby impeding accurate real-world testing. To overcome these challenges, we incorporate regularized constraints during the training process. Firstly, we leverage the local similarity of the natural volume since the points on the hidden object are analogous to their neighbors. We employ a Gaussian window to weigh the neighbors of the points, as [31
32 25 The local similarity loss is formulated as follows:
Lis =. E, Σ. |p v. p(x, y, k,) WIl1,
(6)
where p(z, y. 2) indicates the volume at position (z, v,22, p(z, y, 2, k) represents the volume block centered at (Ir, y, =) with size k. W refers to the Gaussian window with size As shown in Fig. 2 e), when the local similarity loss function acts on the points located at the objects, it brings out more local details. On the contrary, when it acts on the boundary points, blurs and smoothens the edges. Secondly, to suppress background noise and sharpen the edge boundary, we introduce the 3D total
5

---

variation loss function, formulated as follows:
Lr = -x E, >. (1pp(1 + L, v, p(z. - || 1, p(a, v, v, z)Il1
(7)
+ +11p(..
p( y. v)))).
The effectiveness of the regularized loss functions Lis and Leo is demonstrated in 5.5
Experiments
5.1 Datesets and Evaluation Metrics
We evaluate our method on both synthetic and real-world datasets. For the synthetic dataset, we simulate the data using a transient renderer provided by [3], rendering 277 motorbikes from the ShapeNet dataset [33]. The synthetic dataset is composed of 2712 training samples and 291 testing samples. The scanning grid is with a size of 128x 128 and the time resolution is 512 with a bin width of 33ps. For the real-world scenes, we adopt the real-world dataset provided by [14] which contains diverse objects. We conduct preprocessing to discard the direct bounce data and resize the data to 128x x28x512. The total acquisition time of the real-world transient data is about 45min. We directly extract the scanning points from the initial transient data to simulate the under-scanning process. 32x32, 16x 16, and 8x8 scanning grid represents the total acquisition time of 169s, 42s, and 11s respectively. NLOS imaging from USM is able to expedite the data capturing by orders of magnitude. Besides, we also adopt the real-world data from [29 to further validate the generalization capability of our model.
For the intensity recovery results, we evaluate the performance with three metrics: Peak Signal- to-Noise Ratio (PSNR), Structural SIMilarity (SSIM), and ACCuracy (ACC), We utilize ACC to determine the foreground and background classification accuracy.
5.2 Implementation and Baseline Details
The proposed network is implemented in PyTorch. The optimizer AdamW 34 with a learning rate of x 10-44 and weight decay of xx10 10-4, The models are trained on the synthetic dataset for 50 epochs with a batch size 4, and directly tested on the real- world dataset, setting the high-spatial resolution to 128 x 128. For the loss function, the hyperparameters \1, X2, and A3 is 1, 1 10 -5, and 1x All the experiments are conducted on a workstation equipped with 4 NVIDIA GeForce A100 GPUs
Our baseline methods include four traditional methods (FBP three bcep-learning-basee methods (UNet (28]. NeTF 335 LFE
LCT FK 14 RSD [55
5 and one iterative method (CSA (77.. All methods, except CSA [17 perform non-iterative reconstruction. We find the performance of these non-iterative methods, which upsample the input before reconstruction, is better than that of the methods which reconstruct the volume before upsampling the output. Thus, for these non- non-iterative baselines, we interpolate the input under-scanning data to the desired high-spatial resolution using the nearest interpolation approach, as [17. CSA [17 and NeTF (35) are fed with the padded transient data (at the same spatial resolution as the desired transient data) and under-scanning transient data, respectively.
5.3 Experimental Results
5.3.1 Transient Recovery
To validate the effectiveness of TRN, we compare the qualitative histograms of the recovered, direct interpolated, and raw transient measurement which is from the real-world scene (Teaser [14)) at certain scanning points. We select the transient data at three low scanning girds: 32x32, 16x 16, and 8.8. As illustrated in Fig. 3 our predicted results show greater similarity the ground truth terms of curve trend and peak size. Specifically, under the 8x8 scanning condition, the interpolated histogram consistently generates larger outliers, while our method delivers robust performance, indicating the superiority of our proposed TRN and paving the way for subsequent volume reconstruction.

---

interp
Inberp
pred
43
12
Interp
131 pred
pred
pred
32x
16x
8x
Figure 3: Comparisons of the histograms from the recovered and the interpolated transient surement. The rows show the histograms from different recovered transient measurements with the same scanning position. The green, blue, and red lines represent the interpolated, recovered, and ground-truth histograms. The data from the real-world scene (Teaser in [14)).
Table Quantitative comparisons of intensity images from different methods under various scanning grids on the synthetic test set. The output spatial resolution is 128x 128. The best is in bold.
Points
Metries
FBP [1]
LCT [2]
FK [14]
RSD [5]
UNet 28
NeTF
LFE [5] CSA [17]
Ours
PSNR(dB)
21.12
22.07
24.59
25.44
26.71
22.85
27.94
23.22
28.64
ACC(%)
0.52
56.54
15.48
69.09
73.71
68.25
73.05
30.43
74.35
32x32
SSIM
0,2512
0.6314
0.4152
0.7847
0.8808
0.8691
0.8867
0.8223
0.8975
PSNR(BB)
15.69
20.27
18.02
23.53
26.64
19.14
27.04
22.62
28.00
ACC(%)
0.05
20.16
5.99
60.86
73.79
62.38
72.40
25.83
74.19
16x
SSIM
0.0851
0.4335
0.2121
0.7102
0.8827
0.7977
0.8620
0.8064
0.8929
PSNR(dB)
12.31
15.85
17.20
19.77
26.28
17.03
26.10
21.46
27.30
ACC(%)
0.031
2.64
32.54
73.56
54.87
72.17
15.56
73.59
8x8
SSIM
0.0441
0.1981
0.2147
0.5131
0.8775
0.7639
0.8218
0.7976
0,8789
32x32
16x16
8-8
LCT
FK
RSD
NeTF
LFE
CSA
Ours
GT
Figure 4: Qualitative comparisons of intensity images from different methods under various scanning grids the synthetic test The final spatial resolution is 128x 128.
5.3.2 Volume Reconstruction
Synthetic Results. The quantitative results on the synthetic test set are reported Table As number of scanning points reduces, the traditional methods' metrics decrease significantly, while the deep-learning-basse methods except NeTF 35 perform more robustly. Moreover, our method outperforms the deep methods under different scanning grids in terms of all metrics. The qualitative
7

---

32x32
16x16
8x8
32x32
16x16
8x8
LCT
RSD
NETF
CSA
Ouus
GT
Figure 5: Real-world scenes captured by
through three different scanning grids are provided.
32x32
16x16
8x8
CU ENP RRF
32x32
CAP
16x16
8x8
LCT
FK
RSD
NETF
LIE
CSA
Ours
GT
Figure 6: Real-world scenes captured by through three different scanning grids are provided.

---

Table 2:
The inference time and memory of different models. Note that only CSA and our method are specifically designed for NLOS imaging from USM
Method
FBP [1]
LCT [2
FK [4]
RSD [3
UNet 28
NcTF [5
LFE [5]
CSA 17
Ours
Time (8)
0.042
0.034
0.061
0.038
0.162
2h-24h
0.030
20
0.420
Memory (M)
6026
6016
8056
10344
5642
2G-23G
4692
5306
7130
results are presented in Fig. 4 The traditional methods fail to reconstruct the motorbike under low scanning points, particularly under the 8x8 scanning grid, Alternatively, LFE [5] and CSA [17] generate the overall structure with textureless information, whereas our method captures more details even with few scanning points. See more qualitative results in the supplement.
Real-world Results. In Fig. and Fig. 6 we provide qualitative results in the real-world scenes. We exclude FBP [1] and UNet [28] from the results as they cannot perform well under under-scanning conditions. Specifically, under the 32x32 and 16x 16 scanning grids, the traditional methods, LCT 12]. FK [4], and RSD [3], support reconstruction with heavy noise. Meanwhile, NeTT (35] fails under the 16x 16 scanning grid. LFE (5) provides results with lower noise than traditional methods but fails to capture several details such as the tail of the Dragon and the statue near the bookshelf in Fig. 5 By contrast, CSA [17 and our proposed method produce complete and clear results, with our method generally capturing more details and producing less noise, such as the horns of the dragon and the torso of the statue. Under the 8x8 scanning grid, all baselines degrade significantly. Nonetheless, both CSA and our method achieve satisfactory results. Our method performs better than CSA [77 in most scenes and achieves comparable results CSA about the bookshelf. It is worth noting that our proposed method offers faster inference time than CSA, which will be discussed in Sec More qualitative results of real-world transient measurements, which are captured by different imaging systems and under different total acquisition time, are reported in the supplement.
5.4 Inference Time
We conduct experiments to evaluate the inference time and memory requirements of various methods. Specifically, we utilize an NVIDIA GeForce A A100 GPU for all testing experiments. Traditional methods including FBP [T], LCT [2], FK [14, and RSD (15) are reimplemented using PyTorch, To accommodate CSA [7], which was originally implemented in Matlab the CPU, we modify the code to enable computations on the GPU, resulting in a significant acceleration. Table 2 presents the comprehensive results.
The traditional methods and LFE achieve a real-time level of performance. NeTF each measurement and requires a huge computational time. However, these methods can not be applied to NLOS reconstruction from USM well. CSA [17.. the existing solution for NLOS imaging from USM, suffers from iterative computing times and is not well-suited for practical use. By contrast, our method leverages recent advancements deep learning, enabling to perform a single forward pass that takes only 420ms (370ms for transient recovery and 50ms for volume reconstruction). This is nearly 50 times faster than CSA [77. Moreover, our method does not require iterative computing. which is significant advantage over CSA 117. Additionally, our method has reasonable inference memory requirement, feasible for deployment on portable devices to explore real-world scenarios.
35 renders
5.5 Ablation Study
In this section, we assess the effectiveness of the proposed TRN, VRN, and the regularized loss items: Lis and Ltv. respectively. Additionally, we investigate the effectiveness of the kernel size of MKFE in TRN. All the ablation experiments are conducted under the 16x 16 scanning grid condition. We quantitatively evaluate the results on a synthetic test set, while the qualitative results are tested on real-world data.
Table 3 lists the ablation results of the proposed TRN, VRN, and the regularized loss items. Initially, we combine the proposed TRN with LFE (5), and the metrics surpass the original LFE [ which demonstrates the effectiveness of TRN. Subsequently, we combine the interpolation operator and the proposed VRN. The metrics surpass the original LFE again, which demonstrates the effectiveness of VRN, When we integrate the proposed TRN and VRN, the metries improve notably. Finally, we

---

Table Ablation study for TRN, VRN, and regu
larized loss items. TR and VR denote the transient Table 4: Ablation study for the kernel size in mul-
tiple kernel feature extraction module of TRN.
recovery and volume reconstruction components.
Kernel Size
PSNR SSIM ACC
TR
VR
Lla,t0
PSNR
SSIM ACC
3
27.80 0.8744 73.98
Interp LFE [S]
X
27.04 0.8620 72.40
5
27.89 0.8791 73.87
TRN
LFE (5)
27.79
0.8692 72.75
9
27.92 0.8825 73,91
(b)
Interp VRN
X
27.71
0.8646 72.85
17
27.95
0.8696 73.85
(c)
TRN VRN
28.00
0.8899 73.44
3,5,9,17
28.00
0.8929 74.19
(d) TRN
VRN
28.00
0.8929 74.19
(a)
(b)
(c)
(c)
(a)
(b)
(c)
(c)
Figure 7: Reconstructed intensity images of the models, whose indexes correspond to Table 3
introduce the regularized loss items: Lis and Civ. As a result, the SSIM and ACC metrics demonstrate further improvement. Qualitative results corresponding Table 3 can be seen in Fig. 7
Table presents the ablation results of various kernel sizes used in MKFE, can observed that the PSNR metric tends to increase along with the kernel size, whereas the SSIM and ACC metrics do
not mirror that trend. This can be attributed the fact that the larger kernel could recognize distant regions, while the smaller one could focus on local areas. Notably, the multiple-kernel mechanism results in the best performance across all metrics.
Conclusions and Limitations
Conclusions. In this paper, we study the challenging task of non-line-of-sigh imaging from the under-scanning measurement the deep learning perspective. The elaborate network, which is composed of two components: transient recovery network and volume reconstruction network, proposed to recover the transient measurement and reconstruct the hidden volume successfully. Further, the regularized constraints are exploited to address overfitting and enhance more local details while suppressing smoothing. Notably, our method runs 50 times faster than the existing iterative solution. We believe that method is a significant step forward in the field of non-line-of-sight imaging and has the potential to unlock new possibilities for future research.
Limitations. Several limitations exist in our proposed method, which serve as subjects for future work. Firstly, our forward model is constrained generalizing the forward model. Secondly, our proposed method is fully supervised, which may restrain the generalization ability for real-world data. Designing an unsupervised training method for non-line-of-sightt imaging is another future work.
linear confocal system, our future work involves
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grants 62131003 and 61 61901435, and Innovation Program for Quantum Science and Technology under Grant 2021ZD0300300.
References
[1] Nicolas Scheiner, Florian Kraus, Fangyin Wei, Buu Phan, Fahim Mannan, Appenrodt, Werner Ritter, Jurgen Dickmann. Klaus Dietmayer, Bernhard Sick, et al. Seeing around street corners: Non-line-of -fi-shhht detection and tracking in-the-wit wild using doppler radar. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2068-2077, 2020.
[2] Cheng Wu, Jianjiang Liu, Xin Huang. Zheng- Ping Li, Chao Yu, Tun-Tian Ye, Jun Zhang, Qiang Zhang Xiankang Dou, Vivek K Goyal, al. Non mon-line-od of- imaging over 1.43 km. Proceedings of the National Academy of Sciences, (18(10):20224488111 2021.
10

---

[3] Fangzhou Mu, Sicheng Mo, Jiayong Peng, Xiaochun Liu, Ji Hyun Nam. Siddeshwar Raghavan, Andreas Velten, and Yin Li. Physics the rescue: Deep non-line-offsight reconstruction for high-speed imaging. IEEE Transactions on Puttern Analysis and Machine Intelligence, pages 1-12, 2022.
[4] Mariko Isogawa, Yuan, Matthew O'Toole, and Kris M Kitani. Optical non- oo-line-of-siiht physics-based 3d human pose estimation. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7013-7022, 2020.
[5] Wenzheng Chen, Fangyin Wei, Kiriakos N Kutulakos, Szymon Rusinkiewicz, and Felix Heide. Learned feature embeddings for non-line-of-sight imaging and recognition. ACM Transactions Graphics, 39(6):1-88, 2020.
[6] Ping Liu, Yanhua Yu, Zhengqing Pan, Xingyue Peng. Ruiqian Li, Yuehan Wang, Jingyi Yu, and Shiying Li. Hiddenpose: Non-line-of-sight 3d human pose estimation. In IEEE International Conference on Computational Photography, pages 1-12, 2022.
[7] David B Lindell, Gordon Wetzstein, and Vladlen Koltun. Acoustic non-line-of-sight imaging. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6780-6789, 2019.
[8] Sven Bauer, Robin Streiter, and Gerd Wanielik. Non-line-of sight mitigation for reliable urban gnss vehicle localization using a particle filter. In International Conference on Information Fusion, pages 1671, 2015.
[9] Julian Iseringhausen and Matthias B Hullin. Non-line-of-sile reconstruction using efficient transient rendering. ACM Transactions Graphics, 39(1):1-14, 2020.
[10] Daniele Faccio, Andreas Velten, and Gordon Wetzstein. Non-line-of-sight imaging. Nature Reviews Physics, 2(6):318 327, 2020.
[11] Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G Bawendi, and Ramesh Raskar. Recovering three-dimensional shape around a comer using ultrafast time-of-flihtt imaging. Nature Communications, 3(1):1-8, 2012.
[12] Matthew O'Toole, David B Lindell, and Gordon Wezzstein. Confocal non-line-of-sight imaging based on the light-cone transform. Nature, 555(7696):338-341, 2018.
[13] Felix Heide, Matthew O' O'Toole, Kai Zang, David B I. Lindell, Steven Diamond, and Gordon Wetzstein. Non-line-of-sight imaging with partial occluders and surface normals. ACM Transactions on Graphics 38(3):1-10, 2019.
[14] David B Lindell, Gordon Wetzstein, and Matthew O O'Toole. Wave-based non-line non-line-of off-lii--of--ihht imaging using fast fk migration. ACM Transactions Graphics, 38(4):1-13, 2019.
[15] Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jarabo, Diego Gutierrez, and Andreas Velten. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature, 572(7771):620 623, 2019.
[16] Mariko Isogawa, Dorian Chan, Ye Yuan, Kris Kitani, and Matthew O'Toole. Efficient non-line-of-sighe imaging from transient sinograms. In European Conference on Computer Vision, pages 193-208 2020.
[17] Jun-Tian Ye, Xin Huang, Theng-Ping Li, and Feihu Xu. Compressed sensing for active non-line-of-sight imaging. Optics Express, 1929):749-1176 2021.
[18] Zachary Harmany, Roummel F Marcia, and Rebecca M Willett, This is spiral-tap: Sparse poisson intensity reconstruction shorrithms-thore and practice. IEEE Transactions on Image Processing. 21(3):0884-106, 2011.
[19] Dayu Zhu and Wenshan Cai. Fast non-line-of-sight imaging with two-step deep remapping. ACS Photonics, 26))2204- 2022.
[20] Deyang Jiang, Xiaochun Liu, Jianwen Luo, Zhengpeng Liso, Andreas Velten, and Xin Lou. Ring and radius sampling based phasor field diffraction algorithm for non- reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1 (4(11):7841-788 2021.
[21] Shumian Xin. Sotiris Nousias, Kiriakos N Kutulakos, Aswin C Sankaranarayanan, Srinivasa G Narasimhan, and Ioannis Gkioulekas. A theory of fermat paths for non-line-ole tine-of-sight sight shape reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6800-6800 2019.
11

---

[22] Sean I Young, David B Lindell, Bernd Girod, David Taubman, and Gordon Wetzstein. Non-line-of-sighee surface reconstruction using the directional light-cone transform. IEEE/CVF Conference on computer vision and puttern recognition, pages 1407-1416, 2020.
[23] Victor Arellano, Diego Gutierrez. and Adrian Jarabo. Fast kack-projection for non-line of sight reconstruc- tion. Optics Express, 25(10):11574-11)
2017.
[24] Byeongjoo Ahn, Akshat Dave, Ashok Veeraraghavan, loannis Gkioulekas, and Aswin C Sankaranarayanam. Convolutional approximations the general non- ofne-of-sight imaging operator. In IEEE/CVF Interna- tional Conference on Computer Vision, pages 7889-7899, 2019.
[25] Jianjaang Liu, Yijun Zhou, Xin Huang, Phng Li, Feihu Xu. Photon-efficient non-line-of-sightt imaging. IEEE Transactions on Computational Imaging, 8:639-650, 2022.
[26] Xintong Liu, Jianyu Wang. Zhupeng Li, Zuoqiang Shi, Xing Fu, and Lingyun Qiu. Non-line-ot -fne-of-sight reconstruction with signal- object collaborative regularization. Light: Science & Applications, 10(1):198, 2021.
[27] Zhupeng Li, Xintong Liu, Jianyu Wang, Zuoqiang Shi, Lingyun Qiu, and Xing Fu, Fast non-line-of-sight imaging based on first photon event stamping. Optics Letters, 1728):1928 1931, 2022.
[28] Javier Grau Chopite. Matthias B Hullin, Michael Wand, and Julian Iseringhausen. Deep non-line-of-sight reconstruction. In JEEE/CVF Conference Computer Vision and Pattern Recognition, pages 960-969, 2020.
[29] Yue Li, Jiayong Peng. Juntian Ye. Yueyi Zhang, Feihu Xu, and Zhiwei Xiong. Nlost: Non-line-on of imaging with transformer. In JEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13313-13322 2023.
[30] Jianyu Wang, Xintong Liu, Leping Xiao, Zuoqiang Shi, Lingyun Qiu, and Xing Fu. Non-line-of-sight imaging with signal superresolution network. In /EEE/CVF Conference on Computer Vision and Pottern Recognition, pages 17420-17429, 2023.
[31] Weisheng Dong, Lei Zhang, Guangming Shi, and Xiaolin Wu. Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Transactions on Image Processing, 40(7):1838-1855) 2011.
[32] Kan Chang and Baoxin Li. Joint modeling and reconstruction of a compressively-sensedd set of correlated images. Journal of Visual Communication and Image Representation, 33:286-300, 2015.
[33] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint 11200025,. 2015.
[34] Ilya Loshchilov and Frank Hutter. 1711v:1711.05101, 2017.
Decoupled weight decay regularization. arXiv preprint
[35] Siyuan Shen, Zi Wang. Ping Liu, Zhengqing Pan, Ruiqian Li, Tian Gao, Shiying Li, and Jingyi Yu. Non-line-of-sieee imaging via neural transient tields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(7):2257-2268, 2021.
12


Reviews:
[{"review_id": "Q2IvA37NIB", "reviewer": "Reviewer_r4bJ", "summary": "This paper introduces a deep-learning-based method for non-line-of-sight (NLOS) imaging from under-scanning measurements (USM). It comprises two main components: the transient recovery network (TRN) and the volume reconstruction network (VRN). The TRN extracts features and produces high-resolution measurements, while the VRN reconstructs the hidden volume. The method achieves superior performance on synthetic and real-world data, displaying impressive robustness even at an 8x8 scanning grid and providing high-speed inference, 50 times faster than existing solutions.", "questions": "Please see the weakness part", "limitations": "NA", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The paper deals with under-scanning measurements and develops an end-to-end network for high-quality and fast inference.\n(2) The experiments validate the claim. The experimental results show improvements in both qualitative (Fig. 4, and Fig. 5) and quantitative  (Tab. 1) aspects, and fast inference (Tab. 2). The supplementary material contains further qualitative results, providing additional evidence of the effectiveness of the proposed method.\n(3) Ablation is quite comprehensive.", "weaknesses": "(1) The paper does not mention how noise is modeled.\n(2) It is better to show the tradeoff between the number of samplings and noisy level for the final reconstruction quality.\nIn the paper, we see that 32x32 sampling is always better than 8x8.\nBut given the same total acquisition time, 32x32 might be worse than 8x8 under some noise level. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are highly encouraged by the positive recommendation and comments from the reviewer. The main concerns are addressed below.\n\n**Q1:** *The paper does not mention how noise is modeled.*\\\n**Reply:**  For the noise modeling of the NLOS simulation, the formulation is as follows:\\\n$$ \\hat{\\tau}(\\mathbf{s},t) \\sim \\text { Poisson }(\\tau(\\mathbf{s},t) + B), $$\\\nwhere $\\tau$ denotes the raw transient measurement, $\\hat{\\tau}$ means the resulting transient measurement, and $B$ represents the background photons and dark counts. For the data synthesis, we exclude cross-talk and afterpulsing effects, as [1]. The implementation of our noise model adheres closely to the methodology outlined in [2].\\\nWe sincerely appreciate your kind reminder and we will explain the noise model in the manuscript.\n\n[1] Wenzheng Chen, Fangyin Wei, Kiriakos N Kutulakos, Szymon Rusinkiewicz, and Felix Heide. Learned feature embeddings for non-line-of-sight imaging and recognition. ACM Transactions on Graphics, 39(6):1–18, 2020.\\\n[2] Quercus Hernandez, Diego Gutierrez, and Adrian Jarabo. A computational model of a single-photon\navalanche diode sensor for transient imaging. arXiv preprint arXiv:1703.02635, 2017.\n\n**Q2:** *Tradeoff between the number of samplings and noisy level for the final reconstruction quality.*\\\n**Reply:** As suggested, we build a confocal NLOS imaging system (Details in the global rebuttal response) and execute experiments across varying noise levels. We have captured the real-world transient measurements with different acquisition time per point and scanning grids while maintaining the total acquisition time constant. \n\nIn Fig. 1 (left part of the global rebuttal  PDF), we present the results obtained with a total acquisition time of 25.6s and 102.4s, respectively. Through these experiments, two crucial insights have emerged: 1) Under the same total acquisition time, the results obtained from the 16 * 16 scanning grid are better than those from the 32 * 32 scanning grid, revealing that longer exposure times for fewer scanning points yield better imaging quality. 2) However, when comparing the results from 32 * 32 and 8 * 8 scanning grids, this phenomenon is observed to be less pronounced under extremely low scanning grids. \n\nIn light of these insights, we infer that for our specific imaging system, employing a 16*16 scanning grid strikes a balance between the reconstruction quality and the total acquisition time (yet this would vary for different systems). Furthermore, it is noteworthy that our manuscript introduces a universally applicable technique for NLOS imaging utilizing under-scanning measurements. This method is agnostic to the tradeoff scanning grid of any given imaging system, reaffirming its broad applicability."}, {"review_id": "ZAZZ1heWak", "reviewer": "Reviewer_iKMu", "summary": "The paper proposes a deep learning based approach to non-line-of-sight (NLOS) imaging from under-scanning measurements (USM). This is achieved by first using a transient recovery network (TRN) that increases the spatial resolution of the input, and a volume reconstruction network (VRN) that uses linear physics prior for volume reconstruction and intensity image generation. The method uses local similarity and total variation loss to further improve the reconstruction quality.", "questions": "How is the supervised training done on the real-world dataset [14]?\nHow would the method scale to a larger real-world dataset?\n", "limitations": "Main limitation as pointed out in the paper is the supervised training approach. For diverse set of scenes the method will need to generalize for arbitrary shapes. The regularization losses can help with smooth surfaces but will not perform well on objects with holes unless the hyperparameters are tuned carefully.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The proposed approach results in shorter scan time and shows better reconstruction performance at multiple grid resolution compared to all the other methods.", "weaknesses": "-The local similarity and TV losses represent surface smoothness and cause problems with holes. What is the motivation behind using a Guassian weighting instead of bilateral weight which preserves discontinuity.\n\n-There are details missing on the real-world data train/test split. \n\n\n[Minor] Line 128: Fig. 2f should be Fig. 2e\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are highly encouraged by the positive recommendation and comments from the reviewer. The main concerns are addressed below.\n\n**Q1:** *What is the motivation behind using a Gaussian weighting instead of bilateral weight which preserves discontinuity?*\\\n**Reply:** Indeed, bilateral weights are commonly used in computer vision tasks to achieve denoising while preserving image discontinuity when the processed results contain sufficient information. However, in the context of NLOS reconstruction from under-scanning measurements, the primary challenge lies in reconstructing more hidden volume from the scarce input and the secondary goal is to suppress noise. To address this challenge, we take into account the inherent similarity between neighboring pixels in NLOS scenes. We first leverage the Gaussian weighted NLM term to learn and reconstruct more intricate details in the hidden volume. This step aims to capture and preserve the fine features that might be otherwise lost during the reconstruction process. Subsequently, we apply the TV term to denoise the reconstructed results. \n\nThe effectiveness of this approach has also been demonstrated in other areas, including compressed sensing tasks [1,2] and NLOS imaging tasks [3,4]. The success of this methodology in these diverse domains further reinforces its suitability.\n\n[1] Weisheng Dong, Lei Zhang, Guangming Shi, and Xiaolin Wu. Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Transactions on image processing, 20(7):1838–1857, 2011.\\\n[2] Kan Chang and Baoxin Li. Joint modeling and reconstruction of a compressively-sensed set of correlated images. Journal of Visual Communication and Image Representation, 33:286–300, 2015.\\\n[3] Xintong Liu, Jianyu Wang, Zhupeng Li, Zuoqiang Shi, Xing Fu, and Lingyun Qiu. Non-line-of-sight reconstruction with signal–object collaborative regularization. Light: Science & Applications, 10(1):198,2021.\\\n[4] Jianjiang Liu, Yijun Zhou, Xin Huang, Zheng-Ping Li, and Feihu Xu. Photon-efficient non-line-of-sight364\nimaging. IEEE Transactions on Computational Imaging, 8:639–650, 2022.\n\n**Q2:** *There are details missing on the real-world data train/test split. How is the supervised training done on the real-world dataset [14]? How would the method scale to a larger real-world dataset?*\\\n**Reply:** This could be a misunderstanding. The model is NOT trained on real-world data. Instead, our model is first trained in a supervised manner on the synthetic dataset and then directly test on the real-world data. In principle, there would be no difficulty to scale to a larger real-world dataset ( See experiments in Q3 Reply).\n\n**Q3:** *Main limitation as pointed out in the paper is the supervised training approach. For diverse set of scenes the method will need to generalize for arbitrary shapes.*\\\n**Reply:** To further validate the generalization capability of our model, we conduct tests on additional real-world data captured from **different** NLOS imaging systems. In Fig. 2 of the global rebuttal PDF, we showcase two scenarios for comparison. The top scene is captured by [1], while the bottom scene is captured using **our imaging system** ( Details in the global rebuttal response). As evident from the results, our proposed method consistently generates promising outcomes, exhibiting fine details and reduced noise levels. Notably, our method also delivers remarkable robustness even under extremely low scanning grids, highlighting its adaptability to challenging conditions. In contrast to other existing solutions, our method stands out for its superior performance and generalization capabilities across diverse scenes.\n\n[1] Yue Li, Jiayong Peng, Juntian Ye, Yueyi Zhang, Feihu Xu, and Zhiwei Xiong. Nlost: Non-line-of-sight imaging with transformer. In Conference on Computer Vision and Pattern Recognition, pages 13313–13322, 2023.\n\n**Q4:** *Line 178: Fig. 2f should be Fig. 2e*\\\n**Reply:**  Thanks for pointing out this typo, we will update it."}, {"review_id": "UWZx2mT1p5", "reviewer": "Reviewer_9P8E", "summary": "1, This paper proposed a new approach to NLOS reconstruction from USM based on deep learning, and achieved high-quality and rapid inference.\n2, Two main reconstruction modules, dubbed transient recovery network (TRN) and volume reconstruction network (VRN), are designed to produce the hidden volume and the intensity image at a high-spatial resolution.", "questions": "Please refer to Weakness.", "limitations": "Please refer to Weakness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1, This paper proposed a deep learning-based network for NLOS reconstruction from USM, where the key components are the transient recovery network and the volume reconstruction network. The method is reasonable and is an inspirable attempt to apply such an end-to-end framework to deal with this challenging task.\n2, The performance of this method outperforms existing methods both quantitatively and qualitatively, as well as the inference time.", "weaknesses": "1, The main modules in this work are heavily based on existing work. For example, the multiple kernel feature extraction block has already been widely applied in many downstream vision tasks, like “Multiple kernel learning for hyperspectral image classification: A review” (Yanfeng Guo, et al. TGRS 2017). Besides, the volume reconstruction network is very similar to LFE (“Learned feature embeddings for non-line-of-sight imaging and recognition”, Wenzheng Chen, et al. ACM TOG 2020). These points indicate that the contribution of this work is not convincing enough, although the performance is satisfying.\n2, The details of the volume refinement module are unclear.\n3, The loss function in this work is a combination of four loss terms, which may cause generalizability problem to the model, could you please conduct more experiments on more datasets to demonstrate the performance?\n4, In Table 2, it shows that LFE costs much less memory and achieves much faster inference time than this method, which indicates the performance of this method is not that outstanding as authors claimed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for giving us valuable comments. The main concerns are addressed below.\n\n**Q1:** *The main modules in this work are heavily based on existing work. These points indicate that the contribution of this work is not convincing enough, although the performance is satisfying.*\\\n**Reply:** As we know, one advantage of the AI community is gradually building on top of existing works, which eventually leads to both solid and efficient solutions for many practical applications. When we look back, there are a number of works in this community with seemingly incremental components but have made a great impact on future research. From the perspective of \"AI for Science\" (the track we submit), the contribution of being ''an inspirable attempt to apply an end-to-end framework to deal with a challenging task'', as recognized by the reviewer, is nontrivial. Meanwhile, we believe **the strengths of this work are distinct: the comprehensive experiments, thorough ablation, and clearly new state-of-the-art performance have been well recognized by ALL reviewers**. Being confident in our results, we are ready to release the source code to the public. We would appreciate the opportunity to present this work as the new state-of-the-art together with its repository at this conference, which allows other researchers in this scientific field to build on top of it and improve the field further with an enhanced baseline.\n\n**Q2:** *The details of the volume refinement module are unclear.*\\\n**Reply:** The volume refinement module is constructed with a composition of three 3D convolutions and three interlaced residual layers. Each of these residual layers is composed of two 3D convolutions, followed by a ReLU operation, and ultimately a residual connection. This module has been purposefully designed to proficiently accentuate essential features while amplifying the 3D volume. The objective is to enhance fidelity, thereby avoiding a direct projection of the 3D volume onto 2D planes which could potentially result in the loss of crucial inherent features.\n\nThanks for your reminder, and we will update this component in detail.\n\n**Q3:** *Could you please conduct more experiments on more datasets to demonstrate the performance?*\\\n**Reply:** To further validate the generalization capability of our model, we conduct tests on additional real-world data captured from **different** NLOS imaging systems. In Fig. 2 of the global rebuttal PDF, we showcase two scenarios for comparison. The top scene is captured by [1], while the bottom scene is captured using **our imaging system** (Details in the global response). As evident from the results, our proposed method consistently generates promising outcomes, exhibiting fine details and reduced noise levels. Notably, our method also delivers remarkable robustness even under extremely low scanning grids, highlighting its adaptability to challenging conditions. In contrast to other existing solutions, our method stands out for its superior performance and generalization capabilities across diverse scenes.\n\n[1] Yue Li, Jiayong Peng, Juntian Ye, Yueyi Zhang, Feihu Xu, and Zhiwei Xiong. Nlost: Non-line-of-sight imaging with transformer. In Conference on Computer Vision and Pattern Recognition, pages 13313–13322, 2023.\n\n**Q4:** *In Table 2, it shows that LFE costs much less memory and achieves much faster inference time than this method, which indicates the performance of this method is not that outstanding as authors claimed.*\\\n**Reply:** We respectfully disagree with the provided analysis. We would like to address the following three insights: \n\n1) Among all the solutions considered, only CSA and our method are specifically designed for NLOS imaging from under-scanning measurements (USM). It is important to note that LFE is trained to work with sufficient scanning transient measurements, making it less suitable for USM-based NLOS imaging scenarios.\n\n2) As mentioned in Line 261 of the manuscript, our method's total inference time is 420ms (370ms for transient recovery and 50ms for volume reconstruction). On the other hand, the inference time of LFE, which solely involves volume reconstruction, is reported to be 30ms. There is a difference of only 20ms between the two methods for volume reconstruction.\n\n3) In Table 3(b) of the manuscript, the combined approach of our transient recovery network and LFE exhibits improved metrics for PSNR/ACC/SSIM, increasing from 27.04/72.40/0.8620 to 27.79/72.75/0.8692. This improvement indeed demonstrates that there is room for enhancing LFE's performance further.\n\nIn summary, while LFE shows less memory and faster inference time, our method stands out as a specialized solution designed explicitly for NLOS imaging from under-scanning measurements.  Furthermore, the combination of our transient recovery network with LFE showcases the potential for further improving LFE's results. \n\n"}, {"review_id": "UIvKtEQpsF", "reviewer": "Reviewer_VzCp", "summary": "This paper proposes a new deep learning based method for NLOS imaging reconstruction. Their method is an end-to-end deep network that directly acts on under scanning transient measurements to recover the NLOS image. The use of a single pass network enables fast, realtime inference. The key contribution is the extension to under scanning measurement regime, i.e. transient inputs are only available over a sparse grid of scanning points. This is enabled using the “transient recovery network” that performs padding+convolutions using kernels of different sizes to upsample the transient inputs and extract features, and further processing to estimate the sufficient scanning measurements.\n\nThese upsampled transients are transformed from xyt space to xyz space in the “volume reconstruction network” (by inverting the LCT, which is essentially a 3D deconvolution operation), and then projected back to 2D image space to recover the final image. The architecture for this component closely resembles the architecture in LFE [5] (which is also an end-to-end NLOS image reconstruction method, but which is trained to work with sufficient scanning transient measurements).\n\nThe authors also provide extensive experiment results for different baseline methods (both traditional reconstruction methods which don't rely on any learning across multiple image/transient pairs, and deep learning based methods: LFE and U-net). They demonstrate the superior quality as well as reduced inference time of their method. They evaluate the TRN separately against the baseline of simply interpolating the transient histogram from neighboring points.", "questions": "1. Can you comment on any risk of overfitting to the training domain with your method (and deep learning methods in general)? It’s important to quantify that risk since the traditional methods () don’t have any such risk.\n2. You mention in lines 226-227 that \"Due to their time-intensive nature and no need for training on simulated datasets, we exclude the results of NeTF [33] and CSA [17] from the analysis\". I didn't understand the second reason. These methods don't require any training, so it should be easy to include them in the analysis, right? Also if you were able to include these methods in the real-world results, then the first reason (time-intensive nature) also shouldn't hold.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper takes an important step towards realizing the goal of real time NLOS imaging. This is enabled by the use of a fast, end-to-end reconstruction algorithm combined with faster measurement (through underscanning).\n2. Ablation studies are done well (e.g. contribution of regularization loss terms, contribution of TRN and VRN is quantified individually)\n3. Comparison to baseline methods is done in a fair manner, e.g. for the CSA method they reimplement the code to enable acceleration on GPU.\n4. Good writing and clear explanation of method and results", "weaknesses": "The ML contribution of this paper is limited. While the paper makes good incremental progress on an important real world imaging problem, there weren't any novel insights into the nature of the problem (e.g. the compressive sensing aspect, is there any minimum number of scans needed etc). The proposed architecture is also not very novel either.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for giving us valuable comments. The main concerns are addressed below.\n\n**Q1-1:** *The ML contribution of this paper is limited. ... So I'm not sure of the suitability of the paper for a venue like Neurips.*\\\n**Reply:** First of all, we would like to kindly remind that our submission is to the track \"ML for physical sciences\". As recognized by the reviewer, this work \"takes an important step towards realizing the goal of real-time NLOS imaging\", we therefore believe it is suitable for this track in NeurIPS. For the similar reason, the main novelty of this work does not lie in the architecture design, but adapting existing deep learning techniques to this challenging task with an end-to-end framework, which is nontrivial from the perspective of \"AI for Science\". Meanwhile, we believe **the strengths of this work are distinct: the comprehensive experiments, thorough ablation, and clearly new state-of-the-art performance have been well recognized by ALL reviewers**. Being confident in our results, we are ready to release the source code to the public. We hope this cross-discipline work, together with its repository, will contribute to the diversity of this conference, which allows other researchers in this scientific field to build on top of it and improve the field further with an enhanced baseline.\n\n**Q1-2:** *Novel insights into the nature of the problem (e.g. the compressive sensing aspect, is there any minimum number of scans needed etc).*\\\n**Reply:** As suggested by Reviewer r4bJ, we now add a set of experiments to explore the tradeoff between the number of samplings and noisy level for the final reconstruction quality, which we believe provides novel insights into the nature of the problem. In light of these insights, we infer that for our specific imaging system, employing a 16*16 scanning grid strikes a balance between the reconstruction quality and the total acquisition time (yet this would vary for different systems).  Please refer to the global response for details.\n\n**Q2:** *Comment on any risk of overfitting.*\\\n**Reply:**\nAs suggested, we have conducted two experiments to comment on the overfitting risk of our method:\n\n1) Test on the **Unseen** synthetic dataset\\\nIn the manuscript, the training dataset is rendered from 277 motorbikes. Subsequently, we generate an additional **Unseen** testing dataset by rendering 250 transient measurements from other objects (e.g., baskets, cars, cupboards, etc.). The quantitative results, including NeTF and CSA, are listed in Table 1, and the qualitative results are presented in Fig. 3 of the global rebuttal PDF. As evident from the results, our method outperforms the competition in intensity reconstruction, showcasing the superior generalization capability of our network to unseen objects.\n\n2) Test on more real-world data from **different** imaging systems\\\nTo further validate the generalization of our model, we extend the evaluation to include more real-world data captured from different NLOS imaging systems. Fig. 2 of the global rebuttal PDF showcases two scenarios: the top scene captured by [1], and the bottom scene captured by our imaging system ( Details in the global rebuttal response). As observed, our method delivers promising results with fine details and reduced noise levels. Moreover, it exhibits impressive robustness even under extremely low scanning grids. The demonstrated superiority of our method over other solutions underlines its remarkable generalization capability across diverse real-world scenes.\n\n[1] Yue Li, Jiayong Peng, Juntian Ye, Yueyi Zhang, Feihu Xu, and Zhiwei Xiong. Nlost: Non-line-of-sight imaging with transformer. In Conference on Computer Vision and Pattern Recognition, pages 13313–13322, 2023.\n\n**Q3:** *You mention in lines 226-227 that \"Due to their time-intensive nature and no need for training on simulated datasets, we exclude the results of NeTF [33] and CSA [17] from the analysis\". I didn't understand the second reason.*\\\n**Reply:** \nWe are sorry that the explanation provided in the manuscript may be unclear. \n\nNeTF is a test-time training method that renders the transient field on each single measurement, and can be directly applied to the individual real-world data without training on a large amount of synthetic data. For the same reason, it is computationally extensive to generate NeTF results on the large amount of synthetic data, since each scene requires a long rendering time. CSA is a traditional method that does not require training on synthetic data. It is also quite slow, so considerable time is needed to generate CSA results on the large amount of synthetic data.\n\nSince real-world results are the most important, we did not list the synthetic results of NeTF and CSA in the original manuscript due to limited time. However, as expected, we now take abundant computing resources to supplement all synthetic results for NeTF and CSA. For NeTF, we train each under-scanning transient measurement under different scanning grids for testing. For CSA, we also evaluate all the test data. The quantitative results for different scanning grids are as follows:\n| PSNR/ACC/SSIM | NeTF               | CSA                | Ours                   |\n| ------------- | ------------------ | ------------------ | ---------------------- |\n| 32*32         | 22.85/68.25/0.8691 | 23.22/30.43/0.8223 | **28.64/74.35/0.8975** |\n| 16*16         | 19.14/62.38/0.7977 | 22.62/25.83/0.8064 | **28.00/74.19/0.8929** |\n| 8*8           | 17.03/54.87/0.7639 | 21.46/15.56/0.7976 | **27.30/73.59/0.8789** |\n\n\nAs evident from the results, our proposed method consistently outperforms NeTF and CSA across all scanning grid sizes, achieving higher PSNR, ACC, and SSIM metrics.\n\nWe truly appreciate your valuable suggestion and we will revise the explanation. We will include these quantitative results in the manuscript to offer a comprehensive and transparent assessment of our method's capabilities.\n"}]

Author rebuttal:
# This is the global rebuttal for all the reviewers including a PDF containing only figures. Please find individual responses to each reviewer below.

The content is:

**Fig 1**: Qualitative results about the noise level experiments and our imaging system.

As suggested by Reviewer r4bJ, we build a confocal NLOS imaging system and execute experiments across varying noise levels. We have captured the real-world transient measurements with different acquisition time per point and scanning grids while maintaining the total acquisition time constant.

In Fig. 1 (left part of the global rebuttal PDF), we present the results obtained with a total acquisition time of 25.6s and 102.4s, respectively. Through these experiments, two crucial insights have emerged: 1) Under the same total acquisition time, the results obtained from the 16 * 16 scanning grid are better than those from the 32 * 32 scanning grid, revealing that longer exposure times for fewer scanning points yield better imaging quality. 2) However, when comparing the results from 32 * 32 and 8 * 8 scanning grids, this phenomenon is observed to be less pronounced under extremely low scanning grids.

In light of these insights, we infer that for our specific imaging system, employing a 16*16 scanning grid strikes a balance between the reconstruction quality and the total acquisition time (yet this would vary for different systems). Furthermore, it is noteworthy that our manuscript introduces a universally applicable technique for NLOS imaging utilizing under-scanning measurements. This method is agnostic to the tradeoff scanning grid of any given imaging system, reaffirming its broad applicability.


**More details about our NLOS imaging system**. Our system operates in a confocal manner, with a 532 nm laser emitting pulses at 70 ps pulse width and 20 MHz repetition frequency. The emitted pulses pass through a two-axis raster-scanning galvo mirror, transmitting onto the visible wall. Subsequently, both direct and indirect diffuse photons are collected by another two-axis galvo mirror and coupled to a multimode fiber, directed towards a free-running single-photon avalanche diode (SPAD) detector with a detection efficiency of approximately 40\%. A time-correlated single photon counter records the sync signals from the laser and the photon-detection signals from the SPAD. During data collection, the scanning points and sampling points maintain the same direction but are slightly misaligned to avoid capturing the first bouncing signals during scanning. We employ raster scanning across a 2m * 2m area on the visible wall, and each transient measurement's histogram length is set to 512, with a bin width of 32 ps.

**Fig 2**: More real-world qualitative results. 

As suggested by Reviewer iKMu, 9P8E, and VzCp, we extend the evaluation to include more real-world data captured from different NLOS imaging systems. Fig. 2 of the global rebuttal PDF showcases two scenarios: the top scene captured by [1], and the bottom scene captured by our imaging system. As observed, our method delivers promising results with fine details and reduced noise levels. Moreover, it exhibits impressive robustness even under extremely low scanning grids. The demonstrated superiority of our method over other solutions underlines its remarkable generalization capability across diverse real-world scenes.

**Fig 3**: More qualitative results on the Unseen synthetic dataset.

As suggested by Reviewer VzCp, we generate an additional **Unseen** testing dataset by rendering 250 transient measurements from other objects (e.g., baskets, cars, cupboards, etc.). The qualitative results are presented in Fig. 3 of the global rebuttal PDF. As evident from the results, our method outperforms the competition in intensity reconstruction, showcasing the superior generalization capability of our network to unseen objects.

**Table 1**:  Quantitative comparisons on the Unseen dataset.

As suggested by Reviewer VzCp, we generate an additional **Unseen** testing dataset by rendering 250 transient measurements from other objects (e.g., baskets, cars, cupboards, etc.). The quantitative results, including NeTF and CSA, are listed in Table 1 of the global rebuttal PDF. As evident from the results, our method outperforms the competition in intensity reconstruction, showcasing the superior generalization capability of our network to unseen objects.

[1] Yue Li, Jiayong Peng, Juntian Ye, Yueyi Zhang, Feihu Xu, and Zhiwei Xiong. Nlost: Non-line-of-sight imaging with transformer. In Conference on Computer Vision and Pattern Recognition, pages 13313–13322, 2023.


Meta-review: The paper presents a method for non-line-of-sight imaging in the case where a light source and detector are used to illuminate and image a sparse set of points on a relay surface. While the reviews were initially mixed, with concerns about the novelty (i.e., the paper uses a similar reconstruction network to Chen et al. (2020)), evaluation of different scanning resolutions, and lack of baseline comparisons to other methods. The authors addressed these concerns in the rebuttal, with results comparing performance at different resolutions and additional baseline comparisons. 

After the discussion phase, all reviewers had a positive rating, and the AC finds the paper suitable for acceptance. While the paper does build on existing deep learning architectures, there is still novelty in applying these methods to a new problem domain. Overall, the paper does a good job at addressing the problem of NLOS imaging with underscanning measurements: the writing is clear, the evaluation is thorough, and the techniques is applied convincingly to real-world datasets. The authors should incorporate the additional resolution analysis and baseline comparisons into the camera ready version. Additionally, the authors may wish to mention and discuss the concurrent work, appearing at CVPR 2023, referenced below. 

Wang, Jianyu, et al. "Non-line-of-sight Imaging with Signal Superresolution Network." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
Decision: Accept (poster)