SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Quantitative comparisons of intensity images from different methods under various scanning grids on the synthetic test set. The output spatial resolution is  ${128} \times  {128}$  . The best is in bold.
Points	Metrics	FBP [11]	LCT [12]	FK [14]	RSD [15]	UNet [28]	NeTF [35]	LFE [5]	CSA [17]	Ours
32×32	PSNR(dB)	21.12	22.07	24.59	25.44	26.71	22.85	27.94	23.22	28.64
ACC(%)	0.52	56.54	15.48	69.09	73.71	68.25	73.05	30.43	74.35
SSIM	0.2512	0.6314	0.4152	0.7847	0.8808	0.8691	0.8867	0.8223	0.8975
16×16	PSNR(dB)	15.69	20.27	18.02	23.53	26.64	19.14	27.04	22.62	28.00
ACC(%)	0.05	20.16	5.99	60.86	73.79	62.38	72.40	25.83	74.19
SSIM	0.0851	0.4335	0.2121	0.7102	0.8827	0.7977	0.8620	0.8064	0.8929
8×8	PSNR(dB)	12.31	15.85	17.20	19.77	26.28	17.03	26.10	21.46	27.30
ACC(%)	0.031	2.64	7.21	32.54	73.56	54.87	72.17	15.56	73.59
SSIM	0.0441	0.1981	0.2147	0.5131	0.8775	0.7639	0.8218	0.7976	0.8789
Figure 4: Qualitative comparisons of intensity images from different methods under various scanning grids on the synthetic test set. The final spatial resolution is  $128 \times 128$ .
5.3.2 Volume Reconstruction
Synthetic Results. The quantitative results on the synthetic test set are reported in Table 1. As the number of scanning points reduces, the traditional methods' metrics decrease significantly, while the deep-learning-based methods except NeTF [35] perform more robustly. Moreover, our method outperforms the deep methods under different scanning grids in terms of all metrics. The qualitative
Figure 5: Real-world scenes captured by [14] through three different scanning grids are provided.
Figure 6: Real-world scenes captured by [29] through three different scanning grids are provided.
Table 2: The inference time and memory of different models. Note that only CSA and our method are specifically designed for NLOS imaging from USM.
Method	FBP [11]	LCT [12]	FK [14]	RSD [15]	UNet [28]	NeTF [35]	LFE [5]	CSA [17]	Ours
Time (s)	0.042	0.034	0.061	0.038	0.162	2h-24h	0.030	20	0.420
Memory (M)	6026	6016	8056	10344	5642	2G-23G	4692	5306	7130
results are presented in Fig. 4. The traditional methods fail to reconstruct the motorbike under low scanning points, particularly under the  $8 \times 8$  scanning grid. Alternatively, LFE [5] and CSA [17] generate the overall structure with textureless information, whereas our method captures more details even with few scanning points. See more qualitative results in the supplement.
Real-world Results. In Fig. 5 and Fig. 6, we provide qualitative results in the real-world scenes. We exclude FBP [11] and UNet [28] from the results as they cannot perform well under under-scanning conditions. Specifically, under the  $32 \times 32$  and  $16 \times 16$  scanning grids, the traditional methods, LCT [12], FK [14], and RSD [15], support reconstruction with heavy noise. Meanwhile, NeTF [35] fails under the  $16 \times 16$  scanning grid. LFE [5] provides results with lower noise than traditional methods but fails to capture several details such as the tail of the Dragon and the statue near the bookshelf in Fig. 5. By contrast, CSA [17] and our proposed method produce complete and clear results, with our method generally capturing more details and producing less noise, such as the horns of the dragon and the torso of the statue. Under the  $8 \times 8$  scanning grid, all baselines degrade significantly. Nonetheless, both CSA and our method achieve satisfactory results. Our method performs better than CSA [17] in most scenes and achieves comparable results to CSA about the bookshelf. It is worth noting that our proposed method offers faster inference time than CSA, which will be discussed in Sec 5.4. More qualitative results of real-world transient measurements, which are captured by different imaging systems and under different total acquisition time, are reported in the supplement.
5.4 Inference Time
We conduct experiments to evaluate the inference time and memory requirements of various methods. Specifically, we utilize an NVIDIA GeForce A100 GPU for all testing experiments. Traditional methods including FBP [11], LCT [12], FK [14], and RSD [15] are reimplemented using PyTorch. To accommodate CSA [17], which was originally implemented in Matlab on the CPU, we modify the code to enable computations on the GPU, resulting in a significant acceleration. Table 2 presents the comprehensive results.
The traditional methods and LFE [5] achieve a real-time level of performance. NeTF [35] renders each measurement and requires a huge computational time. However, these methods can not be applied to NLOS reconstruction from USM well. CSA [17], the existing solution for NLOS imaging from USM, suffers from iterative computing times and is not well-suited for practical use. By contrast, our method leverages recent advancements in deep learning, enabling it to perform a single forward pass that takes only 420ms (370ms for transient recovery and 50ms for volume reconstruction). This is nearly 50 times faster than CSA [17]. Moreover, our method does not require iterative computing, which is a significant advantage over CSA [17]. Additionally, our method has a reasonable inference memory requirement, feasible for deployment on portable devices to explore real-world scenarios.
In this section, we assess the effectiveness of the proposed TRN, VRN, and the regularized loss items:  $\mathcal{L}_{ls}$  and  $\mathcal{L}_{tv}$ , respectively. Additionally, we investigate the effectiveness of the kernel size of MKFE in TRN. All the ablation experiments are conducted under the  $16\times 16$  scanning grid condition. We quantitatively evaluate the results on a synthetic test set, while the qualitative results are tested on real-world data.
Table 3 lists the ablation results of the proposed TRN, VRN, and the regularized loss items. Initially, we combine the proposed TRN with LFE [5], and the metrics surpass the original LFE [5], which demonstrates the effectiveness of TRN. Subsequently, we combine the interpolation operator and the proposed VRN. The metrics surpass the original LFE again, which demonstrates the effectiveness of VRN. When we integrate the proposed TRN and VRN, the metrics improve notably. Finally, we
Table 3: Ablation study for TRN, VRN, and regu- Table 4: Ablation study for the kernel size in mul- realized loss items. TR and VR denote the transient recovery and volume reconstruction components. tiple kernel feature extraction module of TRN.
TR	VR	\( \mathcal{L}_{ls, tv} \)	PSNR	SSIM	ACC
(-)	Interp	LFE [5]	×	27.04	0.8620	72.40
(a)	TRN	LFE [5]	×	27.79	0.8692	72.75
(b)	Interp	VRN	×	27.71	0.8646	72.85
(c)	TRN	VRN	×	28.00	0.8899	73.44
(d)	TRN	VRN	✓	28.00	0.8929	74.19
Kernel Size	PSNR	SSIM	ACC
3	27.80	0.8744	73.98
5	27.89	0.8791	73.87
9	27.92	0.8825	73.91
17	27.95	0.8696	73.85
3,5,9,17	28.00	0.8929	74.19
(c)
Figure 7: Reconstructed intensity images of the models, whose indexes correspond to Table 3.
introduce the regularized loss items:  $\mathcal{L}_{ls}$  and  $\mathcal{L}_{tv}$ . As a result, the SSIM and ACC metrics demonstrate further improvement. Qualitative results corresponding to Table 3 can be seen in Fig. 7.
Table 4 presents the ablation results of various kernel sizes used in MKFE. It can be observed that the PSNR metric tends to increase along with the kernel size, whereas the SSIM and ACC metrics do not mirror that trend. This can be attributed to the fact that the larger kernel could recognize distant regions, while the smaller one could focus on local areas. Notably, the multiple-kernel mechanism results in the best performance across all metrics.
Conclusions. In this paper, we study the challenging task of non-line-of-sight imaging from the under-scanning measurement in the deep learning perspective. The elaborate network, which is composed of two components: a transient recovery network and a volume reconstruction network, is proposed to recover the transient measurement and reconstruct the hidden volume successfully. Further, the regularized constraints are exploited to address overfitting and enhance more local details while suppressing smoothing. Notably, our method runs 50 times faster than the existing iterative solution. We believe that our method is a significant step forward in the field of non-line-of-sight imaging and has the potential to unlock new possibilities for future research.
Limitations. Several limitations exist in our proposed method, which serve as subjects for future work. Firstly, our forward model is constrained to a linear confocal system, our future work involves generalizing the forward model. Secondly, our proposed method is fully supervised, which may restrain the generalization ability for real-world data. Designing an unsupervised training method for non-line-of-sight imaging is another future work.
This work was supported in part by the National Natural Science Foundation of China under Grants 62131003 and 61901435, and Innovation Program for Quantum Science and Technology under Grant 2021ZD0300300.
[1] Nicolas Scheiner, Florian Kraus, Fangyin Wei, Buu Phan, Fahim Mannan, Nils Appenrodt, Werner Ritter, Jurgen Dickmann, Klaus Dietmayer, Bernhard Sick, et al. Seeing around street corners: Non-line-of-sight detection and tracking in-the-wild using doppler radar. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2068-2077, 2020.
[2] Cheng Wu, Jianjiang Liu, Xin Huang, Zheng-Ping Li, Chao Yu, Jun-Tian Ye, Jun Zhang, Qiang Zhang, Xiankang Dou, Vivek K Goyal, et al. Non-line-of-sight imaging over  $1.43\mathrm{km}$ . Proceedings of the National Academy of Sciences, 118(10):e2024468118, 2021.
[22] Sean I Young, David B Lindell, Bernd Girod, David Taubman, and Gordon Wetzstein. Non-line-of-sight surface reconstruction using the directional light-cone transform. In IEEE/CVF Conference on computer vision and pattern recognition, pages 1407-1416, 2020.
[23] Victor Arellano, Diego Gutierrez, and Adrian Jarabo. Fast back-projection for non-line of sight reconstruction. Optics Express, 25(10):11574-11583, 2017.
[24] Byeongjoo Ahn, Akshit Dave, Ashok Veeraraghavan, Ioannis Gkioulekas, and Aswin C Sankaranarayanan. Convolutional approximations to the general non-line-of-sight imaging operator. In IEEE/CVF International Conference on Computer Vision, pages 7889-7899, 2019.
[25] Jianjiang Liu, Yijun Zhou, Xin Huang, Zheng-Ping Li, and Feihu Xu. Photon-efficient non-line-of-sight imaging. IEEE Transactions on Computational Imaging, 8:639-650, 2022.
[26] Xintong Liu, Jianyu Wang, Zhupeng Li, Zuoqiang Shi, Xing Fu, and Lingyun Qiu. Non-line-of-sight reconstruction with signal-object collaborative regularization. Light: Science & Applications, 10(1):198, 2021.
[27] Zhupeng Li, Xintong Liu, Jianyu Wang, Zuoqiang Shi, Lingyun Qiu, and Xing Fu. Fast non-line-of-sight imaging based on first photon event stamping. Optics Letters, 47(8):1928-1931, 2022.
[28] Javier Grau Chopite, Matthias B Hullin, Michael Wand, and Julian Iseringhausen. Deep non-line-of-sight reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 960-969, 2020.
[29] Yue Li, Jiayong Peng, Juntian Ye, Yueyi Zhang, Feihu Xu, and Zhiwei Xiong. Nlost: Non-line-of-sight imaging with transformer. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13313-13322, 2023.
[30] Jianyu Wang, Xintong Liu, Leping Xiao, Zuoqiang Shi, Lingyun Qiu, and Xing Fu. Non-line-of-sight imaging with signal superresolution network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17420-17429, 2023.
[31] Weisheng Dong, Lei Zhang, Guangming Shi, and Xiaolin Wu. Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Transactions on Image Processing, 20(7):1838-1857, 2011.
[32] Kan Chang and Baoxin Li. Joint modeling and reconstruction of a compressively-sensed set of correlated images. Journal of Visual Communication and Image Representation, 33:286-300, 2015.
[33] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[35] Siyuan Shen, Zi Wang, Ping Liu, Zhengqing Pan, Ruiqian Li, Tian Gao, Shiying Li, and Jingyi Yu. Non-line-of-sight imaging via neural transient fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(7):2257-2268, 2021.