SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yue Li Yueyi Zhang* Juntian Ye Feihu Xu Zhiwei Xiong
{yueli65, jt141884}@mail.ustc.edu.cn
{zhyuey,feihuxu,zwxiong}@ustc.edu.cn
Active confocal non-line-of-sight (NLOS) imaging has successfully enabled seeing around corners relying on high-quality transient measurements. However, acquiring spatial-dense transient measurement is time-consuming, raising the question of how to reconstruct satisfactory results from under-scanning measurements (USM). The existing solutions, involving the traditional algorithms, however, are hindered by unsatisfactory results or long computing times. To this end, we propose the first deep-learning-based approach to NLOS imaging from USM. Our proposed end-to-end network is composed of two main components: the transient recovery network (TRN) and the volume reconstruction network (VRN). Specifically, TRN takes the under-scanning measurements as input, utilizes a multiple kernel feature extraction module and a multiple feature fusion module, and outputs sufficient-scanning measurements at the high-spatial resolution. Afterward, VRN incorporates the linear physics prior of the light-path transport model and reconstructs the hidden volume representation. Besides, we introduce regularized constraints that enhance the perception of more local details while suppressing smoothing effects. The proposed method achieves superior performance on both synthetic data and public real-world data, as demonstrated by extensive experimental results with different under-scanning grids. Moreover, the proposed method delivers impressive robustness at an extremely low scanning grid (i.e.,  $8 \times 8$ ) and offers high-speed inference (i.e., 50 times faster than the existing iterative solution). The code is available at https://github.com/Depth2World/Under-scanning_NLOS.
Non-line-of-sight (NLOS) imaging technology permits the detection of objects located behind occluders, and has numerous real-world applications, such as autonomous driving, remote sensing, and disaster rescue [1, 2, 3, 4, 5, 6, 7, 8, 9]. The widely-known time-of-flight confocal imaging system, composed of a pulse laser, a Single Photon Avalanche Diode (SPAD), and a Time-Correlated Single Photon Counting sensor (TCSPC), is favored for its ability to capture high-quality transient measurements [10]. As illustrated in Fig. 1, the picosecond pulse laser illuminates the relay wall at a certain location. The light scatters at the target object behind the wall and then re-scatters to the relay wall, from which the photons are captured by the SPAD along with TCSPC, forming a histogram. The transient measurement records the histograms of all raster-scanned locations and can be used to derive the hidden object volume by the well-studied reconstruction algorithms such as those developed in [5, 11, 12, 13, 14, 15]. However, the total acquisition time, which is directly proportional to the number of scanning points and the exposure time of each scanning point, hinders the real-time NLOS imaging process.
Figure 1: Overview of the active confocal NLOS imaging. The red circles denote a sufficient-scanning grid at the high-spatial resolution, and the blue triangles denote an under-scanning grid.
The time-consuming dilemma raises the question of how to reconstruct the hidden volume from the under-scanning measurement (USM) without compromising the imaging quality. As shown in Fig. 1, the spherical waves are scattered back by the illuminated object, which indicates that the temporal histogram from each scanning point contains all the geometry information of the hidden object. As a result, it is feasible to recover the high-spatial-resolution hidden volume from USM. Previous studies [16, 17] explored NLOS imaging from USM using circular scanning or the compressed sensing algorithm [18]. These solutions either produce unsatisfactory outcomes or require lengthy numerical iterative computations, leaving a large room for improvement.
In this paper, we propose a new approach to NLOS reconstruction from USM, pioneering the utilization of deep learning to achieve superior quality and swift inference. Specifically, our proposed network consists of two main components: the transient recovery network (TRN) and the volume reconstruction network (VRN). Given the under-scanning transient measurement, a multiple kernel feature extraction module in TRN generates the features at different kernel scales to obtain the pyramid receptive field. Then, the features are concatenated and aggregated by a multiple feature fusion module to produce the high-spatial-resolution transient measurement which undergoes VRN subsequently. The VRN consists of a feature extraction module, a feature propagation module, a volume refinement module, and a projection reconstruction module. Inspired by [19, 5, 3, 6], we integrate the linear physics prior [12] into the feature propagation module. Additionally, we introduce the local similarity and total variation losses to regularize our network. The local similarity loss assists with perceiving more local details, and the total variation loss aids in suppressing smoothing effects. Our proposed network has an end-to-end architecture that necessitates only one forward pass for reconstruction. The high-fidelity reconstruction in an extremely low scanning grid (i.e.,  $8 \times 8$ ) and the high-speed inference (i.e., 50 times faster than the existing iterative solution) demonstrate the superiority of our proposed method. The contributions are summarized as follows:
- We study the challenging task of NLOS imaging from USM and develop an end-to-end network for high-quality and fast inference, which is the first of its kind.
- We exploit the multiple kernel mechanism in the transient recovery network to adapt transient measurements with different under-scanning grids.
- Our proposed method achieves superior performance on both synthetic data and public real-world data, as demonstrated by extensive experimental results.
2.1 NLOS Imaging Algorithms
In recent years, transient-based NLOS imaging has garnered significant attention in the research community. Despite the apparent progress made by direct reconstruction methods [11, 12, 13, 14, 15, 20, 21, 22, 23], these methods remain sensitive to exposure noise, data quality, and variations in regions. Concurrently, iterative methods [2, 24, 25, 26, 27] can effectively reduce imaging noise but suffer considerably from lengthy computation time. In contrast, deep-learning-based techniques [28, 5, 3, 29] offer a balance between the imaging quality and inference time, making them an attractive proposition for NLOS reconstruction. To this end, we approach NLOS reconstruction
from a deep-learning perspective, leveraging its advantages of efficient inference and robustness to noisy data.
2.2 NLOS Imaging From USM
The experiments conducted in [12, 17] indicated that the exposure time per scanning point is more crucial than the number of scanning points for imaging quality. Subsampling the scanning points can be a viable trade-off for reducing the total acquisition time.  $\mathrm{C}^2\mathrm{NLOS}$  [16] advocated for circular scanning methodology which improved the transient capturing efficiency, but the corresponding reconstruction quality left much to be desired. CSA [17] investigated a compressed sensing algorithm called SPITAL [18] for NLOS imaging from USM at a grid shape and demonstrated impressive results at a low scanning ratio, however, its iterative computation required an extended time. In contrast, our proposed method leverages deep-learning technology for NLOS imaging from USM, and necessitates a single forward pass for inference. As a concurrent work, Wang et al. [30] utilized the learning-based method and focused on recovering the high-spatial-resolution transient measurement, rather than NLOS reconstruction from USM.
The travel time of the direct photons between the visible wall and the NLOS imaging system is disregarded for simplicity. The transient measurement, which solely describes the indirect photon bounces, is constructed via a 3D spatial-temporal (2D spatial and 1D temporal) representation. Similar to the previous approaches [12, 16, 22], we assume that light scatters just once at the hidden object in an isotropic manner, without any occlusions in the object. Given the illuminated point at  $t = 0$ , the object o and the scanning point s which is located at the same position as the illuminated point, the active confocal NLOS imaging observation model can be expressed as follows:
$$
\tau (\mathbf {s}, t) = \iiint_ {\Omega} \rho (\mathbf {o}) \frac {\delta (2 \| \mathbf {s} - \mathbf {o} \| - c \cdot t)}{\| \mathbf {s} - \mathbf {o} \| ^ {4}} \mathrm {d} \Omega , \tag {1}
$$
where  $\tau$  denotes the transient measurement,  $\Omega$  denotes the spherical waves of the scattered light,  $\rho$  represents the volume of the albedo of the hidden object surface,  $\delta$  models the light propagation function,  $||\mathbf{s} - \mathbf{o}||^4$  is the quartic radiance falloff of light transport between the relay wall and the object, and  $c$  is the speed of the light. As discussed in [12, 16], Eq. 1 can be derived to a straightforward 3D convolution form via variables changing. After discretized, the formation model can be regarded as a linear model:
$$
\rho = A ^ {- 1} \tau , \tag {2}
$$
where  $\mathbf{A}$  refers to the light transport matrix, the vectorized  $\rho \in \mathbb{R}^{h_x w_y n_z}$ , the vectorized  $\tau \in \mathbb{R}^{h_x w_y n_t}$ ,  $h_x$  and  $w_y$  are the number of the scanning points along the  $x$  and  $y$  axis,  $n_t$  and  $n_z$  are the length of the discretized histogram bins along time  $t$  axis and depth bins along depth  $z$  axis. The determination of  $\rho$  represents an inverse problem, involving 3D deconvolution procedures.
For NLOS imaging from USM, define the under-scanning transient measurement  $\pmb{\tau}_{u} \in \mathbb{R}^{h_{x}w_{y}n_{t}}$  and the corresponding hidden volume  $\rho_{u} \in \mathbb{R}^{h_{x}w_{y}n_{z}}$ , the sufficient-scanning transient measurement  $\pmb{\tau}_{s} \in \mathbb{R}^{H_{x}\bar{W}_{y}n_{t}}$  and the corresponding hidden volume  $\rho_{s} \in \mathbb{R}^{H_{x}W_{y}n_{z}}$ , where  $H_{x} > h_{x}$  and  $\bar{W}_{y} > w_{y}$ . The NLOS imaging from USM is transformed into the following optimization problem:
$$
\hat {\theta} _ {\mathcal {T}}, \hat {\theta} _ {\mathcal {V}} = \underset {\theta_ {\mathcal {T}}, \theta_ {\mathcal {V}}} {\arg \min } \left(\mathcal {L} \left(\boldsymbol {\tau} _ {u} ^ {\uparrow}, \boldsymbol {\tau} _ {s}\right) + \mathcal {L} \left(\boldsymbol {\rho} _ {u} ^ {\uparrow}, \boldsymbol {\rho} _ {s}\right) + \lambda \phi \left(\theta_ {\mathcal {T}}, \theta_ {\mathcal {V}}\right)\right), \tag {3}
$$
where  $\tau_u^\uparrow$  and  $\rho_{\pmb{u}}^{\uparrow}$  are the predicted high-spatial-resolution transient data and hidden volume through the algorithms  $\theta_{\mathcal{T}}$  and  $\theta_{\mathcal{V}}$  respectively,  $\mathcal{L}$  represents the loss function between the generated and ground-truth one,  $\phi$  is the regularization term, and  $\lambda$  is a parameter to weigh the regularization term.
In this section, we present an end-to-end deep network for NLOS imaging from USM. The network takes in under-scanning transient measurement and finally outputs the hidden volume and the intensity image at a high-spatial resolution. The overall network comprises two stages: the Transient Recovery Network (TRN) and the Volume Reconstruction Network (VRN). In the first stage, the TRN recovers
Figure 2: (a) The pipeline of the proposed network. The network consists of a transient recovery network (TRN) and a volume reconstruction network (VRN). (b) An overview of TRN. The blue rectangular blocks denote 3D residual blocks with corresponding kernel sizes. The orange and yellow rectangular blocks represent the 3D convolutions and 3D residual blocks, respectively. (c) An overview of VRN. (d) A diagram for the multiple kernel feature extraction. (e) A diagram for the regularized loss functions. The points on the objects always have similar values with the neighbors, while points on the edges have different values from the outside.
the under-scanning transient data to the desired high-spatial resolution. In the second stage, the VRN reconstructs the hidden volume and then generates the intensity image, as illustrated in Fig. 2(a).
4.1 Transient Recovery Network
Histograms captured from distinct scanning points all contain the complete hidden geometry information. However, each histogram has unique characteristics due to the differences in traveling time. Consequently, the data from even the adjacent scanning points can not be identical, but similar. To explore this information and restore the transient measurement at the high-spatial resolution, we design the transient recovery network (TRN) to recover sufficient-scanning transient measurement.
TRN consists of a Multiple Kernel Feature Extraction module (MKFE) and a Multiple Feature Fusion module (MFF), as illustrated in Fig. 2(b). The MKFE module comprises four residual blocks with different convolution kernels and an interpolation operator. Each residual block consists of two 3D convolutions, two Leaky ReLU activation functions, and one residual connection. The kernel scales in the residual blocks of the MKFE are arranged from large to small to perceive the entire geometry, therefore providing a sizeable receptive field. As illustrated in Fig. 2(d), the scanning points in the padded scanning grid are sparse and distant from each other. To exploit the feature connection between the scanning points in one convolution operation directly, the size of the convolution kernel is set to cover the farthest scanning point distance. The large kernel perceives the distant regions, and the small kernel focuses on the local areas. Various under-scanning grids require different appropriate kernel sizes. For a more general architecture, we further implement a multiple kernel mechanism for TRN to adapt the different under-scanning transient measurements. The MFF module alternately involves three 3D convolutions and three 3D residual blocks with fixed kernels.
Given the under-scanning transient measurement  $\tau_{u}$ , we first pad the under-scanning transient measurement to reach the desired high-spatial resolution. All histograms, except the initial position,
are zeroed. Next, we pass the padded transient data through four residual blocks to obtain fine features. Then, we utilize the nearest interpolation for the input under-scanning transient measurement to get coarse features at the desired resolution, which facilitates stable training and fast convergence. Afterward, the coarse and fine features are concatenated along the channel dimension. Then the combined features are supplied to the fusion recovery decoder to generate the sufficient-scanning transient measurement  $\tau_{s}$ .
4.2 Volume Reconstruction Network
For the hidden volume reconstruction network (VRN), we borrow the idea from the previous works [5, 3] which incorporated the physics prior to feature propagation. Inspired by LFE [5], VRN is composed of a feature extraction module, a feature transform module, a volume refinement module, and a projection reconstruction module, as shown in Fig. 2(c). Different from LFE [5], we utilize the physical model-based approach called LCT [12] to build the feature transform module. LCT [12] adopts deconvolution operation and conducts efficient computation. Moreover, we design a volume refinement module to enhance the 3D features for high fidelity. The module is constructed with three 3D convolutions and three interlaced residual layers. Each of these residual layers is composed of two 3D convolutions, followed by a ReLU operation and a residual connection. This module has been purposefully designed to proficiently accentuate essential features while amplifying the 3D volume. The objective is to enhance fidelity, thereby avoiding a direct projection of the 3D volume onto 2D planes which could potentially result in losing crucial inherent features.
Given the sufficient-scanning transient measurement  $\tau_{s}$  which is recovered by TRN, we first extract the intrinsic spatial-temporal features by the feature extraction module. Then the extracted features are supplied to the feature transform module, where the 3D spatial-temporal features are transformed into 3D spatial features. Afterward, the 3D spatial features proceed to the volume refinement module to produce the reconstructed 3D volume  $\rho$ . For further optimizing the output intensity images, the 3D volume representations are mapped to the 2D domain using maximum intensity projection along the depth axis, and then the projected 2D features are convolved to generate the final intensity images  $I$ .
4.3 Regularized Training
Our proposed network is trained in an end-to-end supervised manner. The overall loss function consists of four components: the measurement loss  $\mathcal{L}_{mea}$ , the intensity loss  $\mathcal{L}_{int}$ , the local similarity loss  $\mathcal{L}_{ls}$  and the total variation loss  $\mathcal{L}_{tv}$ . The total loss is formulated as follows:
$$
\mathcal {L} _ {\text {t o t a l}} = \mathcal {L} _ {\text {m e a}} + \lambda_ {1} \cdot \mathcal {L} _ {\text {i n t}} + \lambda_ {2} \cdot \mathcal {L} _ {\text {l s}} + \lambda_ {3} \cdot \mathcal {L} _ {\text {t v}}, \tag {4}
$$
where  $\lambda_1, \lambda_2$ , and  $\lambda_3$  are the hyper-parameters that weigh the corresponding loss function.
To achieve end-to-end training, the L1 norms for the outputs from TRN and VRN are computed:
$$
\mathcal {L} _ {m e a} = \left\| \boldsymbol {\tau} _ {s} - \boldsymbol {\tau} _ {s} ^ {g t} \right\| _ {1}, \quad \mathcal {L} _ {i n t} = \left\| \boldsymbol {I} - \boldsymbol {I} ^ {g t} \right\| _ {1}, \tag {5}
$$
where  $\tau_{s}$  and  $\tau_{s}^{gt}$  correspond to the predicted sufficient-scanning and the ground-truth transient measurements,  $I$  and  $I^{gt}$  refer to the predicted and ground-truth intensity images of the objects hidden from view. The efficacy of  $\mathcal{L}_{mea}$  is showcased in Sec 5.5.
Under conditions of fewer scanning points, two problems are encountered in deep learning methodology. Firstly, the reduction of transient measurements causes difficulties in volume reconstruction, making small or distant objects mostly unrecoverable or partially recovered. Secondly, effective signal reduction or signal sparsity increase leads to network overfitting to the simulation data, thereby impeding accurate real-world testing. To overcome these challenges, we incorporate regularized constraints during the training process. Firstly, we leverage the local similarity of the natural volume since the points on the hidden object are analogous to their neighbors. We employ a Gaussian window to weigh the neighbors of the points, as [31, 32, 25]. The local similarity loss is formulated as follows:
$$
\mathcal {L} _ {l s} = \sum_ {x} \sum_ {y} \sum_ {z} | | \boldsymbol {\rho} (x, y, z) - \hat {\boldsymbol {\rho}} (x, y, z, k) \cdot W | | _ {1}, \tag {6}
$$
where  $\rho(x, y, z)$  indicates the volume at position  $(x, y, z)$ ,  $\hat{\rho}(x, y, z, k)$  represents the volume block centered at  $(x, y, z)$  with size  $k$ ,  $W$  refers to the Gaussian window with size  $k$ . As shown in Fig. 2(e), when the local similarity loss function acts on the points located at the objects, it brings out more local details. On the contrary, when it acts on the boundary points, it blurs and smoothens the edges. Secondly, to suppress background noise and sharpen the edge boundary, we introduce the 3D total
variation loss function, formulated as follows:
$$
\begin{array}{l} \mathcal {L} _ {t v} = \sum_ {x} \sum_ {y} \sum_ {z} \left(\left| \left| \boldsymbol {\rho} (x + 1, y, z) - \boldsymbol {\rho} (x, y, z) \right| \right| _ {1} + \left| \left| \boldsymbol {\rho} (x, y + 1, z) - \boldsymbol {\rho} (x, y, z) \right| \right| _ {1} \right. \tag {7} \\ + \left\| \boldsymbol {\rho} (x, y, z + 1) - \boldsymbol {\rho} (x, y, z) \right\| _ {1}). \\ \end{array}
$$
The effectiveness of the regularized loss functions  $\mathcal{L}_{ls}$  and  $\mathcal{L}_{tv}$  is demonstrated in Sec 5.5.
5.1 Datasets and Evaluation Metrics
We evaluate our method on both synthetic and real-world datasets. For the synthetic dataset, we simulate the data using a transient renderer provided by [5], rendering 277 motorbikes from the ShapeNet dataset [33]. The synthetic dataset is composed of 2712 training samples and 291 testing samples. The scanning grid is with a size of  $128 \times 128$  and the time resolution is 512 with a bin width of 33ps. For the real-world scenes, we adopt the real-world dataset provided by [14] which contains diverse objects. We conduct preprocessing to discard the direct bounce data and resize the data to  $128 \times 128 \times 512$ . The total acquisition time of the real-world transient data is about  $45\mathrm{min}$ . We directly extract the scanning points from the initial transient data to simulate the under-scanning process. A  $32 \times 32$ ,  $16 \times 16$ , and  $8 \times 8$  scanning grid represents the total acquisition time of 169s, 42s, and 11s respectively. NLOS imaging from USM is able to expedite the data capturing by orders of magnitude. Besides, we also adopt the real-world data from [29] to further validate the generalization capability of our model.
For the intensity recovery results, we evaluate the performance with three metrics: Peak Signal-to-Noise Ratio (PSNR), Structural SIMilarity (SSIM), and ACCuracy (ACC). We utilize ACC to determine the foreground and background classification accuracy.
5.2 Implementation and Baseline Details
The proposed network is implemented in PyTorch. The optimizer is AdamW [34] with a learning rate of  $1 \times 10^{-4}$  and a weight decay of  $1 \times 10^{-4}$ . The models are trained on the synthetic dataset for 50 epochs with a batch size of 4, and directly tested on the real-world dataset, setting the high-spatial resolution to  $128 \times 128$ . For the loss function, the hyperparameters  $\lambda_{1}, \lambda_{2}$ , and  $\lambda_{3}$  is  $1, 1 \times 10^{-5}$ , and  $1 \times 10^{-6}$ . All the experiments are conducted on a workstation equipped with 4 NVIDIA GeForce A100 GPUs.
Our baseline methods include four traditional methods (FBP [11], LCT [12], FK [14], RSD [15]), three deep-learning-based methods (UNet [28], NeTF [35], LFE [5]), and one iterative method (CSA [17]). All methods, except CSA [17] perform non-iterative reconstruction. We find the performance of these non-iterative methods, which upsample the input before reconstruction, is better than that of the methods which reconstruct the volume before upsampling the output. Thus, for these non-iterative baselines, we interpolate the input under-scanning data to the desired high-spatial resolution using the nearest interpolation approach, as [17]. CSA [17] and NeTF [35] are fed with the padded transient data (at the same spatial resolution as the desired transient data) and under-scanning transient data, respectively.
5.3 Experimental Results
5.3.1 Transient Recovery
To validate the effectiveness of TRN, we compare the qualitative histograms of the recovered, direct interpolated, and raw transient measurement which is from the real-world scene (Teaser [14]) at certain scanning points. We select the transient data at three low scanning girds:  $32 \times 32$ ,  $16 \times 16$ , and  $8 \times 8$ . As illustrated in Fig. 3, our predicted results show greater similarity to the ground truth in terms of curve trend and peak size. Specifically, under the  $8 \times 8$  scanning condition, the interpolated histogram consistently generates larger outliers, while our method delivers robust performance, indicating the superiority of our proposed TRN and paving the way for subsequent volume reconstruction.
Figure 3: Comparisons of the histograms from the recovered and the interpolated transient measurement. The rows show the histograms from different recovered transient measurements with the same scanning position. The green, blue, and red lines represent the interpolated, recovered, and ground-truth histograms. The data is from the real-world scene (Teaser in [14]).