Toward Dynamic Non-Line-of-Sight Imaging with Mamba Enforced Temporal Consistency
Authors: Yue Li
Yi Sun
Shida Sun
Juntian Ye
Yueyi Zhang
Feihu Xu
Zhiwei Xiong
Venue: NeurIPS 2024 poster (2024)
Keywords: dynamic
non-line-of-sight imaging
spatial-temporal Mamba
Primary area: machine_learning_for_physical_sciences
TL;DR: We propose a novel  spatial-temporal Mamba based approach for dynamic NLOS reconstruction, as well as  a new dataset for training and testing.

Abstract: Dynamic reconstruction in confocal non-line-of-sight imaging encounters great challenges since the dense raster-scanning manner limits the practical frame rate. A fewer pioneer works reconstruct high-resolution volumes from the under-scanning transient measurements but overlook temporal consistency among transient frames. To fully exploit multi-frame information, we propose the first spatial-temporal Mamba (ST-Mamba) based method tailored for dynamic reconstruction of transient videos. Our method capitalizes on neighbouring transient frames to aggregate the target 3D hidden volume. Specifically, the interleaved features extracted from the input transient frames are fed to the proposed ST-Mamba blocks, which leverage the time-resolving causality in transient measurement. The cross ST-Mamba blocks are then devised to integrate the adjacent transient features. The target high-resolution transient frame is subsequently recovered by the transient spreading module. After transient fusion and recovery, a physical-based network is employed to reconstruct the hidden volume. To tackle the substantial noise inherent in transient videos, we propose a wave-based loss function to impose constraints within the phasor field. Besides, we introduce a new dataset, comprising synthetic videos for training and real-world videos for evaluation. Extensive experiments showcase the superior performance of our method on both synthetic data and real world data captured by different imaging setups. The code and data are available at https://github.com/Depth2World/Dynamic_NLOS.

Toward Dynamic Non-Line-of-Sight Imaging with Mamba Enforced Temporal Consistency
Yue Li
Yi Sun
Shida Sun
Feihu Xu Zhiwei Xiong* University of Science and Technology of China
Juntian Ye
Yueyi Zhang
(yueli65, sunyi1217, , jt141884, sassu,
sdsun) @mail. ustc. edu. en {zhyuey, feihuxu, zwxiong wostongtt.eoo.oooooooooooeeeeeee cn
Abstract
Dynamic reconstruction in confocal non-line-of-sight imaging encounters great challenges since the dense casterscaanning manner limits the practical frame rate. fewer pioneer works reconstruct high-resolution volumes from the under-scanning transient measurements but overlook temporal consistency among transient frames. To fully exploit multi-frame information, we propose the first spatial-temporal Mamba (ST-Mamba) based method tailored for dynamic reconstruction of transient videos. Our method capitalizes neighbouring transient frames to aggregate the target 3D hidden volume. Specifically, the interleaved features extracted from the input transient frames are fed to the proposed ST-Mamba blocks, which leverage the time-resolving causality transient measurement. The cross ST- Mamba blocks then devised to integrate the adjacent transient features. The target high- high-resolution transient frame is subsequently recovered by the transient spreading module. After transient fusion and recovery, physical-based network is employed to reconstruct the hidden volume. To tackle the substantial noise inherent in transient videos, propose a wave-based loss function impose constraints within the phasor field. Besides, we introduce a new dataset, comprising synthetic videos for training and real-world videos for evaluation. Extensive experiments showcase the superior performance of our method on both synthetic data and real- world data captured by different imaging setups. The code and data are available pettps://githnn httpa//cc//ogghoooo/////aaaaaaa
Introduction
Non-Line-of-Sige offSSght (NLOS) imaging revolutionizes our comprehension of the environment by reveal- ing hidden information. Different from conventional cameras, the NLOS system captures indirect light reflections or signals that interact with the hidden object, subsequently rebounding off the relay wall that is visible to the imaging system. By analyzing these reflections, NLOS can reveal critical properties like albedo and surface normal of the hidden objects, unlocking valuable insights. A typical active NLOS imaging setup is illustrated in Fig. [ The pulsed laser emits periodic pulses directed towards a relay wall, serving the dual purpose of illumination and synchronization for the imaging system. The Single Photon Avalanche Diode (SPAD) captures photons reflected from the relay wall, while the Iime-Correlatd Single Photon Counting sensor (TCSPC) records their arrival times within each pulse period. The temporal distribution of each scanning point accumulates over successive pulse periods, termed exposure time. Consequently, the total acquisition time for a transient measurement scales proportionally with the exposure time and the density of the scanning grid. Notably, achieving high-quality reconstructions necessitates dense scanning grids, at the expense of longer acquisition times, typically ranging from minutes to hours.
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Tramsinn Frumes
Relay Was
19
Galro
Lear
Dynamic Scutes
t
Galvo
Syne
Lens
Piber
DAD
TOSPC
Uccluder
Figure I: Left: Active NLOS Imaging Setup. Right: Dynamic NLOS Imaging.
The compromise between acquisition time and data quality poses formidable obstacle to the advancement of fast imaging techniques, making them impractical for real-world applications.. Recent endeavors 1 2 4 addressed this challenge by initiating from under-scanning measurements and striving to reconstruct high-resolution volumes comparable to those derived from sufficient scanning measurements. These methods can significantly accelerate data acquisition by orders of magnitude. Recent research 4 5 6 has further demonstrated that employing
cow-density scanning grid can balance reconstruction quality and total acquisition time, without sacrificing too much information. The sparse scanning points and rapid exposure time offer the potential for dynamic NLOS imaging, a field yet to be fully explored. However, the pursuit of dynamic NLOS reconstruction faces two primary challenges: 1) Insufficient information fusion across adjacent transient frames: Existing methods, whether traditional or deep-based, typically concentrate on individual transient frames, overlooking the temporal consistency between them. 2) Lack of NLOS video datasets, including synthetic data for training and real-world data for evaluation: The rapid exposure time results in diminished signal-to-noise ratio (SNR) of transient measurement, highlighting the critical need for simulation datasets that accurately emulate real-world conditions. Besides, dynamic imaging imposes extremely high requirements on the synchronization accuracy and acquisition efficiency of the hardware system.
Based on these observations. we exploit temporal consistency in transient videos by extracting information from the multiple frames to compensate for the unrecoverable areas for the reference frame, leading to improved quality. Our proposed method consists of two main stages: firstly, integrating the transient frames and expanding the target transient measurement across the spatial dimension, and secondly, reconstructing hidden volume. Specifically, in the first stage, after extracting the features from the input, we introduce the elaborate spatial-tempon sequentially exploit the causality in transient measurement and dig into the inherent long-ranging features along the spatial and temporal dimensions. Subsequently, we devise the cross ST-Mamba to blend complementary features among transient frames towards the target frame. After that, the high-resolution transient frame is recovered by the transient spreading module. For the second stage, we embed the physical prior into the feature transformation module, i.e., transforming the spatial-temporal data into the Fourier domain, element-wise multiplying with the inverse point spread function (PSF) of the imaging system, and then reverting the features into the spatial domain. The final refinement module subsequently enhances the target hidden volume, as well as the derived intensity image and depth map. During the training process, introduce novel virtual wave-based loss function to accentuate effective information in low SNR data, by employing a Gaussian-shaped illumination function constrain transient measurement within the phasor field 7
Mamba (ST-Mamba) to
To bridge the training and testing phases, we present new dataset for NLOS dynamic imaging. The synthetic data comprises the dynamic objects with 32 frames in each sequence with varying quantum efficiency. The real-world NLOS videos are captured at 4 frames per second (FPS) by our imaging prototype. The dataset is publicly available to propel research in dynamic imaging within this field. Comparative evaluation against existing traditional and deep-learning haap--laannnnbaaase based solutions demonstrates that our method achieves superior reconstruction performance and generalization capability to real-world scenarios. Our contribution can be summarised as follows:
For the first time, we introduce Mamba-based method tailed for dynamic NLOS imaging. The proposed spatial-temporal Mamba mechanisms successfully exploit the inherent long- ranging causal features and integrate the temporal consistency across the transient frames.

---

We build a new dynamic NLOS dataset crafted for learning from synthetic data and evaluat- ing models on real-world data for dynamic NLOS reconstruction. which could help advance faster NLOS imaging techniques.
Our proposed method exhibits superior performance on both synthetic and real-works datasets, as evidenced by extensive experimental results.
2
Related Work
NL.S Imaging Systems. Active NLOS imaging systems can be divided into two categories: confocal and non-confocal imaging systems. For the confocal system, the illumination points from the laser and the scanning points collected by the time-resolved sensor coincide. The total acquisition time for transient measurement is proportional to the exposure time per scanning point and the density of the scanning grid, typically ranging from minutes to hours. Different from confocal systems, the detector in non-confocal systems [7 8 9 10] is in array form, such 16x I or x2x The laser illuminates a fixed point on the relay wall and the spad array captures the indirect photon simultaneously. The non-confocal system has the potential for real-time imaging but still faces the following challenges. Accuracy is still traded for speed ( TO 11] Due to the unsatisfactory parameters (low quantum efficiency and fill factor, high dark count and cross-talk effect), non-confocal systems also require relatively long exposure times to achieve reconstruction, e.g., 0.3 FPS in [8) 5 FPS in [TO, [1]. 20 FPS in g More importantly, the price of the SPAD array quite expensive. There are also some special imaging setups using dynamic cues, c.g., key-hole imaging [121, light field tomography [13] and motion deblurring [4.. In this paper, we continue focus on the confocal system and strive to advance the development of dynamic NLOS imaging in terms of imaging compromise and cost expenditure.
Reconstruction Algorithms. The NLOS reconstruction algorithms have made significant progress, encompassing the back-projection [15
T6
171,
linear optimization [18 19
20
21, non-linear
optimization [22 23). wave propagation 7 24 and deep-learning-
methods 25
26 27
28 29]
30
31 32 33 These methods reconstruct promising hidden volumes, contingent upon high-quality
transient measurements. The other studies 2 4 34 attempt to achieve faster system acquisition speeds by using fewer scanning points while still recovering high-quality results. CSA M and FSN 2 explored iterative algorithms with regularization, albeit at the cost of computation time. The deep methods 3 4 address this issue by leveraging the deep-learning technology for a single forward inference. Unfortunately. these methods always neglect the temporal consistency between the neighbouring frames. As an incremental yet crucial advancement, we focus on dynamic NLOS reconstruction and aim to leverage the
information to enhance reconstruction quality. The concurrent works 35 36) employs the dynamic scanning grid and then fuses the multi-frame information, while the scanning grid in this paper is fixed.
3 Preliminary
3.1 Observation Model
The transient measurement, denoted as r, comprises a set of temporal histograms, acquired from the raster-scanning points on the relay wall. We follow the common assumptions of no inter-reflections, no occlusions, and isotropic light scattering within the hidden scene. As depicted in Fig. L given the illuminated point Pt, the continuous transient measurement at the scanning point pa can be expressed as follows:
T(p..) = 1ff rr p(p:) - 5(ri + r. c)d2,
where p denotes the hidden albedo volume, Pt is the target point of the hidden scene Ω, r is the distance between the illuminated and the target points, and r. is the distance between the scanning and the target points. models the light propagation from the relay wall to the hidden object and back the wall. After being captured by the detector within N pulses, the discrete transient measurement i can be accumulated as: F(ps, i) Poisson(E . N [T + 6).. t') + +NNN.) .d).
(2)
3

---

Galvo
SPAD
(a) Synthetic Sequence
(b) Capture Setup
(c) System Setup
Figure 2: (a) Top:
Intensity images of a synthetic sequence. Bottom: Transient slices (x-y) of the transient frames. (b) and (c) presents the capture setup and our self-built imaging system.
where e denotes the quantum efficiency the detector. and d represent the background ambient noise and the dark count of the detector (37), respectively. e indicates that the temporal bin i is sampled from a Gaussian-shaped jitter. By modelling efficiency and jitter, the observation model brought closer the real-world process. For further details about the detection model, refer to [38 26
3.2 Datasets
Synthetic Dataset.
Considering that there are currently no dynamic simulation datasets available for training and evaluation, we modify the time-resolved rendering tool [26) and then simulate the dynamic synthetic dataset. The dataset comprises 265 sequences, which consist of to static objects and 1 dynamic object following 3D helical motion trajectory. Each sequence has 32 frames of transient measurements, with bin width of 33 ps. The toy example is shown in Fig. a). Note that the color is for visualization, the data is in gray-scale. The spatial-temporal resolution of the transient measurement is 128x 128 x512. To enhance the generalization capability of the synthetic data, we incorporate detector jitter provided by [38. during the synthetic sampling process. Additionally, to introduce variability, we randomly assign quantum efficiencies ranging from 1% to 30% for the sequences. To prepare the training data, execute the interval sampling along the spatial dimension for the raw transient to obtain the under-scanning measurement of size 16x 16x512.
Real-world Dataset. 1) System Sctup: For the evaluation real-world data, develop active confocal NLOS imaging system. The prototype is illustrated in Fig. ( The system utilizes a 532 laser (VisUV-532) to generate pulses with a width of 85 picoseconds and a repetition frequency of 20 MHz, delivering an average power of 750 m W. These pulses are directed through a two-axis raster-scanning Galvo mirror (Thorlabs GVS212) towards the relay wall. Subsequently, both direct and indirect diffuse photons are gathered by another two-axis Galvo mirror, coupled into a multimode optical fiber, and then channelled into SPAD detector (PD-100-CTE-FC) with detection efficiency of approximately 45%. The movement of both Galvo mirrors is synchronized and controlled by National Instruments acquisition device (NI-DAQ USB-6343). The TCSPC (Time Tagger Ultra) captures the pixel trigger signals from DAQ, the synchronization signals from the laser, and photon detection signals from the SPAD. The temporal resolution of the overall system is approximately 95 ps. 2) Collection Details: During data collection, the illuminated and sampling points maintain consistent direction but are intentionally offset slightly to prevent interference from directly reflected photons during scanning. We perform a raster scan across 16x 16 square grid of points on the relay wall. Each scanning point is allotted 800 μu for exposure, and the histogram with length of 512 bins and a bin width of 32 ps. Accumulation occurs during the switching process of points in [24]. leading to aliasing in transient measurements. We employ the point-by-poing accumulation method, where data during the jump between scanning points is disregarded. As such setting, we capture 4 video sequences of dynamic NLOS scenes, with each sequence containing approximately 64 frames, and capture rate of 4 FPS.

---

SEManta
SE-aabb
Choss
Spresdiing
Feshure
Extracton
Resuur
Transformation
Refinemant
SF-Mmba
MEMamba
STMembe
SE-Mania
Cioss
Hidden Volume
IMF
Transient Frames
fftn
ifftn
Figure 3: The pipeline of our proposed method. Given three frames of the transient measurements, the target hidden volume, intensity image and depth map, the reference frame are reconstructed.
3.3 State Space Model (SSM)
The State Space Model (SSM) is employed to describe the linear time-invariant systems. The system processes the ID input sequence æx() E R by propagating them through the intermediate hidden states h(t) E R.W, ultimately generating output sequences y(t) e R. Typically, SSM can be expressed as the linear ordinary differential equation:
h (1) Ah(() + Bx((), s(() Ch(t),
(3)
where A E NxN is the state matrix, B, C E Nx are the projection parameters. After discretiza- tion via the timescale parameter [39], Eq i formulated as:
h A + BI, Y/ = Ch+,
where A = exp(AA). (ex))
(AA) (exp(AA
1) AB. Besides, Eq. 4 can be transformed
into convolutional operation which is suitable for hardware. Recently, Mamba [4]] employ the data-dependent mechanisms, including the learnable parameters B, C. and A, as well as the parallel
scanning. Mamba has rapidly become popular in various tasks 41 42
44 due to its linear
computational complexity and global modelling capability.
4 Method
4.1
Overview
We present a groundbreaking approach to dynamic NLOS reconstruction. Our method leverages Mamba to capture Longrrange dependencies within transient data to achieve high-fidelit, recon- structions. To begin, we formalize the dynamic NLOS reconstruction problem. Given a sequence f [Telosesa hxxx where i represents the total number of the transient frames, h and denote the spatial dimensions (height and width) of the t-th transient frame, and T signifies the number of discretized histogram bins along the temporal dimension. The objective is to reconstruct the target hidden volume V [V]iseii- E IR x where H, W, and Z represent the 3D) spatial dimensions of the reconstructed volume. Due to the low spatial resolution of input transient frames, we propose a unified framework that merges transient measurement super-resolution and volume reconstruction. We strategically enhance each transient frame individually and subsequently integrate them before spreading
high spatial resolution, thereby significantly reducing the com- putational burden. Specifically, our method utilizes three adjacent frames as input and predicts the nigh-resolution hidden volume for the reference frame.
Our proposed method leverages multi-stage architecture to achieve high-fidelity dynamic NLOS reconstruction. In the initial stage, the interleaved extraction module utilizes 3D interlaced and dilated convolutions to effectively downsample the temporal dimension of the input transient frames. Next, the spatial-temporal Mamba (ST-Mamba) blocks extract informative features (Fr+a where E { 1, O, }}} by exploiting long-range dependencies within the transient data. Subsequently, the cross ST-Mamba blocks capitalize on the inherent temporal consistency between frames and integrate the multi-fram features to obtain the aligned features F. The target features F, and the aligned features are added together for the enhanced features FÇ Finally, the transient spreading module employs 3D transposed convolution layers with traditional interpolation to generate high-resolution reference transient frame Notably, method recovers data along the spatial dimension well
5

---

MLP
CONY
Tempora
SSM
Spatial
SSM
CONY
Temppe SSM
Spatial
SSM
Norm
Nom
MLP
Activation
MLP
Activation
Fmn
Norm
MLP
CONY
Temporal SSM
Spatial SSM
(a) ST-Mamba
(b) Cross ST-Mamba
Figure 4: The overview of the proposed ST-Mamba (a) and cross ST-Mamba (b).
performs temporal upsampling, resulting in temporal size that matches the input. This recovered temporal information, as highlighted prior work [30], strengthens the constraints between the predicted and ground-truth reference transient frames during training loss construction, ultimately leading to improved reconstruction quality.
After processing transient information, our method proceeds towards volume reconstruction. Given the specific target transient frame -." employ feature extraction module that combines convolu- tional operations with traditional interpolation techniques, ensuring the capture of relevant features while preserving crucial temporal details. Inspired by existing physics-based based methods [26] 45 29 31],
we then utilize a physical prior [46] within a feature transformation module. This module trans- forms the extracted features from spatio-temporal domain into a purely spatial domain. Finally, refinement module refines the transformed features and generates the hidden albedo volume. The intensity image and depth map are then derived from the volume. The details of the ST-Mamba, Cross ST-Mamba, and hidden volume reconstruction will be explored in the following sections.
4.2
Peatial-Temporal Mamba
The Mamba architecture, while powerful for ID sequential data 440] presents challenges for NLOS imaging due unidirectional processing mode. The 3D NLOS transient measurements are inherently partially causal and high-dimensional. Specifically, the histogram of each scanning point along the temporal axis exhibits causality, but the scanning points themselves along the spatial axis non-causal. The non-causality and high dimensionality hinder Mamba from effectively capturing the underlying features. To overcome these limitations, we propose the ST-Mamba and the cross ST-Mamba mechanisms, which specifically designed exploit and integrate the deep features within under-scanning transient measurements for NLOS imaging tasks.
ST-Mamba. The overview of ST-Mamba is presented in Fig. 4a). Given the initial extracted features Fin E Cxxxxxx the ST-Mamba block first conducts the temporal SSM. The normalized input features are reshaped | and then undergo linear projection and ID causal convolution thanks to the histogram of the temporal axis being unidirectional. Then, the output features are reshaped Txxxxxx for the next spatial-SSM, Due the non-causality, we adopt the bidirectional SSM 42 to capture spatial awareness, Finally, the output features are multiplied with gating features from the activation operator and fed to the last linear projection, yielding Ft.. By incorporating mechanisms to handle both causal and non-causal data components, ST-Mamba offers more comprehensive approach for modelling the long-range correlations transient measurement.
Cross ST-Mamba. As discussed above, the deep features from different transient frames are exploited. To integrate the information from adjacent frames, we further introduce the cross ST- Mamba mechanism to align the features for the target frame. As shown in Fig. 4b), the cross ST-Mamba possesses a reference branch and adjacent branch. These two branches share the same gating factor from the reference input. Different from the query mechanism of cross-ane attention, the complementary information between transient frames is integrated by the gating mechanism. Given the reference features F, and neighbouring features Fr+11 the cross ST-Mamba block conducts the temporal and spatial SSM for inputs successively. Then the output reference features and neighbouring features are modulated by the same gating parameter derived from the reference input. Finally, the modulated features are added together for the aligned features. The output features F generated after linear projection and shorteut.

---

4.3 Hidden Volume Reconstruction
According to Eq. [] the forward model can be simplified into 3D convolution form through resampling along the temporal axis for transient data and resampling along the depth axis for hidden volume. The solution to NLOS reconstruction is an inverse problem, involving the PSF of the imaging system [8] Without additional computation, PSF can be expressed explicitly under the specific imaging setup, which commonly introduced to learning- baaning-basedd reconstruction methods [26 45 29 II. In this study, we also incorporate the physical-prior [46,, illustrated in the bottom right part of Fig. 3 For the hidden volume reconstruction, the feature extraction module comprises three 3D residual blocks for extracting shallow features and downsampling along the temporal axis. The volume refinement module is composed of three 3D convolutions and three interlaced 3D residual blocks. Each residual block comprises two 3D convolutions followed by ReLU activation and a residual connection. The extracted features from the target high- high-resolution transient frame are convolved by the illumination function to access the phasor field. After resampling, the
features perform the Fourier transform, element-wiss multiply the inverse PSF in the frequency domain, and then perform the inverse Fourier transform to exhibit the 3D spatial features. Due to the large domain gap between the synthetic and real-world data, this methodology trades the Fourier computational burden for generalizability. which has been widely utilized in deep methods.
4.4 Loss Function
The total loss function Liotat is composed of three components: the measurement recovery loss Lm. the volume reconstruction loss Lu, and the regularized loss Ly:
Ltotal = L. + 3L, + Thr,
(5)
C. - Cpf + tafe, L, Lint + a2Ldep L, Lia+ azLiv:
where the parameters B, and a contribute the corresponding loss. Among these loss items, the phasor field loss Epf. the transient loss L,, the intensity loss Lint, and the depth loss Cdep formulated as follows:
Eps = || E P(L,o) - Tit * P(l,o)ll, L, 4= 1=|| - 7. 112,
(6)
= Lint = -| 10 Lap== 1
where t. I, and D denote the transient measurement, intensity image and depth map of the hidden volume. gl denotes ground truth. o(l,o) represents the illumination function (46) P(L, o)
2.. eict - De the central frequency depended on the wavelength, is the standard deviation
of the Gaussian function. denotes the convolution along the temporal dimension, leading to the highlight of useful information in the frequency domain. Inspired by (4, we utilize the local similarity loss Lts and the total variation loss L.v for constructing the last regularized loss L,., For more details about the loss items, see the supplementary.
5 Experiments
5.1 Experimental Details
Implementation. Our method is implemented using PyTorch, trained on the synthetic data, and then directly tested on the real-world data. During training, we employ the Adam W (47 as the optimizer with learning rate of 10-4 and weight decay of 0.95. To enhance visual clarity, the final output spatial resolution is set to 128x 128, based on the input size of 16x 16. All the experiments are conducted on the NVIDIA A100 GPUs, with batch size of 4. We utilize 150 sequences for training and 17 sequences for synthetic testing. Besides, we utilize 4 sequences for real-world evaluation. The hyper-parameter and 7 are set to and 10-5, ar, a22 0t are set to 0.5, 1, and 0.1, respectively.
Baselines. We compare our method with existing baselines, including the traditional methods LCT [1]. FK (24), and RSD [46), the iterative method CSA [1] as well as the deep- deep-learning based methods including LFE 26). I-K 31I, and USM (4. The baseline methods are implemented following their publicly available codes. Apart from the multi-frame version, we also provide the single-frame version Ours-S (excluding cross ST-Mamba) for a comprehensive comparison. Note that only CSA, USM, and our method are specifically designed for reconstruction from the under-
7

---

Table 1: Quantitative comparison of the existing methods on the synthetic data. The spatial
resolution of the input and output is 16x 16 and x28x 128, respectively. The best in bold. The second
with underline. Note that only methods with gray annotation are designed for reconstruction from under-scanning measurements.
Intensity
Depth
Methods
Architecture
PSNRT
SSIM
RMSE
MAD
LCT
18
Linear Optimation
17.25
8.81
0.4355
0,4103
RSD
46
Phasor Field Waves
19.00
13.48
0.4043
0.3844
FK
24
F-k Migration
20.90
49.84
0.3930
0.3756
LFB
226
Physical-based
23.20
78.02
0.0993
0.0526
I-K
31
Physical-bass
23.22
79.79
0.1011
0.0468
CSA
Linear Optimation
20.70
71.13
0.2647
0,1090
USM
Physical-based
23.80
80.85
0.0945
0.0432
Ours-S
Physical-base -bssed
23.97
81.35
0.0939
0.0400
Ours
Physical-basee
24.46
84.08
0.0880
0.0397
#13
#15
#17
LCT
RSD
FK
CSA
USM
OursS
Ours
Figure 5: Qualitative results of two synthetic sequences. The symbol "#' denotes the frame. The input spatial resolution is 16x 16, and the output spatial resolution is x28x 128.
scanning measurement. For the other baselines, the inputs are interpolated to the target resolution 128x 512 for the final comparison.
Evaluation Metrics. The synthetic quantitative evaluation comprises two categories. For intensity images, we compute the peak signal-to-noise ratio (PSNR) and structural similarity metrics (SSIM), averaged across the corresponding test samples. For depth maps, we calculate the root mean square error (RMSE) and mean absolute distance (MAD).
5.2 Synthetic Results
Our method demonstrates superior performance against existing approaches, as shown by the quanti- tative results in Tab. [ Notably, our method excels in both intensity and depth estimation, achieving significantly better results than baseline methods. The single-frame version of our method, Ours-S, which leverages the core ST-Mamba architecture, also outperforms other methods, demonstrating the effectiveness of ST-Mamba in exploiting long-ranging causal data. Furthermore, the multi-frame model surpasses the single-frre frame model across all metrics, showcasing the strength of the proposed

---

Table Ablation results on the loss items and spatial-temporal Mamba mechanism.
ST-Mamba
Loss Items
Intensity
Depth
Spatial
Temporal
Lint.dep
Lt
Lis,to
PSNRT
SSIM
RMSHL
MADI
S-Mamba
T-Mamba
24.19
82.75
0.0946
0.0409
S-Mamba
T-Mamba
/
/
24.18
83.10
0.0914
0.0409
S- S-Mamba
T-Mamba
/
x
24.47
83.07
0.0905
0.0404
S S-Mamba
T-Mamba
/
/
/
/
24.46
84.08
0.0880
0.0397
T-Mamba
T-Mamba
/
24.32
83.68
0.0898
0.0398
T-Mamba
24.38
82.49
0.0921
0.0478
S- S-Mamba
S-Mamba
24.31
83.08
0.0911
0.0496
S-Mamba
24.36
82.82
0.0938
0.0440
#
V
U
U
V
U
U
U
T
T
T
T
T
T
T
T
T
T
U
U
U
TT
V
UT
68
LCT
RSD
FK
LFE
T-K
CSA
USM
OUES-S
Ours
Scene
Figure 6: Reconstructed results from real-world measurements captured by imaging system. The spatial scanning grid is 16x 16, and the output spatial resolution of the hidden volume is 128x 128.
cross ST-Mamba in integrating information from multiple transient frames. The qualitative results are presented in Fig. Except for the dynamic motorbikes, the synthetic sequences contain the background static objects such as cars, buckets, helmets, and bottles. Due to the various quantum efficiencies of sequences, the traditional methods reconstruct the blurry results lacking details. The dese- method LFE (26) and I-K (31 recover the main structure but still miss details. CSA M generates artifacts around the hidden objects, while USM [4] performs better in reducing background noise but struggles with fine structure. In contrast, our method performs the best in both static and dynamic scenarios, with higher fidelity and more details. The promising reconstruction results underscore the ability of the proposed method to capture the dynamic nature effectively.
5.3 Ablation Studies
To assess the effectiveness of the proposed loss functions and the spatial-temporal Mamba mechanism, we conduct the ablation studies with Tab. listing the quantitative results.
Loss Items. It can be concluded that incorporating the regularized loss generally improves metrics, with a minor trade-off PSNR suggesting resistance to overfitting. This phenomenon has also been verified in USM 4 A significant performance boost is observed when the phasor field loss included, which indicates that enforcing constraints within the phasor field highlights valuable information in the transient measurements, leading to more accurate reconstructions.

---

Temporal Mamba. To investigate the efficiency of individual spatial Mamba (S-Mamba) and temporal Mamba (T-Mamba) components for NLOS reconstruction, we operate the Mamba along the spatial and temporal dimensions, respectively. The temporal Mamba-based model might outperform the spatial Mamba-based model on single scanning points. However, it lacks spatial awareness, hindering its ability to capture the overall structure. When both spatial and temporal Mamba are employed, the model achieves the best performance across all metrics, demonstrating the advantage of capturing information from both spatial and temporal domains.
5.4 Real-world Results
To evaluate the generalizability of our method, we test the models on world transient videos captured by our imaging system. The quantitative results shown in Fig. 6 where the top three rows exhibit three sequences of planar objects in rigid motion, while the bottom three rows depict a sequence with non-rigid motion. Traditional methods reconstruct the single object with considerable noise but struggle to recover the distant moving letter in multi-object scenes. Deep learning approaches, while achieving cleaner backgrounds, often lead to incomplete simplified reconstructions of the hidden scene. Although CSA []] and USM [4] are designed specifically for mnder-scanning measurements, they lose the adaptability and generalization ability under low SNR conditions. In contrast, our method demonstrates superior performance, capturing finer geometric structures and richer details. Furthermore, our methods deliver robustness non- rigid scenarios, as exemplified by the clear recovery of arm movements. Besides, our multi-frame model outperforms the single-frame frame model on more detailed information and less noise, showeasing the effectiveness of cross ST-Mamba. These promising results highlight the strong representation capability and generalization ability of our proposed method to real-world scenarios. More qualitative results can be seen in the supplementary material.
overly
6 Conclusion and Discussion
Conclusion. This work presents novel learning-based framework for dynamic reconstruction in confocal NLOS imaging. By leveraging the powerful ST-Mamba and cross ST--aama,, the proposed method effectively captures both long-ranging causal information while exploiting the natural consistency within transient video sequences. Extensive evaluations on a newly created dataset, encompassing both synthetic and real-world scenarios, demonstrate the superiority of our method in achieving high-quality reconstructions compared to existing approaches. We believe the proposed method presents a significant step forward dynamic NLOS reconstruction, unlocking the possibilities for a wide range of real-world applications in various fields.
Linitation.
The temporal consistency has shown significant potential in dynamic NLOS reconstruc- tion. Fusing information in the spatial-temporal domain and purely spatial domain may be a more effective approach. The feature transformation module relies on Fourier operation utilize the PSF of the specific imaging system, which consumes a computational burden. It is essential to develop lightweight transformation to reduce this burden and allow for more network design.
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grant 62131000 and Innovation Program for Quantum Science and Technology under Grant 2021ZD0300300
References
[1] Jun-Tian Ye, Xin Huang, Zheng-Ping Li, and Feihu Xu. Compressed sensing for active non-line-of-sight imaging. Optics Express. 1949):1749-173. 2021.
[2] Xintong Liu, Jianyu Wang. Leping Xiao, Xing Fu, Lingyun Qiu, and Zuoqiang Shi. Few-shot non- line- of sight imaging with signal-surface collaborative regularization. In Proceedings of the Conference on Computer Vision and Pattern Recognition, 2023.
10

---

[3] Jianyu Wang, Xintong Liu, Leping Xiao, Zuoqiang Shi, Lingyun Qiu, and Xing Fu, Non-line-of-sight imaging with signal superresolution network. In Conference on Computer Vision and Puttern Recognition, 2023.
[4] Yue Li, Yueyi Zhang, Juntian Ye, Feihu Xu, and Zhiwei Xiong. Deep non-line-of-sight imaging from under-seanning measurements. Advances in Neural Information Processing Systems, 36, 2023.
[5] Xingyu Cui, Huanjing Yue, Song Li, Xiangjun Yin, Yusen Hou, Yun Meng. Kai Zou, Xiaolong Hu, and Jingyu Yang. Virtual scanning: Unsupervised non-line-of-sight imaging from irregularly undersampled transients. In Advances in Neural Information Processing Systems, 2024.
[6] In Cho, Hyunbo Shim, and Scon Joo Kim, Learning to enhance aperture phasor field for non-line-of -fighh imaging. In European Conference Computer Vision, pages 72 89, 2024.
[7] Xiaochun Liu, Sebastian Bauer, and Andreas Velten. Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems. Nature Communications, 1 11(1):1-3,, 2020.
[8] Chenfei Jin, Meng Tang, Legeng Jia, Xiaorui Tian, Jie Yang, Kai Qiao, and Siqi 7hang. Scannerless oon-line-of-sight three dimensional imaging with a 32x32 spad array. arXiv preprint arXiv:2011.05122 2020.
[9] Chengquan Pei, Anke Zhang. Yue Deng, Feihu Xu, Jiamin Wu, David, Lei Li, Hui Qiao, Lu Fang. and Qionghai Dai. Dynamic non-site-of-sight imaging system based on the optimization of point spread functions. Optics Express, 29(20):32349 32364, 2021.
[10] Ji Hyun Nam, Eric Brandt, Sehastian Bauer, Xiaochun Liu, Eftychios Sifakis, and Andreas Velten, Real- time non-line-of-sight imaging of dynamic scenes. arXiv preprint arXiv:2010. 12737, 2020.
[11] Ji Hyun Nam, Eric Brandt, Sebastian Bauer, Xiaochun Liu, Marco Renna, Alberto Tosi, Eftychios Sifakis, and Andreas Velten Low-latency time-of-fligee non-line-of-sight imaging at 5 frames per second, Nature communications, 12(1);6526, 2021.
[12] Christopher A Metzler, David Lindell. and Gordon Wetzstein. Keyhole imaging: non-line-of-sige sight imaging and tracking of moving objects along a single optical path. IEEE Transactions on Computational Imaging, 7:1-12, 2020.
[13] Xiaohua Feng and Liang Gao. Ultrafast light field tomography for snapshot transient and oo--lin-off-sght imaging. Nature communications, 12(1):2179, 2021.
[14] Javier Grau Chopite, Patrick Hachn, and Matthias Hullin. Non-line-of-sight estimation of fast human motion with slow scanning imagers. In European Conference on Computer Vision pages 176-194, 2024.
[15] Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Vecraraghavan, Moungi G Bawendi, and Ramesh Raskar. Recovering three-dimensional shape around a comer using ultrafast time-of- of-fighhttt imaging. Nature commanications, 3(1):745, 2012.
[16] Mauro Buttafava, Jessica Zeman, Alberto Tosi, Kevin Elicciri, and Andreas Velten. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Optics express, 23(16):0997-21011, 2015.
[17] Victor Arellano, Diego Gutierrez, and Adrian Jarabo. Fast back pooj-projection for non-line of sight reconstruc- tion. Optics Express. 25(10): 1111100:1154-111 -15-11533, 2017.
[18] Matthew O Toooole, David B Lindell, and Gordon Wetzstein, Confocal non-line-of-sight imaging based on the light-coone transform. Nature, 555(76966:338-41, 2018.
[19] Sean Young, David B Lindell, Bernd Girod, David Taubman, and Gordon Wetzstcin. Non-linc-of-sipp sight surface reconstruction using the directional light-cone transform. In Conference on Computer Vision and Pattern Recognition, 2020.
[20] Xintong Liu, Jianyu Wang, Zhupeng Li, Zuoqiang Shi, Xing Fu, and Lingyun Qiu. Non-line-of-sight reconstruction with sognal-object collaborative regularization. Light: Science & Applications, 10(1): 10(1):198, 2021.
[21] Cheng Wu, Jianjiang Liu, Xin Huang, Zheng-Ping Ping Li, Chao Yu, Jun- Tian Ye, Jun Zhang, Qiang Zhang Xiankang Dou, Vivek K Goyal, et al. ofn--ine-of-siiht imaging over 1.43 km. Proceedings of the National Academy of Sciences, 118(10):20224468) 18, 2021.
[22] Feihu Xu, Gal Shulkind, Christos Thrampoulidis, Jeffrey H Shapiro, Antonio Torralba, Franco NC Wong. and Gregory W Wornell. Revealing hidden scenes by photon-efficient occlusion based opportunistic active imaging. Optics Express, (19999999999222 2018

---

[23] Felix Heide. Matthew O'Toole. Kai Zang, David B Lindell, Steven Diamond, and Gordon Wetzstein. Non-line-of-sighe imaging with partial occluders and surface normals, ACM Transactions on Graphics. 38(3):1-10, 2019.
[24] David Lindell, Gordon Wetzstein, and Matthew O'Toole. Wave-based non con-line-offa sight imaging using fast fk migration. ACM Transactions on Graphics, 38(4):1-13, 2019.
[25] Javier Grau Chopite, Matthias B Hullin, Michael Wand, and Julian Iseringhausen. Deep non-line-of-sight reconstruction. In Conference on Computer Vision and Pattern Recognition, 2020.
[26] Wenzheng Chen, Fangyin Wei, Kiriakos N Kutulakos, Szymon Rusinkiewicz, and Felix Heide. Learned feature embeddings for non- non-line-oie ofne--f--ighh imaging and recognition. ACM Transactions on Graphics, 3-86);1-18, 2020.
[27] Siyuan Shen, Zi Wang. Ping Liu, Zhengqing Pan, Ruiqian Li, Tian Gao, Shiying Li, and Jingyi Yu. Non-line-of-sight imaging via neural transient fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(7):2257-2268, 2021.
[28] Ping Liu. Yanhua Yu, Zhengqing Pan, Xingyue Peng. Ruiqian Li, Yuehan Wang, Jingyi Yu, and Shiying Li. Hiddenpose: Non-line-of-sighh 3d human pose estimation. In IEEE International Conference on Computational Photography, pages 1-12, 2022.
[29] Yue Li, Jiayong Peng, Juntian Ye, Yueyi Zhang, Feihu Xu, and Zhiwei Xiong. Nlost: Non-line-of-sight imaging with transformer. In Conference on Computer Vision and Pattern Recognition, 2023.
[30] Jinye Miao, Enlai Guo, Yingjie Shi. Fuyao Cai, Lianfa Bai, and Jing Han. Super-resolution con-line-of-sipee imaging based on temporal encoding. Opticy Express. (1(4))400235 40225-44888444488 2023.
[31] Yanhua Yu, Siyuan Shen, Zi Wang. Binbin Huang. Yuehan Wang, Xingyue Peng, Suan Xia, Ping Liu, Ruiqian Li, and Shiying Li. Enhancing non-line-of-sight imaging via learnable inverse kernel and attention mechanisms. In Proceedings of the International Conference on Computer Vision, 2023.
[32] Xiongfei Su. Yu Hong, Jun-tian Ye, Feihu Xu, and Xin Yuan. Model-guided iterative diffusion sampling for nlos reconstruction. IEEE Journal of Selected Topics in Quantum Electromics, 2024.
[33] Xiongfei Su, Yu Hong, Juntian Ye, Feihu Xu, and Xin Yuan. Multi-scale iterative model-guides unfolding network for nlos reconstruction. In Computer Graphics Forum, volume 42, page e e14958. Wiley Online Library, 2023.
[34] Mariko Isogawa, Dorian Chan, Ye Yuan, Kris Kitani, and Matthew O'Toole. Efficient -on-line-of-sight imaging from transient sinograms. In European Conference Computer Vision, pages 193-208, 2020.
[35] Juntian Ye, Yu Hong. Xiongfei Su, Xin Yuan, and Feihu Xu. Plug-and-pla algorithms for dynamic non-line-of-sight imaging. ACM Transactions on Graphics, 2024.
[36] Jun-Tian Ye, Yi Sun, Wenwen Li, Jian-Wo Zeng, Yu Hong, Zheng-Ping Xin Huang, Xianghui Xue, Xin Yuan, Feihu Xu, et al. Real-time non-line-of-sight computational imaging using spectrum filtering and motion compensation. Nature Computational Science, pages 1-8, 2024.
[37] Danilo Bronzi, Federica Villa, Simone Tisa, Alberto Tosi, and Franco Zappa. Spad figures of merit for photon-counting, photon-timing, and imaging applications: review. IEEE Sensors Journal, 16(1):3-12, 2015.
[38] Quercus Hernandez, Diego Gutierrez, and Adrian Jarabo. A computational model of single-photon avalanche diode sensor for transient imaging. arXiv preprint arXiv: 020635 2017.
[39] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. The International Conference on Learning Representations, 2022.
[40] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. First Conference on Language Modeling, 2024.
[41] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.1016 2024.
[42] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang. Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In International Conference on Machine Learning, 2024.
12

---

[43] Zifu Wan, Yuhao Wang, Silong Yong, Pingping Zhang. Simon Steppuctis. Katia Sycara, and Yaqi Xie. Sigma: Siamese mamba network for multi-modal semantic segmentation. In Winter Conference Applications of Computer Vision, 2025.
[44] Dingkang Liang, Xin Zhou, Wei Xu, Xingkui Zhu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, and Xiang Bai. Pointmamba: simple state space model for point cloud analysis. In Advances Neural Information Processing Systems, 2024.
[45] Fangzhou Mu, Sicheng Mo, Jiayong Peng, Xiaochun Liu, Hyun Nam, Siddeshwar Raghavan, Andreas Velten, and Yin Li. Physics to the rescue: Deep con-line-off-fee reconstruction for high- imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1- 12, 2022.
[46] Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jarabo, Diego Gutierrez, and Andreas Velten. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature, 572(7771):620 623, 2019.
[47] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018.
13

---

A Supplemental material
A.1 Ablation studies on causality and different modules
Table 3: Ablation studies causality and different modules. * denotes that the method possesses the
same number of SSMs as our final method.
Methods
Intensity
Depth
ID
Encoder
Fusion
Causality
PNSRT
SSIM
RMSE.
MADI
Mamba
Mamba
X
23.94
80.59
0.0964
0.0572
1
VIT
Mamba
23.77
80.13
0.0983
0.0490
2
Mamba*
V
23.97
81.35
0.0939
0.0400
3
Mamba VIT
/
24.32
82.47
0.0886
0.0456
4 Mamba Mamba
24.46
84.08
0.0880
0.0397
As shown in Tab disabling the causal operation our method results in a significant drop in performance metrics, demonstrating the effectiveness of our approach in exploring causality. For more comprehensive ablation studies on ST-Mamba and Cross ST-Mamba, we individually replaced the ST-Mamba (blue blocks in Fig 3 and Cross ST-Mamba (orange blocks in Fi with plain attention block and a cross attention block. By comparing IDs 1, 3, and 4, it is evident that our method employed. This does not imply that the cross-attention mechanism is unsuitable for maintaining spatio-temporal consistency; rather, indicates that using Cross ST-Mamba is more effective for transient data with partial causality. Additionally, we excluded only the Cross ST-Mamba while maintaining the same number of SSMs as in our final method (ID 2). This further demonstrates the effectiveness of the proposed Cross ST-Mamba.
performs best when both ST-Mamba and Cross ST-Mamba
A.2 Comparison of the SR module separately with USM 4
CV
Figure 7: The reconstruction results via the traditional method (RSD) for the high-resolution transient measurements, which are recovered by the SR networks of USM and Ours.
The capabilities of the transient super-resolution networks of USM 4 and our method are further compared. To obtain quantitative results, we first generate the recovered high-resolution transient measurements and then conduct the reconstruction using the traditional method RSD[ZI. instead of the deep neural network. For USM, the metries PSNR/SSIM/RMSE/MAD are 19.25/18.99 0wt../.....22....... 1000. For method, the metrics PSNR/SSIMRMSE/MIAA are 21 21.48/17.0800.2. 10000.0022000001777777 Except for SSIM, our method surpasses USM. The decrease in SSIM may originate from the artifacts interference.
To provide a more comprehensive evaluation, we introduce another metric ACC, from 11, which indicates the foreground recovery degree. The ACC for USM is 10.49, while for our method, it is 31.22, demonstrating that our method generates cleaner and higher-quality transient measurements. Qualitative results provided in Fig. further demonstrating our superiority comprchensively.
14

---

A.3 More real-world results
More reconstructed intensity images from the real-world data are shown in Fig. and Fig. Our method achieves excellent reconstruction results in simple planar scenes, such as the clear letter structures in Fig. 8 and Fig. g
It also delivers outstanding reconstruction performance in complex
scenes as shown in Fig. 6
411
U
U
U
U
U
U
U
U
U
u
0
U
W
u
T
T
T
T
T
T
T
T
T
T
T
T
T
T
T
T
T
T
T
LCT
RSD
FK
LPE
I-K
CSA
USM
Ours-S
Ours
Soene
Figure Reconstructed intensity images from the real-world transient videos captured by our imaging system.
U
u
UT
L
U
U
UT
UT
U
U
U
J
VT
U
U
VT
u
u
LCT
RSD
LFE
I-K
CSA
USM
Ours-S
Qurr
Scene
Figure 9: Reconstructed intensity images from the real-world transient videos captured by our imaging system.
15

---

A.4
Loss Function
As discussed in Sec. as 4
The local similarity loss Lts and the total variation loss Ltv is formulated
Lis = Ex Ex y, p 2, W1l1,
Lio E, +. p(z,, 1a ||, 1,2) p(x, v. U,)1
(7)
11 y: 2)
where p(z, y. 2) indicates the volume at position (zz y, 2), p(a, y, 2, k) represents the volume block centered at (It y, 2) with size k W refers to the Gaussian window with size k.
A.5
Computational Memory and Inference Time
Table 4: The inference time and memory of different models. Note that only methods with gray annotation are specifically designed for NLOS imaging from under-scanning measurement.
Method
LCT [8]
FK [24]
RSD 46
LFE 26
I-K
31
CSA [
USM 4
Ours-S Ours
Time (s)
0.034
0.061
0.038
0.031
0.032
0.149
0.198
0.359
Memory (M)
6016
8056
10344
4646
4934
5306
8362
16684
17162
The inference memory and inference time of the models are listed in Tab, 4 It is indeed that using multi-frame information will significantly increase the inference time, but the inference memory usage still within the range of consumer-grade GPU, making it suitable for real-world applications.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Sec. d
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
No theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: See Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification:
NLOS
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuii wwwtc//ggui////eeeeceoocc CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Sec. 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Tab.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
19

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Tab.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::::///uiis...eeee http:p/.......biii.ttttcooooeee.
Answer: [Yes]
Justification: Follow the ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
20

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthcee....c... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
21

---

Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "tZADpSUYCy", "reviewer": "Reviewer_j9gw", "summary": "The paper proposes a dynamic non-line-of-sight imaging approach that uses Mamba to enforce spatio-temporal consistency (called ST-Mamba). This is done using a mutlistage model that takes in a set of transient frames, performs temporal downsampling, followed by spatial and temporal SSM per frame and across frames to get a high-resolution transient frame. Finally, the volume is reconstructed from the high-resolution transient frame using a physical prior. The paper uses a custom synthetic temporal dataset and a small real-world dataset for quantitative and qualitative evlauation. Experimental results show that the proposed method performs better than other NLOS methods for dynamic scenes and ablation experiments show the effectiveness of ST-Mamba.", "questions": "What’s the motivation for using SSMs to enforce spatio-temporal consistency?\n\nHow much is the cross ST-Mamba component contributing?", "limitations": "The authors identified computational cost as a limitation of the work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Dynamic NLOS is a challenging problem primarily due to sparse transient data. The paper uses spatio-temporal consistency to increase the resolution of the transient image and performs the final reconstruction. The data is causal in the temporal dimension but non-causal in the spatial dimension which limits the effectiveness of standard Mamba. ST-Mamba first performs temporal SSM followed by bidirectional SSM to capture the spatio-temporal features. Furthermore, to aggregate information across frames, the cross ST-Mamba module fuses the output of two ST-Mamba operating on temporal frames.\n\nThe paper utilizes synthetic data for training and real data for testing. The synthetic dynamic dataset can be useful for other dynamic NLOS research.", "weaknesses": "Only qualitative results are provided for real dataset and without any video results difficult to judge the quality of the output.\n\nSSMs are primarily motivated for long sequences. The proposed method uses three frames as input and the paper doesn’t motivate why SSMs are needed for spatio-temporal consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are highly encouraged by the positive recommendations and comments from the reviewer. The main concerns are addressed below.\n\n__Q1:__ Lack video results.\\\n__Reply:__ We sincerely appreciate your kind reminder. After preparing the video as requested, however, we find that external links are not allowed in rebuttal, even with an anonymous one. We will include the qualitative video results in the final supplementary materials.\n\n__Q2:__ Motivation for using SSMs to enforce spatio-temporal consistency.\\\n__Reply:__ There are two points to address:\n\n1) By employing a well-designed structure to fully exploit the causality of transient data, our single-frame method (without cross ST-Mamba) outperforms the existing methods. Our experimental results demonstrate that SSMs have significant application potential for transient data with partial causality. Therefore, we aim to use SSMs to further maintain spatio-temporal consistency. \n\n2) Additionally, we conduct an experiment using cross attention to maintain spatio-temporal consistency, with results listed in Tab. 1 (ID 3) of the rebuttal ‘global’ PDF. It can be seen that using Cross ST-Mamba (ID 4) yields the best results. This does not imply that cross-attention mechanism is unsuitable for maintaining spatio-temporal consistency; rather, it indicates that using Cross ST-Mamba is more effective for transient data with partial causality.\n\n__Q3:__ Contribution in cross ST-Mamba component.\\\n__Reply:__ As shown in Tab.1 of the original manuscript, we provide the results of the single-frame version of our method, Ours-S. Note that Ours-S excludes the cross ST-Mamba \nbut possesses the same number of SSMs as Ours.\nThe multi-frame model (Ours) surpasses the single-frame model across all metrics by a large margin, which showcases the strength of the proposed cross ST-Mamba in integrating information from multiple transient frames.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XNdOeW1kYe", "reviewer": "Reviewer_CVND", "summary": "This paper introduces the first Mamba-based method for dynamic NLOS imaging, featuring three key modules: the spatial-temporal Mamba, the cross ST-Mamba, and the phasor field loss. The spatial-temporal Mamba extracts and integrates both causal and non-causal transient components, while the cross ST-Mamba combines deep features from adjacent frames. The phasor field loss ensures the consistency of transients in the phasor field. Evaluations on synthetic and self-captured real-world transients demonstrate that the proposed method outperforms existing approaches. Ablation studies further validate the effectiveness of the proposed components.", "questions": "Please refer to Weakness.", "limitations": "Please refer to Weakness.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1)\tThe proposed method of leveraging adjacent frame information and cross Mamba to enhance single-frame reconstruction is outstanding. As hidden objects move, fixed few-shot scanning points can capture information that other frames miss, which is crucial for recovering lost signals.\n(2)\tThis paper introduces deep learning into dynamic NLOS imaging for the first time. Its capability for single-pass rapid inference helps advance dynamic NLOS imaging towards real-time imaging.\n(3)\tThe captured real-world dataset will be a valuable resource for future research in dynamic NLOS imaging.", "weaknesses": "(1)\tUsing cross-mamba to extract inter-frame causality in transients is understandable. However, the values in single-frame transients  are based on the distance from the hidden object to the wall. This seems to be the spatial correlation rather than time-resolving causality. Please provide further explanation to justify the motivation for using ST-Mamba to process single-frame transients.\n(2)\tLines 180-184 are unclear. Restoring deep features of transients to the original resolution seems to be a common practice. Can transient recovered be done directly at the original resolution? The temporal information in low spatial-resolution transients might be compromised by down-sampling and up-sampling operations.\n(3)\tThe framework is similar to USM, consisting of a transient super-resolution network and a volume reconstruction network. The proposed components are mainly applied to the transient super-resolution network. Please compare with USM's recovered high spatial-resolution transients to highlight the proposed method’s contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are highly encouraged by the positive recommendations and comments from the reviewer. The main concerns are addressed below.\n\n__Q1:__ The explanation for the causality in transients.\\\n__Reply:__ First, let us review the definition of causal signals: Causal signals refer to signals where the output at any given time depends not only on the current input but also on past inputs. These signals exhibit a cause-and-effect relationship where the effect (output) is directly influenced by prior causes (inputs). In other words, the response of a system or process at any time is a function of its history. \\\nNext, let us apply the definition to transient measurement and explain causality. For each sampling point, the temporal histogram indeed contains all the geometry information of the hidden object because the spherical waves are scattered back by the illuminated object. Considering any point on the temporal axis of the histogram, the value at this point is only related to a target point on the hidden object and any point closer to the relay wall than the target point (shorter flight). Clearly, this conforms to the definition of causality. It is also worth mentioning that the histogram is obtained through periodic cumulative accumulation, which is a practical operation to compromise the dead time of SPAD. This does not affect the causal nature. \\\nWe will include this further explanation in the manuscript.\n\n__Q2:__ The operations on the temporal dimension.\\\n__Reply:__ Of course, the transient data can be processed while always maintaining the temporal dimension.\nHowever, performing convolution operations, dot product operations, etc., on three-dimensional data consumes a significant amount of computational resources. Therefore, it is NOT a common practice not design networks directly and continuously at high dimensions.\\\nTo some extent, we agree that _The\ntemporal information in low spatial-resolution transients might be compromised by down-sampling and up-sampling\noperations_. As discussed above, this is the trade-off between performance and consumption. We emphasize this point to highlight that the reconstruction of the temporal dimension is crucial in transient measurement recovery tasks, as validated by the [1].\n\n[1] Jinye Miao, Enlai Guo, Yingjie Shi, Fuyao Cai, Lianfa Bai, and Jing Han. Super-resolution non-line-of-sight imaging based on temporal encoding. Optics Express, 31(24):40235–40248, 2023.\n\n__Q3:__ Comparison of the SR module separately with USM.\\\n__Reply:__ As suggested, we compare the capabilities of the transient super-resolution networks of USM[1] and our method. \nTo obtain quantitative results, we first generate the recovered high-resolution transient measurements and then conduct the reconstruction using the traditional method RSD[1], instead of the deep neural network. For USM, the metrics PSNR/SSIM/RMSE/MAD are 19.25/18.93/0.2206/0.1000. For our method, the metrics PSNR/SSIM/RMSE/MAD are 21.48/17.08/0.2120/0.0917. Except for SSIM, our method surpasses USM. The decrease in SSIM may originate from the artifacts interference. \\\nTo provide a more comprehensive evaluation, we introduce another metric ACC, from[2], which indicates the foreground recovery degree.\nThe ACC for USM is 10.49, while for our method, it is 31.22, demonstrating that our method generates cleaner and higher-quality transient measurements. Qualitative results are provided in Fig.1 of the rebuttal 'global' PDF, further demonstrating our superiority comprehensively.\n\n[1] Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jarabo, Diego Gutierrez, and Andreas Velten. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature, 572(7771):620–623, 2019.\\\n[2] Jun-Tian Ye, Xin Huang, Zheng-Ping Li, and Feihu Xu. Compressed sensing for active non-line-of-sight imaging. Optics Express, 29(2):1749–1763, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k8Of5KOJsE", "reviewer": "Reviewer_oqBG", "summary": "The paper tackles the problem of non-line-of-sight imaging. As a spin on the commonly attempted task, they solve the problem in dynamic scenes and use transients captured at different time points to aid the recovery of the scene at a specific time point. \n\nTo solve the problem they introduce a Mamba inspired architecture, which takes in 3 transient frames. These three transient frames are processed by ST-Mamba and Cross ST-Mamba blocks to produce a high-resolution transient frame, which is then fed into a physically based network to recover hidden volume, intensity, and depth maps. \n\nThe authors train the model on a custom simulated dataset, the model is also evaluated on a self-captured dynamic transient dataset.", "questions": "1) The authors show the baseline Our-S, where multiple frames are not given to their method. However another apples-to-apples comparison might be if more data was provided to the baseline methods, this could mean testing the other methods not on spatially upsampled data, but on higher res data. In some sense this would mean both methods are tested on “equal” amounts of data. This could be done by keeping the total exposure time of the frame constant, (since otherwise rolling shutter-type motion blur artifacts might crop up I think?), but just using more sampling points. Have the authors tried this/related setups?", "limitations": "Limitations discussed in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: I like the problem being attempted in the paper and enjoy the merger between the fields of computational imaging + machine learning, which I think is underexplored. I think the authors also put a good spin on the problem, by performing NLOS in dynamic scenes. I also think the dataset, given the authors release it would be a useful contribution. \n\nQuality: I think the experiments are well done, the baseline Ours-S is especially informative, as it shows that the architecture is also useful on its own when you don’t have access to transients at other time points.", "weaknesses": "Quality: I have a slight issue with the authors calling the method “Dynamic NLOS”. If I understand correctly the method can use transients from different time points (one before and one after) to reconstruct the scene at a specific time point. The method does not automatically reconstruct the scene in time. To actually recover the scene in time, the authors would have to pass in the 3 sets of frames sequentially and get a separate reconstruction in time, is that right? I think the method would be better called “NLOS imaging by using dynamic cues” or something of this sort. \n\nIn any case even though the method is not doing Dynamic NLOS, I think the authors should discuss prior art on Dynamic NLOS and specify how their work is different. This also goes for the datasets, since it is a contribution the authors should specify what it provides over other dynamic NLOS datasets. Here is a couple of papers the authors should cite, but there are many more: \n\n1) Metzler et al., Keyhole Imaging: Non-Line-of-Sight Imaging and Tracking of Moving Objects Along a Single Optical Path, IEEE Transactions on Computational Imaging, 2021.\n\n2) Nam et al., Real-time Non-line-of-Sight imaging of dynamic scenes\n\nClarity: I think the paper is mainly geared towards the Computational Imaging community, thus I think the authors would be better off having a preliminary section, where they specifically give a recap of Mamba. This would also make it easier to follow the contributions of the paper and better motivate some of the architectural choices made by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are highly encouraged by the positive recommendations and comments from the reviewer. The\nmain concerns are addressed below.\n\n__Q1:__ Suggestions for the title.\\\n__Reply:__ We appreciate your insight that renaming it to \"NLOS Imaging by Using Dynamic Cues\" would more accurately reflect our approach and its capabilities. We will take this into consideration in our revisions to improve clarity and accurately represent our work. \\\nThanks for your valuable feedback.\n\n__Q2:__ Discussion about the prior art on dynamic NLOS.\\\n__Reply:__ Thanks for your insightful comments and the suggested references. \n''Real-time Non-line-of-Sight Imaging of Dynamic Scenes.'' is based on a non-confocal setup for dynamic imaging, which has been discussed in Sec.2.\n''Keyhole Imaging: Non-Line-of-Sight Imaging and Tracking of Moving Objects Along a Single Optical Path'' is the technology using a fixed scanning and sampling point for dynamic imaging, and it indeed represents a third type of prior work in Sec.2.\n\nWe will include ''keyhole imaging'' as a third type and cite additional relevant papers for a more comprehensive discussion.\nThanks again for your valuable suggestions. We believe these additions will significantly strengthen our manuscript and clarify the contributions of our work.\n\n__Q3:__ A preliminary section for Mamba.\\\n__Reply:__ Thanks for your suggestion, we will include a recap as follows:\n\n__State Space Model (SSM)__ \n\nThe State Space Model (SSM) is employed to describe the linear time-invariant systems. The system processes the 1D input sequence $x(t)\\in \\mathbb{R}$ by propagating them through the intermediate hidden states $h(t)\\in \\mathbb{R}^{N}$, ultimately generating output sequences $y(t)\\in \\mathbb{R}$. Typically, SSM can be expressed as the linear ordinary differential equation:\n$$\nh^{'}(t)=\\mathbf{A} h(t) + \\mathbf{B}x(t),\\quad y(t) = \\mathbf{C}h(t) \\quad \\quad \\quad1.\n$$\n\nwhere $A\\in \\mathbb{R}^{N\\times N}$ is the state matrix, $B,C\\in \\mathbb{R}^{N\\times 1}$ are the projection parameters. After discretization via the timescale parameter $\\Delta$[1], Eq.1 is formulated as:\n$$\n    h_{t} =\\mathbf{\\bar{A}}h_{t-1} + \\mathbf{\\bar{B}}x_t,\\quad y_t = \\mathbf{C}h_t \\quad \\quad \\quad \\quad \\quad \\quad \\quad 2.\n$$\n\nwhere $\\mathbf{\\bar{A}}=\\text{exp}(\\Delta \\mathbf{A})$, $ \\mathbf{\\bar{B}}=(\\Delta \\mathbf{A})^{-1}(\\text{exp}(\\Delta\\mathbf{A}-I)\\cdot \\Delta\\mathbf{B}$. Besides, Eq.2 can be transformed into a convolutional operation which is suitable for hardware. Recently, Mamba[1] employes data-dependent mechanisms, including the learnable parameters $B$,$C$, and $\\Delta$, as well as parallel scanning. Mamba has rapidly become popular in various tasks[2,3,4,5,6] due to its linear computational complexity and global modelling capability.\n\n[1] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396, 2021.\\\n[2] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.  arXiv preprint arXiv:2312.00752, 2023.\\\n[3] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.10166, 2024.\\\n[4] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Visionmamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024.\\\n[5] Zifu Wan, Yuhao Wang, Silong Yong, Pingping Zhang, Simon Stepputtis, Katia Sycara, and Yaqi Xie. Sigma: Siamese mamba network for multi-modal semantic segmentation. arXiv preprint arXiv:2404.04256,2024.\\\n[6] Dingkang Liang, Xin Zhou, Xinyu Wang, Xingkui Zhu, Wei Xu, Zhikang Zou, Xiaoqing Ye, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. arXiv preprint arXiv:2402.10739,2024.\n\n__Q4:__ Another comparison with more sampling points.\\\n__Reply:__ Thanks for your thoughtful suggestion. However, we did not explore this specific setup due to the following considerations:\n1) In a confocal setup, there is a trade-off between the number of sampling points and the exposure time for each point, regarding imaging quality. Our current setup achieves 4 FPS with an 800 µs exposure time and a 16x16 scanning grid. Increasing the number of scanning points to maintain an equal amount of data would reduce the frame rate to 1.3 FPS, which compromises our goal of high-speed imaging. \n2) Keeping the total number constant, or increasing the scanning-grid would be unfair to multi-frame algorithms, as their scanning-grid remains unchanged. \n3) It is important to note that the amount of training data used for the single-frame methods is consistent with that used for our final method, ensuring the fairness of the comparison for data-driven approaches.\n\nThanks for your understanding.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lOqB4frUha", "reviewer": "Reviewer_mnC2", "summary": "1.\tThis paper build a new dynamic NLOS dataset crafted for learning from synthetic data and evaluating models on real-world data for dynamic NLOS reconstruction.\n2.\tThis paper introduce a Mamba-based method tailed for dynamic NLOS imaging.", "questions": "See the weaknesses.", "limitations": "See the weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper build a new dynamic NLOS dataset crafted for learning from synthetic data and evaluating models on real-world data for dynamic NLOS reconstruction.\n2.\tThe proposed method exhibits good performance.", "weaknesses": "1.\tThe paper does not adequately explain the motivation for using Mamba.\n2.\tThe proposed ST-Mamba and Cross ST-Mamba merely stack existing modules and lack innovation.\n3.\tSection 4.2 lacks detailed formulas explaining the operations of ST-Mamba and Cross ST-Mamba. Additionally, Section 4.3 is missing formal descriptions of the Hidden Volume Reconstruction and contains overly colloquial language.\n4.\tThe paper lacks ablation experiments to validate the effectiveness of Spatial-Temporal Mamba and Cross ST-Mamba. Note that Section 5.3 does not adequately validate the effectiveness of Spatial-Temporal Mamba or Cross ST-Mamba. These two modules should be individually replaced with other existing feature extraction or fusion modules in ablation experiments to verify their effectiveness.\n5.\tThe paper did not provide detailed settings for the ablation experiments of Spatial-Temporal Mamba. To verify the impact of Spatial Mamba and Temporal Mamba, the cascade structure of Temporal SSM and Spatial SSM should be replaced with two Temporal SSMs or two Spatial SSMs, rather than simply removing the Spatial SSM or Temporal SSM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time with our paper. While we have provided detailed responses to address the main concerns below, one thing needs to be highlighted first: we respectfully cannot agree that our work \"lacks innovation\" due to inheriting certain “existing modules”. In contrast, __the innovation of integrating NLOS and Mamba is a consensus of all other reviewers and has been well recognized__.\n\n__Q1:__ Motivation for Mamba.\\\n__Reply:__ The motivation are twofold:\n1) __Causality in Transient Measurement:__ Our method starts by considering the causality inherent in transient measurement.   To capture this long-ranging causal information, we employ Mamba.  As shown in Tab. 1 (IDs 0 and 4) of the rebuttal 'global' PDF, disabling the causal operation in our method results in a significant drop in performance metrics, demonstrating the effectiveness of our approach in exploring causality. \n\n2) __Efficiency and Scalability of Mamba:__ Mamba has rapidly become popular in various tasks due to its linear computational complexity and global modelling capability. \nFor comparison with VIT, we replace the ST-Mamba and Cross ST-Mamba with the plain attention block and the cross attention block. Note that the number of the core operators is the same. As shown in Fig.2 of the rebuttal 'global' PDF, our method exhibits exceptional linear scaling performance. As the input resolution increases, our method will use significantly less GPU memory. \n\n__Q2:__ Lack innovation.\\\n__Reply:__ There are two points to address:\n\n1) __The challenge of using Mamba for NLOS:__ The Mamba architecture, while effective for 1D sequential data, faces challenges in NLOS imaging due to its unidirectional processing mode. The 3D NLOS transient measurements are inherently partially causal and high-dimensional. Specifically, the histogram of each scanning point along the temporal axis exhibits causality, but the scanning points themselves along the spatial axis are non-causal. This non-causality and high dimensionality hinder Mamba from effectively capturing the underlying features. To overcome these limitations, we propose ST-Mamba and Cross ST-Mamba mechanisms, which are specifically designed to exploit and integrate the deep features within under-scanning transient measurements for NLOS imaging tasks. \n\n2) __The development of NLOS field:__ As we know, one advantage of the AI community is gradually building on top of existing works, which eventually leads to both solid and efficient solutions for many practical applications. When we look back, there are a number of works in this community with seemingly incremental components but have made a great impact on future research. Being confident in our results, we are ready to release the source code to the public. We would appreciate the opportunity to present this work as the new state-of-the-art together with its repository at this conference, which allows other researchers in this scientific field to build on top of it and improve the field further with an enhanced baseline.\n\n\n__Q3:__ Detailed description for Sec. 4.2, 4.3.\\\n__Reply:__ In Sec 4.2, we have provided a detailed explanation of the data flow, dimension change, and operator, accompanied by the schematic diagram Fig.4 for understanding. The core computational operators (SSMs) adhere to the original design, and thus we did not elaborate on them with additional formulas, as it may be redundant.\n\nRegarding the hidden volume reconstruction, the feature extraction module comprises three 3D residual blocks for extracting shallow features and downsampling along the temporal axis. \nThe volume refinement module is composed of three 3D convolutions and three interlaced 3D residual blocks. Each residual block comprises two 3D convolutions followed by a ReLU activation and a residual connection. We will include the explanation in Sec 4.3. Moreover, we will revise the colloquial words, e.g., 'enter', 'pure', 'huge', etc., ensuring clarity and professionalism in our presentation.\n\nThanks for bringing up these points, and we will revise the corresponding components.\n\n\n__Q4:__ Ablations for ST-Mamba and Cross ST-Mamba.\\\n__Reply:__ As suggested, we have conducted more ablation studies on ST-Mamba and Cross ST-Mamba. The quantitative results are listed in Tab.1 of the rebuttal 'global' PDF. \\\nWe individually replaced the ST-Mamba (blue blocks in Fig.2 of the manuscript) and Cross ST-Mamba (orange blocks in Fig.3 of the manuscript) with a plain attention block and a cross attention block.\nBy comparing IDs 1, 3, and 4, it is evident that our method performs best when both ST-Mamba and Cross ST-Mamba are employed. Additionally, we excluded only the Cross ST-Mamba while maintaining the same number of SSMs as in our final method (ID 2). This further demonstrates the effectiveness of the proposed Cross ST-Mamba.\\\nThank you for your valuable suggestion. We will include the additional ablation studies in the revised manuscript.\n\n__Q5:__ Ablations for spatial and temporal Mamba.\\\n__Reply:__ To thoroughly verify the impact of spatial Mamba (S-Mamba) and temporal Mamba (T-Mamba), we have conducted additional comprehensive ablation studies. In these studies, we replaced the spatial Mamba with temporal Mamba and vice versa to validate their individual contributions to the network. The results are listed as follows:\n\n| Spatial | Temporal | PSNR | SSIM | RMSE| MAD |\n| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |\n| S-Mamba|S-Mamba | 24.31\t |83.08\t |0.0911\t |0.0496|\n| T-Mamba| T-Mamba| 24.32\t|83.68\t|0.0898\t|0.0398|\n| S-Mamba| T-Mamba| __24.46__\t|__84.08__\t|__0.0880__\t|__0.0397__|\n|\n\nIt can be seen that spatial and temporal Mamba contribute differently to the reconstruction performance, and fusing both of them boosts the overall performance.\n\nWe truly appreciate your suggestions and will include the detailed settings and descriptions in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# This is the global rebuttal for all the reviewers including a PDF containing only figures and tables. Please find individual responses to each reviewer below.

The content is:

__Fig1:__ The reconstruction results via the traditional method (RSD) for the high-resolution transient measurements, which are recovered by the SR networks of USM and Ours.

As suggested by Reviewer CVND, we compare the capabilities of the transient super-resolution networks of USM[1] and our method. 
To obtain quantitative results, we first generate the recovered high-resolution transient measurements and then conduct the reconstruction using the traditional method RSD[1], instead of the deep neural network. For USM, the metrics PSNR/SSIM/RMSE/MAD are 19.25/18.93/0.2206/0.1000. For our method, the metrics PSNR/SSIM/RMSE/MAD are 21.48/17.08/0.2120/0.0917. Except for SSIM, our method surpasses USM. The decrease in SSIM may originate from the artifacts interference. \
To provide a more comprehensive evaluation, we introduce another metric ACC, from[2], which indicates the foreground recovery degree.
The ACC for USM is 10.49, while for our method, it is 31.22, demonstrating that our method generates cleaner and higher-quality transient measurements. Qualitative results are provided in Fig.1 of the rebuttal 'global' PDF, further demonstrating our superiority comprehensively.

[1] Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jarabo, Diego Gutierrez, and Andreas Velten. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature, 572(7771):620–623, 2019.\
[2] Jun-Tian Ye, Xin Huang, Zheng-Ping Li, and Feihu Xu. Compressed sensing for active non-line-of-sight imaging. Optics Express, 29(2):1749–1763, 2021.

__Fig.2:__ Efficiency comparisons between Ours-VIT and Ours. Ours-VIT is derived from Ours and replaces the Mamba with Attention. Note that the physical prior is excluded when comparing the inference memory.

Mamba has rapidly become popular in various tasks due to its linear computational complexity and global modelling capability. For comparison with VIT, we replace the ST-Mamba (blue blocks in Fig.2 of the manuscript) and Cross ST-Mamba (orange blocks in Fig.3 of the manuscript) individually with the plain attention block and the cross attention block. As shown in Fig.1 of the rebuttal 'global' PDF, our method exhibits exceptional linear scaling performance. As the input resolution increases, our method will use significantly less GPU memory.

__Tab.1__: Ablation studies on causality and different modules.

As suggested by Reviewer mnC2, we have conducted more comprehensive ablation studies on ST-Mamba and Cross ST-Mamba. The quantitative results are listed in Tab.1 of the rebuttal 'global' PDF.
We individually replaced the ST-Mamba (blue blocks in Fig.2 of the original manuscript) and Cross ST-Mamba (orange blocks in Fig.3 of the original manuscript) with a plain attention block and a cross attention block. By comparing IDs 1, 3, and 4, it is evident that our method performs best when both ST-Mamba and Cross ST-Mamba are employed. Additionally, we excluded only the Cross ST-Mamba while maintaining the same number of SSMs as in our final method (ID 2). This further demonstrates the effectiveness of the proposed Cross ST-Mamba.

Meta-review: The paper received consistently positive feedback following the authors’ rebuttal. Reviewers highlighted the paper's originality (oqBG, CVND, j9gw), the quality of the work (oqBG, CVND), and the development of a new dynamic NLOS dataset (mnC2, oqBG, CVND, j9gw). While there were initial concerns about missing ablations and unclear details, these were effectively addressed in the rebuttal, leading to stronger overall reviews.

After carefully reviewing the paper, the reviews, and the rebuttal, The AC concurs with the reviewers' positive consensus and recommend acceptance of this paper.

For the camera-ready version, the authors should integrate all discussions from the rebuttal into the main paper and supplementary materials. Specifically, the authors should implement the following changes:

**1. Ablations**: Include ablations for ST-Mamba and Cross ST-Mamba, and for spatial and temporal Mamba (mnC2), as well as a comparison of the SR module with USM (CVND).

**2. Improved positioning**: Add a preliminary section on Mamba (oqBG, mnC2) and revise the title to reflect “NLOS Imaging by Using Dynamic Cues” (oqBG).

**3. Video results** (j9gw)
Decision: Accept (poster)