Toward Dynamic Non-Line-of-Sight Imaging with Mamba Enforced Temporal Consistency

Authors: yue  li
yi  sun
shida  sun
juntian  ye
yueyi  zhang
feihu  xu
zhiwei  xiong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 15:31:29

Abstract: Dynamic reconstruction in confocal non-line-of-sight imaging encounters great challenges since the dense raster-scanning manner limits the practical frame rate. A fewer pioneer works reconstruct high-resolution volumes from the under-scanning transient measurements but overlook temporal consistency among transient frames. To fully exploit multi-frame information, we propose the first spatial-temporal Mamba (ST-Mamba) based method tailored for dynamic reconstruction of transient videos. Our method capitalizes on neighbouring transient frames to aggregate the target 3D hidden volume. Specifically, the interleaved features extracted from the input transient frames are fed to the proposed ST-Mamba blocks, which leverage the time-resolving causality in transient measurement. The cross ST-Mamba blocks are then devised to integrate the adjacent transient features. The target high-resolution transient frame is subsequently recovered by the transient spreading module. After transient fusion and recovery, a physical-based network is employed to reconstruct the hidden volume. To tackle the substantial noise inherent in transient videos, we propose a wave-based loss function to impose constraints within the phasor field. Besides, we introduce a new dataset, comprising synthetic videos for training and real-world videos for evaluation. Extensive experiments showcase the superior performance of our method on both synthetic data and real world data captured by different imaging setups. The code and data are available at https://github.com/Depth2World/Dynamic_NLOS.

Main paper: 
 introduction: 
 Non-Line-of-Sight (NLOS) imaging revolutionizes our comprehension of the environment by revealing hidden information. Different from conventional cameras, the NLOS system captures indirect light reflections or signals that interact with the hidden object, subsequently rebounding off the relay wall that is visible to the imaging system. By analyzing these reflections, NLOS can reveal critical properties like albedo and surface normal of the hidden objects, unlocking valuable insights. A typical active NLOS imaging setup is illustrated in Fig. 1 . The pulsed laser emits periodic pulses directed towards a relay wall, serving the dual purpose of illumination and synchronization for the imaging system. The Single Photon Avalanche Diode (SPAD) captures photons reflected from the relay wall, while the Time-Correlated Single Photon Counting sensor (TCSPC) records their arrival times within each pulse period. The temporal distribution of each scanning point accumulates over successive pulse periods, termed exposure time. Consequently, the total acquisition time for a transient measurement scales proportionally with the exposure time and the density of the scanning grid. Notably, achieving high-quality reconstructions necessitates dense scanning grids, at the expense of longer acquisition times, typically ranging from minutes to hours. The compromise between acquisition time and data quality poses a formidable obstacle to the advancement of fast imaging techniques, making them impractical for real-world applications. Recent endeavors [1, 2, 3, 4] addressed this challenge by initiating from under-scanning measurements and striving to reconstruct high-resolution volumes comparable to those derived from sufficient scanning measurements. These methods can significantly accelerate data acquisition by orders of magnitude.Recent research [4] has further demonstrated that employing a low-density scanning grid can balance reconstruction quality and total acquisition time, without sacrificing too much information. The sparse scanning points and rapid exposure time offer the potential for dynamic NLOS imaging, a field yet to be fully explored. However, the pursuit of dynamic NLOS reconstruction faces two primary challenges: 1) Insufficient information fusion across adjacent transient frames: Existing methods, whether traditional or deep-based, typically concentrate on individual transient frames, overlooking the temporal consistency between them. 2) Lack of NLOS video datasets, including synthetic data for training and real-world data for evaluation: The rapid exposure time results in a diminished signal-to-noise ratio (SNR) of transient measurement, highlighting the critical need for simulation datasets that accurately emulate real-world conditions. Besides, dynamic imaging imposes extremely high requirements on the synchronization accuracy and acquisition efficiency of the hardware system.Based on these observations, we exploit temporal consistency in transient videos by extracting information from the multiple frames to compensate for the unrecoverable areas for the reference frame, leading to improved quality. Our proposed method consists of two main stages: firstly, integrating the transient frames and expanding the target transient measurement across the spatial dimension, and secondly, reconstructing hidden volume. Specifically, in the first stage, after extracting the features from the input, we introduce the elaborate spatial-temporal Mamba (ST-Mamba) to sequentially exploit the causality in transient measurement and dig into the inherent long-ranging features along the spatial and temporal dimensions. Subsequently, we devise the cross ST-Mamba to blend complementary features among transient frames towards the target frame. After that, the high-resolution transient frame is recovered by the transient spreading module. For the second stage, we embed the physical prior into the feature transformation module, i.e., transforming the spatial-temporal data into the Fourier domain, element-wise multiplying with the inverse point spread function (PSF) of the imaging system, and then reverting the features into the spatial domain. The final refinement module subsequently enhances the target hidden volume, as well as the derived intensity image and depth map. During the training process, we introduce a novel virtual wave-based loss function to accentuate effective information in low SNR data, by employing a Gaussian-shaped illumination function to constrain transient measurement within the phasor field [5] .To bridge the training and testing phases, we present a new dataset for NLOS dynamic imaging. The synthetic data comprises the dynamic objects with 32 frames in each sequence with varying quantum efficiency. The real-world NLOS videos are captured at 4 frames per second (FPS) by our imaging prototype. The dataset is publicly available to propel research in dynamic imaging within this field. Comparative evaluation against existing traditional and deep-learning-based solutions demonstrates that our method achieves superior reconstruction performance and generalization capability to real-world scenarios. Our contribution can be summarised as follows:• For the first time, we introduce a Mamba-based method tailed for dynamic NLOS imaging.The proposed spatial-temporal Mamba mechanisms successfully exploit the inherent longranging causal features and integrate the temporal consistency across the transient frames.• We build a new dynamic NLOS dataset crafted for learning from synthetic data and evaluating models on real-world data for dynamic NLOS reconstruction, which could help advance faster NLOS imaging techniques.• Our proposed method exhibits superior performance on both synthetic and real-world datasets, as evidenced by extensive experimental results.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: See the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 NLOS Imaging Systems. Active NLOS imaging systems can be divided into two categories: confocal and non-confocal imaging systems. For the confocal system, the illumination points from the laser and the scanning points collected by the time-resolved sensor coincide. The total acquisition time for a transient measurement is proportional to the exposure time per scanning point and the density of the scanning grid, typically ranging from minutes to hours. Different from confocal systems, the detector in non-confocal systems [5, 6, 7, 8] is in array form, such as 16×1 or 32×32.The laser illuminates a fixed point on the relay wall and the spad array captures the indirect photon simultaneously. The non-confocal system has the potential for real-time imaging but still faces the following challenges. Accuracy is still traded for speed [7, 8, 9] . Due to the unsatisfactory parameters (low quantum efficiency and fill factor, high dark count and cross-talk effect), non-confocal systems also require relatively long exposure times to achieve reconstruction, e.g., 0.3 FPS in [6] , 5 FPS in [8, 9] , 20 FPS in [7] . More importantly, the price of the SPAD array is quite expensive. There are also some special imaging setups using dynamic cues, e.g., key-hole imaging [10] and light field tomography [11] . In this paper, we continue to focus on the confocal system and strive to advance the development of dynamic NLOS imaging in terms of imaging compromise and cost expenditure.Reconstruction Algorithms. The NLOS reconstruction algorithms have made significant progress, encompassing the back-projection [12, 13, 14] , linear optimization [15, 16, 17, 18] , non-linear optimization [19, 20] , wave propagation [5, 21] , and deep-learning-based methods [22, 23, 24, 25, 26, 27, 28, 29, 30] . These methods reconstruct promising hidden volumes, contingent upon high-quality transient measurements. The other studies [1, 2, 3, 4, 31] attempt to achieve faster system acquisition speeds by using fewer scanning points while still recovering high-quality results. CSA [1] and FSN [2] explored iterative algorithms with regularization, albeit at the cost of computation time.The deep methods [3, 4] address this issue by leveraging the deep-learning technology for a single forward inference. Unfortunately, these methods always neglect the temporal consistency between the neighbouring frames. As an incremental yet crucial advancement, we focus on dynamic NLOS reconstruction and aim to leverage the multi-frame information to enhance reconstruction quality. A concurrent work [32] employs the dynamic scanning grid and then fuses the multi-frame information, while the scanning grid in this paper is fixed.3 PreliminaryQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Sec. 6.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
observation model: 
 The transient measurement, denoted as τ , comprises a set of temporal histograms, acquired from the raster-scanning points on the relay wall. We follow the common assumptions of no inter-reflections, no occlusions, and isotropic light scattering within the hidden scene. As depicted in Fig. 1 , given the illuminated point p l , the continuous transient measurement at the scanning point p s can be expressed as follows:EQUATIONwhere ρ denotes the hidden albedo volume, p t is the target point of the hidden scene Ω, r l is the distance between the illuminated and the target points, and r s is the distance between the scanning and the target points. δ models the light propagation from the relay wall to the hidden object and back to the wall. After being captured by the detector within N pulses, the discrete transient measurement τ can be accumulated as:EQUATIONwhere ε denotes the quantum efficiency of the detector. b and d represent the background ambient noise and the dark count of the detector [33] , respectively. t J indicates that the temporal bin t is sampled from a Gaussian-shaped jitter. By modelling efficiency and jitter, the observation model is brought closer to the real-world process. For further details about the detection model, refer to [34, 23] .Synthetic Dataset. Considering that there are currently no dynamic simulation datasets available for training and evaluation, we modify the time-resolved rendering tool [23] and then simulate the dynamic synthetic dataset. The dataset comprises 265 sequences, which consist of 1 to 3 static objects and 1 dynamic object following a 3D helical motion trajectory. Each sequence has 32 frames of transient measurements, with a bin width of 33 ps. The toy example is shown in Fig. 2 (a). Note that the color is for visualization, the data is in gray-scale. The spatial-temporal resolution of the transient measurement is 128×128×512. To enhance the generalization capability of the synthetic data, we incorporate a detector jitter provided by [34] , during the synthetic sampling process. Additionally, to introduce variability, we randomly assign quantum efficiencies ranging from 1% to 30% for the sequences. To prepare the training data, we execute the interval sampling along the spatial dimension for the raw transient to obtain the under-scanning measurement of size 16×16×512.Real-world Dataset. 1) System Setup: For the evaluation on real-world data, we develop an active confocal NLOS imaging system. The prototype is illustrated in Fig. 2 (c). The system utilizes a 532 nm laser (VisUV-532) to generate pulses with a width of 85 picoseconds and a repetition frequency of 20 MHz, delivering an average power of 750 mW. These pulses are directed through a two-axis raster-scanning Galvo mirror (Thorlabs GVS212) towards the relay wall. Subsequently, both direct and indirect diffuse photons are gathered by another two-axis Galvo mirror, coupled into a multimode optical fiber, and then channelled into a SPAD detector (PD-100-CTE-FC) with a detection efficiency of approximately 45%. The movement of both Galvo mirrors is synchronized and controlled by a National Instruments acquisition device (NI-DAQ USB-6343). The TCSPC (Time Tagger Ultra) captures the pixel trigger signals from DAQ, the synchronization signals from the laser, and photon detection signals from the SPAD. The temporal resolution of the overall system is approximately 95 ps. 2) Collection Details: During data collection, the illuminated and sampling points maintain a consistent direction but are intentionally offset slightly to prevent interference from directly reflected photons during scanning. We perform a raster scan across a 16×16 square grid of points on the relay wall. Each scanning point is allotted 800 µs for exposure, and the histogram is with a length of 512 bins and a bin width of 32 ps. Accumulation occurs during the switching process of points in [21] , leading to aliasing in transient measurements. We employ the point-by-point accumulation method, where data during the jump between scanning points is disregarded. As such setting, we capture 4 video sequences of dynamic NLOS scenes, with each sequence containing approximately 64 frames, and a capture rate of 4 FPS.The State Space Model (SSM) is employed to describe the linear time-invariant systems. The system processes the 1D input sequence x(t) ∈ R by propagating them through the intermediate hidden states h(t) ∈ R N , ultimately generating output sequences y(t) ∈ R. Typically, SSM can be expressed as the linear ordinary differential equation: hEQUATIONwhere A ∈ R N ×N is the state matrix, B, C ∈ R N ×1 are the projection parameters. After discretization via the timescale parameter ∆ [35] , Eq 3 is formulated as:EQUATIONwhere Ā = exp(∆A), B = (∆A) -1 (exp(∆A -I) • ∆B. Besides, Eq. 4 can be transformed into a convolutional operation which is suitable for hardware. Recently, Mamba [36] employ the data-dependent mechanisms, including the learnable parameters B, C, and ∆, as well as the parallel scanning. Mamba has rapidly become popular in various tasks [37, 38, 39, 40] due to its linear computational complexity and global modelling capability.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
overview: 
 We present a groundbreaking approach to dynamic NLOS reconstruction. Our method leverages Mamba to capture long-range dependencies within transient data to achieve high-fidelity reconstructions. To begin, we formalize the dynamic NLOS reconstruction problem. Given a sequence τ = [τ t ] 0≤t≤i ∈ R h×w×T , where i represents the total number of the transient frames, h and w denote the spatial dimensions (height and width) of the t-th transient frame, and T signifies the number of discretized histogram bins along the temporal dimension. The objective is to reconstruct the target hidden volumeV = [V t ] 1≤t≤i-1 ∈ R H×W ×Z, where H, W , and Z represent the 3D spatial dimensions of the reconstructed volume. Due to the low spatial resolution of input transient frames, we propose a unified framework that merges transient measurement super-resolution and volume reconstruction. We strategically enhance each transient frame individually and subsequently integrate them before spreading to a high spatial resolution, thereby significantly reducing the computational burden. Specifically, our method utilizes three adjacent frames as input and predicts the high-resolution hidden volume for the reference frame.Our proposed method leverages a multi-stage architecture to achieve high-fidelity dynamic NLOS reconstruction. In the initial stage, the interleaved extraction module utilizes 3D interlaced and dilated convolutions to effectively downsample the temporal dimension of the input transient frames.Next, the spatial-temporal Mamba (ST-Mamba) blocks extract informative features ( F t+a where a ∈ {-1, 0, 1}) by exploiting long-range dependencies within the transient data. Subsequently, the cross ST-Mamba blocks capitalize on the inherent temporal consistency between frames and integrate the multi-frame features to obtain the aligned features F a t . The target features F t and the aligned features are added together for the enhanced features F e t . Finally, the transient spreading module employs 3D transposed convolution layers with traditional interpolation to generate a high-resolution reference transient frame τ ↑ t . Notably, our method recovers data along the spatial dimension as well as performs temporal upsampling, resulting in a temporal size that matches the input. This recovered temporal information, as highlighted in prior work [27] , strengthens the constraints between the predicted and ground-truth reference transient frames during training loss construction, ultimately leading to improved reconstruction quality. After processing transient information, our method proceeds towards volume reconstruction. Given the specific target transient frame τ ↑ t , we employ a feature extraction module that combines convolutional operations with traditional interpolation techniques, ensuring the capture of relevant features while preserving crucial temporal details. Inspired by existing physics-based methods [23, 41, 26, 28] , we then utilize a physical prior [42] within a feature transformation module. This module transforms the extracted features from a spatio-temporal domain into a purely spatial domain. Finally, a refinement module refines the transformed features and generates the hidden albedo volume. The intensity image and depth map are then derived from the volume. The details of the ST-Mamba, Cross ST-Mamba, and hidden volume reconstruction will be explored in the following sections.The Mamba architecture, while powerful for 1D sequential data [36] , presents challenges for NLOS imaging due to its unidirectional processing mode. The 3D NLOS transient measurements are inherently partially causal and high-dimensional. Specifically, the histogram of each scanning point along the temporal axis exhibits causality, but the scanning points themselves along the spatial axis are non-causal. The non-causality and high dimensionality hinder Mamba from effectively capturing the underlying features. To overcome these limitations, we propose the ST-Mamba and the cross ST-Mamba mechanisms, which are specifically designed to exploit and integrate the deep features within under-scanning transient measurements for NLOS imaging tasks.ST-Mamba. The overview of ST-Mamba is presented in Fig. 4(a) . Given the initial extracted features F in t ∈ R C×T ×h×w , the ST-Mamba block first conducts the temporal SSM. The normalized input features are reshaped to R hw×T ×C and then undergo linear projection and 1D causal convolution thanks to the histogram of the temporal axis being unidirectional. Then, the output features are reshaped to R T ×hw×C for the next spatial-SSM. Due to the non-causality, we adopt the bidirectional SSM [38] to capture spatial awareness. Finally, the output features are multiplied with gating features from the activation operator and fed to the last linear projection, yielding F t . By incorporating mechanisms to handle both causal and non-causal data components, ST-Mamba offers a more comprehensive approach for modelling the long-range correlations in transient measurement.Cross ST-Mamba. As discussed above, the deep features from different transient frames are exploited. To integrate the information from adjacent frames, we further introduce the cross ST-Mamba mechanism to align the features for the target frame. As shown in Fig. 4 (b), the cross ST-Mamba possesses a reference branch and an adjacent branch. These two branches share the same gating factor from the reference input. Different from the query mechanism of cross-attention, the complementary information between transient frames is integrated by the gating mechanism. Given the reference features F t and neighbouring features F t±1 , the cross ST-Mamba block conducts the temporal and spatial SSM for the inputs successively. Then the output reference features and the neighbouring features are modulated by the same gating parameter derived from the reference input. Finally, the modulated features are added together for the aligned features. The output features F a t is generated after a linear projection and a shortcut.According to Eq. 1, the forward model can be simplified into a 3D convolution form through resampling along the temporal axis for transient data and resampling along the depth axis for hidden volume. The solution to NLOS reconstruction is an inverse problem, involving the PSF of the imaging system [15] . Without additional computation, PSF can be expressed explicitly under the specific imaging setup, which is commonly introduced to learning-based reconstruction methods [23, 41, 26, 28] . In this study, we also incorporate the physical-prior [42] , illustrated in the bottom right part of Fig. 3 . For the hidden volume reconstruction, the feature extraction module comprises three 3D residual blocks for extracting shallow features and downsampling along the temporal axis. The volume refinement module is composed of three 3D convolutions and three interlaced 3D residual blocks. Each residual block comprises two 3D convolutions followed by a ReLU activation and a residual connection. The extracted features from the target high-resolution transient frame are convolved by the illumination function to access the phasor field. After resampling, the spatial-temporal features perform the Fourier transform, element-wise multiply the inverse PSF in the frequency domain, and then perform the inverse Fourier transform to exhibit the 3D spatial features. Due to the large domain gap between the synthetic and real-world data, this methodology trades the Fourier computational burden for generalizability, which has been widely utilized in deep methods.The total loss function L total is composed of three components: the measurement recovery loss L m , the volume reconstruction loss L v , and the regularized loss L r :EQUATIONwhere the parameters β, γ and α contribute the corresponding loss. Among these loss items, the phasor field loss L pf , the transient loss L t , the intensity loss L int , and the depth loss L dep are formulated as follows:EQUATIONwhere τ , I, and D denote the transient measurement, intensity image and depth map of the hidden volume. gt denotes ground truth. P (t, σ) represents the illumination function [42] P (t, σ) = e jΩ C t • e -t 2 2σ 2, Ω C is the central frequency depended on the wavelength, σ is the standard deviation of the Gaussian function.t * denotes the convolution along the temporal dimension, leading to the highlight of useful information in the frequency domain. Inspired by [4] , we utilize the local similarity loss L ls and the total variation loss L tv for constructing the last regularized loss L r . For more details about the loss items, see the supplementary. 
experimental details: 
 Implementation. Our method is implemented using PyTorch, trained on the synthetic data, and then directly tested on the real-world data. During training, we employ the AdamW [43] as the optimizer with a learning rate of 10 -4 and a weight decay of 0.95. To enhance visual clarity, the final output spatial resolution is set to 128×128, based on the input size of 16×16. All the experiments are conducted on the NVIDIA A100 GPUs, with a batch size of 4. We utilize 150 sequences for training and 17 sequences for synthetic testing. Besides, we utilize 4 sequences for real-world evaluation. The hyper-parameter β and γ are set to 1 and 10 -5 . α 1 , α 2 , α 3 are set to 0.5, 1, and 0.1, respectively.Baselines. We compare our method with existing baselines, including the traditional methods LCT [15] , FK [21] , and RSD [42] , the iterative method CSA [1] as well as the deep-learning-based methods including LFE [23] , I-K [28] , and USM [4] . The baseline methods are implemented following their publicly available codes. Apart from the multi-frame version, we also provide the single-frame version Ours-S (excluding cross ST-Mamba) for a comprehensive comparison. Note that only CSA, USM, and our method are specifically designed for reconstruction from the underscanning measurement. For the other baselines, the inputs are interpolated to the target resolution 128×128×512 for the final comparison. Evaluation Metrics. The synthetic quantitative evaluation comprises two categories. For intensity images, we compute the peak signal-to-noise ratio (PSNR) and structural similarity metrics (SSIM), averaged across the corresponding test samples. For depth maps, we calculate the root mean square error (RMSE) and mean absolute distance (MAD).Our method demonstrates superior performance against existing approaches, as shown by the quantitative results in Tab. 1. Notably, our method excels in both intensity and depth estimation, achieving significantly better results than the baseline methods. The single-frame version of our method, Ours-S, which leverages the core ST-Mamba architecture, also outperforms other methods, demonstrating the effectiveness of ST-Mamba in exploiting long-ranging causal data. Furthermore, the multi-frame model surpasses the single-frame model across all metrics, showcasing the strength of the proposed cross ST-Mamba in integrating information from multiple transient frames. The qualitative results are presented in Fig. 5 . Except for the dynamic motorbikes, the synthetic sequences contain the background static objects such as cars, buckets, helmets, and bottles. Due to the various quantum efficiencies of sequences, the traditional methods reconstruct the blurry results lacking in details. The deep-based method LFE [23] and I-K [28] recover the main structure but still miss details. CSA [1] generates artifacts around the hidden objects, while USM [4] performs better in reducing background noise but struggles with fine structure. In contrast, our method performs the best in both static and dynamic scenarios, with higher fidelity and more details. The promising reconstruction results underscore the ability of the proposed method to capture the dynamic nature effectively.To assess the effectiveness of the proposed loss functions and the spatial-temporal Mamba mechanism, we conduct the ablation studies with Tab. 2 listing the quantitative results.Loss Items. It can be concluded that incorporating the regularized loss generally improves metrics, with a minor trade-off in PSNR suggesting resistance to overfitting. This phenomenon has also been verified in USM [4] . A significant performance boost is observed when the phasor field loss is included, which indicates that enforcing constraints within the phasor field highlights valuable information in the transient measurements, leading to more accurate reconstructions.Spatial-Temporal Mamba. To investigate the efficiency of individual spatial Mamba (S-Mamba) and temporal Mamba (T-Mamba) components for NLOS reconstruction, we operate the Mamba along the spatial and temporal dimensions, respectively. The temporal Mamba-based model might outperform the spatial Mamba-based model on single scanning points. However, it lacks spatial awareness, hindering its ability to capture the overall structure. When both spatial and temporal Mamba are employed, the model achieves the best performance across all metrics, demonstrating the advantage of capturing information from both spatial and temporal domains.To evaluate the generalizability of our method, we test the models on real-world transient videos captured by our imaging system. The quantitative results are shown in Fig. 6 , where the top three rows exhibit three sequences of planar objects in rigid motion, while the bottom three rows depict a sequence with non-rigid motion. Traditional methods reconstruct the single object with considerable noise but struggle to recover the distant moving letter in multi-object scenes. Deep learning approaches, while achieving cleaner backgrounds, often lead to incomplete or overly simplified reconstructions of the hidden scene. Although CSA [1] and USM [4] are designed specifically for under-scanning measurements, they lose the adaptability and generalization ability under low SNR conditions. In contrast, our method demonstrates superior performance, capturing finer geometric structures and richer details. Furthermore, our methods deliver robustness in nonrigid scenarios, as exemplified by the clear recovery of arm movements. Besides, our multi-frame model outperforms the single-frame model on more detailed information and less noise, showcasing the effectiveness of cross ST-Mamba. These promising results highlight the strong representation capability and generalization ability of our proposed method to real-world scenarios. More real-world qualitative results can be seen in the supplementary material. 
conclusion and discussion: 
 Conclusion. This work presents a novel learning-based framework for dynamic reconstruction in confocal NLOS imaging. By leveraging the powerful ST-Mamba and cross ST-Mamba, the proposed method effectively captures both long-ranging causal information while exploiting the natural consistency within transient video sequences. Extensive evaluations on a newly created dataset, encompassing both synthetic and real-world scenarios, demonstrate the superiority of our method in achieving high-quality reconstructions compared to existing approaches. We believe the proposed method presents a significant step forward in dynamic NLOS reconstruction, unlocking the possibilities for a wide range of real-world applications in various fields.Limitation. The temporal consistency has shown significant potential in dynamic NLOS reconstruction. Fusing information in the spatial-temporal domain and purely spatial domain may be a more effective approach. The feature transformation module relies on Fourier operation to utilize the PSF of the specific imaging system, which consumes a computational burden. It is essential to develop a lightweight transformation to reduce this burden and allow for more network design. As shown in Tab.3, disabling the causal operation in our method results in a significant drop in performance metrics, demonstrating the effectiveness of our approach in exploring causality. For more comprehensive ablation studies on ST-Mamba and Cross ST-Mamba, we individually replaced the ST-Mamba (blue blocks in Fig. 3 and Cross ST-Mamba (orange blocks in Fig. 3 ) with a plain attention block and a cross attention block. By comparing IDs 1, 3, and 4, it is evident that our method performs best when both ST-Mamba and Cross ST-Mamba are employed. This does not imply that the cross-attention mechanism is unsuitable for maintaining spatio-temporal consistency; rather, it indicates that using Cross ST-Mamba is more effective for transient data with partial causality. Additionally, we excluded only the Cross ST-Mamba while maintaining the same number of SSMs as in our final method (ID 2). This further demonstrates the effectiveness of the proposed Cross ST-Mamba. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [NA] Justification: The paper does not use existing assets. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: USM-SR+RSD Ours-SR+RSD GT The capabilities of the transient super-resolution networks of USM [4] and our method are further compared. To obtain quantitative results, we first generate the recovered high-resolution transient measurements and then conduct the reconstruction using the traditional method RSD [5] , instead of the deep neural network. For USM, the metrics PSNR/SSIM/RMSE/MAD are 19.25/18.93/0.2206/0.1000. For our method, the metrics PSNR/SSIM/RMSE/MAD are 21.48/17.08/0.2120/0.0917. Except for SSIM, our method surpasses USM. The decrease in SSIM may originate from the artifacts interference.To provide a more comprehensive evaluation, we introduce another metric ACC, from [1] , which indicates the foreground recovery degree. The ACC for USM is 10.49, while for our method, it is 31.22, demonstrating that our method generates cleaner and higher-quality transient measurements.Qualitative results are provided in Fig. 7 , further demonstrating our superiority comprehensively.More reconstructed intensity images from the real-world data are shown in Fig. 8 and Fig. 9 . Our method achieves excellent reconstruction results in simple planar scenes, such as the clear letter structures in Fig. 8 and Fig. 9 . It also delivers outstanding reconstruction performance in complex scenes as shown in Fig. 6 . As discussed in Sec. 4.4, The local similarity loss L ls and the total variation loss L tv is formulated as [4] :EQUATIONwhere ρ(x, y, z) indicates the volume at position (x, y, z), ρ(x, y, z, k) represents the volume block centered at (x, y, z) with size k, W refers to the Gaussian window with size k. The inference memory and inference time of the models are listed in Tab. 4. It is indeed that using multi-frame information will significantly increase the inference time, but the inference memory usage is still within the range of consumer-grade GPU, making it suitable for real-world applications.NeurIPS Paper Checklist 


Decision: The paper received consistently positive feedback following the authors’ rebuttal. Reviewers highlighted the paper's originality (oqBG, CVND, j9gw), the quality of the work (oqBG, CVND), and the development of a new dynamic NLOS dataset (mnC2, oqBG, CVND, j9gw). While there were initial concerns about missing ablations and unclear details, these were effectively addressed in the rebuttal, leading to stronger overall reviews.

After carefully reviewing the paper, the reviews, and the rebuttal, The AC concurs with the reviewers' positive consensus and recommend acceptance of this paper.

For the camera-ready version, the authors should integrate all discussions from the rebuttal into the main paper and supplementary materials. Specifically, the authors should implement the following changes:

**1. Ablations**: Include ablations for ST-Mamba and Cross ST-Mamba, and for spatial and temporal Mamba (mnC2), as well as a comparison of the SR module with USM (CVND).

**2. Improved positioning**: Add a preliminary section on Mamba (oqBG, mnC2) and revise the title to reflect “NLOS Imaging by Using Dynamic Cues” (oqBG).

**3. Video results** (j9gw)

Reviews: review: paper_summary: paper_summary: The paper proposes a dynamic non-line-of-sight imaging approach that uses Mamba to enforce spatio-temporal consistency (called ST-Mamba). This is done using a mutlistage model that takes in a set of transient frames, performs temporal downsampling, followed by spatial and temporal SSM per frame and across frames to get a high-resolution transient frame. Finally, the volume is reconstructed from the high-resolution transient frame using a physical prior. The paper uses a custom synthetic temporal dataset and a small real-world dataset for quantitative and qualitative evlauation. Experimental results show that the proposed method performs better than other NLOS methods for dynamic scenes and ablation experiments show the effectiveness of ST-Mamba.
strength_weakness: strength_weakness: Only qualitative results are provided for real dataset and without any video results difficult to judge the quality of the output.

SSMs are primarily motivated for long sequences. The proposed method uses three frames as input and the paper doesn’t motivate why SSMs are needed for spatio-temporal consistency.
questions: questions: What’s the motivation for using SSMs to enforce spatio-temporal consistency?

How much is the cross ST-Mamba component contributing?
limitations: limitations: The authors identified computational cost as a limitation of the work.
paper_summary: paper_summary: This paper introduces the first Mamba-based method for dynamic NLOS imaging, featuring three key modules: the spatial-temporal Mamba, the cross ST-Mamba, and the phasor field loss. The spatial-temporal Mamba extracts and integrates both causal and non-causal transient components, while the cross ST-Mamba combines deep features from adjacent frames. The phasor field loss ensures the consistency of transients in the phasor field. Evaluations on synthetic and self-captured real-world transients demonstrate that the proposed method outperforms existing approaches. Ablation studies further validate the effectiveness of the proposed components.
strength_weakness: strength_weakness: (1)	Using cross-mamba to extract inter-frame causality in transients is understandable. However, the values in single-frame transients  are based on the distance from the hidden object to the wall. This seems to be the spatial correlation rather than time-resolving causality. Please provide further explanation to justify the motivation for using ST-Mamba to process single-frame transients.
(2)	Lines 180-184 are unclear. Restoring deep features of transients to the original resolution seems to be a common practice. Can transient recovered be done directly at the original resolution? The temporal information in low spatial-resolution transients might be compromised by down-sampling and up-sampling operations.
(3)	The framework is similar to USM, consisting of a transient super-resolution network and a volume reconstruction network. The proposed components are mainly applied to the transient super-resolution network. Please compare with USM's recovered high spatial-resolution transients to highlight the proposed method’s contributions.
questions: questions: Please refer to Weakness.
limitations: limitations: Please refer to Weakness.
paper_summary: paper_summary: The paper tackles the problem of non-line-of-sight imaging. As a spin on the commonly attempted task, they solve the problem in dynamic scenes and use transients captured at different time points to aid the recovery of the scene at a specific time point. 

To solve the problem they introduce a Mamba inspired architecture, which takes in 3 transient frames. These three transient frames are processed by ST-Mamba and Cross ST-Mamba blocks to produce a high-resolution transient frame, which is then fed into a physically based network to recover hidden volume, intensity, and depth maps. 

The authors train the model on a custom simulated dataset, the model is also evaluated on a self-captured dynamic transient dataset.
strength_weakness: strength_weakness: Quality: I have a slight issue with the authors calling the method “Dynamic NLOS”. If I understand correctly the method can use transients from different time points (one before and one after) to reconstruct the scene at a specific time point. The method does not automatically reconstruct the scene in time. To actually recover the scene in time, the authors would have to pass in the 3 sets of frames sequentially and get a separate reconstruction in time, is that right? I think the method would be better called “NLOS imaging by using dynamic cues” or something of this sort. 

In any case even though the method is not doing Dynamic NLOS, I think the authors should discuss prior art on Dynamic NLOS and specify how their work is different. This also goes for the datasets, since it is a contribution the authors should specify what it provides over other dynamic NLOS datasets. Here is a couple of papers the authors should cite, but there are many more: 

1) Metzler et al., Keyhole Imaging: Non-Line-of-Sight Imaging and Tracking of Moving Objects Along a Single Optical Path, IEEE Transactions on Computational Imaging, 2021.

2) Nam et al., Real-time Non-line-of-Sight imaging of dynamic scenes

Clarity: I think the paper is mainly geared towards the Computational Imaging community, thus I think the authors would be better off having a preliminary section, where they specifically give a recap of Mamba. This would also make it easier to follow the contributions of the paper and better motivate some of the architectural choices made by the authors.
questions: questions: 1) The authors show the baseline Our-S, where multiple frames are not given to their method. However another apples-to-apples comparison might be if more data was provided to the baseline methods, this could mean testing the other methods not on spatially upsampled data, but on higher res data. In some sense this would mean both methods are tested on “equal” amounts of data. This could be done by keeping the total exposure time of the frame constant, (since otherwise rolling shutter-type motion blur artifacts might crop up I think?), but just using more sampling points. Have the authors tried this/related setups?
limitations: limitations: Limitations discussed in the paper.
paper_summary: paper_summary: 1.	This paper build a new dynamic NLOS dataset crafted for learning from synthetic data and evaluating models on real-world data for dynamic NLOS reconstruction.
2.	This paper introduce a Mamba-based method tailed for dynamic NLOS imaging.
strength_weakness: strength_weakness: 1.	The paper does not adequately explain the motivation for using Mamba.
2.	The proposed ST-Mamba and Cross ST-Mamba merely stack existing modules and lack innovation.
3.	Section 4.2 lacks detailed formulas explaining the operations of ST-Mamba and Cross ST-Mamba. Additionally, Section 4.3 is missing formal descriptions of the Hidden Volume Reconstruction and contains overly colloquial language.
4.	The paper lacks ablation experiments to validate the effectiveness of Spatial-Temporal Mamba and Cross ST-Mamba. Note that Section 5.3 does not adequately validate the effectiveness of Spatial-Temporal Mamba or Cross ST-Mamba. These two modules should be individually replaced with other existing feature extraction or fusion modules in ablation experiments to verify their effectiveness.
5.	The paper did not provide detailed settings for the ablation experiments of Spatial-Temporal Mamba. To verify the impact of Spatial Mamba and Temporal Mamba, the cascade structure of Temporal SSM and Spatial SSM should be replaced with two Temporal SSMs or two Spatial SSMs, rather than simply removing the Spatial SSM or Temporal SSM.
questions: questions: See the weaknesses.
limitations: limitations: See the weaknesses.

Comments: title: Look Forward to Feedbacks
Appreciate  the constructive suggestions.
comment: Thank you once again for your feedback and we are highly encouraged by the positive recommendations.
We are glad our response mostly met your expectations. 

Thanks again for dedicating your time to assist us with comprehensive experiments, and we will incorporate the components as per our discussion.
Thanks for the explanations. The authors have addressed my concerns. Therefore, I give a higher rating.
I would like to thank the author for his detailed response to my comments, which addressed most of my concerns. However, I still have a slight concern about the innovation of the proposed method, so I am giving it a weak accept.
Dear Reviewers,

Thank you sincerely for your review. We would greatly appreciate it if you could inform us of any remaining questions or concerns that you may have, so that we can address them promptly prior to the deadline. Alternatively, if you feel that your initial concerns are addressed, we would appreciate updating your evaluation to reflect that. Thank you!
Thank you for your thoughtful review and we are pleased that our response effectively addressed your concerns.

As suggested, we will include detailed descriptions of the imaging rate and the relevant scenes, further highlighting the significance of using a confocal setup for dynamic imaging.

Thanks again for the time you've invested in helping us improve our work.
Thank you very much for the responses to my questions. 


**Q2: Discussion about the prior art on dynamic NLOS.**

Thank you for the reply, I guess I didn't fully appreciate the point of the *NLOS Imaging systems* section in the related works before, but I guess this is just mainly a comment on trade-offs between a confocal and non-confocal setup, especially in dynamic scenes? I guess to me currently the related works section seems just like a comment on the current state of NLOS imaging, without being properly related to the themes in the paper i.e. NLOS imaging in dynamic scenes. 

Although this is not a must, in my opinion, this paragraph could be a bit more useful if the authors add the discussion on the FPS in the confocal setup and how that relates to the scenes that can be captured, then also the discussion with the non-confocal setup would make more sense. 

I think this is also the case with the *Reconstruction algorithms* section -- here I think the third paragraph the authors mention could help. 


**Q3. A preliminary section for Mamba.**

Thank you very much, I think this helps a lot. I also think in line with the comment from reviewer mnC2, the authors could probably better relate Mamba here itself to the current paper. For example, this preliminary section could be combined with the *4.1 Overview* to set up the problem more in line with the *Eq 1, 2* in the author's rebuttal comment to me.