Title: Learnable Burst-Encodable Time-of-Flight Imaging for High-Fidelity Long-Distance Depth Sensing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Time-of-Flight; End-to-End Learning
Abstract: Long-distance depth imaging holds great promise for applications such as autonomous driving and robotics. Direct time-of-flight (dToF) imaging offers high-precision, long-distance depth sensing, yet demands ultra-short pulse light sources and high-resolution time-to-digital converters. In contrast, indirect time-of-flight (iToF) imaging often suffers from phase wrapping and low signal-to-noise ratio (SNR) as the sensing distance increases. In this paper, we introduce a novel ToF imaging paradigm, termed Burst-Encodable Time-of-Flight (BE-ToF), which facilitates high-fidelity, long-distance depth imaging. Specifically, the BE-ToF system emits light pulses in burst mode and estimates the phase delay of the reflected signal over the entire burst period, thereby effectively avoiding the phase wrapping inherent to conventional iToF systems. Moreover, to address the low SNR caused by light attenuation over increasing distances, we propose an end-to-end learnable framework that jointly optimizes the coding functions and the depth reconstruction network. A specialized double well function and first-order difference term are incorporated into the framework to ensure the hardware implementability of the coding functions. The proposed approach is rigorously validated through comprehensive simulations and real-world prototype experiments, demonstrating its effectiveness and practical applicability. The code is available at: https://github.com/ComputationalPerceptionLab/BE-ToF.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Burst-Encodable Time-of-Flight (BE-ToF) and an end-to-end learnable reconstruction framework that jointly optimizes the coding function and ToF signals to obtain a better quality depth map.\n\nBE-ToF uses a short-period Modulation signal to have the advantage of using a high-frequency modulation signal (e.g. better details in the long distance) while adding a time delay in the demodulation signal to get over a phase wrapping issue on the far distance region.\n\nThe reconstruction network is based on RSCF-Net and takes synthetically generated raw ToF images using the BE-ToF Imaging model. This network can optimize both learnable coding functions that can optimize the demodulation signal of the ToF, and the network itself, which denoises and improves the ToF signals to produce high-quality depth in the end.\n\nIn the end, the paper shows the prototype of BE-ToF depthmap that uses the learned coding function and BE-ToF principle.\n\nSTRENGTHS AND WEAKNESSES:\nStrength :\n1. The paper proposes a fundamental solution to the existing issue with i-ToF sensing\n2. The authors built a real-life prototype of the sensor that is proposed in the paper and shows it actually works in a real-life scenario\n3. Both Qualitative and Quantiative experiments show superior quality of the proposed modality and the nework\n\nWeakness (suggestions for each weakness are written in the Qestions part)\n1. Lack of Clarity: The paper includes many details in different parts, but the connection is very weak between the details. \n2. BE-ToF's limitation: BE-ToF is able to acquire only in a specific range of depth maps.\n3. Limited Originality:  The paper's writing focuses a lot on the network and training part, while the network and training rather seem to be an incremental work from [10], which is rather chaing the backbone and adding more losses\n4. Limited real ToF dataset experiment: Experiments are mainly done on the synthetic training while only one image is added for the real life scenario.\n\nQUESTIONS:\n1. Lack of Clarity\n\n- Somehow, the explanation regarding the advantage of BE-ToF in Sec.3 is not really clear. Personally, what is written in the first and second paragraphs of the supplementary material helped me more to understand the purpose of BE-ToF. It would be great to merge the Suppl. material's BE-ToF principle to Sec.3\n\n- How the differential BE-ToF Imaging Model is used is not well written in the paper. It would be great to refer to Fig.2 in Section 3.1, then the use of the model will be clearer.\n\n- Also, It would be great to mention in the dataset paragraph  (4.1)  that the raw ToF images are generated/synthesized from Sec. 3.1, Eq.5. It was not really clear how the train is done from the conventional depth dataset but with the input of raw ToF signal in the first iteration of reading. Mentioning that briefly will make readers understand the paper much more easily. \n\n- It would be great to mention the reasoning behind the selection of RSCF-Net briefly.\n\n- The depth range in Fig.5 is created by rescaling the GT depth? Also, I see the range is always 3m. Is it because the short-period signal that goes up to 3m is used for the experiment? Then it would be great to mention briefly in Sec.4.2.\n\n- It's hard to understand lines 189-190 regarding albedo. In Eq2, roh_s is reflectance (line 123), but in lines 189-190 it is mentioned as albedo. Also, it's hard to understand why using the R channel of albedo is used as roh_s\n\n2. BE-ToF's limitation of being able to acquire only in a specific range of depth maps\n\n- This is more of a limitation of BE-ToF functionality. It's great to have a ToF that can capture the far distance without having issues with phase wrapping. But having a small distance window limits its real-life use case. It would be great to have some way to improve this issue or have a discussion on this as future work.\n\n3. Limited Originality\n\n- This paper mainly shows details about the reconstruction network and relatively small details about BE-ToF, but the network part seems to be an incremental work from [10] (in the main paper), which changed the reconstruction network backbone and added more loss in the forward model, and the network is not necessarily tied to the BE-ToF modality as well. \n- As proposing BE-ToF is the main differentiation from [10] (in the main paper), i'd love to see experiments that run more with real-life BE-ToF instead of synthetic training. I posted more comments regarding the experiment in the next section.\n\n4. Limited real ToF dataset experiment\n\n- It's understandable that doing a real-life experiment is challenging in this setup. But the experiments are mostly synthetic, and only one picture of a real-life ToF image is included. In a real-life scenario, raw ToF images are really noisy, sparse, having physically induced noise like MPI, Motion Blur, and artifacts from surface material of the objects in the scene, which can make the input of the raw ToF signal significantly different from the simple ToF model introduced in the paper. \n- Also, experiment on Sec.5 doesn't seem to use what was trained from the network training (at least its not mentioned), if the network is not used in real, the readers can understand that the real raw signal is not really close to what is used in the synthetic trained input. Synthetic-based trained ToF models still show results on the real ToF setup, for example, from DeepToF or Gu et al [1]. If the depth in Fig.8 is from the BE-ToF and Network, then please specify. And if not, it would be great to see how the result looks on the trained network.\n- As far as I see, the reconstruction network itself is independent to how the raw ToF signal is acquired. One suggestion is to test the reconstruction network on the real-life ToF dataset that contains raw ToF and depth map as GT. This way, the authors can prove the performance of the proposed network compared to the existing ToF pipelines. As far as I checked, there are two real datasets with raw ToF included (CroMo dataset [2], HAMMER dataset [3])\n- Another possible experiment is to modify the system prototype to make the signal compatible with other trained baselines (e.g. changing frequency and without delay to try FisherToF model, adding dual frequency model to test on DeepToF model) and show the qualitative experiment on the depth images. It would be great to see whether the real-life quality of depth on different setups acts similarly to what is shown in Fig.5.\n\nMinor suggestion but, there is not much related work on ToF imaging. There are two more works that are quite relevant to this paper to be added in the related work: [1] (as multi-frequency modulation, and learnable kernel), [4] (RGB-based phase unwrapping from raw ToF)\n\n[1] Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset, Guo. et a.\n\n[2] CroMo: Cross-Modal Learning for Monocular Depth Estimation, Verdié et.al.\n\n[3] On the Importance of Accurate Geometry Data for Dense 3D Vision Tasks, Jung et.al.\n\n[4] Wild ToFu: Improving Range and Quality of Indirect Time-of-Flight Depth with RGB Fusion in Challenging Environments, Jung et.al\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of long-distance high-fidelity depth imaging with time-of-flight (ToF) systems. Mainly, this work proposes a joint optimization of the coding function with a depth estimation neural network in an end-to-end manner. The work uses a differentiable modeling of the acquisition model, including two regularization functions to achieve implementable coding functions, i) a polynomial function to achieve a square wave, i.e., binary valued functions ii) total variation, so that the coding function does not transition from 1 to 0 so fast. The paper also introduces a restormer-based depth estimation network which includes, beyond the traditional MSE recovery loss function, an information-theory loss function based on the Fisher information matrix to overcome the SNR in the measurements. A comprehensive evaluation of the proposed method, along with SOTA comparison, is provided, showing improvements of the paper in determined distance ranges. This is also validated with an experimental ToF prototype, showcasing the method in real acquisition scenarios.  Supplementary material includes a detailed mathematical derivation of the sensing model and additional results with other datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. The proposed method addresses the physical constraints for the coding functions, which are required for implementation. Three different regularizations are proposed to obtain wide binary coding functions and maximize the depth quality.\n\n2. The results are superior to the state of the art in all cases, both quantitatively and qualitatively. The results in terms of MAE are better for different levels of signal-to-noise ratio, which guarantees the robustness to noise and general improvement.\n\n3. The proposed method bridges a gap present in dToF and iToF to achieve accurate and long-distance depth imaging in a limited depth range.\n\nWeaknesses\n\n1. An in-depth analysis of the coefficients of the regularizers ($\\gamma_1, \\gamma_2, \\gamma_3$) used in Eq. (11) is required. Ablation studies and convergence analysis could be added.\n\n2. Although higher quality is achieved for long distances, the range of action is limited by $T_m$. The difference between the minimum and maximum depth is only 3 meters. A discussion of how this could be increased would improve the work.\n\n3. While the method is compared to other state-of-the-art approaches, the evaluation is limited to using their reconstruction networks with the proposed coding functions. A more comprehensive and fair comparison would involve training the coding functions from other methods with the same reconstruction networks used in this paper, to better isolate and assess the contribution of the coding stage itself.\n\n4. There should be an analysis of the “optimality” of the resulting coding functions. Is there a way to interpret them? Additionally, there should be a comparison with other coding functions  [1,2].\n\n[1] Gupta, Mohit, et al. \"What are optimal coding functions for time-of-flight imaging?.\" ACM Transactions on Graphics (TOG) 37.2 (2018): 1-18.\n[2] Gutierrez-Barragan, Felipe, et al. \"Practical coding function design for time-of-flight imaging.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\nQUESTIONS:\n1. Why in Figs 5, 6 of the main manuscript and Figs 2 and 3 of the supplementary material not include a quantitative metric to fairly evaluate the quality of the proposed approach? \n\n2. Since for promoting values a degree-two polynomial is used, is it guaranteed that this polynomial achieves exact 0 and 1 values? Why not simply use a binary activation function with a Straight-Through Estimator for the gradient as widely used in binary neural networks? [1]\n\n3. Did you try with fewer measurements? $k<4$? \n\n4. Usually, loss functions are chosen to have range $[0,\\infty]$, but the used regularizer has negative values. Why not simply choose a polynomial function $f(x) = (x-1)^2 (x)^2$? Is there any advantage to the proposed approach?\n\n[1]  Le, Huu, et al. \"Adaste: An adaptive straight-through estimator to train binary neural networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\nLIMITATIONS:\nAlthough the paper mentions that the method is constrained to a narrow range of distance $\\sim 3 $m,  it does not mention how to address those limitations regarding the limited range in future work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Burst-Encodable Time-of-Flight (BE-ToF) imaging, a novel approach for long-distance depth sensing. By transmitting light in burst patterns and capturing the phase shift over an entire burst period, BE-ToF overcomes limitations of traditional iToF systems such as phase ambiguity. The method employs a learnable, end-to-end framework that optimizes optical coding and the neural network for depth reconstruction. The effectiveness of BE-ToF is demonstrated through both simulated experiments and a prototype setup showing superior results in long-range scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper proposes a technically novel method for long-range depth imaging by introducing the burst-encodable time-of-flight (BE-ToF) paradigm, enabling phase unwrapping with just a single modulation frequency. This significantly simplifies system complexity. The approach integrates a hardware-aware, end-to-end learning framework optimizing binary code design and depth reconstruction jointly. The authors demonstrate clear performance improvements over baselines on synthetic data. Additionally, comprehensive ablation studies and validation using a real-world prototype indicate promising practical applicability for high-precision long-range depth sensing.\n\n\nWeaknesses:\n\nThe evaluation methodology for indoor long-range scenarios lacks clarity. It appears that the dataset involves artificially extending indoor scenes to represent longer distances, but this is not explicitly stated or adequately explained. It is uncertain whether simulated indoor long-range scenes sufficiently capture conditions typically found in long-range (usually outdoor) scenarios, such as differing ambient illumination, diverse materials, and atmospheric effects. Furthermore, important practical factors like real sensor noise or beam divergence seem unaccounted for. Thus, the strong quantitative results reported might not accurately reflect real-world performance, necessitating additional validation with genuine long-range data.\n\nWhile the hardware prototype provides a convincing qualitative depth map demonstration for distances between 22–25 meters, it lacks quantitative evaluation against ground-truth data (e.g., LiDAR). Quantitative assessment on real-world scenarios and comparative analysis with existing methods would significantly strengthen the reliability and practical relevance of the results.\n\nMinor:\n\nOptimizing coding and reconstruction through learning follows previous work, limiting novelty primarily to the burst-encodable concept itself.\n\nThe stated motivation targets autonomous driving and robotics; however, the depth window presented is narrow. Practical automotive applications would require extending this range dynamically, which is not demonstrated. Additionally, experiments involving dynamic or moving scenes are absent but crucial to understanding potential method limitations.\n\nWhile hardware specifications are provided, comparisons regarding laser power, measurement number (K), and network complexity against baselines remain unclear. Clarifying these details and including runtime analysis would improve understanding of the method's practical feasibility.\n\nQUESTIONS:\nCould the authors explicitly explain the methodology behind the synthetic long-range indoor dataset? Are conditions such as lighting, materials, and atmospheric factors reflective of outdoor long-range environments? Addressing this clearly would strengthen the confidence in reported results.\n\nCould the authors provide quantitative evaluation results of the hardware prototype using common ground-truth methods (e.g. LiDAR)? Demonstrating quantitative comparisons with previous methods on real-world data would significantly enhance credibility.\n\nHave the authors considered dynamically shifting the time delay to extend the practical depth range for applications like autonomous driving? \n\nCould the authors include experiments demonstrating performance in dynamic or moving scenarios?\n\nCould the authors clarify whether system parameters such as laser power, measurement count (K), and network complexity match those of baseline methods? Including a runtime analysis would help readers assess the practical implications and feasibility of the proposed approach.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes a long-distance, high-depth-resolution indirect time-of-flight (iToF) imaging system. The core contribution is the design of learnable modulation and demodulation codes that improve depth resolution, particularly under low and spatially varying signal-to-noise ratios. These codes are burst-type, targeting specific long-distance depth ranges rather than continuous coverage. The authors validate their approach through simulations and demonstrate hardware implementation, presenting a single real-world depth image.\n\nWhile the proposed method shows promise, I am currently inclined against accepting the manuscript due to (1) lack of simpler and fairer baseline comparisons and (2) lack of more real world results despite the availability of hardware.\n\nSTRENGTHS AND WEAKNESSES:\n**strengths**\n- The manuscript is well written and easy to follow. It builds a hardware prototype for validation. \n- Usage of the double well loss function to encourage the decoding function to be binary valued at any time is quite interesting. I haven't seen this loss function before.  \n- first-order difference loss to avoid narrow peaks (high BW) is also interesting. However, why not sample the D(t) functions at the fastest switching rates possible by hardware and thereby avoid this loss function altogether. Also, loss functions encourage a behaviour but do not guarantee that it will happen. How is that handled? \n**weaknesses**\n- While the proposed technique enhances long-distance sensing, it does not extend the depth range---unlike dual-modulation, frequency switching, or M-sequence-based methods, which are specifically designed for that purpose. As a result, the current comparisons may be somewhat unfair to these techniques, as they are evaluated on metrics outside their intended strengths. \n-  A more fair and complete comparison---currently missing---is an ablation study where a single-frequency waveform (e.g., a sinusoid with a frequency chosen to match the same depth range as used in this manuscript) is employed, but temporally delayed by appending zeros. This raises the question: is the use of neural-optimized codes truly necessary to achieve good performance?\n- Comparison with space filling codes (optimal codes for ToF imaging [1]) or M-sequences [2] which can also lead to long distance (and range) is missing.\n     - [1] Gupta, M., Velten, A., Nayar, S.K. and Breitbach, E., 2018. What are optimal coding functions for time-of-flight imaging?. ACM Transactions on Graphics (TOG), 37(2), pp.1-18.\n     - [2] Kadambi, A., Whyte, R., Bhandari, A., Streeter, L., Barsi, C., Dorrington, A. and Raskar, R., 2013. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Transactions on Graphics (ToG), 32(6), pp.1-10.\n- While the manuscript builds the hardware, it includes only a single experimental result, even in the supplementary material. Moreover, the result lacks quantitative or qualitative comparisons with alternative coding strategies. Capturing and evaluating multiple scenes under varying conditions---such as different times of day---and comparing the performance across different codes would significantly strengthen the manuscript and provide a more comprehensive validation of the proposed system.\n\n- Eq. 5 is inaccurate. The light itself is Poisson distributed. Given that the system is operating in low-light conditions, this is quite important. \n   - I am at Eq. 9 and noticed that light is considered Poisson accurately here. So, it is just a typo in Eq. 5. Please fix it. \n\n**Composition: **\n-  Equations are part of the text and should therefore include appropriate punctuation. Specifically, Eqs. 1–5, 7, 9, and 11 should end with commas, while Eqs. 6, 9, and 10 should end with periods.\n- Lines 133–134: Names derived from proper nouns must be capitalized. Use 'Poisson' and 'Gaussian' instead of 'poisson' and 'gaussian'.\n- Move Fig 5 to top of the page as well, like the rest of the figures. If not, it comes after the text and appears like it is part of the text.\n\nQUESTIONS:\nPlease answer the questions asked in Strengths And Weaknesses section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-07-02"}