Title: ToF-IP: Time-of-Flight Enhanced Sparse Inertial Poser for Real-time Human Motion Capture
Venue: NeurIPS 2025
Accepted: yes
Keywords: IMU

ToF

Motion Capture
Abstract: Sparse inertial measurement units (IMUs) provide a portable, low-cost solution for human motion tracking but struggle with error accumulation from drift and sensor noise when estimating joint position through time-based linear acceleration integration (i.e., indirect measurement). 
To address this, we propose ToF-IP, a novel 3D full-body pose estimation system that integrates Time-of-Flight (ToF) sensors with sparse IMUs. 
The distinct advantage of our approach is that ToF sensors provide direct distance measurements, effectively mitigating error accumulation without relying on indirect time-based integration. 
From a hardware perspective, we maintain the portability of existing solutions by attaching ToF sensors to selected IMUs with a negligible volume increase of just 3\%.
On the software side, we introduce two novel techniques to enhance multi-sensor integration: (i) a Node-Centric Data Integration strategy that leverages a Transformer encoder to explicitly model both intra-node and inter-node data integration by treating each sensing node as a token; and (ii) a Dynamic Spatial Positional Encoding scheme that encodes the continuously changing spatial positions of wearable nodes as motion-conditioned functions, enabling the model to better capture human body dynamics in the embedding space.Additionally, we contribute a 208-minute human motion dataset from 10 participants, including synchronized IMU-ToF measurements and ground-truth from optical tracking.
Extensive experiments demonstrate that our method outperforms state-of-the-art approaches such as PNP, achieving superior accuracy in tracking complex and slow motions like Tai Chi, which remains challenging for inertial-only methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduced Time-of-flight sparse inertial pose estimation.\n\nBy introducing four cameras at the limbs, the paper obtains absolute distance measurements between the body and the environment, which is supposed to stabilize body pose estimation.\n\nThe main contributions are:\n1) the use of ToF cameras at the limbs\n2) a 200min dataset for pose estimation from ToF sensors \n3) a method that improves pose estimation slightly\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- Novel sensors for pose estimation\n- a 200min dataset of ToF and inertial sensors for pose estimation\n- good results, the new sensors together with the architecture deliver state of the art results\n\nWeakness:\n- the method does not seem very novel, despite showing slightly improved results\n- the method uses slightly different sensor layout, making comparisons hard\n- the paper compares to baselines that do not have access to ToF data, making comparisons less fair (comparing with baselines that get the tof data as an additional input would allow a fairer comparison of the method)\n- the sensor hardware and noise could be discussed in more dteail\n\nQUESTIONS:\nAbstract:\n\nWill the training code be released as well?\n\n27: the sentence sounds like there is only one way of doing that. IMUs measure local accelerations and local angular velocities. One naive way of integrating that is through linear acceleration. Typically more complex state estimation with calibration schemes, e.g. Kalman Filters, are required. I recommend a rephrasing.\n\n28-30: This is not an amplification of error that is related to IMU, but an error source due to human modeling. I recommend to rephrase it slightly. \n\n44: Please clarify what kind of distances are measured, between sensors, between environments?\n\n48-50: why are those locations chosen for the ToF sensors? \n\n53: how is multisensor data hierarchical? Please elaborate. Multisensor data from different limbs does not follow a clear hierarchy in my opinion.\n\n56: unclear what is meant by intra-node? Inbeween sensors? Or just one sensor? Maybe define it prior or name it differently, e.g. per-sensor individual integration?\n\n56: What do you integrate? Is it numerical integration? Why do you integrate orientations? How do you obtain orientations? IMU measure angular velocities. If you like previous methods assume orientations as input, please state so clearly :)\n\n60: The dynamic spatial positional encoding - sounds simply like learnable positional encodings of 3D positions in that part. Later in the method section, it becomes a bit clearer. Can you clarify the motivation on why your positional encoding is superior to learning an encoding based on sensor position?\n\n119: maybe cite the very closely related baseline ultra inertial poser here. \n\n141: how do you obtain the original joint position which is needed for integration? How do you obtain the original joint velocity which is needed when double integrating accelerations?\n\n147: Do you mean $\\tau$? t seems to be overloaded as the limit of the interval and the current timesamp\n\nSuppl Eq 1 - seems to ignore starting conditions, if the original velocity estimation is not 0, the velocity error should have a constant bias. Please add starting conditions as an assumption, i.e. the original position is always (0,0,0) and the starting velocity is always 0. In the same regard, your theoretical derivation seems irrelevant if you discard your obtained results (cubic variance) in favour of a quadratic relationship between variance and time. Also the explanation and reasoning why the relationship is actually quadratic and not cubic as you derived does not become clear to me. Can you clarify where and how you use your theoretical findings, why other works and practical findings come up with σ^2 t^2 instead of your theoretical findings and why going from continuous to discrete time sampling drops a factor of t in the variance (after equation 6 in supplementary)\n\nMethod 3: I am not sure what insight you generate from section 3, and how it is related to your contribution\n\n158: you do not use the standard layout used in previous work but instead place the sensors traditionally placed at the knees, at the feet. While there is no inherent disadvantage of this setup, you dont discuss the change of layout and also other methods. Additionally, a comparison to other methods becomes questionable when a different layout is used in your evaluation.\n\n158: what is the energy consumption of your hardware? How much runtime would a wearable device have with just ACC, ACC+GYRO and ACC+GYRO+TOF\n\n160: what is the orientation of your TOF sensors, how are they pointing? Can you also indicate that in the figure 3\n\n170: What is the reason for partially using transformers and partially LSTMs, what is your finding on the suitability of each architecture. What if you only use LSTMs in you model. What if you only use transformers in your architecture?\n\n175: Why does orientation need to be integrated?\n\n179: “Integrating measurements from other sensing nodes”. How are you integrating distances? Are you talking about the mathematical integration?\n\n186-187: This argument does not become clear to me for once all the values from one node, can interact with all other nodes through the MLP, but also all measurements from a single node can interact with all measurements from that same sensor node. Additionally, along the temporal axis previous works employ LSTMs, and thus both the inter sensor node (spatial) interaction and single node interaction (temporally) is possible. Your claimed limitation of previous work does not become apparent.\n\n218: can you clarify on your positional encoding. The encoding seems to come purely from orientation, acceleration and distances. How would you distinguish sensors when their orientations, accelerations and distances are the same? What is the reason to force your learned sensor embedding $\\phi_n$ thorough the cosine embedding again? Do you have results that show that this additional passing though the sinusoidal encoding is needed?\n\n231: what subsets of AMASS do you use?\n\n233: do you assume shaped input or mean shape?\n\n240: what are the 20 types of motion? Can you provide more examples of your dataset. Both your dataset and the one from DIP are rather slow paced. Does your dataset contain fast sequences? How does your method perform on fast paced data?\n240: Can you report results on TotalCaptureReal dataset? The DIP dataset does not contain translation, has inaccurate ground truth, very noisy sensor data and \n\n251: How do you obtain orientations from sensor data? What is the drift of your sensor data over time?\n252: does your method work window based, or frame based? How do you do inference?\n\n256: can you report the units of your metrics? Can your report translation errors?\n\nTable 1: How do you train the other methods, considering that you have different sensor layout of your method?\n\nTable 1: Do you ever finetune on the FIP-DB dataset or is it used purely for evaluation?\n\nTable 1: Can you train the closest baselines with additional ToF input to investigate their benefit of ToF data. On DIP your architecture seems to be much worse (15.52 sip error vs 13.71 sip error of PNP) when not using ToF data.\n\n4.2.3: Do you predict smpl angles or do you represent only the joint positions directly? What is the velocity output and angle output used for? Do you evaluate those outputs?\n\nSupp 1.2) How does the ToF sensor work in the wild. How is the noise? Do you have an analysis of the noise of the ToF sensor. You just highlight the sensors characteristic within one sequence. How much base noise does the sensor have? What is the mean distance error?\n\nSupp 4.1) do you add noise in the simulation to model real world sensor noise?\n\nSupp 4.5) Why do you not report global translation error, if your method is capable of global translation tracking? Also \n\nVideo Qual results) the results of your method look surprisingly jittery (especially in the first part of the video), why is your jitter metric lower than other methods jitter?\n\nThanks for your work. Despite my many critique points I am happy to adjust my score depending on how the questions are answered during the rebuttal.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a hardware and algorithmic pipeline that augments the standard 6‑node sparse‑IMU mocap setup with four on‑board time‑of‑flight (ToF) sensors mounted on the wrists and ankles.\n\nOn the hardware side, the authors claim only a 3 % volume increase while keeping the canonical 6‑IMU attachment pattern, so the system remains lightweight and untethered. \n\nOn the algorithmic side, they introduce\n\nNode‑Centric Data Integration (NCI): Each sensing node forms a token that first fuses its own ToF depth patch, IMU acceleration, and orientation (“intra‑node”), then exchanges information through a Transformer encoder (“inter‑node”). \n\nDynamic Spatial Positional Encoding (Dyn‑PE): A motion‑conditioned positional encoding that replaces the static index‑based encodings typically used in Transformers, letting token positions evolve with the body’s motion across time. \n\nThree RNNs predict joint velocities, positions, and rotations from the Transformer embeddings.\n \nA new 208-minute paired ToF‑IMU‑optical dataset (10 subjects, 20+ movements) is introduced, and the authors also provide a synthetic ToF extension of the public DIP‑IMU dataset.\n\nSTRENGTHS AND WEAKNESSES:\n## **Strengths**\n\ni) The paper is well written and easy to follow.\n\nii) The problems of the existing purely IMU‑based systems (integration and drift errors) are clearly explained, and the motivation for the proposed approach is compelling.\n\nii) The architecture is well-suited for the task.\n\niii) Experiments and ablations are thorough and justified.\n\niv) The limitations of the proposed method are discussed in detail.\n\nv) The proposed dataset, ToF‑IP‑DB, that pairs body‑mounted ToF with IMU and optical ground truth, will be a valuable resource for the community.\n\n## **Weaknesses**\n**i) Unclear Runtime & Power Metrics:**\n- The claim of “real‑time” performance is unsupported. No FPS, per‑module latency (Transformer vs. LSTM), or end‑to‑end throughput is reported.\n- Power consumption of the IMU–ToF nodes is not provided.\n\n**ii) Occlusion Robustness:**\n\nTime‑of‑flight ranging can fail when body parts occlude one another; no discussion about this is present.\n\n**iii) Figure Legibility:**\n\nFig. 4 is too small or low‑contrast to read clearly in print.\n\n**iv) Missing Units:**\n\nBenchmark results lack metric units (e.g., mm, cm), making quantitative comparisons ambiguous.\n\n**v) Typographical Errors:**\n- Table 1 headers contain typos.\n- The caption of Fig. 5 mentions Case 1 and Case 4; is that correct?\n\nQUESTIONS:\n**i) Performance & Power:**\n\nWhat is the inference throughput (FPS) of the full pipeline on your target hardware?\n\n  - What are the per‑module latencies for the Transformer and LSTM components?\n\n  - What is the typical battery life or power draw of a single IMU–ToF node under continuous operation? (if possible to obtain)\n\n\n**ii) Dynamic Positional MLP (L. 221):**\n\nIs the MLP that computes each node’s dynamic positional encoding shared across all sensing nodes, or is there a separate MLP per node?\n\n**iii) Supplementary Video Lag:**\n\nIn the supplementary video (timestamps 2:10–2:13), why is there a visible lag between the actor’s movement and the estimated pose during fast actions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new framework to capture egocentric 3D human motion using the integrated Time-of-Flight (ToF) and Inertial-Measurement-Unit (IMU) sensing system. The authors propose a unified Transformer-based framework featuring a Node-Centric Data Integration strategy and a Dynamic Spatial Positional Encoding scheme to explicitly model intra-node and inter-node sensor interactions. They also contribute a new ToF-IP-DB dataset containing more than 3 hours of human motion data captured by their proposed sensing system paired with optical motion capture. Authors conducted extensive evaluation against state-of-the-art methods (although none of them shares the same sensor input) and ablation studies to support their argument.\n\nSTRENGTHS AND WEAKNESSES:\n**Stengths**\n– The authors design and build a wearable motion capture sensing system by integrating 4 ToF sensors into a standard 6 IMU setup, and they validate the hardware end-to-end, greatly boosting the paper’s practical credibility.\n– They propose a novel Transformer-RNN architecture with Node-Centric Data Integration and Dynamic Spatial Positional Encoding, enabling principled fusion of sparse depth maps and IMU streams to recover full 3D kinematics.\n– The authors collected the real data of human movement captured from their proposed sensing system paired with optical motion capture, providing both training and evaluation data to the community.\n\n**Weaknesses**\n– The measured accuracy gains over IMU-only approaches are modest relative to the added hardware complexity, raising questions about the cost–benefit trade-off in practical deployments. Considering the fact that ToF sensor may not be generalized to the real-world scenario including uneven terrain, this accuracy gain seems insufficient.\n\nQUESTIONS:\n– I wonder how much cost and battery inefficiency we need to consider by adding the sparse resolution ToF sensors. Since the authors mentioned that they carefully balanced between the power consumption and the performance, I would like the authors to provide this information in the main paper more comprehensively.\n\n– From their supplementary video, it seems like their system runs in real time. Since this is an important factor for the wearable motion tracking system, I would recommend the authors to state their time and computational complexity (FPS, FLOPs) in the main paper.\n\nLIMITATIONS:\nI do not have any other concern than the practicality of the new sensor addition with insufficient accuracy gain.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ToF-IP, a system for human motion capture that fuses sparse Inertial Measurement Units (IMUs) with body-mounted Time-of-Flight (ToF) sensors. The authors claim that direct distance measurements from ToF sensors can mitigate the inherent drift accumulation in IMU-only systems, which is a well-documented and critical challenge in the field. The authors present three main contributions: a wearable hardware prototype, a new motion capture dataset (ToF-IP-DB), and a Transformer-based software framework featuring Node-Centric Integration and Dynamic Spatial Positional Encoding. The authors also contribute a new 208-minute motion dataset with synchronized IMU-ToF measurements and demonstrate that their method achieves state-of-the-art accuracy, especially for complex and slow motions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n**+** The work addresses a fundamental and persistent problem in wearable motion capture: the quadratic error accumulation resulting from the double integration of noisy accelerometer data in IMU-only systems.   \n\n**+**  The design of a lightweight, wearable prototype that integrates ToF sensors onto a standard 6-IMU layout with minimal volume increase is a practical and valuable engineering contribution.   \n\n**+** The creation and planned release of the ToF-IP-DB dataset, which includes synchronized IMU, body-worn ToF, and optical ground-truth data, is a significant service to the research community, enabling future work and benchmarking in this specific subfield.\n\nAreas for improvement\n\n**-** The claims of superior performance are not supported by a detailed statistical analysis. The absence of error bars, confidence intervals, or significance testing makes it impossible to determine if the reported improvements are meaningful or the result of random variance.   \n\n**-** The results indicate that while ToF integration may reduce positional error, it may not improve or even worsen, pose jitter. While ToF-IP shows improvements in positional and angular errors, it performs poorly on the Jitter metric. In Table 1, the jitter of ToF-IP on the DIP dataset (0.17) is no better than that of PIP (0.17) or PNP (0.17). The ablation study in Table 2 shows that adding ToF data without the proposed NCI module (Case 2) increases jitter from 0.07 to 0.13 on the ToF-IP-DB dataset compared to an IMU-only baseline (Case 1). This suggests that while ToF's direct distance measurements may help correct for drift, the inherent noise in ToF sensors introduces high-frequency errors, or jitter, into the final pose estimation. The paper fails to discuss this drift-jitter trade-off, presenting a potentially incomplete and overly optimistic picture of the proposed method's performance.\n\n**-** The contribution of Dyn-PE requires more explanation. The authors argue that static positional encodings, common in NLP, are ill-suited for wearable sensors whose spatial relationships are constantly changing. Their proposed solution is to make the positional value for each node a learned function of the entire sensor input at that timestep, i.e., PE_dyn (t) is a function of X_all (t). This approach raises a fundamental question: if the positional encoding already depends on the full input, what new relational information is the Transformer's self-attention mechanism expected to learn? The core purpose of self-attention is precisely to compute dynamic, input-dependent relationships between tokens. Dyn-PE appears to be pre-computing a simplified version of this relationship and adding it to the input, which feels redundant. While the ablation study in Table 3 shows a marginal improvement in some metrics (e.g., Positional Error on ToF-IP-DB drops from 7.36 to 6.31), this result is presented without any statistical validation. It is therefore impossible to know whether this small gain is a genuine contribution of the architecture or simply a product of random initialization or training dynamics. The claim that this technique \"enhances spatial awareness\" requires more rigorous proof to be considered a significant contribution.\n\nQUESTIONS:\n**-** Could you provide a more robust statistical analysis of your key results, particularly for the ablation studies (Table 2) and SOTA comparison (Table 1)?\n\nActionable Guidance: We suggest running the experiments multiple times (e.g., 3-5 times with different random seeds) and reporting the mean and standard deviation for key metrics like Positional Error and SIP Error. This would provide crucial insight into the stability and reliability of your method's improvements.\n\n\n**-** How does the ToF-IP system behave when one or more ToF sensors are temporarily occluded? Does performance degrade gracefully, or does it fail catastrophically?\n\nActionable Guidance: For the rebuttal, I would find it very helpful to see an analysis of this failure mode. This could be a quantitative experiment where you synthetically block the ToF signal for a portion of a test sequence, or a qualitative analysis of the system's output during self-occlusion events. A discussion of whether the Transformer's self-attention mechanism learns to down-weight unreliable (occluded) sensor inputs would be particularly insightful.\n\n\n**-** Could you elaborate on this trade-off between improved positional accuracy and output jitter? Is the jitter a direct result of the raw ToF sensor noise, and have you explored post-processing or filtering techniques to mitigate it?\n\nActionable Guidance: We encourage a brief discussion on the source of this jitter. Furthermore, demonstrating the effect of a simple filtering technique (e.g., a Savitzky-Golay filter or a Kalman filter) on the output poses could be very effective. A comparison of the smoothness of ToF-IP's output with and without filtering, and against an inertial-only baseline, would be highly illustrative.\n\n\n**-** Could you provide more detail on the domain gap between your simulated ToF data and the real data from your hardware prototype?\n\nActionable Guidance: To help with this question, please provide a side-by-side qualitative comparison of a real ToF depth map  and a simulated one for a similar body pose. Please also clarify if your Unity-based ToF simulation  models common real-world noise sources, such as multi-path interference or absorption by certain clothing materials.\n\nLIMITATIONS:\nIn Section 6, the authors acknowledge several limitations: the system's performance is contingent on ToF sensor reliability, it may be compromised by occlusions, ToF sensor noise introduces jitter, and the approach needs validation in more diverse real-world scenarios. While these are valid points, the discussion is brief and lacks depth. Several other limitations are not acknowledged by the authors:\n\n**-** The performance of optical ToF sensors is known to be highly dependent on the environment. They can struggle with highly reflective surfaces (e.g., mirrors, polished floors) or highly absorbent surfaces (e.g., black fabrics), which can corrupt distance measurements. These are common conditions that are not addressed.\n\n**-** The proposed Transformer-based architecture is computationally intensive. The authors report running experiments on a high-end NVIDIA RTX 4080 GPU. While they claim \"real-time\" performance, they provide no concrete metrics like frames-per-second (FPS) or processing latency in milliseconds. Without these metrics, the claim is unsubstantiated and the practicality of deploying this model on a portable or embedded device remains unclear.   \n\n**-** The paper mentions that the collected data is \"synchronized\" but provides no details on the hardware or software mechanisms used to achieve this synchronization between the IMUs and ToF sensors. For high-quality sensor fusion, precise, low-latency time-stamping is critical, and the lack of detail on this implementation aspect is a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}