Title: Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgery
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Reconstruction

Ophthalmic Surgery

Hand–Instrument Interaction

Surgical Vision
Abstract: Accurate 3D reconstruction of hands and instruments is critical for vision-based analysis of ophthalmic microsurgery, yet progress has been hampered by the lack of realistic, large-scale datasets and reliable annotation tools. In this work, we introduce OphNet-3D, the first extensive RGB-D dynamic 3D reconstruction dataset for ophthalmic surgery, comprising 41 sequences from 40 surgeons and totaling 7.1 million frames, with fine-grained annotations of 12 surgical phases, 10 instrument categories, dense MANO hand meshes, and full 6-DoF instrument poses. To scalably produce high-fidelity labels, we design a multi-stage automatic annotation pipeline that integrates multi-view data observation, data-driven motion prior with cross-view geometric consistency and biomechanical constraints, along with a combination of collision-aware interaction constraints for instrument interactions. Building upon OphNet-3D, we establish two challenging benchmarks—bimanual hand pose estimation and hand–instrument interaction reconstruction—and propose two dedicated architectures: H-Net for dual-hand mesh recovery and OH-Net for joint reconstruction of two-hand–two-instrument interactions. These models leverage a novel spatial reasoning module with weak-perspective camera modeling and collision-aware center-based representation. Both architectures outperform existing methods by substantial margins, achieving improvements of over 2mm in Mean Per Joint Position Error (MPJPE) and up to 23\% in ADD-S metrics for hand and instrument reconstruction, respectively.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the first large-scale, real-world, high-quality dataset for 3D reconstruction of hand–instrument interactions in ophthalmic surgery settings. Based on this dataset, it further presents two benchmarks for bimanual hand pose estimation and two-hand–two-tool pose estimation. The two baselines achieve comparable or better performance than SOTA methods in general settings.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n1. This is the first work that considers hand-object pose estimation in a surgery setting, which may benefit future researchers from both the general and medical areas.\n2. The multi-stage automatic annotation pipeline is reasonable and cost-effective.\n\n### Weakness\n1. No detailed application is discussed in this paper. What could the model be used for after training on this dataset, more than reconstructing hand and object motion?\n2. When it comes to hand motion estimation, the temporal consistency should be considered. This could be measured using the Acceleration and Jerk error, like in [1].\n3. For hand performance, any comparison with the most advanced model, HaMeR [2]? (both **without domain-specific training** and **exclusively trained on this dataset**)?\n4. Lastly but most importantly, considering the medical background of this paper, any metrics to measure surgical quality could be reported and analyzed in benchmarks, like *time and motion*, *flow of operation*,  *overall performance*, etc., in [3]?\n\n[1] Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera, CVPR 2025.\n[2] Reconstructing hands in 3D with transformers, CVPR 2024.\n[3] JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS): A Surgical Activity Dataset for Human Motion Modeling, MICCAI workshop 2014.\n\nQUESTIONS:\nPlease refer to the weaknesses in **Strengths And Weaknesses** part. For a medical-related paper, it is important to consider more medical-related evaluations and applications.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new dataset for bi-manual hand and instrument motion in surgical settings. The authors design a multi-stage optimization pipeline to obtain ground-truth MANO hand parameters and 6-DoF object motion sequences. In addition, they propose H-Net and OH-Net for reconstructing hand and object motions. The paper provides extensive implementation and annotation details, making the dataset and methods reproducible and informative for future research.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths:**\n\n- The modeling of bi-manual hand and instrument motion is novel and relevant, especially in the context of surgical applications.\n- The dataset is valuable, containing multi-view RGB-D sequences with dense MANO hand meshes and 6-DoF instrument annotations.\n- The authors provide extensive implementation details and thorough documentation of the data collection and annotation pipeline.\n\n---\n\n**Weaknesses:**\n\n- While Line 8 claims the annotation pipeline is automatic, Line 103 reveals that it still relies on manual corrections for per-frame instance masks.\n- While a few visual examples are provided, the overall quality of the ground-truth annotations is not systematically evaluated. This is not a clear weakness, as large-scale annotation quality remains a challenging open problem, but it would benefit from further discussion.\n- No video results are shown for the reconstructed motions from the proposed OH-Net.\n\nQUESTIONS:\nThe paper claims the annotation pipeline is automatic (Line 8), but Line 103 mentions that manual corrections are required for per-frame instance masks. To what extent is the pipeline automated in practice? Could the authors clarify how much manual intervention is still involved?\n\nWhile the authors provide video examples to demonstrate the quality of the generated annotations, these individual cases are insufficient to assess the overall quality of the dataset. Evaluating the accuracy and consistency of large-scale automatically annotated data is a challenging but essential problem, particularly when the main contribution lies in the proposed annotation pipeline.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work aims to facilitate dynamic 3D reconstruction of hands and surgical instruments in ophthalmic surgery. The authors (i) introduce an automatic annotation pipeline, (ii) construct a large-scale dataset using 8 RGB-D cameras, (iii) establish two benchmarks, and (iv) present two baseline methods. The main contribution of this paper is the OphNet-3D dataset, which contains 7.1M frames featuring 40 surgeons, 12 surgical phases, and 10 surgical instruments. Based on this dataset, the authors introduce two benchmarks and baseline methods. The proposed methods achieve state-of-the-art results for hand reconstruction and hand-instrument joint reconstruction.\n\nSTRENGTHS AND WEAKNESSES:\n< Strengths >\n\nThis paper has the following strengths:\n\n(i) The OphNet-3D dataset has a significant contribution to the research field. It is carefully constructed with 8 synchronized RGB-D cameras using the proposed annotation pipeline. As shown in Table 1 and Figure 1, the dataset is substantially larger and more diverse than existing datasets. It includes detailed annotations such as MANO hand meshes and 6-DoF instrument poses. \n\n- As described in NeurIPS Paper Checklist, the authors will open-source all data, code, and model weights. This contribution can facilitate kinematic analysis of critical operational details (e.g., surgeon hand movements, surgical tool usage).\n\n(ii) The authors establish two benchmarks and introduce baseline methods for bimanual hand motion estimation and two-hand-instrument interactions. They also evaluate existing methods on the proposed OphNet-3D dataset, which supports the dataset’s utility as a benchmarking tool. \n\n(iii) This paper is interesting and well-organized. The techniques used for dataset construction and baseline methods are generally appropriate. \n\n---\n\n< Weaknesses >\n\nThis paper has the following weaknesses:\n\n(i) Despite the dedicated efforts for instrument motion annotation (Sec. 3.2), the proposed baseline struggles to capture fine-grained instrument articulation, as shown in the supplementary material. This limitation is understandable for now, given the challenges posed by the fine-grained motion scale in ophthalmic surgery.\n\n(ii) This paper contains several incomplete or unclear sentences, which may hinder readability and clarity:\n\n- L228-230: \"Based on the data split, bimanual hand pose estimation and (2) hand-instrument interactions.\"\n\n- L257-258: \"Mean Per Joint Positional Error between each hand-instrument interaction pair (MRRTEh,o)\"\n\n- L367: \"on all metrics ()\"\n\n- L840: \"CoordConv []\"\n\n- Equation (4) defines \"2d\", but equations (9), (10), and (22) use \"pj2d\".\n\n(iii) This paper lacks ablation studies to analyze the contribution of individual components in the proposed method, limiting assessment of design choices and their effectiveness.\n\nQUESTIONS:\n(i) How effective are the losses L_inter and L_sdf in addressing interpenetration? An ablation study on these losses would be valuable, given that hand-instrument interaction is the central focus of this work.\n\n(ii) The proposed OH-Net is trained with monocular training images from all 8 camera views. Is the same training protocol applied to the compared methods? Clarification is important for fair benchmarking.\n\n(iii) Can the authors provide a comparison of inference speeds between their baselines and other methods? Since the baselines rely on multiple large pre-trained models, they may be much slower than existing approaches.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}