Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgery
Authors: Ming Hu
Zhengdi Yu
Feilong Tang
Kaiwen Chen
Yulong Li
Imran Razzak
Junjun He
Tolga Birdal
Kaijing Zhou
Zongyuan Ge
Venue: NeurIPS 2025 spotlight ()
Keywords: 3D Reconstruction
Ophthalmic Surgery
Hand–Instrument Interaction
Surgical Vision
Primary area: applications

Abstract: Accurate 3D reconstruction of hands and instruments is critical for vision-based analysis of ophthalmic microsurgery, yet progress has been hampered by the lack of realistic, large-scale datasets and reliable annotation tools. In this work, we introduce OphNet-3D, the first extensive RGB-D dynamic 3D reconstruction dataset for ophthalmic surgery, comprising 41 sequences from 40 surgeons and totaling 7.1 million frames, with fine-grained annotations of 12 surgical phases, 10 instrument categories, dense MANO hand meshes, and full 6-DoF instrument poses. To scalably produce high-fidelity labels, we design a multi-stage automatic annotation pipeline that integrates multi-view data observation, data-driven motion prior with cross-view geometric consistency and biomechanical constraints, along with a combination of collision-aware interaction constraints for instrument interactions. Building upon OphNet-3D, we establish two challenging benchmarks—bimanual hand pose estimation and hand–instrument interaction reconstruction—and propose two dedicated architectures: H-Net for dual-hand mesh recovery and OH-Net for joint reconstruction of two-hand–two-instrument interactions. These models leverage a novel spatial reasoning module with weak-perspective camera modeling and collision-aware center-based representation. Both architectures outperform existing methods by substantial margins, achieving improvements of over 2mm in Mean Per Joint Position Error (MPJPE) and up to 23\% in ADD-S metrics for hand and instrument reconstruction, respectively.

Towards Dynamic 3D Reconstruction of Hand-Instrument Interaction in Ophthalmic Surgery
Ming Hu
Zhengdi Yu
Feilong Tang 1,,3+ Kaiwen Chen Yulong Li Tolga Birdal Kaijing Zhou Zongyuan Ge
Imran Razzak Junjun He?
Monash University
2 SShangaai Al Laboratory MBZUAI "Eye Hospital, Wenzhou Medical Univeristy
Imperial College London ming. hu@monash. edu, 2. y.y220@mmeriil... ac. uk https:/ 3d, github.
Abstract
Accurate 3D reconstruction of hands and instruments is critical for vision-based analysis of ophthalmic microsurgery, yet progress has been hampered by the lack of realistic, large-scale datasets and reliable annotation tools. In this work, we in- troduce OphNet-3D, the first extensive RGB-D dynamic 3D reconstruction dataset for ophthalmic surgery, comprising 41 sequences from 40 surgeons and totaling 7.1 million frames, with fine-grained annotations 12 surgical phases, 10 instru- ment categories, dense MANO hand meshes, and full 6-DoF instrument poses. To scalably produce high-fidelity labels, we design a multi-stage automatic annotation pipeline that integrates multi-view data observation, data-driven motion prior with cross-view geometric consistency and biomechanical constraints, along with a combination of collision-awer aware interaction constraints for instrument interactions. Building upon OphNet-3D, we establish two challenging benchmarks bimanual hand pose estimation and pose two dedicated architectures: H-Net for dual-hand mesh recovery and OH-Net interactions. These models leverage a novel spatial reasoning module with perk-perspective camera modeling and collision-aware center-based representation. Both architectures outperform existing methods by substantial margins, achieving improvements of over 2mm in Mean Per Joint Position Error (MPJPE) and up to 23% ADD-S metrics for hand and instrument reconstruction, respectively.
interaction reconstruction and pro-
for joint reconstruction
wwo-hand-two-instmene
Introduction
Modern ophthalmic microsurgery represents one of the most delicate surgical paradigms in medicine, requiring sub-millimeter precision in instrument manipulation under restricted workspace condi- tions 63 75). While advances in robotic tools and surgical training platforms have improved treatment outcomes, current skill assessment methods still rely heavily expert supervision and subjective feedback, limiting their scalability and objectivity [[5] 39 54). In current ophthalmic surgical training paradigms, trainees predominantly rely direct supervision from instructors for skill acquisition and performance evaluation. However, this approach imposes significant demands instructional resources, particularly considering the time-intensive nature of surgical mentorship and the critical requirement for real-tine weal-time feedback in complex microsurgical procedures. Recent studies in computer-assisted surgery [37] 73 [7 [1] reveal that kinematic analysis of surgical tools and operator hand movements could enable objective skill evaluation, personalized training feedback,
Equal contribution, "Coreeponndig author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Table 1: Comparison with existing 3D hand reconstruction datasets. OphNet-3D is the first surgi- cal RGB-D dataset, offering high-resolution videos and rich annotations of complex hand-instrument interactions. It far exceeds prior datasets in scale and diversity and uniquely supports real-time dual- hand and multi-object reconstruction tasks. Task abbreviations: HPE: Hand Pose Estimation, OPE: Object Pose Estimation, HR: Hand Reconstruction, HOI: Hand-Object Interaction Reconstruction, HMOI: Hand and Multi-Object Interaction Reconstruction, Video: Video-level Reconstruction.
Datasel Properties
Task Support
Medalite
Views
Resolution
Patticpantts
Obejects
Mations
Valco
RAND pocy
Genera
Reel RGB
224w 254
13
130.7K
Gencral
Real RGB
256-116
150K
Generi
RGB
S0-140
100
Genral
Real RCII--
1760):711
471.8KC
DeaYCB (ooce
1
Gestral
Roal RGB D
640)-401
20
552K
ARCTIC (008622
19
Oescral
Real RGB
2800)2200
11
2.154
evosss
Geseral
Redl RGBhmmaap))
Clinical
Systh RGBDDD
3
1920- 1080
88 3K
HLP 3D pEcoars
X
Clitical
Synth RCBB
90
$48x-400
31.7K
OptNas 11 (Curd)
Chinical
Roal RCil--D
848--100
10 12
7.1M
and even real-time intraoperative guidance 73 [0]. Goodman et al. 21 curated the 1,997-video
AVOS corpus and trained real-time multitask model that parses hands, tools, and actions. Building on AVOS, Vaid et al. [79] reframed surgeon-hand hand recognition as semi-supervised, single-class 2D detection task that mixes many noisy unlabeled frames with few labeled ones. Both efforts still rely on 10-frame 2D snippets and lack long-range temporal, depth, pose, or multi-view cues. Meanwhile, other approaches [4 58 continue to depend on external motion-capture rigs or wearable sensors, introducing constraints that conflict with sterile surgical environments and disrupt natural workflow.
Recent advances in monocular [13
94 82 59 and multi-view [84 85 RGB-based 3D hand-object
interaction reconstruction have demonstrated notable success in general-purpose scenarios, offering promising avenue for contactless skill assessment in ophthalmic surgical training. Estimating surgeons' hand and instrument poses from single RGB image enables the quantification of critical operational details- as grip posture and tool orientation- that are closely linked to surgical quality and clinical outcomes. However, directly applying these methods to ophthalmic microsurgical settings remains challenging due to the highly constrained operating space, fine-graines motion scale, and frequent occlusions and complex interactions between both hands and multiple instruments. These factors pose significant difficulties for accurate motion structure reconstruction using existing algorithms. Moreover, the lack of high-precision, realistically annotated 3D datasets specific to ophthalmic surgery further limits methodological development and evaluation in this domain.
To address the aforementioned limitations, we introduce OphNer-3D,, the first large-scale dataset capturing dynamic 3D hand-instrumen interactions in real-world ophthalmic surgeries. Collected with eight synchronized RGB-D cameras, it comprises 41 cataract surgery sequences by 40 surgeons (avg. >12 minssequence). totaling over 7.1 M aligned RGB-D frames annotated for 12 surgical phases and 10 instrument categories. We apply a multi-stage automatic annotation pipeline to recover dense 3D hand meshes and 6D tool poses from these multi videos. Finally, we define two evaluation benchmarks maachnakks-bbmanual hand-pose estimation and hand instrrummttnt interaction and propose a unified baseline, OH-Net, for joint reconstruction of woo-hand-two-tte interactions. Our contributions are:
We present OphNet-3D, the first large-scale, real-world, high-quality dataset for 3D reconstruction of hand-instrument interactions in clinical surgical settings. OphNet-3D delivers an unprece- dented combination of dataset scale, camera views, participant diversity, instrument, motion and object variety, and supported task types, at 2.5x the size of the largest existing general 3D hand reconstruction dataset and 70x that of the largest surgical 3D hand reconstruction dataset.
We propose a multi-stage automatic annotation pipeline that reconstructs 3D hand meshes and 6D tool poses from multi-view RGB-D videos using optimization with data-driven hand motion prior combined with biomechanical constraints and interaction-aware refinement.
Based on the proposed dataset, we establish two benchmarks: one for bimanual hand pose esti- mation and another for hand-tool interaction. We further introduce a unified baseline, OH-Net, which jointly reconstructs wo-hhandtwo-tee interactions with effective spatial reasoning, and demonstrate its performance through extensive quantitative and qualitative results.

---

Carger Secinding (-
OphNet-3D's acquisition framework, comparisons with other datasets, and phase-frame distributions. Left: (a) synchronized multi-camera rig with 8 calibrated RGB- D cameras and directional LED lights; (b) participants perform standardized cataract surgery maneuvers on pig-eye simulators under an ophthalmic microscope; (c) boxplots of pixel errors for eight cameras across three calibration runs. Upper Right: comparative visualization of OphNet-3D and existing 3D hand datasets. The horizontal axis denotes the total number of RGB frames, while the vertical axis indicates the number of categories across the participants, objects, and motion settings. Circle diameters encode the number of provided segmentation-man instances; datasets without mask annotations are represented by dashed outlines. Lower Right: distribution of frame counts and clip durations for each phase.
Figure 1:
OphNet-3D Dataset
Data Collection. OphNet-3D is captured in multi-camers studio consisting of 8 Intel@ RealSense D435 RGB-D cameras recording at 30 FPS, along with 3 high-powered directional LED lights aimed at the hands ensure uniform illumination (Fig. T left). The cameras capture at resolution of 848x480, and the multi-view system calibrated using an ArUco calibration board. The detailed setup of the recording platform can be found in B.]
We adhered to standard cataract surgical protocols, segmenting the procedure into 12 distinct phases. Detailed definitions and demonstration for each phase are provided in B.3 During the procedures. 10 different surgical instruments were employed and all instruments were scanned using a ZEISS ATOS Q bluc-light 3D scanner, with corresponding images of the physical instruments and their 3D CAD models presented in B.5 During each recording session, two additional assistants were present operated the recording system, while the other assisted with the surgical workflow, such as instrument handover, to ensure procedural continuity. All surgical actions were recorded on ophthalmic surgical simulation platform utilizing pig eyes, accompanied by synchronized video captured from an ophthalmic surgical microscope perspective. Finally, all videos were temporally ophthalmologist, who also annotated the
aligned between the hand-vier and microscope-vic surgical phase locations and performed a secondary verification.
Data Statisties. We recorded total of 41 sequences from 40 unique participants (one participant contributed two sequences, wearing blue and white gloves respectively), of whom 20 have more than one year of surgical experience and 20 have less than one year. Raw videos in dataset have average duration of 16 minutes, comprising over 9.5M RGB frames. After filtering out transitional segments via phase localization annotation (B.4). the final OphNet-3D contains 565 phase segments with a total duration 300 hours and over I.M RGB frames, as detailed in Tab. [] In addition, OphNet-3D provides segmentation annotations for more than 2 instances.
Automatic Annotation Method
Given the input videos {>*) from multiple views with T frames containing two hands interacting with possibly two surgical instruments, we aim to reconstruct the 3D hand-instrument interacting motions by recovering the hand meshes and 6D instrument poses. To efficiently auto-label the captured RGB-D videos with accurate mesh and pose annotations from multiple views, we design an optimization-ban based multi-stage automatic annotation pipeline shown in Fig. 2

---

Multi-viow RGB-D
Observation
Optimization
Joint Optimization
H
SDI
M
[pl.e
Calibration (PnP)
Surgical Instrument Scan
Point Cloud Generation
Tool Pose Initialization
(H)
Regioration & ICP
Deyth
Soute
Crea van
Figure 2: Our automatic annotation pipeline. Given a multi-view RGB-D video sequence input, our pipeline reconstructs the 3D hand mesh and 6D instrument pose in multi- manner. H and W represent the initialization network for hands 61 and instance segmentation masks 667.
In the first stage, we use the 3D) CAD models scanned as described in Sec. 2 to track per-frame 6l) instrument poses. Moreover, we leverage the state-of-the-art SAM2 67 with manual corrections to obtain per-frame accurate instance masks for hands ( (MM2) and instruments (* ('M,)) for further point cloud segmentation. For per-frame scene point cloud generation, we first compute point cloud from depth images for each view and merge the point clouds across views. To segment out the region of interest (i.e. two hands and instruments) with cross-view filtering, we project the merged point cloud back the 8 views and keep the points that project onto the hand innndinnttuumet region for more than half of the views to get the final per-frame scene point cloud P,, which can be further split into hand Phand and instrument P In the second stage, our goal is to reconstruct the 3D hand and 6D instrument pose from the multi-view RGB-D videos recorded by calibrated cameras. To this end, we leverage the state-of-the-art 2D & 3D hand pose estimation method [6] [2] to initialize per-frame hand motion state in the camera coordinate system, as well utilizing the masks for global registration and ICP 42 to estimate an initial instrument pose for each camera view. Recovering accurate
interaction challenging due frequent occlusions, truncation and mutual confusion. As remedy, we propose a
joint optimization scheme with a hand motion prior model HMP [4] and biomechanical constraints in the third stage inspired by (T..
3.1 Hand Motion Annotation
Hand Representation. We parametrize the hand shape and pose using the MANO hand model (69 which uses standard vertex-based linear blend skinning with learned blend shapes. At each time step the hand motion state is represented as:
de 10 "h -h.T")
where 0f E 3 denotes the local pose of 15 hand joints, 3 1O represents the hand shape parameters, and is expressed using the axis-angle representation. while the translation T." E R3 specifies the wrist position 3D space. The handedness indicated by h E (l.r). representing left or right hand, respectively. Using these MANO parameters and the skinning function W(.), we can reconstruct the 3D hand mesh V E 3x38778 and the 3D hand keypoints J E pax21 with V = W[ B*) P(g*), S) + 7/1 and J = LV where W(-) denotes the skinning function, H represents the posed parametrie hand template, and 1778 E R x is row vector of ones. The function P(-) returns the hand joint positions in the rest pose, S defines the skinning weights, and L is pre-trained linear regressor for estimating joint locations from mesh vertices.
define the global wrist state. Specifically, the orientation of e R
Hand Initialization. For each camera view, we initialize per-frame 3D hand motion state in camera coordinate system leveraging an efficient two-hand motion tracking system based on [61] with hallucinated detection handling, We further obtain per-frame 2D hand keypoints from ViTPose [33 and hand bounding box extract image patches, feeding into [6 for a coarse-to-fin 3D motion state prediction. Finally, we compute the weighted sum of the motion state based on the visibility to obtain the final initilization of the global motion state qql the world coordinate system. Next, we convert the motion state into the world coordinate system using the calibrated camera information (Ri, T(} of camera view
w " = RR 1.ch
and
" "7 =
R

---

where " and " are the hand wrist orientation in world and camera space. "T and T are the translation. Here we omit for simplicity. For 2D observations, we initialize from ViTPose [83]. MediaPipe [52 and the 2D re-projection from 61 with confidence filter to extract final per-frame
2D keypoint h I2221 for each view as observation for following optimization, where the 2D re-projection is performed with weak-perspective camera parameters predicted by [61.. Moreover, we provide more details regarding the post-processing of the observations, including hallucination handling and missing detection infilling in the D.1
Optimization. To recover the hand meshes from multi-view RGB-D videos, we propose an iterative fittigg algorithm by minimizing the following objective with regularization and biomechanical hand constraints (11.
as well as hand motion prior [14].
N.
Ei Bi -.)
E C2zaC2a + AsnCau) + Ag + Asd + Cbto + Epriors
(3)
where N₁ is the number of camera views and L2a is the joint 2D re-projection loss minimizing the difference between 2D hand keypoints observations (J/,]T Tt=o and the re re-projection of the 3D
keypoints obtained from MANO model with parameters {0): Br, of b1 in current global state y.:
T Caa = Σ Σ'(c (J)-
kEiL.r 1=
where pl-) is the Geman-McClure robust function
C is a confidence filter mask for joint J R,, T,, T) ) the re-projected 3D keypoints, and is the perspective camera projection for each view with collected camera intrinsics K E R3x3 and extrinsics (R), T/). To improve the pixel alignment across views, we propose silhouette-baad basee error term Lall
19
visibility. J I.
Cal Σ M NR(V. K R T!)1
(5)
where NR(.) is differentiable renderer that renders the 3D mesh for the hands into the 21) mask and M;C is the clean segmentation mask for the hand h at timestep t. To have precise 3D alignment between the mesh reconstruction and the real scene, we compute the 3D mesh loss and minimize the distance between the frame point cloud P and the MANO hand mesh:
T Laa ΣΣ Ny
(PE ((()
vi
n,lb where j(i) arg min lIp v/ll
(6)
hE(l.r) 1= i=1
where p/ the j-th point the hand point cloud P at timestep and oi is the i-th vertex of the hand mesh V The reconstructed hand mesh normal of the vertex is represented as n/. Moreover, we reduce the jitter of the hand motion and improve the temporal smoothness by integrating Smmothh
Lymooth
Σ
Jhh
J/l12 + g(( 0/
AE(L.) 1=
where g(-) represents the geodesic distance. To further improve the plausibility of hand motion qualityand reduce jitter for natural movement, compute the Sprior Lz + AgLo + Agha by utilizing data-driven motion prior [14] where the latent code is z" Inspired by [9]]. It ensures the motion constrained under the learnt prior space, by penalizing the negative log-likelihood:
C.
Σ
Σ
log. o
hE(l.] t=
To explicitly prevent implausible poses produced during optimization, we further propose Lbio (Saa( + Cbr AApalm + >angle<Cnggle) which consists of angle regularization terms with biomechanical constraints [71] and and an angle limitation constraint. More details regarding the loss calculation are provided in D.2
5

---

3.2 Instrument Motion Annotation
By leveraging multi-view RGB-D frames together with camera pose information, our method can accurately annotate per-frame 6D instrument pose of the instruments. To recover 3D hand motion under the challenging surgical scenario (e.g. light, occlusion).
Obtaining Canonical Local Instrument Geometry. We laser-scanned each instrument to obtain high-rest nigh-resolution 3D meshes (Sec.2) For articulated instruments (e.g. phacoenulsification handpiece), additionally separate and scan them into two articulated parts, well their rest pose and maximum relative pose articulation as shown in Fig. Please see 16 for the detailed visualization of the instruments.
Acquiring Instrument Articulation. The articulated instrument (e.g. handpiece) surface is pa- rameterised by the 6D pose of each base part and ID articulation relative to a canonical pose. In particular, the 6D instrument pose can be represented as [RG, T For each instrument, we define a 3D parametric model OC-) leveraging the scanned instrument parts and relative pose state. Given the 6D pose 8f E I6 and the ID relative articulation factor a E R', where a E [o, 1] uniformly parameterizes pose deformation across different models. Here, a 0 corresponds the rest pose, and a = 1 to the maximally articulated pose. representing the deformation state. The instrument 3D mesh O(09, a.) E 3 can be reconstructed, where N, is the instrument vertices number.
Initialization of 6D instrument pose. We obtain accurate per-fiaee per-frame 6D instrument pose leveraging the multi-view RGB-D information. As mentioned, we first perform instrument segmentation and 2D tracking for each camera view using SAM2 [67] with manual correction to obtain clean 2D segmentation masks for both hands and instruments. Moreover, we merge the depth image across views to generate the point cloud for the whole scene. After that, segment out the region of interest leveraging a cross-vice view filter, which projects the point cloud into all camera views and keeps the points projected onto the hand-in instrumettmmnt region for more than half of the views to get the point cloud for instruments pabjj By running RANSAC- alignment of the 3D instrument mesh and P After that, we refine the alignment with ICP [42 to obtain the initial rigid transformation from instrument canonical coordinate system to the world coordinate system. Finally, we run a simple Chamfer distance-based optimizer for better alignment and to obtain the articulation a and the final 6D instrument pose {R; Ti). By applying the 6D pose to the instrument model, we can obtain the 3D mesh in the world coordinate system. Note that our surgical scenario contains various two-hand- thhand-tw--insttee two-instrument interactions, thus we represent the instrument in each hand as O
global registration, we have a coarse global
3.3 Joint Optimization
Naively putting the hand and instrument together may result in implausible hand-instrumen interac- tions such as enter-penetration and unnatural contact. To jointly optimize 3D hand pose and instrument pose and introduce more constraints for the interaction, propose the following objectives:
N.
En lh. TE Ri, Ti, ) = Er + > (aan + Aza Cad + Ainter Linter Asat Lastf
(8)
1m
Specifically, Lui represents the silhouette loss term that computed between the combined hand- object mask 'M..0 and the rendered mask NR(V., O K Rİ Ti) of the 3D hand-object mesh. Laa is calculated between ground truth point cloud P. and the predicted hand and object mesh. Moreover, we leverage the interaction loss Linter to constrain the hand-instrument contact following 28 as Linter ARLR + AA where L and LA are the attraction loss and the repulsion loss, which penalize the interpenetration between hand and instruments, and minimize the distance between the interacting hand and instrument in the possible contact region, respectively. We provide more details regarding the calculation in the Appendix. Finally, we refine the interaction by applying a modified version of Signed Distance Field (SDF) loss Cadr. for which we provide more details and an ablation study in D.2
4 Baseline and Experiments
Our dataset can enable various downstream tasks for pose estimation and recognition. this section, introduce two benchmarks built upon our dataset. We first propose the evaluation protocols of

---

each benchmark and provide a detailed analysis of our dataset. Furthermore, we present baseline methods corresponding to the benchmarks with comparison against the state-of-the oof-the-art art methods to demonstrate the effectiveness of our approach. More implementation details are provided in
4.1 Evaluation Protocol
Data Split. To ensure each phase has balanced samples, we split training, validation, and test sets by sub- jects, which have 30, 3, 8 subjects sep- arately. Based on the data splít, biman- ual hand pose estimation and (2) hand- instrument interactions. We provide more details regarding the data distribution and data quality analysis in the Appendix. Note that in our experiments on both bench- marks, we train the model on the monoc- ular training images from all views, in- cluding both egocentric and allocentric for rich supervision.
dataset into
Dilernee
W
Denile
Buckhome
Read
Input
Devd of
OC-)
Object Roordd cadtl HN
Deiuder
Evaluation Metries. Our goal is to recon- struct accurate 3D motion of hands and instruments during complex surgical opera- tions from video. Specifically, we propose metrics to quantify estimate quality and compare our baseline method against state- of-the-an art hand-object pose approaches.
Figure 3: Overview of the OH-Net. The backbone image encoder outputs the image feature, which is then used to decode the hand/instrument centre heatmap and segmentation mask. MANO decoders predict their cor- responding weak-perspective camera parameters. De- coupling the instrument branch forms H-Net.
Bimanual Hand Pose Estimation: To evaluate the accuracy and plausibility of the hand reconstruc- tion pipeline, we report the Mean Per Joint Positional Error (MPJPE), Mean Per Vertex Positional Error (MPVPE) in mmm after root (hand wrist) joint alignment. To explicitly measure the relative translation error, we report Mean Relative Root Translation Error (MRRTE).
Hand-instrumer Interaction: To quantify the reconstruction quality, we report the same evalua- tion metries (MPJPE, MPVPE, MRRTE) as bimanual hand pose estimation benchmarks. For instrument pose estimation quality, we evaluate the commonly used ADD-S score, which measures the average distance between the model vertices transformed by the ground truth the estimated poses, following [48
72 76 192. Specifically, we report the percentage of the transformed instruments with a vertex positional error less than 10% of the instrument diameter. We further report the Mean Articulation Error (MAE) to evaluate the articulation of instruments, calculating the absolute error between the ground-truth articulation factor and the prediction percentage, excluding the rigid instruments without articulation. For the interaction quality, we evaluate Mean Per Joint Positional Error between each inter-eenet Volume (Pen) in cm3
interaction pair (MRRTEh, o) and Mean
4.2 Bimanual Hand Motion Estimation
We now set up the benchmark for biman- ual hand pose estimation from a monocular RGB input image. Acquiring accurate 3D hand pose is essential the surgical sce- narios during manipulation.
Table 2: Quantitative evaluation results for bimanual hand motion estimation. We compare method with state-of-the-ant the-art hand reconstruction methods on local hand poses. MPJPE, MPVPE, and MRRTE are reported in millimeters (mm) after root alignment.
Parametric Representation. In the task of (monocular) bimanual hand pose estima- tion, our goal is to reconstruct the 3D pose of the two hands from RGB input video. In order to obtain the detailed geometry of the two hands, adopt the parametric model MANO [69] as our mesh representa- tion to predict (0) B Φi T following
Method
MPIPE
MPNDE
MRRTF.
MPPPP:
MINTFF |
MRRTE
18.67
18.91
18.89
18:75
Ince
Wil
19.08
19.87
37.27
20.19
20.34
38.75
Inaefaand
18.02
17.06
32:43
19.38
19.14
32.77
ACR 50
18.15
18.57
33.20
18.85
10.28
33.59
H-Nett Onte N-Net)
18.57
18:48
33.92
18.95
19 14
34.15
H.Net
17.39
18.72
31.66
17.65
18.65
31.89
H-NosDD
18.28
16.37
26,86
15.97
16.18
26.39
the dataset settings. Given the MANO parameters, the 3D hand mesh V/2 and the 21 hand keypoints
7

---

Table Quantitative evaluation results for two-hand interactions. We compare our
method with the state-of- -ftee----thh-rrrr art hand reconstruction methods on local hand poses.
Split
Method (mos))
MPJPE (mmm)
MPVPE (nun) 2 MRRTE (mm))
ADDS (16)
Pen (mmmm)
MRRTE. (mmm)
Hasson et al.
19.57
21.45
39.78
56.66
7:87
27.58
HFL-Not
4R
17:51
18:27
37.45
55. 64
7.06
25.67
Val
HOISDE
17.05
18:22
34,36
50.91
5.69
24.13
OH-Net (Wle T-Net)
17.48
18:31
32.61
66.87
14.83
(:4)
21.50
OH-Net
1841
19:34
OH--Nei-D
76.88
5.14
17.33
Hasson
20.04
56
HFL
17.66
61
5.27
OHI-Nel (wlo T-Net)
17.89
18.06
33.25
65
14.51
6.45
21.97 9.
OH--Net (Ours)
17.34
18.36
31.58
20.79
11.17
5.91
20.11
Ol--Net-D
15.94
16.13
26.44
76.31
9.62
5.18
17.45
J can be regressed. We adopt the commonly used weak-perspective camera model following 94
61 44 90
56
to estimate the 3D translation.
Baseline. To address the problem of reconstructing bimanual hands from a monocular RGB(-D)
image. we introduce H-Net the baseline approach, without the instrument branch. As shown in Fig. 3 the model takes as input the image at timestep to extract the image feature ft E R Dx H W where D is the dimension of the feature map. Subsequently. the following 3 regression heads predict the segmentation mask M E 3 (i.e. left hand, right hand, background), MANO parameter map MILano E 2 hand center heatmap Chand 2xxHx respectively. Leveraging the collision aware center-based representation [74] 90) for hands, we disentangle the bimanual hand features while pushing away the centers that are too close in the repulsion field. In the following, the MANO parameters q² for each hand is extracted by combining with the Hand Center map and the instance segmentation mask. After obtaining the 3D mesh by MANO model W(-) with keypoints, we use the output relative translation AT E R² from T-Net to model the fine-grained relative transformation from the left hand the right hand, incorporating the strong spatial features as prior knowledge. The weak- peak-peeseeetive camera parameter is represented as ct' Moreover, we denote the RGB-D input based version as H-Net-D. Finally, the network supervised by the weighted sum of the hand center loss and the mesh parameter loss:
L A + ApI2d + Asa + AoilCall + Emano + Aeeg
where focol is the focal loss for the hand center map. Lmano = Agea + Agha is the weighted sum L2 loss of the MANO parameters. We provide the training and implementation details in App. [D3
Results. We evaluate the performance of our baseline models on surgical bimanual hand reconstruc- tion tasks, and compare them against state-of-the-a oo-the--rt hand pose estimation methods. As shown in Tab. 2 our method significantly outperforms prior approaches such as InterWild (55) DIR [68], and IntagHand [4), achieving the best overall performance across different metries, including MPJPE, MPVPF, and MRRTE. These results highlight the importance of domain-specin design: our hand center detector and tailored parameterization improve robustness in surgical environments, where factors such as gloves, occlusions, and instrument-indueee hand articulation pose unique challenges. Notably, the T-Net module contributes to finer pose refinement by learning spatial alignment from segmentation masks, leading to consistent improvements across both joint-wise and veveex-level metrics. These findings validate the effectiveness of our baseline in modeling surgical hands with reasoning.
high precision, serving as a strong foundation for subsequent
4.3 Hoo--Hand-Instrumen Interactions
In this section, we propose the benchmark for hand-instrument interaction, which aims to reconstruct the 3D mesh for two hands and the 6D pose for the surgical instruments.
Parametric Representation In terms of two-hand-instrument Interaction baseline, our task is to reconstruct the 3D meshes of the two hands as well as the 6D pose of the in-hand surgical instruments. We keep the hand representation as MANO [69] for consistency. For 6D instrument pose estimation, we leverage the parametric model ( a₂) ) introduced in Sec. 3.2 represent the surgical instruments with articulation. Specifically, the 3D mesh is regressed using the parameters of 6D pose 8g E R and the ID relative articulation factor a, E [0, 1] which controls the articulation.
Baseline. As discussed in Sec. 4.2 integrating the instrument 6D pose estimation branch forms the full model of OH-Net. As the first method to reconstruct wwo-hand-two-ohject, we propose to disentangle the features and mesh representation along with an explicit handler for interaction. As
8

---

illustrated in Fig. 3 the extracted feature map is followed by 5 regression heads, yielding hand center map Chand E 2x and object center map C E R The segmentation head predicts the instance mask M R We extend the collision-aware center-based representation mechanism to fit in the task and push away the two close instrument center and hand centers. The instrument parameter map is regressed as M R I14HHxW which contains the 6D pose and ID articulation factor for both instruments. Next, T-Net predicts ÅT E I for the relative translation between the left hand, right hand, and between their interacting instruments. OH-Net supervised with the objectives below:
L Aooaa focat + Apjdd + Azd Sall fmsno + >aeg Seef Jobj Gool
(10)
where Ctool represents the loss function for instrument supervision. Specifically, Ltool is composed of Ln and L, for the 61) instrument pose and L., for the ID articulation. Moreover, Lej2d and L is also defined for instrument and optimized with pre-defined bounding box of the instrument mesh following [50) We refer the readers to the appendix for detailed implementation details.
Results. We evaluate the proposed OH-Net frame- work on the hand-instrument interaction benchmark. As shown in Tab. [ our method achieves state-of- the-art performance across all evaluated metrics, in- cluding MPJPE, MPVPE, MRRTE, and ADD-S. The full OH-Net benefits significantly from the joint mod- cling of hands and instruments, with joint training improving both interaction accuracy and articulation consistency. Notably, OH-Net is the first method ca- pable of reconstructing hands and two interacting instruments simultaneously, with explicit disentan- glement and spatial reasoning. The RGB-D version, OH-Net-D, shows further gains across all metrics, especially in interaction tration (Pen) and MRRTE".0. The MAE and ADD-S results highlight reliable articulation estimation and instrument localization. Moreover, Fig. 4 showcases qualitative examples of predicted meshes and multi- view renderings, demonstrating our model's ability to preserve hand-instrumen contact and recover detailed interactions even under occlusions. Additional results and visualizations are provided DD.
Site
Figure 4: Qualitative results on the Hand- instrument interaction benchmark. Each row shows a sample from the test set, with columns displaying: (1) input image, (2) mesh prediction, (3) rendered mesh from side view, and (4) bird s-eee view.
scores such as Pene-
5 Discussion
Related Work. Surgical vision research has recently evolved beyond traditional tasks 57 33
toward 3D perception, including scene reconstruction [93, I. depth estimation 12 87 and navigation 166, 86). Despite advances in datasets like AVOS (1 and MM-OR (58), most sys- tems remain limited to passive monitoring rather than real-time surgical assistance. Mcanwhile, 3D hand reconstruction has progressed from general benchmarks [96] (6) to synthetic medical datasets 813. though lacking clinical realism. Monocular approaches have evolved from single- hand models 13 61 to two-hand systems [94] (6) and hand-object interactions [22 89, using template- maapette-based methods [18 25 and geometric constraints [5] (95). Our work bridges the clinical gap with a targeted 3D perception framework for ophthalmic microsurgery using real RGB-D data capturing fine-grained, two-hand, multi-instrument interactions. See Tab. and A for details.
Conclusion. In this paper, we introduce OphNet-3D, the first large-scale, real-world RGB-D dynamic 3D reconstruction dataset for ophthalmic microsurgery, comprising 41 sequences from 40 surgeons (7.1 M frames), annotated with 12 surgical phases, 10 instrument classes, detailed MANO hand meshes, and 6D instrument poses. We develop multi-stage automatic annotation pipeline that integrates monocular hand-prior models, segmentation poiniiccooo alignment, and biomechanics- based joint hand-in indt-inmmmmmet optimization. Leveraging this dataset, establish two new benchmarks- bimanual hand pose estimation and inntrimnutmment interaction and propose H-Net and OH-Net, which achieve state-of-the-art performance all metrics ()
Limitations & Future Work. This study offers new insights into dynamic 3D reconstruction in ophthalmic surgery but has three main limitations: it relies on data from single center (requiring

---

multi-center validation for better generalizability); strong microscope illumination causes instrument- tip overexposure (which could addressed with synchronized motion-capiin capture RGB or infrared imaging); and it has not yet integrated the microscope view for joint reconstruction of the ocular surface, hands, and instruments (may be a key direction for future work boost clinical relevance).
Acknowledgements
This work supported by the Center of Excellence for Antimicrobial Therapeutics Discovery and Innovation (CEATDI) [grant 8002003].
T. Birdal acknowledges support from the Engineering and Physical Sciences Research Council [grant EP/X01 1364/1]. T. Birdal was supported by a UKRI Future Leaders Fellowship [grant number MR/Y018818/1).
References
[1] P. Banerjee. S. Shkodrani. P. Moulon, S. Hampali, S. Han, F. Zhang, L. Zhang, J. Fountain, E. Miller. S. Basol, R. Newcombe, R. Wang, J. Engel, and T. Hodan. HOT3D: Hand and object tracking in 3D from egocentric multi-view videos. CVPR, 2025.
[2] R. A. Bartholomew, H. Zhou, M. Boreel, K. Suresh, S. Gupta, M. B. Mitchell, C. Hong. S. E. Lee, T. R. Smith, J. P. Guenette, al. Surgical navigation the anterior skull base using 3-dimensional endoscopy and surface reconstruction. JAMA Orolaryggooogy-Henn & Neck Surgery, 150(4):3183366, 2024.
[3] M. Birlo, R. Caramalau, P. J. Edwards, Dromey, M. J. Clarkson, D. Stoyanov, al. Hup-3d: 3d multi-view synthetic dataset for assisted-eoocentric hand-ultrasound pose estimation. arXiv preprint arXiv:2407.092/6 2024.
[4] H. Borgli, Thambawita,
Smedsrud, S. Hicks. Jha, S. L. Eskeland, R. Randel, K. Pogorelov, M. Lux, D. T. D. Nguyen, et al. vaperk,asir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific data, 7(1):283, 2020.
[5] S. Brahmbhatt, C. Tang, C. D. Twigg, C. C. Kemp, and J. Hays. Contactpose: A dataset of grasps with object contact and hand pose. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XIII 16, pages 361-378. Springer, 2020.
[6] Z. Cao, I. Radosavovic, A. Kanazawa, and J. Malik. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 12417-12426, 2021.
[7] Y.-W. Chao, W. Yang, Y. Xiang, P. Molchanov, A. Handa, J. Tremblay, Y. S. Narang, K. Van Wyk, U. Iqbal, S. Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9044 9053, 2021.
[8] P. Chen, W. Li, N. Gunderson, J. Ruthberg, R. Bly, Z. Sun, W. M. Abuzcid, and E. J. Seibel. Endoperfect: A hybrid nerf-sterco vision approach pioneering monocular depth estimation and 3d reconstruction in endoscopy, 2025.
[9] Y. Chen, 7. Tu, D. Kang, R. Chen, I.. Bao, 7. Zhang, and J. Yuan. Joint hand-object 3d reconstruction from single image with cross-branch feature fusion. IEFF Transactions on Image Processing, 30:4008-4021, 2021.
[10] E. Corona, A. Pumarola, G. Alenya, F. Moreno-Noguer, and G. Rogez. Ganhand: Predicting human grasp affordances multi-object scenes. Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pages 0031-501,, 2020.
[11] C. Cotsoglou, Granieri, S. Bassetto, V. Bagnardi, Pugliese, G. L. Grazi, A. Gugliclmi, Ruzzenente, L. Aldrighetti, Ratti, et al. Dynamic surgical anatomy using 3d reconstruction technology complex hepato-biliary surgery with vascular involvement. results from an international multicentric survey. HPB, 26(1):83-90, 2024.
[12] B. Cui, M. Islam. L. Bai, and H. Ren. Surgical-dino: adapter learning of foundation models for depth estimation in endoscopic surgery. International Journal of Computer Assisted Radiology and Surgery. pages 1-8, 2024.
[13] H. Dong. A. Chharia, W. Gou, F. V. Carrasco, and F. De la Torre. Hamba: Single-view 3d hand reconstruc- tion with graph-guided bi bi-scanning mamba. arXiv preprint arXiv:2407.09646. 2024.
10

---

[14] E. Duran, M. Kocabas, V. Choutas, Z. Fan, and M. J. Black. Hmp: Hand motion priors for pose and shape estimation from video. In Proceedings the IEEE/CVF Winter Conference Applications of Computer Vision, pages 6353-6363, 2024.
[15] P. Entezami, L. E. Franzblau, and K. C. Chung. Mentorship in surgical training: a systematic review. Hand, 711:300-36, 2012.
[16] Z. Fan, O. Taheri, D. Tzionas, M. Kocabas, M. Kaufmann, M. J. Black, and O Hilliges. ARCTIC: A dataset for dexterous bimanual hand-ohject manipulation. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[17] R. Fujii, R. Hachiuma, H. Kajita, and H. Saito. Surgical tool detection in open surgery videos. Applied Sciences, 12(20):10473, 2022.
[18] G. Marcia-Hernando, S. Yuan, S. Back. and T.-K. Kim. First-person hand action benchmark with rgb-d videos and 3d hand pose annotations, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 409 409-419,77 2018.
[19] S. Geman, Statistical methods for tomographic image restoration. Bull. Internat. Statist. Inst., 52:5 21. 1987.
[20] G. Gkioxari, Girshick, P. Dollár, and K. He. Detecting and recognizing human-object intcractions. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 8359-83611 8867,, 2018
[21] F. D. Goodman, K. K. Patel, Y. Zhang. W. Locke, C. J. Kennedy, R. Mchrotra, S. Ren, M. Guan, O. Zohar, Downing, et al. Analyzing surgical technique diverse open surgical videos with multitask machine learning. JAMA surgery, 159(2):185-19)) 2024.
[22] P. Grady, C. Tang, D. Twigg, M. Vo, S. Brahmbhatt, and C. C. Kemp. Contactopt: Optimizing contact to improve grasps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1471-1488 2021.
[23] M. Grammatikopoulou, E. Flouty, A. Kadkhodamohanmadi, G. Quellec, A. Chow, J. Nchme, Luengo, and D. Stoyanov. Cadis: Cataract dataset for surgical rgb-image segmentation. Medical Image Analysis, 71:10205: 2021.
[24] H. Hamer, J. Gall, T. Weise, and I.. Van Gool. An object-dependent hand pose prior from sparse training data. In 2010 JEEE Computer Suciety Conference Computer Vision and Puttern Recognition, pages 671-678. IEEE, 2010.
[25] S. Hampali, M. Rad, M. Oberweger, and V. Lepetit. Honnotate: A method for 3d annotation of hand and object poses. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3196-3206, 2020.
[26] Y. Hasson, B. Tekin, F. Bogo, I. Laptev, M. Pollefeys, and C. Schmid. Leveraging photometric consisteney over time for sparsely supervised hand-object reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 571 571-580, 2020.
[27] Y. Hasson, G. Varol, D. Tzionas, Kalevatykh, M. J. Black, Laptev, and C. Schmid. Learning joint reconstruction hands and manipulated objects. In CVPR, 2019.
[28] Y. Hasson, G. Varol, D. Tzionas, I. Kalevatykh, M. J. Black, Laptev, and C. Schmid. Leamingg joint reconstruction of hands and manipulated objects. Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 11807-11816, 2019.
[29] Hayoz, C. Hahne, Kurmann, Allan, G. Beldi, Candinas, P. Márquez-Neila, and R. Sznitman. Online 3d reconstruction and dense tracking in endoscopic videos. In International Conference on Medical Image Computing and Computer-Assister Intervention, pages 444-554. Springer, 2024.
[30] K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[31] Hein, M. Seibold, F. Bogo, M. Farshad, M. Pollefeys, Fürnstahl, and N. Navab. Towards markerless surgical tool and hand pose estimation. International journaa of computer assisted radiology and surgery. 16:799-808 2021.
[32] M. Hu, L. Wang, S. Yan, D. Ma, Q. Ren, P. Xia. W. Feng. P. Duan, L. Ju, and Z. Ge. Nurvid: A large expert-level video database for nursing procedure activity understanding. Advances in Neural Information Processing Systems, 36:18146 481644 2023.

---

[33] M. Hu, P. Xia, L. Wang. S. Yan, F. Tang, Z. Xu. Y. Luo, K. Song, J. Leitner, X. Cheng, al. Ophnet: A large- scale video benchmark for ophthalmic surgical workflow understanding. arXiv preprint 4rXiv:2406.07471 2024.
[34] M. Hu, K. Yuan, Y. Shen, F. Tang, X. Xu, L. Zhou, W. Li. Y. Chen, Z. Xu, Z. Peng, et al. Ophelip: Hierarchical retrieval-augmmented learning for ophthalmic surgical video-language pretraining. arXiv preprint ar 4rXiv:2411.1522,, 2024.
[35] D. Huang, X. Ji. X. He, J. Sun. T. He. Q. Shuai, W. Ouyang, and X. Zhou. Reconstructing hand-held objects from monocular video. In SIGGRAPH Asia Conference Proceedings. 2022.
[36] K. Karunratanakul, J. Yang. Y. Zhang, M. J. Black, K. Muandet, and S. Tang. Grasping field: Learning implicit representations for human grasps. In 2020 International Conference on 3D Vision (3DV), pages 533-344. IEEE, 2020.
[37] S. Khalid, M. Goldenberg, T. Grantcharov, B. Taati, and F. Rudzicz. Evaluation of deep learning models for identifying surgical actions and measuring performance. JAMA network open, 3(3xx201664-e200 c201664, 2020.
[38] J. Kim, M.-G. Gwon, H. Park, H. Kwon, G.-M. Um, and W. Kim. Sampling is matter; Point-guided 3d human mesh reconstruction. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages
2023.
[39] S. V. Kotsis and K. C. Chung. Application of the "see one, do one, teach one" concept in surgical training. Plastic and reconstructive surgery. (31(5))1194-11)))) 1201, 2013.
[40] T. Kwon, Tekin, Stühmer. Bogo, and M. Pollefeys. H2o: Two hands manipulating objects first person interaction recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10138-10148 October 2021.
[41] T. Kwon, Tekin, Stühmer, Bogo, and M. Pollefeys. H2o: Two hands manipulating objects first person interaction recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10138- 10138-10148, 2021.
[42] V. Lepetit, Moreno-Noguer, and P. Fua. Ep n p: An accurate o (n) solution to the p n p problem. International journal of computer vision, 81:155- 166, 2009.
[43] C. Li, Y. Tong. Y. Long, W. Si, D. C. M. Yeung, Y.-K. Chan, and Q. Dou. Extended reality with hind-assisted guidance and console 3d overlay for robotic surgery remote mentoring. IEEE Robotics and Automation Letters, 2024.
[44] M. Li. L. An, H. Zhang. L. Wu, F. Chen, T. Yu, and Y. Liu. Interacting attention graph for single image two-hand reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2761-2770 2022.
[45] M. Li, H. Zhang, Y. Zhang. R. Shao, T. Yu, and Y. Liu. Hhmr: Holistic hand mesh recovery by enhancing the multimodal controllability of graph diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 645-654, 2024.
[46] D. Lin, Y. Zhang, M. Li, Y. Liu, W. Jing, Q Yan, Q Wang. and H. Zhang. 4dhands: Reconstructing interactive hands
with transformers. arXiv preprint arXiv:2405. 20330. 2024.
[47] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980 2988, 2017.
[48] Z. Lin, C. Ding, H. Yao, Z. Kuang, and S. Huang. Harmonious feature learning for interactive hand-object pose estirattion. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 12989-12 12998, 2023.
[49] D. Liu, Q. Li, T. Jiang. Y. Wang, R. Miao, F. Shan, and Z. Li. Towards unified surgical skill assessment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9522-9531, 2021.
[50] S. Liu, H. Jiang. J. Xu, S. Liu, and X. Wang. Semi-supervised 3d hand-object poses estimation with interactions in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14687-14697, 2021.
[51] X. Liu, A. Sinha, M. Ishii, G. D. Hager. A. Reiter, R. H. Taylor, and M. Unberath. Dense depth estimation in monocular endoscopy with self supervipee learning methods. IEEE Transactions on Medical Imaging. 39(5):1438 -4441447444, 2020.
12

---

[52] C, Lugaresi, J. Tang, H. Nash. C. McClanahan, E. Uboweja, M. Hays, F. Zhang, C.-L. Chang, M. G. Yong. J. Lee, et al. Mediapipe: A framework for building perception pipelines. arXiv preprint arXiv: 1XXx:1906.0817.. 2019.
[53] G. Manni, C. Lauretti, F. Prata, R. Papalia, I.. Zollo, and P. Soda. Bodyslam: A generalized monocular visual slam framework for surgical applications. arXiv preprint arXiv:2408. 030788 2024.
[54] R. Q. Mao, L. Lan, J. Kay, R. Lohre, O. R. Ayeni, D. P. Goel, et al. Immersive virtual reality for surgical training: systematic review. Journal of Surgical Research, 268:40-58, 2021.
[55] G. Moon. Bringing imputs shared domains for 3d interacting hands recovery in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17028 17037. 2023.
[56] G. Moon, S.-L. Yu, H. Wen, T. Shiratori. and M. Lee. Interhand2. 6m: A dataset and baseline for 3d interacting hand pose estimation from a single rgb image. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 548-564, Springer, 2020.
[57] C. I. Nwoye, T. Yu, C. Gonzalez, B. Seeliger, P. Mascagni, D. Mutter, J. Marescaux, and N. Padoy. Rendezvous: Attention mechanisms for the recognition of surgical action triplets in endoscopic videos. Medical Image Analysis, 78:102433, 2022.
[58] E. Özsoy, C. Pellegrini, T. Czempiel, F. Tristram, K. Yuan, D. Bani-Harouni, U. Eck, B. Busam, M. Keicher, and N. Navab. Mm-or: A large multimodal operating room dataset for semantic understanding of high- intensity surgical environments. arXiv preprint 25i::2533002579, 2025.
[59] Y. L.. Pang. C. Oh, and A. Cavallaro. Sparse multi-view hand-object reconstruction for unseen environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 803-810, 2024.
[60] A. Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01700 2019.
[61] G. Pavlakos, D. Shan, I. Radosavovic, A. Kanazawa, D. Fouhey, and J. Malik. Reconstructing hands in 3d with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9826-9836, 2024.
[62] T.-H. Pham, N. Kyriazis, A. A. Argyros, and A. Kheddar. Hand-ohject contact force estimation from markerless visual tracking. IEEE transactions on pattern analysis and machine intelligence, 40( (2)):28883- 2896, 2017.
[63] J. D. Pitcher, J. T. Wilson, T.-C. Tsao, S. D. Schwartz, and J.-P. Hubschman. Robotic eye surgery: past, present, and future. J Comput Sci Syst Biol, 3(1):337, 2012.
[64] A. Prakash, M. Chang, M. Jin, R. Tu, and S. Gupta. 3d reconstruction of objects in hands without real world 3d supervision. In European Conference on Computer Vision, pages 126-145. Springer, 2024.
[65] H. Qi. C. Zhao, M. Salzmann, and A. Mathis. Hoisdf: Constraining 3d hand-object pose estimation with global signed distance fields. In 2024 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 10392-10402. IEEE, 2024.
[66] I.. Qiu and H. Ren. Endoscope navigation with slam-based registration to computed tomography for transoral surgery. International Journal of Intelligent Robotics and Applications, 4(2):252-266 263, 2020.
[67] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. Girshick, P. Dollár, and C. Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 00714, 2024.
[68] P. Ren, C. Wen, X. Zheng, 7. Xue, H. Sun, Q. Qi, J. Wang, and J. Liao. Decoupled iterative refinement framework for interacting hands reconstruction from single rgb image. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 8014-8025, 2023.
[69] J. Romero, D. Tzionas, and M. Black. Embodied hands: Modeling and capturing hands and bodies together. ACM Transactions on Graphics, (Proc. SIGGRAPH Asiaj, 36(6), Nov. 2017.
[70] D. Shan, J. Geng, M. Shu, and D. F. Fouhey. Understanding human hands in contact at internet scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9869-9878, 2020.
13

---

[71] A. Spurr, U. Iqbal, P. Molchanov, O. Hilliges, and J. Kautz. Weakly supervised 3d hand pose estimation via biomechanical constraints. In European conference on computer vision, pages 211-228. Springer, 2020.
[72] S. Stevsic and O. Hilliges. Spatial attention improves iterative 6d object pose estimation. In 2020 international conference on 3D vision (3DV), pages 1070-1078, IEEE, 2020.
[73] T. Sugiyama, S. Lama, L. Gan, Y. Maddahi, K. Zarcinia, and G. R. Sutherland. Forces of tool-tissue interaction to assess surgical skill level. JAMA surgery, 153(3)2234-243 2018.
[74] Y. Sun, Q. Bao, W. Liu, Y. Fu, M. J. Black, and T. Mci. Monocular, one-stage, regression of multiple 3d people. In Proceedings of the IEEE/CVF international conference computer vision, pages 11179 11188, 2021.
[75] S. Tamai, History of microsurgery from the beginning until the end of the 1970s. Microsurgery. 14(1):6-13, 1993.
[76] B. Tekin, F. Bogo, and M. Pollefeys. H+ o: Unified egacentric recognition of 3d hand-object poses and interactions. In Proceedings of the IEEE/CVF conference computer vision and pattern recognation, pages
2019.
[77] Q. Tian, 7. Chen, H. Liao, X. Huang. I., Li, Ourselin, and H. Liu. Endoommi: Zero-shot cross-dataset depth estimation endoscopy by robust self-lea from noisy labels. arXiv preprint arXiv:2409.05442 2024.
[78] D. Tzionas, L. Ballan, A. Srikantha, P. Aponte, M. Pollefeys, and J. Gall. Capturing hands in action using discriminative salient points and physics simulation. International Journal of Computer Vision, 118:172-193, 2016.
[79] P. Vaid, Yeung, and A. Rau. Robust semi-supervised detection of hands in diverse open surgery environments. K. Deshpande, M. Fiterau, S. Joshi, 7. Lipton, R. Ranganath, I. Urteaga, and Yeung, editors, Proceedings of the Sth Machine Learning for Healthcare Conference, volume 219 of Proceedings of Machine Leurning Research, pages 736-753 PMLR, 11-12 Aug 2023.
[80] H. Wang, Y. Long, Y. Chen, H.-C. Yip. M. Scheppach, P. W.-Y. Chiu, Y. Yam, H. M.-L. Meng, and Q. Dou. Learning dissection trajectories from expert surgical videos via imitation learning with equivariant diffusion, Medical Image Analysis, 103:99 2025.
[81] R. Wang, S. Ktistakis, S. Zhang, M. Mcholdt, and Q. Lohmeyer. Pov-surgery: A dataset for egocentric hand and tool pose estimation during surgical activitics. In International Conference on Medical Image Computing and Computer-Assister Intervention, pages 440-450. Springer. 2023.
[82] J. Wu, G. Paylakos, G. Gkioxari, and J. Malik. Reconstructing hand-held objects in 3d. arXiv preprint arXiv:2404.0m 065007 2024.
[83] Y. Xu. J. Zhang, Q. Zhang, and D. Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in Neural Information Processing Systems, 35:38571 1335844, 2022.
[84] J. Yang. J. Li, G. Li, Z. Shen, H.-Y. Wu, Z. Fan, and H. Huang. MIphand: Real time multi-view 3d hand mesh reconstruction via mlp modeling. arXiv preprint arXiv:2406. 16137, 2024.
[85] L. Yang. J. Xu, L.. Zhong, X. Zhan, 7. Wang, K. Wu, and C. Lu. Poem: reconstructing hand in point embedded multi-view stereo. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 21108-21111111111 2117, 2023.
[86] Y. Yang, L. Guoliang, Q. Li, and R. Song, slam framework based spinal endoscopic localization method. Procedia Computer Science, 250:81-58 2024.
[87] 7. Yang, J. Pan, J. Dai, 7. Sun, and Y. Xiao. Self-supervised endoscopy depth estimation framework with clip-guidance segmentation, Biomedical Signal Processing and Control, 95: 106410, 2024.
[88] Y. Ye, A. Gupta, and S. Tulsiani. What's in your hands? 3d reconstruction of generic objects in hands. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3895-3905, 2022.
[89] Y. Ye, P. Hebbar, A. Gupta, and S. Tulsiani. Difffusion Diffusion-guided reconstruction of everyday hand-object interaction clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19717- 19728, 2023.
14

---

[90] Z. Yu, S. Huang, F. Chen, T. Breckon, and J. Wang. Acr: Attention collaboration-based regressor for arbitrary two-hand reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). June 2023.
[91] Z. Yu. S. Zafeiriou, and T. Birdal. Dyn-hamr: Recovering 4d interacting hand motion from dynamic camera, arXiv preprint arXiv:2412.1288 2024.
[92] S. Zakharov, Shugurov, and S. Ilie. Dpod: 6d pose object detector and refiner. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1941 1950, 2019.
[93] R. Zha, X. Cheng, H. Li, M. Harandi, and Z. Ge. Endosurf: Neural surface reconstruction of deformable tissues with stereo endoscope videos. International conference on medical image computing and computer-assisted intervention, pages 13-23. Springer, 2023.
[94] B. Zhang, Wang. X. Deng, Zhang, Tan, Ma, and Wang. Interacting two-hand 3d pose and shape reconstruction from single color image. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1 1354 100-111666 11363, 2021.
[95] J. Y. Zhang, S. Pepose. H. Joo, D. Ramanan, J. Malik, and A. Kanazawa. Perceiving 3d human-object spatial arrangements from a single image in the wild. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XII 16, pages 34 51. Springer, 2020.
[96] C. Zimmermann, D. Ceylan. J. Yang. B. Russell, M. Argus, and T. Brox, Freihand: A dataset for markerless capture of hand pose and shape from single rgb images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822, 2019.
15

---

A Related Work
Surgical 3D Perception. Beyond conventional tasks in surgical assistance such phase recogni-
tion (57,33 34), anatomical segmentation 4 (3 and instrument detection [34] 57 recent efforts
have increasingly focused 3D perception for open surgical environments, addressing challenges
11 29 8), depth estimation [1] 87 77 [5]], navigation [66 86 [33,, and skill assessment 49 21 A growing body of work leverages multi-view camera arrays, RGB-D sensors, and IMUs to reconstruct dense surgical scenes with both static anatomical structures and dynamic hand-tool interactions. Goodman et al. [21] proposed AVOS, large-scale annotated video dataset of open surgeries, and developed a multitask model to extract procedural signatures and quantify surgeon skill from real-world surgical videos. MM-OR (58) introduces a comprehensive multimodal operating room dataset featuring RGB-D, audio, speech transcripts, and robotic logs, annotated with panoptic segmentation and semantic scene graphs. While these systems demonstrate technical progress on specific tasks and in multimodal fusion capabilities, their clinical impact remains limited, with most systems designed primarily for passive action recognition or offline documentation. Few are optimized for real-time use or tailored to support specific training objectives in microsurgical procedures. In this work, we address this gap by developing a clinically-oriente 3D perception framework for ophthalmic microsurgery, integrating task task-specific data, model design, and system-level considerations for weal-world deployment.
in scene reconstruction 993
3D Hand Dataset, Recent 3D hand reconstruction datasets have driven progress in hand pose, object pose, and interaction reconstruction (see Tab. D General benchmarks such FreiHAND (96) and InterHand2.66 (56) provide multi-view real RGB images for reliable 3D pose recovery. Con- tactPose (5] further leverages RGB-D to capture detailed contact patterns. Banerjee al. present HOT3D M a large-scale egocentric, multi-view hand-object interaction (HOI) dataset comprising = 11.5 M synchronized frames captured with head- wead-mounted Aria and Quest 3 cameras across 19 participants manipulating 33 objects; each frame is paired with mocap-grade 3D hand and object poses. Medical datasets extend these ideas to surgery. Hein et al. [1 released a synthetic surgical RGB set with limited views and simple actions. POV-Surgery [8] and HUP-3D B enrich viewpoint count and interaction diversity, yet their synthetic origin still curbs realism. Ophthalmic surgery, notably cataract removal, follows fixed workflows but requires both hands to manipulate several tools confined field-for instance, holding an iris retractor while guiding capsulorhexis forceps. These dense, dual-hand motions strain current reconstruction methods. OphNet-3D addresses this gap with real RGB-D clinical dataset that records fine-grained, two-hand, multi-instrument interactions during live ophthalmic operations.
Monocular 3D Hand Mesh Reconstruction. Compared with multi-view approaches, monocular reconstruction is more practical in clinical theatres: single, non-contact RGB camera minimises equipment, preserves sterility, and avoids obstructing the surgeon's 's workspace. Research has pro- gressed from single-hand models [13
45
61
38
90 to two-hand systems [94|
46 68] and, most
recently, HOI reconstruction 82 64
89 Monocular HOI remains difficult because occlusion is severe and annotated data are scarce. Many studies therefore assume known instance-specific templates [18 24 25]; with the template place, object recovery reduces to 6D pose estima- tion and joint hand objjett pose inference. Joint reasoning is implemented via implicit feature fusion [9 20, 50 70 76), explicit geometric constraints such as contact or collision [5 10 22 95).
35
or physics-based consistency [62 78). To lift the template assumption, newer work directly predicts object either as explicit genus-O meshes [28] or via a joint hand-object implicit field [36].
Conditional reconstruction strategies further exploit hand joint cues refine object geometry [8]]
B
OphNet-3D Construction
B.1
Synchronized Recording Configuration
We capture using the Intel RealSense SDK and synchronize all cameras with the official software- trigger method. Because we simultaneously acquire and store high-resolution RGB and depth streams from eight cameras, the system's USB bandwidth and I/O performance are pushed to their limits, and sustained data throughput may exceed its capacity, causing occasional frame drops. Therefore, on the hardware side, we've configured a high-core-count host machine, enterprise- storage drives with high write speeds, and fiber-optic USB cables to ensure stable data transmission.
16

---

In fige-freueency image I/O scenarios, traditional storage formats such as JPEG and PNG despite their widespread compatibility and ease of visualization-inng innur compression artifacts and processing overhead that can significantly degrade overall system performance. To address these limitations and improve data throughput, we adopt a sinay--frrmat-base image encoding approach that directly serializes ev::Mat matrix data into raw binary files. Specifically, our method first writes essential matrix metadata (rows, columns, and ev::at type) as integers in the file header to ensure accurate reconstruction of the image's s dimensions and layout; it then appends the unaltered pixel byte stream to the file body without any compression or encoding conversion. During decoding, the file is read in the same order to restore the original ev::at structure exactly. In implementation, color frames are stored as 848 x 480 three-channel matrices of type CV SV_8UC3, while depth frames are 848 x 480 single-chanme matrices of 16-bit unsigned integers (UINT16), with each color- depth pair spatially aligned. For acquisition, the main thread initializes eight camera-capture subthreads; upon receiving capture command, image saving begins and the preview display is disabled conserve resources, whereas the preview re-enabled when capture is inactive.
Figure 5: Synchronized calibration of 8 cameras.
B.2 Synchronized Calibration
In our multi-camera calibration pipeline, each Intel RealSense D435's intrinsic parameters-innlinne focal lengths, principal point coordinates and lens distortion coefficients are retrieved at runtime via the official SDK and assembled into the 3x3 camera matrix K; for extrinsic calibration, a planar ArUco marker board of precisely known marker size and layout serves the world reference, and for each synchronized color capture a simple green-channel mask first isolates the board region, from which the 2D corner coordinates of each detected ArUco marker are paired with their exact 3D positions on the board plane. A non-lineat Perspective- -------PP-tti solver then computes the rigid-body rotation and translation that best align the 3D marker positions to their 2D observations, yiclding a 4x4 homogeneous transform from board (world) to camera coordinates. During the recording process, due to force majeure the platform moved three times. After each move, performed recalibration; Fig. 5 shows one such calibration example.
B.3 Phase Definition and Demonstration
Referring to the standard cataract surgical procedure, we divided the entire surgical process into 12 phases: (1) main incision creation, (2) viscoelastic injection, (3) paracentesis, (4) capsulorhexis, (5)
hydrodissection and hydrodelimeation, (6) pre-phaco, (7) nucleus sculpting, (8) nucleus cracking, (9)
17

---

nuclear rotation, (10) phacoenulsification, (1I) cortex removal, and (12) incision hydration. Detailed definitions of each phase and the instruments used are listed in Tab. from two different camera viewpoints.
Fig. 10 shows the 12 phases
Index
Phuse
Phuse Delinition
Instruments Used
procise ontyy point through the Use anterior chamber the eye.
toothed foroaps, kerstome
Visceelattc injection
Injection ophilalmie viscoelastic devices (OVDs astesior chamber viscoelastic
viscoclastic syrigge
facilisate subsequent surgical chamber deyti, prosect intracuular
3
jumsontto's
Crorion of small, selforaling silesper incivirs the limbas using 18 stsh black tolloww socess for socurd indruments such the dupper into the autzrior clandees.
157 usb hask.. tooteed forcups
capsulortexiis
Creating continoous curvilinear opening the anterior less capsule using capsslorhesis tonceps allow sate access to the lens nocleus for remoral.
itss repositur, capoulochexis forceps
5
hydmodioutcionn and by-
Injor belanced sill solition (RSS) using 10ml syring to seperate the lone sucloss from the curies and sppaiiic, allowing coeer mitaiin and remeval.
ml syrings
drvolcimaation
6
pce placo
Use a şhacoemaldicaiio handpiece pentls remove loose contical spisuclou manerial froe the anertior lens surface before maclets sculpting begass.
nucleus chopper, placomuliiitatio handpooce
2
micless scalpting
Heing a phasemalsiiiaaion handpicce carve growess into the lens mellns fa live mnlles divicin and ramoval.
nucleus chopper, phacoemulaiicaii handpocce
nucleus cracking
Using the placemabillaaaoo handgicee conhitation sucleus chopper meshunically split the grooved suctess into smaller fraguenns for casier phacoemdstiiatioo and removal.
nucleus chapper, placoemubiifaatio handpooce
muclear mistiio
Using
capualar Fey, emouring qprimal positoning coninuel placommulliiition
chopper to genely rober
choppor phacoemllsiiaaiio handpocce
10
pluscemalalication
Invovves placeaubsiiiatio
with ultrasonic break up nucleus chayper, placoemuliiicatioa handpiccs
and emulsity the aucleus fragmeuss while simuluneously aspirating material maintaning amerior chamber sabliny.
11
removal
Use iniiation
handgiece gendly
residual contiell
repositor, iriigation aspiration handpiece
the capsilez bag following phacoemulssiicatio of mucleus.
12
incision hydration
Using balanced will solution (RSS) through a 10ml syrings to swell the cumeall stmms at incision (both msin incision pumsentisin) edgos. scaling the surgical incivionss the and of crlames sunaey.
10 ml syringe
Table 4: Definition of each phase and the instruments used in each phase
B.4 Phase Localization Annotation
We performed boundary annotation on each video sequence as follows. First, because the microscope and hand-view recordings were acquired separate devices and thus lack intrinsic temporal synchronization, we aligned them manually. Immediately before each trial, a rigid printed marker bearing the legend "Start Recording" was displayed concurrently across all camera views; the frame in which this marker first appeared in each view served as the synchronization point. Next, an experienced ophthalmologist reviewed the temporally aligned microscope and hand-view videos to delineate the onset and offset of each surgical phase. To improve label purity, segments corrupted by visual mrise-such field of view-wers were excluded. A second ophthalmologist then independently verified and revised all boundary annotations. Finally, to ensure precise demarcation of phase transitions, we uniformly contracted each annotated interval by removing one second from both its start and end.
instrument exchanges during which the hand left the camera
B.5 Instrument Demonstration
During the procedures, 10 different surgical instruments were employed: (I) 15" stab blade, (2) keratome blade, (3) iris repositor. (4) nucleus chopper, (5) toothed forceps. (6) capsulorhexis forceps. (7) phacoemulsificatio viscoelastic syringe. F shoow photographs of the ten instruments used during surgery, and Fig. [6 presents the 3D model scan files. For the forceps, we scanned both the open and closed configurations. For the syringe, we scanned two states cattes-plunerr rod at its maximum and minimum extension and additionally scanned the plunger rod and the syringe body separately as individual components.
handpiece, (8) prrigation-aspiaaaii
handpiece, (9) 10 mL syringe, and (10)
C OphNet-3D Statistics
Tab. 5 reports, for each split of OphNet-3D, the number of video clips and the distribution of frame counts per surgical phase. Since some phase operations may involve third hand entering the scene, we remove those segments during annotation; this can truncate a complete phase into multiple shorter clips. Such clips are counted separately in our statistics but are linked via sequential index identifiers.
18

---

Figure 6: 10 different instruments. From left right they are: capsulorhexis forceps, viscoelastic syringe, toothed forceps, iris repositor, nucleus chopper, 15 stab blade, keratome blade, irrigation- aspiration handpiece, phacoemulsification handpiece, and 10 mL syringe.
No. of Clips
No. of Frames
Phase Label
train
val
train
main incision creation
32
8
312,224
21,600
48,960
382,784
viscoelastic injection
59
519,120
132,960
paracentesis
35
4
13
52
196,080
13,440
71,520
281,000
capsulorhexis
1,116,960
292,080
1,514,400
hydrodissection and hydrodelinsaaiion
30
8
42
321,864
21.120
79,920
422,904
pre-phaco
31
435,600
mucleus sculpting
30
8
367,920
20,400
106,080
494,400
cracking
29
127,200
nuclear rotation
12
5
18
51,120
2,640
31,440
85,200
phacoemull sification
556,080
46,320
237,600
cortex removal (I/A)
33
10
46
751,832
98.640
290.640
1,141,112
incision hydration
406,168
80,672
all
401
45
119
565
4,955,272
468,240
1,511,312
7,141,720
Table 5: Phase distribution across splits for clips and frames.
D More Implementation Details and Results
D.I Data processing
Hand Mesh Initialization. To initialize the per-frame hand motion for each view, we adopt three-stage process inspired by DynHaMR (91], incorporating 2D detection fusion, per-view tracking, and global fusion.
We first extract 2D hand keypoints by applying ViTPose 83 to each RGB frame. Keypoints below a confidence threshold e 0.5 are discarded. Cropped patches around detected hand regions are then reprocessed through ViTPose for local refinement. To address unreliable or missing joints, we additionally apply MediaPipe [52) and fuse the results: for each joint, we retain the ViTPose prediction if above threshold, and replace it with MediaPipe's output otherwise. If entire hands are undetected in some views, we infill the missing detections by copying hand motion from nearby frames with high visibility and smoothing their trajectory with temporal window. This approach
19

---

ensures full joint set J per frame for each view. To reduce noise from hallucinated detections or wong handedness, we adopt filtering strategy. For each frame, we keep only the bounding box with the highest loU (> 0.9) among overlapping detections and discard those that appear in fewer than 10 frames across the sequence. Additionally, we track bounding box continuity to detect erroneous handedness flips or duplicated hands - if bounding box IoU with the previous frame drops below 0.1, we mark the frame as invalid and exclude it from subsequent fitting. These invalid frames are later recovered via generative infilling.
Next, we estimate the 3D hand pose per view using the coarse-to to-fine regression pipeline of [6]] which returns MANO parameters , 8² T.). The 3D wrist translation T is obtained via depth sampling and back projjecco::
2(u Cx)
z(U 100-o) cv)
I
fx
fg
where (u, u) are 2D keypoints, and (fz: fu) are focal lengths. We choose the optimal 2 minimizing reprojection error to initialize depth.
Finally, per-view MANO parameters are transformed to world coordinates using known camera extrinsics (R,, ) and merged across views using a weighted average. View weights are derived from the per-frame visibility scores computed from 2D keypoint confidence. This results in a globally consistent, temporally smooth initialization of hand pose across all frames and views. The resulting motion serves as the input to our multi-stage RGB-D optimization pipeline Sec. 3.2 where temporal, geometric, and interaction constraints are jointly optimized.
D.2 Annotation pipeline
Implementation details. We implement the annotation pipelien with PyTorch [60] During the optimization of stage II and stage III (Sec. we use L--FGG algorithm with Ir = 1 and optimizing
the loss functions using below weights:
For stage II, we have: A2d = 0.001, Asmooth = 10, Ag = 0.04, A3 = 0.05.
For stage III, we have: X 200, X 2, 10, Xpen
10, Ag 0.05, Aja
1, Apalmm = 1, Ab1 =
To better model the hand plausibility, we propose to leverage a biomechanical constraints and an angle limitation constraint to objective function:
Lja
Σ [do,n(a), H)), Lbi =
Σ I(lIb//la; Umin Uyaax)
(11)
Gallm Σ I([lc]llz;cc enin maxx)
Σ I(| di dmin Prax),
(12)
Gaugle =110; ll2 + I(10] I1a; min ) + I( ((11))122 thii s
(13)
where j is the index of the hand joint. Lja is for joint angle priors that constrains the joint angle sequence af (af az)) by approximating the convex hull on (a/ a.)) plane with the point set H². I(-) is the interval loss that penalizes outliers. Lb represents the loss term for bone length penalizing the finger bone length b, that lie outside valid bone length range [b'min bl Similarly, further constrain the curvature |c/ 2 and angular distance ||d ll2 for the palm root bones, Cpalm by penalizing the outliers if the ranges [cmin cmax] and [diim d- Moreover, we constrain a specific subset of hand poses 6% (e.g. twist rotation of Distal Interphalangeal (DIP) joints) and penalize the outliers of the pre-defined range [Ob 0b
MANO Regularization. We regularize the predicted MANO parameters during optimization using prior on both pose and shape:
Lyara XgLo Ag
(14)
The pose regularization term Lo penalizes deviations from rest pose (assumed to be all-zero) using an & norm over the pose parameters:
T
Lo =
Σ RE(l.))
Σ
(15)
1=0
20

---

The shape prior Lg similarly penalizes the shape coefficients 3, encouraging plausible hand geometry:
L3 -3= Σ 1B"
(16)
be(lr)
These terms serve as soft constraints that prevent drift during optimization and help enforce physical realism.
Interaction Loss. To model physical plausibility and guide the relative spatial arrangement of the hand and tool, we incorporate interaction loss Linter comprising attraction loss LA and repulsion loss LR. following prior work [28). Specifically, the attraction term encourages contact between hand and object surfaces when interaction is expected, while the repulsion term penalizes interpenetration. We define the set of hand contact vertices Co by computing the proximity of each MANO vertex on the hand to the object mesh. For each frame, we mark as contact those hand vertices within mm of the object surface. From this set, we extract six regions of contact based on anatomical structure: five fingertips and the palm base, following 28 These six anchor regions provide soft guidance for maintaining realistic contact.
The attraction loss is computed from the set of anchor points the hand and their closest points on the object mesh Vobj:
L (Voby, Vhand) = Σ Laist (Ext(OO)), Vobj)),
(17)
where Cf: (Ob)) are the hand anchor vertices corresponding to region i, and Ediat is distance loss (e.g., LI) between those vertices and the nearest points on the object. We define the distance loss Cdist as the average Euclidean distance between a set of hand contact vertices and their nearest neighbors on the object mesh:
Edist (C, Vobj) = Σ
min ||v 1lv-,
(18)
|C7
HE Volu
VEC
where C' is the set of contact vertices on the hand and Vobj is the object mesh. To discourage unnatural interpenetration, we define repulsion loss LR between the hand vertices and the inside of the object:
LR (Vonj, Vhand)
Σ (in(vs) d(vi, Vobj),
(19)
where (-n is an indicator function marking vertices that lie inside the object, and d(-, Vobi) is the shortest distance to the object surface. The final interaction loss is a weighted combination:
Linter = An + (1 CA,
(20)
where AR 0.5 balances repulsion and attraction. Following (28). we empirically set AR 1.0
during early training to resolve interpenetration first, then reduce to allow attraction.
This loss encourages anatomically plausible contact while suppressing mesh collisions, improving the realism of hand-tool interaction.
Signed Distance Field Loss Ludt. To penalize interpenetration between the hand mesh and the tool surface, we adopt signed distance field loss that queries the SDF defined over the tool volume. For each time step, we precompute a voxelized signed distance field (-) around the tool mesh O Then, the SDF loss is defined as:
Casl
Σ -()()))) VEVA
(21)
where V/8 is the hand mesh and dv(v) returns the signed distance of hand vertex to the tool surface - negative values indicate penetration. The max(0, -2 term ensures that only intrusions (i.e.,
where o < 0) are penalized, encouraging the hand to remain outside the tool surface.
Runtime Our network is agnostic to the initialization method and is not restricted to using 61. which affects the processing time. Therefore we conduct runtime experiment excluding the stage (hand and instance mask initialization time) and Pyrender offscreen rendering, which are not included in the optimization pipeline. On an NVIDIA A A100 GPU the optimization pipeline can take 15 minutes for a video with 1000 frames, The stage II (the initialization of instrument 6D pose and hand pose optimization) takes around 10.9 minutes to process due to the bottle neck in ICP processing and registration. Finally, the last stage of joint optimization only takes around 4.1 minutes.
21

---

D.3 Baselines and Experiments
Implementation details. We implement baseline methods based on PyTorch [60] We ResNet-50 [30] as the backbone network. All the imput image and segmentation maps are resized to 512x512 while keeping the same aspect ratio with 0 paddings, which are then used to extract the feature maps f E x(x+ with CoordConv []. We train our network using 1 A100 GPU with batchsize of 64. The size our backbone feature is 128 x 128 and the size of our pixel-aligned output maps is 64 64. We applied random scale, rotation, flip, and colour jitter augmentation during training.
Loss functions. We supervise our baseline models using weighted sum of losses that account for 2D keypoint projection, 3D reconstruction accuracy, silhouette alignment, segmentation, and parameter regression. The total loss is formulated as:
= Ssocccafooca + Appdd + Asa + XaaCa + LMANO + >aee + NobyProol (22)
Liocal is focal loss used supervise the predicted hand and object center heatmaps. The projection loss CpDd penalizes the re- re-projection error between the predicted 3D keypoints (via MANO) and the 2D annotations using a robust Geman-MeClure function. The 3D loss L3a measures the vertex-to-surface distance between the predicted mesh and the observed point cloud from multi- view fusion. The LpjaD and L3a are also computed for the instrument vertex and pre-defined 3D bounding box around it. The silhouette loss Cail compares the predicted hand and tool silhouette masks (from a differentiable renderer) against the ground-trun -tuth masks to enforce pixel-wise consistency. The MANO loss mann is composed of L2 losses over the predicted hand pose and shape parameters:
mann Xe 0 1- + A 1²
(23)
where and 3 denote pseudo ground-truth values from the annotation pipeline. The segmentation loss Laeg is pixel-wise cross-entropy loss over the hand and tool instance masks. The tool loss Looo supervises both the 6D pose and the 1D articulation parameters via parameter map regression and point cloud alignment. We use the following weights in all experiments: Aoocal 80, Apj2d 400,
Aza = 300, Aeil= = 50, Ag = 80, Ag = 10, Aseg 160.
Figure 7: Qualitative results on the hand pose estimation benchmark. Each image is an overlay from each camera view.
D.4 More Qualitative Results
In this section, we provide qualitative visualizations of our model predictions on the Hand- Instrument Interaction benchmark for different phases. Each row in Fig. @illustrates three temporally adjacent frames from representative video clips, capturing various surgical manipulation phases and interaction types. For each frame, we show: (1) the input RGB image, (2) the mesh overlay with predicted hand and instrument meshes, and two alternative views highlight the spatial relationship between hands and tools. These results demonstrate that our method generates consistent, physically plausible reconstructions across frames despite visual challenges such as occlusion, rapid tool motion, and complex hand articulation. The visual continuity across time confirms that our model not only produces accurate per-frame predictions but also maintains coherent temporal behavior, which essential for understanding fine-grained surgical actions.
22

---

E Discussion
Limitation. While our dataset and method offer new insights into dynamic 3D reconstruction in ophthalmic surgery, several limitations remain. First, the data collection was conducted based on single surgical procedure and within a single-center setting, potentially limiting the generalizability to datasets with varying surgical workflows, surgeon-spest -ppeciic operational habits, and illumination conditions; future work will expand to multi-center studies. Second, due to strong illumination from the surgical microscope, some instrument tips are overexposed in RGB views, affecting visibility and downstream pose estimation. Incorporating mocap mocap-synchronizer RGB capture or infrared cameras may help mitigate this issue. Third, we have not yet explored integrating the microscope view for joint reconstruction the ocular surface, hands, and instruments, which may enable more clinically meaningful applications.
23

---

Input
Mesh Overlay
Alt. View
Alt. View
Figure 8: Qualitative results on the hand-instrument interaction benchmark. Each row shows sample from the test set, with columns displaying: (I) input RGB image, (2) mesh overlay prediction, and (3)(4) for alternative view.
24

---

Input
Mesh Overlay
Alt. View
Alt. View
Figure 9: Qualitative results on the hand-instrument interaction benchmark. Each row shows sample from the test set, with columns displaying: (1) input RGB image, (2) mesh overlay prediction. and (3)(4) for alternative view.
25

---

main incision creation
vistoelastic injection
paracentesls
capsularhexis
bydrodissection and bydrodelineation
pre-phaco
nucleus sculpting
nucleus cracking
nuclear rolation
phacoenmlsification
coctex removal (I/A)
incision hydration
26

---

main incision creation
viscoelastic injection
paracentesis
capsulathexis
bydrodissection and hydrodelineation
pre-phacu
nucleus sculpting
nucleus cracking
nuclear rotation
phacoemmisifiiation
cortex remaval (I(A)
Incision hydration
Figure 10: 12 phases from different views.
27

---

28

---

Figure 11: Instance mask examples for phacoemulsification and viscoclastic injection.
29

---

keratome blade
15c stah blade
nucleus chopper
phacoemulsification handpicce
iris repositor
irrigation-aspiration handpiece.
toothed forceps
capsulorhexis forceps
30

---

10 mL syringe Figure 16: Scanned model files for 10 types of instruments.
discellaste syringe
31

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, clearly outline our contributions data collection, data annotation, and methodological innovation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Sec. 5 and we analyze the current limitations and their corresponding solutions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
32

---

Answer: [Yes]
Justification: For the data annotation pipeline and methodology, we provide clear mathemat- ical proofs and citations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We supplement each experiment with detailed model descriptions and ex- perimental parameter settings to ensure reproducibility. and all datasets and code will be
open-sourced
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
33

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Once the paper is accepted, we will open-source all data, code, and model weights.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
mublic//gudess/Cdeettbmisss www.aaammmiioooooooooooooooeeiie for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //ipsscc/puut
guides ProodeSubmissiooool....o
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We explain the data splits and some training and validation details in the main text, and provide more detailed experimental setup information in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We employ a fair evaluation protocol and metrics, and discuss the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
34

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details of the model training platform and training time in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
This study adheres to the NeurIPS ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
This work has the potential to substantially benefit the field of ophthalmic surgery by enabling accurate, real-time 3D understanding of complex hand-instrument interactions. Our dataset and methods can facilitate objective skill assessment, personalized surgical training, and intraoperative guidance, potentially improving surgical outcomes and reducing the need for extensive expert supervision.
Justification:
35

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our data and code will be hosted on major open-source platforms such as Hugging Face and GitHub.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
This study does not involve any data or code that require prior permission.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
36

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
We provide visualizations for some of the datasets and ensure they are anonymized.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
In the data collection section, we describe the participants' basic information.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
37

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy http:e///eerrrpooccc///oeeeerere for what should or should not be described.
38


Reviews:
[{"review_id": "yF4BI3NGis", "reviewer": "Reviewer_MyX8", "summary": "This work aims to facilitate dynamic 3D reconstruction of hands and surgical instruments in ophthalmic surgery. The authors (i) introduce an automatic annotation pipeline, (ii) construct a large-scale dataset using 8 RGB-D cameras, (iii) establish two benchmarks, and (iv) present two baseline methods. The main contribution of this paper is the OphNet-3D dataset, which contains 7.1M frames featuring 40 surgeons, 12 surgical phases, and 10 surgical instruments. Based on this dataset, the authors introduce two benchmarks and baseline methods. The proposed methods achieve state-of-the-art results for hand reconstruction and hand-instrument joint reconstruction.", "questions": "(i) How effective are the losses L_inter and L_sdf in addressing interpenetration? An ablation study on these losses would be valuable, given that hand-instrument interaction is the central focus of this work.\n\n(ii) The proposed OH-Net is trained with monocular training images from all 8 camera views. Is the same training protocol applied to the compared methods? Clarification is important for fair benchmarking.\n\n(iii) Can the authors provide a comparison of inference speeds between their baselines and other methods? Since the baselines rely on multiple large pre-trained models, they may be much slower than existing approaches.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "< Strengths >\n\nThis paper has the following strengths:\n\n(i) The OphNet-3D dataset has a significant contribution to the research field. It is carefully constructed with 8 synchronized RGB-D cameras using the proposed annotation pipeline. As shown in Table 1 and Figure 1, the dataset is substantially larger and more diverse than existing datasets. It includes detailed annotations such as MANO hand meshes and 6-DoF instrument poses. \n\n- As described in NeurIPS Paper Checklist, the authors will open-source all data, code, and model weights. This contribution can facilitate kinematic analysis of critical operational details (e.g., surgeon hand movements, surgical tool usage).\n\n(ii) The authors establish two benchmarks and introduce baseline methods for bimanual hand motion estimation and two-hand-instrument interactions. They also evaluate existing methods on the proposed OphNet-3D dataset, which supports the dataset’s utility as a benchmarking tool. \n\n(iii) This paper is interesting and well-organized. The techniques used for dataset construction and baseline methods are generally appropriate. \n\n---\n\n< Weaknesses >\n\nThis paper has the following weaknesses:\n\n(i) Despite the dedicated efforts for instrument motion annotation (Sec. 3.2), the proposed baseline struggles to capture fine-grained instrument articulation, as shown in the supplementary material. This limitation is understandable for now, given the challenges posed by the fine-grained motion scale in ophthalmic surgery.\n\n(ii) This paper contains several incomplete or unclear sentences, which may hinder readability and clarity:\n\n- L228-230: \"Based on the data split, bimanual hand pose estimation and (2) hand-instrument interactions.\"\n\n- L257-258: \"Mean Per Joint Positional Error between each hand-instrument interaction pair (MRRTEh,o)\"\n\n- L367: \"on all metrics ()\"\n\n- L840: \"CoordConv []\"\n\n- Equation (4) defines \"2d\", but equations (9), (10), and (22) use \"pj2d\".\n\n(iii) This paper lacks ablation studies to analyze the contribution of individual components in the proposed method, limiting assessment of design choices and their effectiveness.", "was_revised": true, "final_justification": "Most concerns were resolved during the discussion period. I believe this paper makes a valuable contribution and is well-qualified for acceptance. Therefore, I maintain my original rating of `5: Accept`.", "rebuttal": "We thank the reviewer for their thoughtful and constructive feedback. We are pleased that they recognize the impact of our proposed dataset and its significant contribution to the research community and applications such as the analysis of critical operational workflows. We also appreciate the reviewer’s constructive assessment of our benchmark system design, which they found appropriate and supportive of the dataset’s utility as a benchmarking instrument. We are encouraged that the reviewer has a clear understanding of the goals and contributions of our work.\n\n### **W1 - Challenges in fine-grained motion estimation:**\nWe agree with the reviewer that, as the pioneer in the field to recover 3D bimanual instrument interactions in the challenging context of ophthalmic surgery, capturing fine-grained instrument articulation, particularly during delicate procedures (e.g., *capsulorhexis*) involving subtle motions and instruments such as *toothed forceps* and *capsulorhexis forceps*, under extreme lighting & occlusion conditions may exhibit slightly reduced alignment accuracy compared to other phases. We would especially polish the dataset and keep updating with the newest features.\n\n### **W2 - Minor corrections & typos:**\nWe thank the reviewer for bringing these up. We have now carefully corrected them and incorporated them into our final revision of the paper.\n\n### **W3 & Q1 - Ablation study:**\nWe absolutely agree with the reviewer that, due to the novelty of the task and our benchmark, a detailed ablation study could further benefit the community. This is the reason why we ablates the on the essential components of the baseline OH-Net in Table. 3 of the main paper. Moreover, we hereby also provide the ablation study for the loss terms used in the annotation pipeline including $L_{inter}$ and $L_{sdf}$, by analysing the **Acceleration, Jerk** for hands following Dyn-HaMR [1] and Penetration rate between hand and instrument:\n| Terms | $L_{sil}$ | $L_{sil} + L_{2D}$ | $L_{sil} + L_{2D} + L_{3D}$ | $L_{sil} + L_{2D} + L_{3D} + L_{inter}$ | $L_{sil} + L_{2D} + L_{3D} + L_{inter} + L_{sdf}$ |\n|:------|:---------:|:------------------:|:---------------------------:|:----------------------------------------:|:------------------------------------------------:|\n| Acc ($m/s^{2}$) ↓ | 1.89 (± 0.08) | 1.92 (± 0.09) | 1.29 (± 0.13) | 1.47 (± 0.10) | 1.45 (± 0.11) |\n| Pen ($cm^3$) ↓ | 4.46 (± 0.15) | 4.12 (± 0.11) | 3.45 (± 0.08) | 2.98 (± 0.13) | 1.39 (± 0.09) |\n| Jerk ($10m/s^{3}$) ↓ | 0.98 (± 0.07) | 1.02 (± 0.07) | 0.72 (± 0.05) | 0.78 (± 0.06) | 0.76 (± 0.05) |\n\nThe table shows the impact of different error terms in our joint loss function. Specifically, we found $L_{3D}$ can speed up the convergence and increase the stability. Among them, $L_{inter}$ and $L_{sdf}$ can help with better interaction modelling with less inter-penetration. Other regularization terms (e.g. $L_{prior}$, $L_{bio}$) are included in all experiments to avoid unrealistic hand poses. The results are obtained by randomly selecting 100 clips on test split and repeating for 5 times.\n\n### **Q2 - Model training clarification:**\nFirst of all, we would like to kindly point out that only the proposed annotation pipeline relies on multiple large pre-trained models, while the baseline methods (e.g. OH-Net, H-Net) do not require any pre-training. In terms of the annotation pipeline, the methods used for initialization were using the officially released checkpoints without any additional pre-training. For the comparison with baseline OH-Net, all the methods in the experiments were trained and evaluated on our dataset with the same data split and protocols.\n\n### **Q3 - Runtime analysis:**\nFor our baseline method OH-Net, it takes about **0.1 ~ 0.3 seconds** on a monocular RGB image to predict the poses using a single NVIDIA A100 GPU whilst the SoTA method HaMeR takes around 1.4 seconds. As we mentioned that only our annotation pipeline needs a reasonable initialization from existing pose estimation models, the average runtime for it could vary depending on different initialization methods. In terms of the runtime of annotation pipeline, please refer to **Section D.2 of the Appendix**.\n\n\n[1] Z. Yu et al., “Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera,” CVPR ’25."}, {"review_id": "QbqKHB6Yqc", "reviewer": "Reviewer_feMU", "summary": "This paper introduces a new dataset for bi-manual hand and instrument motion in surgical settings. The authors design a multi-stage optimization pipeline to obtain ground-truth MANO hand parameters and 6-DoF object motion sequences. In addition, they propose H-Net and OH-Net for reconstructing hand and object motions. The paper provides extensive implementation and annotation details, making the dataset and methods reproducible and informative for future research.", "questions": "The paper claims the annotation pipeline is automatic (Line 8), but Line 103 mentions that manual corrections are required for per-frame instance masks. To what extent is the pipeline automated in practice? Could the authors clarify how much manual intervention is still involved?\n\nWhile the authors provide video examples to demonstrate the quality of the generated annotations, these individual cases are insufficient to assess the overall quality of the dataset. Evaluating the accuracy and consistency of large-scale automatically annotated data is a challenging but essential problem, particularly when the main contribution lies in the proposed annotation pipeline.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "*Strengths:**\n\n- The modeling of bi-manual hand and instrument motion is novel and relevant, especially in the context of surgical applications.\n- The dataset is valuable, containing multi-view RGB-D sequences with dense MANO hand meshes and 6-DoF instrument annotations.\n- The authors provide extensive implementation details and thorough documentation of the data collection and annotation pipeline.\n\n---\n\n**Weaknesses:**\n\n- While Line 8 claims the annotation pipeline is automatic, Line 103 reveals that it still relies on manual corrections for per-frame instance masks.\n- While a few visual examples are provided, the overall quality of the ground-truth annotations is not systematically evaluated. This is not a clear weakness, as large-scale annotation quality remains a challenging open problem, but it would benefit from further discussion.\n- No video results are shown for the reconstructed motions from the proposed OH-Net.", "was_revised": true, "final_justification": "We increased the final score given the rebuttal by the authors.", "rebuttal": "We appreciate the reviewer’s recognition of the contributions of our work. We are glad that the novelty of modeling bi-manual hand and instrument motion in surgical contexts was well received, and that the dataset were found valuable and significant.\n\n### **W1 & Q1 - About the annotation pipeline:**\nWe thank the reviewer for raising this point and apologize for any confusion. Our annotation pipeline does not rely on manual corrections during the optimization process after the initialization. Overall, manual correction was applied only to the **mask initialization in the first frame** of each video to improve hand-instrument tracking. All subsequent masks were automatically inferred by SAM2. We have now clarified this more explicitly in the revision.\n\n### **W2 & Q2 - More systematic evaluation on dataset:**\nWe appreciate the reviewer’s comment regarding the systematic evaluation of annotation quality. Since we are constructing the ground truth itself rather than evaluating against an existing dataset, it is inherently challenging to perform standard quantitative evaluation. Nevertheless, we have implemented multiple constraints to ensure annotation reliability, including thorough visual inspection across all sequences, cross-view 3D consistency checks, and sanity checks on temporal smoothness. Our method is based on the SotA initialisations and refines the hand instrument motion in the aspects of temporal consistency, contact and the hand pose accuracy in the following stages. To support our claim, we hereby provide a comparison on the test split between our data and SotA model (e.g. HaMeR [1] + instrument initialized as mentioned Section 3.2 and transformed into HaMeR's coordinate system) in terms of motion quality using Acceleration, Jerk and Penetration (between hand and instrument) metrics in the table below following Dyn-HaMR [2], as well as the ablation study provided in the response to [MyX8], which demonstrates the effectiveness of the annotation method and the quality. Note that although other standard error metrics (e.g. MPJPE, MPVPE, Acc Err) for them cannot be computed in this case as we are constructing the ground truth, we can still analyse the **Acceleration, Jerk** for hands and Penetration rate between hand and instrument.\n\n| Method | Acc ($m/s^{2}$) $\\downarrow$ | Pen ($cm^3$) $\\downarrow$ | Jerk ($10m/s^{3}$) $\\downarrow$ |\n|:-------|:-----------------------------:|:-------------------------:|:-------------------------------:|\n| HaMeR + Instrument (Init.) | 8.72 ($\\pm$ 0.21) | 6.79 ($\\pm$ 0.57) | 52.76 ($\\pm$ 8.23) |\n| OphNet-3D (Our dataset) | 1.45 ($\\pm$ 0.11) | 1.39 ($\\pm$ 0.09) | 0.76 ($\\pm$ 0.05) |\n\nWe can observe significant improvements over the SoTA (e.h. HaMeR), where much better Acceleration and Jerk proves stability of our annotation pipeline and the Penetration shows the effectiveness of our physical constraints and regularisation terms. The results are obtained by randomly selecting 100 clips on test split and repeating for 5 times.\n\n### **W3 & Q2 - More qualitative result:**\nWe fully agree with the reviewers that more visualizations can offer a better understanding of the dataset. In addition to the operational-frame visualizations in the appendix, our supplementary video presents, beginning at 33s, 8 operation clips along with their reconstruction visualizations (4 shorter and 4 longer clips). While we already included video samples of the predictions in our original submission, we regret that the NeurIPS rebuttal policy does not allow us to share additional materials at this stage. We acknowledge the value of presenting more examples and agree that doing so can help further demonstrate the effectiveness and limitations of our method. We now provided a comprehensive and diverse set of results in our final version. We invite the reviewer to follow our final release, where we have also share extended visualizations on the project page.\n\n**References:**\n\n[1] G. Pavlakos et al., “Reconstructing Hands in 3D with Transformers,” CVPR ’24.\n\n[2] Z. Yu et al., “Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera,” CVPR ’25."}, {"review_id": "mWRDInMXo3", "reviewer": "Reviewer_3Ziv", "summary": "This paper proposes the first large-scale, real-world, high-quality dataset for 3D reconstruction of hand–instrument interactions in ophthalmic surgery settings. Based on this dataset, it further presents two benchmarks for bimanual hand pose estimation and two-hand–two-tool pose estimation. The two baselines achieve comparable or better performance than SOTA methods in general settings.", "questions": "Please refer to the weaknesses in **Strengths And Weaknesses** part. For a medical-related paper, it is important to consider more medical-related evaluations and applications.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strength\n1. This is the first work that considers hand-object pose estimation in a surgery setting, which may benefit future researchers from both the general and medical areas.\n2. The multi-stage automatic annotation pipeline is reasonable and cost-effective.\n\n### Weakness\n1. No detailed application is discussed in this paper. What could the model be used for after training on this dataset, more than reconstructing hand and object motion?\n2. When it comes to hand motion estimation, the temporal consistency should be considered. This could be measured using the Acceleration and Jerk error, like in [1].\n3. For hand performance, any comparison with the most advanced model, HaMeR [2]? (both **without domain-specific training** and **exclusively trained on this dataset**)?\n4. Lastly but most importantly, considering the medical background of this paper, any metrics to measure surgical quality could be reported and analyzed in benchmarks, like *time and motion*, *flow of operation*,  *overall performance*, etc., in [3]?\n\n[1] Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera, CVPR 2025.\n[2] Reconstructing hands in 3D with transformers, CVPR 2024.\n[3] JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS): A Surgical Activity Dataset for Human Motion Modeling, MICCAI workshop 2014.", "was_revised": true, "final_justification": "I was on a trip in the previous days and missed the last chance to discuss with the authors, yet I trust they have addressed my proposed concerns. I also hope to see the dataset released to benefit the research community, so there are no strong reasons to reject this paper.", "rebuttal": "We appreciate that the reviewer acknowledges the significance of our work. We would also like to thank the reviewer for the suggestions on clinical applications, evaluation, and experiments - these have been really crucial for improving our work.\n\n### **W1 & Q1- Clinical applications:**\nWe sincerely thank the reviewer for their insightful comments regarding the practical applications of our work. As we emphasized in the introduction, accurate three-dimensional reconstruction provides substantially richer information compared to traditional two-dimensional imaging methods. Capturing the precise spatial relationships, depth information, and temporal dynamics of hand-instrument interactions forms the foundation for numerous downstream clinical applications. Our 3D reconstruction framework captures critical information inherently missing from 2D methods: precise positioning of instrument tips in 3D space, accurate hand joint poses under occlusion, authentic spatial relationships in bimanual interactions, and motion trajectories with depth perception. These 3D features are essential prerequisites for the following applications:\n\n(1) **Quantitative surgical skill assessment**: Unlike 2D video analysis, our 3D reconstruction enables measurement of precise motion metrics including 3D path length, economy of motion, millimeter-level tremor amplitude, and authentic bimanual coordination patterns. Additionally, we have included synchronized ophthalmic microscope videos along with surgical/practice experience duration data from 40 surgeons, facilitating more comprehensive and systematic analysis.\n\n(2) **Intelligent surgical training systems**: Detailed 3D motion trajectories support the development of AI-driven training systems with real-time feedback on instrument angles, force vectors, and spatial precision—elements that cannot be accurately assessed from 2D projections alone.\n\n(3) **Surgical workflow optimization**: Our phase-annotated 3D data can be used to analyze motion efficiency, workspace utilization, and ergonomic patterns, thereby providing evidence for improving surgical techniques and reducing operator fatigue.\n\n(4) **Next-generation robot-assisted surgery**: Millimeter-precision 3D hand-instrument coordination data provides crucial training data for developing semi-autonomous surgical robots capable of replicating expert-level minimally invasive surgical movements.\n\n(5) **Risk assessment and safety monitoring**: 3D reconstruction can detect potentially hazardous movements, proximity to critical structures, and predict complication risks based on deviations from optimal trajectories.\n\nWe recognize these application domains need comprehensive discussion to fully demonstrate our work's clinical impact. In the final version, we will expand the discussion section to include detailed use cases, implementation pathways, and clinical validation strategies, highlighting the transformative potential of 3D surgical scene understanding in advancing ophthalmic surgical practice.\n\n### **W2 & W3 – Temporal consistency and comparison with HaMeR:**\nWe thank the reviewer for these valuable suggestions, we fully agree that assessing temporal motion quality is important. We would like to answer this from two aspects: 1) data annotation pipeline and 2) baseline method, by comparing both methods to HaMeR [1].\n\n(1) **Annotation pipeline vs HaMeR.** Since we are constructing the ground truth itself, it is  inherently challenging to perform standard quantitative evaluation like MPJPE or acceleration error without a reference. Nevertheless, we have implemented multiple physical constraints and regularisation terms with temporal smooth to guarantee the annotation reliability. We now provide evaluations using **Acceleration (not error term)**, **Jerk** following Dyn-HaMR [2], and **Penetration** (between hands and instruments) metrics where each of them designed to quantify motion stability and interaction plausibility (also in response to MyX8, feMU). Although traditional error metrics like MPJPE or acceleration error cannot be computed without a reference, we can still observe the significant improvements over HaMeR, where it presents much lower Acceleration, Jerk and Penetration, indicating the high **temporal consistency** and the effectiveness of the annotation method. Additionally, the strong performance of downstream models trained on our annotations, and the superior values across the temporal metrics, offer indirect yet compelling evidence of annotation quality.\n\n| Method | Acc ($m/s^{2}$) $\\downarrow$ | Pen ($cm^3$) $\\downarrow$ | Jerk ($10m/s^{3}$) $\\downarrow$ |\n|:-------|:-----------------------------:|:-------------------------:|:-------------------------------:|\n| HaMeR + Instrument (Init.) | 8.72 ($\\pm$ 0.21) | 6.79 ($\\pm$ 0.57) | 52.76 ($\\pm$ 8.23) |\n| OphNet-3D (Our dataset) | 1.45 ($\\pm$ 0.11) | 1.39 ($\\pm$ 0.09) | 0.76 ($\\pm$ 0.05) |\n\n(2) **OH-Net vs HaMeR.** We appreciate the reviewer highlighting HaMeR. As noted above, HaMeR is used as initialization in our data annotation pipeline and is also evaluated separately, where we demonstrate significantly improved **temporal consistency**, stability, and plausibility.\n\nHowever, it is important to note that HaMeR is a **large-scale foundation model** trained on 2.7M samples using a huge ViT backbone, and is designed primarily for per-frame local hand pose estimation. In contrast, our baseline methods are specifically developed to reconstruct *joint hand–instrument* and support interaction-aware reasoning. \nAs HaMeR is a large-scale foundation model designed exclusively for hand pose estimation, without accounting for object interactions, we instead benchmark our approach against state-of-the-art hand-object (instrument) pose estimation baselines, as reported in Tables 2 and 3.\n\nMoreover, both HaMeR and our baseline models are per-frame methods and do not explicitly model temporal consistency. Nevertheless, to provide additional insight, we evaluated HaMeR using its official checkpoint on our dataset under the same protocol. Specifically, we randomly selected 100 clips from the test split and repeated the evaluation 5 times (totalling 500 sequences) to ensure consistency and robustness. Due to the scale of HaMeR and the substantial computational & time cost associated with retraining a foundation model, domain-specific retraining was not feasible during the limited time and the scope. We thank the reviewer for the advice and carefully add this into the final version.\n\nAs shown in the table below, our methods consistently achieve lower **Acceleration** and **Jerk**, indicating superior temporal stability. This further supports the value of our annotation pipeline and task-specific modeling.\n\n| Method | Acc Err ($m/s^{2}$) $\\downarrow$ | Jerk ($10m/s^{3}$) $\\downarrow$ |\n|:-------|:---------------------------------:|:-------------------------------:|\n| HaMeR (official checkpoint) | 5.24 ($\\pm$ 0.16) | 52.76 ($\\pm$ 7.23) |\n| H-Net (Ours) | 3.76 ($\\pm$ 0.07) | 35.34 ($\\pm$ 4.55) |\n| OH-Net (Ours) | 4.13 ($\\pm$ 0.09) | 39.45 ($\\pm$ 5.23) |\n\n### **W4 - No surgical quality metrics:**\nIn JIGSAWS, the authors conducted quantitative assessment of technical skills for three fundamental surgical tasks (Suturing, Knot-Tying, and Needle-Passing) using six specific dimensions (Respect for tissue, Suture/needle handling, Time and motion, Flow of operation, Quality of final product, and Overall performance), with each dimension rated on a 1-5 point Likert scale (where 1 point represents the poorest performance, 3 points indicates \"competent/adequate\" performance, and 5 points denotes the best performance), and the total score calculated as their sum. The annotation was performed by an experienced gynecological surgeon with extensive expertise in robotic and laparoscopic surgery through video review. It is worth noting that JIGSAWS' assessment method still relies on subjective expert scoring, where this direct rating approach based on human observation not only has limited interpretability but also inevitably suffers from raters' subjective biases. In contrast, our work provides a more fundamental and objective analytical framework. As emphasized in the introduction and W1, we establish a new technical foundation for the entire surgical skill assessment field through constructing a large-scale dataset and developing advanced 3D reconstruction techniques. Though space constraints prevent us from incorporating additional annotations or experiments, our core contributions in this work are pioneering. \n\nMore importantly, our framework represents a paradigm shift: from 2D video analysis relying on subjective scoring to objective data analysis based on precise 3D reconstruction. This transformation is particularly crucial in ultra-high precision ophthalmic surgery, where millimeter-level operational errors can lead to severe consequences. By integrating 3D hand joint positions, 3D instrument poses, depth information, and visualizations, we not only provide quantifiable, verifiable, and reproducible objective measurements, but more importantly, establish a technical platform that can eliminate human biases and support standardized assessment. This objectivity and precision, unattainable through traditional subjective scoring methods, enables: (1) surgeons to make more scientific decisions based on precise spatial data; (2) training systems to provide more accurate feedback; and (3) skill assessment to be based on objective metrics rather than subjective impressions. We will cite JIGSAWS and include a comparative analysis in the final manuscript. We believe this foundational work will drive a new generation of surgical education, training, and assessment systems, while offering essential tools and data resources for researchers in the field.\n\n**References:**\n\n[1] Reconstructing Hands in 3D with Transformers\n\n[2] Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera"}]

Author rebuttal:

Meta-review: (a) The paper proposed a large‑scale RGB‑D dataset for dynamic 3D reconstruction in ophthalmic microsurgery (41 sequences from 40 surgeons; ~7.1M frames) with fine‑grained labels (12 surgical phases, 10 instrument categories, dense MANO meshes, 6‑DoF instrument poses). It further proposes a multi‑stage automatic annotation pipeline and establishes two benchmarks: 1) bimanual hand pose estimation and 2)  hand–instrument interaction, together with baseline models H‑Net (dual‑hand) and OH‑Net (two‑hands–two‑instruments) that incorporate weak‑perspective spatial reasoning and collision‑aware, center‑based representations. Reported gains include >2 mm MPJPE for hands and up to 23% ADD‑S for instruments over prior baselines.

(b) + The dataset and pipeline fill a clear gap in surgical vision.
     + The benchmarks and baselines are thoughtfully constructed; and the rebuttal resolved the core technical and evaluation concerns raised by reviewers.

(c) - Need clearer architectural specifications (e.g., segmentation head).
     - Need more comprehensive ablation and clarification on the temporal stability.

(d)(e) The paper got positive rating from all reviewers after rebuttal {5,5,4}. The three reviews focused on dataset value, fairness/metrics, automation claims, and practicality. The rebuttal was thorough and empirical, and resolved most of concerns from reviewers. AC agreed with the acceptance decision, and suggest authors to should include their clarification in the final camera-ready version.
Decision: Accept (spotlight)