Title: PolyPose: Deformable 2D/3D Registration via Polyrigid Transformations
Venue: NeurIPS 2025
Accepted: yes
Keywords: deformable image registration

2D/3D registration

differentiable rendering

surgical computer vision
Abstract: Determining the 3D pose of a patient from a limited set of 2D X-ray images is a critical task in interventional settings. While preoperative volumetric imaging (e.g., CT and MRI) provides precise 3D localization and visualization of anatomical targets, these modalities cannot be acquired during procedures, where fast 2D imaging (X-ray) is used instead. To integrate volumetric guidance into intraoperative procedures, we present PolyPose, a simple and robust method for deformable 2D/3D registration. PolyPose parameterizes complex 3D deformation fields as a composition of rigid transforms, leveraging the biological constraint that individual bones do not bend in typical motion. Unlike existing methods that either assume no inter-joint movement or fail outright in this under-determined setting, our polyrigid formulation enforces anatomically plausible priors that respect the piecewise-rigid nature of human movement. This approach eliminates the need for expensive deformation regularizers that require patient- and procedure-specific hyperparameter optimization. Across extensive experiments on diverse datasets from orthopedic surgery and radiotherapy, we show that this strong inductive bias enables PolyPose to successfully align the patient's preoperative volume to as few as two X-rays, thereby providing crucial 3D guidance in challenging sparse-view and limited-angle settings where current registration methods fail. Additional visualizations, tutorials, and code are available at https://polypose.csail.mit.edu.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PolyPose, a novel method for estimating the 3D pose and shape of deformable anatomical structures from sparse 2D X-ray projections. The authors introduce a polyrigid transform model, which decomposes the deformable object into rigid parts and allows independent pose estimation for each part. The method leverages a set of 2D observations and a learned canonical template to localize 3D points via pose-parameterized correspondences. To enable differentiable supervision, the framework integrates a custom X-ray rendering pipeline and a visibility network. The method is validated on human spine and pelvis data from clinical X-ray images and outperforms previous methods in both pose estimation and 3D reconstruction quality.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- Innovative approach:Introducing multi-rigid body transformation into 2D/3D registration and Use anatomical priors to reduce the number of parameters for optimization degrees of freedom.\n- Methodology: Differentiable X-ray renderer and visibility network are valuable contributions. \n- Empirical results: Strong benchmarks and ablation studies. \n- Clinical relevance: Supports low-dose imaging workflows.\n\nWeaknesses:\n- Scalability: Generalization to other anatomies (especially soft tissue) is untested.\n- Manual part annotation required.\n- Limited dataset and anatomical diversity.\n- Real-time performance is not fully evaluated.\n\nQUESTIONS:\n1、Can the method generalize to anatomies without clear rigid substructures? \n2、How does performance change with fewer 2D views? \n3、Are the polyrigid parts interpretable in anatomical terms? 4. Could prior knowledge or statistical shape models help?\n4、Could prior knowledge or statistical shape models help?\n5、Anatomical plausibility of the weighting function: Is the assumption of uniform bone mass reasonable?\n\nLIMITATIONS:\nThe paper briefly acknowledges the potential limitation on generalization and requirement of predefined parts, but lacks a full discussion on modeling soft-tissue deformations, dependence on number of X-rays, or robustness to noise/occlusion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presented PolyPose for deformable 2D/3D registration. PolyPose parameterizes complex 3D deformation fields as a composition of rigid transforms, leveraging the biological constraint that individual bones do not bend in typical motion. The model enforces anatomically plausible priors that respect the piecewise rigid nature of human movement. PolyPose solves sparse-view deformable registration and aligns preoperative 3D images to intraoperative 2D X-rays using a locally rigid deformation model. It requires only X-rays and eliminates complex regularization. Extensive experiments on diverse datasets from orthopedic surgery and radiotherapy align the patient’s preoperative volumes to two X-rays.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThis paper presented a regularization-free framework for deformable 2D/3D registration that estimates polyrigid deformation fields using differentiable X-ray rendering.　The proposed anatomically motivated motion model is robust in sparse-view and limited-angle settings and produces smooth, invertible, and accurate deformation fields through construction.\n\nWeaknesses\nThe model relied on the assumption that anatomical structures are rigid. It would be helpful to discuss its generalization capacity when confronted with subtle bending or complex joint interactions. Moreover, the model relied on manual or semi-supervised segmentation of rigid bodies in the preoperative volumetric images and inaccurate deformation fields. The initial camera pose estimation relied on rigid registration of an anchor structure.\n\nQUESTIONS:\n1. The model relied on the assumption that anatomical structures are rigid, being limited to bones with subtle bending or complex joint interactions. Whether the proposed model can be extended to soft tissue deformation and non-skeletal contexts?\n2. The model required precise segmentation of rigid bodies in the preoperative volumetric images. The segmentation errors can cause inaccurate deformation fields. The manual and semi-automatic segmentation impose additional time complexity.\n3. The initial camera poses relied on rigid registration of an anchor structure. The obscured anchor structure can cause pose estimation errors.\n4. Tangent space averaging (Eq. 4) assumes transforms combine linearly in se(3). It is unclear whether such computation can handle large-scale transformations or deformations.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors tried to address the problem of deformable 2D/3D registration between a preoperative 3D scan (e.g. a volumetric CT) with 2D X-ray images acquired during the procedure (intra-operative). Instead of going full rigid or full deformable, they new idea is to leverage an anatomical prior that individual bones move as rigid bodies to simplify the deformation model. By combining these local rigid motions, the method produces a locally rigid warp field that is smooth and diffeomorphic by construction. A differentiable X-ray rendering procedure is used o compare the warped 3D volume against the 2D X-rays, allowing gradient-based optimization of the rigid body poses. The authors demonstrates that the proposed approach, PolyPose, can align a patient's preoperative 3D volume to as few as two 2D X-rays, even from limited angles, without additional regularization. Extensive experiments on diverse datasets from orthopedic surgery and radiotherapy show that PolyPose outperforms prior approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- A novel approach to apply a locally-rigid deformation model in the 2D/3D registration setting.\n- The model can work as few as 2 X-ray images, and with limited angle scenarios\n- The proposed method does not require expensive deformation regularization or hyperparameter tuning. This greatly improves the ease of use in real world applications.\n- Strong performance comparing to deep learning based or optimization-based methods. \n- The code is publicly available with all implementation details.\n\nWeakness\n- The proposed method requires accurate segmentation of rigid bones. A study of how the performance depends on segmentation accuracy would be helpful. While there are many segmentation tools nowadays, they still make notable errors from time to time in real world application, esp. on diseased situation or anomality.\n- As the authors mentioned, diffeomorphic deformation cannot be handled here, such as jaw opening. In fact, during neurosurgery, there might be minor deformation of the patient's head against the pre-operative scan, due to the head clamp. In other words, it is unclear how polypose can handle such situation when there exists even minor non-rigid deformation in the local rigid piece of bones.\n- The paper does not explicitly discuss runtime or computational efficiency, which could be important for intraoperative use.\n\nQUESTIONS:\n- The method relies on treating certain anatomical parts as rigid bodies. How are these rigid structures defined and obtained in practice? (Manually?)\n- As mentioned, it would be helpful to analyze the robustness of the proposed method against segmentation errors.\n- In the limited-angle scenarios, some bones might be barely visible or even completely occluded in the available X-ray views. How does the model handle a rigid component with little direct 2D supervision? And since there is no explicit regularization, it would be helpful to discuss this corner cases\n- The proposed method didn't optimize soft tissue motion, and such motion are carried along by blended warp field. The results show improved alignment of some soft organs without modeling them. However, one might wonder about scenarios with significant non-rigid soft tissues change? Is this still under scope of PolyPose?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present PolyPose, a simple and robust method for deformable 2D/3D registration. PolyPose parameterizes complex 3D deformation fields as a composition of rigid transforms. First by estimating the camera matrcis through an anchored sub-structure that is reliably visible across all views and rigidly aligned, the authors get the accurate extrinsic matrix estimation for every angled X-ray image. Then, after obtaining the correct camera matrics, the authors construct the polyrigid deformation field by leveraging comparable sized structures as separate warped coordinate systems, and weighted by respective mass and distance controlled weight field for each structure and 2D X-ray point combination to represent the global rigid transformation.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The novelty is clear and prominent. By simplifying a non-rigid patient and procedure specific transform into the polyrigid warps at all voxels of the 3D pre-volume which is decided by a linear mass and distance weighted combination of all rigid articulated bodies.\n2. Estimating the camera matrices through an anchor structure is very clever. The criteria for selecting such anchor structure is also important, such as (a) is reliably visible across all views in X-ray 2D sparse registration, as well as (b) could be rigidly aligned via its static property, like pelvis in Figure 3A.\n3. The formalization is clear and complete, although the methodology is complex, the description is clear and precise.\n4. The experiment resutls clearly show the superiority of proposed technique, including larger overlay in dice score and lower deformation error and topological error in tangent space, for most views of X-ray sparse registration, over multiple baselines on two large body data sets.\n\nWeakness:\n1. In equation (6), why only structure mass and distance are taken into consideration, how about their size and the distance to the mass center point?\n2. With equation (3) and rigidly aligned S_{anchor} which is available at all views, all angled camera matrices could be estimated, but how many sampled 2D points $p$ in X-ray images are needed for a correct evaluation for the X-ray projection operator P? And what is the confidence we should keep for such a discrete approximation of first-order approximation following Beer-Lambert law? Any alternative evaluation loop between camera matrics and acquisition deformation field needed?\n3. Equation (9) I can understand the SE(3) to se(3) transformation, but why it is isomorphic to R^6?\n4. For experiment, I am more interested in a general body data set where different articulated bodies or soft tissues are in different size and mass, and no rigid transform or global alignment guarantee for any body or structure.\n\nQUESTIONS:\nPlease see my weakness section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a locally-rigid registration framework (PolyPose) for CT to X-ray (2D/3D) registration. PolyPose is a two-stage method. In stage 1, PolyPose estimates the initial camera poses for each X-ray scan with differentiable X-ray rendering and gradient descent. Subsequently, PolyPose leverages segmented rigid structures (bones) to create a precomputed weight field and estimate the local rigid transform for each structure. The final deformation field is a weighted combination of the resulting local rigid transforms. The proposed method was evaluated using synthetic Head&Neck datasets (19 test subjects) and real clinical datasets (DeepFluoro, comprising 6 test subjects with CT-X-ray pairs). PolyPose achieves the best overall registration with no folding in the deformation field among 3D registration methods (uniGradICON, multiGradICON, FireANTs, anatomix) and 2D/3D registration methods (DiffPose, LiftReg and 2D3D-RegNet).\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\nA practical and elegant solution for CT to multi-X-ray registration. This paper proposes a promising solution for CT to multi-X-ray registration, a less-explored yet important topic in the context of image registration. By leveraging strong domain priors (bones are rigid, articulated motion is piecewise rigid) and differentiable X-ray rendering, PolyPose achieves robust and accurate CT to multi-X-ray registration without the need for a large training dataset. \n\nThorough and rigorous evaluation. The registration performance is comprehensively evaluated with both synthetic and real clinical images. Moreover, an ablation study on the deformation parameterization, weight function and rigid components is provided.\n\nSOTA results in 2D/3D registration. PolyPose achieves superior results over strong competitive methods such as DiffPose and FireANTs.\n\nStrong reproducibility. The source code is provided in the attachment and is well-written.\n\nGood clarity. This paper is well-written. The figures and mathematical formulations are clear and easy to follow.\n\n**Weakness**\n\nReliance on differentiable X-ray rendering and rigid structure segmentations: PolyPose comprises differentiable X-ray rendering and rigid structure segmentations, which hinders its potential for other applications such as MRI to X-Ray registration. Moreover, the differentiable X-ray rendering is tailored for CT to X-ray projection, and it is not applicable to other modalities, which further limits the application of PolyPose.\n\nLimited novelty. The success of PolyPose has been greatly fueled by differentiable X-ray rendering, rigid structure segmentations, and polyrigid registration, which are not new in the literature. Although an improvement over the weight field is introduced, the performance gain is rather marginal, as shown in Table 3. \n\nRobustness to exceptional cases. PolyPose may fail to handle soft tissue deformation and metal artefacts. First, as discussed in the limitation, since PolyPose assumes rigid bodies capture all motion, the local deformation of soft tissue, such as breathing and muscle contraction, may degrade the quality of the solution. Second, in real clinical scenarios, the metal artefact is common in post-treatment CT scans. Can PolyPose register images with serious metal artifacts? Can the differentiable X-ray projection accurately simulate the metal artefact?\n\nQUESTIONS:\nCan PolyPose register images with serious metal artefacts? Can the differentiable X-ray projection accurately simulate the metal artefact?\n\nGiven 1 CT and 1 X-ray scan, will there be any difference between PolyPose and DiffPose?\n\nIn what situations will the DiffPose fail but the PolyPose not?\n\nLIMITATIONS:\nPolyPose relies on rigid structure segmentation. The potential limitations and impact are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}