PolyPose: Deformable 2D/3D Registration via Polyrigid Transformations
Authors: Vivek Gopalakrishnan
Neel Dey
Polina Golland
Venue: NeurIPS 2025 poster ()
Keywords: deformable image registration
2D/3D registration
differentiable rendering
surgical computer vision
Primary area: machine_learning_for_sciences
TL;DR: We present PolyPose, a simple locally-linear method for deformably aligning a high-resolution preoperative 3D volume to as few as two intraoperative 2D X-rays without explicit regularizers or hyperparameter optimization.

Abstract: Determining the 3D pose of a patient from a limited set of 2D X-ray images is a critical task in interventional settings. While preoperative volumetric imaging (e.g., CT and MRI) provides precise 3D localization and visualization of anatomical targets, these modalities cannot be acquired during procedures, where fast 2D imaging (X-ray) is used instead. To integrate volumetric guidance into intraoperative procedures, we present PolyPose, a simple and robust method for deformable 2D/3D registration. PolyPose parameterizes complex 3D deformation fields as a composition of rigid transforms, leveraging the biological constraint that individual bones do not bend in typical motion. Unlike existing methods that either assume no inter-joint movement or fail outright in this under-determined setting, our polyrigid formulation enforces anatomically plausible priors that respect the piecewise-rigid nature of human movement. This approach eliminates the need for expensive deformation regularizers that require patient- and procedure-specific hyperparameter optimization. Across extensive experiments on diverse datasets from orthopedic surgery and radiotherapy, we show that this strong inductive bias enables PolyPose to successfully align the patient's preoperative volume to as few as two X-rays, thereby providing crucial 3D guidance in challenging sparse-view and limited-angle settings where current registration methods fail. Additional visualizations, tutorials, and code are available at https://polypose.csail.mit.edu.

PolyPose: Deformable 2D/3D Registration via Polyrigid Transformations
Vivek Gopalakrishnan MIT
Neel Dey MIT, MGH, and HMS harvard.
Polina Golland MIT
vivek@csail. mit.edu
polina@csail. mit. edu
Intraoperative Xray
Intraoperative Xrayy
Polyrigid deformation O
Before PolyPase
After PolyPose
Xray (2D)
CT (D)
Preoperative
(A) Misaligned 2D and images
(B) PalyPase: ricil 3D warp field
(C) Regularizer-free nonrigid registration
Figure 1: PolyPose Pose is locally-rigid framework for sparse-view deformable 2D/3D registration. (A) PolyPose can deformably align high-resolution preoperative 3D volume as few as two intraoperative 2D X-rays without the need of expensive regularizers or hyperparameter optimization. (B) To tackle this highly ill-posed problem, we estimate the poses (L) of rigid bodies in the volume and smoothly interpolate them in space to produce a topologically consistent locally-rigid warp. (C) Using the estimated warps, PolyPose provides 3D volumetric guidance procedures where only minimal supervision is available from intraoperative 2D X-rays.
Abstract
Determining the 3D pose of a patient from limited set of 2D X-ray images is critical task in interventional settings. While preoperative volumetric imaging (e.g., CT and MRI) provides precise 3D localization and visualization of anatomical targets, these modalities cannot be acquired during procedures, where fast 2D imaging (X-ray) is used instead. To integrate volumetric guidance into intraoperative procedures, we present PolyPose, simple and robust method for deformable 2D/3D registration. PolyPose parameterizes complex 3D deformation fields as a composition of rigid transforms, leveraging the biological constraint that individual bones do not bend in typical motion. Unlike existing methods that either assume no inter-joint movement or fail outright in this under-determined setting, our polyrigid formulation enforces anatomically plausible priors that respect the piecewise-rige rigid nature of human movement. This approach eliminates the need for expensive deformation regularizers that require patient- and procedure-specific hyperparameter optimization. Across extensive experiments on diverse datasets from orthopedic surgery and radiotherapy, we show that this strong inductive bias enables PolyPose to successfully align the patient's preoperative volume to as few two X-rays, thereby providing crucial 3D guidance challenging sparse-view and limited-angle settings where current registration methods fail. Additional visualizations, tutorials, and code are available at https:/polyposeeccall www.seeesssll....... mit edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Skuft
Tangent space interpolation of rigid transforms
2D
rigid motion
Skull
2
T6
Clavidle Humenus
R. Clavidle
Scapula (
R. Humerus
A. Scappulaa
T6
(C Construct locally-rigid warp
Segment rigid bodies from CT
Estimate structure-specifee specific rigid transforms
Figure 2: Illustration of polyrigid deformation fields. We visualize 2D slices of the rigid motion acting on every articulated structure. Linearly combining these transforms in the tangent space yields smooth and invertible deformation field, which color by the relative contribution from every structure. PolyPose enables the recovery of this 3D deformation field via differentiable rendering.
Introduction
Estimating the 3D position of anatomical structures from 2D X-ray images is critical task for clinical interventions that require millimeter-leve precision, such as image-guides surgery [1-5] or the deliv- ery of radiotherapy in cancer treatment [6-10]. The number of 2D X-rays available for 3D volumetric pose estimation is directly proportional to the radiation exposure to the patient and clinical team, as well as the time available for the procedure, thereby reducing the number of X-rays acquired [11, 12]. Furthermore, the geometry of X-ray scanners limits the angular range of acquisitions, introducing spa- tial ambiguities along the projection direction and challenges for 3D localization [13]. While patients undergoing surgery and radiotherapy typically have previously acquired 3D volumes, such as com- puted tomography (CT) scans, their use is confounded by their misalignment with the intraoperative 2D X-rays as patients move between acquisitions (see the misaligned outlines in Figure IA).
Several parameterizations of 2D/3D motion have been proposed to align these modalities. For example, rigid 2D/3D registration methods align global structure [14-17], but do not account for the soft tissue deformation or articulated inter-joint motion that occurs during procedures and creates localization challenges. Other work estimates point-wise displacement fields using either deep learn- ing [18-22] or optimization [23-25]. However, given the minimal supervision available for estimating 3D deformations in 2D sparse-view and limited- angle settings, deformable models require extensive application-speeee -ppecfffin regularization to generate anatomically faithful warps. thereby introducing new modeling decisions and hyperparameter tuning for every subject, procedure, and anatomical region.
Our approach is instead motivated by generic anatomical prior: bones are rigid bodies. We parame- terize deformable 2D/3D registration using a low-dimensional polyrigid model with limited degrees of freedom (Figure 2), where transformations are composed from individually estimated rototranslations of multiple articulated structures that are linearly combined in the tangent space se(3) [26]. This reduces the number of optimizable parameters from the order of voxels in the CT volume to the order of the number of rigid components. Furthermore, unlike other oow-dimensional deformation models (e.g.. splines [27] or linear bases [18]). polyrigid transforms have several desirable properties by construction, such as smoothness, invertibility, and coordinate frame invariance [26].
Our method, PolyPose, enables the estimation of highly accurate non-rigid deformations that are anatomically plausible and topologically consistent. We do this via differentiable X-ray rendering, providing piece wise 2D/3D registration targets from which to construct a polyrigid warp. Empirically, across diverse datasets, PolyPose is robust even for a small number of input views from limited angles. Furthermore, given its strong inductive priors, PolyPose does not require any regularization and has no tunable hyperparameters other than the step size of the optimizer. Our method outperforms both deep learning and optimization-base 2D/3D registration methods and enables the 3D localization of critical structures during medical interventions from intraoperative 2D images.
Contributions. To summarize, PolyPose contributes:
A regularization-fre framework for deformable 2D/3D registration that estimates polyrigid deformation fields using differentiable X-ray rendering.

---

A hyperparameter- free weighting function for linearly combining multiple rigid transforma- tions, providing out-of-the-box generalization to new surgical and therapeutic procedures.
An anatomically motivated motion model that robust in sparse-view and anitedd-ngge settings and produces smooth, invertible, and accurate deformation fields by construction.
2
Related Work
Rigid 2D/3D registration. Given a 2D X-ray and 3D CT volume, rigid registration methods estimate a global rigid transformation in SE(3) that optimally aligns the two images [28, 29]. While state-of- the-art methods can now determine the pose of rigid bodies with less than millimeter of error [15, 16] (which, in a different reference frame, is equivalent to estimating the extrinsic matrix of the image), they fail to describe the motion of volumes subject non-rigid deformable transformations.
Deformable 2D/3D registration. Non-rigid deformable 2D/3D registration is crucial to radiation oncology, where dense displacement field is needed align a preoperative planning CT volume with multiple intraoperative X-ray images [20, 23J. As deformably aligning a 3D volume to a set of sparse 21) X-rays is severely ill-posed, deformable 2D/3D methods rely on complex regularization schemes (e.g., diffusion [30], total variation [31], elastic penalties [32]), introducing numerous hyperparameters that must be carefully tuned for every procedure, subject, and anatomical region.
Deformable 3D/3D registration. Many methods exist to reconstruct 3D cone-beam computed tomog- raphy (CBCT) volumes from multiple 2D X-rays [33]. As such, one could reformulate multi-view 2D/3D registration as a 3D/3D registration task, an active research area, and use recent foundation models for multimodal 3D/3D registration [34-36] improved solvers for iterative deformable 3D/3D registration [37-39]. Unfortunately, the reconstructed CBCTs produced from sparse (< 10) X-rays have very low SNR and suffer severe streaking artifacts [40, 41], complicating their use as registration targets. In parallel, the broader vision literature has proposed several alternative represen- tations of 3DD deformation fields for large deformations. For instance, methods such Nerfies [32] and RAFT-3D [42] estimate dense SE(3) fields in which each spatial location is assigned an inde- pendent rigid transformation. While expressive, these dense deformation models are severely under- constrained in clinical settings characterized by sparse-view and limited-angle X-ray acquisitions.
Learning- based deformable 2D/3D registration. To avoid solving an expensive optimization problem for every new pair of 2D X-rays and 3D volume, numerous deep learning methods have been proposed for deformable 2D/3D registration. For example, methods like LiftReg [18] and 2D3D-RegNet [19] rely on convolutional architectures that directly regress parameterizations of 3D deformation fields from imaging. While some of these methods can be trained in self-supervised fashion, they require longitudinal datasets with multiple CT volumes for every patient and/or proce- dure, which is infeasible for many clinical and surgical settings.
Marker-based colpiecomponent tracking. Unlike the registration methods described above, some animal biomechanics studies use implanted fiducial markers to track and study the motion of bony structures in X-ray videos [43, 44]. However, this technique is impractical clinical settings due to the invasive nature of implanting markers, well its inability to track deformable soft tissue.
3 Methods
Let _ (IR*) define the of bounded and compact functions g R R and V E (RR) represent a 3D CT volume of a patient. Additionally, let = {In E (R*)) represent a set of N 2D X-ray images of the same patient at different time point (we assume all images in I are acquired simultancously). Specifically, assume the patient is in different positions for the acquisitions of V and I (c.g.. supine vs. standing).
The geometry underlying X-ray image formation can be modeled using a pinhole camera [45]. Let each image I, be associated with camera matrix II, K, [R, tx]. where K,, and [R, [x,] the intrinsic and extrinsic matrices, respectively. We model the relationship between V and I as
(I) where P(II,): T (R*) - L (R2) is the X-ray projection operator whose geometry is defined by the camera matrix II, and + R > 3R3 3D deformation field. Given V and I, our goal is to solve for the camera matrices [II,..... II and the deformation field Φ.
In -(II,) Vo
3

---

(A) Stage Estimate the pose (II) of each X-ray image
Operation How
Gradient Rlo
Rendered X-ray
Roal Kray
P(L1)
meder
Eq. (9)
P(DL
Extract structure
(3)
Sady
Prepperative CT(V)
(B) Stage Optimize structure speciffc poses ()) to estimate the polyrigid deformation field (1)
Projection.peeaae
Articulated
Image function
Camera matrix
$ Dog L log(T)) [ogt-1
Rendered K-ray
Real K-ryy
P(TT)) neraier
C
Eq.
P(UL
C
0.24
nedeer Eq. (3)
0.75
0.01
Precompute weight (6)
deformation warp
W(x)
Eq. (4)
Figure 3: Overview of PolyPose. (A) We estimate the camera pose II for each X-ray by registering the structure anchor across all input views (Section 3.2). (B) Using these camera matrices, we jointly optimize the poses of the rigid bodies in V by producing locally linear polyrigid warp field and maximizing the similarity of warped differentiably rendered and real X-rays (Section 3.3).
3.1 Preliminaries
Differentiable X-ray rendering. Given the camera matrix II, K, RR, t. 3x34 the location of the X-ray source in world coordinates is given by R [46, p. 158]. For pixel in I, with coordinates p E IR2, its location the X-ray detector plane is given by P S where is the X-ray machine's focal length (derived from K,, [46, 162]). f the pseudoinverse, and P E pp2 is p in bomogeneous coordinates. A construction of the intrinsic matrix K,, given in Appendix A.
The 3D ray back poojeereeeeted from p to the camera center can be parameterized as F(A) = S A(P S) for all X E [0, 1]. The negative log-intensity measured at p is given by the Beer- law [47]:
/ v(x)dx= S/'' (FF(I)) |P-S|] v(s X(P X (2)
I, (p)
JXEF
where V--) represents the linear attenuation coefficient (LAC) at every point in space, a physical property proportional to the density. The line integral in Eq. (2) defines the first-order continuous approximation of the X-ray projection operator P (II,,,, i.e., scattering, beam hardening, etc.
We implement Eq. (2) by modeling V with a discrete CT volume (i.e., a voxelgrid of LACs). This discrete line integral can be approximated with interpolating quadrature
(p)
PP S|| Σ
v Am (P
-S)] (Am+1 Am)
where Am+1 Am is the distance between adjacent samples F' and V|-] represents sampling operation (e.g., trilinear interpolation) on the discrete volume [48, 49]. Here, we rely on open-source implementations of the rendering equation (3) as series of vectorized tensor operations [50].
Parameterizing the deformation field. Let (S,...., .kt C represent a set of disjoint binary masks for the articulated rigid bodies within the volume (e.g., the bones of the skeleton). Each structure S₂ is associated with a corresponding rigid transformation Tç E SE(3) that represents the displacement of S& between the acquisitions of V and I. In the polyrigid framework, the deformation

---

field Φ is parameterized as a convex combination of the K rigid transforms represented in the tangent
space se(3) [26]. Specifically, the polyrigid deformation at any point IR is computed as
Σ kk=1 wx(() log Ts Ek=1 wx(x)
+ ( ....... Tx](x) T()),, where T(x) & exp
SE
is the locally-rigid transformation at x (represented as a 4 x matrix), & E ps is the representation of x R in homogeneous coordinates, (xxxx) is the weight of structure Sy x, and log(+) and exp(.) are the logarithm and exponential maps for (3),), respectively.
By fusing log-tra trg-trannformed versions of the pose for each structure, opposed simply averaging their associated displacements, the resulting polyrigid warp is diffeomorphic, anatomically constrained, and well-suited to our ill-posed setting. Eq. (4) can also be efficiently computed using closed forms for log(.) and exp(-) maps on (3(3), which are provided in Appendix B.
3.2 Estimating the Camera Matrices
Given preoperative 3D volume V and intraoperative 2D X-ray images Ii...., IN. we aim to estimate the camera matrices I..,.., IIN. While patients move non-rigidly between the acquisitions of V and I, there exists global rigid transform for individual articulated structure. Therefore, using a rigid 2D/3D registration framework (xvr [16]), we anchor pose representations by first rigidly aligning structure Sachor that is reliably visible across all views in I, such the pelvis in Figure 3A. Using Sanchor, estimate the extrinsic matrix for every X-ray image [R,, t.]. Finally, as X-ray imaging systems used in clinical practice are calibrated, the intrinsic parameters Kı,..., KN can easily be obtained from each image's metadata, yielding camera matrices II, K,, Rn 6x)
3.3 Constructing the Polyrigid Deformation Field
Constructing the weight field. Prior formulations of 3D/3D polyrigidreiissraioon registration [51] have proposed defining the weight each structure S₃ at any point x E R using the reciprocal distance function
ww (x) 1 edk (x)
where de is the minimum Euclidean distance from Sk, and is hyperparameter controlling the rate of decay of wP as x moves further away from S₁. However, we found that Eq. (5) produced inaccurate deformation fields for volumes containing articulated bodies with very different sizes (Ta-
ble 3). To our knowledge, Eq. (5) has largely only been used when the constituent substructures have comparable volumes, such as certain brain regions [51] or the carpal bones [52, 53].
Instead, loosely inspired by the influence of mass in gravitational attraction [54], we define the weight field for each structure
Wi wx(xx) (x) 1 (x)
m&
(6)
where my is the normalized mass of Sş relative to all structures. We estimate my using the volume of S₁ (i.e., assuming a constant density for all bones). This formulation eliminates challenging hyperparameter optimization while still producing topologically valid deformations (Table 3). An example of our proposed weight field visualized in Figure 3B (left).
Joint optimization. Given the camera matrices II, Πn estimated Section 3.2, we jointly optimize the pose for every rigid body by maximizing image similarity metric L (e.g., normalized cross correlation, mutual information, etc.) between the rendered and real X-ray images:
N
(T Tx) = argmaa
Σ L (In, (L..)(IL,) o V T..... Tkl). 1=1
TI..... T N
where constructed from T Ti...., TR via Eq. (4).
Efficient computation with vectorized forward model. Let X E Mr3 be the coordinates of every voxel in V where M is the number of voxels. For each structure Sg, we evaluate Eq. (6) to precompute w& (x) at every x E X. Concatenating the structure-specific weights, we construct the
5

---

discretized weight field W E RM with its rows normalized to sum to 1. Additionally, since the codomain of the logarithm map log: (E + se(3) homeomorphic to E (see Appendix B), we succinctly represent all structure-specific transformations Ti..... Tk with the matrix
log T
$
E RR
(8)
log Tr
Then, using batched matrix multiplication, we construct the polyrigid warp at voxel coordinates:
(Xx) exp(WT)XXXXX
where exp( W SE(3) represents set of M rigid transforms computed with vectorized imple- mentation the exponential map. The computational flow in PolyPose is illustrated in Figure 3B.
3.4 Implementation Details
To measure the similarity between rendered and real X-rays (C in Figure 3), we use a variant of the patch-wis wish normalized cross correlation loss [55] that computes the similarity between raw and Sobel-filtered images at multiple scales [15, 56]. For both camera and structure-specin pose estimation, we perform gradient
optimization on rigid transforms parameterized in the tangent space se(3). Across all experiments, we use the Adam optimizer [57] with step sizes 8rot 10-2 and Bxyz = 10 for the rotational and translational components of se(3), respectively. PolyPose and all baseline methods were trained (if applicable) and evaluated using single NVIDIA RTX A6000. All further implementation details are provided in Appendix C.
Experiments
4.1 Datasets and Experimental Setup
Neck We first perform experiments on a longitudinal dataset of CT scans of patients undergoing radiotherapy for head and neck squamous cell carcinoma [58] using 10/2/19 subject- wise training, validation, and testing split. Each patient had one CT volume from the pre-, peri-, and post-treatment periods, respectively [59]. To simulate deformable 2D/3D registration task, we generated a small set of synthetic X-ray images (2-9 images) in a 180 orbit from either the peri- or post-treatment CTs (fixed image) to be registered to the preoperative CT (moving image). To assess registration accuracy, we measure the 3D volume overlap between the warped labelmaps of rigid and soft tissue structures and their corresponding ground truth labelmaps in the peri- or post- post-treatment CT. The poses of soft tissue structures are not optimized, thereby serving to assess PolyPose's 's extrapolation outside rigid bodies.
DeepFluoro. To measure performance on real X-ray images, we use DeepFluoro, a cadaveric ortho- pedic surgery dataset of six preoperative CT volumes with associated intraoperative X-ray images (between 24-111 per subject) [60]. As typical image-guided interventions, the intraoperative X-ray images were acquired from a limited viewing angle (approximately 30") as unconventional oblique views are often not useful for human operators. Additionally, DeepFluoro provides manual segmentations of the femurs and pelvis in the real X-ray images. As such, for each subject, we estimate a deformation field using two X-rays capturing the left and right femurs, and quantitatively evaluate registration accuracy using 2D segmentation metrics computed on X-ray images not used to estimate the deformation field.
4.2 Baselines
We evaluate several 2D/3D and 3D/3D registration approaches as points of reference with implemen- tation details provided in Appendix D. We first compare against xvr [16], the current of-t--of-hhe-a art method for estimating single global rigid transformation. Next, we evaluate two convolutional deep learning methods for deformable 2D/3D registration: LiftReg [18] and 2D3D-RegNe [19]. LiftReg regresses the coefficients for a low-rank approximation of the deformation field whose basis obtained via PCA on a training set of ground truth 3D/3D warps, while 2D3D-RegNer directly estimates a dense translation field using VoxelMorph-style approach [61].

---

xer [16)
LittReg (18)
203D-RegNet (19)
mGI [35]
mGI+10
FireANTs (37)
anatornix [38)
PolyPase (ours)
(A) Warped CT
(8) Overiay
Dice
Dic
Dice
Dice
Dice 0.23
Dice 0.27
Dice 0.75
0.35
0.28
0.25
0.02
0.06
FDK(3D)
FD)
(G Target
K-nys
Keay
(D)
FreANTs
found Faldi
Foled 00715 odag2h 122
Folds 271%
Food DAIN OCoglT L.2
etogfit: o0
Figure 4: Qualitative evaluations of sparse-view 2D/3D registration Head&Neck. (A) Re- sulting warped CT volumes by different registration methods. (B) We visualize registration error by overlaying the warped CT (green) the ground truth CT (red). Baseline methods incur registration erross in the skull, spine, and surrounding soft tissue. (C) 2D/3D registration methods take stacks of X-ray images input, while 3D/3D) registration methods require reconstructed volume. (D) Visual- izations of the estimated deformation fields, superimposed on renderings of the warped CT volumes. PolyPose estimates smooth, localized deformations with minimal topological errors. Visualizations of the deformation fields for all other baselines are provided in Appendix E.
As 3D volumes be rapidly reconstructed from intraoperative 2D X-rays to serve registration targets, we also compare PolyPose to four 3D/3D registration methods [34, 35, 37, 38]. To match the speed requirements of intraoperative settings, we reconstruct 3D volumes using the FDK algorithm [62] implemented the ASTRA Toolbox [33]. Both uniGradICON (uGI) [34] and multiGradICON (mGI) [35], pair of foundation models for unimodal and multimodal image registration, contain variants with post-hoc iterative optimization (+IO). For each experiment, we report the two best- rest-performing variants from uGI, uGI+IO, mGI, and mGI+IO. FireANTs [37] and anatomix [38] are iterative solvers that provide state-of-the-art 3D/3D registration via improved optimization techniques and feature representations, respectively.
4.3 Results
Sparse-view registration. Figure 4 visualizes the warped CT volumes and deformation fields esti- mated from three input views distributed across 180 viewing angle range and Figure reports quan- titative evaluation metrics for the Head&Neck dataset. Of all evaluated methods, PolyPose estimates most accurate deformation fields across numbers of input X-rays available as registration tar- gets. PolyPose achieves the highest 3D Dice on both rigid structures and important soft tissue organs, even though the pose of these organs was not directly estimated during optimization. This is crucial as non-target organs
be avoided much possible in the delivery of radiotherapy. Of particular note, PolyPose outperforms both deep learning-basee 2D/3D methods [18, 19]. suggesting that train- ing the limited datasets available in interventional settings produces models that fail to generalize.
PolyPose also estimates deformation fields with minimal topological defects. Our construction from small number of rigid components yields interpretable deformation fields that are more anatomically plausible than baselines. For example, a subject with only minimal head motion, PolyPose recovers the exact underlying deformation (Figure 4D), whereas anatomix [38], the second-most accurate method, produces topologically-defee -defeccte and irregular warps as measured by the percentage of folds in the deformation, &Fllds and the standard deviation of volume changes, (log (J]) [63].
7

---

Rigid Bodles
Soft Tissue
Warp Regularity
Runtime
0.8
PolyPase (ours)
0m59s
anatomix [38]
1m05s
1.0
[37]
oflogDD
mGI I0 [35]
1m10s
0.4
mGI [35]
0m28s
0.6
RD3D-RegNet [19)
0m01s
0.2
LiftReg [18)
0m02s
0.0
0.2
[16)
0m06s
VVlaws (180")
Vlews (180")
(180")
Figure 5: Quantitative results of sparse-view 2D/3D registration on the Head&Neck dataset. We evaluated the accuracy of estimated deformation fields by computing the 3D Dice 21 rigid structures (L/R humerus, L/R scapula, L/R clavicles, thoracic and cervical vertebrae, and skull) and five soft tissue structures (thyroid, spinal cord, brain, esophagus. and trachea). PolyPose is the most accurate registration method and also exhibits the most regular deformable warps for almost all numbers of views. 2D/3D and 3D/3D registration methods shown with solid and dashed lines, respectively. Lastly, we report the average runtime for each method.
Limited-angle registration. Certain baselines are not applicable the DeepFluoro dataset. The deep learning methods LiftReg [18] and 2D3D-ReeNet [19] cannot be trained on this dataset since they require multiple CT's from each patient, while each subject in DeepFluoro only has a single volume. Therefore, we also evaluate regularized dense deformation model from radiotherapy, which optimizes a displacement vector for every voxel [23]. In Figure 3, we visualize a preoperative CT and two intraoperative X-rays spaced about 30 apart and the deformation field estimated by PolyPose.
Visualizations of the estimated deformation fields and warped CTs show that PolyPose produces interpretable warps, e.g., modeling the external rotation of the femurs (Figure 6A and B). In contrast, the dense parameterizations yield anatomically implausible and unintelligible deformations as their objective prioritizes memorizing the appearance of the training views. Additionally, dense deforma- tion models can only influence voxels on which they have direct pixel supervision (see the broken femurs in Figure 6B), whereas PolyPose extrapolates to unsupervised anatomy via piecewise-rigidity
To measure the accuracy of the estimated deformation fields, warp the input CTs, render synthetic X-rays from them, and compare the positions of bones in the rendered X-rays with their ground truth segmentations in the real X-rays (Figure 6D). Table reports the 2D Dice and 95th percentile Hausdorff Distance (HD95) for the pelvis, left femur, and right femur, as well as the %Folds in the estimated deformation fields. We used the pelvis as the anchor when estimating the camera poses for the X-ray images (Figure 3A). As such, nearly all baselines (evaluated using our camera matrices) exhibit high accuracy on the pelvis. However, for the femurs, PolyPose produces the highest accuracy, Additional visualizations of all baselines in Table are provided in Appendix E.2.
4.4 Ablations and Analyses
Choice of deformation parameterization. In Table 2, we compare our polyrigid formulation to per- voxel translations [23] and SE(3) transformations [32, 42], also optimized via differentiable rendering. Given minimal supervision, only our low-dimensional deformation model enables the localization of
Table 1:
Quantitative results limited-angle registration with the DeepFluoro dataset. Given only two X-ray images with 30 of separation, PolyPose recovers the most accurate 3D deformation field relative all baselines, while also having topological defects. We color the best and second-bee best methods and report all metrics as mean(sd).
Pelvis
Femur (L)
Femur (R)
Dice (f)
HD95 ())
Dice (*)
ND99 (L)
Dice (t)
IID95 (L)
% Folds ())
PolyPose (ours)
0.99(0.00)
10000.00)
0.98(0.01)
1.48(1.99)
0.98(0.01)
1.56(1.02)
0.0000.00) %
Dense R [23]
0.98(0.01)
3.60(5.47)
0.97(0.02)
3.29(2.62)
0.96(0.02)
3.4((2.78)
0.44(0.12)9) 12
Xyr [16]
0.99(0.00)
1.01(0.07)
0.96(0.02)
4.03(3.07)
0.94(0.02)
6.51(4.21)
0.00X0.00) %
FireANTs 37
0.99(0.00)
1.01(000))
0.96(0.02)
4.03(3.07)
0.93(0.02)
9.6314.26)
0.00K0.00)
anatomix
0.9510.01)
3.63(0.50)
0.93/0.02)
5.4(22777)
0.92(0.2)
6.8914.13)
2.0(1.211
multiGrradICON [35]
0.83(0.05)
16.37(6.75 75)
0.86(0.04)
8.69(4.84)
0.77(0.08)
18.18(3.54)
0.00(0.00)
uniGradICON [34]
0.66(0.07)
21.98(4.57)
0.50(0.12)
28.51(12.71)
0.83(0.04)
13.74(0.98)
0.00(0.00) %
8

---

Preaperative CT
Dense R [23)
Dense SE(3) [32]
PolyPase (ours)
(A) Deformation Field
(B) Warpad
(D) Test X-ryss (C) Train X ctys
NCC-0.99
NCG-055
NCC88.7
Figure 6: Qualitative evaluations of linited-angle 2D/3D registration on DeepFluoro. (A and B) PolyPose's 's anatomical priors recover realistic motion even without direct supervision. (C) Dense deformation parameterizations [23, 32] estimate warps that reproduce the appearance of the training X-rays, yielding near near-pertee pearect image similarity metrics (NCC x 0.99/1), (D) However, the dense deformations do not generalize to held-out views, as demonstrated by the misalignment of the ground truth (red) and estimated (white) segmentation labels in unseen images.
the misaligned femurs without topological defects. PolyPose has only O(K) optimizable parameters and is thus well suited for ill-posed settings, whereas the under-constrained dense representations have (M) parameters with K < M. Here, K 3 and M 398 x 197 398 10".
Choice of weight function. In Table 3, we compare different parameterizations of the weight field. Our hyperparameter-free weighting function in Eq. (6) outperforms the widely used formulation in Eq. (5). Note that the optimal performance for the left and right femurs achieved for vastly different hyperparameter values (e 10 vs.
10-3) when using Eq. (5). Thus, Eq. (5) has a large hyperparameter search space, requiring a different for every rigid body. contrast, our hyperparameter-five function in Eq. (6) uses the mass of each rigid body as effective heuristic.
Number of rigid components. PolyPose memory-efficient, capable of jointly optimizing the poses of 26 rigid bodies in a large CT scan on a single GPU with 48 GB of vRAM. However, this may be too computationally expensive for snoource-limited medical settings. We therefore perform an ablation on the Head&Neck dataset where we systematically increase the number of structures whose poses we optimize. Starting from rigid aliganigmennt (i.e., without PolyPose). we progressively add structures until reaching the full configuration used in Figure 5. Including the skull, cervical spine, and thoracic spine stabilizes the deformation fields estimated by PolyPose (Figure 7A). Adding further rigid bodies yields only marginal improvements, demonstrating that PolyPose is expressive even when constrained to a subset of rigid bodies in an anatomical region.
Table 2: Performance of different deformation parameterizations on DeepFluoro. PolyPose successfully recovers the position of the femurs, while the dense representations fail to do so.
Pelvis
Femur (L)
Femur (R)
Dice (1)
HD95 (1)
Dice (f) HD95 (1)
Dice (f) HD95 (1)
Folds (1)
PolyPose (ours)
0.99(0.00)
1.00(0.00)
0.98(0.01)
0.98(0.01) 1.48(1.09)
1.56(1.02)
0.00(0.00)4 %
Dense R [23]
0.98(0.01)
3.60(5.47)
0.97(0.02)
3.29(2.62)
0.96(0.02)
3.43(2.78)
0.44(0.1 (2)4%
Dense SE(3) [32]
0.93(0.02)
9.42(5.69)
0.90(0.02)
6.07(2.01)
0.88(0.03)
9.29(3.41)
44.08(00.00)$

---

Table 3: Performance of different weight functions on DeepFluoro. Our hyperparameter-fre
weighting function (6) outperforms the previously proposed Eq. (5), which achieves optimal perfor- mance for various anatomical structures at different hypermeter values.
Weight
Pelvis
Femur (L)
Femur (R)
function
Dice (1)
HD95 (L)
Dice (1) HD95 (1)
Dice (1) HD95 ())
% Folds (L)
Eq. (6)
0.99(0.00)
0.98(0.01) 1.000.000)
1,48(1.09)
0.98(0.01)
1.56(1.02)
0.00(0.00) %
Eq. (5)
10
(.99()
1.38(0.41)
0.93(0.02)
5.60(3.29)
0.96(0.01)
3,29(3.48)
0.03(0.01)%
Eq.
10-1
0.99(0.00)
1.58(0.41)
0.93(0.02)
5.31(3.27)
0.96(0.01)
3.53(3.55)
0.02(0.01) )%
Eq. (5)
10-2
0.99(0.00)
1.49(0.37)
0,94(0.01)
4.24(2.55)
0.95(0.01)
4.27(.75)
0.00(0.00) %
Eq. (5)
10-3
0.98(0.00)
1.6((0.36)
0.95(0.01)
2.87(1.18)
0.95(0.01)
4.34(3.71)
0.00(0.01)% %
Robustness to label corruption. PolyPose requires segmentations of the relevant rigid bodies in a CT scan to construct the weight field. While there exist numerous jemmi)aautomaed tools that make CT segmentation relatively straightforward task [64-66], they can make notable errors and miss fine details. We therefore analyze PolyPose's performance To simulate typical annotation errors, we systematically corrupt the ground truth segmentations in the DeepFluoro dataset with increasing radii of erosion (Figure 7B). A radius of mm corresponds to no erosion and is equivalent to the experimental setting in Table 1. We find that PolyPose is robust to extreme segmentation corruption, even outperforming the baselines in Table over range of erosion radii from 1 mm to 3 mm, both in terms of Dice and Hausdorff Distance.
function of segmentation accuracy.
Discussion
Limitations and future work. PolyPose's ability to generalize to extreme deformations in soft tissue far away from skeletal structures remains to be fully evaluated. Preliminary experiments in Appendix E.3 show that PolyPose successfully models free-breathing respiratory motion between maximum inhalation and maximum expiration acquisitions. However, PolyPose may fail to generalize to settings where there insufficient rigid bodies to constrain the deformation of soft tissues (e.g.. within the abdomen). Additionally, while our method produces diffcomorphisms by construction (typically highly desirable property), this does not cover every type of deformation. For example, separating a rigid body into two (e.g.. opening the jaw) cannot be represented by a diffeomorphism and thus cannot be modeled by PolyPose. We visualize such a failure case in Appendix F. This limitation could be mitigated by the incorporation of a kinematic chain into the rigid body parameterization.
Conclusion. Deformable 2D/3D registration holds immense promise in localizing critical organs from intraoperative images. However, the accuracy of previous methods fails to meet the standards for clinical deployment. We present PolyPose, optimization-base under-determined registration problem with a polyrigid field. Throughout extensive experiments on publicly available datasets from diverse clinical specialties, PolyPose estimated the most accurate and topologically correct warps in both sparse-view and limited-angle settings. addition to its high performance, PolyPose's lack of need for regularization and near-absence of hyperparameters make it generically applicable across a broad set of medical procedures.
method that solves this extremely
(A) Ablation on number of rigid components.
(B) Ablation on corruption given 3D segmentation.
Dice (1)
0.99
Rigid Bodies
Soft
Structures
Tissuev
HD95 (4)
Rigid pre-alignment [16]
0.51
0.49
0.97
4-
Dice
Skull
0.63
0.96
C-spine
0.64
0.76
T-spine
0.70
0.77
0.95
Pelvis
Humerus (L/R)
0.70
0.81
Femur (L)
Scapula (L/R)
0.71
0,80
Femur (R)
Clavicles (R)
0.74
0.81
6
0
Erosion radius (mm)
Erosion radius (mm)
Figure 7: PolyPose is robust to label restriction and corruption. (A) PolyPose remains expressive even when optimizing a limited subset of the rigid bodies an anatomical region. (B) PolyPose's
performance is relatively stable to 3 mm of label erosion (corresponding to a 40-60% volume reduction), beyond which registration accuracy degrades gracefully.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by NIH NIBIB 5T32EB001680) NIH NIBIB RO1EB033773, and the Chou Family Transformative Research Fund.
References
[1] Terry M Peters.
surgery: from X-rays to virtual reality. Computer methods in biomechanics and biomedical engineering, 4(1):27-57, 2001.
[2] Chris Schulz, Stephan Waldeck, and Uwe Max Mauer. Intraoperative image guidance in neurosurgery: development, current indications, and future trends. Radiology research and practice, 2012()):197344, 2012.
[3] R Phillips, WJ Viant, AMMA Mohsen, JG Griffiths, MA Bell, TJ Cain, KP Sherman, and MRK Karpinski. Image guided orthopaedic surgery design and analysis. Transactions of the Institute of Measurement and Control, 17(5):251-264, 1995.
[4] Stephen Rudin, Daniel R Bednarek, and Kenneth R Hoffmann. Endovascular image-guided interventions (EIGIs). Medical physics, 35(1):301-309 2008
[5] Vania Tacher, MingDe Lin, Pascal Desgranges, FeanFFaacciis Deux Thijs Grünhagen, Jean- Pierre Becquemin, Alain Luciani, Alain Rahmouni, and Hicham Kobeiter. Image guidance for endovascular repair of complex aortic aneurysms: comparison of two-dimensional and three-dimensional angiography and image fusion. Journal of Vascular and Interventional Radiology, 14911):1698-1700 2013.
[6] Catherine McBain, Ann M Henry, Jonathan Sykes, Ali Amer, Tom Marchant, Christopher M Moore, Julie Davies, Julia Stratford, Claire McCarthy, Bridget Porritt, et al. X-ray volumetric imaging in image-guided radiotherapy: the new standard in on-treatment imaging. International Journal of Radiation Oncology* Biology* Physics, 64(2):625-634, 2006.
[7] Laura A Dawson and David A Jaffray. Advances image-guided radiation therapy. Journal of clinical oncology, 25(8):9389944 -96966,,88899999 2007.
[8] Florian Sterzing. Rita Engenhart- Cabbilli,,, Michael Flentje, and Jürgen Debus. Image-guided guided radiotherapy: a new dimension in radiation oncology. Deutsches Aerzteblatt International, 108 (16):274, 2011.
[9] Mare I. Kessler. Image registration and data fusion in radiation therapy. The British Institute of Radiology, 79:$99-SI1 2006.
[10] Elizabeth Huynh, Ahmed Hosny, Christian Guthier, Danielle S Bitterman, Steven F Petit, Daphne A Kogan, Benjamin Kann, Hugo JWL Aerts, and Raymond Mak. Artificial intelligence in radiation oncology. Nature Reviews Clinical Oncology. 17(12):771-711, 2020.
[11] Vano, R Sanchez, JM Fernandez, Rosales, MA Garcia, Sotil, Hernandez, Carrera, J Ciudad, MM Soler, et al. Importance of dose settings the x-ray systems used for interven- tional radiology: national survey. Cardiovascular and interventional radiology, 32:121-126. 2009.
[12] Mahadevappa Mahesh, Armin J Ansari, and Fred A Mettler Patient exposure from radiologic and nuclear medicine procedures in the united states and worldwide: 2009-2018. Radiology. 307(1):e221233, 2022.
[13] Jürgen Frikel and Eric Todd Quinto. Characterization and reduction artifacts in limited angle tomography. Inverse Problems, 29(12); 1220)7, 2013.
[14] Robert Grupp, Rachel Hegeman, Ryan Murphy, Clayton Alexander, Yoshito Otake, Benjamin A McArthur, Mehran Armand, and Russell H Taylor. Pose estimation of periacetabular osteotomy fragments with intraoperative x-ray navigation. IEEE Transactions on Biomedical Engineering, 4-12):441-452 2019.

---

[15] Vivek Gopalakrishnan, Neel Dey, and Polina Golland. Intraoperative 2D/3D image registration via differentiable X-ray rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages I 11662-11677 2024.
[16] Vivek Gopalakrishnan, Neel Dey, David-Dimitris Chlorogiannis, Andrew Abumoussa, Anna M Larson, Darren B Orbach, Sarah Frisken, and Polina Golland. Rapid sprccfff neural networks for intraoperative X-ray to volume registration. arXiv preprint arXiv:2503. 2025.
[17] Andrew Abumoussa, Vivek Gopalakrishnan, Benjamin Succop, Michael Galgano, Sivakumar Jaikumar, Yueh Z Lee, and Deb A Bhowmick. Machine learning for automated and real- time winddimensional to three-dimensional registration of the spine using a single radiograph. Neurosurgical Focus, 54(6):E16, 2023.
[18] Lin Tian, Yuch Z Lee, Raúl San José Estépar, and Marc Niethammer. LiftReg: limited angle 2D/3D deformable registration. In International Conference on Medical Image Computing and Computer-Assister Intervention, pages 207-216. Springer, 2022.
[19] You Zhang. An unsupervised 2D-3D deformable registration network (2D3D-RegNet) for cone-beam CT estimation. Physics Medicine & Biology, 66(7):074001, 2021.
[20] Markus D Foote, Blake E Zimmerman, Amit Sawant, and Sarang C Joshi. Real-time 2D-3D deformable registration with deep learning and application to lung radiotherapy targeting. In Information Processing Medical Imaging: 26th International Conference, IPMI 2019, Hong Kong, China, June 2-7, 2019, Proceedings 26, pages 265-276. Springer. 2019.
[21] Megumi Nakao, Mitsuhiro Nakamura, and Tetsuya Matsuda. mmage-to-grape convolutional network for 2D/3D deformable model registration of low-contrast organs. IEEE Transactions on Medical Imaging, 41 41(12):3747-3761, 2022.
[22] François Lecomte, Pablo Alvarez, Stéphane Cotin, and Jean-Louis Dillenseger. Beyond respi- ratory models: physics-enhanced synthetic data generation method for 2D-3D deformable registration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2413-2421, 2024.
[23] Lin Tian, Connor Puett, Peirong Liu, Zhengyang Shen, Stephen R Aylward, Yueh Z. Lee, and Marc Niethammer. Fluid registration between lung CT and stationary chest tomosynthesis images. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part 11I 23, pages 307-317. Springer, 2020.
[24] Qingyu Zhao, Chen-Rui Chou, Gig Mageras, and Stephen Pizer. Local metric learning in 2D/3D deformable registration with application in the abdomen. IEEE transactions on medical imaging, 33(8):1592 1ee1-199116000 1600, 2014.
[25] Thomas SY Tang and Randy E Ellis. 2D/3D deformable registration using hybrid atlas. In International Conference on Medical Image Computing and Computer-Assisted Intervention. pages 223-230 Springer, 2005.
[26] Vincent Arsigny, Olivier Commowick, Nicholas Ayache, and Xavier Pennec. A fast and log- Euclidean polyaffine framework for locally linear registration. Journal of Mathematical Imaging and Vision, 33:222-233 2009.
[27] Weimin Yu, Moritz Tannast, and Guoyan Zheng. Non-rigid free-form 2D-3D registration using a B-spline-based statistical deformation model. Pattern recognition, 63:689-699, 2017.
[28] Primoz Markelj, Dejan Tomaževič, Bostjan Likar, and Franjo Pernuš. review of 3D/2DD registration methods for image-guided interventions. Medical image analysis, 16(3):642-661, 2012.
[29] Mathias Unberath, Cong Gao, Yicheng Hu, Max Judish, Russell H Taylor, Mehran Armand, and Robert Grupp. The impact of machine learning on 2D/3D registration for image-guided interventions: systematic review and perspective. Frontiers Roboties and AI, 8:716007, 2021.
12

---

[30] Danielle F Pace, Stephen R Aylward, and Mare Niethammer. A locally adaptive regularization based on anisotropic diffusion for deformable image registration of sliding organs. IEEE Transactions on Medical Imaging, 32(1 1):21 14-2114-2126 2013.
[31] Valery Vishnevskiy, Tobias Gass, Gabor Szekely, Christine Tanner, and Orcun Goksel. Isotropic total variation regularization of displacements in parametric image registration. IEEE Transac- lions on Medical Imaging, 36(2):385-395, 2016.
[32] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo BrutinnBrualll. Nerfies: Deformable neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[33] Wim Van Aarle, Willem Jan Palenstijn, Jeroen Cant, Eline Janssens, Folkert Bleichrodt, Andrei Dabravolski, Jan De Beenhouwer, K Joost Batenburg, and Jan Sijbers. Fast and flexible X-ray tomography using the ASTRA toolbox. Optics Express, 2212222552222211142 2016.
[34] Lin Tian, Hastings Greer, Roland Kwitt, François-Xavier Vialard. Raúl San José Estépar, Sylvain Bouix, Richard Rushmore, and Mare Niethammer. uniGradICON: A foundation model for medical image registration. In International Conference Medical Image Computing and Computer- Assisted Intervention, pages 749-760. Springer, 2024.
[35] Başar Demir, Lin Tian, Hastings Greer, Roland Kwitt, François-Xaan Xavier Vialard, Raúl San José Estépar, Sylvain Bouix, Richard Rushmore, Ebrahim Ebrahim, and Marc Niethammer. Multi- GradICON: A foundation model for multimodal medical image registration. In International Workshop Biomedical Image Registration, pages 3-18. Springer, 2024.
[36] Zi Li, Jianpeng Zhang, Tai Ma, Tony CW Mok, Yan-Jie Zhou, Zeli Chen, Xianghua Ye, Le Lu, and Dakai Jin. UniReg: Foundation model for controllable medical image registration. arXiv preprint arXiv:2500.. 12868, 2025.
[37] Rohit Jena, Pratik Chaudhari, and James C Gee. FireANTs: Adaptive riemannian optimization for multi-scale diffeomorphic matching. arXiv preprini arXiv:2404.01244 2024.
[38] Neel Dey, Benjamin Billot, Hallee E Wong, Clinton J Wang, Mengwei Ren, P Ellen Grant, Adrian V Dalca, and Polina Golland. Learning general-purpose biomedical volume representa- tions using randomized synthesis. arXiv preprint arXiv:2411.00
2024.
[39] Hanna Siebert, Christoph Großbröhmer, Lasse Hansen, and Mattias P Heinrich. ConvexAdame Self-contiguring dual-optimisstionn-aass based 3D multitask medical image registration. IEEE Transactions on Medical Imaging, 2024.
[40] Junguo Bian, Jeffrey H Siewerdsen, Xiao Han, Emil Y Sidky, Jerry L Prince, Charles A Pelizzari, and Xiaochuan Pan. Evaluation of sparse-view reconstruction from Hlt--panel-detectr cone-beam CT. Physics in Medicine & Biology, 55(22):6575, 2010.
[41] Mohammadhosseen Momeni, Vivek Gopalakrishnan, Neel Dey, Polina Golland, and Sarah Frisken. Voxel-based differentiable X-ray rendering improves sulf-supereieed 3D CBCT recon- struction. arXiv preprint arXiv:2411.192: 2024.
[42] Zachary Teed and Jia Deng. RAFT-3D: Scene flow using rigid-motion embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8375 8384, 2021.
[43] Elizabeth L Brainerd, David B Baier, Stephen M Gatesy, Tyson L. Hedrick, Keith A Met- zger, Susannah L Gilbert, and Joseph J Crisco. X-ray reconstruction of moving morphology (XROMM): precision, accuracy and applications in comparative biomechanics research. Journal of Experimental Zoology Part A: Ecological Genetics and Physiology, 313(5):262-279 2010.
[44] Benjamin J Knörlein, David B Baier, Stephen M Gatesy, JD Laurence- Chassen, and Elizabeth Brainerd. Validation of XMALab software for marker-based XROMM. Journal of Experimental Biology, 219(23):3701-3711 2016.
[45] Paul Kirkpatrick and Albert Vincio Baez. Formation of optical images by X-rays. Journal of the Optical Society of America, 7869):66--774, 1948.
13

---

[46] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cam- bridge University Press, 2nd edition, 2004.
[47] Donald F Swinehart. The Beer-Lambert law. Journal of Chemical Education, 39(7):333, 1962.
[48] Nelson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 1(2):99-108 108, 1995.
[49] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[50] Vivek Gopalakrishnan and Polina Golland. Fast auto- dite-differentialle digitally reconstructed radiographs for solving inverse problems in intraoperative imaging. In Workshop on Clinical Image-Basee Based Procedures, pages 1-11. Springer, 2022.
[51] Olivier Commowick, Vincent Arsigny, Aurélie Isambert, Jimena Costa, Frédéric Dhermain, François Bidault, P-Y Bondiau, Nicholas Ayache, and Grégoire Malandain. An efficient locally affine framework for the smooth registration of anatomical structures. Medical Image Analysis, 12(4):427-44 2008
[52] Russell Wustenberg. Carpal bone rigid-body kinematics by log-Euclidean polyrigid estimation. Master's thesis, New York University Tandon School of Engincering, 2022.
[53] Batool Abbas, James Fishbaugh, Catherine Petchprapa, Riccardo Lattanzi, and Guido Gerig. Analysis of the kinematic motion of the wrist from 4D magnetic resonance imaging, In Medical Imaging 2019; Image Processing, volume 10949, pages 351-356 SPIE, 2019.
[54] Isaac Newton, I Bernard Cohen, and Anne Whitman. The Principia: mathematical principles of natural philosophy. University of California Press, 1999.
[55] Brian B Avants, Charles L Epstein, Murray Grossman, and James C Gee. Symmetric diffco- morphic image registration with cross-cor:elation: evaluating automated labeling of elderly and neurodegenerative brain. Medical image analysis, 12(1):26-41, 2008
[56] Robert B Grupp. Mehran Armand, and Russell H Taylor. Patch-based image similarity for intra- operative 2d/3d pelvis registration during periacetabular osteotomy. In OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Based Procedures, and Skin Image Analysis: First International Workshop. OR 2.0 2018, 5th International Work- shop, CARE 2018, 7th International Workshop, CLIP 2018, Third International Workshop. ISIC 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16 and 20, 2018, Proceedings 5, pages 153-163. Springer, 2018
[57] Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint urXiv::142.. 6980, 2014.
[58] Tatiana Bejarano, Mariluz De Ornelas-Coute and Ivaylo B Mihaylov. Head-and-neck squamous cell carcinoma patients with CT taken during pe-treatment, mid-treatment, and post- post-treatment (HNSCC-3DCT-RT), https://doo.org/11.773/////////
88 13upr2xf, 2018.
[59] Tatiana Bejarano, Mariluz De Ornelas- CCutoo, and Ivaylo B Mihaylov. Longitudinal fan-beam computed tomography dataset for head-and-neck squamous cell carcinoma patients. Medical Physics, 2665):252-- 2019.
[60] Robert B Grupp, Mathias Unberath, Cong Gao, Rachel A Hegeman, Ryan J Murphy, Clayton P Alexander, Yoshito Otake, Benjamin A McArthur, Mehran Armand, and Russell Taylor. Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration. International Journal of Computer Assisted Radiology and Surgery, 15:759-769, 2020.
[61] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Voxel- Morph: a learning framework for deformable medical image registration. IEEE Transactions on Medical Imaging. 1788-1178818000 2019.
14

---

[62] Lee A Feldkamp, Lloyd C Davis, and James W Kress. Practical cone-beam algorithm. Journal of the Optical Society of America A, 1(6):612-619 1984.
[63] Alex Leow, Igor Yanovsky, Ming-Chang Chiang, Agatha D Lee, Andrea D Klunder, Allen Lu, James T Becker, Simon W Davis, Arthur W Toga, and Paul M Thompson. Statistical properties of Jacobian maps and the realization of unbiased targe-deformation nonlinear image registration. IEEE Transactions Medical Imaging, 26(6).822 822-832, 2007.
[64] Jakob Wasserthal, Hanns-Christial Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel Boll, Joshy Cyriac, Shan Yang, et al. TotalSeg- mentator: robust segmentation of 104 anatomic structures in CT images, Radiology: Artificial Intelligence, 5(5):2230024, 2023.
[65] Hallee E Wong, Marianne Rakic, John Guttag, and Adrian V Dalca. ScribblePrompt: fast and flexible interactive segmentation for any biomedical image. European Conference on Computer Vision, pages 207-229. Springer, 2024.
[66] Fabian Isensee, Maximilian Rokuss, Lars Krämer, Stefan Dinkelacker, Ashis Ravindran, Flo- rian Stritzke, Benjamin Hamm, Tassilo Wald, Moritz Langenberg, Constantin Ulrich, al. anInteractive: Redefining 3D promptable segmentation. arXiv preprint urXiv:2503.88373 2025.
[67] Jose-Luis Blanco. A tutorial on SE(3) transformation parameterizations and on-manifold optimization. University of Malaga, Tech. Rep, 3(6):1, 2010.
[68] Geoffrey D Hugo, Elisabeth Weiss, William C Sleeman, Salim Balik, Paul Keall, Jun Lu, and Jeffrey F Williamson. Data from 4d lung imaging of nscle patients (version 2) [data set]. http://bo..rr/11..9997////////// 2016 60.......YGLE.... 2016.
[69] Geoffrey Hugo, Flisabeth Weiss, William C Sleeman, Salim Balik, Paul Keall, Jun Lu, and Jeffrey F Williamson. A longitudinal four-dimensional computed tomography and cone beam computed tomography dataset for image-guided radiation therapy research lung cancer. Medical physics, 44(2):622-771, 2017.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our experimental results demonstrate the accuracy and topological consistency of our estimated deformation fields in sparse-view and limited-angle settings (Section 4.3), well as universally improved performance over all baselines.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations and future work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [Yes]
Justification:
We carefully list the minor assumptions we make in Section 3.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In Section 3, we provide all necessary technical details for the proposed PolyPose algorithm. including the theoretical framework and our vectorized implementation. We also provide closed-form formulas for the log and exp maps on SE(3), necessary to implement Eq. (4) in Appendix B. Py Torch implementation of PolyPose, along with data preprocessing, training, and testing scripts, is provided in the Supplement. The datasets we analyze in this work (Head&Neck and DeepFluoro) are publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
17

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: A pip-installable version of PolyPose. along with all training and testing scripts, are provided in the attached code repository. All our experiments are performed using publicly available datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines https://nips.cc/pbbicc
Suudes/CooeSSummissioPooiiic for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (htips: hhtppuulllgguidde/oooeeebbbebeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All details of the train/test splits, optimizer, and optimizer hyperparameters can be found the descriptions of the datasets (Section 4.1) and in our implementation details (Section 3.4 and Appendix C).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We specify in Table that all metries are reported mean(standard deviation).
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In our implementation details (Appendix C), we specify that all experiments were conducted on single NVIDIA RTX A6000. We provided memory and speed analyses of our methods in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't 'tttt make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have carefully reviewed the NeurIPS Code of Ethics and determined that our research is in accordance with the outlined standards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
19

---

This work proposes a state-of-the-arr algorithm for delivering 3D image guidance during challenging image-guided surgeries and interventions. The utility of our method in real-world clinical settings is discussed in Section 5.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
The research herein poses no risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and benchmark methods analyzed in this work have been appro- priately cited in accordance with their respective licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
20

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assett are released, the license, copyright information, and terms of the package should be provided. For popular datasets, paperswithcode. https/wwwooeoco//tttaseeeeee com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This work does not release new assets besides documented code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This research does not involve crowdsourcing. All analyses were retrospec- tively performed on publicly available datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We retrospectively analyze publicly available datasets. The collection of these datasets approved by IRBs at their original institutions.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This research does not involve LLMs in any manner.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy httt:://eeeiiis.cccccoffeeeeeees for what should or should not be described.
22

---

A Projective X-ray Geometry
To complete the derivation of the forward model for the negative log-intensity at a pixel p in an X-ray image, we must specify how to construct the intrinsic matrix K from the image's metadata.
The intrinsic matrix represents the mapping from camera to pixel coordinates [46]. This can be decomposed as first mapping from camera to image coordinates and second mapping from image pixel coordinates:
1/8x
0
W
2
f
0
0z
K
1/sv
H
2
0
f
0y
(10)
0
0
1
where f the camera's focal length, (0z: 0y) is the camera's optical center, (82, 8y) is the pixel spacing, and (H, W) is the image's 's height and width, respectively [16].
These intrinsic parameters for each X-ray image can readily be identified from the image's 's metadata
encoded in the DICOM (Digital Imaging and Communications in Medicine) header. Specifically,
The focal length f given by the DistanceSourceToobttettoor (0018,11100 attribute.
The optical center (Oas oy) is given by the DetectorkctiveOriggnn (0018,7028) attribute.
The pixel spacing (#m; 5y) is the given by the ImagerPixelSpacing (0018, 164) attribute.
The image dimensions (H, W' are given by the Rows (0028,0010) and Columns (0028,001] attributes, respectively.
B Lie Theory for Polyrigid Transforms
We summarize the Lie theory of SE(3) from Blanco [67] needed to implement PolyPose. We start by defining the logarithmic map, which maps any rigid transformation
R 0
E SE(3), where
R E SO(3) and t e RR
(11)
to the vector v = سا u/² E se(3) = R This vector corresponds to the matrix
W3
W2
log(T)
W3
U2
(12)
W2
0
Ug
0
which itself the generator of an infinitesimal rototranslation about the axis defined by u.
To efficiently write the formulas for w and u, it is convenient to first define the exponential map:
exp(v) = [elwix Du
(13)
0
where
[
sin
[w] +
cos 0
[w]
(14)
0
cos
- sin
Ω
I
ww) +
( x
(15)
0
g3
and ||>||| and Hx constructs skew-symmetric matrix from 3-vector.
Then, w is given by Rodrigues' rotation formula
R32 R
(race R) -
R13 R31
where 0 arccos
(16)
ssin 0
R21 R
2
and u -
23

---

C Additional Implementation Details
C.1 Estimating Camera Poses
To recover camera poses in an accurate and automatic manner, we use XVI, a -ptient-specific machine learning framework for state-of of-the-art rigid 2D/3D registration [16]. Specifically, given train patient-specific convolutional network fo I - [R t| to predict initial camera pose estimate for a given X-ray image using self-supervised synthetic pretraining. At inference time, we refine these initial pose estimates using differentiable rendering, protocol known as test-time optimization (Figure 3A). However, we modify the original test-time optimization protocol and instead optimize the pose of single anatomical structure Sanchor E .S....., Sx).. representation of the camera poses by rigidly registering the left clavicle the Head&Neck dataset and the pelvis in the DeepFluoro dataset.
We anchor our
Optimization problem. Given an image similarity loss function (e.g.. normalized cross correlation, mutual information, etc.). we estimate the extrinsic parameters of each X-ray image by independently solving the following optimization problem:
Rn, t
argmax (LT,, P(K,, t.l)) aaachr @ v))
(17)
[R..It.]ESE(3)
where @ is clement-wise multiplication used to mask the CT volume and render the structure Sanchor- This optimization is performed in the langent space of SE(3) using gradient descent. Finally,
n K, [R., t..).
(18)
We use the hybrid loss function gradient multiscale normalized cross correlation (gmNCC) to
guide 2D/3D rigid registration. This composite loss function is the average of multiscale NCC
(mNCC) [15], which averages NCC across the global and local scales, and gradient NCC (gNCC) [60], which computes NCC on Sobel-filtered versions of the image. This image similarity metric is advantageous for 2D/3D registration tasks mNCC encourages global alignment while gNCC encourages alignment of edges of bones.
C.2 Polyrigid Pose Optimization
Weight field. Given a labelmap for the preoperative CT scan, we first compute structure-specific Euclidean distance transforms for each rigid body whose pose will optimize. Examples of these per-structure distance fields are illustrated for a subject in the DeepFluoro dataset (Figure 8). Finally, these distance transform combined using Eq. (6). Since the weights fixed during optimized in PolyPose, this field can be precomputed before estimating the warp field.
Optimization. We represent the poses of every rigid body in the tangent space sc(3). Since translational parameters (u) in units millimeters are typically two orders magnitude larger than angular parameters (w) in units of radians,
the Adam optimizer with step sizes Brot = 10 for rotations and 8xyz = 10 for translations across all experiments, which is the same optimizer setup use for estimating camera poses in Eq. (17). We use the same gmNCC metric to compute image similarity the objective function (7).
two separate step sizes. Specifically,
Segmentations (8,)
Per-structure Euclidean distance field (d))
Weight field (w))
(R)
Figure 8: A slice of the weight field produced by Eq. (6). We visualize the weight field as the relative contribution of each structure every pixel in the slice.

---

D Implementations of Baselines
Below, we detail the implementation of all baselines compared this work. Note that all baselines, except for xvr, depend on accurate camera pose estimates, but do not specify protocols for calibrating the input X-ray images. Therefore, all methods (including our own) were evaluated using the same camera poses that we estimated using PolyPose.
[16] is rigid 2D/3D registration framework comprising (1) patient-specific neural network pretrained on synthetic data to produce accurate initial pose estimates and (2) a test-time optimization protocol to refine initial pose estimates. We train patient-specific neural networks and perform test-time optimization for each subject using the default architecture and training hyperparameters.
Lifteeg [18] is deep dictionary learning method for deformable 2D/3D registration. It uses PCA to construct a low-rank vector space of 3D deformations given a dataset of patients with multiple CTs. Since patients in DeepFluoro do not have multiple CT scans, we can only evaluate LiftReg the Head&Neck dataset. Specifically, we use FireANTss [37] to compute ground truth 3D deformations from pairs of CTs in the training of Head& Neck. Then, train CNN regress coefficients of the basis vectors, reconstruct the resulting deformation field, warp the moving CT, and compute the loss using 3D MSE and diffusion regularizer.
2D3D- AeeD-RegNet [19] uses VoxeIMorph-style [61] architecture to directly estimate 3D deformation field given moving CT and fixed CBCT reconstructed from the input 2D X-rays. It is supervised using an image similarity loss on X-rays rendered from the warped CTs and the real X-rays, as well inverse consistency regularizer and energy regularizer.
uniGradICON [34] and multiGradICON [35] are foundation models for intra- and inter- modrlty registration, respectively, trained large datasets. We use the pretrained models available in their repositories in our experiments. These neural networks not have any hyperparameters, and we optimize hyperparameters for their iterative variants on the validation set.
FireANTss [37] and anatomix [38] are improved solvers for 3D/3D registration that leverage novel optimization techniques and feature representations. We install the binaries available in their respective repositories and optimize the requisite hyperparameters on the validation set.
Dense 3 [23] places an optimizable displacement vector at every voxel in the moving CT scan. Similarly, Dense SE(3) [32] places optimizable rototranslation generator at every voxel. We optimize both dense parameterizations with the same differentiable rendering setup as in PolyPose.
E
Additional Results
We present further visualizations of the Head&Neck and DeepFluoro datasets and new analysis of lung radiotherapy dataset with extreme soft tissue deformation.
E.1 &Neck
In Figure we render the deformation fields produced by 2D3D- Re3D-RegNet [19] and multiGradICON (with and without IO) [35] for the same subject visualized in Figure 4. The warps produced by 2D3D-RegNet are well-behaved from a topological perspective, but fail accurately capture the inter-scal scan motion of the patient's 's head. Deformations produced by the multiGradICON variants display an interesting failure mode, with the warp field radiating away from the isocenter the CT scan. This dilation results in the anatomically implausible warped volumes visualized in Figure 4A.
2030 Net Folss 6:1
HOT % Taldue ologDD) 0.99
GOT & Folds 0.00 ofoggh UAB
PolyPose (our))
Faldds O 0
ologgh 0.51
alogiti) 0.70
Figure 9: 3D renderings of the deformation fields produced by 2D3D-RegNet [19] and multi- GradICON [35]. These visualizations are complementary to the examples shown in Figure
25

---

xvr [16]
Dense R² [23]
Dense Sens) [32]
PolyPose (ours)
FireANTs [37]
anatomix [38]
uniGradICON [34]
multiGradICON [35]
Figure 10: Visualizations of the warped CTs produced by various methods on the same subject. The top rows visualize central slices of the warped CTs and the bottom rows visualize maximum intensity projections along the coronal direction. Only PolyPose successfully recovers the anatomical motion (external rotation of the femurs) from minimal supervision (two X-ray images).
E.2 DeepFluoro
In Figure 10, visualize central slices and maximum intensity projections of the warped CTs produced by PolyPose as well as the baseline methods. This figure exemplifies many the common failure modes for previous 2D/3D and 3D/3D registration methods:
Dense parameterizations of the deformation field, such as R [23] and SE(3) [32, 42], can only influence voxels on which they have direct pixel supervision. As such, both of these methods break the femurs, outlining the bounded subregion of the preoperative volume that can be deformed.
Both FireANTs [37] and anatomix [38] produce very small deformations, failing to capture the inter-scan motion of the patient. While the deformations these methods produce are not physiologically implausible, they are inaccurate and not useful in real-world settings.
While uniGradICON [34] comes the closest of all the baselines to recovering the motion of the femurs, both it and multiGradICON [35] are affected by the streaking artifacts present in sparse-view CBCT reconstructions and produce dramatic dilations of the preoperative volume. This is analogous the dilating warps produced by multiGradICOON in the &eck dataset (Figures 4 and 9).
26

---

E.3 ThoraXCBCT
We perform additional experiments on dataset 13 patients undergoing chemoradiotherapy for advanced, non-small cell lung cancer [68]. Each patient had one pre-therapeutic fan-beat CT (FBCT) volume, used plan the radiotherapy, and two intra- inraainterventional cone-beam CT (CBCT) volumes, used to reorient the plan to the patient's current position and morphology [69]. Specifically, we registered pre-therapeutic FBCTs to synthetic X-rays generated from interventional CBCTs. This dataset is particularly challenging given the significant non-rigid respiratory motion between maximum inhalation (FBCT) and maximum expiration (CBCT) acquisitions.
Figure 11 visualizes exemplar registrations for the ThoraXCBCT dataset and reports quantitative evaluation metrics. Despite the substantial non-rigid deformation present in these volumes, PolyPose achieved superior registration accuracy compared to state-of-thee -ottte--rtt-ttttt- 2D/3D and 3D/3D baselines. Notably, by modeling the motion of rigid bodies, PolyPose successfully captured the deformation of the soft tissue structures in the scene, including the liver, heart, aorta, trachea, esophagus, and all lobes of the lungs. This evaluation further demonstrates that the skeleton provide sufficient geometric constraints for accurate soft tissue registration, even under extreme respiratory conditions.
FireANTs [37]
anatomix [38]
xvr [16]
PolyPose (ours)
Overlay Warped CT
(1000.12)/0.36(0..0
0.25(0.15) 0.58(0.16) 0.51(0.20) / (.74(0.18) Dice (Rigid Bodies Soft Tissues)
0.73(0.19) 0.81(0.11)
Figure 11: PolyPose accurately models extreme deformations, including respiratory motion. PolyPose achieves the highest 3D Dice for both rigid bodies and soft tissue, respectively.
F
Failure Cases
By construction, PolyPose produces diffeomorphisms. While this is intentional and generally a desirable property (the majority of human motion is smooth and invertible). diffeomorphisms do not represent all types of motion. We visualize this failure mode using a CT scan from an internal dataset of neurosurgical patients where the patient's mouth closed in the preoperative CT, while is open in the intraoperative X-rays. To represent opening the jaw, PolyPase repositions the patient's ' mandible in the warped CT. However, as the patient's 's top and bottom rows of teeth were touching in the preoperative CT, this downward warp creates an anatomically implausible stretching of the teeth in the lower jaw (see the red box in Figure 12). This is because, diffeomorphisms are invertible. they cannot model the creation of empty space occurs when the mouth opens. This defect results the creation of a third row of teeth, as seen the volume rendering.
Preoperative CT
Warped CT from PolyPose (ours)
An exemplar failure mode of diffeomorphisms. The diffeomorphisms produced by PolyPose cannot represent certain motions, such the opening of the mouth, as the top and bottom rows teeth are touching in the preoperative CT scan and would require the creation of topologically- inconsistent empty space match the target intraoperative X-rays.
Figure 12:
27


Reviews:
[{"review_id": "f9Z0fLzYbk", "reviewer": "Reviewer_7gVY", "summary": "This paper introduces a locally-rigid registration framework (PolyPose) for CT to X-ray (2D/3D) registration. PolyPose is a two-stage method. In stage 1, PolyPose estimates the initial camera poses for each X-ray scan with differentiable X-ray rendering and gradient descent. Subsequently, PolyPose leverages segmented rigid structures (bones) to create a precomputed weight field and estimate the local rigid transform for each structure. The final deformation field is a weighted combination of the resulting local rigid transforms. The proposed method was evaluated using synthetic Head&Neck datasets (19 test subjects) and real clinical datasets (DeepFluoro, comprising 6 test subjects with CT-X-ray pairs). PolyPose achieves the best overall registration with no folding in the deformation field among 3D registration methods (uniGradICON, multiGradICON, FireANTs, anatomix) and 2D/3D registration methods (DiffPose, LiftReg and 2D3D-RegNet).", "questions": "Can PolyPose register images with serious metal artefacts? Can the differentiable X-ray projection accurately simulate the metal artefact?\n\nGiven 1 CT and 1 X-ray scan, will there be any difference between PolyPose and DiffPose?\n\nIn what situations will the DiffPose fail but the PolyPose not?", "limitations": "PolyPose relies on rigid structure segmentation. The potential limitations and impact are discussed in the paper.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\nA practical and elegant solution for CT to multi-X-ray registration. This paper proposes a promising solution for CT to multi-X-ray registration, a less-explored yet important topic in the context of image registration. By leveraging strong domain priors (bones are rigid, articulated motion is piecewise rigid) and differentiable X-ray rendering, PolyPose achieves robust and accurate CT to multi-X-ray registration without the need for a large training dataset. \n\nThorough and rigorous evaluation. The registration performance is comprehensively evaluated with both synthetic and real clinical images. Moreover, an ablation study on the deformation parameterization, weight function and rigid components is provided.\n\nSOTA results in 2D/3D registration. PolyPose achieves superior results over strong competitive methods such as DiffPose and FireANTs.\n\nStrong reproducibility. The source code is provided in the attachment and is well-written.\n\nGood clarity. This paper is well-written. The figures and mathematical formulations are clear and easy to follow.\n\n**Weakness**\n\nReliance on differentiable X-ray rendering and rigid structure segmentations: PolyPose comprises differentiable X-ray rendering and rigid structure segmentations, which hinders its potential for other applications such as MRI to X-Ray registration. Moreover, the differentiable X-ray rendering is tailored for CT to X-ray projection, and it is not applicable to other modalities, which further limits the application of PolyPose.\n\nLimited novelty. The success of PolyPose has been greatly fueled by differentiable X-ray rendering, rigid structure segmentations, and polyrigid registration, which are not new in the literature. Although an improvement over the weight field is introduced, the performance gain is rather marginal, as shown in Table 3. \n\nRobustness to exceptional cases. PolyPose may fail to handle soft tissue deformation and metal artefacts. First, as discussed in the limitation, since PolyPose assumes rigid bodies capture all motion, the local deformation of soft tissue, such as breathing and muscle contraction, may degrade the quality of the solution. Second, in real clinical scenarios, the metal artefact is common in post-treatment CT scans. Can PolyPose register images with serious metal artifacts? Can the differentiable X-ray projection accurately simulate the metal artefact?", "was_revised": true, "final_justification": "The rebuttal has addressed all my concerns. There is a clear difference between DiffPose and PolyPose. The proposed method is a practical solution for CT to X-Ray registration. The proper combination of differentiable X-ray rendering, rigid structure segmentations and transformation model shows well-grounded effort. Since the proposed method cannot directly generalise to all the 2D-3D registration tasks, its impact is slightly hindered. Hence, I will keep my positive rating ('Borderline accept').", "rebuttal": "We thank the reviewer for their insightful comments and questions. We are glad that they found the problem to be under-explored yet important, and our solution to be practical, elegant, and rigorously evaluated.\n\n> _”PolyPose comprises differentiable X-ray rendering and rigid structure segmentations, which hinders its potential for other applications such as MRI to X-Ray registration..”_\n\nWe agree that our work does not apply to the MRI to X-ray registration problem. For context, CTs and X-rays are directly related via the Radon transform, which enables differentiable rendering of 2D X-rays from 3D CTs. In contrast, MRI and X-ray share no analytical relationship, changing the problem to *multi-modal* 2D/3D registration of slices to a volume, which is different from the problem we tackle.\n\nWe respectfully posit that, as CT volumes are nearly universally acquired for planning various interventional procedures (e.g., radiotherapy, orthopedic surgery), our problem formulation accepts a broad scope of potential operative applications of PolyPose. We agree that this should be made clearer to the reader and we will revise our discussion to do so. Thank you for the clarifying question!\n \n> _”Limited novelty. The success of PolyPose has been greatly fueled by differentiable X-ray rendering, rigid structure segmentations, and polyrigid registration, which are not new.”_\n\nWe agree that the listed sub-components of our framework are foundational concepts in CT and X-ray image analysis. To clarify, we do not claim to contribute new methods for segmentation or X-ray rendering. Our main contribution is a new problem formulation: deformable piecewise-rigid alignment of preoperative volumes to intraoperative X-rays. Quantitatively, our work sets a new empirical state-of-the-art for this under-explored yet important problem (as noted by the reviewer), enabling surgical navigation applications that were not possible with previous methods. We will explicitly clarify this in the revision.\n\n> _”As discussed in the limitation, since PolyPose assumes rigid bodies capture all motion, the local deformation of soft tissue, such as breathing and muscle contraction, may degrade the quality of the solution.”_\n\nThis may be a potential miscommunication. While our method does not explicitly represent soft tissue deformations, soft tissue structures directly contribute to the image-based loss used in alignment (Figure 3B) and are thus incorporated into the deformation estimation. To clarify, our only assumption is that bones move rigidly; no rigidity is assumed for other structures. \n\nFor example, in the highly ill-posed setting of CT to X-ray registration of the Head&Neck dataset containing both rigid bodies and soft tissue, methods that do not use this prior produce unrealistic and suboptimal results (Figure 4), as soft tissue structures are very difficult to distinguish in X-ray and the deformation model is unconstrained. In contrast, our prior on human motion enables PolyPose to successfully extrapolate the deformations to soft tissue structures that were not explicitly modeled to achieve the best overall alignment (Figure 5).\n\n> _”In real clinical scenarios, the metal artefact is common in post-treatment CT scans. Can PolyPose register images with serious metal artifacts? Can the differentiable X-ray projection accurately simulate the metal artefact?”_\n\nThank you for raising this important point about real-world deployment. While metal implants cause serious artifacts on CT, the implants are well-defined on X-ray images, and differentiably rendered X-rays accurately match the appearance of real X-rays. For example, in a different context, [16] successfully used such differentiable renderings from CTs with metal implants for rigid 2D/3D registration. \n\nAs a result, PolyPose can register images with metal artifacts. However, PolyPose requires segmentations of rigid bodies in the CT and automated segmentation tools like TotalSegmentator [64] may degrade under severe metal artifacts. Fortunately, as shown in our responses to reviewers `Jfu5` and `CMp5`, PolyPose is robust even with degraded segmentations. We will clarify in the revision that experiments on images with metal implants are an important area of future work.\n\nLastly, if the reviewer is interested in observing an X-ray rendered from a metal-implanted CT, we would be happy to share a code snippet to do so during the discussion period, as figures are not permitted in the rebuttal. \n\n> _”Given 1 CT and 1 X-ray scan, will there be any difference between PolyPose and DiffPose?”_\n\nYes, DiffPose and PolyPose will produce very different outputs in this scenario:\n- DiffPose will estimate a single rigid transform to align the two images;\n- PolyPose will estimate multiple rigid transforms (one for each rigid structure in the CT) and interpolate them to produce a dense diffeomorphic deformation field.\n\nBroadly, DiffPose and PolyPose do not differ in the number of scans used, but in the number of rigid transforms estimated.\n\n> _”In what situations will the DiffPose fail but the PolyPose not?”_\n\nAs DiffPose is a rigid-only 2D/3D registration method, DiffPose will fail whenever there is no *single* global rotation and translation that captures the patient’s motion between their preoperative and intraoperative positions. On the other hand, PolyPose performs *deformable* 2D/3D registration and can accurately capture the complex articulated multi-joint movements common in human movement. Examples include the articulated motion of the skull, spine, and shoulder complex in the Head&Neck dataset (Figure 5) and the independent movement of the femurs relative to the pelvis in the DeepFluoro dataset (Figure 6 and Figure 10 in Appendix E.2)."}, {"review_id": "LnzJ2uFUey", "reviewer": "Reviewer_Y5jy", "summary": "The authors present PolyPose, a simple and robust method for deformable 2D/3D registration. PolyPose parameterizes complex 3D deformation fields as a composition of rigid transforms. First by estimating the camera matrcis through an anchored sub-structure that is reliably visible across all views and rigidly aligned, the authors get the accurate extrinsic matrix estimation for every angled X-ray image. Then, after obtaining the correct camera matrics, the authors construct the polyrigid deformation field by leveraging comparable sized structures as separate warped coordinate systems, and weighted by respective mass and distance controlled weight field for each structure and 2D X-ray point combination to represent the global rigid transformation.", "questions": "Please see my weakness section.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The novelty is clear and prominent. By simplifying a non-rigid patient and procedure specific transform into the polyrigid warps at all voxels of the 3D pre-volume which is decided by a linear mass and distance weighted combination of all rigid articulated bodies.\n2. Estimating the camera matrices through an anchor structure is very clever. The criteria for selecting such anchor structure is also important, such as (a) is reliably visible across all views in X-ray 2D sparse registration, as well as (b) could be rigidly aligned via its static property, like pelvis in Figure 3A.\n3. The formalization is clear and complete, although the methodology is complex, the description is clear and precise.\n4. The experiment resutls clearly show the superiority of proposed technique, including larger overlay in dice score and lower deformation error and topological error in tangent space, for most views of X-ray sparse registration, over multiple baselines on two large body data sets.\n\nWeakness:\n1. In equation (6), why only structure mass and distance are taken into consideration, how about their size and the distance to the mass center point?\n2. With equation (3) and rigidly aligned S_{anchor} which is available at all views, all angled camera matrices could be estimated, but how many sampled 2D points $p$ in X-ray images are needed for a correct evaluation for the X-ray projection operator P? And what is the confidence we should keep for such a discrete approximation of first-order approximation following Beer-Lambert law? Any alternative evaluation loop between camera matrics and acquisition deformation field needed?\n3. Equation (9) I can understand the SE(3) to se(3) transformation, but why it is isomorphic to R^6?\n4. For experiment, I am more interested in a general body data set where different articulated bodies or soft tissues are in different size and mass, and no rigid transform or global alignment guarantee for any body or structure.", "was_revised": true, "final_justification": "After reading authors' rebuttal and discussion with other reviewers, my concerns have been addressed and thanks for that.\nHowever, this work is still limited to fixed setting of targeted 2d-3d scenarios, which hinders its general application to all tasks. Considering all above aspects, I decide to keep my original positive rating.", "rebuttal": "We thank the reviewer for their interesting mathematical questions, and are happy that they found our method to be clever and our formalization of a mathematically complex methodology to be clear, complete, and precise.\n\n> _”In equation (6), why only structure mass and distance are taken into consideration, how about their size and the distance to the mass center point?”_\n\nRegarding size, we want to respectfully clarify that our weight function (Equation 6) does take the relative size of rigid structures into consideration. For all rigid bodies, we assume that bone density is constant (Line 156), such that a structure’s mass is directly proportional to its volume (i.e., size). Regarding distance, we use the distance to the nearest point on the surface and not the distance to the center of mass, because soft tissue is most influenced by the nearest piece of bone that it is attached to. \n\nLastly, we would like to emphasize that PolyPose is modular and alternative weight field formulations can be easily integrated. We will emphasize these points in the revised methods section.\n\n> _”With equation (3) and rigidly aligned S_anchor which is available at all views, all angled camera matrices could be estimated, but how many sampled 2D points p in X-ray images are needed for a correct evaluation for the X-ray projection operator P?”_\n\nWe sample as many 2D points as there are pixels in the X-ray image. For example, if the X-ray image is $512 \\times 512$ pixels, then we cast $512^2 = 262,144$ rays using Equation 3. We note that this rendering operation requires only tens of milliseconds to render a $512 \\times 512$ X-ray, as it is efficient and can be parallelized over rays on the GPU.\n\n> _”What is the confidence we should keep for such a discrete approximation of first-order approximation following Beer-Lambert law?”_\n\nA first-order approximation of the Beer-Lambert law is sufficient for registration tasks because\n- The resulting synthetic images are sufficiently similar to real X-rays (see Figure 3); and\n- A discrete renderer implemented in this manner faithfully recapitulates the projective geometry underlying X-ray image formation.\n\nSimulating second-order photonic effects, such as beam hardening or scattering, may further improve small textural details within the rendered X-ray, but these details do not change the image geometry relevant to registration. Lastly, this discrete first-order approximation has been successfully used in many widely adopted 2D/3D X-ray/CT registration approaches (e.g., DiffPose, CVPR 2024 [15]).\n\n> _”Any alternative evaluation loop between camera matrics and acquisition deformation field needed?”_\n\nWe are unsure of what the reviewer meant by this question. If they are asking whether it is possible to alternate between optimizing the camera matrices used for polyrigid deformations and the raw unconstrained deformation field itself, this can be done, yes. However, we did not find this to be necessary for our experiments, as the polyrigid deformation widely outperforms baseline methods that operate directly on the deformation field in Table 1 and Figure 5. We would be happy to discuss further in the discussion period if we misunderstood the question.\n\n> _”Equation (9) I can understand the SE(3) to se(3) transformation, but why it is isomorphic to R^6?”_\n\nThank you for catching this typo. We will update the text to instead say: “the codomain of the logarithmic map is *homeomorphic* to $R^6$” in the revision, where this six-dimensional space corresponds to three rotational and three translational parameters (Appendix B).\n\n> _”I am more interested in a general body data set where different articulated bodies or soft tissues are in different size and mass, and no rigid transform or global alignment guarantee for any body or structure.”_\n\nWe respectfully clarify that the Head&Neck dataset exactly satisfies the conditions that the reviewer wished to see tested. Within that dataset, all the articulated bodies and soft tissue structures are of vastly different sizes and masses, including the 21 articulated bodies (L/R humerus, L/R scapula, L/R clavicles, thoracic and cervical vertebrae, and skull) and five soft tissue structures (thyroid, spinal cord, brain, esophagus, and trachea). Furthermore, no global rigid or affine transform describes the motion of patients in this dataset, nor any of their constituent soft tissues."}, {"review_id": "LQr7yUCrxo", "reviewer": "Reviewer_CMp5", "summary": "In this paper, the authors tried to address the problem of deformable 2D/3D registration between a preoperative 3D scan (e.g. a volumetric CT) with 2D X-ray images acquired during the procedure (intra-operative). Instead of going full rigid or full deformable, they new idea is to leverage an anatomical prior that individual bones move as rigid bodies to simplify the deformation model. By combining these local rigid motions, the method produces a locally rigid warp field that is smooth and diffeomorphic by construction. A differentiable X-ray rendering procedure is used o compare the warped 3D volume against the 2D X-rays, allowing gradient-based optimization of the rigid body poses. The authors demonstrates that the proposed approach, PolyPose, can align a patient's preoperative 3D volume to as few as two 2D X-rays, even from limited angles, without additional regularization. Extensive experiments on diverse datasets from orthopedic surgery and radiotherapy show that PolyPose outperforms prior approaches.", "questions": "- The method relies on treating certain anatomical parts as rigid bodies. How are these rigid structures defined and obtained in practice? (Manually?)\n- As mentioned, it would be helpful to analyze the robustness of the proposed method against segmentation errors.\n- In the limited-angle scenarios, some bones might be barely visible or even completely occluded in the available X-ray views. How does the model handle a rigid component with little direct 2D supervision? And since there is no explicit regularization, it would be helpful to discuss this corner cases\n- The proposed method didn't optimize soft tissue motion, and such motion are carried along by blended warp field. The results show improved alignment of some soft organs without modeling them. However, one might wonder about scenarios with significant non-rigid soft tissues change? Is this still under scope of PolyPose?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- A novel approach to apply a locally-rigid deformation model in the 2D/3D registration setting.\n- The model can work as few as 2 X-ray images, and with limited angle scenarios\n- The proposed method does not require expensive deformation regularization or hyperparameter tuning. This greatly improves the ease of use in real world applications.\n- Strong performance comparing to deep learning based or optimization-based methods. \n- The code is publicly available with all implementation details.\n\nWeakness\n- The proposed method requires accurate segmentation of rigid bones. A study of how the performance depends on segmentation accuracy would be helpful. While there are many segmentation tools nowadays, they still make notable errors from time to time in real world application, esp. on diseased situation or anomality.\n- As the authors mentioned, diffeomorphic deformation cannot be handled here, such as jaw opening. In fact, during neurosurgery, there might be minor deformation of the patient's head against the pre-operative scan, due to the head clamp. In other words, it is unclear how polypose can handle such situation when there exists even minor non-rigid deformation in the local rigid piece of bones.\n- The paper does not explicitly discuss runtime or computational efficiency, which could be important for intraoperative use.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful questions. These led to interesting additional experiments that have improved our understanding of the robustness of PolyPose. We are also glad that they found our novel approach to be extensively validated and easy to use in real-world applications due to its lack of expensive regularization or optimizable hyperparameters.\n\n> _”The proposed method requires accurate segmentation of rigid bones. A study of how the performance depends on segmentation accuracy would be helpful. While there are many segmentation tools nowadays, they still make notable errors from time to time in real world application, esp. on diseased situation or anomality.”_\n\nThis is an important question to investigate – thank you for raising it. To simulate errors typical of automated segmentation tools like TotalSegmentator, we present an experiment below where we systematically erode the ground truth segmentations in the DeepFluoro experiments by increasing the radius of erosion. Here, radius=0mm corresponds to no erosion and is equivalent to the experiment in Table 1 (registration given two X-rays from a limited viewing angle). We find that as we increase the strength of corruption, our method remains robust against extreme corruption and still outperforms the baselines in Table 1 both in terms of Dice and Hausdorff Distance.\n\n| **Erosion Radius** | **Pelvis - Dice (↑)** | **Pelvis - HD95 (↓)** | **Femur (L) - Dice (↑)** | **Femur (L) - HD95 (↓)** | **Femur (R) - Dice (↑)** | **Femur (R) - HD95 (↓)** |\n|--------------------|-----------------------|-----------------------|--------------------------|--------------------------|--------------------------|--------------------------|\n| 0 mm               |       0.99(0.00)      |       1.00(0.00)      |        0.99(0.00)        |        1.02(0.10)        |        0.98(0.00)        |        1.43(0.36)        |\n| 1 mm               |       0.99(0.00)      |       1.00(0.00)      |        0.98(0.00)        |        1.11(0.49)        |        0.98(0.00)        |        1.43(0.46)        |\n| 2 mm               |       0.99(0.00)      |       1.00(0.00)      |        0.98(0.00)        |        1.07(0.24)        |        0.98(0.01)        |        1.66(0.43)        |\n| 3 mm               |       0.99(0.00)      |       1.00(0.00)      |        0.98(0.00)        |        1.15(0.27)        |        0.97(0.01)        |        1.55(0.45)        |\n| 4 mm               |       0.99(0.00)      |       1.01(0.07)      |        0.97(0.00)        |        1.71(0.50)        |        0.97(0.01)        |        1.84(0.42)        |\n| 5 mm               |       0.99(0.00)      |       1.11(0.23)      |        0.95(0.00)        |        3.20(1.25)        |        0.96(0.01)        |        2.28(0.69)        |\n| 6 mm               |       0.99(0.00)      |       1.21(0.34)      |        0.95(0.01)        |        2.89(1.18)        |        0.96(0.01)        |        2.95(1.76)        |\n| 7 mm               |       0.99(0.00)      |       1.41(0.46)      |        0.94(0.01)        |        3.64(1.27)        |        0.95(0.01)        |        3.89(3.08)        |\n| 8 mm               |       0.99(0.00)      |       1.64(0.49)      |        0.93(0.01)        |        5.13(3.19)        |        0.95(0.01)        |        3.70(3.17)        |\n| 9 mm               |       0.99(0.00)      |       1.83(0.61)      |        0.94(0.01)        |        3.47(1.50)        |        0.94(0.01)        |        4.24(3.36)        |\n\nThe results are stable to 4 mm of erosion, beyond which point the accuracy degrades gracefully. However, an erosion radius of just 4mm corresponds to a significant corruption of the original segmentation with its volume decreasing by 40%-60%. For example, the volume of the left femur label decreases from 186,953 mm³ to 117,440 mm³ with an erosion radius of 4 mm.\n\n> _”“The method relies on treating certain anatomical parts as rigid bodies. How are these rigid structures defined and obtained in practice? (Manually?)”_\n\nRegarding the definition of rigid pieces, our only assumption is that bones do not bend in typical motion. Thus, bone segmentations are defined as rigid pieces, and no other anatomical regions are treated as rigid bodies. \n\nRegarding obtaining these bone (rigid body) segmentations on our CT volumes, individual bone segmentations are easily obtained using off-the-shelf tools such as TotalSegmentator [64]. This requires only a few seconds per volume without any manual intervention, adding minimal time complexity. For example, this is the exact procedure we used to obtain bone segmentations for PolyPose for the Head&Neck dataset.\n\n> _”As the authors mentioned, diffeomorphic deformation cannot be handled here, such as jaw opening. In fact, during neurosurgery, there might be minor deformation of the patient's head against the pre-operative scan, due to the head clamp. In other words, it is unclear how polypose can handle such situation when there exists even minor non-rigid deformation in the local rigid piece of bones.”_\n\nWe respectfully clarify that our method *specifically* produces diffeomorphic deformations by construction. While each bone is assumed to move rigidly, the overall deformation field is estimated by a tangent space interpolation of rigid transforms that best fits the image alignment losses and does not require each bone to move with exact rigidity in the final deformation. Further, while diffeomorphisms cannot cover edge cases that are non-diffeomorphic (e.g., the jaw opening, a limitation that the reviewer noted from Appendix G), it is sufficiently accurate to describe subtle topological changes, like the indentations caused by a head clamp.\n\nDue to this diffeomorphic constraint, we do not claim to be accurate if there are major topological changes between the preoperative and intraoperative imaging, e.g., if the patient’s skull appears intact in the CT, but is fractured in the X-ray. These scenarios could be addressed in PolyPose by masking the loss function in regions without topological correspondence; however, this is beyond the scope of our work and  there is no publicly available paired CT/X-ray dataset with major topological changes for us to evaluate such a variant. We will expand the discussion section to emphasize these points—thank you for raising them.\n\n> _”The paper does not explicitly discuss runtime or computational efficiency, which could be important for intraoperative use.”_\n\nThank you for raising this. We report the runtimes of all methods evaluated in our paper on the Head&Neck cancer dataset in the table below and will include this information in the paper. While PolyPose is slower than the inference-only LiftReg and 2D3D-RegNet methods, it requires no training and is much more accurate. Runtimes and Dice metrics are averaged over all numbers of views (2-9) evaluated in Figure 5. All methods are evaluated on a single NVIDIA RTX A6000 GPU.\n\n|                  | **Dice (Rigid Bodies)** | **Dice (Soft Tissues)** | **Runtime (min:sec)** |\n|------------------|:-----------------------:|:-----------------------:|:---------------------:|\n| PolyPose         |        0.71(0.15)       |        0.76(0.08)       |       0:59(0:20)      |\n| FireANTS         |        0.39(0.13)       |        0.52(0.17)       |       0:41(0:08)      |\n| anatomix         |        0.57(0.14)       |        0.66(0.13)       |       1:05(0:12)      |\n| multiGradICON    |        0.03(0.02)       |        0.09(0.03)       |       0:28(0:03)      |\n| multiGradICON+IO |        0.20(0.14)       |        0.32(0.17)       |       1:10(0:08)      |\n| 2D3D-RegNet      |        0.58(0.12)       |        0.67(0.09)       |       0:01(0:01)      |\n| LiftReg          |        0.47(0.13)       |        0.57(0.12)       |       0:02(0:01)      |\n| DiffPose         |        0.53(0.15)       |        0.61(0.13)       |       0:06(0:02)      |\n\n> _”In the limited-angle scenarios, some bones might be barely visible or even completely occluded in the available X-ray views. How does the model handle a rigid component with little direct 2D supervision? And since there is no explicit regularization, it would be helpful to discuss this corner cases”_\n\nThis scenario occurs within our experiments on the DeepFluoro dataset, which is both limited-angle and has barely visible structures (the L/R femurs) due to the restricted field-of-view. PolyPose extrapolates well to these structures and outperforms existing widely-used methods for this task. For more details, please see Figure 6 and Table 1, as well as Figure 10 in Appendix E.2.\n\n> _”The proposed method didn't optimize soft tissue motion, and such motion are carried along by blended warp field. The results show improved alignment of some soft organs without modeling them. However, one might wonder about scenarios with significant non-rigid soft tissues change? Is this still under scope of PolyPose?”_\n\nThank you for this interesting question! In general, PolyPose is effective for soft tissues that deform along with the movement of rigid structures (e.g., the esophagus and skull / cervical spine, the lungs’ upper lobes and the ribs / sternum / spine, etc.). For these cases, extreme soft tissue motion will also be coupled with significant motion of the rigid structures, which our experiments show can be faithfully modeled by PolyPose. We do not claim applicability to procedures that are entirely non-rigid with no rigid substructures visible that could be used as anchors. We will make sure to expand the limitations to discuss this."}, {"review_id": "5gARx5sVVG", "reviewer": "Reviewer_Jfu5", "summary": "This paper presented PolyPose for deformable 2D/3D registration. PolyPose parameterizes complex 3D deformation fields as a composition of rigid transforms, leveraging the biological constraint that individual bones do not bend in typical motion. The model enforces anatomically plausible priors that respect the piecewise rigid nature of human movement. PolyPose solves sparse-view deformable registration and aligns preoperative 3D images to intraoperative 2D X-rays using a locally rigid deformation model. It requires only X-rays and eliminates complex regularization. Extensive experiments on diverse datasets from orthopedic surgery and radiotherapy align the patient’s preoperative volumes to two X-rays.", "questions": "1. The model relied on the assumption that anatomical structures are rigid, being limited to bones with subtle bending or complex joint interactions. Whether the proposed model can be extended to soft tissue deformation and non-skeletal contexts?\n2. The model required precise segmentation of rigid bodies in the preoperative volumetric images. The segmentation errors can cause inaccurate deformation fields. The manual and semi-automatic segmentation impose additional time complexity.\n3. The initial camera poses relied on rigid registration of an anchor structure. The obscured anchor structure can cause pose estimation errors.\n4. Tangent space averaging (Eq. 4) assumes transforms combine linearly in se(3). It is unclear whether such computation can handle large-scale transformations or deformations.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\nThis paper presented a regularization-free framework for deformable 2D/3D registration that estimates polyrigid deformation fields using differentiable X-ray rendering.　The proposed anatomically motivated motion model is robust in sparse-view and limited-angle settings and produces smooth, invertible, and accurate deformation fields through construction.\n\nWeaknesses\nThe model relied on the assumption that anatomical structures are rigid. It would be helpful to discuss its generalization capacity when confronted with subtle bending or complex joint interactions. Moreover, the model relied on manual or semi-supervised segmentation of rigid bodies in the preoperative volumetric images and inaccurate deformation fields. The initial camera pose estimation relied on rigid registration of an anchor structure.", "was_revised": true, "final_justification": "The authors have provided detailed clarifications to main concerns in the review, including those regarding the rigid anatomical structure alignment, segmentation of rigid bodies in CT/X-rays, and camera pose estimation. Additional discussions on the model performances under different strengths of corruption and extrapolation to soft tissue structures help improve clarity. I still suggest that this work has discussions on structures that are not strictly rigid. Discussions on the reliability of extrapolating to soft tissue structures would be helpful in demonstrating the generalization capacity to diverse structures. Considering the overlapping of anatomical structures on 2D X-rays and image artifacts, the discussions on the effects of segmentation errors of anchor structures and rigid bodies are required to validate the proposed methods.", "rebuttal": "Thank you for the feedback and detailed questions and comments – they are addressed individually below. The question about robustness to segmentation errors led to an interesting ablation study that we will include in the updated manuscript. We are also glad the reviewer appreciated the anatomical motivation behind our method, which leads to a regularization-free framework with strong priors.\n\n> _”The model relied on the assumption that anatomical structures are rigid, being limited to bones with subtle bending or complex joint interactions. Can the proposed model can be extended to soft tissue deformation and non-skeletal contexts?”_\n\nThis may be a potential miscommunication. Our method does not explicitly represent soft tissue deformations. However, soft tissue structures directly contribute to the image-based loss used during registration (Figure 3B) and are thus incorporated into the deformation estimation. To clarify, our only assumption is that bones move rigidly; no rigidity is assumed for other structures. \n\nIn practice, methods that do not have this prior produce unrealistic and suboptimal results in the highly ill-posed setting of CT to X-ray registration (Figure 4), as soft tissue structures are very difficult to distinguish in X-ray. In contrast, our anatomically motivated prior on human motion enables PolyPose to successfully extrapolate to soft tissue structures that were not explicitly modeled and achieve the best overall alignment (Figure 5).\n\nWe do not claim applicability to CT to X-ray registration tasks that do not have identifiable rigid substructures. We will clarify this in the revised discussion, thank you for the clarifying question.\n\n> _”The model required precise segmentation of rigid bodies in the preoperative volumetric images. The segmentation errors can cause inaccurate deformation fields.”_\n\nThis is an important question to investigate – thank you for raising it. To simulate errors typical of automated segmentation tools like TotalSegmentator, we present an experiment below where we systematically erode the ground truth segmentations in the DeepFluoro experiments by increasing the radius of erosion. Here, radius=0mm corresponds to no erosion and is equivalent to the experiment in Table 1 (registration given two X-rays from a limited viewing angle). We find that as we increase the strength of corruption, our method remains robust against extreme corruption and still outperforms the baselines in Table 1 both in terms of Dice and Hausdorff Distance.\n\n| **Erosion Radius** | **Pelvis - Dice (↑)** | **Pelvis - HD95 (↓)** | **Femur (L) - Dice (↑)** | **Femur (L) - HD95 (↓)** | **Femur (R) - Dice (↑)** | **Femur (R) - HD95 (↓)** |\n|--------------------|-----------------------|-----------------------|--------------------------|--------------------------|--------------------------|--------------------------|\n| 0 mm               |       0.99(0.00)      |       1.00(0.00)      |        0.99(0.00)        |        1.02(0.10)        |        0.98(0.00)        |        1.43(0.36)        |\n| 1 mm               |       0.99(0.00)      |       1.00(0.00)      |        0.98(0.00)        |        1.11(0.49)        |        0.98(0.00)        |        1.43(0.46)        |\n| 2 mm               |       0.99(0.00)      |       1.00(0.00)      |        0.98(0.00)        |        1.07(0.24)        |        0.98(0.01)        |        1.66(0.43)        |\n| 3 mm               |       0.99(0.00)      |       1.00(0.00)      |        0.98(0.00)        |        1.15(0.27)        |        0.97(0.01)        |        1.55(0.45)        |\n| 4 mm               |       0.99(0.00)      |       1.01(0.07)      |        0.97(0.00)        |        1.71(0.50)        |        0.97(0.01)        |        1.84(0.42)        |\n| 5 mm               |       0.99(0.00)      |       1.11(0.23)      |        0.95(0.00)        |        3.20(1.25)        |        0.96(0.01)        |        2.28(0.69)        |\n| 6 mm               |       0.99(0.00)      |       1.21(0.34)      |        0.95(0.01)        |        2.89(1.18)        |        0.96(0.01)        |        2.95(1.76)        |\n| 7 mm               |       0.99(0.00)      |       1.41(0.46)      |        0.94(0.01)        |        3.64(1.27)        |        0.95(0.01)        |        3.89(3.08)        |\n| 8 mm               |       0.99(0.00)      |       1.64(0.49)      |        0.93(0.01)        |        5.13(3.19)        |        0.95(0.01)        |        3.70(3.17)        |\n| 9 mm               |       0.99(0.00)      |       1.83(0.61)      |        0.94(0.01)        |        3.47(1.50)        |        0.94(0.01)        |        4.24(3.36)        |\n\nThe results are stable to 4 mm of erosion, beyond which point the accuracy degrades gracefully. However, an erosion radius of just 4mm corresponds to a significant corruption of the original segmentation with its volume decreasing by 40%-60%. For example, the volume of the left femur label decreases from 186,953 mm³ to 117,440 mm³ with an erosion radius of 4 mm.\n\n> _” The manual and semi-automatic segmentation impose additional time complexity.”_\n\nWe respectfully clarify that manual annotation is not needed for our work and for the majority of use cases in radiotherapy and orthopedic surgery. PolyPose only uses bone annotations on CT volumes, which can easily be segmented using off-the-shelf tools such as TotalSegmentator [64]. This automatic process requires only a few seconds per volume without any manual intervention, adding minimal time complexity.\n\n> _”The initial camera poses relied on rigid registration of an anchor structure. The obscured anchor structure can cause pose estimation errors.”_\n\nWe agree that the choice of anchor structure is critical to ensure the success of the initial rigid registration used for preprocessing. In CT and X-ray images, most bones are clearly visible and not obscured in practice due to their higher X-ray attenuation and we only use bones as rigid structures in our formulation. For example, this is why we selected the pelvis for our DeepFluoro experiments, as it is both central and clearly identifiable. We will update the manuscript to include this recommendation for readers.\n\n> _”Tangent space averaging (Eq. 4) assumes transforms combine linearly in se(3). It is unclear whether such computation can handle large-scale transformations or deformations.”_\n\nWe respectfully clarify that, as the tangent space is a Euclidean vector space (and not an embedded manifold like SE(3)), it is valid to linearly combine rigid transforms represented in se(3). As a result, Equation 4 produces diffeomorphisms by construction [26, 51], meaning our method can model everything from a global rigid transform to a large-scale dense deformation field."}, {"review_id": "8i95Wvo5dN", "reviewer": "Reviewer_5y16", "summary": "This paper proposes PolyPose, a novel method for estimating the 3D pose and shape of deformable anatomical structures from sparse 2D X-ray projections. The authors introduce a polyrigid transform model, which decomposes the deformable object into rigid parts and allows independent pose estimation for each part. The method leverages a set of 2D observations and a learned canonical template to localize 3D points via pose-parameterized correspondences. To enable differentiable supervision, the framework integrates a custom X-ray rendering pipeline and a visibility network. The method is validated on human spine and pelvis data from clinical X-ray images and outperforms previous methods in both pose estimation and 3D reconstruction quality.", "questions": "1、Can the method generalize to anatomies without clear rigid substructures? \n2、How does performance change with fewer 2D views? \n3、Are the polyrigid parts interpretable in anatomical terms? 4. Could prior knowledge or statistical shape models help?\n4、Could prior knowledge or statistical shape models help?\n5、Anatomical plausibility of the weighting function: Is the assumption of uniform bone mass reasonable?", "limitations": "The paper briefly acknowledges the potential limitation on generalization and requirement of predefined parts, but lacks a full discussion on modeling soft-tissue deformations, dependence on number of X-rays, or robustness to noise/occlusion.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- Innovative approach:Introducing multi-rigid body transformation into 2D/3D registration and Use anatomical priors to reduce the number of parameters for optimization degrees of freedom.\n- Methodology: Differentiable X-ray renderer and visibility network are valuable contributions. \n- Empirical results: Strong benchmarks and ablation studies. \n- Clinical relevance: Supports low-dose imaging workflows.\n\nWeaknesses:\n- Scalability: Generalization to other anatomies (especially soft tissue) is untested.\n- Manual part annotation required.\n- Limited dataset and anatomical diversity.\n- Real-time performance is not fully evaluated.", "was_revised": true, "final_justification": "I think the quality of the article is quite good, which basically solves my doubts. I have decided to maintain my score.", "rebuttal": "We thank the reviewer for their feedback and thoughtful questions! Their feedback and the responses below will be integrated into the manuscript. We are glad they found our approach to this clinically relevant problem to be innovative and well-supported by strong experimental results.\n\n> _”Weakness: Generalization to other anatomies (especially soft tissue) is untested.”_\n\nThis is a potential miscommunication. We specifically test generalization to soft tissue structures in our experiment on the Head&Neck cancer dataset (Figure 5). These experiments demonstrate that our method achieves state-of-the-art accuracy in the localization of soft tissues. We will clarify the text to emphasize this point further.\n\n> _”Weakness: Manual part annotation required.”_\n\n\nWe respectfully clarify that manual annotation is not needed for our work. PolyPose only needs bone annotations from CT volumes, which are easily obtained using off-the-shelf tools such as TotalSegmentator [64]. Segmentation can thus be performed automatically, requiring only a few seconds per volume without any manual intervention. For example, our experiments on the Head&Neck cancer dataset only use TotalSegmentator for annotating rigid bodies. \n\n> _”Weakness: Limited dataset and anatomical diversity.”_\n\n\nWe respectfully posit that our evaluation datasets come from highly disparate regions of the body with no overlap, such as the pelvis and the head-neck-shoulder complex in the DeepFluoro and Head&Neck cancer datasets, respectively. \n\n> _”Weakness: Real-time performance is not fully evaluated.”_\n\nThank you for raising this. We report the runtimes of all methods evaluated in our paper on the Head&Neck cancer dataset in the table below and will include this information in the paper. While PolyPose is slower than the inference-only LiftReg and 2D3D-RegNet methods, it requires no training and is much more accurate. Runtimes and Dice metrics are averaged over all numbers of views (2-9) evaluated in Figure 5. All methods are evaluated on a single NVIDIA RTX A6000 GPU.\n\n|                  | **Dice (Rigid Bodies)** | **Dice (Soft Tissues)** | **Runtime (min:sec)** |\n|------------------|:-----------------------:|:-----------------------:|:---------------------:|\n| PolyPose         |        0.71(0.15)       |        0.76(0.08)       |       0:59(0:20)      |\n| FireANTS         |        0.39(0.13)       |        0.52(0.17)       |       0:41(0:08)      |\n| anatomix         |        0.57(0.14)       |        0.66(0.13)       |       1:05(0:12)      |\n| multiGradICON    |        0.03(0.02)       |        0.09(0.03)       |       0:28(0:03)      |\n| multiGradICON+IO |        0.20(0.14)       |        0.32(0.17)       |       1:10(0:08)      |\n| 2D3D-RegNet      |        0.58(0.12)       |        0.67(0.09)       |       0:01(0:01)      |\n| LiftReg          |        0.47(0.13)       |        0.57(0.12)       |       0:02(0:01)      |\n| DiffPose         |        0.53(0.15)       |        0.61(0.13)       |       0:06(0:02)      |\n\n> _”Question: Can the method generalize to anatomies without clear rigid substructures?”_\n\nThis is an important question. In our experiments on the Head&Neck dataset, our method accurately generalizes to and aligns soft tissue structures without explicitly modeling them. We do not claim that our solution applies to the extreme scenario where there are no skeletal structures available to constrain the deformation field. We will further emphasize this in the discussion.\n\n> _”Question: How does performance change with fewer 2D views?”_\n\nWe conducted this experiment in Figure 5, which shows that PolyPose’s performance remains robust as the number of 2D views/X-rays decreases. Compared to the baselines, our method has the highest accuracy across all numbers of views for both rigid and soft tissue structures.\n\n> _”Question: Are the polyrigid parts interpretable in anatomical terms?”_\n\nThis is an important advantage of our method that we did not emphasize in the initial submission. Thank you for raising it! Yes, the polyrigid parts in our work are the bones in the patient’s skeleton, which leads to the estimated deformation fields being interpretable in anatomical terms by construction. For example, in Figure 2 (right), each displacement vector is colored by the relative contribution of the rigid movement of each bone. One can observe that the estimated warp field is *locally-linear*, with the deformation vectors being most influenced by the nearest rigid structures (Figure 2). \n\n> _”Question: Could prior knowledge or statistical shape models help?”_\n\nWe agree that prior knowledge helps our problem context. To re-emphasize, our model already incorporates prior knowledge through the biological constraint that individual bones do not bend in typical motion. As a result, our polyrigid deformation model automatically enforces anatomically plausible priors that respect the piecewise rigid nature of human movement. This is why our method does not require any explicit regularization in such an ill-posed and underdetermined setting. Statistical shape models could also be potentially integrated in future work, but we found volumetric bone segmentations to be easy to obtain and to provide robust results in our framework. We would be happy to discuss any alternative priors or parameterizations the reviewer is interested in during the discussion period.\n\n> _”Question: Anatomical plausibility of the weighting function: Is the assumption of uniform bone mass reasonable?”_\n\nWithin the context of registration, we believe that assuming uniform density within an individual bone is a reasonable approximation. While mineral density can change subtly across subregions of a bone, this variation is not well characterized in the literature and would require specialized imaging, such as a DEXA scan for each subject. As demonstrated by the experiments, our simplifying assumption is robust across multiple anatomical regions and leads to large improvements over existing baselines."}]

Author rebuttal:

Meta-review: Summary: Approach to model deformable CT to X-ray registration by weighted composition of locally rigid bone transforms.  Experiments show SOTA alignment starting with as few as two views.

Strengths: Strong anatomical prior to regularize an ill-posed task.  Gives consistent gains over alternative methods on head-neck and pelvis.  Approach has fast runtime.

Weaknesses: Needs accurate bone segmentations and visible skeletal anchors; therefore not applicable to MRI X-ray alignment. or purely soft-Case of severe metal artifacts/anchor occlusion is partly discussed and not fully validated.

Accept: well-justified method with clear empirical wins in sparse/limited-angle settings and added robustness analyses.

(e) Rebuttal & Discussion: Reviewers asked about soft-tissue generalization, segmentation errors, occlusions/metal artifacts, and runtime; authors added ThoraxCBCT results, a segmentation-corruption study, runtime table, and clarified scope, after which all reviewers maintained positive ratings.  Strong consensus for acceptance from reviewers and myself.
Decision: Accept (poster)