<figure><img src="image_1.png" /></figure>

HAL
open science

# Domain agnostic 2D-3D deformable registration Application to fluoroscopic guidance without contrast agent

François Lecomte-Denis, Juan Verde, Jean-Louis Dillenseger, Stéphane Cotin

► To cite this version:

François Lecomte-Denis, Juan Verde, Jean-Louis Dillenseger, Stéphane Cotin. Domain agnostic 2D-3D deformable registration Application to fluoroscopic guidance without contrast agent. Medical Image Analysis, 2025, 105, pp.103688. 10.1016/j.media.2025.103688. hal-05132987v2

HAL Id: hal-05132987

https://hal.science/hal-05132987v2

Submitted on 11 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Domain agnostic 2D-3D deformable registration
Application to fluoroscopic guidance without contrast agent

François Lecomte-Denis¹,*, Juan Verde², Jean-Louis Dillenseger³, Stéphane Cotin¹

¹Inria, Strasbourg, 67000, France

²IHU Strasbourg, 67000, France

³LTSI - UMR 1099, University of Rennes, Inserm, Rennes, 35000, France

*Corresponding author: francois.lecomte@inria.fr

**Abstract**

We present a method for estimating, in real time, a 3D displacement field from a single fluoroscopic image. Our approach uses a fully convolutional network architecture to solve the associated inverse problem. Supervised learning is performed on synthetic data, using Digitally Reconstructed Radiographs as input and displacement fields as output. We use randomized Gaussian kernels to produce a synthetic training dataset with displacement fields that are smooth and diffeomorphic. In contrast to other 2D-3D registration methods, our novel data generation approach does not rely on a statistical motion model. This enables our model to accurately predict deformations unrelated to breathing or other predetermined motion patterns. As an example of clinical application, we show that our model is able to predict deformations related to percutaneous needle insertions accurately, potentially removing the need for contrast agent injection.

**Keywords:** 2D-3D registration; Image-guided intervention; contrast agent; liver; deformation; deep learning; fluoroscopy; X-ray; CT.

# 1 Introduction

The liver, a complex solid organ, houses three distinct vascular systems and the biliary tree. It is unfortunately targeted by a wide variety of diseases, including tumors, both primary ( affecting the liver itself) and secondary (resulting from diseases in other organs). The evolution of image-guided techniques has significantly improved diagnostic, theragnostic, and therapeu- tic procedures ([1]). These new techniques have also reduced invasiveness and enriched the information available for guiding the interventions.

Liver therapies guided by imaging techniques can be categorized into three primary groups of procedures: those targeting the liver parenchyma, those involving vascular structures, and those related to the biliary tree. The most performed procedures for each group are; for the **parenchyma**: liver or tumor biopsy, and tumoral ablation; for the **vessels**: portal or hep- atic vein embolization, transjugular intrahepatic portosystemic shunt (TIPS), and transarterial chemoembolization (TACE); and for the **biliary system**: cholangiography, cholecystostomy, biliary drainage, endocanalicular biopsy, and stent placement.

X-ray-based imaging is one of the most frequently used imaging modalities for guiding the above-mentioned liver therapies. It includes CT, 2D fluoroscopy, and 3D fluoroscopy (i.e.,
cone-beam CT). However, low tissue contrast is a prominent limitation of X-ray-based imaging, requiring the injection of contrast agents (CA) into the bloodstream to enhance contrast between structures, enabling recognition, and thus, facilitating the procedures. During procedures, the liver is subject to deformations caused by cardiac and breathing motion, and the instrumentation performed by the clinician. This means that, in the absence of contrast injection, their position becomes indeterminate, with a lack of context, leading to risks of unintended damage. However, the volume of CA that can be injected during an intervention is limited due to its potential toxic effects when injected intravascular, notably nephrotoxicity ([2]). Furthermore, the rapid dissipation of the contrast effect after injection makes the visualization of low-contrast tissues only possible shortly after injection. Consequently, while fluoroscopy offers real-time capabilities, this transient contrast effect renders fluoroscopy most of the time an asynchronous image guidance method, available either before or after any gesture.

Therefore, the objective of this work is to predict the motion of intrahepatic structures (e.g., vessels, tumors, liver segments) in fluoroscopy-guided interventions without the need for CA injection. It is a challenging task due to the poor contrast between the structures and the surrounding tissue in fluoroscopic images. This task is further complicated in clinical settings where organs are deformed due to surgical manipulations or needle-based (percutaneous) interventions. Thus, we propose a *domain-agnostic* 2D-3D deformable registration method able to recover *arbitrary deformations* of the anatomy from a single fluoroscopic image. Our approach is based on the generation of a randomized, synthetic dataset to train a neural network that predicts deformations. Using a preoperative CT scan as a prior to generate a bespoke training dataset, our approach has the potential to provide real-time guidance in fluoroscopy-guided interventions while avoiding contrast agent injection, thus enhancing the safety and efficiency of procedures.

## 2 Previous works

Outside of the medical field, extracting 3D information from one or several 2D projective images is a widely encountered problem. Some works adopt a global approach, aiming to recover a 3D scene from a single 2D optical image ([3]). Other works focus on reconstructing a 3D mesh of an object from one or more 2D projections ([4–6]).

In the medical field, 3D CT and MRI scans are ubiquitous as they bring crucial information to clinicians. When two 3D CT scans are available, 3D-3D registration can be employed to compare the two scans and identify the differences between them, bringing additional information to clinicians. In [7], the authors proposed Quicksilver, a deep learning-based method to accelerate traditional Large Deformation Diffeomorphic Metric Mapping (LDDMM) registration. Instead of obtaining LDDMM parameters via optimization, Quicksilver directly computes these parameters from a pair of images using a neural network, speeding up registration by several orders of magnitude. In [8], the authors introduced an unsupervised learning-based registration framework, where a Convolutional Neural Network (CNN) predicts a dense displacement field from two images, along with a diffeomorphic variant ensuring topology preservation. To better handle large 3D image volumes, [9] proposed a novel coarse-to-fine optimization strategy, iteratively refining the registration transform from a low to high resolution. In [10], Hoffmann et al. advanced the field by employing synthetic data generation to enable contrast-invariant registration without real training images or displacement fields. Instead, randomized images and displacement fields are used as training data, where the displacement fields are obtained by integrating a random stationary velocity field.

Relying on CT images is, however, not possible in a number of medical interventions for two main reasons: a CT scan induces a non-negligible radiation dose for the patient, and its acquisition is not in real-time. To address these limitations, a variety of works ([11–15])
tackle the problem of 3D CT reconstruction from one or few 2D fluoroscopic images. Since reconstructing a 3D image from a few projections is an ill-posed problem, some of these works rely on a previously acquired CT image. These approaches are, however, not adapted to clinical scenarios where preoperative tumor or vessel segmentations are available. They reconstruct an (unsegmented) CT volume from scratch instead of updating existing segmentations, which would be useful to visualize during interventions.

In a related application domain, radiotherapy faces similar challenges, when the position of a tumor from the preoperative CT scan must be updated to follow the breathing motion of the patient. This problem has been solved with success by tracking an implanted radio-opaque marker near the tumor in a bi-plane fluoroscopic image while the patient breathes ([16, 17]). However, marker implantation requires an additional procedure, potentially leading to complications. For this reason, recent works aim to directly predict the tumor location from two or more fluoroscopic images ([15, 18, 19]).

Another, more general approach is to predict a displacement field to update the position and shape of preoperatively segmented structures. Like our approach, such methods are referred to as 2D-3D deformable registration methods. We refer the interested reader to the following surveys for a more exhaustive list of 2D-3D registration methods [20, 21]. In this category, some methods use several fluoroscopic images to recover breathing motion ([12, 22–24]), while others predict breathing motion using only a single fluoroscopic image ([25–29]). All of these methods demonstrate clinically acceptable target localization accuracy in the context of breathing motion prediction, yet they are restricted to known motion patterns.

Finally, it is important to mention the importance of rigid registration as an initial step of the registration process. In rigid registration, the objective is to find a transformation matrix to align the preoperative and intraoperative images. In the case of 2D-3D registration, this transformation is often referred to as the *pose* of the fluoroscopic acquisition device with respect to the CT scan. Recent advances in deep learning-based 2D-3D rigid registration include DiffPose ([30]), which introduces a self-supervised framework leveraging differentiable X-ray rendering. This method achieves sub-millimeter accuracy for rigid registration by combining a patient-specific ResNet-based CNN pre-trained on synthetic X-rays with test-time optimization. For 2D-3D deformable registration, DiffPose would be used once, after positioning the fluoroscopic imager, to retrieve its pose with respect to the preoperative CT scan, before proceeding with deformable registration.

While previous works, particularly single-plane methods, have demonstrated the feasibility of 2D-3D fluoroscopy to CT real-time registration in radiotherapy, a critical gap remains in addressing arbitrary deformation recovery in other interventional procedures. These deformations result not only from respiratory motion but also from the mechanical influence of surgical instruments on tissue, leading to complex and unpredictable changes. Traditional approaches for tracking anatomical structures in fluoroscopy-guided interventions have relied on markers or contrast agents, posing limitations and potential risks. Meanwhile, all existing markerless 2D-3D tracking methods are based on pre-established motion models, suitable only for scenarios where the movement, such as periodic breathing, is known in advance. Thus, no solution exists for contrast-free, real-time tracking of anatomical structures in fluoroscopy-guided interventions.

This work directly addresses this unmet need by introducing a novel, domain-agnostic method for single-view 2D-3D deformable registration. Our key contributions are threefold. First, we propose a novel registration method able to recover deformations commonly encountered in fluoroscopy-guided interventions, moving beyond the limitations of existing methods focused on respiratory motion. In this aspect, our main contribution is the domain agnostic data generation method (Sec. 3.2), which eliminates the need for prior knowledge of motion during the procedure. Second, we developed and evaluated a novel back projection module for
2D-3D registration that transforms 2D features into 3D features while considering projective geometry (Sec. 3.3.1). Third, we propose a novel loss combination (Sec. 3.3.2), specifically designed for 2D-3D deformable registration, where the network is supervised in projective space rather than anatomical space, leading to improved performances.

We validated the robustness and versatility of our method on an experimentally acquired porcine dataset with operator-induced deformations and a synthetic human dataset with breathing motion. Additionally, we performed a sensitivity analysis to evaluate the impact of the backprojection module and loss function on performance. Our results underscore the clinical relevance of our method, demonstrating real-time, contrast-free tracking of intrahepatic vessels, which could reduce the need for contrast agents in fluoroscopy-guided interventions. With this approach, we aim to contribute to the development of safer, more broadly applicable markerless and contrast-free tracking solutions for fluoroscopy-guided interventions.

# 3 Method

## 3.1 Overview

Our framework is based upon the most common steps of fluoroscopy-guided interventions, outlined in Fig. 1. First, a preoperative CT image is acquired, and structures of interest are segmented, from which the clinicians can plan the intervention. We assume that the pose of the C-arm with respect to the patient is determined during this step. During the intervention, the C-arm is positioned as planned and is used for intra-operative image guidance.

In the context of image-guided liver therapies, contrast agents are injected before acquiring the preoperative CT. From the preoperative Contrast Enhanced CT scan (CECT), the vessel tree is segmented to obtain a 3D segmentation volume. Using the CECT and the planned C-arm pose, we generate a synthetic training dataset composed of synthetic deformations and synthetic fluoroscopic images, as described in Sec. 3.2 and as shown in Fig. 1.

The neural network is then trained on the synthetic dataset (see Sec. 3.3). Next, during the intervention, a fluoroscopic image of the patient is acquired without injecting a contrast agent. The fluoroscopic image is processed in real time by the network to compute the deformation between the preoperative CT scan and the intraoperative anatomy. The deformation computed by the network is used to warp the preoperative vascular tree. Finally, the warped tree can be projected on top of the fluoroscopic image (i.e., augmented fluoroscopy) using ray-casting techniques, to obtain a result similar to the image in the lower right part of Fig. 1.

## 3.2 Data generation

To train the network to estimate a deformation $\phi$ from a non-contrasted fluoroscopy $p$, we generate a training dataset composed of pairs of synthetic $\phi_i$ and $p_i$. To generate synthetic non-contrasted fluoroscopic images, we first transform the preoperative Contrast Enhanced CT image $I_{CE}$ into a non-contrasted CT image $I$ via inpainting ([31]). This operation preserves the information outside the segmentation and removes as much contrast information as possible. We can then generate deformed, non-contrasted CT images $I'_i$ from $I$.

For each sample of the dataset, the process goes as follows: The deformation $\phi_i$ is generated using a sum of randomized Gaussian kernels (Sec. 3.2.1). Then, the deformed CT image $I'_i$ is generated from $I$ and $\phi_i$ (Sec. 3.2.2). Finally, following (4), we generate synthetic fluoroscopic images $p_i$ from $I'_i$. (Sec. 3.2.3).
<figure><img src="image_3.png" /><figcaption>Figure 1: Overview of the proposed method. First, the intervention is planned from a 3D CT scan of the patient, where structures of interest are segmented, and the C-arm pose is determined (top left). Second, the neural network is trained on non-rigid deformations of the 3D CT and synthetic fluoroscopic images (top, middle, and right). Here, non-rigid deformations are represented schematically in 2D (in reality, deformations are 3D vector fields), with color to indicate the amplitude of the displacement. Third, during the intervention, the C-arm is positioned, and a fluoroscopic image is acquired (bottom left). Fourth, the network computes the deformation from the fluoroscopic image (bottom middle) and the warped vessel tree is used to augment the fluoroscopy (bottom right).</figcaption></figure>
### 3.2.1 Deformation generation

A non-rigid deformation is defined on the 3D image $I(\mathbf{x})$ by $\phi(\mathbf{x}) : \mathbb{R}^3 \to \mathbb{R}^3$ with $\mathbf{x}$ a point in the image volume and $\phi(\mathbf{x}) = \mathbf{x} + \varphi(\mathbf{x})$, with $\varphi(\mathbf{x})$ a displacement vector field. We restrict the region where $\phi$ is defined to a sub-region of the volume, which will be referred to as the field domain. The key characteristics we seek in the displacement field are smoothness and invertibility. A good candidate for producing such displacement fields is the LDDMM framework ([32]), which demonstrated very good performance in non-rigid registration problems ([33]). In this framework, the non-rigid deformation $\phi$ that registers an image $I$ to an image $I'$ is obtained by integrating a velocity field $V(t, \mathbf{x})$ over time, following a set of differential equations to drive the evolution of $V(t, \mathbf{x})$.

The authors demonstrate that $V(t, \mathbf{x})$ can be expressed as:

$$V(t, \mathbf{x}) = \sum_{k=1}^{N_{cp}} \alpha_k(t) K_k(\mathbf{x}, \mathbf{y}_k(t)) \quad (1)$$

where $K_k(t)$ are elements of a Reproducing Kernel Hilbert Space, such as Gaussian kernels, located at the $N_{cp}$ control points $\mathbf{y}_k \in \mathbb{R}^3$ and associated with weights $\alpha_k \in \mathbb{R}^3$. The Displacement Vector Field (DVF) $\varphi$ is then given by $\varphi(\mathbf{x}) = \int_0^1 V(t, \mathbf{x}) dt$.

In our framework, we instead directly compute the DVF $\varphi(\mathbf{x})$ by randomizing the control points $\mathbf{y}_k$, covariance matrices $\sigma_k \in \mathbb{R}^{3 \times 3}$ and weights $\alpha_k$ of the Gaussian kernels.

$$\varphi(\mathbf{x}) = \sum_{k=1}^{N_{cp}} \alpha_k \odot K(\mathbf{x}, \mathbf{y}_k, \sigma_k) \quad (2)$$

To sample $\mathbf{y}_k$, we generate a set of random points in the field domain. We discard points that are within a threshold distance $\Delta_y$ of each other to prevent sharp variations in $\varphi$, and re-generate rejected points until the desired number of control points is obtained. $\alpha_k$ are sampled from a 2D uniform distribution and then multiplied by a random common scaling factor between 0 and 1 to ensure samples with small overall displacements are represented in the dataset. Finally, $\sigma_k$ is generated as $N_{cp} \times 3 \times 3$ i.i.d variables with values between 15% and 30% of the size of the field domain. While this new formulation does not preserve the diffeomorphic nature of $\phi$ by construction, we can compute its spatial Jacobian $J$ and verify that it is positive, and if not, regenerate $\phi$, thus ensuring diffeomorphic deformations.

### 3.2.2 Image warping

As stated above, we model the non-rigid deformations of the anatomy as coordinate transforms $\phi(\mathbf{x})$. The warped CT image $I'(\mathbf{x}) = I \circ \phi(\mathbf{x})$ is obtained by linearly interpolating the values of $I$ at $\mathbf{x}' = \phi(\mathbf{x})$:

$$I'(\mathbf{x}) = \sum_{\mathbf{z} \in \mathcal{Z}(\mathbf{x}')} I'(\mathbf{z}) \prod_{d \in \{0,1,2\}} (1 - |\mathbf{x}'_d - \mathbf{z}_d|) \quad (3)$$

where $\mathbf{z}$ are the 8 voxels nearest to $\mathbf{x}'$ and $d$ iterates through the 3 spatial components of $\mathbf{x}'$ and $\mathbf{z}$. This operation is known as backward warping.

### 3.2.3 Digitally Reconstructed Radiographs

We generate Digitally Reconstructed Radiographs (DRR) using the DeepDRR framework ([34]).

This framework models the C-arm as a pinhole camera, parameterized by a projection matrix $\mathbf{P}$ composed of an intrinsic matrix $\mathbf{H} \in \mathbb{R}^{3 \times 3}$ and an extrinsic matrix $\mathbf{E} \in \mathbb{R}^{3 \times 4}$. $\mathbf{E}$ is obtained from the planned pose of the C-arm and $\mathbf{H}$ is obtained from the characteristics of the
C-arm detector panel. In the DeepDRR framework, the fluoroscopic image $p$ observed during the intervention is approximated by:

$$p(\mathbf{u}) \approx \int I'(\mathbf{x})d\mathbf{l}_{\mathbf{u}} \qquad (4)$$

With $l_{\mathbf{u}}(\mathbf{x}) = \mathbf{P} \cdot \mathbf{x}$ the ray connecting the point $\mathbf{u} \in \mathbb{R}^2$ on the detector plane to the emission source.

Eqn. 4 shows that $p(\mathbf{u})$ is invariant to shifts of the distribution of $I(\mathbf{x})$ along the path of the ray, as long as the integral of $I(\mathbf{x})$ is preserved. This means that displacements collinear to the projection rays cannot be directly observed in the projection image, since such displacement do not incur changes in $\int I'(\mathbf{x})d\mathbf{l}_{\mathbf{u}}$, leaving the intensity of the projected image unchanged. A direct consequence of this result is that, in fluoroscopic images or DRRs, it is impossible to recover displacements along the direction of projection.

## 3.3 Network architecture

Our architecture builds upon ResNet ([35]), drawing inspiration from its success in volume reconstruction from a 2D projection ([11]). We incorporated a novel backprojection module that takes the projection geometry into account when transforming 2D features to 3D, and modified the decoder to predict a 3D displacement field instead of a 3D image.

The key characteristic of our fully convolutional network is the direct translation from a dense 2D input to a dense 3D output (see Fig. 2). The encoder first produces 2D feature maps from the input fluoroscopy. Then, the backprojection module employs the projection matrix $\mathbf{P}$ to map the 2D features from input space to output 3D anatomical space (detailed in Sec. 3.3.1). Finally, the decoder transforms these 3D feature maps into the final displacement field $\hat{\mathbf{\phi}}$.

Critically, the backprojection module performs the 2D-to-3D feature translation at the end of the encoder. This strategic placement occurs after the encoder has extracted rich semantic information from the 2D input. This minimizes the semantic gap between 2D features extracted from the fluoroscopy and 3D features that will be decoded into a 3D displacement field in anatomical space. In contrast with alternative architectures such as U-net [36] or FPN [37], which establish connections between early encoder layers and late decoder layers, our architecture avoids such connections. Indeed, these connections would have to bridge a large semantic gap when attempting to combine 2D features close to input space with 3D features close to output space, as input and output spaces present important semantic differences.

In the encoder, the first layer transforms the grayscale 256 × 256 input image into feature maps of shape 96 × 128 × 128. Then, five ResNet blocks composed of two convolutional layers, each followed by a BatchNorm and a PReLU activation, process the data, with each block dividing the spatial resolution and multiplying the number of features by two. Finally, the last convolutional layer is applied without changing the number of features or resolution, to obtain 2D feature maps of shape 1536 × 8 × 8. After the encoder, the backprojection module transforms the features from 2D projective space to 3D anatomical space, as described in Sec. 3.3.1, to obtain 3D features of shape 768 × 4 × 2 × 4, conserving the total number of features in the process. In the decoder, 10 deconvolutional layers are employed to decode these 3D features into a 3D displacement field of shape 3 × 128 × 64 × 128. These layers are not set up in ResNet blocks, with the number of features and spatial resolution being halved and doubled every two layers, respectively, and the last layer transforming the 24-dimensional feature map into a 3-dimensional displacement field.

During training, a reprojection loss $\mathcal{L}_{\phi}^{2D}$ and a 2D dice loss $\mathcal{L}_{s}^{2D}$ are used to optimize the network parameters in a supervised manner, as described in Sec. 3.3.2.

The network is implemented in PyTorch 2.4.1, with a memory footprint of ~ 335 MB. A forward pass is computed in 12 ms on an Nvidia RTX 4090 GPU. The Adam algorithm ([38])
was used to optimize the network weights. The network was trained for 5 epochs, using the One Cycle learning rate schedule ([39]) to vary the learning rate $\eta$, with $\eta_{max} = 5.10^{-3}$. Training the network for more epochs (and a proportionally larger scheduler step size) did not lead to increased performance.

<figure><img src="image_4.png" /><figcaption>Figure 2: The encoder part of the network compresses the image into a low-resolution, high-dimensional feature space. The 2D features are backprojected to 3D using the projection matrix in the backprojection module. In the decoder, the network upscales the feature maps into a 3D displacement field. After the network, the loss is computed on the displacement of grid points projected in 2D space and on the warped vessels segmentation, projected in 2D space.</figcaption></figure>

### 3.3.1 Backprojection module

Inspired by the LiftReg network ([24]), we designed a backprojection module to handle 2D-3D spatial correspondence. An important difference between the LiftReg network and our approach is that we backproject the feature maps instead of backprojecting the input image. This largely reduces the number of parameters and inference time of the network because the encoder is composed of 2D convolutional layers instead of 3D convolutional layers.

Depending on the pose of the C-arm, the input projection image may be oriented arbitrarily with respect to the preoperative CT volume. The goal of this module is to ensure that the value of the displacement field at a voxel $\mathbf{x}$ will be determined by the value of the pixel at the coordinate $\mathbf{u} = \mathbf{P}\mathbf{x}$.

The backprojection module (in green in Fig. 2) contains no trainable parameters and relies on the projection matrix $\mathbf{P}$, obtained from the pose of the C-arm camera, to sample 3D features from 2D features via trilinear interpolation. The 2D feature map $\mathbf{F}$ is a tensor of shape $(N, C, H, W)$, which can be treated as a batch of $N$ 3D images with $(C, H, W)$ dimensions. Since convolutions are approximately local operations, the $(H, W)$ dimensions span the same spatial extent as the input 2D image. The channel dimension $C$ is treated as a pseudo depth dimension, where each channel represents a different image plane. This is the first step in Fig. 3. It is important to note that these 3D features are not in the same space as the preoperative 3D anatomical image, because the input of the network is a perspective projection of the anatomy.
Thus, these features must be mapped to the preoperative anatomical space by performing a backprojection operation. First, a regular grid of 3D points G is sampled in the bounding box of the region covered by the projection. This is the second step in Fig. 3. The grid G, of shape (3, C, H, W), can be converted to 2D coordinates using the projection matrix P by computing G<sub>u</sub> = PG. To index the pseudo-depth dimension C of F, the first coordinate of G<sub>u</sub> is set to an array of values varying from 0 to 1. along C. Finally, F can be interpolated using G<sub>u</sub> to transform the 2D projective features into 3D features F<sub>3D</sub> aligned with the preoperative anatomical space. Elements of F<sub>3D</sub> outside of F are simply set to 0 since they are not visible in the projection. This is the last step in Fig. 3. F<sub>3D</sub> is further transformed by splitting the first dimension into a spatial and a channel dimension, to form a 5D feature map of shape (C<sub>3D</sub>, D, H, W). After the backprojection, F<sub>3D</sub> is input in the decoder (in red in Fig. 2) to predict a displacement field in the same space as the preoperative anatomy.

<figure><img src="image_5.png" /><figcaption>Figure 3: The 2D feature maps are reshaped into a volume in ray space. The volume is then sampled using ray space coordinates to obtain a volume in voxel space. Volume elements that are outside of the projection are set to zero.</figcaption></figure>

3.3.2 Loss computation

Before computing the loss, we first mask the prediction and ground truth to exclude locations
that are not visible in the image or outside the body. Because of the finite size of the detector,
p is restricted to the domain Ω<sub>p</sub>, defined by the size of the C-arm detector. This implies that
the network can only predict φ̂ at the points x that have a projection in the 2D domain Ω<sub>p</sub>:
{ x | ∃t ∈ ℝ, ∃u ∈ Ω<sub>p</sub>, tPx + x<sub>0</sub> = u } with x<sub>0</sub> the origin of the C-arm camera matrix. Thus,
we restrict the computation of L to the voxels that have a projection in Ω<sub>p</sub>. Furthermore,
the displacement at locations outside the body cannot (and should not) be recovered by the
network due to the lack of contrast at those voxels, which typically contain air. Thus, we build
a body mask using TotalSegmentator ([40]), and warp it with the ground truth displacement
field to mask out locations that are not in the deformed body.

We formulate the loss function $\mathcal{L}$ as combination of two losses:

$$
\mathcal{L} = \mathcal{L}_{\varphi^{2D}}(\varphi_i^{2D}, \hat{\varphi}_i^{2D}) + \lambda \mathcal{L}_{s^{2D}}(\mathcal{P}_{2D}(I_s \circ \phi_i), \mathcal{P}_{2D}(I_s \circ \hat{\phi}_i)) \quad (5)
$$

$\varphi_i^{2D}$ and $\hat{\varphi}_i^{2D}$ are the ground truth and predicted displacements, respectively, projected in the image plane. $\mathcal{P}_{2D}(I_s \circ \phi_i)$ and $\mathcal{P}_{2D}(I_s \circ \hat{\phi}_i)$ are the ground truth and predicted segmentations, projected in the image plane. $\mathcal{L}_{\varphi^{2D}}$ is the Mean Square Error (MSE) on the projected displacement, and $\mathcal{L}_{s^{2D}}$ is the Soft Dice loss ([41]) on the projected segmentation.

To compute $\mathcal{L}_{\varphi^{2D}}$, a 3D grid of points defined on the field domain is first warped with the predicted and ground truth displacement fields. Then, the warped 3D points are projected using $\mathbf{P}$ to obtain warped 2D points. Finally, $\mathcal{L}_{\varphi^{2D}}$ is computed between the ground truth and predicted 2D points.
To compute $\mathcal{L}_{s^{2D}}$, the vessels segmentation is warped with the predicted and ground truth displacement fields. Then, the fully differentiable renderer DiffDRR ([42]) is used to obtain 2D projections of the warped segmentations. Finally, $\mathcal{L}_{s^{2D}}$ is computed between the ground truth and predicted 2D segmentations.

This novel combination of loss in 2D image space is better suited to the clinical objective of visualizing 3D preoperative data registered on 2D intraoperative images. In contrast, supervising the network in 3D anatomical space does not take into account the fact that there is a loss of information in the 2D image formation process (see Sec. 3.2.3). Furthermore, our approach specifically supervises the network to predict deformations that minimize displacement errors rather than image intensity errors. This design choice ensures that all anatomical structures are weighted equally in the loss function, regardless of their intensity values, preventing high-contrast structures (such as bones) from dominating the optimization over lower contrast structures (such as soft tissues). Our experimental results showed consistent performance across all anatomical regions, indicating that this approach is effective in training the network to predict deformations everywhere in the anatomy, independently of contrast variations. The performance impact of this loss over alternative formulations is studied in Sec. 4.6.1.

## 3.4 Data augmentation

To improve the network robustness to changes in input image appearance unrelated to deformations, such as the ‘sim-to-real’ domain gap, we used a data augmentation method described in ([43], post-processing section). This data transformation is a composition of perturbations, each successively applied at random to the input with a probability of 50%. Perturbations include noise addition, smoothing, and contrast change. In ([43]), authors demonstrate that this approach enables a 2D-3D rigid registration network trained on DRRs to perform well on real fluoroscopic images. In this work, we find that this data augmentation scheme improves the generalization capabilities of the network (see Sec. 4.6.3).

# 4 Results

We present here several results to demonstrate the robustness, accuracy, and genericity of our method.

## 4.1 Datasets description

Our method was evaluated on three datasets:

1. A clinically acquired porcine dataset, with four pairs of CECTs, before and after intervention-related deformations.

2. A breathing motion dataset, composed of a baseline, clinically-acquired preoperative CECT, and 50 deformed CTs replicating 5 breathing motion sequences.

3. A needle insertion dataset, composed of the same baseline CECT as the breathing motion dataset and 50 deformed CTs mimicking deformations induced by needle insertion.

These datasets are designed to evaluate the ability of our method to recover deformations that would occur during interventions, either due to breathing, or caused by the interaction between surgical tools and tissues. The testing data inputs are DRRs generated from the deformed CTs, with the deformed vessel segmentations serving as ground truth.
With the first dataset (the IHUdeLiver10 dataset), in Sec. 4.3, we evaluated our method on DRRs generated from CT scans acquired after interventions. In our experiments, we used four pairs of {baseline; deformed} CECTs experimentally acquired on four different porcine subjects, to illustrate the flexibility of our approach to recover various intervention-related deformations. Specifically, the four samples we used represent the following interventional scenarios: needle-tissue interactions, inter-fractional motion and laparoscopic surgery, respectively. As opposed to other publicly available datasets, this dataset allows us to evaluate the ability of our method to recover real, intervention-related deformations. A limitation of this dataset is that, due to contrast agent dissipation, the deformed intrahepatic vessels do not match the rest intrahepatic vessels in the number and length of vessels. We mitigate this issue by manually processing the segmentations to obtain vessel trees as similar as possible.

To completely eliminate this uncertainty, we further validate our method on a clinically acquired human CECT, with synthetic deformations (Sec. 4.4 and Sec. 4.5). We created two datasets to evaluate our method on both naturally occurring and intervention-related deformations. The first dataset contains five 10-phase synthetic 4DCT, replicating breathing motion. With this dataset, our objective is to show that our method can compete with other works specifically designed for breathing motion recovery. The second dataset contains a simulated needle insertion over the course of 50 time steps. This dataset allows us to perform an accurate, quantitative assessment of our method on intervention-related deformations without experimental sources of errors. The process of generating these datasets is described below.

**Breathing motion dataset:** To create a semi-synthetic test dataset that represents breathing motion, we used a data generation process different from the training data generation process. First, the thorax bones and liver were segmented automatically using TotalSegmentator ([40]). A 3D grid of control points was also created, covering the image at a resolution of 20 mm in every direction. A random displacement, biased to have a large vertical component, was generated uniformly for each control point. This displacement was then projected on the vectors tangential and orthogonal to the liver at each control point. Then, a distance map was built from the thorax bones segmentation and used to filter the displacement at control points. For each control point, if its distance to the bones was inferior to 10 mm, the tangential component was set to 0 and the orthogonal component was set to 10% of its value. This models both the sliding motion of organs against bones and outward motion during breathing. Finally, the full displacement field was scaled by a factor varying between 0 and 1 following a $cos^4$ schedule to replicate a breathing pattern. To interpolate this displacement smoothly at every voxel, B-splines were used instead of linear interpolation to better handle the discontinuous nature of the displacement between control points. We generated five 10-phase synthetic 4D-CT with this process for a total of 50 samples. It is important to note that these deformations are generated in a completely different way than the training data, which is fully randomized, so our method is not biased to favor the reconstruction of these deformations.

**Needle insertion dataset:** Due to the action of surgical tools on the organ, intraoperative liver motion can be large, with displacements of up to 6 cm ([44]). In this dataset, we simulated the insertion of a synthetic needle 2 mm in diameter 50 mm into the liver over the course of 50 time steps. The amplitude of displacement was 30 mm along the direction of needle insertion. This displacement was applied at control points inside of the liver that were closer than 80 mm to the tip of the needle. With this approach, the deformation remained centered on the needle tip during insertion. A Deep Inspiration Breath Hold (DIBH) motion was also added, with the internal organs slowly moving upwards as the lungs slightly compress over the course of the breath hold.
## 4.2 Experimental setup

Sections 4.3, 4.4, and 4.5 present the results of our method on the IhuDeLiver10 dataset, the breathing motion dataset and the needle insertion dataset, respectively. A sensitivity analysis (Sec. 4.6) was also performed to evaluate the impact of the proposed loss function and backprojection module on the IHUdeLiver10 dataset. Even though our method is agnostic to the choice of anatomical target structure, the portal vein tree was chosen as the target for both the porcine and human datasets, due to its clinical relevance, as introduced in Sec. 1. Here, the proposed application of our method is to replace contrast injection by superimposing the predicted vessel shape on fluoroscopic images. Thus, to evaluate the accuracy of our method, we measure the 2D Dice coefficient between the deformed and predicted vascular trees projected on the fluoroscopic image plane.

The parameters for the data generation were not tuned between the use cases and are common for all cases. The networks were trained from scratch for each case. The voxel resolution was different for each CT, but was always < 1 mm³. The DVFs were generated at a resolution of 128 × 64 × 128 (in LR-AP-SI order) to reduce memory usage and processing time. To deform the baseline CT, the DVFs were upsampled linearly to the shape of the field domain before warping. The virtual C-arm was positioned automatically such that the liver was in the center of the projection, with a margin of (100., 50., 100.) mm around the liver bounding box to ensure the liver is always visible in the projection. The DRRs were generated at a resolution of 512 × 512 pixels, with a pixel size of 0.67 mm, and downsampled to 256 × 256 before being input to the network. We did not observe performance gains when processing the input image at full resolution and downsampling reduced memory usage and computing time. The field domain was set to the bounding box of the projection in the volume. The number of random control points in the ROI was set to $N_{cp} = 30$. This number was chosen arbitrarily to provide enough variability in the generated displacement fields. We did not study the impact of this parameter on network performance. The norm of the generated displacement vectors was comprised between 0 and 100 mm, in order to cover sufficiently large deformations. For each baseline CT, 20,000 samples were generated and randomly split into 18,000 training samples and 2,000 validation samples. Generating each training dataset took approximately 10 hours and 80 GB on a computer equipped with an Nvidia RTX 4090 GPU. Thus, due to time and space constraints, we did not generate a greater number of training samples.

## 4.3 Porcine dataset

The IHUdeLiver10 dataset<fnref n="1" /> is a comprehensive collection of high-quality liver CT images, primarily focusing on organ deformations in 10 swine subjects undergoing various image-guided procedures. Specifically, four cases were selected from the dataset based on their procedures and the ensuing deformations. The deformation mechanism was needle-tissue interactions in subjects 1 and 2, inter-fractional motion in subject 3, and deformations before and after laparoscopic surgery in subject 4. Each case is composed of pairs of {baseline; deformed} multi-phase contrast-enhanced CT scans (MPCECT). Each MPCECT includes a non-contrasted phase and a portal-enhanced phase (after injection). Portal phases were processed by a liver surgeon with extensive knowledge of swine anatomy, including a detailed segmentation of the portal tree.

To remove contrast information, both the baseline and deformed scans were inpainted as described in 3.2. As the contrast injections of baseline and deformed CTs were performed at different times, the number and size of branches do not match. To alleviate this issue, we used the vascular modeling toolkit ([45]) plugin in 3D Slicer ([46]), to compute the centerlines of each portal tree. We then voxelized the centerlines using a constant diameter of ~ 5 mm. Finally,

<footnote n="1">¹IHUdeLiver10 is available at https://doi.org/10.57745/EUBXGH</footnote>
<table><caption>Table 1: Accuracy of our method on the vessel tree registration for each subject in the porcine dataset.</caption><thead><tr><th rowspan="2">Subject id</th><th colspan="2">2D Dice</th></tr><tr><th>Registered</th><th>Baseline</th></tr></thead><tbody><tr><td>1</td><td>0.651</td><td>0.416</td></tr><tr><td>2</td><td>0.694</td><td>0.558</td></tr><tr><td>3</td><td>0.655</td><td>0.278</td></tr><tr><td>4</td><td>0.593</td><td>0.476</td></tr></tbody></table>

the same surgeon removed vessels that were absent from either the baseline or the deformed scan, ensuring that the Dice coefficient was computed between comparable structures.

We evaluated the network's performance for each case by separately generating a training dataset and evaluating its performance on the deformed CT. We used the optimal parameters detailed in Sec. 4.6 for each case. We report the 2D Dice coefficient before and after registration in Table 1. On this dataset, we obtain a mean Dice of $0.649 \pm 0.036$, from $0.432 \pm 0.102$ before registration.

## 4.4 Breathing motion dataset

During an intervention, the deformation of the anatomy is either introduced by the clinician or by physiology, such as breathing. In the above experiment, we evaluated the ability of our method to recover deformations caused by the clinician's action on the anatomy. In this experiment, we evaluate the ability of our method to recover another important type of deformation occurring during interventions, respiratory motion. Fig. 4 shows an example of three phases in the testing dataset (left), overlaid with the vessel centerlines deformed by the network prediction (right).

We compare our method with the IGCN+ method ([28]), developed for 2D-3D breathing motion prediction. This approach uses a U-net CNN to extract the 3D motion of an organ as a 2D, 3-component deformation field. This 2D motion is then projected onto the organ mesh, and a Graph Convolutional Network is used to compute the motion on the occluded parts of the organ. In [28], the networks are trained and tested on a private 4DCT dataset composed of several cases. Here, we train IGCN+ following the implementation described in ([28])<fnref n="2" />, on a dataset generated from the first 4DCT in the breathing motion dataset. Specifically, we used a PCA on the first two motion components of the dataset (as in [28]) to generate the training dataset composed of ground truth 2D 3 components deformation fields and ground truth deformed organ meshes. In this case, the 'organ' on which the IGCN+ method was trained and evaluated is the portal vein tree.

The performance of both methods was evaluated by measuring the 2D Dice coefficient between the ground truth and predicted portal veins segmentations. Fig 5 shows the performance of our method versus the IGCN+ method on the breathing motion dataset. On this dataset, our method obtained a mean Dice after registration of $0.863 \pm 0.047$ and the IGCN+ method obtained a mean Dice after registration of $0.879 \pm 0.044$, from $0.651 \pm 0.074$ before registration.

## 4.5 Needle insertion dataset

On this dataset, we evaluate the ability of our method to recover surgically induced deformations. In this dataset, the amplitude of displacement ranged from 0 to ≈ 30 mm over the course of the needle insertion. We used the same weights as in the above experiment for both our

<footnote n="2">²Original implementation available at https://github.com/meguminakao/IGCN</footnote>
<figure><img src="image_6.png" /><figcaption>Figure 4: From top to bottom, DRR images of samples 0, 2, and 5 of the breathing motion dataset overlaid with the displacement field (left) and with the predicted vessel centerlines (right). The length and color of the arrows represent the magnitude of the displacement field. The vessel centerlines are colored for visualization purposes.</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>Figure 5: 2D Dice of our method and the IGCN+ method on the test set for each sample in the dataset. While the IGCN+ method obtains a slightly better performance on this problem thanks to the high similarity between its training and testing data, our method achieves an excellent result despite the lack of similarity between the training data and the testing data.</figcaption></figure>

network and the IGCN+ network. Fig 6 shows the performance of our method and the IGCN+ method on the needle insertion dataset. On this dataset, our method demonstrated very similar performances as in the breathing motion dataset, with a mean Dice after registration of 0.800 ± 0.006, while the IGCN+ performances were largely degraded, with a mean Dice after registration of 0.696 ± 0.023, nearly the same as before registration (0.688 ± 0.031).

We also evaluated whether our model could be trained to be robust to the presence of a needle in the image, as it occurs in percutaneous interventions. To train the network, we randomly overlaid needles of varying diameters and positions on the input images during training. The translation varied from 0 to 50 mm around the initial position of the needle in the liver and the rotations between 0 and $0.3\pi$. For testing, we rendered the needle as it was inserted into the liver. This training and testing process is specific to the variant denoted ‘Ours with visible needle’ in Fig. 6. With visible needles, the performance of our method is very similar, with a mean dice of 0.805 ± 0.009.

## 4.6 Sensitivity analysis

In order to determine common parameters for all datasets, we designed several experiments to explore optimal design choices. These experiments were performed on the porcine dataset. We first explored 4 design choices for the loss function: 1) Mean Squared Error (MSE) loss between the predicted and ground truth displacement field 2) Reprojection MSE loss between the predicted and ground truth displacement field 3) Reprojection MSE loss + 3D soft Dice loss between the predicted and ground truth 3D segmentations 4) Reprojection MSE loss + 2D soft Dice loss between the predicted and ground truth 2D segmentations (see Sec. 3.3.2 for loss computation details).

In 5), we performed a hyperparameter search to determine the optimal value of the mixing coefficient $\lambda$ between the reprojection loss and the 2D soft Dice loss.
<figure><img src="image_8.png" /><figcaption>Figure 6: 2D Dice on the test set for each sample in the dataset. Our method (in blue) obtains similar performances on the needle insertion dataset and breathing motion dataset. In contrast, the IGCN+ method (in orange) fails on this task due to being trained solely on breathing motion, performing only slightly better than no registration (in red). Our method is also robust to the presence of surgical tools, such as needles, in the image when trained accordingly. The performance of our method is similar when trained and tested with needles in the image (in green) and without (in blue).</figcaption></figure>

We also evaluated the impact of the Backprojection module (see Sec. 3.3.1) over other design choices such as 2D-3D reshape without pooling and 2D-3D reshape with average pooling. These variants are denoted Backproj, NoBackproj, and NoBackproj-AvgPool respectively. In the NoBackproj and NoBackproj-AvgPool variants, the reshape was performed such that the spatial size of features input to the decoder was the same as in the Backproj variant. For the NoBackproj variant, this resulted in 3072 (instead of 768) 3D feature maps, and due to memory constraints, the batch size was reduced from 12 to 3. For the NoBackproj-AvgPool variant, average pooling was performed on the spatial dimensions before reshaping, such that the 3D feature map shape was the same as in the Backproj variant.

Finally, we compared the performances of the network with and without the data augmentation process described in Sec. 3.4.

For each case, the network was trained from scratch and the performance of each network was measured at the last epoch. The training data generation process was kept constant for all experiments and datasets. The best overall parameters found with the sensitivity analysis on the porcine dataset (loss function, architecture variant, and use of data augmentation) were used for all other datasets (breathing motion and needle insertion), which can be considered as held-out data with respect to the sensitivity analysis.

### 4.6.1 Loss comparison

Fig. 7 shows the results of experiments 1), 2), 3), and 4) where our proposed loss combination is compared against alternatives. Fig. 8 shows the results of experiment 5), where different values of the hyperparameter λ for our loss combination are compared. Based on this experiment, we chose λ = 0.5 for other experiments, because it was the most robust across cases. Other values
of $\lambda$ demonstrated suboptimal performances for at least one case and were thus rejected.

<figure><img src="image_9.png" /><figcaption>Figure 7: 2D Dice for each case in the porcine test dataset, for experiments 1) to 4), where different loss functions are used to train the network. The loss function used in 4) demonstrates the best average performances across cases.</figcaption></figure>

### 4.6.2 2D-3D translation

To transform 2D feature maps into 3D feature maps, an additional dimension must be introduced. Fig. 9 illustrates the performance difference between the three design choices for the 2D to 3D transformation of the feature maps. Fig. 10 shows the mean difference between the displacement field predicted for the base input and the displacement fields predicted for 100 perturbed inputs, projected on the input image.

### 4.6.3 Data augmentation

As introduced in 3.4, we used a data augmentation scheme to improve the robustness of the network to changes in image appearances. Fig. 11 shows the performance difference between the network trained with and without the proposed data augmentation scheme.

Based on the results of this sensitivity analysis, the parameters we used to measure the performance of our network on the porcine and human datasets are $\mathcal{L} = \mathcal{L}_{\phi^{2D}} + \lambda \mathcal{L}_{s^{2D}}$ with $\lambda = 0.5$, use of Backprojection to transform 2D feature maps to 3D, and data augmentation to improve network robustness to appearance changes.

## 5 Discussion

In this work, we presented a CT to fluoroscopy registration method to recover both breathing and surgically induced deformations. We evaluated our method on a porcine dataset representing common types of deformation in the context of image-guided liver therapy. Table 1 demonstrates the potential of our method for contrast-free, fluoroscopy-guided liver therapies.
<figure><img src="image_10.png" /><figcaption>Figure 8: Mean, minimum and maximum 2D Dice across all cases in the porcine test dataset, with different values for the weight of $L_{S2D}$, $\lambda$. The value of $\lambda = 0.5$ was chosen as it demonstrated the best robustness across cases, as other values demonstrated low performances, and thus low generalizability, on at least one test case.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 9: Mean, minimum and maximum 2D Dice across all cases in the porcine test dataset for the NoBackproj, NoBackproj-AvgPool, and Backproj variants. The Backproj variant obtains better minimum, mean, and maximum Dice than all other variants.</figcaption></figure>

On this dataset, our method was able to recover the shape of the vascular trees for three different types of intervention-related deformations with good accuracy.

Our results in Fig. 5 on the breathing dataset additionally showed that our method is able to recover breathing motion without requiring preoperative 4D-CT acquisition for training data. We obtain comparable performances with a state-of-the-art method, IGCN+, which was trained on patient-specific breathing motion. Our results in Fig. 6 show that, thanks to its domain-agnostic training process, our method is able to register a surgically induced deformation with good accuracy, while the IGCN+ method, trained on breathing motion, fails on this unseen deformation. This validates our domain-agnostic approach in the context of fluoroscopy-guided interventions, as it is able to recover surgically-induced deformations, which can potentially be of large amplitudes ([44]). We also demonstrated in Fig. 6 that training the network with needles randomly overlaid on the input image makes it robust to the presence of a needle in the image at inference time, as occurs during percutaneous interventions.

We performed a sensitivity analysis in Sec. 4.6 to justify and evaluate the impact of our design choices. We measured the performance improvement obtained when formulating the segmentation recovery as a training objective using a combination of a 2D reprojection loss and a 2D soft Dice loss. Fig. 7 shows that the greatest improvement is obtained when replacing the 3D MSE loss with the 2D reprojection loss. Adding the 2D soft Dice loss brings an additional moderate performance improvement. An explanation for this result is that the 2D Dice loss $L_{S2D}$ specifically targets the registration accuracy on the vessels, while the loss $L_{\varphi^{2D}}$ aims to indiscriminately recover the displacement field everywhere, teaching the network the relationship between the 2D input and associated 3D motion. In Fig. 8, an hyperparameter search was presented to determine the best mixing coefficient $\lambda$ between the two losses. Our criterion for hyperparameter selection was that our method should perform comparably across all cases, rather than maximize average performance. Thus, we chose $\lambda = 0.5$ as this value
<figure><img src="image_12.png" /><figcaption>Figure 10: In this experiment, the network input is perturbed in a small square with random noise. For each of the NoBackproj, NoBackproj-AvgPool, and Backproj variants, the mean output perturbation is measured and projected on the image. The circles represent the radius in which the output perturbation is at least 0.5 mm. The Backproj variant shows the least spatial extension of the perturbation, demonstrating great spatial correspondence between variations in the input and output.</figcaption></figure>
<figure><img src="image_13.png" /><figcaption>Figure 11: 2D Dice with and without data augmentation, for each case in the porcine test dataset. The network always performs better when data augmentation is used.</figcaption></figure>

offered the best compromise between average performance and robustness across cases. Fig. 11 validates the effectiveness of the data augmentation method proposed in ([43]) in our case, with important performance improvements on some cases. We hypothesize that slight changes in image appearance due to inter-fractional anatomical variations, caused, for example, by the digestive process, may induce prediction errors that are mitigated by this data augmentation approach. Finally, we evaluated whether our back projection layer was useful to translate 2D features into 3D features. A natural choice would be to use the *C* channels of the feature maps as a depth dimension. However, Fig. 9 clearly illustrates that this approach is too naive and leads to performance degradation. To respect the projective aspect of the 2D-3D transformation, the Backproject module introduced in Sec. 3.3.1 transforms the 2D feature maps using the projection matrix. This introduces a direct correspondence between a position **x** in the volume and a pixel **u** = **P****x**, which can be observed by randomly perturbing the input at a position **u** and measuring the change in the output displacement field. In Fig. 10, only the Backproj variant displays a local perturbation of the displacement field around the perturbed input, while the NoBackproj and NoBackproj-AvgPool variants display global perturbations of the displacement field. This indicates that the Backproj variant indeed incorporates a 2D-3D spatial correspondence.

While we used the Dice coefficient to evaluate our method, care must be taken to interpret
the value of the Dice coefficient of thin structures such as blood vessels. For example, a Dice
of 0.6 may seem low when compared with other values in the state of the art, but this value
is highly influenced by the small size of the vessels and the differences in the acquisition of
the baseline and deformed vessel trees. Notably, the lengths of corresponding branches in the
baseline and deformed vessel trees are different, which affects the Dice coefficient. It is, in turn,
more informative to look at the relative change before and after registration to evaluate results.
Fig. 12 shows how the value of the 2D Dice coefficient changes when the intrahepatic vessels
and the liver are translated. Due to their long and thin shape, the Dice metric varies far more
rapidly for the intrahepatic vessels than the liver. This is showcased in the top right part of
Fig. 12, which shows how a small shift (5mm translation) affects alignment for the liver and intrahepatic vessels.

<figure><img src="image_14.png" /></figure>

Figure 12: This figure is here to illustrate how the Dice coefficient evolves when 2 shapes, initially perfectly overlapped, are progressively misaligned. We particularly emphasize how the Dice value evolves as a function of the shift when the shape is a relatively large structure (e.g. an organ) and when the shape is a thin structure (e.g. blood vessels). This puts our results into perspective: a Dice value of 0.65 (obtained on the porcine dataset) would correspond to a Dice of 0.92 on a whole organ, while a Dice value of 0.81 (obtained on the breathing motion dataset) would correspond to a Dice of 0.96 on the liver.

Even though our results show that our method has the potential to enable augmented fluoroscopy-guided interventions, challenges remain before it can be used in the operating room. First, we did not validate the effectiveness of our method on real fluoroscopic images, as no paired fluoroscopy/CT datasets are readily available to test our deformable registration method. Obtaining a fluoroscopic image with a perfectly corresponding CT scan volume is not easily done due to practical constraints, but it could be the subject of future work with the use of a robotized CBCT device. However, other works focusing on rigid registration problems demonstrated that specialized synthetic training techniques could be used to bridge the domain gap between DRR images and fluoroscopic images ([43, 47, 48]) reporting clinically relevant performances. In our case, using the data augmentation technique in ([43]), originally developed to bridge this domain gap, improved performances even though DRRs were used as testing data.

Additionally, while our method is suitable for augmenting fluoroscopic images in real-time, it is still limited by the weak depth information in the fluoroscopy. To remediate this, a biomechanical model of the organ (in this case the liver) could be used to generate physically accurate training data. The biomechanical model could also be used in the loss to guide the prediction of the network towards physically plausible displacements.

Finally, while we used a soft Dice loss in our loss function, application-specific losses, such as the Centerline Boundary Dice Loss ([49]) specifically designed for vascular structures, could be used for potentially improved performances.
# 6 Conclusion

The aim of this study was to introduce a domain-agnostic 2D-3D deformable registration method that extends beyond breathing motion prediction. Our findings showcase the versatility of our proposed method, successfully applying it to various types of intervention-related deformations. This application is especially significant as it could potentially eliminate the necessity for contrast agents in fluoroscopy-guided interventions, thus offering a safer alternative. Furthermore, our method proves valuable in scenarios where contrast injection is technically challenging or results in asynchronous information. A key strength of our approach is its independence from a statistical motion model, rendering it adaptable to situations where motion or deformation is unpredictable, as opposed statistical motion model-based methods, which fail in such situations. Moving forward, our focus will be allocated to refine the method's accuracy by incorporating a physics-based deformation model. Additionally, we plan to evaluate its performance using real fluoroscopic images, aiming to validate and enhance its practical applicability in diverse clinical settings.

# 7 Credit authorship contribution statement

François Lecomte Conceptualization, Methodology, Software, Investigation, Data curation, Writing - Original draft, Visualization. Juan Verde Investigation, Resources, Data Curation, Writing - Original draft. Jean-Louis Dillenseger Conceptualization, Supervision, Funding acquisition. Stéphane Cotin Conceptualization, Resources, Writing - Review & Editing Supervision, Project administration, Funding acquisition.

# 8 Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

# 9 Data availability

The IHUdeLiver10 dataset is available at https://doi.org/10.57745/EUBXGH. The authors do not have permission to share the synthetic human dataset.

# 10 Acknowledgments

This work was funded by the French national research agency ANR (VATSOP project, reference ANR-20-CE19-0015). Juan Verde's contributions were made possible through funding from French state funds, managed within the "Plan Investissements d'Avenir" and by the ANR (reference ANR-10-IAHU-02). Additionally, the authors extend their sincere appreciation to the preclinical unit of IHU Strasbourg for their support in the development of the IHUdeLiver10 dataset.

# References

[1] Andrew J Epstein, Peter W Groeneveld, Michael O Harhay, Feifei Yang, and Daniel Polsky. Impact of minimally invasive surgery on medical spending and employee absenteeism. JAMA surgery, 148(7):641–647, 2013.
[2] Charalampos Mamoulakis, Konstantinos Tsarouhas, Irini Fragkiadoulaki, Ioannis Heretis, Martin F Wilks, Demetrios A Spandidos, Christina Tsitsimpikou, and Aristides Tsatsakis. Contrast-induced nephropathy: Basic concepts, pathophysiological implications and prevention strategies. Pharmacology & therapeutics, 180:99-112, 2017.

[3] Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Simon Chen, Yifan Liu, and Chunhua Shen. Towards accurate reconstruction of 3d scene shape from a single monocular image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):6480-6494, 2022.

[4] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV), pages 52-67, 2018.

[5] Andrey Salvi, Nathan Gavenski, Eduardo Pooch, Felipe Tasoniero, and Rodrigo Barros. Attention-based 3d object reconstruction from a single image. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2020.

[6] Lei Li and Suping Wu. Dmifnet: 3d shape reconstruction based on dynamic multi-branch information fusion. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 7219-7225. IEEE, 2021.

[7] Xiao Yang, Roland Kwitt, Martin Styner, and Marc Niethammer. Quicksilver: Fast predictive image registration-a deep learning approach. NeuroImage, 158:378-396, 2017.

[8] Guha Balakrishnan, Amy Zhao, Mert R Sabuncu, John Guttag, and Adrian V Dalca. Voxelmorph: a learning framework for deformable medical image registration. IEEE transactions on medical imaging, 38(8):1788-1800, 2019.

[9] Tony CW Mok and Albert CS Chung. Large deformation diffeomorphic image registration with laplacian pyramid networks. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part III 23, pages 211-221. Springer, 2020.

[10] Malte Hoffmann, Benjamin Billot, Douglas N. Greve, Juan Eugenio Iglesias, Bruce Fischl, and Adrian V. Dalca. Synthmorph: Learning contrast-invariant registration without acquired images. IEEE Transactions on Medical Imaging, 41(3):543-558, March 2022.

[11] Liyue Shen, Wei Zhao, and Lei Xing. Patient-specific reconstruction of volumetric computed tomography images from a single projection view via deep learning. Nature Biomedical Engineering, 3(11):880-888, 2019. Publisher: Springer US ISBN: 4155101904.

[12] You Zhang. An unsupervised 2d-3d deformable registration network (2d3d-regnet) for cone-beam ct estimation. Physics in Medicine & Biology, 66(7):074001, 2021.

[13] Yang Lei, Zhen Tian, Tonghe Wang, Justin Roper, Kristin Higgins, Jeffrey D Bradley, Walter J Curran, Tian Liu, and Xiaofeng Yang. Deep learning-based 3d image generation using a single 2d projection image. In Medical Imaging 2021: Image Processing, volume 11596, pages 516-521. SPIE, 2021.

[14] Chih-Wei Chang, Yang Lei, Tonghe Wang, Jun Zhou, Liyong Lin, Jeffrey D Bradley, Tian Liu, and Xiaofeng Yang. A deep learning approach to transform two orthogonal x-ray images to volumetric images for image-guided proton therapy. In Medical Imaging 2022: Image Processing, volume 12032, pages 484-490. SPIE, 2022.
[15] Yang Lei, Zhen Tian, Tonghe Wang, Marian Axente, Justin Roper, Kristin Higgins, Jeffrey D Bradley, Tian Liu, and Xiaofeng Yang. Fast 3d imaging via deep learning for deep inspiration breath-hold lung radiotherapy. In *Medical Imaging 2022: Image-Guided Procedures, Robotic Interventions, and Modeling*, volume 12034, pages 628–633. SPIE, 2022.

[16] John R. Adler, Steven D. Chang, Martin J. Murphy, James Doty, Paul Geis, and Stephen L. Hancock. The Cyberknife: A frameless robotic system for radiosurgery. In *Stereotactic and Functional Neurosurgery*, volume 69, pages 124–128, 1997. Issue: 1-4 ISSN: 10116125.

[17] Yvette Seppenwoolde, Wouter Wunderink, Erasmus Mc, and Alejandra Méndez Romero. Treatment precision of image-guided liver SBRT using implanted fiducial markers depends on marker-tumour distance. *Article in Physics in Medicine and Biology*, 2011.

[18] Yang Lei, Zhen Tian, Richard Qiu, Tonghe Wang, Justin Roper, Kristin Higgins, Jeffrey D Bradley, Tian Liu, and Xiaofeng Yang. Deep-learning-based markerless tumor localization using 2d kv/mv image. In *Medical Imaging 2022: Image Processing*, volume 12032, pages 914–919. SPIE, 2022.

[19] Chih-Wei Chang, Yang Lei, Tonghe Wang, Liyong Lin, Jun Zhou, Jeffrey D Bradley, Tian Liu, and Xiaofeng Yang. Using orthogonal 2d kv images for target localization via central matching networks. In *Medical Imaging 2022: Image-Guided Procedures, Robotic Interventions, and Modeling*, volume 12034, pages 439–444. SPIE, 2022.

[20] P. Markelj, D. Tomaževič, B. Likar, and F. Pernuš. A review of 3D/2D registration methods for image-guided interventions. *Medical Image Analysis*, 16(3):642–661, April 2012. Publisher: Elsevier.

[21] Mathias Unberath, Cong Gao, Yicheng Hu, Max Judish, Russell H Taylor, Mehran Armand, Robert Grupp, Ka-Wai Kwok, Luigi Manfredi, and Changsheng Li. The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective. *Frontiers in Robotics and AI*, 8, 2021. ISBN: 2021.716007.

[22] Y Zhang, X Huang, and J Wang. Automatic cone beam projection-based liver tumor localization by deep learning and biomechanical modeling. *International Journal of Radiation Oncology, Biology, Physics*, 108(3):S171, 2020.

[23] Guoya Dong, Jingjing Dai, Na Li, Chulong Zhang, Wenfeng He, Lin Liu, Yinping Chan, Yunhui Li, Yaoqin Xie, and Xiaokun Liang. 2d/3d non-rigid image registration via two orthogonal x-ray projection images for lung tumor tracking. *Bioengineering*, 10(2):144, 2023.

[24] Lin Tian, Yueh Z Lee, Raúl San José Estépar, and Marc Niethammer. Liftreg: Limited angle 2d/3d deformable registration. In *Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VI*, pages 207–216. Springer, 2022.

[25] Ran Wei, Fugen Zhou, Bo Liu, Xiangzhi Bai, Dongshan Fu, Bin Liang, and Qiuwen Wu. Real-time tumor localization with single x-ray projection at arbitrary gantry angles using a convolutional neural network (cnn). *Physics in Medicine & Biology*, 65(6):065012, 2020.

[26] Ran Wei, Bo Liu, Fugen Zhou, Xiangzhi Bai, Dongshan Fu, Bin Liang, and Qiuwen Wu. A patient-independent ct intensity matching method using conditional generative adversarial networks (cgan) for single x-ray projection-based tumor localization. *Physics in Medicine & Biology*, 65(14):145009, 2020.
[27] Hua-Chieh Shao, Jing Wang, Ti Bai, Jaehee Chun, Justin C Park, Steve Jiang, and You Zhang. Real-time liver tumor localization via a single x-ray projection using deep graph neural network-assisted biomechanical modeling. *Physics in Medicine and Biology*, 67(11):115009, may 2022.

[28] Megumi Nakao, Mitsuhiro Nakamura, and Tetsuya Matsuda. Image-to-graph convolutional network for 2d/3d deformable model registration of low-contrast organs. *IEEE Transactions on Medical Imaging*, 41(12):3747-3761, 2022.

[29] Hua-Chieh Shao, Yunxiang Li, Jing Wang, Steve Jiang, and You Zhang. Real-time liver motion estimation via deep learning-based angle-agnostic x-ray imaging. *Medical Physics*.

[30] Vivek Gopalakrishnan, Neel Dey, and Polina Golland. Intraoperative 2d/3d image registration via differentiable x-ray rendering. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 11662–11672, 2024.

[31] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A randomized correspondence algorithm for structural image editing. *ACM Trans. Graph.*, 28(3):24, 2009.

[32] Alain Trouve, Mirza Faisal Beg, Michael I Miller, and Laurent Younes. Computing Large Deformation Metric Mappings via Geodesic Flows of Diffeomorphisms. *International Journal of Computer Vision*, 61(2):139–157, 2005.

[33] Stanley Durrleman, Marcel Prastawa, Nicolas Charon, Julie R. Korenberg, Sarang Joshi, Guido Gerig, and Alain Trouvé. Morphometry of anatomical shape complexes with dense deformations and sparse parameters. *NeuroImage*, 101:35–49, November 2014.

[34] Mathias Unberath, Jan Nico Zaech, Sing Chun Lee, Bastian Bier, Javad Fotouhi, Mehran Armand, and Nassir Navab. DeepDRR – A Catalyst for Machine Learning in Fluoroscopy-Guided Procedures. *Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)*, 11073 LNCS:98–106, September 2018.

[35] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 770–778, 2016.

[36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In *International Conference on Medical Image Computing and Computer-Assisted Intervention*, pages 234–241. Springer, 2015.

[37] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pages 2117–2125, 2017.

[38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. *arXiv preprint arXiv:1412.6980*, 2014.

[39] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In *Artificial intelligence and machine learning for multi-domain operations applications*, volume 11006, pages 369–386. SPIE, 2019.
[40] Jakob Wasserthal, Hanns-Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al. Totalsegmentator: robust segmentation of 104 anatomic structures in ct images. *Radiology: Artificial Intelligence*, 5(5), 2023.

[41] Fausto Milletarì, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. *2016 Fourth International Conference on 3D Vision (3DV)*, pages 565–571, 2016.

[42] Vivek Gopalakrishnan and Polina Golland. Fast auto-differentiable digitally reconstructed radiographs for solving inverse problems in intraoperative imaging. In *Workshop on Clinical Image-Based Procedures*, pages 1–11. Springer, 2022.

[43] Matthias Grimm, Javier Esteban, Mathias Unberath, and Nassir Navab. Pose-dependent weights and domain randomization for fully automatic x-ray to ct registration. *IEEE transactions on medical imaging*, 40(9):2221–2232, 2021.

[44] Oleg Heizmann, Stephan Zidowitz, Holger Bourquain, Silke Potthast, Heinz-Otto Peitgen, Daniel Oertli, and Christoph Kettelhack. Assessment of intraoperative liver deformation during hepatic resection: Prospective clinical study. *World Journal of Surgery*, 34:1887–1893, 2010.

[45] Richard Izzo, David Steinman, Simone Manini, and Luca Antiga. The vascular modeling toolkit: a python library for the analysis of tubular structures in medical images. *Journal of Open Source Software*, 3(25):745, 2018.

[46] Andriy Fedorov, Reinhard Beichel, Jayashree Kalpathy-Cramer, Julien Finet, Jean-Christophe Fillion-Robin, Sonia Pujol, Christian Bauer, Dominique Jennings, Fiona Fennessy, Milan Sonka, et al. 3d slicer as an image computing platform for the quantitative imaging network. *Magnetic resonance imaging*, 30(9):1323–1341, 2012.

[47] Cong Gao, Benjamin D Killeen, Yicheng Hu, Robert B Grupp, Russell H Taylor, Mehran Armand, and Mathias Unberath. Synthetic data accelerates the development of generalizable learning-based algorithms for x-ray image analysis. *Nature Machine Intelligence*, 5(3):294–308, 2023.

[48] Srikrishna Jaganathan, Maximilian Kukla, Jian Wang, Karthik Shetty, and Andreas Maier. Self-supervised 2d/3d registration for x-ray to ct image fusion. In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pages 2788–2798, 2023.

[49] Pengcheng Shi, Jiesi Hu, Yanwu Yang, Zilve Gao, Wei Liu, and Ting Ma. Centerline boundary dice loss for vascular segmentation. In *International Conference on Medical Image Computing and Computer-Assisted Intervention*, pages 46–56. Springer, 2024.