# Towards real-time vessel guided augmented reality for liver surgery

Sidaty El Hadramy, Juan Verde, Nicolas Padoy, Stéphane Cotin

► To cite this version:

Sidaty El Hadramy, Juan Verde, Nicolas Padoy, Stéphane Cotin. Towards real-time vessel guided augmented reality for liver surgery. IEEE International Symposium on Biomedical Imaging (ISBI 2024), May 2024, Athens, Grece, Greece. 10.1109/ISBI56570.2024.10635104. hal-04387242

HAL Id: hal-04387242

https://inria.hal.science/hal-04387242v1

Submitted on 11 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# TOWARDS REAL-TIME VESSEL GUIDED AUGMENTED REALITY FOR LIVER SURGERY

Sidaty El Hadramy*†

Juan Verde*‡

Nicolas Padoy†‡

Stéphane Cotin*†

* Inria, Strasbourg, France

† ICube, University of Strasbourg, CNRS, Strasbourg, France

‡ IHU Strasbourg, France

## ABSTRACT

This work presents a novel method for augmented reality in liver surgery. Leveraging advanced reconstruction and segmentation techniques, our approach transfers diverse intraoperative data to Euclidean space, using vessel centerlines as a common representation. Hence, overcoming limitations in operation room equipment and clinician expertise. In addition, we capitalize on the graph-like structure of intraoperative vascular trees to ensure a robust and automatic initial registration. Our approach combines physics-based modeling accuracy with neural network speed for real-time elastic registration. The proposed method is trained on data synthesized from real human acquisitions and validated against a biomechanical model. Our approach achieves a mean target registration error less than 0.6 mm for different synthesized intraoperative cases. Moreover, it demonstrates robustness over different modalities and segmentation errors.

**Index Terms—** Augmented Reality, Deep Learning, Registration, Liver Surgery, Computer-Assisted Intervention

## 1. INTRODUCTION

Due to the highly deformable nature of the liver, hepatic surgery presents unique challenges in retrieving the location of relevant landmarks, securing tumor resection while avoiding undesired damage to vascular structures. Recent developments in real-time registration methods [1] have made it possible to bring Augmented Reality (AR) into the operating room. Numerous studies have showcased the advantages of physics-based digital twins in this context [2], in particular biomechanical models for their capability at dealing with sparse and/or noisy data [1, 3]. Different intraoperative imaging modalities can be used. Considering laparoscopic surgery, the most frequent modality is video streams, either monocular or stereo [4]. Using stereo vision techniques or SLAM approaches, it is possible to extract a point cloud from the organ surface as an intraoperative input [5], which will then serve as constraints for the biomechanical model. A general limitation of these methods is the need for an initial (rigid) registration with the pre-operative data, i.e, bringing both inputs into the same reference frame.

Other possible intraoperative imaging modalities are Ultrasound, and Fluoroscopy, and in high-tech operative rooms, Cone Beam Computed Tomography (CBCT), and Computed Tomography (CT). A common point of these modalities is their ability to acquire the state of the internal structures, such as vessels. Image registration typically relies on image features to estimate rigid or non-rigid transformations. Cheng et al. [6] demonstrated that neural networks can learn the similarity metric that outperforms normalized multi-information (NMI) and local cross-correlation (LCC) metrics commonly used in classical registration methods. Recent deep learning approaches [7] have proved to be a successful alternative to solve multimodel image registration problems, even when a large non-linear mapping is required. To cite a few, Sedghi et al. [8] performed the registration of 3D US/MR abdominal scans by using a neural network to learn a similarity metric. Haskins et al. [9] learned a similarity metric for multimodal registration of MR and transrectal US (TRUS) volumes by using a Convolution Neural Networks (CNN). El Hadramy, Verde et al. [10] suggested an approach to learn elastic registration of intra-patient CT scans across various contrast phases. Liao et al. [11] introduced a method to register abdominal 3D CT and cone-beam (CBCT) images. In these approaches, the networks are specifically designed to extract features from a particular type of intraoperative modality, thus limiting their flexibility.

To address these limitations, we present a novel method that offers generalization across intraoperative imaging modalities (assuming we can observe and segment internal structures) while benefiting from an accurate and robust initial registration. Our method relies on advanced state-of-the-art reconstruction [12] and segmentation [13] methods to transfer any intraoperative data from image to Euclidian space. The centerlines of the acquired intraoperative vessels are used as a common representation of the intraoperative modalities. The proposed approach leverages the accuracy and extrapolation capabilities of biomechanical models as well as Neural networks inference speed for a real-time non-rigid registration. Moreover, we exploit the graph-like structure of the vessel tree visible in intraoperative data to obtain a robust and accurate initial registration.
## 2. METHOD

The proposed method is patient-specific and works under the assumption that regardless of the intraoperative modality, the root portal vein (i.e., main trunk) and its branches are visible. This assumption is in line with the clinical workflow, as clinicians use such branches as anatomic landmarks to guide the intervention. The method is made of three essential steps (Figure 1). First, an initial registration to bring preoperative and intraoperative data onto the same reference. Then, we create a digital twin of the organ from the preoperative data. Finally, we train a neural network on the non-rigid registration task using data generated with the biomechanical model.

<figure><img src="image_2.png" /><figcaption>Fig. 1: A liver digital twin is used to train a network on the non-rigid registration task. The initial registration is performed thanks to the graph-like structure of the vascular trees.</figcaption></figure>

### 2.1. Digital Twin

#### 2.1.1. Parenchyma

Hyperelastic formulations are commonly utilized to model the behavior of soft tissues experiencing significant deformations. For the parenchyma model, we formulate a boundary value problem for computing the deformation of a hyperelastic material under both Dirichlet and Neumann boundary conditions. As illustrated in Fig 2, the parenchyma occupies a volume $\Omega$ with boundary $\Gamma$. We assume Dirichlet boundary conditions on $\Gamma_D$, a subset of $\Gamma$, are known a priori, while Neumann boundary conditions on $\Gamma_N$ can vary at any time step. In our case, $\Gamma_D$ corresponds to the intersection between the parenchyma and the portal vein (Fig 2.) We discretize the parenchyma domain $\Omega$ with first order hexahedral elements. This is motivated by their good convergence property, lock-free behavior [14] and regular structure that fits CNN inputs well.

The material behavior is approximated with the Saint-Venant-Kirchhoff constitutive model for its simplicity but also its ability to handle nonlinear deformations. With the Lagrangian formulation, the schema of the problem is written as follows:

$$ \left\{ \begin{array}{ll} \nabla \cdot (FS) = b & \text{on } \Omega \\ u(X) = 0 & \text{on } \Gamma_D \\ (FS) \cdot n = t & \text{on } \Gamma_N \end{array} \right. \quad (1) $$

<figure><img src="image_3.png" /><figcaption>Fig. 2: Formulation of the problem. The parenchyma occupies a domain $\Omega$, We consider both Dirichlet and Neumann boundaries $\Gamma_D$ and $\Gamma_N$ respectively. The domain $\Omega$ is discretized using hexahedron elements.</figcaption></figure>

Where $F$ represents the deformation gradient tensor and $S$ the second Piola-Kirchoff stress tensor. $b$ is the external body forces, $n$ the unit normal to $\Gamma_N$, $u$ the displacement field and $X$ is the material coordinates. For given boundary values, we consider the weak form of the problem 1 and solve it using the finite element method (FEM). In our case, $\Gamma_D$ is fixed, and the liver deformations are simulated by applying various traction forces on the Neumann domain. This process is described in the section 2.3.

#### 2.1.2. Internal Structures

The parenchyma model allows to describe the physical behavior of the organ surface. However, in our application, the internal structures are our main interest. The parenchyma model acting as a master imposes its displacements to the slaves, i.e, internal structures. We use a linear operator between the slave ($x_s$) and the master ($x_m$) discretization, where $J$ is the Jacobian matrix written as follows:

$$ J = \frac{\partial x_s}{\partial x_m} \quad (2) $$

### 2.2. Initial registration

We recall that the root portal vein is assumed to be visible in the intraoperative data. Given two vascular trees, we identify and match the root branch of both. This allows to have a first global registration by matching those branches. Then, the affine registration is performed through an Iterative Closest Point (ICP) [15] algorithm between both vessel trees surface points. The global registration ensures a precise and fast convergence of the ICP, which is known to be a local registration method and rely on a rough alignment as initialization [15]. The identification of the root branch is done by considering a graph structure originating from the portal vein. The graph is constructed from the vessel tree centerlines, where nodes and edges correspond respectively to bifurcations and branches in the tree. From an arbitrary orientation of the graph, we identify branches with either no parent or no children in the oriented graph. The root branch of the portal vein is determined as the one with the highest radius. This strategy aligns with
the gross anatomy of the portal vein, and its branching pattern. We demonstrate in the results section that this approach is robust to partial and noisy intraoperative data.

## 2.3. Non-rigid registration

Given the interoperative vessel tree centerlines, in this section, we aim to update the preoperative data in order to match the intraoperative liver state. For the sake of clarity, in the following, let $p_p$ be the preoperative parenchyma, $p_i$ the intraoperative parenchyma and $c_i$ the intraoperative vessels centerlines. We would like to find an operator $f$ that minimizes the following equation:

$$ \|p_p \circ f(c_i) - p_i\|_2 \quad (3) $$

In the equation 3, given a set of points $c_i$, $f$ outputs a displacement field that allows a non-rigid registration between $p_p$ and $p_i$ discretized domains. We approximate $f$ using the U-Net architecture, similar to the one proposed by Brunet, Mendizabal et al. [5]. The neural network takes as input a regular grid in which we encode the intraoperative centerlines. This grid has a high resolution and encodes the position of the closest points of the centerlines at each of its nodes. The output of the neural network, which describes the displacement field, has the same shape as the input.

The training set is generated with the physical model. Each sample corresponds to a deformed state of the model, which is obtained by choosing a set of points on the liver preoperative surface. Then, for each point, we consider a sphere of radius $r$. The intersection between these spheres and the liver surface is a region where we apply a random Gaussian distribution of forces. An example of this process is illustrated in Fig 3.

<figure><img src="image_4.png" /><figcaption>**Fig. 3:** Example of a training set generation sample. Spheres (in blue) centers are randomly chosen on the surface initial state (in gray). Forces are applied at their intersection with the surface. In red, the resulting deformation is shown.</figcaption></figure>

The linear operator described in section 2.1.2 also updates the vessel tree position, which constitute the intraoperative state in the training process. We use the method proposed by ntiga et al. [16] to extract their centerlines, then we apply random noise to the centerlines in order to simulate segmentation errors. The network is trained by minimizing the mean squared error between the ground truth (from the biomechanical model) and predicted displacement fields.

# 3. RESULTS

To evaluate our method, the liver parenchyma and its internal structures are segmented from a real patient data [17]. We build a physical model considering 4000 Pa for the Young modulus and 0.4 for the Poisson's ratio, we then generate training and validation datasets. The external force amplitudes vary up to a maximum value of 30N, to ensure we cover both small and large deformations of the organ. The dataset contains 3600 different deformations, 3200 are used for training and 400 for validation. We consider two cases according to the available intraoperative modality. The first case corresponds to modalities where we can observe the full vascular tree, for instance, CT, CBCT or MR, while the second case simulates modalities where usually only partial parts of the vessel tree are seen after the volume reconstruction, this is the case when dealing with modalities like US or intravascular US (IVUS). The following subsections describe the validation of our method in both cases. We use the physical model as a baseline and validate our networks' prediction using a target registration errors (TRE) on the tumors' localization. The implementation of the physical model is done through SOFA Framework [18]. The neural network is implemented with PyTorch<fnref n="1" />.

## 3.1. Full Intraoperative tree

Without loss of generality, in this case, we consider a scenario where surgeons have access to an intraoperative modality where the full preoperative vessel tree can be observed during the intervention. The full vascular tree and centerlines are considered intraoperatively in both training and validation. Table 1 shows the results of our network over the 400 validation samples. We achieve a mean target registration error of 0.6 mm. The predictions are computed in 34 ms. Figure 4 shows the results of a validation sample, where the matching between prediction and ground truth can be observed. Moreover, we have compared our results to the U-Mesh approach proposed by Brunet, Mendizabal et al. [5], their intraoperative modality is laparoscopic video, where a 3D point cloud on the surface of the organ is reconstructed from the video frames. We used the implementation proposed in their work to synthesize the point clouds, train and validate the network, the results are shown in the table 1.

<table><caption>**Table 1:** Results of our method in terms of TREs for the scenarios: Full, Partial (50%) and Partial (30%) vessel tree. The ground truth is represented by the digital twin.</caption><thead><tr><th></th><th>e<sub>mean</sub>(mm)</th><th>e<sub>max</sub>(mm)</th><th>e<sup>relative</sup><sub>mean</sub> (%)</th><th>e<sup>relative</sup><sub>max</sub> (%)</th></tr></thead><tbody><tr><td>U-Mesh [5]</td><td>0.83 ± 0.42</td><td>3.01</td><td>0.22 ± 0.11</td><td>0.78</td></tr><tr><td>Ours (30%)</td><td>0.59 ± 0.37</td><td>2.36</td><td>0.16 ± 0.10</td><td>0.63</td></tr><tr><td>Ours (50%)</td><td>0.63 ± 0.24</td><td>1.28</td><td>0.17 ± 0.06</td><td>0.33</td></tr><tr><td>Ours (full)</td><td>0.60 ± 0.37</td><td>2.97</td><td>0.16 ± 0.09</td><td>0.81</td></tr></tbody></table>

<footnote n="1">¹https://pytorch.org/docs/stable/index.html</footnote>
<figure><img src="image_5.png" /><figcaption>**Fig. 4:** Visualization of a validation sample with full intra-operative vessel tree. Initial state (in gray), ground truth (in green) and prediction (in orange).</figcaption></figure>

## 3.2. Partial intra-operative tree

The generated dataset is modified to reproduce scenarios where only partial data is available. In our experiments, we have considered two sub-trees consisting of respectively 50% and 30% of the initial vessel tree, branches in the area of the tumors were chosen. We have trained the network on both scenarios. Results on the validation datasets are reported in the table 1. The experiments show that the method still performs in the case of sparse intraoperative data, achieving a mean TRE of 0.59 mm and 0.63 mm for 30% and 50% cases respectively. The computation time is 34 ms.

## 3.3. Initial registration

Let $v_p$ and $v_i$ be two vessel trees surface points, located in the same reference frame, with a non-rigid deformation between them. A random rigid transformation is applied to $v_p$ to obtain a third vessel tree surface points, denoted $v_p^T$. Following our initial registration method, we register $v_p^T$ to $v_i$ and compare the obtained result to $v_p$. We consider four classes of $v_i$, denoted respectively "Full and clean", "full and noisy", "partial and clean" and "partial and noisy". Figure 5 illustrates the results of our initial registration approach in each of these classes. Our experiments show that the initial registration is robust to partial vascular trees and different level of noise that may occur due to segmentation errors.

<figure><img src="image_6.png" /><figcaption>**Fig. 5:** Results of the initial registration.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 6:** Segmentation sensibility results for full, 30% and 50% partial intraoperative vascular trees</figcaption></figure>

## 3.4. Segmentation sensibility

The centerlines are highly dependent on the intraoperative images' segmentation. Despite current advances in medical image segmentation, vascular segmentation remains very challenging and highly operator or software dependent. Therefore, during the training of the network, the input centerlines were subject to random noise with a maximum magnitude of 1 mm on x, y and z space directions, this allows to create a variability on the segmentation. We have then tested the network by applying noises of magnitude 1.5 mm and 2 mm on the centerlines. These levels of noises have not been seen during the training of the network. Figure 6 illustrates the results of partial intraoperative trees. As the figure shows, the variability of centerlines in the training set made it robust to even unseen level of noise, this demonstrates the robustness of our approach over segmentation errors.

# 4. CONCLUSION

The proposed method is trained on synthetically generated data from real human cases and validated against the FEM solution of a biomechanical model of the liver. Our approach demonstrated very good performance in preoperative to intraoperative non-rigid registration tasks. Moreover, by using the vascular tree as the main registration feature, our method presents several benefits. First, the robust and accurate initial registration is facilitated by the graph-like structure of intraoperative data. Second, it can handle different intraoperative modalities and shows robustness to partial and noisy data. Finally, upon completion of training, the neural network makes it possible to obtain real-time inference for the elastic registration. To further validate our approach, in future work, we plan to acquire preoperative and intraoperative data from real ex-vivo human liver. This allows to have better control of the Diriclet boundary conditions, a crucial factor for an accurate digital twin.
## 5. COMPLIANCE WITH ETHICAL STANDARDS

This research study was conducted retrospectively using human subject data made available in open access. Ethical approval was *not* required as confirmed by the license attached with the open access data.

## 6. ACKNOWLEDGEMENTS

This work was partially supported by French state funds managed by the ANR under reference ANR-10-IAHU-02 (IHU Strasbourg). The authors would like to express their deepest appreciation to Robin Enjalbert, Vincent Italiano, Valentina Scarponi and Pablo Alvarez.

## 7. REFERENCES

[1] Suwelack S, Röhl S, Bodenstedt S, Reichard D, Dillmann R, dos Santos T, Maier-Hein L, Wagner M, Wünscher J, Kenngott H, Müller BP, and Speidel S, “Physics-based shape matching for intraoperative image guidance,” *Med Phys.* 41(11):111901, 2014.

[2] Alvarez P., Chabanas M., Rouzé S., Castro M., Payan Y., and J. L. Dillenseger, “Lung deformation between preoperative ct and intraoperative cbct for thoracoscopic surgery: a case study,” *Medical Imaging, Vol. 10*576D, 2018.

[3] N. Haouchine, J. Dequidt, I. Peterlik, E. Kerrien, M. O. Berger, and S. Cotin., “Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery.,” *ISMAR*, pp. 199-208, 2013.

[4] L. M. Su, B. P. Vagvolgyi, R. Agarwal, C. E. Reiley, R. H. Taylor, and G. D. Hager, “Augmented reality during robot-assisted laparoscopic partial nephrectomy: Toward real-time 3d-ct to stereoscopic video registration,” *Urology*, vol. 73, no. 4, pp. 896–900, 2009.

[5] Brunet J. N., Mendizabal A., Petit A., Golse N., Vibert E., and Cotin S., “Physics-based deep neural network for augmented reality during liver surgery,” *Medical Image Computing and Computer Assisted Intervention. Cham: Springer International Publishing*, 2019.

[6] Cheng X., Zhang L., and Y. Zheng, “Deep similarity learning for multimodal medical images,” *In International conference on medical image computing and computer-assisted intervention*, 2016.

[7] Balakrishnan G, Zhao A, Sabuncu M R, Guttag J, and Dalca A V, “Voxelmorph: a learning framework for deformable medical image registration,” *IEEE Trans. Med. Imaging* 38 1788–800, 2019.

[8] Sedghi A., Luo J., Mehrtash A., Pieper S., Tempany C. M., Kapur T., Mousavi P., and Wells III W. M., “Semi-supervised deep metrics for image registration,” *arXiv preprint arXiv:1804.01565*, 2018.

[9] Haskins G., Kruecker J., Kruger U., Xu S., Pinto P. A., B. J. Wood, and P Yan, “Learning deep similarity metric for 3d mr-trus image registration,” *International Journal of Computer Assisted Radiology and Surgery*, 14:417–425, 2019.

[10] El Hadramy S., Verde J., Padoy N., and S Cotin, “Intraoperative ct augmentation for needle-based liver interventions,” *In International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland.*, 2023.

[11] Liao R., Miao S., Tournemire P., Grbic S., Kamen A., Mansi T., and D Comaniciu, “An artificial agent for robust image registration,” *AAAI*, pages 4168–4175, 2017.

[12] El Hadramy S., Verde J., Beaudet K. P., Padoy N., and Cotin S., “Trackerless volume reconstruction from intraoperative ultrasound image,” *In International Conference on Medical Image Computing and Computer-Assisted Intervention* (pp. 303-312). Cham: Springer Nature Switzerland, 2023.

[13] Wasserthal J., Breit H.-C., Meyer M.T., Pradella M., Hinck D., Sauter A.W., Heye T., Boll D., Cyriac J., Yang S., Bach M., and Segeroth M., “Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images,” *Radiology: Artificial Intelligence*, 2023.

[14] Benzley S. E., Perry E., Merkley K., Clark B., and Sjaardama G, “A comparison of all hexagonal and all tetrahedral finite element meshes for elastic and elasto-plastic analysis,” *4th international meshing roundtable*, (17) pp. 179-191., 1995.

[15] Paul J. Besl and Neil D. McKay, “A method for registration of 3d shapes,” *PAMI*, 1992.

[16] Antiga L. and B. Ene Iordache., “Centerline computation and geometric analysis of branching tubular surfaces with application to blood vessel modeling,” *WSCG*, 2003, 2003.

[17] Soler L., A. Hostettler, V. Agnus, A. Charnoz, J. Fasquel, J. Moreau, A. Osswald, M. Bouhadjar, and J. Marescaux., “3d image reconstruction for comparison of algorithm database: A patient specific anatomical and medical image database,” *Tech. Rep*, vol. 1, no 1, 2010.

[18] Faure F., Duriez C., Delingette H., Allard J., Gilles B., Marchesseau S., and Cotin S., “Sofa: A multi-model framework for interactive physical simulation,” *Soft tissue biomechanical modeling for computer assisted surgery*, 283-321, 2012.