<figure><img src="image_1.png" /></figure>

HAL
open science

# Marker-based Registration for Large Deformations
-Application to Open Liver Surgery

Yinoussa Adagolodjo, Nicolas Golse, Eric Vibert, Michel de Mathelin, Stéphane Cotin, Hadrien Courtecuisse

► To cite this version:

Yinoussa Adagolodjo, Nicolas Golse, Eric Vibert, Michel de Mathelin, Stéphane Cotin, et al. Marker-based Registration for Large Deformations -Application to Open Liver Surgery. ICRA 2018 - International Conference on Robotics and Automation, May 2018, Brisbane, Australia. hal-01792837

HAL Id: hal-01792837

https://inria.hal.science/hal-01792837

Submitted on 15 May 2018

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Marker-based Registration for Large Deformations
- Application to Open Liver Surgery -

Yinoussa Adagolodjo<fnref n="1" />,<fnref n="2" />, Nicolas Golse<fnref n="3" />, Eric Vibert<fnref n="3" />, Michel De Mathelin<fnref n="1" />,
Stéphane Cotin<fnref n="2" />, Hadrien Courtecuisse<fnref n="1" />,<fnref n="2" />

**Abstract**— This paper introduces an Augmented Reality (AR) system for open liver surgery. Although open surgery remains the gold-standard for the treatment of complex tumors and central lesions, technological issues actually prevent using AR with sufficient accuracy for clinical use. We propose a markers-based method allowing for the tracking and the deformation of a preoperative model in real-time during the surgery. Markers are manually placed on the surface of the organ after opening the abdominal cavity, and tracked in real-time by a set of infrared cameras. Our framework is composed of both a non-rigid initial registration method, providing an estimation of the location of the markers in the preoperative model, and a real-time tracking algorithm to deform the model during the surgery (even for large deformation or partial occlusion of the organ). The method is validated on both synthetic and ex-vivo samples; in addition, we demonstrate its applicability in the operating room during a liver resection surgery on a human patient. Preliminary studies provided promising results to improve the location of tumors, and to help surgeons into planning the ideal resection intraoperatively.

## I. INTRODUCTION

Primary liver cancer is the sixth most common cancer, the third most common cause of death from cancer and liver metastases result in an estimated 600,000 deaths per years worldwide. For these hepatobiliary malignancies, hepatic resection represents one of the few potentially curative treatments, for large tumors (i.e. > 3 cm [1]);

In spite of recent advances [2], liver surgery faces several challenges, mainly due to wide anatomical complexity and variations inside an opaque organ [3]. Moreover, surgeons have to deal with two opposite goals: a complete removal of all the tumor tissue while preserving 1) enough remnant parenchyma [4] and 2) the required vascularization (inflow/outflow) and bile ducts for a functional tissue [5]. The future resected area needs to be carefully (preoperatively) anticipated according to CT scan and/or MRI findings and then faithfully applied in the operative field. To this end, hepatectomy requires precise section surface location and intraoperative navigation within the vascular bed. To guide surgeons, the main tool used for decades remains the ultrasonography [6]. However, this requires radiological skills and mental effort to rebuild the 3-dimensional anatomy. Because of all these difficulties, more than half patients with liver metastasis receive a non-optimal resection, even in tertiary centers [7].

For the most complex and central (perihilar) lesions, the open surgery remains the gold-standard approach and it seemed useful to develop augmented reality (AR) in this way. AR may also be useful in case of missing lesions (disappeared metastasis after chemotherapy) because it is well known that tumor tissue may remain in spite of normal imagery [8], [9]. Planning the ideal resection area and applying it intraoperatively (without visible targets) thanks to AR may help the surgeon to minimize the recurrence risk. However, unlike neurosurgery or orthopedic surgery in which organs do not suffer from the deformation, real-time intraoperative AR is very rarely used for routine hepatic surgery. Indeed, open liver surgery faces important technical issues that actually prevent using AR with sufficient accuracy for clinical use: mainly a huge plastic deformation due to the mobilization of the liver (soft tissue) and a limited access to the organ's surface area (anterior surface hidden by costal arches, posterior face not seen).

The goal of this work is to develop a new technique of intraoperative augmented reality for open liver surgery, based on the preoperative CT scan (3D reconstruction), accurate surface registration thanks to landmarks and infrared cameras, and then apply a deformation to the numerical reconstruction. Since open surgery implies unavoidable occlusions and significant deformations of the organs, we rely on optical markers tracked by infrared cameras, and a biomechanical model to provide robust tracking of the organ intraoperatively.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Initial Problem: 1) The optical markers are placed intraoperatively after the opening of the abdominal cavity 2) Their 3D positions are extracted thanks to an optical tracking system. 3) Since the markers were not present in the preoperative CT, we propose a method to estimate their position intraoperatively.</figcaption></figure>

The markers are placed and sutured on the liver at the beginning of the surgery (after opening the abdominal cavity),

<footnote n="1">¹AVR/ICube at CNRS. Strasbourg, France</footnote>

<footnote n="2">²Mimesis at Inria. Strasbourg, France</footnote>

<footnote n="3">³Hôpital Paul-Brousse. Paris, France</footnote>
which requires to estimate their position on the preoperative model (see Figure 1). The contributions of this paper are a semi-automatic method to solve this problem plus a tracking algorithm to deform the model in real-time (including during partial occlusion or large deformations) based on the 3D displacement of the markers. We demonstrate the feasibility of this approach both in terms of accuracy of the registration of the model (even for large deformations) and in terms of compatibility with the clinical workflow.

## II. RELATED WORKS

Attempts of AR have been realized in laparoscopy to improve the visualization of the screen, which appeared as an ideal interface between the numerical reconstruction and the surgeon [10], [11]. A 3D visualization and surgical planning software tools are proposed in [12] to overlay a 3D preoperative model on top of the stereoscopic camera of the Da Vinci robot; model registration is manually performed, but [13] proposed an automatic method to estimate the location of the endoscope by locating the tip of the volume data. Using only endoscopic cameras, Clements et al. [14] proposes a rigid alignment approach based on salient anatomical features, extracted both in preoperative images and endoscopic data.

In laparoscopy, liver undergoes large deformations invalidating the assumption of a rigid alignment between preoperative and intraoperative image space. Therefore, non-rigid registration has been a very active research topic, and the use of a Finite Element model, built from preoperative images, has been largely employed due to its ability to predict physically consistent deformations, and its robustness against unavoidable outliers detected in intraoperative images. Since pneumoperitoneum represents the main source of shape variation, some work aimed at pre-computing deformations to reduce the initialization problem [15]. The method proposed in [16] simulates the pneumoperitoneum as external pressures applied inside the abdominal cavity, which is modeled using a biomechanical model. However, these assumptions are invalidated as soon as the organ is manipulated by the surgeon.

Physics-based shape-matching algorithm is introduced in [17] where the registration problem is formalized as electrostatic-elastic a problem where an elastic model is charged electrically to slide into an oppositely charged organ shape representation. Nonetheless, the method assumes that at least 50% of the organ surface is captured by the camera, which is generally not possible. Plantefeve et al. [18] extended this method to the non-rigid alignment using a biomechanical model generated from a preoperative CT-scan and anatomical atlas to pre-compute ligaments position. Beside anatomical landmarks, organ silhouette was recently considered by exploiting the organ's rigidity [19], [20] or using multiple views silhouettes [21] or by estimating a 3D contour from the stereo-endoscope.

Although open surgery still represents the main surgical approach, the limited work-space, unavoidable occlusions during manipulations, and significant deformations after the mobilization of the organ raise other technical difficulties.

Yet, even if surgeons can directly access the organ, the mental fusion of planning models with the current surgical view remains challenging and only relies on surgeon's experience. In [22], a fusion technique is introduced in order to register the ultrasound images with a preoperative CT volume thanks to an electromagnetic sensor, the intraoperative CT remains static and not deformed. [23] uses a portable image overlay device to display virtual information directly on the liver surface. The method is used for the estimation of needle insertion trajectories, but deformations are not considered in this article. In [14] they propose a robust surface registration using salient anatomical features for the initial registration and a laser range scanner (LRS) is used to deform the model intraoperatively. The depth information of LRS is matched with the model thanks to a modified Iterative Closest Point (ICP) algorithm. In [24] authors provide evidence of benefits of deformation correction with an evaluation across 20 patients. Nevertheless, the method remains sensitive to unavoidable occlusions of the liver (for instance by the hand of the surgeons), and the ICP may provide miss-correspondences for large deformations.

## III. MATERIALS AND METHODS

The workflow of our method is introduced in Fig. 2. We rely on a Finite Element (FE) model that is driven by the displacement of a set of markers tracked by infrared cameras. The tracking system provides in real-time the 3D positions of the markers which are physically attached to the surface of the organ. The FE model is obtained from segmentation of preoperative CT images, commonly used nowadays for clinical routine diagnostic, embedding as well as all the internal structures of interest for the surgeon (such as a tumor, vessels, ...).

<figure><img src="image_3.png" /><figcaption>Fig. 2. Workflow: Preoperatively, a FE model is built based on the CT scan of the patient. The video stream of a monocular camera is displayed on monitors in the operating room, on which is overlaid the pre-operative model. The model deformed in real-time thanks to a set of markers that are attached to the liver and tracked in real-time by infrared cameras.</figcaption></figure>
An additional standard monocular camera is added to the system, in order to project the FE model on top of the images acquired from this camera. This augmented view is then displayed on the monitors in the operating room. In order to compute a consistent coordinate system frame between monocular and infrared cameras, the so-called Perspective-n-Point problem is solved providing the projection matrix M that maps 3D points to 2D image coordinates. The monocular and tracking cameras are physically mounted on a rigid support; M is therefore assumed to be constant and known during the surgery.

During the surgery, after the initial incision, the surgeon uniformly positions and sutures the markers around the tumor area. Our method is then composed of two steps: 1) an initial non-rigid registration allowing for the alignment and the deformation of the preoperative model with respect to the intraoperative configuration, 2) a tracking phase allowing for the deformation of the model in real-time following the real modifications the organ is subjected to. The two steps consist of the application of a set of constraints on the surface of the FE model, in order to fit the information extracted from the images (see below). The FE model is used as a robust regularization technique against outliers extracted from the images (see [25] and [26] for details), and as an extrapolation method to estimate the displacement of internal structure based on surfaces' displacement.

### A. Biomechanical Model

The FE model is meshed with linear tetrahedral elements. We choose a co-rotational formulation [27] for its stability, its fast computational time, possible thanks to pre-computations, and its ability to handle large displacements. With this formulation the local stiffness matrix can be written with the synthetic formulation:

$$ \mathbf{K}_e = \mathbf{R}_e \int_{V_e} (\mathbf{C}_e \mathbf{D}_e \mathbf{C}_e \partial V_e) \mathbf{R}_e^T \quad (1) $$

where $\mathbf{D}_e$ corresponds to the stress-strain matrix parametrized by the Young's modulus E and the Poisson's ratio $\nu$ and $\mathbf{C}_e$ is the strain-displacement matrix. $\mathbf{R}_e$ is a block-diagonal rotation matrix composed of the rotation of the tetrahedral element [28].

At each simulation step, an Implicit Euler integration is used and Lagrangian Multipliers are employed to impose constraints provided by image features:

$$ \mathcal{F}(\mathbf{q}) + \mathcal{H}(\mathbf{q}, \mathbf{c}, \mathbf{m})\lambda = 0 \quad (2) $$

where $\mathcal{F}$ are internal forces of the FE model, $\mathbf{q}$ are the position of the model, $\mathbf{c}$ are the contour of the organ in the first image and $\mathbf{m}$ are the 3D markers obtained by the tracking system. $\mathcal{H}$ is a non-linear function that maps the data extracted from the imaging systems to the model and $\lambda$ are the constraints forces to enforce the registration constraints. At each step, the problem is linearized leading to a non-linear constraint problem (NLCP) that is solved in real time thanks to GPU parallelization using the method described in [29]. The linearization of $\mathcal{H}$ is specific for each phase of the

registration method and is detailed in the next two sections. The linearized version of $\mathcal{H}$ is a matrix $\mathbf{H}$ known as the jacobian of the constraints which encode for each constraint imposed to satisfy image data, the associated degree of freedom of the FE model impacted by this constraint and the direction of the constraint force (see [29] for details).

### B. Initial Registration

The initial registration is performed after positioning the markers on the liver. During this step (of approximately 5 minutes), the surgeon is asked to remain off-camera and not to manipulate the organ. In this interval, the deformation of the liver remains limited to the breathing motion, and the global shape of the liver is similar to the segmented model. We propose a semi-automatic method that takes as input a coarse initial rigid alignment of the model and a manual segmentation of the contour of the liver in the first image of the monocular camera view. This procedure can be easily performed manually by an operator during the surgery (whereas it is difficult to automatize their acquisition). On the opposite, our method automatically refine the rigid alignment and fit the observed deformation in the per-operative image, whereas it would be complex and not compatible with clinical constraints to be performed manually.

Our method is an extension of work introduced in [20] where a set of protective constraints is applied to the model to fit the outline observed in the image. At each simulation step, the visible contour of the FE model is computed using the projection matrix M of the camera, and an ICP is employed to bind the 2D points of the image with their closest points in the contour of the model. The method is improved adding sliding constraints to enforce the surface of the model to meet the 3D position of the markers. Indeed, even if the location of the markers is not known at this step, we use the fact that they are attached to the surface of the liver. Sliding constraints are applied in the direction of the normal of the surface of the model, letting this way the mechanics of the FE to slide in the tangential plane (see [25] for details). At equilibrium (i.e. where constraints are applied to the same DOF during several simulation steps), the position of the tracked markers are stored into the biomechanical model and a Jacobian matrix $\mathbf{J}$ is defined to provide the position $\mathbf{n} = \mathbf{J}\mathbf{q}$ of the markers according to the position $\mathbf{q}$ of the model.

### C. Intraoperative Tracking

The intraoperative tracking is simplified to 3D points cloud matching. Although the number of markers in the model remains constant, the number of markers tracked by infrared cameras varies a lot during this phase (due to occlusions) and is subject to noise and false positive (due to the presence of instruments). In addition significant deformation may be observed due to the mobilization of the organ. In order to provide robust correspondences between tracked markers and the model, we exploit the time coherency to propose a modified ICP method (see Algorithms 1). Each point of the model is associated with its closest point in the tracked markers (line 8), but we enforce that each marker is associated with
at most one point (line 10–19). If a maker is associated with
several points on the model (line 10 – 17), only the closest
one is kept (line 15 – 17) and the other markers perform
a new search between the remaining available points (line
8). In order to avoid too large binding a distance parameter
dmax is set above which no correspondence can be found.

<figure><img src="image_4.png" /><figcaption>**Algorithm 1:** Binding process between point clouds</figcaption></figure>

The uniqueness of correspondences is essential to im-
pose displacements and avoid over constrained problems
with Lagrangian Multipliers of equation (2). This way, we
enforce that only the most relevant ones will contribute to
the deformation of the model. If markers disappear due to
occlusion, the resulting deformation may be approximated
but it is sufficient to find new associations as soon as the
markers are visible again.

IV. RESULTS

We now validate our approach in terms of accuracy and we
demonstrate its applicability in the operating room in order
to provide Augmented Reality during an open liver resection
surgery on a human patient<fnref n="1" />. The experimental setup was
composed of a Logitech webcam C920, a MacBook Pro Mi-
2014 with a Nvidia GeForce GT 750M and infrared Optitrack
tracking system composed of 4 Flex13 cameras.

Synthetic validations are performed using a silicone Phan-
tom Gel composed of Ecoflex® with Slacker, two products
both from Creation Silicone, in the ratio 1 A : 1,5 B :
1/8 Slacker (where A and B are respectively the base and
the catalyst of the silicone). This ratio allows to reproduce
biomechanical properties of soft tissues, the Young's mod-
ulus is estimated around 1-10 MPa. A generic vessel tree
model has been printed in 3D with deformable material and

incorporated inside the phantom gel during its fabrication.
Finally contrast agent has been inserted inside the gel and is
used as a target location inside the volume.

A. Estimation of markers’ position

We first evaluate the error in the positioning of the
3D markers after the initial registration. The phantom gel
has been scanned and segmented with 10 optical markers
uniformly repatriated on its surface. The gel has then been
placed under our tracking system and 4 deformations have
been generated (see Fig. 3). We segmented the outline of the
monocular camera and applied sliding constraints to register
the model and we measured the position of markers between
their estimated and real positions in the CT scan (see III-B).

<figure><img src="image_5.png" /><figcaption>Fig. 3. Error (m) between estimated and real marker positions versus simulations time (sec). Deformation 1 the gel is placed on the table with no deformation except gravity; Deformation 2 the gel is compressed on each side and a strong deformation is generated orthogonal direction of the camera plane; Deformation 3 and 4 the gel is stretched on the right side. Blue contours are the re-projected contour of the model after the registration. Red and orange dots are the estimated and real positions of markers.</figcaption></figure>

Our method converges rapidly (after 2 sec) toward an
average error of 5mm in the estimation of the location of the
markers, except for deformation 2 where the error is around
10mm. Nevertheless, the deformation 2 is not representative
of the deformation of the liver during hepatic surgery and
the main part of the error is in the direction of the depth of
the camera view.

B. Registration of internal structures

In this section we evaluate the ability of our method to
predict the displacement of internal structures based on the
displacement of markers located at the surface of the model.
We performed a first experiment where the silicone gel has
been scanned in two different configurations with 28 markers
on its surface. The two CT scans have been segmented
including the internal vessels (VE) and the radiopaque target
in the volume (T2) that are used for the validation. In
addition, one marker located on the surface is used for the
validation (T1) in order to evaluate the error at the surface
after the registration.

The point cloud registration method is applied between
both models and in Fig. 4 shows the error between the
internal structures. The error remains very small (between 1
and 3mm) including targets located far from the constraints
and even when only 7 markers are used for the registration.

An evaluation with an explanted liver was performed after
transplantation (see Fig 5). After the extraction of the organ

<footnote n="1">¹The patient agreed to participate in this research project but no medical decisions were taken based on the results of the registration.</footnote>
<figure><img src="image_6.png" /><figcaption>Fig. 4. Average error after the marker-based registration on silicon sample for different number of markers used for the registration.</figcaption></figure>

11 pins have been inserted on the surface and scanned in 3 different configurations. Our point cloud registration method is then employed using the segmentation of the pins to align the 3 models with each other *M1*, *M2* and *M3*. The average Hausdorff distance between the main vessels visible in all the CTs are respectively *M1* → *M2* : 3.83mm, *M2* → *M3* : 2.11mm and *M1* → *M3* : 2.48mm.

<figure><img src="image_7.png" /><figcaption>Fig. 5. Data generation using an explanted liver after transplantation.</figcaption></figure>

### C. Application to liver surgery

Finally, our method was applied in the operating room during a liver resection of a human patient. A biomechanical model composed of 1900 tetrahedral element has been generated from the preoperative CT, including the segmentation of the tumor and the vessel tree. The cameras have been fixed on a metallic bar mounted transversely on top of the patient with a direct view from above the patient (see Fig. 6). The calibration has been performed on the day before the surgery<fnref n="2" />.

After opening of the patient<fnref n="3" />, 7 markers of 0.5mm of diameter have been sutured on the liver of the patient around the area of the tumor in approximately 10 minutes. Their position has been chosen randomly to cover the maximum area of the organ visible in the optitrack system. The initial registration (including segmentation of the contour and initial rigid alignment) has been performed in approximately 5 minutes (with a frame rate in the simulation of 40 FPS). Then the tracking method was applied for 10 minutes providing

convincing registration results even for large deformations and occlusions by the hands of surgeons or when the organ is pushed inside the abdominal cavity. The simulation was completely interactive with a frame rate above 100 FPS thanks to GPU parallelization (see [29] for details).

<figure><img src="image_8.png" /><figcaption>Fig. 6. Augmented view during the liver surgery.</figcaption></figure>

Since no intraoperative CT scan was available, we do not have any ground truth associated with this data to evaluate the accuracy of the registration for internal structures. The only metric available is to measure the 2D re-projection error of the outline of the liver and the projected contour of the model. We manually segmented the visible contour of the organ every 100 frame of the video and measured the Hausdorff distance in pixels between the segmented contour and the projected contour of the model. Independently of the sequence of the video we reported an error varying between 5 to 10 pixels with maximum values of 20 pixels.

## V. CONCLUSIONS

We proposed an augmented reality system for open liver surgery. The method is based on a semi automatic initial registration and an intraoperative tracking method based on optical 3D makers. We demonstrated the applicability of this approach in the operating room during a liver resection surgery. This preliminary study provided promising results to improve location of tumors. In the future, this new technology will also be useful for further development of virtual reality in educational and training aim.

## REFERENCES

[1] D. M. Parkin, F. Bray, J. Ferlay, and P. Pisani, "Global cancer statistics, 2002." *CA: a cancer journal for clinicians*, vol. 55, no. 2, pp. 74–108.

[2] G. Torzilli, R. Adam, L. Viganò, K. Imai, J. Goransky, A. Fontana, C. Toso, P. Majno, and E. de Santibañes, "Surgery of Colorectal Liver Metastases: Pushing the Limits." *Liver cancer*, vol. 6, no. 1, pp. 80–89, nov 2016.

[3] P. Majno, G. Mentha, C. Toso, P. Morel, H. O. Peitgen, and J. H. Fasel, "Anatomy of the liver: An outline with three levels of complexity. A further step towards tailored territorial liver resections," *Journal of Hepatology*, vol. 60, no. 3, pp. 654–662, mar 2014.

[4] S. Truant, O. Oberlin, G. Sergent, G. Lebuffe, L. Gambiez, O. Ernst, and F.-R. Pruvot, "Remnant Liver Volume to Body Weight Ratio 0.5%: A New Cut-Off to Estimate Postoperative Risks after Extended Resection in Noncirrhotic Liver," *Journal of the American College of Surgeons*, vol. 204, no. 1, pp. 22–33, jan 2007.

<footnote n="2">²For later investigation, the hardware part could be replaced by all in one commercial system such as the system used in [24]</footnote>

<footnote n="3">³The study was approved by the local ethics committee, and the patient gave informed consent prior to the procedure.</footnote>
<figure><img src="image_9.png" /><figcaption>Fig. 7. (Top) Augmented view projected on the monitor for different steps of the surgery. (Down) Projected contour of the model (blue) with the closest distance to the manually segmented contour in the images.</figcaption></figure>

[5] F. Faitot, E. Vibert, C. Salloum, D. L. Gorden, F. Coscas, R. Adam, and D. Castaing, "Importance of conserving middle hepatic vein distal branches for homogeneous regeneration of the left liver after right hepatectomy." *HPB: the official journal of the International Hepato Pancreato Biliary Association*, vol. 14, no. 11, pp. 746-53, nov 2012.

[6] O. Traynor, D. Castaing, and H. Bismuth, "Peroperative ultrasonography in the surgery of hepatic tumours," *British Journal of Surgery*, vol. 75, no. 3, pp. 197-202, mar 1988.

[7] L. Viganò, G. Torzilli, M. Cimino, K. Imai, E. Vibert, M. Donadon, D. Castaing, and R. Adam, "Drop-out between the two liver resections of two-stage hepatectomy. Patient selection or loss of chance?" *European Journal of Surgical Oncology (EJSO)*, vol. 42, no. 9, pp. 1385-1393, sep 2016.

[8] J. W. Owen, K. J. Fowler, M. B. Doyle, N. E. Saad, D. C. Linehan, and W. C. Chapman, "Colorectal liver metastases: disappearing lesions in the era of Eovist hepatobiliary magnetic resonance imaging," *HPB*, vol. 18, no. 3, pp. 296-303, mar 2016.

[9] P. Chauvet, T. Collins, C. Debize, L. Novais-Gameiro, B. Pereira, A. Bartoli, M. Canis, and N. Bourdel, "Augmented reality in a tumor resection model," *Surgical Endoscopy*, aug 2017.

[10] X. Kang, M. Azizian, E. Wilson, K. Wu, A. D. Martin, T. D. Kane, C. A. Peters, K. Cleary, and R. Shekhar, "Stereoscopic augmented reality for laparoscopic surgery," *Surgical Endoscopy*, vol. 28, no. 7, pp. 2227-2235, jul 2014.

[11] N. C. Buchs, F. Volonte, F. Pugin, C. Toso, M. Fusaglia, K. Gavaghan, P. E. Majno, M. Peterhans, S. Weber, and P. Morel, "Augmented environments for the targeting of hepatic lesions during image-guided robotic liver surgery," *Journal of Surgical Research*, vol. 184, no. 2, pp. 825-831, oct 2013.

[12] L. Soler, S. Nicolau, P. Pessaux, D. Mutter, and J. Marescaux, "Real-time 3D image reconstruction guidance in liver resection surgery." *Hepatobiliary surgery and nutrition*, vol. 3, no. 2, pp. 73-81, apr 2014.

[13] S. Bernhardt, S. A.Nicolau, V. Agnus, L. Solera, C. Doignon, and J. Marescaux, "Automatic localization of endoscope in intraoperative CT image: A simple approach to augmented reality guidance in laparoscopic surgery," *Medical Image Analysis*, vol. 30, pp. 130-143, may 2016.

[14] L. W. Clements, W. C. Chapman, B. M. Dawant, R. L. Galloway, and M. I. Miga, "Robust surface registration using salient anatomical features for image-guided liver surgery: Algorithm and validation," *Medical Physics*, vol. 35, no. 6Part1, pp. 2528-2540, may 2008.

[15] F. M. Sánchez-Margallo, J. L. Moyano-Cuevas, R. Latorre, J. Maestre, L. Correa, J. B. Pagador, L. F. Sánchez-Peralta, J. A. Sánchez-Margallo, and J. Usón-Gargallo, "Anatomical changes due to pneumoperitoneum analyzed by MRI: an experimental study in pigs," *Surgical and Radiologic Anatomy*, vol. 33, no. 5, jul 2011.

[16] J. Bano, A. Hostettler, S. A. Nicolau, S. Cotin, C. Doignon, H. S. Wu, M. H. Huang, L. Soler, and J. Marescaux, "Simulation of pneumoperitoneum for laparoscopic surgery planning." *MICCAI Conference*, vol. 15, no. Pt 1, pp. 91-8, 2012.

[17] S. Suwelack, S. Röhl, S. Bodenstedt, D. Reichard, R. Dillmann, T. dos Santos, L. Maier-Hein, M. Wagner, J. Wünscher, H. Kennngott, B. P. Müller, and S. Speidel, "Physics-based shape matching for intraoperative image guidance," *Medical Physics*, vol. 41, no. 11, p. 111901, oct 2014.

[18] R. Plantefève, I. Peterlík, N. Haouchine, and S. Cotin, "Patient-specific Biomechanical Modeling for Guidance during Minimally-invasive Hepatic Surgery," *Annals of Biomedical Engineering*, 2015.

[19] N. Haouchine, F. Roy, L. Untereiner, and S. Cotin, "Using Contours as Boundary Conditions for Elastic Registration during Minimally Invasive Hepatic Surgery," *IEEE/RSJ International Conference on Intelligent Robots and Systems; IROS*, 2016.

[20] Y. Adagolodjo, R. Trivisonne, N. Haouchine, and H. Courtecuisse, "Silhouette-based Pose Estimation for Deformable Organs Application to Surgical Augmented Reality," *IEEE/RSJ International Conference on Intelligent Robots and Systems; IROS*, 2017, pp. 1-6.

[21] A. Saito, M. Nakao, Y. Uranishi, and T. Matsuda, "Deformation Estimation of Elastic Bodies Using Multiple Silhouette Images for Endoscopic Image Augmentation," *International Symposium on Mixed and Augmented Reality*, 2015.

[22] M. Diana, P. Halvax, D. Mertz, A. Legner, J.-M. Brulé, E. Robinet, D. Mutter, P. Pessaux, and J. Marescaux, "Improving Echo-Guided Procedures Using an Ultrasound-CT Image Fusion System." *Surgical innovation*, vol. 22, no. 3, pp. 217-22, jun 2015.

[23] K. A. Gavaghan, S. Anderegg, M. Peterhans, T. Oliveira-Santos, and S. Weber, "Augmented Reality Image Overlay Projection for Image Guided Open Liver Ablation of Metastatic Liver Cancer." Springer, Berlin, Heidelberg, 2012, pp. 36-46.

[24] L. W. Clements, J. A. Collins, J. A. Weis, A. L. Simpson, T. P. Kingham, W. R. Jarnagin, and M. I. Miga, "Deformation correction for image guided liver surgery: An intraoperative fidelity assessment," *Surgery*, vol. 162, no. 3, pp. 537-547, sep 2017.

[25] H. Courtecuisse, I. Peterlík, R. Trivisonne, C. Duriez, and S. Cotin, "Constraint-Based Simulation for Non-Rigid Real-Time Registration," *Medicine Meets Virtual Reality*, 2014.

[26] I. Peterlík, H. Courtecuisse, C. Duriez, and S. Cotin, "Model-Based Identification of Anatomical Boundary Conditions in Living Tissues," *Information Processing in Computer-Assisted Interventions*, pp. 175-187, 2014.

[27] M. Müller, J. Dorsey, L. Mcmillan, R. Jagnow, and B. Cutler, "Stable Real-Time Deformations," *Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation*, pp. 49-54, 2002.

[28] M. Nesme, Y. Payan, and F. Faure, "Efficient, physically plausible finite elements," *EUROGRAPHICS*, 2005.

[29] H. Courtecuisse, J. Allard, P. Kerfriden, S. P. A. Bordas, S. Cotin, and C. Duriez, "Real-time simulation of contact and cutting of heterogeneous soft-tissues," *Medical Image Analysis*, vol. 18, no. 2, pp. 394-410, 2014.