<figure><img src="image_1.png" /></figure>

HAL
open science

Silhouette-based Pose Estimation for Deformable
Organs Application to Surgical Augmented Reality

Yinoussa Adagolodjo, Raffaella Trivisonne, Nazim Haouchine, Stéphane Cotin, Hadrien Courtecuisse

<figure><img src="image_2.png" /></figure>

► To cite this version:

Yinoussa Adagolodjo, Raffaella Trivisonne, Nazim Haouchine, Stéphane Cotin, Hadrien Courtecuisse. Silhouette-based Pose Estimation for Deformable Organs Application to Surgical Augmented Reality. IROS 2017 - IEEE/RSJ International Conference on Intelligent Robots and Systems, Sep 2017, Vancouver, Canada. hal-01578815

HAL Id: hal-01578815

https://hal.science/hal-01578815

Submitted on 29 Aug 2017

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.
# Silhouette-based Pose Estimation for Deformable Organs
## Application to Surgical Augmented Reality

Yinoussa Adagolodjo<fnref n="1" />,<fnref n="2" />, Raffaella Trivisonne<fnref n="2" />, Nazim Haouchine<fnref n="2" />, Stéphane Cotin<fnref n="2" /> and Hadrien Courtecuisse<fnref n="1" />,<fnref n="2" />

***Abstract***— In this paper we introduce a method for semi-automatic registration of 3D deformable models using 2D shape outlines (silhouettes) extracted from a monocular camera view. Our framework is based on the combination of a biomechanical model of the organ with a set of projective constraints influencing the deformation of the model. To enforce convergence towards a global minimum for this ill-posed problem we interactively provide a rough (rigid) estimation of the pose. We show that our approach allows for the estimation of the non-rigid 3D pose while relying only on 2D information. The method is evaluated experimentally on a soft silicone gel model of a liver, as well as on real surgical data, providing augmented reality of the liver and the kidney using a monocular laparoscopic camera. Results show that the final elastic registration can be obtained in just a few seconds, thus remaining compatible with clinical constraints. We also evaluate the sensitivity of our approach according to both the initial alignment of the model and the silhouette length and shape.

## I. INTRODUCTION

Augmented Reality (AR) is becoming an increasingly helpful tool for guidance and navigation during laparoscopic surgery. During such surgery, organs are manipulated indirectly by the surgeon via instruments inserted through trocars while watching a monitor displaying the view of the surgical field captured via an endoscopic camera. This procedure reduces risks of hemorrhaging and shortens the post-operative recovery time which, making it an appealing technical approach. Its main disadvantage, however, resides in the limited (2D) visual feedback and lack of direct tissue manipulation. For these reasons, certain procedures remain quite challenging and require additional decision support especially for unexpected surgeons. In this context, AR can be used to visualize in-depth structures, thus helping surgeons locate tumors and avoid vessels, which results in safer procedures [12].

Providing a surgical AR often required to perform a pre-operative to intra-operative registration, since abdominal organs usually undergo (potentially large) deformations at the time of surgery. An important number of methods have been developed, either as 3D-3D registration, taking advantage of intra-operative 3D reconstruction methods (based on ultrasound, CT scanning, or stereoscopic images) [6] [17], [20], [8], [21], [2], [9], or as 3D-2D, registration using only a single image from monocular endoscopic camera [7], [4], [18]. The latter, which can also be seen as finding the camera

pose w.r.t to a deformable object (for instance the organ) is fundamentally ill-posed due to projection ambiguities where several 3D shape configurations can have the same 2D projection in the image.

In this paper, we propose a method for the 3D-2D initial alignment step that generally precedes any surgical augmented reality algorithm. Our approach uses as input a 3D pre-operative mesh and the organ's silhouette extracted from an intra-operative images. Using an underlying biomechanical model, our approach estimates the organ deformation, requiring minimal user interaction. Being generic and fast, we believe it can be used for various types of surgeries, and fits well the surgical work flow.

## II. RELATED WORKS

State-of-the-art methods that tackle initial alignment problem for surgical AR can be sorted into two categories: (i) methods that attempt to simulate the deformation leading to the intra-operative organ shape and (ii) methods that enrich the intra-operative data by adding information from anatomical landmarks, markers, position sensors or additional imaging modalities.

The aim of pre-computing deformations is to reduce the initialization problem to a rigid model-base alignment. Pre-computing model deformation requires simulating the impact of the intra-operative pressure (pneumoperitoneum) onto the abdominal organ, as this pressure represents the main source of shape variation [19]. The method proposed in [1] simulates the pneumoperitoneum as external pressures applied inside the abdominal cavity, which is modeled using a biomechanical model. This is mainly used to optimize the trocar's placement, rather than solving a registration problem. In a similar way, a deformed mesh is computed using a mass-spring-damper model [15] as a result of pressure insulation, which is applied to the volume as boundary conditions. Recently, a database-based approach was proposed to simulate organs deformation under pneumoperitoneum [10]. This database is built from intra-operative images and pre-operative segmentation to later be used as an atlas for new organs. These methods, however, still remains approximations and can hardly estimate the true intra-operative deformation and fail at reducing the problem to a rigid alignment.

In the second category, the strategy is not to pre-compute the deformation, but to use intra-operative imaging to directly estimate it. This data can be directly acquired from endoscopic cameras or via additional sensors and hardware. Using

<footnote n="1">¹CNRS, Strasbourg France</footnote>

<footnote n="2">²Inria, Strasbourg France</footnote>
only endoscopic cameras, Clements et al. [3] propose a rigid alignment approach based on salient anatomical features, extracted in both the pre-operative images and endoscopic data. Plantefeve et al. [17] extended this method to non-rigid alignment using anatomical atlases to pre-compute ligaments position. Beside anatomical landmarks, organ silhouette was recently considered by exploiting the organ's rigidity [4] or using multiple view silhouettes [18] or by estimating a 3D contour from the stereo-endoscope [9]. The organ silhouette can indeed bring powerful information and better constrain the registration process. These methods however lack of robustness due to inter-patient organ texture variability. To avoid uncertainties that emanates from endoscopic images based approaches, additional imaging can be used in the operating room to facilitate the initial alignment. Intra-operative ultrasound has been considered in [6] to register the liver vessel tree on a three-dimensional liver model. The registration of the vessels drives the deformation of the whole organ and thus approximates the deformation. Oktay et al. [16] suggested exploiting intra-operative CT-scans after insufflation as an additional constraint to drive the simulation. Recently [2], a method proposed to scan the tip of the endoscopic camera to find its rigid pose w.r.t in the intra-operative scanned organs. These methods provide accurate registration but intra-operative scans are for the moment not available in actual clinical routines.

We propose a semi-automatic method that takes advantages of both approaches. The deformation of the organ is initially estimated based on a numerical simulation. Intra-operative contours of the organ are then used to refine the solution and estimate 3D model pose. In contrast to previous methods that use 3D intra-operative reconstruction our approach relies solely on a single-view image acquired from a monocular endoscopic camera.

### III. METHODOLOGY

The registration of a pre-operative 3D segmentation S with intra-operative 2D contours c is an ill posed problem, known to be very difficult to solve since several 3D shapes can correspond to the same 2D projection. It can be formalized as finding a 3D non-rigid transformation T which maps S into the camera frame. However, since all the information available intra-operatively is two-dimensional, the solution can only be measured in 2D:

$$ \min_{\mathbf{q}} (\|\overline{T(S)} - \mathbf{c}\|) \qquad (1) $$

where q are the positions of S, and $\overline{T(S)}$ is the projection of T(S) in the camera view. For rigid transformation scenarios, this process is known as a perspective-n-point problem which can be solved given a set of correspondences between 3D points and their 2D projections [11]. However, the rigid assumption is not valid for our purpose and the deformations generate a problem having an infinite number solutions.

In order to decrease the number of solutions, we assume an elastic behavior of the organ with preservation of the volume after the deformation. Therefore, we propose to rely on a

non-linear biomechanical Finite Element (FE) model coupled with a set of projective constraints H to solve the problem of equation (1). This leads to a non-linear problem whose solution is the positions q of the physical model providing the equilibrium between the internal forces and the external forces applied by the projective constraints H:

$$ \mathcal{F}(\mathbf{q}) + \mathcal{H}(\mathbf{c}, \mathbf{q}) = \mathbf{0} \qquad (2) $$

where F are internal forces of the FE model. At this point it is important to note that the solution of the equation (2) is not unique (see figure 1).

<figure><img src="image_3.png" /><figcaption>Fig. 1. Multiple solutions: both configurations have the same projection in the camera view and minimize the energy between internal forces of the FE model and projective constraints. Both of them are a local minimum of equation (2). The simulation will converge towards the closest solution from the initial positions.</figcaption></figure>

In practice, the Newton-Raphson employed to solve this problem converges towards the closest local minimum from q₀, being the initial positions of the model. Nevertheless, the solution at equilibrium still does not necessarily correspond to the global solution of equation (1). It is therefore necessary to have a sufficient knowledge about the mechanical actions responsible for the deformation of the tissue (external forces, boundary conditions,...), to choose carefully q₀ such that the local solution of equation (2) corresponds to the global solution of the problem defined in equation (1).

Formally speaking, a transformation D(S) is computed based on an initial direct simulation of the deformation between pre-operative and intra-operative steps. For laparoscopy, D corresponds to the simulation of the pneumoperitoneum which represents the main source of shape variation. A strong assumption of our work is therefore to be able to simulate the main deformation of the organ compared to the segmented model S. We won't detail this aspect in this paper, instead we refer interested readers to the large number of publications on this topic [19], [1], [15], [10], [5].

The resulting model D(S) is assumed to be closer to the global solution but due to unavoidable errors in the model, it prevents us from using a purely rigid registration to fit the organ's contour in the image. Therefore, in the following sections we propose a non-rigid registration to perform this step while meeting the intra-operative constraints. The method takes as input a biomechanical mesh built from the deformed D(S) shape, an initial rigid registration R and the segmented contour in the image c. The overall process of our approach is shown in figure 2.
<figure><img src="image_4.png" /><figcaption>Fig. 2. Overview of the method: 1) A direct simulation $D(S)$ is applied to transform the reconstructed model obtain from the segmentation (red) in a shape close to the 3D position observed in the image (green). 2) A Rigid transformation (blue) is provided by the user to roughly align the model with the contour of the organ segmented in the image (yellow). 3) Projective constraints are applied to the biomechanical model to fit the organ's contour and finally provide the 3D shape with respect to the camera position.</figcaption></figure>

### A. Contour extraction and initialization

The method takes as input the contours c of the organ that is manually segmented in the 2D image. This operation is easily performed by sketching the silhouette of the organs in the 2D view and it can even be simplified using a tactile surface, sometimes already used in the operating room. Contours are then re-sampled and smoothed to facilitate the registration. For liver surgery, laparoscopic view is often partial and only a sub part of the organ's contour is visible. We will show that our method can deal with incomplete contours but obviously a sufficient part of the organ must be visible to obtain reliable solutions.

Without loss of generality, we also assume that the position of the camera is known with its intrinsic parameters. Indeed, our goal is to estimate the shape of the organ with respect to the camera position that can be chosen arbitrarily. The intrinsic parameters are not patient specific. Most of the time they are provided by constructors or they can be retrieved with an offline calibration. The position and camera parameters allow the definition of the so-called projection matrix P which maps 3D points to 2D image coordinates.

Since the frame coordinates of $D(S)$ and the camera may be significantly different, a rigid initialization $R$ is provided by the user. Although surgeons can immediately provide correspondences between the visible part of the organ in the 2D image on the 3D segmented model, this operation must be simplified as much as possible since it is performed intra-operatively. Manipulations of the model are therefore applied around the gravity center of the object following axes of the camera (for translations and rotations), allowing this way for a rigid alignment of the 3D model with the 2D view within several seconds.

### B. Biomechanical models

The biomechanical constitutive law follows a co-rotational formulation [13]. This model is chosen for its stability and

the fast computation time possible thanks to precomputations. Moreover, the model is not restricted to small displacements. With this formulation, the local stiffness matrix can be written with the synthetic formulation:

$$K_e = R_e \int_{V_e} (C_e D_e C_e \partial V_e) R_e^T \quad (3)$$

where $D_e$ corresponds to the stress-strain matrix parametrized by the Young's modulus $E$ and the Poisson's ratio $\nu$ and $C_e$ is the strain-displacement matrix. $R_e$ is a block-diagonal rotation matrix of the tetrahedral element which is obtained from a QR decomposition of the nodal displacements [14], and is recomputed at each simulation step.

### C. Projective Constraints

Projective constraints are applied to the biomechanical model. The first difficulty is to find the primitives of the model that correspond to the visible contour of the image (see fig. 3). Indeed, the 3D model is a close surface for which there is no outline.

The algorithm is performed in two steps. We first select all the front-face triangles with respect to the camera direction (i.e. $dot(n, z) > 0$, with n the normal of the triangle and z the direction of the camera). Since the convexity of the model cannot be assumed, this test is not sufficient because many triangles remain selected (red triangles in fig. 3(a)) while they are not visible as other triangles are closer to the camera (green triangles). In order to keep only the visible triangles, let a be the camera position and $q^{(i)}$ the set of points that belongs to the front-face triangles (red and green). An intersection test is performed between all the front-face triangles and $q^{(i)}a$ being a ray from the node $q^{(i)}$ to the camera. If this ray intersect any front-face triangle, then all the triangles connected to $q^{(i)}$ are discarded. Finally, the visible contour of the model (yellow in 3(a)) is obtained by selecting all the edges that connect one visible and non-visible triangles.

<figure><img src="image_5.png" /><figcaption>Fig. 3. Projective constraints definition. The visible outline of the model is extracted with respect to the camera position 3(a). An ICP method is performed to bind the segmented contour in the image with the projected contour of the 3D model 3(b). Finally constraints are projected back on the 3D model without any constraints along the depth of the camera 3(c).</figcaption></figure>

Once the contour of the model is obtained, it's projected in the image using the projection matrix $P$. Thanks to the initial rigid alignment, both outlines are similar in the 2D view. The Iterative Closest Point method is then used to bind each point of the contour c, manually segmented by the user, with the closest point on the projected contour of the model
(see Fig. 3(c)). In order to avoid outliers that may appear due to strong deformations or wrong initial alignment, we check that the curvature of both contours are similar. For that, we compute the 2D normal of both contours and reject any binding if the dot product is below a given threshold. In practice, the threshold is fixed to 0.5 to enforce that both the contours have the same curvature.

Finally, the constraints are projected back in 3D to the model (see Fig. 3(c)). By construction, these constraints will impose displacements in the 2D images in order to deform the contour, while the model is entirely free in the depth direction of the camera. Therefore, its motion will be only driven by internal elastic forces to reach equilibrium with constraints forces. If the contour of the model is smaller than the image's contour, stretching forces will tend to bring the model closer to the camera, whereas compression will push away the model. We show that iteratively repeating this process leads to a solution where the model appears at the correct 3D location including deformations compared to the initial shape $D(S)$ and error in the initial alignment $R$.

### D. Constraint-based registration

We now detail the numerical simulation used to link image-based data with the FE model. Initial positions of the FE model are obtained from successive transformations of the segmented model as explained previously $q_0 = R(D(S))$.

A dynamic backward Euler Implicit formulation is chosen to integrate the system over time. Indeed, even if the solution of our problem only requires the static equilibrium that fits the image contours, a dynamic integration scheme allows to numerically stabilize the system. At equilibrium, velocity and acceleration terms vanishes providing this way an equivalent solution to a static formulation. The governing equation of FE models is given by the dynamic equation:

$$M\ddot{q} + B\dot{q} + F(q) + H(q, c)\lambda = 0 \quad (4)$$

where $M$ and $B$ are respectively the mass and the damping matrices used to numerically stabilize the system. $H(q, c)$ is a non-linear function gathering the projective constraints previously defined. Lagrange multipliers $\lambda$ are used to impose displacements on FE models.

At each simulation step $i$, the non-linear problem defined in equation (4) is linearized which is equivalent to a first order Taylor expansion (see [5] for details):

$$\left\{ \begin{array}{l} A\mathbf{x} + H\lambda = \mathbf{b} \\ H^T\mathbf{x} = \delta \end{array} \right. \quad (5)$$

with $A = \frac{1}{h}M + B + h\frac{\partial F}{\partial q}$, $\mathbf{x} = h\Delta\dot{\mathbf{q}}$, $\mathbf{b} = -h^2\frac{\partial F}{\partial q}\dot{\mathbf{q}}$ the solution of the Newton-Raphson solver and $h$ the time step. $H = \frac{\partial H}{\partial q}$ is the linearization of the projective constraints, i.e. it includes the direction in which the contact forces $\lambda$ will be applied to cancel the violation of the constraints $\delta$. These directions are obtained, as explained in the previous section, such that no constraints are applied in the depth direction of the camera. $\delta$ is proportional to the distance in pixels between $c$ and the projected contour of the 3D model. This

equation is solved in two steps using the Schur complement method and an iterative Gauss-Seidel algorithm providing stable solutions of constraints (see [5] for details).

## IV. RESULTS

Since the only information available per-operatively are 2D, we do not have enough data to validate the 3D shape of the model after the registration. Therefore, we validated our approach in a controlled environment where 3 optical cameras (corresponding to axial, coronal and sagittal views) are calibrated and located in the same coordinate system (see Fig. 4). For that, we used 4 infrared Optitrack cameras<fnref n="1" /> capable of providing the location of a set of 3D markers with respect to Optitrack frame coordinates. The positions of the optical cameras and intrinsic parameters are estimated solving the $PnP$ problem based on the location of 3D positions of the markers. Note that even if the localization of the camera is not necessary for our purpose (i.e. providing AR with respect to an arbitrary position of the camera), this operation is necessary for the validation.

<figure><img src="image_6.png" /><figcaption>Fig. 4. Validation setup. A silicone gel is seen by 3 webcams in different 2D views. Markers are placed on the surface of the gel and tracked by Optitrack system providing 3D positions used for the validation.</figcaption></figure>

A phantom gel has been prepared with a silicone-based mixture mimicking biological tissue properties. We combined Ecoflex® with Slacker, two products both from CreationSilicone, in the ratio 1 A : 1,5 B : 1/8 Slacker (where A and B are respectively the base and the catalyst of the silicone). This ratio allows to reproduce biomechanical properties of soft tissues, the Young's modulus is estimated around 1-10 MPa. A CT scan has been acquired and segmented providing the geometry $S$ of the gel in the rest configuration.

The gel was then placed on a table in a position visible by all the cameras. An initial FE simulation $D(S)$ have been performed to simulate the effect of gravity and contact with the table.

We validate our approach using 4 deformations of the gel (see fig. 5). For each scenario, we stored the images

<footnote n="1">¹http://optitrack.com/</footnote>
<figure><img src="image_7.png" /><figcaption>Fig. 5. Average error (mm) of Manual rigid registration and our method in the 3 views of the cameras. 4 deformations are evaluated and all the method is performed using both *S* and *D(S)* models.</figcaption></figure>

obtained in the 3 camera views, and a set of 3D positions of 33 markers located on the gel's surface (using the Optitrack system). These markers are used only for the validation and considered as the *ground truth*. The contour of the gel has been segmented in 2D images and all the examples are initialized with both $R_1(S)$ and $R_2(D(S))$. We compared our method versus the initial rigid registration provided by the user. Every scenario shows a significant improvement of our approach compared to the manual transformations. The average 3D error (over all the deformations and cameras) are **Manual Rigid S**: 7mm, **Manual Rigid D(S)**: 7mm, **Our method S**: 1.6mm and **Our Method D(S)**: 1.2mm. The method provides accurate solutions for *S*, but the error can even be decreased with the knowledge of the contacts with the table using *D(S)*.

In order to evaluate the sensitivity of our method with respect to the initial rigid alignment, we performed several simulation adding perturbations on the initial transformation provided by the user. Translation of ±2 cm and rotations of ±20° were applied along the *X*, *Y* and *Z* axes of the camera. In Fig. 6 we report the convergence rate of our method during the simulation steps for the various perturbations. We found that the method is relatively insensible to these perturbations and converges toward the same solutions except for rotations along *X* + 20° and *X* − 20° axes. However the error remains limited (15 mm) and is still much smaller than the initial error provided by the manual alignment (26 mm).

<figure><img src="image_8.png" /><figcaption>Fig. 6. Error in mm during the simulation steps for various perturbations of the initial rigid transformation provided.</figcaption></figure>

Most of the time, the entire organ is not visible in the laparoscopic views. Therefore, we evaluated the sensitivity of

our approach with respect to incomplete contours. In figure 7 we show the resulting deformations where the entire contour is used to apply constraints on the model and 3 other simulations where 10,20 and 30% of the contour have been omitted. We show that the global shape of the organ is preserved and the global error remains acceptable even for 30% of the contour ignored. Moreover, the contour of the 3D model remains close to the real contour in the image at the location where even where there is no constraints applied.

<figure><img src="image_9.png" /><figcaption>Fig. 7. Sensitivity to partial contours. Yellow points are the contour that is conserved to apply constraints and red dots are the *ground truth* 3D positions.</figcaption></figure>

In terms of performances, we believe that the method fits the clinical time constraints. The segmentation and meshing steps are not considered since they are performed pre-operatively. A user study with 10 volunteers have been performed to evaluate the time required by the segmentation of the contour, the initial alignment and the simulation. The average time to draw the contours in the image is around 18 seconds, the rigid transformation required 22 seconds on average and the simulation is performed in 13 seconds. The
<figure><img src="image_10.png" /><figcaption>Fig. 8. Application to liver surgery (left) and kidney (right).</figcaption></figure>

entire process performed intra-operatively can therefore be
performed in less than one minute.

Finally, we applied our method to surgical data (see Fig. 8). Since no intra-operative CT scan was available, these results only prove the feasibility of our method with real data. We first applied our method to the registration of the liver in laparoscopy where the contour is partially occluded by other organs. Using only contours located on the top and bottom of the organ, our method quickly converges to a realistic configuration. We also applied the same approach for kidney surgery where the entire contour of the organ was visible in the image. In order to provide visual clues of the shape of the organ, the surgeon was asked to draw straight lines on the organ (purple) in order to see the curvature of the surface. Before the registration, a texture was applied on the 3D model (dashed red lines in Fig. 8) which correspond to straight in the rest configuration. After registration, the curvature of the texture is similar to what is observed in the medical image.

V. CONCLUSION

To conclude, we proposed a method to retrieve the 3D
shape and position of an organ only given its 2D contour
in a medical image. Being performed intra-operatively, our
method is intuitive and fast, and therefore fits well in a
clinical environment. In addition, it does not require any
specific hardware, or calibration protocol. Beside its clinical
application, the main scientific contribution is a mathematical
formalism of constraints which allow to define the defor-
mation of a 3D object given only 2D features. Although
already tested on a variety of scenarios, on which our method
provided very good results, it remains difficult to evaluate its
sensitivity to the initial alignment, and more importantly, to
the difference between the pre-operative and intra-operative
shapes. This is a direction we will pursue, along with
collecting clinical data for which we have a ground truth.

REFERENCES

[1] J. Bano, A. Hostettler, S. A. Nicolau, S. Cotin, C. Doignon, H. S. Wu, M. H. Huang, L. Soler, and J. Marescaux. Simulation of pneumoperitoneum for laparoscopic surgery planning. In *Proceedings of the 15th MICCAI: Part I*, pages 91–98, 2012.

[2] Sylvain Bernhardt, Stéphane A Nicolau, Vincent Agnus, Luc Soler, Christophe Doignon, and Jacques Marescaux. Automatic localization of endoscope in intraoperative ct image: a simple approach to augmented reality guidance in laparoscopic surgery. *Medical image analysis*, 30:130–143, 2016.

[3] Logan W. Clements, William C. Chapman, Benoit M. Dawant, Robert L. Galloway, and Michael I. Miga. Robust surface registration using salient anatomical features for image-guided liver surgery: Algorithm and validation. *Medical Physics*, 35(6):2528–2540, 2008.

[4] T. Collins, D. Pizarro, A. Bartoli, M. Canis, and N. Bourdel. Computer-assisted laparoscopic myomectomy by augmenting the uterus with pre-operative mri data. In *Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on*, pages 243–248, Sept 2014.

[5] Hadrien Courtecuisse, Jérémie Allard, Pierre Kerfriden, Stéphane P. A. Bordas, Stéphane Cotin, and Christian Duriez. Real-time simulation of contact and cutting of heterogeneous soft-tissues. *Medical Image Analysis*, 18(2):394–410, 2014.

[6] Benoit Dagon, Charles Baur, and Vincent Bettschart. A framework for intraoperative update of 3d deformable models in liver surgery. In *IEEE EMBS*, pages 3235–3238. IEEE, 2008.

[7] N. Haouchine, J. Dequidt, M.-O. Berger, and S. Cotin. Single view augmentation of elastic objects. In *Mixed and Augmented Reality (ISMAR), 2014 IEEE International Symposium on*, pages 199–208, Sep 2014.

[8] Nazim Haouchine, Jérémie Dequidt, Igor Peterlik, Erwan Kerrien, Marie-Odile Berger, and Stéphane Cotin. Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery. In *ISMAR 2013*, pages 199–208, 2013.

[9] Nazim Haouchine, Frederick Roy, Lionel Untereiner, and Stéphane Cotin. Using contours as boundary conditions for elastic registration during minimally invasive hepatic surgery. In *Intelligent Robots and Systems (IROS), 2016 IEEE/RSJ International Conference on*, pages 495–500. IEEE, 2016.

[10] S. F. Johnsen et al. Database-based estimation of liver deformation under pneumoperitoneum for surgical image-guidance and simulation. In *Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015*, pages 450–458, 2015.

[11] Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua. Epnp: An accurate o(n) solution to the pnp problem. *International Journal of Computer Vision*, 81(2):155, 2008.

[12] J. Marescaux and M. Diana. Next step in minimally invasive surgery: hybrid image-guided surgery. *Journal of Pediatric Surgery*, 50(1):30–36, 2015.

[13] Matthias Müller, Julie Dorsey, Leonard McMillan, Robert Jagnow, and Barbara Cutler. Stable real-time deformations. In *Proceedings of the 2002 ACM SIGGRAPH/Eurographics Symposium on Computer Animation*, SCA '02, pages 49–54, New York, NY, USA, 2002. ACM.

[14] Matthieu Nesme, Yohan Payan, and François Faure. Efficient, physically plausible finite elements, august 2005.

[15] Yukitaka Nimura et al. Pneumoperitoneum simulation based on mass-spring-damper models for laparoscopic surgical planning. *Journal of Medical Imaging*, 2(4):044004, 2015.

[16] Ozan Oktay et al. Biomechanically driven registration of pre- to intra-operative 3d images for laparoscopic surgery. In *Proceedings of the 16th MICCAI: Part II*, pages 1–9, 2013.

[17] Rosalie Plantefève, Igor Peterlik, Nazim Haouchine, and Stéphane Cotin. Patient-specific biomechanical modeling for guidance during minimally-invasive hepatic surgery. *Annals of Biomedical Engineering*, 44(1):139–153, 2016.

[18] Akira Saito, Megumi Nakao, Yuki Uranishi, and Tetsuya Matsuda. [poster] deformation estimation of elastic bodies using multiple silhouette images for endoscopic image augmentation. In *Proceedings of the 2015 IEEE International Symposium on Mixed and Augmented Reality*, ISMAR '15, pages 170–171, 2015.

[19] F. M. Sánchez-Margallo, J. L. Moyano-Cuevas, R. Latorre, J. Maestre, L. Correa, J. B. Pagador, L. F. Sánchez-Peralta, J. A. Sánchez-Margallo, and J. Usón-Gargallo. Anatomical changes due to pneumoperitoneum analyzed by mri: an experimental study in pigs. *Surgical and Radiologic Anatomy*, 33(5):389–396, 2011.

[20] Dos Santos et al. Pose-independent surface matching for intra-operative soft-tissue marker-less registration. *Medical image analysis*, 18(7):1101–1114, 2014.

[21] Stefan Suwelack et al. Physics-based shape matching for intraoperative image guidance. *Medical physics*, 41(11):111901, 2014.