# Real-Time Augmented Reality for mini-invasive Liver Surgery: a realistic 3D real time follow up for the surgeons

Michael Kugler, Alexandre Hostettler, Domenico Borzacchiello, Francisco Chinesta, Daniel George, Yves Rémond

## ► To cite this version:

Michael Kugler, Alexandre Hostettler, Domenico Borzacchiello, Francisco Chinesta, Daniel George, et al.. Real-Time Augmented Reality for mini-invasive Liver Surgery: a realistic 3D real time follow up for the surgeons. FFQIS - 1er Forum Franco-Quebecois d'Innovation en Santé, 2016, Montréal, Canada. FFQIS - 1er Forum Franco-Quebecois d'Innovation en Santé, 2016. hal-05397735

HAL Id: hal-05397735

https://univoak.hal.science/hal-05397735v1

Submitted on 15 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

FFQIS

1 ER FORUM
FRANCO-QUÉBÉCOIS
D'INNOVATION EN SANTÉ

<figure><img src="image_3.png" /></figure>

# Real-Time Augmented Reality for mini-invasive Liver Surgery : a realistic 3D real time follow up for the surgeons

Michaël Kugler (1), Alexandre Hosttetler (2), Luc Soler (2), Domenico Borzacchiello (3), Francisco Chinesta (3), Daniel George (1), Yves Rémond (1)

## Before : Large Incision

<figure><img src="image_4.png" /></figure>

The evolution of surgery from large incision to minimally-invasive techniques improved considerably the success rate as well as the patient post-operation recovery. Although those techniques improve drastically the patient experience, they simultaneously increased the technical difficulty for the surgeon losing both 3D vision and the full scale overview of the operated organ.

New augmented reality techniques used in per-operative conditions aim to provide the surgeons with a real-time 3D representation of a deformable organ. Here, liver mini invasive surgery accounting with its internal structures like blood vessels and tumour is investigated to cover the mini-invasive drawbacks, integrate real time deformation and improve the surgeon working environment.

## Now : Mini-invasive

<figure><img src="image_5.png" /></figure>

## Augmented reality

- 3D deformable representation of the full organ, blood vessels and tumour

- Superposition with the patient body or real-time laparoscopic video retransmission

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

## What surgeons will see tomorrow

<figure><img src="image_9.png" /></figure>

## Real-time treatment

- Pre-treatment to obtain a deformation reference database to avoid size complexity and time dependence

- Real time reconstruction using the data-base knowledge.

## Real deformations

- Computation of the internal and external deformations coming from breathing, heart beat stimulus and surgeon gestures

- Use of the real tissue mechanic (liver, vascularisation network and distribution and blood pressure) and homogenization into the numerical model

- Max geometric error < 1mm

<figure><img src="image_10.png" /></figure>

## Patient dependence

- Acquisition of the patient liver and vascularisation geometries from Ct-scan or MRI.

- Treatment of the acquire geometries with a computational learning method to integrate with the already known data-base.

- Express the new geometry in real-time through the learning-method

## Conclusion

- True mechanical behaviour including internal structures is integrated through homogenization and enable to compute internal and external deformations of the liver

- A liver database is being constructed through learning method. Preliminary results show the possible extraction of patient dependency and real time per-surgery response.

- Extension of the patient database through learning method is still required to improve the accuracy of the numerical tools.

<figure><img src="image_11.png" /></figure>

## References

- A. Ammar, et al., "Proper generalized decomposition of time-multiscale models," Int. J. Num. Meth. Eng., **90**, 2012, 569-596.

- Y. Rémond, et al., "Applied RVE Reconstruction and Homogenization of Heterogeneous Materials", Wiley-ISTE, 2016.

## Acknowledgement

Thanks to IRCAD (Institut de Recherche contre les Cancers de l'Appareil Digestif) through 3D-Surg and Visible Patient teams.

<footnote>(1) ICube, 2 rue Boussingault, Université de Strasbourg, CNRS, 6700 Strasbourg, FRANCE</footnote>

<footnote>(2) IRCAD, 1 Place de l'Hôpital, 67000 Strasbourg, FRANCE</footnote>

<footnote>(3) École Centrale de Nantes, IUF, 1 rue de la Noë, 44300 Nantes, FRANCE</footnote>