# Augmented reality in liver surgery

Belkacem Acidi, Mohammed Ghallab, Stéphane Cotin, Eric Vibert, Nicolas Golse

► To cite this version:

Belkacem Acidi, Mohammed Ghallab, Stéphane Cotin, Eric Vibert, Nicolas Golse. Augmented reality in liver surgery. Journal of Visceral Surgery, 2023, 160 (2), pp.118-126. 10.1016/j.jviscsurg.2023.01.008. hal-04414274

HAL Id: hal-04414274

https://inria.hal.science/hal-04414274v1

Submitted on 9 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# AUGMENTED REALITY IN LIVER SURGERY, WHERE WE STAND IN 2023

ACIDI B¹,²,³, GHALLAB M¹,², COTIN S²,³, VIBERT E¹,²,⁴,⁵, GOLSE N¹,²,³,⁴,⁵

1 : Department of Surgery, AP-HP Hôpital Paul-Brousse, Hepato-Biliary Center,
94800 Villejuif, France

2 : Augmented Operating Room Innovation Chair (BOPA)

3 : Inria « Mimesis », Strasbourg, France

4 : DHU Hepatinov, 94800 Villejuif, France

5 : Inserm, Paris-Saclay University, UMRS 1193, Pathogenesis and treatment of liver diseases; FHU Hepatinov, 94800, Villejuif, France

**Auteur correspondant :**

Dr Nicolas Golse
Hepato-Biliary Center, Hôpital Paul Brousse
12 Avenue Paul Vaillant Couturier 94804 Villejuif Cedex, France

**Email** : nicolas.golse@aphp.fr

**Conflit d'intérêts** : aucun

**Financement** : aucun
# ABSTRACT

**Introduction:** During an operation, augmented reality (AR) enables surgeons to enrich their vision of the operating field by means of digital imagery, particularly as regards tumors and anatomical structures. While in some specialties, this type of technology is routinely utilized, in liver surgery due to the complexity of modeling organ deformities in real time, its applications remain limited. At present, numerous teams are attempting to find a solution applicable to current practice, the objective being to overcome difficulties of intraoperative navigation in an opaque organ.

**Objective:** To identify, itemize and analyze series reporting AR techniques tested in liver surgery, the objectives being to establish a state of the art and to provide indications of perspectives for the future.

**Methods:** In compliance with the PRISMA guidelines and availing ourselves of the PubMed, Embase and Cochrane databases, we identified English-language articles published between January 2020 and January 2022 corresponding to the following keywords: augmented reality, hepatic surgery, liver and hepatectomy.

**Results:** Initially, 102 titles, studies and summaries were preselected. Twenty-eight corresponding to the inclusion criteria were included, reporting on 183 patients operated with the help of AR by laparotomy (n=31) or laparoscopy (n=152). Several techniques of acquisition and visualization were reported. Anatomical precision was the main assessment criterion in 19 articles, with values ranging from 3mm to 14mm, followed by time of acquisition and clinical feasibility.

**Conclusion:** While several AR technologies are presently being developed, due to insufficient anatomical precision their clinical applications have remained limited. That much said, numerous teams are currently working toward their optimization, and it is highly likely that
in the short term, the application of AR in liver surgery will have become more frequent and
effective. As for its clinical impact, notably in oncology, it remains to be assessed.

**Keywords:** augmented reality, hepatic surgery, hepatectomy, hepatic resection
# INTRODUCTION

Otherwise known as operative removal, surgical resection is of key importance in the curative therapeutic arsenal for primary and secondary hepatic lesions. That said, one of the major difficulties of liver surgery consists in the organ's complex anatomy, which is subject to numerous inter-individual variations (1). The scarcity of external morphological landmarks (blood vessels, round ligaments...) renders intraoperative imagery indispensable. At present, intraoperative sonography remains the reference examination (2), contributing to orientation in the vasculo-biliary structures, to delineation of tumors, and to determination of resection planes. And yet, notwithstanding this tool, hepatic resection is obviously far too often incomplete, even in high-volume referral centers (3, 4), the reason being that a considerable level of expertise is required not only in performance and interpretation of bidimensional sonography, but also in terms of the intellectual effort needed in tridimensional reconstruction.

Over the last twenty years, several technological advances providing assistance for the liver surgeon have come into being as means of attenuating the difficulties of intraoperative navigation: 1/ Augmented reality (AR) corresponds to the different methods facilitating the embedding of digital data during visualization of a real-life scene. That said, the display of anatomical, biological and clinical information that it offers necessitates an interface between the actual object (in this case, the operating theater) and the observer (masks, glasses, screen, projection, hologram...) (5)), and essentially permits relatively "transparent" observation of the liver, the objective being to visualize its internal anatomical structures. 2/ After intravenous injection of fluorescent dye (need for near-infrared excitation), fluorescence imagery using indocyanine green (6) facilitates localization of superficial lesions, the biliary tree (7, 8), and vascular territories (9, 10). It bears mentioning that
fluorescence with indocyanine green has also been shown to favor detection of organ contours and thereby ameliorate intraoperative mapping or registration ("superposition", cf. infra.) of AR (11).

In this paper, we will not deal with: 1/ virtual reality (RV) corresponding to an artificially generated environment that enables simulation of a “fictional” scene for pedagogical or training-related purposes, but does not permit intraoperative guidance (immersive world without connection to a real-life environment); 2/ imagery by means of intraoperative slices (TDM or IRM), which is reserved for “few and far between” expert centers, and is not applicable to the majority of resected patients.

As of now and to our knowledge, there exists no exhaustive review of the literature specifically dedicated to AR in hepatic surgery. In this paper, we propose a systematic review designed to provide an update on the presently available technological advances, in which we will underline their interest, their limits, and the technological constraints explaining their restricted availability; we will also dwell on the perspectives they afford, the objective being to establish a state of the art on AR in its applications to hepatic surgery.

# MATERIALS AND METHODS

## 1) Study selection

A systematic review of the literature published between January 2000 and January 2022 was carried out from three data bases: PubMed, Embase and Cochrane. The following MESH terms were used: ((liver) AND (surgery)) AND ((augmented reality) AND (hepatectomy)).
Only studies dealing with AR in hepatic surgery and published in English in a peer-reviewed journal were pre-selected; inclusive and redundant terms were subsequently applied for the purposes of research exhaustivity. By application of the "snowballing methodology", any article identified among the references of a pre-selected article was likewise proposed for selection. In this review, we utilized the PRISMA "Reporting Items for Systematic Reviews and Meta-analysis" (PRISMA) recommendations (12).

## 2) Eligibility criteria and data selection

The titles and abstracts of the identified articles were independently studied by one of the co-authors (BA). Unpublished articles, research protocols, summaries and duplicated publications were excluded.

The complete texts of the presumably relevant articles dealing with AR-guided hepatic surgery were analyzed by two co-authors (BA and MG) in view of determining their eligibility. In the event of disagreement, decisions to include or not to include were made following interchange with two "senior" co-authors (EV, NG). As regards studies dealing with the application of AR to more than one organ, only the results concerning AR on the liver were taken into consideration.

The co-authors went on to extract the main information provided in each of the selected studies: 1) *ex vivo* or *in vivo* acquisition; 2) open or mini-invasive surgery; 3) human or animal model; 4) study year; 5) study country; 6) number of subjects included; 7) primary endpoint; 8) results; 9) software used; 10) required acquisition time; and 11) AR degree of precision.

We chose to exclude the following types of work: 1) studies dealing with VR; 2) studies conducted on an *in-silico* model alone; 3) studies conducted on *ex vivo* phantoms alone; 4) studies describing percutaneous treatments.
## 1) The eligible studies (Figure 1)

Ninety-nine articles were initially (pre)selected, and three studies (13-15) were included by "snowballing methodology". Twenty-eight articles were excluded because they did not deal with AR, as well as: 14 publications dealing with an organ other than the liver, 8 reviews of the literature, 9 studies based on mathematical models or synthetic livers, 7 articles describing percutaneous treatments, 3 commentaries, 2 letters to the editors, and 3 studies in Chinese (n=2) or German (n=1). All in all, seventy-four studies were excluded.

Finally, 28 studies were included (13-40)(table1); among them, 18 dealt with AR in mini-invasive (laparoscopic or robotic) surgery (13-16, 18, 20, 23-26, 32-39), and 9 with AR in open surgery (17, 19, 21, 22, 27, 28, 30, 31, 40), while one study described the two surgical approaches (29), 6 studies utilized an animal model (25, 28, 33, 35, 37, 38), 21 were carried out in humans (13-15, 17-24, 26, 27, 29-32, 34, 36, 39, 40), and 1 study in human and animal models (16). Twenty-one studies were conducted in vivo (13-15, 17-24, 26, 27, 29-33, 36, 38, 39), 2 were ex vivo (25, 28), and 5 involved a mixed in vivo and ex vivo (16, 34, 35, 37, 40) model. All in all, 183 livers were studied.

## 2) Review of the literature consisting in the main publications on hepatic surgery (Table 1)

### a) DEFINITION OF THE PERFORMANCE TOOLS
Also occasionally referenced as “mapping” or “resetting” or “alignment”, registration corresponds to the adjusted superposition of the digital distortion model to the actual position of the organ of interest. Registration is always necessary and particularly complex with regard to a flexible, deformable organ; it is generally based on either anatomical landmarks (inferior vena cava, gall bladder, anterior border of the liver) (22-26), or surface landmarks (27-30). Anatomical precision was the primary endpoint in 19 selected articles (13, 14, 19, 21, 22, 24, 26-29, 31, 33-40), translating the observed difference between the actual anatomy of the patient and the registered virtual models. This gap was quantified by several types of measurement (Table 2):

* TRE (Target Registration Error) (15) defined as the distance between the actual location of a tumor and its projection,

* FRE (Fiducial Registration Error) (14) corresponding to the mean difference of the distance errors,

* RPE (Reprojection Error) (26) corresponding to the mean distance between the projected 2D image and its actual 3D position visible by means of laparoscopy.

Seven studies used TRE as the tool to determine anatomical precision (max 13.4mm – min 3mm) (20, 24, 26-28, 30, 32), while 3 studies used FRE (max 14mm – min 6.5mm) (23, 25, 29), and 2 studies used RPE (max 12mm - min 10mm) (22, 33). In 9 studies, precision ranging from 1.1mm to 7.9mm (mean precision: 4.4mm) was reported without predefined tools (34-41).

All in all, precision ranged from less than 1mm to more than 1 cm:

- Maximum precision was reported by Kenngott & al (15) in an AR application during laparoscopic segmentectomy for hepatocellular carcinoma, with TRE of 0.96mm (+/-
5mm). This degree of precision was achieved by measuring the distance between 19 markers positioned on the liver and by comparing the distances f between augmented reality and actual reality. In another, more recent study, in which the authors carried out partial hepatectomy on ex vivo pig livers, Zhang & al (18) experimented with an augmented-reality helmet and recorded precision of 1mm (+/- 0.5mm).

- Maximum imprecision was reported by a Norwegian team having carried out manual AR alignment and calculating TRE using radiopaque markers (scanner) positioned on *in vivo* animal livers, with 13.4mm precision (35). Lastly, Prevost et al. tested the clinical feasibility of AR in surgery by means of manual registration in mini-invasive surgery, and reported 14mm precision (14).

## b) SURGICAL APPROACH AND TIME OF ACQUISITION

Among eighteen studies dealing with AR in mini-invasive surgery, nine concerned AR in open surgery, while one described the two approaches. Precision variables were calculated in open and mini-invasive surgery, with mean distances of 1.9mm and 3.7mm respectively, while mean time needed for acquisition was 5.6 minutes in mini-invasive surgery and 4.2 minutes in open surgery.

The time needed to obtain AR guidance from a preoperative 3D model, defined as time of registration, was indicated in 16 studies (17, 20, 22-26, 32-35, 37, 40-43), with values ranging from 1 to 10 minutes (mean of 5.2 minutes) (Table 2).

## c) SOFTWARE AND MEANS OF REGISTRATION
The most widely reported registration method was manual registration (n= 11) (14, 17, 20, 24, 27, 31, 32, 34-36, 40), while 9 teams described semi-automatic registration (initial, subsequently automatically refined manual registration) (13, 15, 16, 19, 22, 23, 25, 29, 33), and only 3 teams reported entirely automatic registration (18, 26, 37). The advantages and drawbacks of each software application are summarized in **Table 3**. In 2020, in an *in vivo* experiment including 9 pigs, H Luo & al (37) applied an entirely automatic technique in laparoscopic surgery and achieved precision of 8.7mm +/- 2.4mm. A less recent study by Buchs et al (30) reported on robotization adopting the Cas-One guidance system for laparoscopic procedures by AR via automatic registration based on different anatomical landmarks (gall bladder bed, tumor, left hepatic vein, free edge of segment III). The precision of this study was not indicated.

Acquisition of virtual models was carried out by means of different software applications, thereby enabling automatic, semi-automatic or manual registration (Table 2).

In 3 studies, Hepataug software was used (16, 23, 25), and it enabled semi-automatic acquisition of AR during laparoscopic surgery; a 3D model was created via preoperative imagery using MITK software. Anatomical landmarks (superior and inferior borders of the liver, falciform ligament) were manually contoured on the 3D (preoperative) model and the laparoscopic image. Once registration had been completed, the internal anatomical structures obtained by preoperative imagery and the deformable 3D models were superposed across the laparoscopic images. The software thereby provided the surgeon with information concerning the lesions initially detected on the imagery and, most importantly, projected a ring around the lesion corresponding to the optimal margin of resection.
Use of Virtual Reality (VR) Rendering software, which enables manual acquisition of AR in open and mini-invasive surgery, was reported in 3 studies (20, 31, 36). A virtual 3D anatomical model is generated by means of preoperative imagery (scanner or MRI). Resection planes are then delineated on the virtual model using the VR Render plug-in application and virtual surgery planning (VSP, IRCAD). Utilization of simulation open framework architecture (SOFA) (41) enabled simulation of the deformation due to pneumoperitoneum in three robot-assisted hepatectomies, leading to prediction of the position of the organ after insufflation in the peritoneal cavity. The virtual model was then superposed in the operating theater.

d) VISUALIZATION SYSTEMS

The visualization systems correspond to the different means of displaying, on the actual organ, the “clone” of reconstructed AR. There exist several available systems: AR helmets in which the 3D reconstruction is superposed to reality; the laparoscopy screen, which presents the advantage of placing AR at the disposal of the different operators; the console screen of the surgical robot; and, lastly, the video-projector projecting organ reconstruction in the operative field.

The majority of teams (n=14) employed an intraoperative visualization system merging the preoperative 3D reconstructions with the images acquired via the laparoscope on the laparoscopy screen (13, 14, 16, 18, 21, 25, 26, 34, 35, 37, 39). Six teams utilized several cameras including an RGB-D infrared depth camera, thereby superposing the AR model on the operating field via a video display on an external 2D monitor (17, 19, 22, 29, 31, 40). In 3 articles, AR was merged with the robot's laparoscopic camera, thereby achieving visualization of the model across the robotic console (20, 30, 36). In two cases, the AR
models were projected directly on the patient by means of a video-projector (24, 32). In one
article, the HoloLens (Microsoft®) helmet was used as a system of visualization (28). Finally,
one team used a commercialized tablet (27).

DISCUSSION

Nearly 20 years after initial utilization of augmented reality (AR) in neurosurgery (42), AR
in hepatic surgery is essentially confined to research (17). By contrast with orthopedic
surgery and neurosurgery, in which AR is currently utilized in routine practice (43, 44), the
main difficulty in hepatic surgery consists in the plasticity of the organ to model, which is
soft, deformable, and not framed or contained by a rigid bone structure. The deformations
brought about by handling of the organ, transsection or pneumoperitoneum complexify
registration of the models (superposition of the 3D reconstructions adjusted to the actual
intraoperative position of the organ).

In this paper, we analyzed 28 articles reporting on 183 subjects (138 humans, 45 animals) in whom AR had been tested. In 2018, a review of the literature included only 9 articles (5). While this significant increase in published experiments illustrates the growing interest of AR in hepatic surgery, it remains difficult to compare the different and heterogeneous techniques applied. In point of fact, AR is still largely limited to research settings, resulting in a lack of standardization of acquisition techniques, means of evaluation and visualization and, last but not least, modalities of registration.

While anatomical precision was more often than not the primary endpoint, the reported
distances were highly variable (3mm to 14mm) and, more importantly, did not suffice to
ensure reliable navigation in the hepatic parenchyma without a complementary means of
guidance. Once again, it bears mentioning that the liver is a plastic organ of which the
deformations are not necessarily homogeneous within the different anatomical zones; their
composition is heterogeneous (tumors, vessels, ligament attachments...), and they markedly
vary from one patient to another (fibrosis, cholestasis...), thereby partially explaining the
aforementioned anatomical imprecisions; in brief, the deformations in digital/numerical
models do not lend themselves to modeling in a real-life setting. Moreover, few of the
existing studies have described tests showing sizable deformation, which means that the
imprecision observed in real-life conditions is highly likely to be underestimated. That said,
numerous teams are presently at work on real-life deformations and they have been calling
for utilization of: 1/ surface markers, captured by a camera and enabling reconstruction of
the organ in real time (Video 1) or 2/ intraoperative imagery enabling continuous AR
registration through the acquisition of images (sonography, CT, MRI...) throughout the
intervention, thereby effectively capturing the actual position of the organ (area and depth)
at each and every instant. Theoretically ideal, this solution nonetheless raises issues
concerning cost, clutter, irradiation potentially affecting patient and caregiver alike ... and
limited availability.

The techniques reported in the literature are varied and differ according to approach. In
all cases, however, while AR implementation time during a surgical procedure seems limited
(<10 minutes according to most of the authors), it no longer represents an obstacle to AR in
routine practice. On the other hand, laparoscopy entails a number of potentially bothersome
constraints: 1/ low degree of perspective (and correspondingly low percentage of hepatic
surface susceptible to analysis in view of achieving reliable mapping/registration), which
renders it difficult to effectively capture a tridimensional scene; 2/ the way in which a single
luminous source entails shadows (occlusions) and non-visualizable zones; 3/ the
miniaturization and sterilization of depth capture systems (3D), 4/ creation of a depth image from simple (2D) optics or double (3D) optics (Figure 2). All of these factors manifest formidable technical complexity and counteract expansion of the tool.

All in all, robot-assisted surgery entails the same constraints as conventional laparoscopy, except for its including 3D optics that simplify creation of a depth image of a given scene. In the future, integration of AR algorithms in the robot may be facilitated and rendered more transparent for the user. When the above-mentioned constraints will have been attenuated, AR will be of pronounced interest in mini-invasive surgery as a means of compensating for the handling of laparoscopic ultrasound, which remains a highly demanding tool.

A model of totally automatic/robotic registration was studied in one article dealing with open hepatic surgery (21) and in the rare studies concerning laparoscopic surgery (26, 30, 38), in which higher precision than that achieved by other means of registration is reported; this is probably largely due to absence of the subjectivity characterizing semi-manual or manual models, in which the surgeon is called upon to superpose the AR clone, leading to inter-individual variability, and bringing about the most serious errors (14, 35).

From a practical standpoint, AR in hepatic surgery could yield greater resection margins (less incomplete R1/R2 resection), with potentially lower variability than when ultrasonography is used as a reference examination, especially in difficultly accessible areas (posterior or superior segments) or deep lesions. As regards laparoscopic surgery in an animal model, 30 ultrasound-guided atypical hepatic resections were compared to 30 resections guided by AR alone in another group, and also to 30 resections using sonography and AR (25).
In addition, AR seems to present pronounced interest in the treatment of metastases having disappeared during chemotherapy, which render identification problematic for the surgeon (5, 6). This indication (missing metastases) was reported by Ntourakis et al (31), who developed virtual hepatic models of 3 patients before and after chemotherapy, with the pre-chemotherapy anatomical model being projected in the operating theater. As a result, and due to augmented reality, all of the metastases detected on pre-chemotherapy images were superposed in real time on the intraoperative images, and all of the missing metastases were identified (by AR) and successfully resected, with confirmation of the presence of residual neoplastic tissue by means of anatomo-pathological study.

In the future, other indications could be imagined once the relevant technology has matured and registration has become more precise and reliable:

* Display of dangerous zones of “denied” access (virtual walls surrounding the bile ducts and major vascular structures);

* Assisted performance of anatomical resection, especially in cases of liver cancer, so as to achieve improved oncologic outcomes;

* Performance of AR-guided, percutaneous or intraoperative destructive actions.

One of the major future areas of development will consist in AR display at the disposal of all surgeons/assistants. This major point, which has seldom been assessed, entails supplementary difficulties, particularly as regards differing viewpoints (surgeon on the left, assistant on the right), which can lead to major parallax errors. This problem has been reported not only in projection by laparotomy, but also in laparoscopy in cases where the viewers' viewpoints differ, even though they are viewing the same image (45, 46). Moreover, when a robotic approach is implemented, surgically clothed assistants are denied 3D vision of the console and may, as a result, be excluded from visualization. Last but not
least, an AR mask is at present problematically compatible with intraoperative magnifying glasses -- and after several hours of utilization, it becomes uncomfortable.

## CONCLUSION

The utilization of augmented reality presently remains limited, especially insofar as the existing degrees of precision do not suffice to suffuse the surgeon with total confidence in the models proposed. As a result, intraoperative sonography may still be considered as the reference navigation. That said, numerous teams are currently striving to improve AR in terms of robotization and precision of registration, and it is highly likely that in the short to middle term, AR in hepatic surgery shall have become more effective and be utilized more frequently, whether in open or in mini-invasive surgery. As regards its clinical, oncological impact, it remains to be appraised prior to validating its routine use.
<table><caption>**Table 1: Characteristics of the studies selected in this review**</caption><thead><tr><th>Study</th><th>Country</th><th>Year</th><th>MI/O</th><th>In/ex vivo</th><th>Human/Animal</th><th>n</th></tr></thead><tbody><tr><td>Nicolau S.A (24)</td><td>France</td><td>2009</td><td>O</td><td>in</td><td>Human</td><td>5</td></tr><tr><td>Volonté F (32)</td><td>Switzerland</td><td>2011</td><td>MI</td><td>in</td><td>Human</td><td>1</td></tr><tr><td>Buchs NC (30)</td><td>France</td><td>2013</td><td>MI</td><td>in</td><td>Human</td><td>2</td></tr><tr><td>Kenngott H (15)</td><td>Germany</td><td>2013</td><td>MI</td><td>in</td><td>Human</td><td>1</td></tr><tr><td>Okamoto T (22)</td><td>Japan</td><td>2013</td><td>O</td><td>in</td><td>Human</td><td>3</td></tr><tr><td>Onda S (19)</td><td>Japan</td><td>2013</td><td>O</td><td>in</td><td>Human</td><td>2</td></tr><tr><td>Souzaki R (29)</td><td>Japan</td><td>2013</td><td>O-MI</td><td>in</td><td>Human</td><td>6</td></tr><tr><td>Kang X (33)</td><td>USA</td><td>2014</td><td>MI</td><td>in</td><td>Animal</td><td>2</td></tr><tr><td>Hallet J (20)</td><td>France</td><td>2015</td><td>MI</td><td>in</td><td>Human</td><td>1</td></tr><tr><td>Haouchine N(34)</td><td>France</td><td>2015</td><td>MI</td><td>in - ex</td><td>Human</td><td>1</td></tr><tr><td>Pessaux P (36)</td><td>France</td><td>2015</td><td>MI</td><td>in</td><td>Human</td><td>3</td></tr><tr><td>Conrad C (21)</td><td>USA</td><td>2016</td><td>O</td><td>in</td><td>Human</td><td>1</td></tr><tr><td>Ntourakis D (31)</td><td>France</td><td>2016</td><td>O</td><td>in</td><td>Human</td><td>3</td></tr><tr><td>Yasuda J (27)</td><td>Japan</td><td>2016</td><td>O</td><td>in</td><td>Human</td><td>9</td></tr><tr><td>Tang R (17)</td><td>China</td><td>2017</td><td>O</td><td>in</td><td>Human</td><td>1</td></tr><tr><td>Lau LW (38)</td><td>USA</td><td>2018</td><td>MI</td><td>in</td><td>Animal</td><td>2</td></tr><tr><td>Ozgur E (13)</td><td>France</td><td>2018</td><td>MI</td><td>in</td><td>Human</td><td>2</td></tr><tr><td>Thomson S (26)</td><td>England</td><td>2018</td><td>MI</td><td>in</td><td>Human</td><td>9</td></tr><tr><td>Zhang P (18)</td><td>China</td><td>2019</td><td>MI</td><td>in</td><td>Human</td><td>30</td></tr><tr><td>Espinel Y (16)</td><td>France</td><td>2020</td><td>MI</td><td>in/ex</td><td>H/A</td><td>8</td></tr><tr><td>Le Roy B (23)</td><td>France</td><td>2020</td><td>MI</td><td>in</td><td>Human</td><td>17</td></tr><tr><td>TLuo H (37)</td><td>China</td><td>2020</td><td>MI</td><td>ex - in</td><td>Animal</td><td>9</td></tr><tr><td>Prevost GA (14)</td><td>Switzerland</td><td>2020</td><td>MI</td><td>in</td><td>Human</td><td>10</td></tr><tr><td>Schneider C (39)</td><td>England</td><td>2020</td><td>MI</td><td>in</td><td>Human</td><td>18</td></tr><tr><td>Abdallah M (25)</td><td>France</td><td>2021</td><td>MI</td><td>ex</td><td>Animal</td><td>30</td></tr><tr><td>Golse N (40)</td><td>France</td><td>2021</td><td>O</td><td>in - ex</td><td>Human</td><td>5</td></tr><tr><td>Teatini A (35)</td><td>Norway</td><td>2021</td><td>MI</td><td>in - ex</td><td>Animal</td><td>1</td></tr><tr><td>Zhang F (28)</td><td>China</td><td>2021</td><td>O</td><td>ex</td><td>Animal</td><td>1</td></tr></tbody></table>

MI : Mini-invasive (approach); O: open (approach)
<table><caption>Table 2: Summary of the methods and results of the selected studies</caption><thead><tr><th>Study</th><th>Country</th><th>Year</th><th>Tool</th><th>Precision (mm)</th><th>Time (min)</th><th>Manual/SemiA/A</th><th>Software</th><th>n</th></tr></thead><tbody><tr><td>Nicolau SA (24)</td><td>France</td><td>2009</td><td></td><td>5</td><td>1</td><td>Manual</td><td></td><td>5</td></tr><tr><td>Volonté F (32)</td><td>Switzerland</td><td>2011</td><td></td><td>-</td><td>10</td><td>Manual</td><td>Osirix</td><td>1</td></tr><tr><td>Buchs NC (30)</td><td>France</td><td>2013</td><td></td><td>-</td><td>2</td><td>Automatic</td><td>Mevis</td><td>2</td></tr><tr><td>Kenngott H (15)</td><td>Germany</td><td>2013</td><td>TRE</td><td>0.93</td><td></td><td>SA</td><td>Singo ipilot</td><td>1</td></tr><tr><td>Okamoto T (22)</td><td>Japan</td><td>2013</td><td></td><td>5</td><td>-</td><td>SA</td><td>Analyze</td><td>3</td></tr><tr><td>Onda S (19)</td><td>Japan</td><td>2013</td><td>FRE</td><td>10.6 and 6.5</td><td>1.5</td><td>SA</td><td>Analyze</td><td>2</td></tr><tr><td>Souzaki R (29)</td><td>Japan</td><td>2013</td><td>TRE</td><td>5.0</td><td>-</td><td>SA</td><td>Virtual Place</td><td>6</td></tr><tr><td>Kang X (33)</td><td>USA</td><td>2014</td><td>TRE</td><td>3.3 and 2.7</td><td>0.14</td><td>SA Manual</td><td>Polaris</td><td>2</td></tr><tr><td>Hallet J (20)</td><td>France</td><td>2015</td><td></td><td>-</td><td>-</td><td>Manual</td><td>VR RENDER</td><td>1</td></tr><tr><td>Haouchine N (34)</td><td>France</td><td>2015</td><td></td><td>6</td><td>-</td><td>Manual</td><td>SURF</td><td>1</td></tr><tr><td>Pessaux P (36)</td><td>France</td><td>2015</td><td></td><td>2</td><td>8</td><td>Manual</td><td>VR RENDER</td><td>3</td></tr><tr><td>Conrad C (21)</td><td>USA</td><td>2016</td><td></td><td>3</td><td>1</td><td>Automatic</td><td>CASination</td><td>1</td></tr><tr><td>Ntourakis D (31)</td><td>France</td><td>2016</td><td>TRE</td><td>5</td><td>5</td><td>Manual</td><td>VR RENDER</td><td>3</td></tr><tr><td>Yasuda J (27)</td><td>Japan</td><td>2016</td><td>FRE</td><td>7.4</td><td>-</td><td>Manual</td><td>Optotrak</td><td>9</td></tr><tr><td>Tang R (17)</td><td>China</td><td>2017</td><td></td><td>-</td><td>-</td><td>Manual</td><td>IQQA</td><td>1</td></tr><tr><td>Lau LW (38)</td><td>USA</td><td>2018</td><td></td><td>7</td><td>3</td><td>Automatic</td><td>Open CV</td><td>2</td></tr><tr><td>Ozgur E (13)</td><td>France</td><td>2018</td><td></td><td>2.5</td><td>2</td><td>SA</td><td>Agisoft</td><td>2</td></tr><tr><td>Thompson (26)</td><td>England</td><td>2018</td><td>RPE</td><td>12</td><td>1 to 3</td><td>Automatic</td><td>Smart liver</td><td>9</td></tr><tr><td>Zhang P (18)</td><td>China</td><td>2019</td><td></td><td>-</td><td>-</td><td>Automatic</td><td>VTK</td><td>30</td></tr><tr><td>Espinel Y (16)</td><td>France</td><td>2020</td><td>TRE</td><td>11</td><td>5</td><td>SA</td><td>hepataug</td><td>8</td></tr><tr><td>Le Roy B (23)</td><td>France</td><td>2020</td><td></td><td>-</td><td>-</td><td>SA</td><td>Hepataug</td><td>17</td></tr><tr><td>Luo H (37)</td><td>China</td><td>2020</td><td>RPE</td><td>10</td><td>5</td><td>Automatic</td><td>Einstein vision</td><td>9</td></tr><tr><td>Prevost GA (14)</td><td>Switzerland</td><td>2020</td><td>FRE</td><td>14</td><td>8,5</td><td>Manuel</td><td>Cas one</td><td>10</td></tr><tr><td>Schneider C (39)</td><td>England</td><td>2020</td><td>TRE</td><td>3</td><td>10</td><td>SA/Manual</td><td>smart liver</td><td>18</td></tr><tr><td>Abdallah M (25)</td><td>France</td><td>2021</td><td></td><td>-</td><td>-</td><td>SA</td><td>Hepataug</td><td>30</td></tr><tr><td>Golse N (40)</td><td>France</td><td>2021</td><td></td><td>7.9</td><td>10</td><td>Manual</td><td></td><td>5</td></tr><tr><td>Teatini A (35)</td><td>Norway</td><td>2021</td><td>TRE</td><td>13.4</td><td>-</td><td>Manual</td><td>Polaris spectra</td><td>1</td></tr><tr><td>Zhang F (28)</td><td>China</td><td>2021</td><td></td><td>1.2</td><td>-</td><td></td><td></td><td>1</td></tr></tbody></table>
A: automatic; SemiA: semi-automatic; FRE: Fiducial projection error; RPE: Reprojection error; TRE: Target registration error

<table><caption>**Table 3: Advantages and drawbacks of the main augmented reality software applications reported in the literature**</caption><thead><tr><td>Software</td><td>Advantages</td><td>Drawbacks</td></tr></thead><tbody><tr><td>Osirix</td><td>Precision: 5</td><td>Registration: M<br>Approach: O<br>Time: 10</td></tr><tr><td>Mevis</td><td>Approach: MI<br>Registration: A<br>Time: 2</td><td>—</td></tr><tr><td>Singo ipilot</td><td>Approach: MI<br>Registration: SA<br>Precision: 0.9</td><td>—</td></tr><tr><td>Analyze</td><td>Time: 1.5</td><td>Precision: 10<br>Approach: O</td></tr><tr><td>Virtual Place</td><td>Registration: SA<br>Approach: MI</td><td>Precision: 5</td></tr><tr><td>Polaris</td><td>Registration: SA<br>Approach: MI<br>Precision: 2.7</td><td>Time: 0.14</td></tr><tr><td>VR RENDER</td><td>Approach: MI<br>Precision: 5</td><td>Registration: M Time: 5</td></tr><tr><td>SURF</td><td>Approach: MI</td><td>Registration: M<br>Precision: 6</td></tr><tr><td>CASination</td><td>Temps: 1<br>Precision: 3<br>Registration: A</td><td>Approach: O</td></tr><tr><td>Optotrak</td><td>—</td><td>Approach: O<br>Registration: M<br>Precision: 7.4</td></tr><tr><td>IQQA</td><td>—</td><td>Registration: M<br>Approach: O</td></tr><tr><td>Open CV</td><td>Temps: 3<br>Registration: A</td><td>Precision: 7<br>Approach: O</td></tr><tr><td>Agisoft</td><td>Approach: MI<br>Registration: SA<br>Precision: 2.5 Time: 2</td><td></td></tr><tr><td>Smart liver</td><td>Approach: MI<br>Temps: 1<br>Registration: A</td><td>Precision: 12</td></tr><tr><td>VTK</td><td>Registration: A<br>Approach: MI</td><td>—</td></tr><tr><td>Hepataug</td><td>Registration: SA<br>Approach: MI</td><td>Precision: 5<br>Time: 11</td></tr><tr><td>Einstein vision</td><td>Registration: A<br>Approach: MI</td><td>precision: 10 Time: 5</td></tr><tr><td>Cas one</td><td>Approach: MI</td><td>Registration: M<br>Temps: 8.5<br>Precision: 14</td></tr><tr><td>Polaris</td><td>Approach: MI</td><td>Registration: M<br>Precision: 13.4</td></tr></tbody></table>

A: Automatic; M: manual; MI: Mini-invasive; O: open; Precision in mm; SA: Semi-Automatic; Time: in minutes
REFERENCES:

1. Majno P, Mentha G, Toso C, Morel P, Peitgen HO, Fasel JH. Anatomy of the liver: an outline with three levels of complexity--a further step towards tailored territorial liver resections. J Hepatol. 2014;60(3):654-62.

2. Castaing D, Kunstlinger F, Bismuth H. [Intraoperative sonography in hepatic surgery]. J Belge Radiol. 1986;69(1):1-8.

3. Farges O, Fuks D, Boleslawski E, Le Treut YP, Castaing D, Laurent A, et al. Influence of surgical margins on outcome in patients with intrahepatic cholangiocarcinoma: a multicenter study by the AFC-IHCC-2009 study group. Ann Surg. 2011;254(5):824-29.

4. Memeo R, de Blasi V, Adam R, Goere D, Azoulay D, Ayav A, et al. Parenchymal-sparing hepatectomies (PSH) for bilobar colorectal liver metastases are associated with a lower morbidity and similar oncological results: a propensity score matching analysis. HPB (Oxford). 2016;18(9):781-90.

5. Quero G, Lapergola A, Soler L, Shahbaz M, Hostettler A, Collins T, et al. Virtual and Augmented Reality in Oncologic Liver Surgery. Surg Oncol Clin N Am. 2019;28(1):31-44.

6. Kokudo N, Ishizawa T. Clinical application of fluorescence imaging of liver cancer using indocyanine green. Liver Cancer. 2012;1(1):15-21.

7. Lehrskov LL, Westen M, Larsen SS, Jensen AB, Kristensen BB, Bisgaard T. Fluorescence or X-ray cholangiography in elective laparoscopic cholecystectomy: a randomized clinical trial. Br J Surg. 2020;107(6):655-61.

8. Hong SK, Lee KW, Kim HS, Yoon KC, Ahn SW, Choi JY, et al. Optimal bile duct division using real-time indocyanine green near-infrared fluorescence cholangiography during laparoscopic donor hepatectomy. Liver Transpl. 2017;23(6):847-52.

9. Aoki T, Koizumi T, Mansour DA, Fujimori A, Kusano T, Matsuda K, et al. Ultrasound-Guided Preoperative Positive Percutaneous Indocyanine Green Fluorescence Staining for Laparoscopic Anatomical Liver Resection. J Am Coll Surg. 2020;230(3):e7-e12.

10. Inoue Y, Arita J, Sakamoto T, Ono Y, Takahashi M, Takahashi Y, et al. Anatomical Liver Resections Guided by 3-Dimensional Parenchymal Staining Using Fusion Indocyanine Green Fluorescence Imaging. Ann Surg. 2015;262(1):105-11.

11. Nishino H, Hatano E, Seo S, Nitta T, Saito T, Nakamura M, et al. Real-time Navigation for Liver Surgery Using Projection Mapping With Indocyanine Green Fluorescence: Development of the Novel Medical Imaging Projection System. Ann Surg. 2018;267(6):1134-40.

12. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg. 2010;8(5):336-41.

13. Ozgur E, Koo B, Le Roy B, Buc E, Bartoli A. Preoperative liver registration for augmented monocular laparoscopy using backward-forward biomechanical simulation. Int J Comput Assist Radiol Surg. 2018;13(10):1629-40.

14. Prevost GA, Eigl B, Paolucci I, Rudolph T, Peterhans M, Weber S, et al. Efficiency, Accuracy and Clinical Applicability of a New Image-Guided Surgery System in 3D Laparoscopic Liver Surgery. J Gastrointest Surg. 2020;24(10):2251-8.

15. Kenngott HG, Wagner M, Gondan M, Nickel F, Nolden M, Fetzer A, et al. Real-time image guidance in laparoscopic liver surgery: first clinical experience with a guidance system based on intraoperative CT imaging. Surg Endosc. 2014;28(3):933-40.

16. Espinel Y, Ozgur E, Calvet L, Le Roy B, Buc E, Bartoli A. Combining Visual Cues with Interactions for 3D-2D Registration in Liver Laparoscopy. Ann Biomed Eng. 2020;48(6):1712-27.
17. Tang R, Ma L, Xiang C, Wang X, Li A, Liao H, et al. Augmented reality navigation in open surgery for hilar cholangiocarcinoma resection with hemihepatectomy using video-based in situ three-dimensional anatomical modeling: A case report. Medicine (Baltimore). 2017;96(37):e8083.

18. Zhang P, Luo H, Zhu W, Yang J, Zeng N, Fan Y, et al. Real-time navigation for laparoscopic hepatectomy using image fusion of preoperative 3D surgical plan and intraoperative indocyanine green fluorescence imaging. Surg Endosc. 2020;34(8):3449-59.

19. Onda S, Okamoto T, Kanehira M, Fujioka S, Suzuki N, Hattori A, et al. Short rigid scope and stereo-scope designed specifically for open abdominal navigation surgery: clinical application for hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci. 2013;20(4):448-53.

20. Hallet J, Soler L, Diana M, Mutter D, Baumert TF, Habersetzer F, et al. Trans-thoracic minimally invasive liver resection guided by augmented reality. J Am Coll Surg. 2015;220(5):e55-60.

21. Conrad C, Fusaglia M, Peterhans M, Lu H, Weber S, Gayet B. Augmented Reality Navigation Surgery Facilitates Laparoscopic Rescue of Failed Portal Vein Embolization. J Am Coll Surg. 2016;223(4):e31-4.

22. Okamoto T, Onda S, Matsumoto M, Gocho T, Futagawa Y, Fujioka S, et al. Utility of augmented reality system in hepatobiliary surgery. J Hepatobiliary Pancreat Sci. 2013;20(2):249-53.

23. Bertrand LR, Abdallah M, Espinel Y, Calvet L, Pereira B, Ozgur E, et al. A case series study of augmented reality in laparoscopic liver resection with a deformable preoperative model. Surg Endosc. 2020;34(12):5642-8.

24. Nicolau SA, Pennec X, Soler L, Buy X, Gangi A, Ayache N, et al. An augmented reality system for liver thermal ablation: design and evaluation on clinical cases. Med Image Anal. 2009;13(3):494-506.

25. Adballah M, Espinel Y, Calvet L, Pereira B, Le Roy B, Bartoli A, et al. Augmented reality in laparoscopic liver resection evaluated on an ex-vivo animal model with pseudo-tumours. Surg Endosc. 2022;36(1):833-43.

26. Thompson S, Schneider C, Bosi M, Gurusamy K, Ourselin S, Davidson B, et al. In vivo estimation of target registration errors during augmented reality laparoscopic surgery. Int J Comput Assist Radiol Surg. 2018;13(6):865-74.

27. Yasuda J, Okamoto T, Onda S, Futagawa Y, Yanaga K, Suzuki N, et al. Novel navigation system by augmented reality technology using a tablet PC for hepatobiliary and pancreatic surgery. Int J Med Robot. 2018;14(5):e1921.

28. Zhang F, Zhang S, Sun L, Zhan W, Sun L. Research on registration and navigation technology of augmented reality for ex-vivo hepatectomy. Int J Comput Assist Radiol Surg. 2022;17(1):147-55.

29. Souzaki R, Ieiri S, Uemura M, Ohuchida K, Tomikawa M, Kinoshita Y, et al. An augmented reality navigation system for pediatric oncologic surgery based on preoperative CT and MRI images. J Pediatr Surg. 2013;48(12):2479-83.

30. Buchs NC, Volonte F, Pugin F, Toso C, Fusaglia M, Gavaghan K, et al. Augmented environments for the targeting of hepatic lesions during image-guided robotic liver surgery. J Surg Res. 2013;184(2):825-31.

31. Ntourakis D, Memeo R, Soler L, Marescaux J, Mutter D, Pessaux P. Augmented Reality Guidance for the Resection of Missing Colorectal Liver Metastases: An Initial Experience. World J Surg. 2016;40(2):419-26.

32. Volonte F, Pugin F, Bucher P, Sugimoto M, Ratib O, Morel P. Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion. J Hepatobiliary Pancreat Sci. 2011;18(4):506-9.
33. Kang X, Azizian M, Wilson E, Wu K, Martin AD, Kane TD, et al. Stereoscopic augmented reality for laparoscopic surgery. Surg Endosc. 2014;28(7):2227-35.

34. Haouchine N, Cotin S, Peterlik I, Dequidt J, Lopez MS, Kerrien E, et al. Impact of Soft Tissue Heterogeneity on Augmented Reality for Liver Surgery. IEEE Trans Vis Comput Graph. 2015;21(5):584-97.

35. Teatini A, Perez de Frutos J, Eigl B, Pelanis E, Aghayan DL, Lai M, et al. Influence of sampling accuracy on augmented reality for laparoscopic image-guided surgery. Minim Invasive Ther Allied Technol. 2021;30(4):229-38.

36. Pessaux P, Diana M, Soler L, Piardi T, Mutter D, Marescaux J. Towards cybernetic surgery: robotic and augmented reality-assisted liver segmentectomy. Langenbecks Arch Surg. 2015;400(3):381-5.

37. Luo H, Yin D, Zhang S, Xiao D, He B, Meng F, et al. Augmented reality navigation for liver resection with a stereoscopic laparoscope. Comput Methods Programs Biomed. 2020;187:105099.

38. Lau LW, Liu X, Plishker W, Sharma K, Shekhar R, Kane TD. Laparoscopic Liver Resection with Augmented Reality: A Preclinical Experience. J Laparoendosc Adv Surg Tech A. 2019;29(1):88-93.

39. Schneider C, Thompson S, Totz J, Song Y, Allam M, Sodergren MH, et al. Comparison of manual and semi-automatic registration in augmented reality image-guided liver surgery: a clinical feasibility study. Surg Endosc. 2020;34(10):4702-11.

40. Golse N, Petit A, Lewin M, Vibert E, Cotin S. Augmented Reality during Open Liver Surgery Using a Markerless Non-rigid Registration System. J Gastrointest Surg. 2021;25(3):662-71.

41. Allard J, Cotin S, Faure F, Bensoussan PJ, Poyer F, Duriez C, et al. SOFA--an open source framework for medical simulation. Stud Health Technol Inform. 2007;125:13-8.

42. Cuchet E, Knoplioch J, Dormont D, Marsault C. Registration in neurosurgery and neuroradiotherapy applications. J Image Guid Surg. 1995;1(4):198-207.

43. Bernard F, Haemmerli J, Zegarek G, Kiss-Bodolay D, Schaller K, Bijlenga P. Augmented reality-assisted roadmaps during periventricular brain surgery. Neurosurg Focus. 2021;51(2):E4.

44. Keating TC, Jacobs JJ. Augmented Reality in Orthopedic Practice and Education. Orthop Clin North Am. 2021;52(1):15-26.

45. Ferrari V, Cattari N, Fontana U, Cutolo F. Parallax Free Registration for Augmented Reality Optical See-Through Displays in the Peripersonal Space. IEEE Trans Vis Comput Graph. 2022;28(3):1608-18.

46. Cahais J, Schwarz L, Bridoux V, Huet E, Tuech JJ. Is the image "right" for everyone? Introduction to the parallax effect in laparoscopic surgery. J Visc Surg. 2017;154(1):11-4.
screening

n=99

n=3 (snowballing methodology)

eligible

n=102

n=28 irrelevant

n=14 other organ

n=8 reviews

n=9 phantoms

n=7 percutaneous

n=3 commentaries

n=2 letters to the editor

n=2 Chinese

n=1 German

inclus

74 exclusions

n=28
**Figure 2:**

Example of deep mapping (=tridimensional, B) from a bi-
dimensional laparoscopic acquisition (A) to enable registration of
the digital model.

(Personal data, Chaire BOPA/Inria)

<figure><img src="image_2.png" /></figure>