# Interobserver Variability in the Assessment of Fluorescence Angiography in the Colon

Antonio S Soares, Neil T Clancy, Sophia Bano, Imran Raza, Michele Diana,

Laurence B Lovat, Danail Stoyanov, Manish Chand

► To cite this version:

Antonio S Soares, Neil T Clancy, Sophia Bano, Imran Raza, Michele Diana, et al. Interobserver Variability in the Assessment of Fluorescence Angiography in the Colon. Surgical Innovation, 2022, 30 (1), pp.45 - 49. 10.1177/15533506221132681. hal-05129101

HAL Id: hal-05129101

https://univoak.hal.science/hal-05129101v1

Submitted on 25 Jun 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Interobserver Variability in the Assessment of Fluorescence Angiography in the Colon

Antonio S. Soares, MD<sup>1,2 ID</sup>, Neil T. Clancy, PhD<sup>1,3</sup>, Sophia Bano, PhD<sup>1,4</sup>, Imran Raza, MD, PhD<sup>5</sup>, Michelle Diana, MD, PhD<sup>6,7 ID</sup>, Laurence B. Lovat, BSc, MBBS, PhD, FRCP<sup>1,2</sup>, Danail Stoyanov, PhD<sup>1,4</sup>, and Manish Chand, FRCS, MBA, FASCRS, PhD<sup>1,2</sup>

Surgical Innovation
2023, Vol. 30(1) 45–49
© The Author(s) 2022

<figure><img src="image_2.png" /></figure>

Article reuse guidelines:
sagepub.com/journals-permissions
DOI: 10.1177/15533506221132681
journals.sagepub.com/home/sri

<figure><img src="image_3.png" /></figure>

S sage

## Abstract

**Background:** Fluorescence angiography in colorectal surgery is a technique that may lead to lower anastomotic leak rates. However, the interpretation of the fluorescent signal is not standardised and there is a paucity of data regarding interobserver agreement. The aim of this study is to assess interobserver variability in selection of the transection point during fluorescence angiography before anastomosis.

**Methods:** An online survey with still images of fluorescence angiography was distributed through colorectal surgery channels containing images from 13 patients where several areas for transection were displayed to be chosen by raters. Agreement was assessed overall and between pre-planned rater cohorts (experts vs non-experts; trainees vs consultants; colorectal specialists vs non colorectal specialists), using Fleiss' kappa statistic.

**Results:** 101 raters had complete image ratings. No significant difference was found between raters when choosing a point of optimal bowel transection based on fluorescence angiography still images. There was no difference between pre-planned cohorts analysed (experts vs non-experts; trainees vs consultants; colorectal specialists vs non colorectal specialists). Agreement between these cohorts was poor (<.26).

**Conclusion:** Whilst there is no learning curve for the technical adoption of FA, understanding the fluorescent signal characteristics is key to successful use. We found significant variation exists in interpretation of static fluorescence angiography data. Further efforts should be employed to standardise fluorescence angiography assessment.

## Keywords

colorectal surgery, image guided surgery, ergonomics and/or human factors study

## Introduction

Anastomotic leak (AL) in colorectal surgery is still common today<fnref n="1" /> and causes significant patient morbidity<fnref n="2" /> and costs to the health system.<fnref n="3" /> Several strategies have been employed to avoid this complication and minimize its effects, ranging from better patient stratification<fnref n="4" /> to closer monitoring postoperatively (both biochemical<fnref n="5" /> and radiological<fnref n="6" />). Decision making with regard to the anastomotic level and thus perfusion of the bowel conduit is a main determinant in AL. Intraoperatively, the point of proximal transection is based on the extent of disease that is being resected and by assessment of the proximal colon viability. This decision is most commonly based on clinical judgement through the detection of marginal artery pulse, bleeding from the cut edges of the mesentery and tissue colour.

## Corresponding Author

Manish Chand, University College London, Charles Bell House, 43-45 Foley Street, London W1W 7JN, UK.

Email: m.chand@ucl.ac.uk

<footnote n="1">¹Wellcome EPSRC Centre for Interventional and Surgical Sciences (WEISS), London, UK</footnote>

<footnote n="2">²Division of Surgery and Interventional Sciences, University College London, London, UK</footnote>

<footnote n="3">³Department of Medical Physics and Biomedical Engineering, University College London, London, UK</footnote>

<footnote n="4">⁴Department of Computer Science, University College London, London, UK</footnote>

<footnote n="5">⁵University College London Hospital, University College London Hospitals NHS Trust, London, UK</footnote>

<footnote n="6">⁶IRCAD, Research Institute Against Digestive Cancer, Strasbourg, France</footnote>

<footnote n="7">⁷ICube Lab, Photonics for Health, University of Strasbourg, France</footnote>
Recently, fluorescence angiography with indocyanine green (ICG) have been used to better assess bowel perfusion and subsequently, choice of transection point. After intravenous injection, ICG binds to albumin in the intravascular compartment and travels through the circulation. It will be present in all tissue that is perfused, and it can be visualised using a specialised camera system incorporating near infrared imaging. The use of fluorescence angiography has been shown to significantly reduce the odds ratio for anastomotic leak in observational studies.⁷,⁸

However, for FA to be useful it relies on meaningful interpretation of the displayed visual image. There is no documented learning curve for the technical aspects of FA, ie administering the injection, which is understandable but there is also no recommended learning curve for the interpretation of the images. And it is ultimately image interpretation that leads to the clinical decision making. This is often inaccurately referred to as the ‘number of cases used’. Previous work has compared expert assessment with non-experts using a dynamic video assessment tool,⁹ however no consistent cut off was used to define expert. A better understanding of how to standardize the interpretation of ICG images and potentially, build a quantitative model to help in clinical decision making, is a current unmet need.

The aim of this study was to identify user variability in the interpretation of ICG fluorescence angiography and to further determine whether level of seniority and/or experience influenced clinical decision-making when using ICG.

## Methods

### Setting

An online survey, containing 15 pre-selected images showing the proximal colon before anastomosis, was created to assess user interpretation. Users were asked to select the optimal point of transection from a series of options.

### Image selection

Intraoperative video from a random selection of patients that had undergone surgery and bowel resection using ICG to help in decision making was used. All videos were from a single surgical team at a tertiary referral centre for colorectal surgery. The technique for use of ICG has been extensively described elsewhere.¹⁰ A still image was selected from the video footage that corresponded to maximal fluorescence intensity for that case as assessed by the operating surgeon. For each image, the length of colon was divided into equal segments representing the transection options as shown in Figure 1 (full survey shown in supplementary figure 1). These were manually segmented along the longitudinal axis of the colon with an

approximate distance of 1 cm between segments. 1 cm cut-off has been shown to be consistent with the change in transection point when using fluorescence angiography.¹¹,¹² Randomly ordered letters were attributed to these segments to allow unbiased selection.

In total, 15 images were included, corresponding to 13 different patients. 2 patients had the same view pre- and post-fluorescence angiography (cases #8 and #10; and cases #9 and #14 respectively, supplementary figure 1), allowing a direct comparison of choices made on the same specimen.

### Participants

An online image bank was created. Participants were sent an invitation to join the study through the membership of the European Fluorescence Imaging-Guided Surgery (EURO-FIGS) registry. Each user was required to state their surgical and FA experience - previous experience with fluorescence (novice if under 5 uses of fluorescence angiography, expert otherwise); level of training (trainee or consultant); and area of specialisation (colorectal or not colorectal). The question “Would you use an automated system to standardise the decision of the transection point?” was completed by the study participants.

### Data Analysis

Users' responses were analysed and compared by case, experience, level of seniority, and area of surgical specialization. The areas selected for transection were analysed as categorical variables using a one-way ANOVA test. A P-value under .05 was considered significant. Interrater agreement was assessed between raters for the different images.¹³ The Fleiss' kappa statistic was calculated for every image.¹⁴ For comparison, 95% confidence intervals were obtained by bootstrapping using 100 repetitions with replacement. Stata (StataCorp., Texas, USA) version 14.1 was used for statistical analysis. Raters were excluded for agreement analysis if not all selections were entered.

### Ethics

All patients provided written informed consent for video data collection during surgery according to institutional guidelines and no patient identifiable data was used for the present study. No identifiable data was collected about participants in the survey and therefore no ethical approval was necessary.

## Results

The study included 108 participants. 7 participants were excluded from agreement analysis due to incomplete
<figure><img src="image_4.png" /><figcaption>**Figure 1.** Case #1 image with categories to be selected.</figcaption></figure>

<table><caption>**Table 1.** Details of Study Participants.</caption><thead><tr><th>Level of Training</th><th>% (n)</th></tr></thead><tbody><tr><td>Consultants</td><td>64.8% (70)</td></tr><tr><td>Trainees</td><td>33.3% (36)</td></tr><tr><td>Missing data</td><td>1.9% (2)</td></tr><tr><td>Fluorescence experience</td><td></td></tr><tr><td>Novice</td><td>42.6% (46)</td></tr><tr><td>Expert</td><td>56.5% (61)</td></tr><tr><td>Missing data</td><td>.9% (1)</td></tr><tr><td>Area of specialisation</td><td></td></tr><tr><td>Colorectal</td><td>79.6% (86)</td></tr><tr><td>Not colorectal</td><td>19.4% (21)</td></tr><tr><td>Missing data</td><td>.9% (1)</td></tr><tr><td>Exclusions</td><td>6.5% (7)</td></tr></tbody></table>

<table><caption>**Table 3.** Agreement Analysis.</caption><thead><tr><th></th><th># Raters</th><th>Agreement</th><th>95% CI</th></tr></thead><tbody><tr><td>Consultants</td><td>66</td><td>.24</td><td>.169 - .342</td></tr><tr><td>Trainees</td><td>32</td><td>.205</td><td>.153 - .262</td></tr><tr><td>Fluorescence experience</td><td></td><td></td><td></td></tr><tr><td>Novice</td><td>43</td><td>.206</td><td>.154 - .260</td></tr><tr><td>Expert</td><td>57</td><td>.251</td><td>.198 - .308</td></tr><tr><td>Area of specialisation</td><td></td><td></td><td></td></tr><tr><td>Colorectal</td><td>80</td><td>.224</td><td>.170 - .300</td></tr><tr><td>Not colorectal</td><td>19</td><td>.258</td><td>.173 - .337</td></tr></tbody></table>

<table><caption>**Table 2.** Differences in Mean Normalised Score Between Different Rater Cohorts.</caption><thead><tr><th></th><th># Ratings/Group</th><th>Mean Normalised Score</th><th>P-Value</th></tr></thead><tbody><tr><td>Level of training</td><td></td><td></td><td></td></tr><tr><td>Consultants</td><td>903</td><td>.622</td><td>.37</td></tr><tr><td>Trainees</td><td>454</td><td>.634</td><td></td></tr><tr><td>Fluorescence experience</td><td></td><td></td><td></td></tr><tr><td>Novice</td><td>597</td><td>.626</td><td>.98</td></tr><tr><td>Expert</td><td>786</td><td>.627</td><td></td></tr><tr><td>Area of specialisation</td><td></td><td></td><td></td></tr><tr><td>Colorectal</td><td>1115</td><td>.627</td><td>.98</td></tr><tr><td>Not colorectal</td><td>255</td><td>.628</td><td></td></tr></tbody></table>
ratings on all images. The details of the participants are included in Table 1.

Overall, there was no significant difference in the point of transection between any of the groups – consultant vs trainees; novice vs expert; or colorectal vs general – Table 2.

The participant scores for each individual case and statistical analysis can be found in supplementary table 1.

Within each cohort, there was a poor level of agreement in responses. This was consistent throughout the different cohort categories as can be seen in Table 3.

Of all participants, 68.5% replied yes to the question "Would you use an automated system to standardise the decision of the transection point?".

## Discussion and Conclusions

The results of the present study show that there was no significant difference between the users when choosing a point of optimal bowel transection based on fluorescence angiography. There was no difference between consultants and trainees, fluorescence novices and experts, or colorectal subspecialists within colorectal surgery and generalists. Furthermore, there was a poor level of agreement within each cohort. These results reflect that the interpretation of fluorescence angiography images is not associated with expertise, seniority, or sub-specialism.

There is increasing evidence to suggest fluorescence angiography has a beneficial effect in reducing the risk of anastomotic leak in colorectal surgery. There have been a number of review articles recently¹⁵,¹⁶ which have shown this to be the case despite the lack of a large scale, formal randomised controlled trial. However, as with any technology or new technique it is important to be able to safely deploy so that the benefits may be obtained without any concerns over safety or complications. In the case of fluorescence angiography, the technique may be simple however any associated complications would be associated with misinterpretation of the images and thus sub-optimal clinical decision making. This is a novel situation for surgical technologies and is perhaps more attuned with radiology training which is determined by image interpretation. An example would be examining an x-ray and being able to interpret a fracture from normal bone based on the image characteristics.

Intuitively, one would have expected there to have been a difference between senior surgeons and trainees in determining the optimal transection point but that has not been the case in this study. That may be explained by the fact that optimal transection point could be argued to be a function of perfused bowel and maximal colonic length rather than just perfusion. It may be that the more junior surgeons have been more cautious whereas the more senior surgeons are attempting to maximise colonic length as well as perfusion.

A similar study by Nial et al.⁹ has also shown the variability amongst users of fluorescence angiography. In their study, video footage was used rather than still images with the raters being asked to position the optimal transection point rather than choosing from pre-selected options. Despite the slightly different methodology the results of the study were similar. This suggests that there is a need to incorporate formal image interpretation training as part of the implementation of fluorescence angiography for new users. Developing such a program would further augment the user's ability to deploy fluorescence angiography in a more effective manner.

This study has certain limitations. The first is that we used still images rather than video. This means that the rater is unable to appreciate the dynamic flow of ICG into the tissue which could perhaps influence their decision-making. Using still images is possible as we had chosen frames which showed maximal peak intensity. The second limitation is the quality of the images themselves. As is the case in fluorescence angiography, the intensity of the fluorescence is not always reflective of perfusion, and this can be difficult to judge in such a survey. The rater may feel that the bowel is not adequately perfused when indeed this is the maximal peak intensity for that case. The third limitation is the rather rudimentary and arbitrary categorisation of experts, trainees, specialists, and generalists. There is currently no universal definition for this. Understanding what makes a user and expert is part of the challenge in fluorescence angiography.

In conclusion, the interpretation of fluorescence angiography images remains variable. There is no validated oral consensus agreement on the number of cases required to reach competency or expertise in interpretation. We suggest that in addition to training in the actual technique of administering ICG, there should be image interpretation training in parallel.

## Author Contribution

Conceptualization: AS Soares, Neil T Clancy, S Bano, M Chand

Methodology: AS Soares, M Chand, LB Lovat

Writing - Review & Editing: AS Soares, Neil T Clancy, S Bano, M Diana, LB Lovat, D Stoyanov, M Chand

## Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

## Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS) (203145Z/16/Z); Engineering and Physical Sciences Research Council (EPSRC) [EP/P027938/1, EP/R004080/1, EP/P012841/1] and The Royal
Academy of Engineering Chair in Emerging Technologies Scheme.

## Disclosures

Michelle Diana has received travelling support from Diagnostic Green and consulting fees from Intuitive Surgical, unrelated to the present study. Manish Chand has received consulting fees from Stryker, unrelated to the present study. Antonio S Soares, Sophia Bano, Neil T Clancy, Imran Raza, Laurence B Lovat and Danail Stoyanov have no relevant conflicts of interest to be disclosed. No industry funding was used for this study.

## ORCID iDs

Antonio S. Soares https://orcid.org/0000-0001-7773-2427

Michelle Diana https://orcid.org/0000-0002-1390-8486

## Supplemental Material

Supplemental material for this article is available online.

## References

1. Bhangu A. An international multicentre prospective audit of elective rectal cancer surgery; operative approach versus outcome, including transanal total mesorectal excision (TaTME). *Colorectal Dis*. 2018;20 Suppl 6:33-46.

2. The National Emergency Laparotomy Project Team. Fourth patient report of the national emergency laparotomy audit (NELA). 2018;1-137.

3. Ashraf SQ, Burns EM, Jani A, et al. The economic impact of anastomotic leakage after anterior resections in English NHS hospitals: Are we adequately remunerating them? *Colorectal Dis*. 2013;15:e190-e198.

4. Frasson M, Flor-Lorente B, Rodríguez JL, et al. Risk factors for anastomotic leak after colon resection for cancer: Multivariate analysis and nomogram from a multicentric, prospective, national study with 3193 patients. *Ann Surg*. 2015;262:321-330.

5. Reynolds IS, Boland MR, Reilly F, et al. C-reactive protein as a predictor of anastomotic leak in the first week after anterior resection for rectal cancer. *Colorectal Dis*. 2017;19: 812-818.

6. Kauv P, Benadjaoud S, Curis E, Boulay-Coletta I, Loriau J, Zins M. Anastomotic leakage after colorectal surgery: diagnostic accuracy of CT. *Eur Radiol*. 2015;25:3543-3551.

7. Blanco-Colino R, Espin-Basany E. Intraoperative use of ICG fluorescence imaging to reduce the risk of anastomotic leakage in colorectal surgery: a systematic review and meta-analysis. *Tech Coloproctol*. 2017;22:15-23. doi:10.1007/s10151-017-1731-8

8. Degett TH, Andersen HS, Gögenur I. Indocyanine green fluorescence angiography for intraoperative assessment of gastrointestinal anastomotic perfusion: a systematic review of clinical trials. *Langenbeck's Arch Surg*. 2016;401: 767-775.

9. Hardy NP, Dalli J, Khan MF, Andrejevic P, Neary PM, Cahill RA. Inter-user variation in the interpretation of near infrared perfusion imaging using indocyanine green in colorectal surgery. *Surg Endosc*. 2021;1:3-7081.

10. Keller DS, Ishizawa T, Cohen R, Chand M. Indocyanine green fluorescence imaging in colorectal surgery: overview, applications, and future directions. *Lancet. Gastroenterol. Hepatol*. 2017;2:757-766.

11. Ris F, Liot E, Buchs NC, et al. Multicentre phase II trial of near-infrared imaging in elective colorectal surgery. *Br J Surg*. 2018;105:1359-1367.

12. Jafari MD, Wexner SD, Martz JE, et al. Perfusion assessment in laparoscopic left-sided/anterior resection (PILLAR II): A multi-institutional study. *J Am Coll Surg*. 2015;220: 82-e1.

13. Mitani AA, Freer PE, Nelson KP. Summary measures of agreement and association between many raters' ordinal classifications. *Ann Epidemiol*. 2017;27:677e4-e4.

14. Hernaez R. Reliability and agreement studies: A guide for clinical investigators. *Gut*. 2015;64:1018-1027.

15. Lin J, Zheng B, Lin S, Chen Z, Chen S. The efficacy of intraoperative ICG fluorescence angiography on anastomotic leak after resection for colorectal cancer: a meta-analysis. *Int J Colorectal Dis*. 2020;361 36:27-39.

16. Liu D, Liang L, Liu L, Zhu Z. Does intraoperative indocyanine green fluorescence angiography decrease the incidence of anastomotic leakage in colorectal surgery? A systematic review and meta-analysis. *Int J Colorectal Dis*. 2020;361 36:57-66.