# Concordance between SIVA, IVAN, and VAMPIRE Software Tools for Semi-Automated Analysis of Retinal Vessel Caliber

Thibaud Mautuit, Pierre Cunnac, Carol Cheung, Tien Wong, Stephen Hogg, Emanuele Trucco, Vincent Daien, Thomas Macgillivray, José Labarere, Christophe Chiquet

<figure><img src="image_1.png" /></figure>

## To cite this version:

Thibaud Mautuit, Pierre Cunnac, Carol Cheung, Tien Wong, Stephen Hogg, et al. Concordance between SIVA, IVAN, and VAMPIRE Software Tools for Semi-Automated Analysis of Retinal Vessel Caliber. Diagnostics, 2022, 12 (6), pp.1317. 10.3390/diagnostics12061317. hal-04471675

HAL Id: hal-04471675

https://hal.science/hal-04471675v1

Submitted on 21 Feb 2024

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Article

# Concordance between SIVA, IVAN, and VAMPIRE Software Tools for Semi-Automated Analysis of Retinal Vessel Caliber

Thibaud Mautuit<fnref n="1" />,<fnref n="2" />,*, Pierre Cunnac<fnref n="1" />,<fnref n="2" />, Carol Y. Cheung<fnref n="3" />, Tien Y. Wong<fnref n="4" />, Stephen Hogg<fnref n="5" />, Emanuele Trucco<fnref n="5" />, Vincent Daien<fnref n="6" />, Thomas J. MacGillivray<fnref n="7" />, José Labarère<fnref n="8" />,<fnref n="9" />, and Christophe Chiquet<fnref n="1" />,<fnref n="2" />,*

<figure><img src="image_3.png" /></figure>

check for
updates

**Citation:** Mautuit, T.; Cunnac, P.; Cheung, C.Y.; Wong, T.Y.; Hogg, S.; Trucco, E.; Daien, V.; MacGillivray, T.J.; Labarère, J.; Chiquet, C. *Concordance between SIVA, IVAN, and VAMPIRE Software Tools for Semi-Automated Analysis of Retinal Vessel Caliber.* *Diagnostics* **2022**, *12*, 1317. https://doi.org/10.3390/diagnostics12061317

Academic Editors: Jae-Ho Han and
Isabel Pinilla

Received: 8 March 2022

Accepted: 17 May 2022

Published: 25 May 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_4.png" /></figure>

**Copyright:** © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

**Abstract:** We aimed to compare measurements from three of the most widely used software packages in the literature and to generate conversion algorithms for measurement of the central retinal artery equivalent (CRAE) and central retinal vein equivalent (CRVE) between SIVA and IVAN and between SIVA and VAMPIRE. We analyzed 223 retinal photographs from 133 human participants using both SIVA, VAMPIRE and IVAN independently for computing CRAE and CRVE. Agreement between measurements was assessed using Bland–Altman plots and intra-class correlation coefficients. A conversion algorithm between measurements was carried out using linear regression, and validated using bootstrapping and root-mean-square error. The agreement between VAMPIRE and IVAN was poor to moderate: The mean difference was 20.2 µm (95% limits of agreement, LOA, –12.2–52.6 µm) for CRAE and 21.0 µm (95% LOA, –17.5–59.5 µm) for CRVE. The agreement between VAMPIRE and SIVA was also poor to moderate: the mean difference was 36.6 µm (95% LOA, –12.8–60.4 µm) for CRAE, and 40.3 µm (95% LOA, 5.6–75.0 µm) for CRVE. The agreement between IVAN and SIVA was good to excellent: the mean difference was 16.4 µm (95% LOA, –4.25–37.0 µm) for CRAE, and 19.3 µm (95% LOA, 0.09–38.6 µm) for CRVE. We propose an algorithm converting IVAN and VAMPIRE measurements into SIVA-estimated measurements, which could be used to homogenize sets of vessel measurements obtained with different software packages.

**Keywords:** central retinal artery equivalent; central retinal vein equivalent; SIVA software; IVAN software; VAMPIRE software; conversion algorithm; retinal vessel measurements

## 1. Introduction

Since the retinal vasculature reflects systemic microvascular damage, the analysis of retinal fundus photographs offers a noninvasive method for estimating cardiovascular risk. Several health outcomes are associated with changes in the diameter of retinal vessels, such as systemic hypertension, stroke, and non-pathological cognitive aging [1–9].

A quantitative description of the retinal vessel network can be provided by several semi-automatic software tools. Three common software packages reported in the literature are IVAN (Integrative Vessel Analysis), [10–14] SIVA (Singapore I Vessel Assess-

<footnote n="1">¹ HP2 Laboratory, INSERM U1300, Univ. Grenoble Alpes, 38700 La Tronche, France; t.mautuit21@orange.fr (T.M.); pierre.cunnac@gmail.com (P.C.)</footnote>

<footnote n="2">² Department of Ophthalmology, University Hospital of Grenoble Alps, CEDEX 09, 38043 Grenoble, France</footnote>

<footnote n="3">³ Department of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong, China; carolcheung@cuhk.edu.hk</footnote>

<footnote n="4">⁴ Singapore Eye Research Institute, Yong Loo Ling School of Medicine, National University of Singapore, Singapore 119077, Singapore; ophwty@nus.edu.sg</footnote>

<footnote n="5">⁵ VAMPIRE Project, School of Computing, University of Dundee, Dundee DD1 4HN, UK; s.c.z.hogg@dundee.ac.uk (S.H.); e.trucco@dundee.ac.uk (E.T.)</footnote>

<footnote n="6">⁶ Department of Ophthalmology, Gui De Chauliac Hospital, 34295 Montpellier, France; v-daien@chu-montpellier.fr</footnote>

<footnote n="7">⁷ VAMPIRE Project, Clinical Research Imaging Centre, University of Edinburgh, Edinburgh EH8 9YL, UK; t.j.macgillivray@ed.ac.uk</footnote>

<footnote n="8">⁸ Clinical Epidemiology Unit, Grenoble University Hospital, 38043 Grenoble, France; jlabarere@chu-grenoble.fr</footnote>

<footnote n="9">⁹ TIMC-IMAG UMR 5525, CNRS, Univ. Grenoble Alpes, 38041 Grenoble, France</footnote>

<footnote>* Correspondence: christophe.chiquet@inserm.fr; Tel.: +33-4-7676-8457</footnote>
ment), [15,16] and VAMPIRE (Vascular Assessment and Measurement Platform for Images of the Retina) [17–19]. All of them include three well-known parameters summarizing vessel widths around the optic disc: CRAE (central retinal artery equivalent), CRVE (central retinal vein equivalent), and arteriovenous ratio (AVR). However, the software packages use different algorithms for vessel segmentation, for vessel labeling (artery or vein), and for estimating vascular diameter. Two additional features are estimated with SIVA and VAMPIRE but not with IVAN: tortuosity and fractal dimension.

Previous studies suggest poor-to-moderate concordance between these software packages. A comparison between SIVA and VAMPIRE on 655 images indicated poor-to-limited agreement for all parameters (0.16–0.41) and the presence of proportional and systematic bias in the majority of the parameters [20]. IVAN, SIVA, and RA (Retinal Analysis) have been compared on a series of 120 retinal photographs, but the intra-class correlation to assess concordance was not reported [21]. Nevertheless, the authors showed that IVAN yields significantly larger retinal vessel measurements than SIVA and they proposed an algorithm to convert IVAN measurements into estimated SIVA measurements. This approach seems attractive for pooling data from different studies.

The poor concordance that has been reported implies that the software packages are not interchangeable. There is a need to clarify the quantitative differences between their measurements, which may lead to different conclusions in statistical studies using the same image set. This calls for a systematic and structured comparison of these three software applications.

The primary aim of our study was to determine the agreement of CRAE, CRVE, and AVR estimates from the three software tools. Our secondary aim was to propose an algorithm converting measurements between these packages.

## 2. Materials and Methods

### 2.1. *Study Population*

We included 242 retinal photographs of eyes exempt of ocular condition from 133 patients from the Department of Ophthalmology of University Hospital of Grenoble Alps. All participants had a complete clinical examination, including refraction, visual acuity, measurement of intraocular pressure, slit-lamp examination for the anterior and posterior segment. All the funduscopy color photographs were 45-degree images and obtained with the same CR-2 Canon fundus camera (Canon™) after pupil dilation, at a resolution of 5184 × 3456 pixels. Overall, 19 images were excluded because of bad quality or because they were not centered on the optic disc. Good photographic quality was determined subjectively on the basis of the following criteria: focus (perceived sharpness of the vessel edge against the background retinal pigment epithelium), illumination (even and consistent lighting across the zone of vessel measurement), and color balance (equal saturation of color channels). Therefore, 223 images were considered for the analysis. This study complied with the Declaration of Helsinki guidelines for research involving human subjects. The Institutional review board (IRB# 5921) reviewed and approved the study protocol, as part of the ongoing prospective observational IMAGEYE cohort study. All study participants provided informed consent for all the ophthalmologic examinations and agreed that anonymized data could be used for clinical research.

### 2.2. *Retinal Image Analysis*

The measurement procedure has been described previously for VAMPIRE [22–24], IVAN [25], and SIVA [15]. Each image selected was analyzed with the three software tools. The interface of each software is briefly represented in Figure 1. A single trained software operator from the Grenoble team performed IVAN analysis and VAMPIRE analysis, while SIVA analysis was carried out independently by a trained operator from the Singapore Eye Research Institute. For IVAN, a standardized ARIC [25] grid was calibrated to a fixed size according to the photograph resolution and was then manually centered on the optic disc. For SIVA and VAMPIRE, the optic disc (and also the macula for VAMPIRE) was identified
automatically by the software and adjusted manually in the case of incorrect location. Each package identified and traced the retinal arterioles and venules automatically. A trained grader then examined the traced vessels and manually corrected any incorrect vessel labels (artery or vein). IVAN and SIVA enable an operator to also modify the length of the traced vessels so as to obtain a better match.

<figure><img src="image_5.png" /><figcaption>**Figure 1.** Presentation of the interface of each software. (A) VAMPIRE software interface, (B) IVAN interface, and (C) SIVA interface.</figcaption></figure>

In all cases, the vessels are analyzed in the Zone B annulus, from 0.5 to 1 disc diameter from the disc margin. The caliber of the six main arteries and veins was summarized with the CRAE and CRVE based on the revised Knudtson-Parr-Hubbard formula [26]. AVR was then calculated.
## 2.3. Pixel-to-Micron Conversion

We applied the commonly used pixel-to-micron conversion procedure based on the assumption that the adult human optic disc diameter is 1800 µm on average [27]. Using VAMPIRE, we computed the mean optic disc diameter in pixels of the entire set of images, while the IVAN and SIVA procedure is based on an image-converting factor (ICF) calculated in a subsample of images (10%).

## 2.4. Statistical Analysis

All analyses were performed with R (version 3.5.0) [28]. Results are presented as mean ± standard deviation (SD). Pearson's correlation test was performed to evaluate the correlation between sets of measurements. The consistency in vessel diameter measurement between packages was estimated with intra-class correlation coefficients (ICC) using a two-way model, consistency definition, and single rater unit [29]. For Pearson's correlation and ICC, the results were interpreted using the following scale: 0.00-0.39 = poor; 0.40-0.69 = moderate; and 0.70-1.00 = excellent. For ICC, single-measure coefficients and 95% confidence intervals (CIs) are reported.

To assess the agreement between software, we performed Bland-Altman analysis, where the 95% limits of agreement (LOA) were defined as mean difference ±1.96 × SD [30,31].

The three software applications were compared in pairs. A one-sample *t* test comparing the mean differences (between retinal vessel caliber of two different software applications) and zero value was run to indicate the presence of systematic bias. Pearson's correlation analysis was conducted between the difference and the average (i.e., the axis of the Bland-Altman plot) to indicate the presence of proportional bias.

## 2.5. Conversion Algorithm

We propose a conversion algorithm (converting sets of measurements from different packages) derived using all the data of our sample and based on linear regression. For internal validation of our conversion model, we performed the bootstrapping procedure described by Labarère et al. [32]. Following this procedure, we generated a bootstrap sample of 1000 samples from our original sample. In order to assess the accuracy of our conversion algorithm, we studied the variation of the root-mean-square error (RMSE) in the bootstrap samples compared with the original RMSE in the original sample. Here, "error" means "difference between values from different packages." The results are expressed in 95% CI, in which the RMSE of 95% of the samples is included.

# 3. Results

The 223 retinal photographs retained for analysis were obtained from 133 participants. The mean age of the participants was 47.9 ± 22.9 years and the sex ratio was 0.6 (50 men, 83 women). The mean axial length of the retinal photographs in the sample was 23.7 ± 0.9 mm. The characteristics of the participants are listed in Table 1.

Table 2 shows the mean absolute value and standard deviation of the CRAE, CRVE, and AVR measurements derived from VAMPIRE, IVAN, and SIVA for the 223 images. The agreement between values computed using VAMPIRE, IVAN, and SIVA is described by scatterplots and Bland-Altman plots (Figures 2–4).
<table><caption>**Table 1.** Clinical characteristics of the study population.</caption><thead><tr><th>Characteristics</th><th>Participants (n = 133)<br>Mean (SD)/n (%)</th></tr></thead><tbody><tr><td>Age, years</td><td>47.9 (22.9)</td></tr><tr><td>Sex ratio male/female</td><td>0.6</td></tr><tr><td>Male</td><td>50 (37.6)</td></tr><tr><td>Female</td><td>83 (62.4)</td></tr><tr><td>Axial length, mm</td><td>23.7 (0.9)</td></tr><tr><td>Intra-ocular pressure, mmHg</td><td>14.3 (2.8)</td></tr><tr><td>Systolic blood pressure, mmHg</td><td>125.9 (12.5)</td></tr><tr><td>Diastolic blood pressure, mmHg</td><td>75.7 (9.7)</td></tr><tr><td>Heart rate, bpm</td><td>73.1 (10.7)</td></tr><tr><td>Body mass index, kg/m<sup>2</sup></td><td>22.5 (7.1)</td></tr></tbody></table>

Bpm: beat per minute.

<table><caption>**Table 2.** Retinal vessel parameters with VAMPIRE, IVAN, and SIVA. CRAE: central retinal artery equivalent, CRVE: central retinal vein equivalent, AVR: arteriole-to-venule ratio.</caption><thead><tr><th rowspan="2">Retinal Vessel Parameter</th><th colspan="3">Mean (±SD)</th></tr><tr><th>VAMPIRE</th><th>IVAN</th><th>SIVA</th></tr></thead><tbody><tr><td>CRAE (µm)</td><td>174.9 (±11.3)</td><td>154.7 (±15.7)</td><td>138.3 (±10.2)</td></tr><tr><td>CRVE (µm)</td><td>239.7 (±19.1)</td><td>218.7 (±18.8)</td><td>199.4 (±14.2)</td></tr><tr><td>AVR</td><td>0.73 (±0.054)</td><td>0.71 (±0.072)</td><td>0.69 (0.04)</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>**Figure 2.** Agreement between each software for CRAE measurements. The three software applications are compared two by two: (A) IVAN and VAMPIRE, (B) SIVA and VAMPIRE, (C) SIVA and IVAN. Scatterplots are represented above with regression lines; Bland–Altman plots are represented below.</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>**Figure 3.** Agreement between each software for CRVE measurements. The three software applications are compared two by two: (A) IVAN and VAMPIRE, (B) SIVA and VAMPIRE, (C) SIVA and IVAN. Scatterplots are represented above with regression lines; Bland–Altman plots are represented below.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>**Figure 4.** Agreement between each software for AVR. The three software applications are compared two by two: (A) IVAN and VAMPIRE, (B) SIVA and VAMPIRE, (C) SIVA and IVAN. Scatterplots are represented above with regression lines; Bland–Altman plots are represented below.</figcaption></figure>

Statistical relationships are summarized in Table 3.
<table><caption>**Table 3.** Agreement between software applications according to retinal vessel parameter. * *p* value of one-sample *t* tests comparing mean difference and zero (systematic bias). † *p* value of Pearson's correlation coefficients of the regression line (proportional bias). *r*, Pearson's correlation between the parameter values obtained with the two software applications considered. AVR: arteriole-to-venule ratio, LOA: limits of agreement.</caption><thead><tr><th>VAMPIRE and IVAN</th><th>Mean Difference</th><th>95%LOA</th><th>p Value *</th><th>r</th><th>ICC</th><th>IC 95</th><th>p Value <sup>†</sup></th></tr></thead><tbody><tr><td>CRAE (µm)</td><td>20.2</td><td>(-12.2 to 52.6)</td><td>&lt;0.001</td><td>0.29</td><td>0.27</td><td>(0.15-0.39)</td><td>&lt;0.001</td></tr><tr><td>CRVE (µm)</td><td>21.0</td><td>(-17.5 to 59.5)</td><td>&lt;0.001</td><td>0.46</td><td>0.46</td><td>(0.35-0.56)</td><td>0.82</td></tr><tr><td>AVR</td><td>0.023</td><td>(-0.12 to 0.16)</td><td>&lt;0.001</td><td>0.36</td><td>0.35</td><td>(0.23-0.46)</td><td>&lt;0.001</td></tr><tr><td>VAMPIRE and SIVA</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>CRAE (µm)</td><td>36.6</td><td>(12.8 to 60.4)</td><td>&lt;0.001</td><td>0.38</td><td>0.37</td><td>(0.25-0.48)</td><td>0.09</td></tr><tr><td>CRVE (µm)</td><td>40.3</td><td>(5.6 to 75.0)</td><td>&lt;0.001</td><td>0.47</td><td>0.45</td><td>(0.33-0.55)</td><td>&lt;0.001</td></tr><tr><td>AVR</td><td>0.037</td><td>(-0.071 to 0.15)</td><td>&lt;0.001</td><td>0.38</td><td>0.37</td><td>(0.25-0.48)</td><td>&lt;0.001</td></tr><tr><td>IVAN and SIVA</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>CRAE (µm)</td><td>16.4</td><td>(-4.25 to 37.0)</td><td>&lt;0.001</td><td>0.75</td><td>0.69</td><td>(0.61-0.75)</td><td>&lt;0.001</td></tr><tr><td>CRVE (µm)</td><td>19.3</td><td>(0.09 to 38.6)</td><td>&lt;0.001</td><td>0.86</td><td>0.83</td><td>(0.78-0.87)</td><td>&lt;0.001</td></tr><tr><td>AVR</td><td>0.015</td><td>(-0.09 to 0.11)</td><td>&lt;0.001</td><td>0.68</td><td>0.61</td><td>(0.52-0.68)</td><td>&lt;0.001</td></tr></tbody></table>

### 3.1. Between VAMPIRE and IVAN

The mean difference was 20.2 µm (95% LOA, -12.2-52.6 µm) for CRAE measurements, 21.0 µm (95% LOA, -17.5-59.5 µm) for CRVE measurements, and 0.023 (95% LOA, -0.12-0.16) for AVR values. Correlation was poor for CRAE and AVR (ICC = 0.29 and 0.35, respectively) and moderate for CRVE (ICC = 0.46). A proportional bias for CRAE and AVR was shown by a significant correlation between the difference and the average (*p* < 0.001 for CRAE and AVR), as represented in the Bland-Altman plot. This proportional bias was not observed for CRVE (*p* = 0.82).

### 3.2. Between VAMPIRE and SIVA

The mean difference was 36.6 µm (95% LOA, -12.8-60.4 µm) for CRAE measurements, 40.3 µm (95% LOA, 5.6-75.0 µm) for CRVE measurements, and 0.037 (95% LOA, -0.071-0.15) for AVR values. The correlation for CRAE and AVR between each software was poor (ICC = 0.37 and 0.38, respectively) and moderate for CRVE (ICC = 0.47). A significant proportional bias was observed for CRVE and AVR (*p* < 0.001) but not for CRAE (*p* = 0.09).

### 3.3. Between IVAN and SIVA

The mean difference was 16.4 µm (95% LOA, -4.25-37.0 µm) for CRAE measurements, 19.3 µm (95% LOA, 0.09-38.6 µm) for CRVE measurements, and 0.015 (95% LOA, -0.09-0.11) for AVR values. The correlation for CRAE and AVR between each software was good (ICC = 0.69 and 0.61, respectively) and excellent for CRVE (ICC = 0.83). A proportional bias was observed for all the retinal vessel parameters (*p* < 0.001 for all).

For each pair of packages and for each parameter, the presence of systematic bias was demonstrated by the significance (*p* < 0.001) of one-sample *t* tests comparing mean differences and zero value.

To determine a conversion method between measurements from different packages, we computed linear regression relationships for CRAE and CRVE measurements.

### 3.4. Between SIVA and IVAN

$$ \text{IVAN-derived SIVA CRAE} = 0.4895 \times \text{IVAN-measured CRAE} + 62.6017 $$

$$ \text{IVAN-derived SIVA CRVE} = 0.6518 \times \text{IVAN-measured CRVE} + 56.8346 $$
### 3.5. Between SIVA and VAMPIRE

$$ \text{VAMPIRE-derived SIVA CRAE} = 0.3388 \times \text{VAMPIRE-measured CRAE} + 79.0913 $$

$$ \text{VAMPIRE-derived SIVA CRVE} = 0.3503 \times \text{VAMPIRE-measured CRVE} + 115.4105 $$

Table 4 describes the comparisons between SIVA parameters and VAMPIRE-derived SIVA parameters or IVAN-derived parameters. In all cases, the mean of the differences between each sample was insignificant. Between SIVA measurements and VAMPIRE-derived SIVA approximates, the RMSE was 7.27 µm (6.48–8.15) for CRAE and 9.69 µm (8.71–10.79) for CRVE. Between SIVA measurements and IVAN-derived SIVA approximates, the RMSE was 4.88 µm (4.35–5.60) for CRAE and 5.19 µm (4.66–6.03) for CRVE.

<table><caption>**Table 4.** Assessment of the robustness of the conversion algorithm using bootstrapping. The root-mean-square error between (RMSE) SIVA measurements and VAMPIRE-derived SIVA approximates or IVAN-derived SIVA approximates (RMSE in µm) is included in the ICC interval obtained in 1000 bootstrap samples. AVR: arteriole-to-venule ratio, CRAE: central retinal artery equivalent, CRVE: central retinal vein equivalent.</caption><thead><tr><th colspan="3">Between SIVA Measurements and VAMPIRE-Derived SIVA Approximate</th></tr><tr><th></th><th>RMSE (µm)</th><th>95% CI</th></tr></thead><tbody><tr><td>CRAE</td><td>7.27</td><td>(6.48–8.15)</td></tr><tr><td>CRVE</td><td>9.69</td><td>(8.71–10.79)</td></tr></tbody></table><table><thead><tr><th colspan="3">Between SIVA Measurements and IVAN-Derived SIVA Approximate</th></tr><tr><th></th><th>RMSE (µm)</th><th>95% CI</th></tr></thead><tbody><tr><td>CRAE</td><td>4.88</td><td>(4.35–5.60)</td></tr><tr><td>CRVE</td><td>5.19</td><td>(4.66–6.03)</td></tr></tbody></table>

## 4. Discussion

Our study analyzed the agreement and correlation between CRAE, CRVE, and AVR as measured by three widely used software tools—IVAN, SIVA, and VAMPIRE—with 233 fundus camera images of 133 retinas of healthy adults. Our analysis showed an excellent agreement between SIVA and IVAN, but a poor-to-moderate agreement between VAMPIRE and the two other software tools. In addition, we proposed a simple method for converting the measurements from one tool into estimates from another.

The evaluation of software tools such as SIVA, IVAN, and VAMPIRE can follow two different approaches. First, one can concentrate on morphological measurements of the retinal vascular tree, typically vessel diameters (as carried out here), tortuosity, junction/bifurcations, and fractal dimension. Second, one can study the association between vascular parameters and systematic parameters, or disease state or risk. In previous studies, the correlation with systematic parameters was found not to be significantly different between SIVA and IVAN [21], and between SIVA and VAMPIRE for the main vascular parameters (hypertension, systolic blood pressure, diastolic blood pressure, and HbA1c) [20,33].

For ophthalmologists, retinal vessel diameter can be considered a standard measurement from fundus images, such as, intraocular pressure, anterior chamber depth, corneal thickness, and axial length in routine practice. In this sense, using morphological measurements of retinal vessel diameters (the first approach) is of major interest. As a quantitative measure, this parameter provides objective information on the vascular retinal tree and could prove useful for the monitoring of ocular diseases, such as retinal vascular occlusion, diabetic retinopathy, or glaucoma. From this point of view, it may become mandatory to compare these retinal measurements in patients for vascular evaluation at baseline, during follow-up, and before and after treatment.

The agreement between IVAN and SIVA was higher than the agreement between IVAN and VAMPIRE or SIVA and VAMPIRE. When considering IVAN and SIVA, the mean difference in CRAE measurements of 16.4 µm (95% LOA, -4.25–37.0 µm) in our study was higher than -6.7 µm (95% LOA, -23.8–10.4 µm) reported in a previous study [21]; and the
mean difference in the CRVE measurement of 19.3 µm (95% LOA, 0.09–38.6 µm) in our study was similar to that reported previously, –18.2 µm (95% LOA, –36.7–0.4 µm). Proportional bias was observed in both of these studies and, overall, both studies are consistent.

The poor agreement between SIVA and VAMPIRE has already been reported [20]. Interestingly, when comparing VAMPIRE and other software packages, the correlation was found to be higher for CRVE than for CRAE. This could be explained by the easier recognition of the venule edges by each software, due to their larger diameter and their more contrasted appearance in a retinal photograph. In our recent study comparing retinal photographs and adaptive optics (AO) imaging, we demonstrated a better agreement and correlation of vein measurements using IVAN when compared with the gold standard AO [34].

To counter the consequences of the discrepancy between measurements taken by different packages, we have proposed a conversion algorithm based on linear regression. The good correlation between SIVA and IVAN values of CRAE, CRVE, and AVR make this choice sensible, but less so if VAMPIRE values are involved. We did not use split-sampling for internal validation as reported in other publications [21] but we performed the bootstrapping procedure described by Labarère et al. [32]. The robustness of the algorithms is well assessed through the bootstrapping procedure. Between SIVA and IVAN, our algorithms are not strictly comparable to the algorithms previously reported (IVAN-derived SIVA CRAE = 0.7176 × IVAN-measured CRAE + 34.3984 IVAN-derived SIVA CRVE = 0.7102 × IVAN-measured CRVE + 44.8717) [21]. This can be explained by many factors, such as the retinal photograph resolution and compression, which could differ in the samples studied. The influence of these two parameters on IVAN measurements has been studied previously and found to be considerable [35].

The aforementioned findings highlight the need for standardization of retinal vascular imaging with fundus cameras (resolution, compression, size, instruments, acquisition protocol, quality among others) if we are to expect similar conclusions from a statistical analysis of measurements obtained with different software packages. One should consider conclusions of this study when analyzing images at high resolution (5184 × 3456 pixels) in a JPEG format. Standardization in the field of acquisition of fundus images and treatment of images by cameras is an urgent goal [36]. Development of deep-learning system [37] may produce fully automated measurement of retinal-vessel calibers and therefore may become a useful tool in the future for clinicians.

We should acknowledge some limitations of this study. First, the pixels-to-micron conversion procedure was the same for SIVA and IVAN, but not exactly the same for VAMPIRE, which could be a cause of systematic bias. Second, images were obtained from the instrument in a JPEG compressed format, which is known to distort image-based measurements; our results should be confirmed with uncompressed images [38–40]. Third, neither agreement nor correlation among packages attests to the accuracy of measurements with respect to objective ground truth. This kind of study has been reported elsewhere for the three software packages considered here. Fourth, SIVA and VAMPIRE generate a large number of parameters and we have studied only CRAE, CRVE, and AVR. Since several other parameters (e.g., tortuosity, fractal dimension) as well as width measurements in regions other than Zone B are used in biomarker studies, a more extensive comparison would be necessary for a proper assessment of the concordance between the three packages. Fifth, our sample is modest in size and comprises adults with healthy retinas. Larger studies including statistical adjustment modeling potentially important factors (e.g., camera operator, image quality, patient characteristics etc.) are required. Finally, the limited correlation of VAMPIRE values with those of SIVA and IVAN limits the validity of the conversion model, which could, however, have value in trend-oriented investigations (i.e., decreasing or increasing outcome).

**5. Conclusions**

In conclusion, our study on the agreement and correlation of measurements from the three most widely used software tools for retinal biomarkers found an excellent agreement
between SIVA and IVAN and a poor-to-moderate agreement between VAMPIRE and the
two other software tools.

Comparing measurements of the retinal vasculature obtained from different software
tools remains a challenge requiring, arguably, a considerable standardization effort on algo-
rithms, image acquisition protocols and quality at least. Standardization would guarantee
not only consistent measurements but also, and importantly, comparable findings in terms
of biomarkers for specific conditions obtained from statistical analysis. In our study, a
solution suggested by our findings is to convert IVAN and VAMPIRE measurements into
SIVA approximate equivalents before use for pooled data analysis.

**Author Contributions:** Conceptualization, T.M., P.C., C.C., C.Y.C., J.L.; methodology, C.C., J.L., E.T.; software, V.D., S.H., E.T., T.J.M., C.Y.C., T.Y.W.; validation, E.T., V.D., C.Y.C.; investigation, T.M., P.C.; resources, C.C.; data curation, S.H., E.T.; writing—original draft preparation, T.M., C.C., P.C.; writing—review and editing, T.M., P.C., C.Y.C., T.Y.W., S.H., E.T., V.D., T.J.M., J.L., C.C.; supervision, C.C.; project administration, C.C.; funding acquisition, C.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by ARFO (Association for Research and Training in Ophthalmology, Grenoble, France, Funding number: ARFO-2019-1).

**Institutional Review Board Statement:** This study complied with the Declaration of Helsinki guidelines for research involving human subjects. The Institutional review board (IRB# 5921) reviewed and approved the study protocol, as part of the ongoing prospective observational IMAGEYE cohort study.

**Informed Consent Statement:** All study participants provided informed consent for all the ophthalmologic examinations and agreed that anonymized data could be used for clinical research.

**Data Availability Statement:** Data may be available on request.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

**References**

1. Harjasouliha, A.; Raiji, V.; Garcia Gonzalez, J.M. Review of hypertensive retinopathy. *Dis. Mon.* **2017**, *63*, 63–69. [CrossRef] [PubMed]

2. Yatsuya, H.; Folsom, A.R.; Wong, T.Y.; Klein, R.; Klein, B.E.K.; Sharrett, A.R. Retinal Microvascular Abnormalities and Risk of Lacunar Stroke: Atherosclerosis Risk in Communities Study. *Stroke* **2010**, *41*, 1349–1355. [CrossRef]

3. McGrory, S.; Taylor, A.M.; Kirin, M.; Corley, J.; Pattie, A.; Cox, S.R.; Deary, I.J. Retinal microvascular network geometry and cognitive abilities in community-dwelling older people: The Lothian Birth Cohort 1936 study. *Br. J. Ophthalmol.* **2017**, *101*, 993–998. [CrossRef] [PubMed]

4. Seidelmann, S.B.; Claggett, B.; Bravo, P.E.; Gupta, A.; Farhad, H.; Klein, B.E.; Solomon, S.D. Retinal Vessel Calibers in Predicting Long-Term Cardiovascular Outcomes: The Atherosclerosis Risk in Communities Study. *Circulation* **2016**, *134*, 1328–1338. [CrossRef] [PubMed]

5. MacGillivray, T.J.; Trucco, E.; Cameron, J.R.; Dhillon, B.; Houston, J.G.; van Beek, E.J.R. Retinal imaging as a source of biomarkers for diagnosis, characterization and prognosis of chronic illness or long-term conditions. *Br. J. Radiol.* **2014**, *87*, 20130832. [CrossRef] [PubMed]

6. Cheung, C.Y.; Ikram, M.K.; Klein, R.; Wong, T.Y. The clinical implications of recent studies on the structure and function of the retinal microvasculature in diabetes. *Diabetologia* **2015**, *58*, 871–885. [CrossRef]

7. Cheung, C.Y.; Lui Ikram, M.K.; Chen, C.; Wong, T.Y. Imaging retina to study dementia and stroke. *Prog. Retin. Eye Res.* **2017**, *57*, 89–107. [CrossRef]

8. Ikram, M.K.; Cheung, C.Y.; Lorenzi, M.; Klein, R.; Jones, T.L.; Wong, T.Y.; NIH/JDRF Workshop on Retinal Biomarker for Diabetes Group. Retinal Vascular Caliber as a Biomarker for Diabetes Microvascular Complications. *Diabetes Care* **2013**, *36*, 750–759. [CrossRef]

9. Cheung, C.Y.L.; Ikram, M.K.; Sabanayagam, C.; Wong, T.Y. Retinal Microvasculature as a Model to Study the Manifestations of Hypertension. *Hypertension* **2012**, *60*, 1094–1103. [CrossRef]

10. Wong, T.Y.; Islam, F.M.A.; Klein, R.; Klein, B.E.; Cotch, M.F.; Castro, C.; Shahar, E. Retinal vascular caliber, cardiovascular risk factors, and inflammation: The multi-ethnic study of atherosclerosis (MESA). *Investig. Ophthalmol. Vis. Sci.* **2006**, *47*, 2341–2350. [CrossRef]
11. Klein, R.; Myers, C.E.; Knudtson, M.D.; Lee, K.E.; Gangnon, R.; Wong, T.Y.; Klein, B.E. Relationship of blood pressure and other factors to serial retinal arteriolar diameter measurements over time: The beaver dam eye study. *Arch. Ophthalmol.* **2012**, *130*, 1019. [CrossRef] [PubMed]

12. Iwase, A.; Sekine, A.; Suehiro, J.; Tanaka, K.; Kawasaki, Y.; Kawasaki, R.; Araie, M. A New Method of Magnification Correction for Accurately Measuring Retinal Vessel Calibers From Fundus Photographs. *Investig. Ophthalmol. Vis. Sci.* **2017**, *58*, 1858–1864. [CrossRef] [PubMed]

13. Wei, F.F.; Zhang, Z.Y.; Petit, T.; Cauwenberghs, N.; Gu, Y.M.; Thijs, L.; Staessen, J.A. Retinal microvascular diameter, a hypertension-related trait, in ECG-gated vs. non-gated images analyzed by IVAN and SIVA. *Hypertens. Res.* **2016**, *39*, 886–892. [CrossRef] [PubMed]

14. Dervenis, N.; Coleman, A.L.; Harris, A.; Wilson, M.R.; Yu, F.; Anastasopoulos, E.; Topouzis, F. Factors associated with retinal vessel diameters in an elderly population: The thessaloniki eye study. *Investig. Ophthalmol. Vis. Sci.* **2019**, *60*, 2208–2217. [CrossRef]

15. Cheung, C.; Wong, T.Y. Quantitative and qualitative retinal microvascular characteristics and blood pressure. *J. Hypertens.* **2011**, *29*, 1380–1391. [CrossRef]

16. Cheung, C.Y.L.; Hsu, W.; Lee, M.L.; Wang, J.J.; Mitchell, P.; Lau, Q.P.; Wong, T.Y. A new method to measure peripheral retinal vascular caliber over an extended area. *Microcirculation* **2010**, *17*, 495–503. [CrossRef]

17. Perez-Rovira, A.; MacGillivray, T.; Trucco, E.; Chin, K.S.; Zutis, K.; Lupascu, C.; Dhillon, B. VAMPIRE: Vessel assessment and measurement platform for images of the RETina. *Conf. Proc. IEEE Eng. Med. Biol. Soc.* **2011**, *2011*, 3391–3394. [CrossRef]

18. Trucco, E.; Ballerini, L.; Relan, D.; Giachetti, A.; MacGillivray, T.; Zutis, K.; Dhillon, B. Novel VAMPIRE algorithms for quantitative analysis of the retinal vasculature. In *Proceedings of the 2013 ISSNIP Biosignals and Biorobotics Conference: Biosignals and Robotics for Better and Safer Living (BRC), Rio de Janerio, Brazil*, 18–20 February 2019; IEEE: Piscataway, NJ, USA, 2013; pp. 1–4. [CrossRef]

19. Trucco, E.; Giachetti, A.; Ballerini, L.; Relan, D.; Cavinato, A.; MacGillivray, T.J. *Morphometric Measurements of the Retinal Vasculature in Fundus Images with VAMPIRE. Biomedical Image Understanding: Methods and Applications*; John Wiley & Sons: Hoboken, NJ, USA, 2015.

20. McGrory, S.; Taylor, A.M.; Pellegrini, E.; Ballerini, L.; Kirin, M.; Doubal, F.N.; MacGillivray, T.J. Towards standardization of quantitative retinal vascular parameters: Comparison of SIVA and VAMPIRE measurements in the lothian birth cohort 1936. *Transl. Vis. Sci. Technol.* **2018**, *7*, 12. [CrossRef]

21. Yip, W.; Tham, Y.C.; Hsu, W.; Lee, M.L.; Klein, R.; Klein, B.; Cheung, C.Y.L. Comparison of common retinal vessel caliber measurement software and a conversion algorithm. *Transl. Vis. Sci. Technol.* **2016**, *5*, 11. [CrossRef]

22. Giachetti, A.; Ballerini, L.; Trucco, E. Accurate and reliable segmentation of the optic disc in digital fundus images. *J. Med. Imaging* **2014**, *1*, 024001. [CrossRef]

23. Relan, D.; MacGillivray, T.; Ballerini, L.; Trucco, E. Automatic retinal vessel classification using a Least Square-Support Vector Machine in VAMPIRE. *Conf. Proc. IEEE Eng. Med. Biol. Soc.* **2014**, *2014*, 142–145. [CrossRef]

24. Pellegrini, E.; Robertson, G.; Trucco, E.; MacGillivray, T.J.; Lupascu, C.; van Hemert, J.; Houston, G. Blood vessel segmentation and width estimation in ultra-wide field scanning laser ophthalmoscopy. *Biomed. Opt. Express* **2014**, *5*, 4329. [CrossRef] [PubMed]

25. Hubbard, L.D.; Brothers, R.J.; King, W.N.; Clegg, L.X.; Klein, R.; Cooper, L.S.; Atherosclerosis Risk in Communities Study Group. Methods for evaluation of retinal microvascular abnormalities associated with hypertension/sclerosis in the Atherosclerosis Risk in Communities Study. *Ophthalmology* **1999**, *106*, 2269–2280. [CrossRef]

26. Knudtson, M.D.; Lee, K.E.; Hubbard, L.D.; Wong, T.Y.; Klein, R.; Klein, B.K. Revised formulas for summarizing retinal vessel diameters. *Curr Eye Res.* **2003**, *27*, 143–149. [CrossRef]

27. Jonas, J.B.; Gusek, G.C.; Naumann, G.O.H. Optic Disc, cup and neurorefinal rim size, configuration and correlations in normal eyes. *Investig. Ophthalmol. Vis. Sci.* **1988**, *29*, 8.

28. R Core Team. *R: A Language and Environment for Statistical Computing*; R Foundation for Statistical Computing: Vienna, Austria, 2018.

29. Koo, T.K.; Li, M.Y. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. *J. Chiropr. Med.* **2016**, *15*, 155–163. [CrossRef]

30. Bland, J.M.; Altman, D.G. Statistical methods for assessing agreement between two methods of clinical measurement. *Int. J. Nurs. Stud.* **2010**, *6*, 307–310. [CrossRef]

31. Bland, J.M.; Altman, D.G. Comparing methods of measurement: Why plotting difference against standard method is misleading. *Lancet* **1995**, *346*, 1085–1087. [CrossRef]

32. Labarère, J.; Bertrand, R.; Fine, M.J. How to derive and validate clinical prediction models for use in intensive care medicine. *Intensive Care Med.* **2014**, *40*, 513–527. [CrossRef]

33. Arnould, L.; Binquet, C.; Guenancia, C.; Alassane, S.; Kawasaki, R.; Daien, V.; Creuzot-Garcher, C. Association between the retinal vascular network with Singapore “I” Vessel Assessment (SIVA) software, cardiovascular history and risk factors in the elderly: The Montrachet study, population-based study. *PLoS ONE* **2018**, *13*, e0194694. [CrossRef]

34. Mautuit, T.; Semecas, R.; Hogg, S.; Daien, V.; Gavard, O.; Chateau, N.; Chiquet, C. Comparing measurements of vascular diameter using adaptive optics imaging and conventional fundus imaging. *Diagnostics* **2022**, *12*, 705. [CrossRef] [PubMed]

35. Pauli, T.W.; Gangaputra, S.; Hubbard, L.D.; Thayer, D.W.; Chandler, C.S.; Peng, Q.; Danis, R.P. Effect of image compression and resolution on retinal vascular caliber. *Investig. Ophthalmol. Vis. Sci.* **2012**, *53*, 5117. [CrossRef] [PubMed]
36. Mookiah, M.R.K.; McGrory, S.; Hogg, S.; Price, J.; Forster, R.; MacGillivray, T.J.; Trucco, E. Towards Standardization of Retinal Vascular Measurements: On the Effect of Image Centering. In *Computational Pathology and Ophthalmic Medical Image Analysis*; Stoyanov, D., Taylor, Z., Ciompi, F., Xu, Y., Martel, A., Maier-Hein, L., Rajpoot, N., van der Laak, J., Veta, M., McKenna, S., et al., Eds.; Lecture Notes in Computer Science; Springer International Publishing: Berlin, Germany, 2018; Volume 11039, pp. 294–302. [CrossRef]

37. Cheung, C.Y.; Xu, D.; Cheng, C.Y.; Sabanayagam, C.; Tham, Y.C.; Yu, M.; Wong, T.Y. A deep-learning system for the assessment of cardiovascular disease risk via the measurement of retinal-vessel calibre. *Nat. Biomed. Eng.* **2021**, *5*, 498–508. [CrossRef] [PubMed]

38. Cree, M.J.; Jelinek, H.F. *The Effect of JPEG Compression on Automated Detection of Microaneurysms in Retinal Images*; Niel, K.S., Fofi, D., Eds.; SPIE: Munich, Germany, 2008; p. 68130M. [CrossRef]

39. Conrath, J.; Erginay, A.; Giorgi, R.; Lecleire-Collet, A.; Vicaut, E.; Klein, J.C.; Massin, P. Evaluation of the effect of JPEG and JPEG2000 image compression on the detection of diabetic retinopathy. *Eye* **2007**, *21*, 487–493. [CrossRef] [PubMed]

40. Eikelboom, R.H.; Yogesan, K.; Barry, C.J.; Constable, I.J.; Tay-Kearney, M.L.; Jitskaia, L.; House, P.H. Methods and limits of digital image compression of retinal images for telemedicine. *Investig. Ophthalmol. Vis. Sci.* **2000**, *41*, 9.