Reliability and Reproducibility of Retinal Oxygen Saturation Measurements using a Predefined Peri-papillary Annulus (2013)

Purpose: To determine the reliability and reproducibility of the Vesselmap Oximetry Module for arteriolar and venular oxygen saturation (SO2) of the same retinal area, specified by a peripapillary annulus, in healthy subjects. Methods: Fundus oximetry images were obtained, using a standardized protocol by a single observer, from the right eye of 20 healthy individuals. Age range was 19–45 years old, and images were analysed using the oximetry module of the Vesselmap System (Imedos, UG, Germany). Intra-observer reliability (assessment of two measurements of SO2 values performed 5 days apart); interobserver reliability (assessment of SO2 performed by two different trained observers); and intrasubject reproducibility (comparison of SO2 measurements of two different images of the same subject and retinal area, taken 10 min apart) were assessed. The standard deviations between the repeated measurements (SDr), together with the intraclass correlation coefficient (ICC), of these three parameters were calculated. Results: The SDr for intra-observer reliability was 0.56% and 0.55% for arteriolar and venular SO2, respectively. The results were similar for intrasubject reproducibility (0.69% and 0.79% for arteriolar and venular SO2, respectively); interobserver reliability, however, was higher (SDr 1.22% and 1.01% for arteriolar and venular SO2, respectively). The ICC values for intra-observer reliability were 0.99 for both arteriolar and venular SO2. The results were similar for both interobserver reliability (0.94 for arteriolar SO2 and 0.96 for venular SO2) and intrasubject reproducibility (0.98 for both arteriolar and venular SO2). Conclusion: Retinal oxygen saturation values taken using the oximetry module of the Vesselmap System are highly reliable and reproducible, provided the image quality is standardized, the same measurement area is analysed in each image, and the number of observers analysing the images is kept to a minimum. Oxygen (O2) is critical to the retina, and any alteration to supply may have serious functional and pathogenic implications (Vanderkooi et al. 1991). The advent of modern spectrophotometric retinal oximetry techniques, utilizing differences in absorption wavelengths of oxygenated and deoxygenated haemoglobin, allows for the noninvasive measurements of oxygen saturation (SO2) in retinal blood vessels (Delori 1988; Harris et al. 2003). One of these instruments is the Oximetry module of the Vesselmap System (Imedos UG, Jena, Germany) (Hammer et al. 2009a). This automatic oximeter utilizes the principle whereby the SO2 in a retinal vessel is linearly correlated with the ratio of the logarithmic contrast of the vessel versus its surrounding at two specific wavelengths of 548 and 610 nm (optical density ratio or ODR) (Beach et al. 1999). Intrasubject reproducibility data for this instrument have been demonstrated previously (Hammer et al. 2009a,b; Lasta et al. 2012). However, we have observed noticeable variations in the amount of light illuminating different areas of the fundus depending on the camera distance and/or angle between each image. This is mainly due to light scatter resulting from the modified light source necessary for the correct functioning of the Dynamic Vessel Analyser (DVA) module of the system. Therefore, the above reproducibility data may have been limited by these variations in illumination especially for vessel segments towards the periphery and optic disc. Data from the above studies may also have mitigated the reproducibility and clinical usefulness, as reproducibility was calculated only in certain branch vessels as opposed to the average of all retinal vessels, which may not take into account the possible variability of reproducibility measurements between retinal quadrants due to the uneven illumination on the fundus. Importantly, the vesselmap analysis software is not fully automated, and utilization of the software requires trained personnel. To date, however, no data have been published on ‘intra-observer’ and ‘interobserver’ reliability parameters. In this study, we therefore assessed (i) the intra- and interobserver reliability of the Vesselmap Oximetry Module for arteriolar and venular SO2 values in healthy subjects; and (ii) the intrasubject reproducibility for the above parameters, in a predetermined area 1–1.5 disc diameters from the optic disc in the peripapillary region of the retina. Twenty healthy Caucasian individuals, with a mean age ±SD of 30.1 ± 7.56 years (range 19–45 years), were recruited as part of a larger trial. The subjects were predominantly female (60%) and were all nonsmokers. Subjects were excluded if they had intraocular pressure (IOP) >21 mmHg, presence of cataract, presence of ametropia of more than 3 dioptres, or other media opacities or any history or signs of retinal or optic nerve disease. All subjects were over the age of 18 years, and did not have any systemic conditions that might affect blood flow such as diabetes mellitus and hypertension. All subjects were also advised to refrain from consuming caffeinated products and alcohol for at least 12 hr before the study. Subjects’ pupils were dilated with 1% tropicamide, and images were taken of the right eye. The images were graded at the Centre for Eye Research Australia, and images with pigment abnormalities were excluded from the study. The parent study was approved by the human ethics committee of the Royal Victorian Eye and Ear hospital and abided by the tenets of the Declaration of Helsinki. Retinal vessel oxygenation measurements in both retinal arteries and veins were estimated using the Oximetry module of the Vesselmap System (Imedos UG, Jena, Germany). In short, two monochromatic fundus images were obtained using a double-band pass filter (light transmission at 548 ± 10 and 610 ± 10 nm) inserted in the illumination path of a fundus camera FF450 (Carl Zeiss, Jena, Germany). Only one observer (REKM) was assigned to take the images. The ratio of the optical densities at 610 nm to that at the isosbestic wavelength 548 nm is proportional to the vessel haemoglobin SO2 (Beach et al. 1999) after compensation for vessel diameter and fundus pigmentation (Hammer et al. 2009a,b). More details of the technique and calibration can be found elsewhere (Hammer et al. 2009a,b). Optic disc-centred images were taken from the right eye of each subject (30-degree field, mydriasis). Care was taken to ensure that the images were sufficiently illuminated (as indicated by the ‘illumination indicator’ algorithm provided in the latest vesselmap Software), the magnification and flash settings were not changed between images and that the background illumination remained constant. For each image, a peripapillary annulus (specifically designed by Imedos to be adjustable to account for different optic disc sizes as per our request) was used to mark an area for analysis (Fig. 1). Within this area (inner radius of 1 and outer radius of 1.5 disc diameters), the oxygen saturation was measured for all arterioles and venules. Typically, about 20 measurements consisting of between 8 and 10 arterioles and 8 and 10 venules above 50 μm in diameter were averaged. This value was chosen as it was observed that the algorithm had difficulty tracing and measuring the oxygen saturation of vessels smaller than 50 μm without repeated efforts by the observer to mark the edges of the vessel walls. Values along the vessel that were more than 20% over the mean value were excluded to eliminate the confounding effects of specular reflection. A fundus photograph showing the peripapillary annulus where oxygen saturation readings were taken. A false color map showing saturation values has been superimposed on the image. Saturation values are color coded according to the scale bar. In this study, intra-observer reliability refers to the comparison of initial and repeated measurements of SO2 on the same images by one trained observer (REKM), approximately 5 days apart. Data were utilized from the first reliably gradable scan per eye (n = 20 images, total of 20 measurements). Interobserver reliability refers to the comparison of measurements of SO2 from two different trained observers (REKM and ZCW) of the same image (n = 20 images, total 40 measurements). Intrasubject reproducibility refers to comparison of measurements of SO2 from two different oximetry images of the same persons. Repeated images were taken 10 min apart on the same day and analysed once by the same observer (REKM) (n = 20 images, total 40 measurements). All analyses were performed using stata 12.1 (StataCorp, College station, TX, USA). To assess intra- and interobserver reliability and intrasubject reproducibility, we used three methods. Firstly, Bland–Altman (BA) plots were utilized to examine agreement between measurements by plotting the difference between the initial and subsequent measurements of SO2 against the mean of the two measurements; limits of agreement were calculated as the mean difference plus minus two standard deviations (SD). Secondly, one-way analysis of variance (anova) was used to estimate the standard deviation between repeated measurements (SDr) as a means of quantifying reliability and reproducibility (Bland 2000) to allow for comparison with the SDr obtained by earlier studies (Hardarson et al. 2006; Hammer et al. 2009a,b). One-way anova partitions the variability between all measurements into variability between subjects and variability between repeated measurements on the same subject. More specifically, the square root of the mean square within measurements is an estimate of the standard deviation between repeated measurements. Last, intraclass correlation coefficient (ICC) with exact 95% confidence interval (CI) was used to assess both intra- and interobserver reliability, as well as intrasubject reproducibility of the measurements. The mean ± SD of initial SO2 values were 94.03 ± 4.89% for arterioles and 61.83 ± 4.99% for venules (Table 1). Table 1 shows the mean (SD) of the initial and repeated analysis 5 days later of arteriolar and venular SO2. It also shows the SDr as analysed by one-way anova. We found consistently low SDr values across the board for all parameters (0.56% for arterioles and 0.55% for venules), which indicated good reliability. Using ICC, we found very high intra-observer reliability when analysing the same image twice, 5 days apart, for both arterioles and venules (ICC: 0.99, 95% CI: 0.97–1.00 for both, Table 2). Figure S1 shows the BA plot of the initial and repeated measures of SO2 from the same image for arteriolar and venular SO2 values. For each of these parameters, the BA plot shows very good agreement between the two measures. The middle section of Table 1 shows the mean (SD) of the measurements by observers 1 and 2 of arteriolar and venular SO2. Although the interobserver SDr values for arteriolar and venular SO2 levels were low, they were higher across the board for these two parameters (1.22% for arterioles and 1.01% for venules) as compared with SDr values for intra-observer reliability. ICC was high when analysing measurements by two different observers of the same image. Values for arterioles (ICC: 0.94, 95% CI: 0.85–0.98) and venules (ICC: 0.96, 95% CI: 0.86–0.99) are shown in Table 2. Figure S2 shows the BA plot for reproducibility of oximetry readings (observer 1 versus observer 2) of arterioles and venules. For these two parameters, the agreement between observer 1 and observer 2 was high. The bottom section of Table 1 shows the mean (SD) of arteriolar and venular SO2 measurements of images taken at 0 and 10 min. It also shows the SDr of the two measurements of images taken initially and 10 min later. We found very little deviation between the SO2 values (0.69% for arterioles and 0.79% for venules) as analysed by one-way anova. Table 2 shows that intrasubject reproducibility was high for arterioles and venules (ICC: 0.98, 95% CI: 0.95–0.99 for both) for this instrument. Figure S3 shows the BA plot for reproducibility of oximetry readings (initial versus subsequent measurements taken 10 min later) of arterioles and venules. The results were highly agreeable for these two parameters. In this study, we assessed the intra- and interobserver reliability of the Vesselmap analysis software, as well as the intrasubject reproducibility of SO2 measurement values in 20 normal subjects. Our results show that the oximetry module of the Vesselmap system provides highly reliable and reproducible SO2 readings for arterioles and venules, provided that image quality is standardized via an established protocol, a predefined location is consistently selected for analysis between images and that the number of observers analysing the images is kept to a minimum. Both intra- and interobserver reliability can be affected by different image analysis factors such as choice and location of vessel segments, as well as image clarity. As the algorithm of the Vesselmap analysis program does not differentiate between vessel crossings and is also affected by artefacts such as a bright vessel reflex or vessels being too close to each other, the role of the observer becomes important in choosing the appropriate vessel segment for measurement. As such, reliability is important due to the introduction of subjectivity to the measurements. Our results, using both the ICC and SD of repeated analyses, have established the high reliability of this instrument both between and within the same observer. It is particularly interesting to note that venular SO2 measurements yielded more reproducible results for both intra- and interobserver reliability. We speculate that this is due to a larger vessel reflex from the arterioles, which might have given rise to greater variation in readings along the length of the vessel. Also, even though interobserver reliability is high for both arteriolar and venular SO2, the increased SDr for interobserver reliability as compared with intra-observer reliability, of these vascular parameters have demonstrated that this technique is subjected to observer interpretation. Hence, it is suggested that the number of observers analysing the data be minimized to two or less to reduce interobserver variability. All image analysis issues that affect reliability will also affect reproducibility. This is in addition to imaging-related issues such as choice of location within the eye, sharpness of image, the quality and evenness of illumination, the choice of vessel segments and the ability of the subject to concentrate and fixate. Compared with results obtained by Hammer et al. (2009a) who analysed SO2 globally, our reproducibility for both arterioles and venules were higher (SD of 2.52% versus our 0.69% SO2 for arterioles; SD of 3.25% as compared with our 0.79% SO2 for venules). This is likely related to illumination issues, especially in the periphery (too dark) or near the optic disc (too much reflection; Fig. 1). As the light source is ring-shaped in order to accommodate for the DVA module on this system, any changes in camera distance and/or angle will potentially result in light scattering, especially if the edges of the ring hits the pupil edge. Using a predefined area between 1 and 1.5 disc diameters away from the optic disc appears to give the most consistent results and also makes it easier to track the same vessel segments in subsequent images, leading to fewer variations. This is corroborated by the high ICC values in our analysis. Hence, SO2 measurements should be carried out using a predetermined retinal area, instead of globally, especially for future research and clinical implementations of this instrument. Interestingly, intrasubject reproducibility for both arteriolar and venular SO2 appears to be slightly better than interobserver reliability (SDr of 0.69% versus 1.22% for arterioles; SDr of 0.79% versus 1.01% for venules). It can be argued that intrasubject reproducibility ‘includes’ intra-observer reliability; it should therefore be less reproducible. We believe that this is mainly due to observer interpretation, for example, how much of the vessel segment to cut off near a vessel branching, as well as how the observer consistently places the peripapillary annulus on the optic disc. All these results in slightly different lengths of vessel segments measured, which may have led to a larger variation in SO2 values between observers. It is also worth noting that our results are more reproducible than the recently published data reported by Lasta et al. (2012). Their study group followed similar protocols to the one used in this report, in that they measured branch vessels between one to two disc diameters away from the margin of the optic disc, included short-term reproducibility testing (15 min between sets of images), and utilized ICC, which accounts for correlation between measurements, to assess reproducibility. We would expect comparable results given these similarities; however, this was not the case (ICC of 0.94 compared with our 0.98 for arteriolar SO2; ICC of 0.88 versus our 0.98 for venular SO2). As the software used is presumably the same, it is possible that the variability in image quality and, possibly, measurement area between subjects may be associated with this discrepancy. Our strong reliability and reproducibility values may be related to the fact that all the retinal images in this study were captured by a single observer and that care was taken to establish and adhere to a protocol designed to minimize the illumination issues mentioned above. This highlights the importance of ensuring that the quality of retinal images, as well as the predefined measurement area, is standardized across the board when performing retinal oximetry measurements. Strengths of this study include the use of healthy subjects with no media opacities that might interfere with the quality of the images, and who had not consumed caffeine, alcohol or smoked on the day of the study. However, several limitations are also noted including the risk of a ‘learning effect’ in reliability analyses, whereby the observer remembers the vessel segments chosen especially when the measurements are repeated within a short time frame; these lead to falsely high reliability readings. To combat that, the subsequent measurement was made 5 days apart. As noted above, there was also the possibility of introducing further variations into the results with different observers taking the fundus images. We therefore restricted the image taking to only one person to ensure consistency. The two observers were also familiar with the vesselmap software prior to analysis of measurements, hence avoiding a ‘training effect’, where a newly trained observer gradually improves in accuracy while analysing the images thus skewing results. Finally, the effect of the cardiac cycle on reproducibility has yet to be determined. In conclusion, we have shown that the oximetry module of the Vesselmap System provides highly reliable and reproducible measures of SO2 provided that image quality is standardized; the readings are taken in a predefined peripapillary region to minimize the effect of illumination and that the numbers of observers analysing the data are kept to a minimum to reduce interobserver variability. Provided that the recommendations above are followed, these results support the use of this instrument in clinical settings, as a device for monitoring the development and progression of ocular diseases such as glaucoma (Olafsdottir et al. 2011) and diabetic retinopathy (DR) (Hammer et al. 2009b; Hardarson & Stefánsson 2012). We wish to thank Mr. Thomas Riemer, of Imedos UG, Germany, for his invaluable contribution and insights. The Centre for Eye Research Australia receives Operational Infrastructure Support from the Victorian Government. Figure S1. Bland-Altman plot for the first and second analysis of oxygen saturation values (SO2 in %) in arterioles (top) and venules (bottom). Figure S2. Bland-Altman plot for the saturation values derived by observer 1 versus observer 2 (SO2 in %) in arterioles (top) and venules (bottom). Figure S3. Bland-Altman plot for initial arteriolar oxygen saturation measurements (SO2 in %) versus measurements 10 min later for arterioles (top) and venules (bottom). As a service to our authors and readers, this journal provides supporting information supplied by the authors. Such materials are peer-reviewed and may be re-organized for online delivery, but are not copy-edited. Technical support issues arising from supporting information (other than missing files) should be addressed to the authors. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.