# Automatic cell identification and analysis on in vivo reflectance confocal microscopy images of the human epidermis

Imane Lboukili, Georgios Stamatas, Xavier Descombes

► To cite this version:

Imane Lboukili, Georgios Stamatas, Xavier Descombes. Automatic cell identification and analysis on in vivo reflectance confocal microscopy images of the human epidermis. SPIE Photonics Europe 2022, Apr 2022, Strasbourg, France. 10.1117/12.2626777. hal-03681154

HAL Id: hal-03681154

https://hal.science/hal-03681154

Submitted on 15 Jun 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Automatic cell identification and analysis on *in vivo* reflectance
confocal microscopy images of the human epidermis

Imane Lboukili*<sup>a,b</sup>, Xavier Descombes<sup>b</sup>, Georgios Stamatas<sup>a</sup>

<sup>a</sup>Johnson & Johnson Santé Beauté France, 1 Rue Camille Desmoulins, 92787 Issy-les-
Moulineaux FRANCE; <sup>b</sup>UCA – INRIA – I3S/CNRS Les Algorithmes – bât. Euclide B, 2000
route des Lucioles, 06900 Sophia Antipolis FRANCE

ABSTRACT

Reflectance confocal microscopy (RCM) allows real-time *in vivo* visualization of the skin at cellular level. The study of RCM images provides information on the topological and geometrical properties of the epidermis. These may change in each layer of the epidermis, depending on the subject's age and the presence of certain dermatological conditions. Studying RCM images requires manual identification of cells to derive these properties which is time-consuming and subject to human error, highlighting the need for an automated cell identification method. We propose an automated pipeline to analyze the structure of the skin in RCM images. The first step is to identify the region of interest (ROI) containing the epidermal cells. The second step is to identify individual cells in the segmented tissue area using an image filter. We then use prior biological knowledge to process the resulting detected cells, removing cells that are too small and reapplying the used filter locally on detected regions that are too big to be considered as a single cell. The results are evaluated both on simulated data and on manually annotated real RCM data. This study shows that automatic cell identification can be achieved, with an accuracy (precision and recall) that matches the inter-expert variability.

**Keywords:** epidermis, image segmentation, reflectance confocal microscopy, object detection

**1. INTRODUCTION**

Reflectance Confocal Microscopy<sup>1,2</sup> (RCM) is a real-time non-invasive *in vivo* technology that allows the visualization of the skin epidermis and upper layers of the dermis at cellular level. It is non-invasive thus making it a technique of choice for repeated sampling on a skin site without damage, when studying the changes in skin structure over-time, or when an invasive biopsy cannot be considered, e.g., cosmetology, study of baby skin, etc. It is made of the collection of signals arising from light reflections at the interface of microstructures with different indices of refraction. In skin, such microstructures are keratin fibers, melanosomes, collagen fibers and cell membranes. Therefore, it provides information on the geometrical and topological properties of the skin, which play important roles in the makeup of the skin barrier.

In most cases, analysis of RCM stacks is done manually, providing qualitative observations. However, manual analysis is time-consuming, intensive, and subject to human interpretation and inter-expert differences. Thus, we could benefit from automated methods to quantitatively analyze RCM images. An important first step in any quantitative study of skin is cell detection. Unfortunately, it is challenging and requires a robust generic algorithm to alleviate images non-uniformity and noise.

The epidermis is made of four distinct layers. From deepest to superficial, they are *stratum basale* (SB), *stratum spinosum* (SS), *stratum granulosum* (SG), and *stratum corneum* (SC).

In RCM images of light-pigmented skin, the SC appears as a large bright island surrounded by dark grooves. It is made of dead but biochemically active cells³. As we cannot observe individual cells on RCM images of the SC, our method will not be applied to these images.

The SG and SS appear as islands of viable cells, called keratinocytes, arranged in a honeycomb pattern. Granular cells are typically larger than spinous cells and as such they have a higher density⁴.

Finally, the SB is made of the smallest keratinocytes, as their differentiation starts in the SB and continues as cells migrate towards the skin surface getting bigger and flatter. The SB is attached to the dermis on the dermal-epidermal
junction, and thus we can sometimes observe the top of dermal papillae on RCM images of the SB. Additionally, melanin-producing melanocytes are scattered through the basal layer. Organelles filled with melanin, called melanosomes, are transferred from melanocytes to keratinocytes. Melanin has a high reflectance⁵,⁶ thus making RCM images of the SB more noisy and of lower quality than images of the SG and SS. For these reasons, we focus on the automated detection of keratinocytes in the granular and spinous layers.

On RCM images of the SG and SS, keratinocytes are characterized by a dark center and a grainy cytoplasm due to micro-structures, surrounded by bright grainy membranes (see Figure 1).

Attempts at the automated identification of epidermal cells on RCM images have been made⁷,⁸. Unfortunately, the amount of noise and heterogeneity of RCM images hinders the development of accurate segmentation methods. We propose a method to automatically detect keratinocytes centers on RCM images of the SG and SS. We compare our results to a manually obtained ground truth and obtain an accuracy on par with cross-experts identification.

<figure><img src="image_2.png" /><figcaption>Figure 1. RCM image of the *stratum spinosum* of light-pigmented skin. In blue, the border between tissue and background. In pink, non-informative areas. In orange, bright spots. And in red, epidermal cells.</figcaption></figure>

## 2. DATA

*In vivo* RCM images were captured on the volar forearm of 80 participants: 60 children aged 3 months to 10 years old and 20 adults aged 25 to 40 years old. All participants have light-pigmented skin, with Fitzpatrick types between I and IV. Inclusion criteria required that the participants were in good health, with no history of skin disease and had not applied any products on the observed area the day of the study.

Images were captured using a Viva scope 1500 reflectance confocal microscope with a z-resolution of 5 micrometers and xy-resolution of 1 micrometer. Images started at the SC and progressed down towards the SB. The size of each image is 1000*1000 pixels.

Image classification in one of the four epidermal layers was obtained using a hybrid deep learning algorithm⁹, allowing us to focus only images of the *stratum granulosum* and *stratum spinosum*.
# 3. IMAGE ANALYSIS WORKFLOW

## 3.1. Identification of the region of interest

RCM images tend to be noisy and non-uniform, which hinders the development of automated segmentation methods. To guide our cell detection, we started by identifying the region of interest (ROI), i.e., the regions containing epidermal cells. To do so, the black background was first identified (see Figure 2). Indeed, islands of cells surrounded by dark empty areas are observed on RCM images. They are due to grooves on the skin surface called micro-relief lines¹⁰ (see Figure 1). In order to identify these furrows and begin building a binary mask of the ROI, a Morphological Geodesic Active Contour¹¹ algorithm (so called Snake) was applied to each image. This method employs morphological operators to detect visible contours based on their intrinsic geometric measures, even if they are noisy or partially unclear.

After identifying the micro-relief lines on each RCM image, non-informative areas were detected. These areas can be seen in the tissue on RCM images and are due to low contrast and a drop in signal-to-noise ratio (see Figure 1). A texture classification was applied to the images, by training a Support Vector Machine (see Figure 2) on four features of the Grey Level Co-occurrence Matrix¹² (GLCM) which successfully discriminates between the informative and non-informative areas. These features are, (a) homogeneity which measures the closeness of the GLCM distribution to its diagonal (reflecting correlation), (b) contrast which measures the local variations in the GLCM; (c) dissimilarity which measures the similarity between pixels, and (d) energy which measures the uniformity of the area.

Our third step in ROI identification was to remove the bright spots sometimes observable in RCM images (Figure 2). Indeed, RCM images of the SG and SS may contain bright areas due to the presence of keratin in hair shafts, from cornified cells at the periphery of the islands (see Figure 1). This was accomplished by applying a succession of dilations and erosions on the RCM image where the background and the non-informative areas were removed, and which had been blurred with a Gaussian filter and binarized.

<figure><img src="image_3.png" /><figcaption>Figure 2. Steps of the identification of the region of interest. A morphological snake was used to identify the borders with the background, followed by a Support Vector Machine algorithm to detect the non-informative areas, and a succession of morphological operations to remove bright spots to an RCM image at *stratum granulosum* level. In blue the region of interest mask.</figcaption></figure>

## 3.2. Identification of individual cells

After identifying the ROI on the RCM image, a median filter was used to remove noise, followed by a local normalization which renders the variance and mean of the denoised image stationary (see Figure 3). Then, the resulting image was filtered with a Sobel filter¹³ to detect edges of the cell membrane (see Figure 3). The filter parameters were chosen to approximate the width and length of a cell membrane in the SG and SS. To the filter output, a median filter and a local normalization were applied, while making sure that the ROI binary mask is respected (see Figure 3). A Gabor filter was then applied to the previous image to refine membrane detection by convoluting the image by a sinusoidal signal of varying frequencies and orientations (see Figure 3). The output of the Gabor filter was equalized with a histogram equalization to adjust the image contrast, followed by a Gaussian adaptive thresholding (see Figure 3) which dynamically and locally changes the binarization threshold over the entire image to account for changes in contrast and brightness, assuming that smaller regions of an RCM image are more likely to be similar. A connected-components analysis was used on the obtained binary image to remove any small blobs in the detected membranes, followed by a second connected-components analysis on the inverse of the image
to close any holes in the membranes due to the graininess of the image and of the cell membranes (see Figure 3). Finally, the clean binary image was skeletonized, and any spurious branches were removed from the skeleton (see Figure 3).

<figure><img src="image_4.png" /><figcaption>Figure 3. Steps of the identification individual cells. A median filter and a local normalization were applied to the image with the ROI mask, followed by a Sato filter. Its output was filtered with a median filtered and locally normalized, and a Gabor filter was applied to it. A threshold was applied on the output after histogram equalization and small blobs were removed with a connected components analysis. The result was then skeletonized, and spurious branches were removed. ROI: region of interest.</figcaption></figure>

### 3.3. Post-processing

After obtaining the skeleton, the ROI mask was cleaned to remove any remaining spurs. This was accomplished by applying a morphological closing to the skeleton. Individual contours, i.e., detected keratinocytes, were detected on the skeleton. To improve the detection, very small contours were removed ($area < 100$ for SG and $area < 50$ for SS) as well as long contours at the border with the background, i.e., eccentricity > 0.85 (see Figure 4 A). These thresholds were determined empirically. The remaining contours were divided into two groups, (1) large contours with an $area > 1000$ for SG and $area > 120$ for SS, and (2) small contours with an $area \le 1000$ for SG and $area \le 120$ for SS. On each area of the original image determined by a large contour, a Gabor filter followed by a Sato filter¹³ were applied with different parameters than previously used. The output was then binarized with Otsu thresholding¹⁴ for SG images and with Gaussian adaptive thresholding for SS images, and small blobs were removed with a connected-components analysis. The subsequent binary image was skeletonized, and its contours detected. On images of the SG, obtained contours with an area smaller than 110 were merged with their neighbors (see Figure 4 B). On images of the SS, where cells, and therefore detected contours, are much smaller, if the second filter iteration still failed to detect more than one contour, as many ellipses as possible were fitted within the detected contour, with these ellipses having the median minor and major axis length of the small contours (see Figure 4 C). These new contours were then combined with the previously found small ones and their cell centers were detected.
<figure><img src="image_5.png" /><figcaption>Figure 4. Post-processing steps. (A) The skeleton obtained after the previous step was cleaned, and contours were detected. Small contours were removed, as well as long contours close to the border with the background. The remaining contours were divided into two groups, small and big contours. Big contours were filtered again to</figcaption></figure>
improve the detection locally. The new resulting contours were then combined to the small contours and their
centers were detected. (B) Example of large contours improvement for a *stratum granulosum* image. (C) Example
of large contours improvement for a *stratum spinosum* image.

**3.4. Accuracy evaluation**

The obtained cell centers were used to initiate a marker-controlled watershed¹⁵ on the ROI. This method considers the input image as a topographic surface which is flooded starting from set seeds or markers, *i.e.*, the detected cell centers, and returns a labelled gray-scale image, where each label is a catching basin, *i.e.*, a detected cell. This labeled image was then compared against manually detected cell centers using the software d-accuracy¹⁶ which evaluates several indexes of the detection quality (see Figure 5). Two accuracy metrics were evaluated, (a) precision, (fraction of correctly detected cells among all the detected cells) and (b) recall (the fraction of accurately detected keratinocytes among all cells defined in the ground truth).

<figure><img src="image_6.png" /><figcaption>Figure 5. Detection accuracy evaluation steps. A marker-controlled watershed was applied to the detected cell centers and the resulting labels were compared to the manually detected ground truth, in pink. The returned metrics were precision and accuracy.</figcaption></figure>

**4. RESULTS**

We evaluate the performance of our approach with respect to two experts. When compared to the first expert, our cell detection approach has a precision of 77% and of 85% on SG and SS images respectively and a recall of 85% and 67% on SG and SS confocal images respectively. And when compared to the second expert, our cell detection approach has a precision of 64% and of 75% on SG and SS images respectively and a recall of 82% and 30% on SG and SS confocal images respectively (see Table 1).

When looking into the differences between the experts, we notice that expert 2 is less sensitive in his detection especially in SS images. The obtained results are more consistent with expert 1 and prove to be accurate compared to inter-expert variability.
<table><caption>Table 1. Table of the cell detection accuracy on *stratum granulosum* and *stratum spinosum* RCM images for two different experts</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th rowspan="2"></th><th>Precision =</th><th>Recall =</th></tr><tr><th>True Positives</th><th>True Positives</th></tr></thead><tbody><tr><td rowspan="3">Stratum<br>granulosum<br>RCM image</td><td>Detection vs.<br>Expert 1</td><td></td><td>77%</td><td>85%</td></tr><tr><td>Detection vs.<br>Expert 2</td><td></td><td>64%</td><td>82%</td></tr><tr><td>Expert 1 vs.<br>Expert 2</td><td></td><td>52%</td><td>48%</td></tr><tr><td rowspan="3">Stratum<br>spinosum<br>RCM image</td><td>Detection vs.<br>Expert 1</td><td></td><td>85%</td><td>67%</td></tr><tr><td>Detection vs.<br>Expert 2</td><td></td><td>75%</td><td>30%</td></tr><tr><td>Expert 1 vs.<br>Expert 2</td><td></td><td>20%</td><td>28%</td></tr></tbody></table>

## 5. DISCUSSION

Our approach gives reasonable results on RCM images of the granular and spinous layers. Performance can be hindered by the presence of multiple layers per image, which makes parametrization of the different steps complicated, although, parameters for a given layer are the same. Our method is a multi-step approach, with multiple parameters each one influencing cell detection and its accuracy. Additionally, the noise and non-uniformity of the images has a great impact on the method performance. Steps like median filtering, local normalization and ROI determination decrease the impact of noise on the results but do not remove it completely.

Computational time is about 10mn depending on the size of the ROI, which is a major advantage when compared to the time required to identify keratinocytes manually on RCM images.

We have shown that the automated detection of keratinocytes on RCM images of the SG and SS is achievable, which is an important step towards the quantitative study of these images and of skin. This prevents from tedious manual work, which is the current practice in RCM images analysis.

The presented approach on confocal images will be useful in uncovering new insights in the study of skin physiology, skin maturation and skin ageing<sup>4,17-20</sup> and skin diseases observable with RCM, e.g., melanomas<sup>21-27</sup>.

## ACKNOWLEDGMENTS

This research was fully funded by Johnson & Johnson Santé Beauté France.

## REFERENCES

1. M. Minsky, "Microscopy apparatus," US3013467A (1961).

2. M. Minsky, "Memoir on inventing the confocal scanning microscope," Scanning **10**(4), 128-138 (1988) [doi:10.1002/sca.4950100403].

3. M. B. Murphrey, J. H. Miao, and P. M. Zito, "Histology, Stratum Corneum," in StatPearls, StatPearls Publishing, Treasure Island (FL) (2021).

4. J. Bensaci et al., "Geometrical and topological analysis of in vivo confocal microscopy images reveals dynamic maturation of epidermal structures during the first years of life," J. Biomed. Opt. **20**(9), 095004 (2015) [doi:10.1117/1.JBO.20.9.095004].
5. M. Rajadhyaksha et al., "In vivo confocal scanning laser microscopy of human skin: melanin provides strong contrast," J. Invest. Dermatol. **104**(6), 946-952 (1995) [doi:10.1111/1523-1747.ep12606215].

6. J. L. S. Sánchez-Mateos et al., "Reflectance-Mode Confocal Microscopy in Dermatological Oncology," in Lasers in Dermatology and Medicine, K. Nouri, Ed., pp. 285-308, Springer, London (2012) [doi:10.1007/978-0-85729-281-0_22].

7. M. A. Harris et al., "A Pulse Coupled Neural Network Segmentation Algorithm for Reflectance Confocal Images of Epithelial Tissue," PLoS ONE **10**(3), e0122368 (2015) [doi:10.1371/journal.pone.0122368].

8. D. Gareau, "Automated identification of epidermal keratinocytes in reflectance confocal microscopy," J. Biomed. Opt. **16**(3), 030502 (2011) [doi:10.1117/1.3552639].

9. P. Kaur et al., "Hybrid deep learning for Reflectance Confocal Microscopy skin images," in 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 1466-1471 (2016) [doi:10.1109/ICPR.2016.7899844].

10. M. Rajadhyaksha et al., "In vivo confocal scanning laser microscopy of human skin II: advances in instrumentation and comparison with histology," J. Invest. Dermatol. **113**(3), 293-303 (1999) [doi:10.1046/j.1523-1747.1999.00690.x].

11. V. Caselles, R. Kimmel, and G. Sapiro, "Geodesic Active Contours," Int. J. Comput. Vis. **22**(1), 61-79 (1997) [doi:10.1023/A:1007979827043].

12. R. M. Haralick, K. Shanmugam, and I. Dinstein, "Textural Features for Image Classification," IEEE Trans. Syst. Man Cybern. **SMC-3**(6), 610-621 (1973) [doi:10.1109/TSMC.1973.4309314].

13. Y. Sato et al., "Three-dimensional multi-scale line filter for segmentation and visualization of curvilinear structures in medical images," Med. Image Anal. **2**(2), 143-168 (1998) [doi:10.1016/s1361-8415(98)80009-1].

14. N. Otsu, "A Threshold Selection Method from Gray-Level Histograms," IEEE Trans. Syst. Man Cybern. **9**(1), 62-66 (1979) [doi:10.1109/TSMC.1979.4310076].

15. F. MEYER and S. Beucher, "Morphological segmentation," J. Vis. Commun. Image Represent. - JVCIR **1**, 21-46 (1990) [doi:10.1016/1047-3203(90)90014-M].

16. E. Debreuve, "DAccuracy," GitLab, <https://gitlab.inria.fr/edebreuvdaccuracy/-/tree/master/daccuracy> (accessed 25 January 2022).

17. G. N. Stamatas et al., "Infant skin physiology and development during the first years of life: a review of recent findings based on in vivo studies," Int. J. Cosmet. Sci. **33**(1), 17-24 (2011) [doi:10.1111/j.1468-2494.2010.00611.x].

18. G. Stamatas and N. Kollias, "Infant skin maturation: Structural changes revealed by in vivo reflect," in *Reflectance Confocal Microscopy of Cutaneous Tumors*, 2nd ed. (2017).

19. E. Cinotti et al., "Structural skin changes in elderly people investigated by reflectance confocal microscopy," J. Eur. Acad. Dermatol. Venereol. JEADV **34**(11), 2652-2658 (2020) [doi:10.1111/jdv.16466].

20. K. Kawasaki, K. Yamanishi, and H. Yamada, "Age-related morphometric changes of inner structures of the skin assessed by in vivo reflectance confocal microscopy," Int. J. Dermatol. **54**(3), 295-301 (2015) [doi:10.1111/ijd.12220].

21. D. Gareau et al., "Automated detection of malignant features in confocal microscopy on superficial spreading melanoma versus nevi," J. Biomed. Opt. **15**(6), 061713 (2010) [doi:10.1117/1.3524301].

22. A. Bozkurt et al., "A Multiresolution Convolutional Neural Network with Partial Label Training for Annotating Reflectance Confocal Microscopy Images of Skin," ArXiv180202213 Cs (2018).

23. S. Zorgui et al., "A Convolutional Neural Network for Lentigo Diagnosis," Impact Digit. Technol. Public Health Dev. Dev. Ctries. **12157**, 89-99 (2020) [doi:10.1007/978-3-030-51517-1_8].

24. A. Halimi et al., "An unsupervised Bayesian approach for the joint reconstruction and classification of cutaneous reflectance confocal microscopy images," 1 August 2017, 241-245 [doi:10.23919/EUSIPCO.2017.8081205].
25. K. Kose et al., "A machine learning method for identifying morphological patterns in reflectance confocal microscopy mosaics of melanocytic skin lesions in-vivo," 968908 (2016) [doi:10.1117/12.2212978].

26. S. Koller et al., "In vivo reflectance confocal microscopy: automated diagnostic image analysis of melanocytic skin tumours," J. Eur. Acad. Dermatol. Venereol. JEADV 25(5), 554-558 (2011) [doi:10.1111/j.1468-3083.2010.03834.x].

27. N. Kollias and G. N. Stamatas, "Optical non-invasive approaches to diagnosis of skin diseases," J. Investig. Dermatol. Symp. Proc. 7(1), 64-75 (2002) [doi:10.1046/j.1523-1747.2002.19635.x].