# Segmentation of Alexis retractors in the context of augmented reality in video-assisted thoracoscopic surgery

The Cong Luong, Simon Rouzé, Jean-Louis Dillenseger

► To cite this version:

The Cong Luong, Simon Rouzé, Jean-Louis Dillenseger. Segmentation of Alexis retractors in the context of augmented reality in video-assisted thoracoscopic surgery. Image-Guided Procedures, Robotic Interventions, and Modeling, Feb 2025, San Diego, France. pp.8, 10.1117/12.3044346. hal-05028087

HAL Id: hal-05028087

https://hal.science/hal-05028087v1

Submitted on 9 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Segmentation of Alexis Retractors in the context of augmented reality in Video-Assisted Thoracoscopic Surgery

The Cong Luong<sup>a</sup>, Simon Rouzé<sup>a,b</sup>, and Jean-louis Dillenseger<sup>a</sup>

## ABSTRACT

This study takes place in the context of video-assisted thoracic surgery (VATS) for lung nodule resection (wedge surgery). The implementation of an augmented reality solution requires registration between the endoscopic intraoperative video image and a CBCT also acquired in an intraoperative context and used for internal localization of the nodule. To achieve this, we need to automatically segment features common to both modalities, namely the Alexis retractors in the operative scene. In this paper we present how to segment retractors Alexis in video images and in intraoperative CBCT. Thanks to their color, segmentation of Alexis retractors is fairly straightforward in video images. A simple thresholding in the Hue-Saturation-Luminosity (HSL) color space is sufficient. However, the segmentation of Alexis retractors is more complicated on CBCT, as there is no contrast in gray-scale values between them and the surrounding tissues. We had therefore to rely only on their surface properties. So we propose the following framework: 1) extract the mesh of the internal surface of the rib-cage, 2) smooth this mesh; 3) calculate the mean curvatures on the vertexes of this mesh. The vertexes of the Alexis retractors boundaries can then be characterized within a specific range of mean curvature values; 4) automatically separate potential Alexis regions by Gaussian Mixture Model and 5) for each Alexis, fit two concentric ellipses on the vertexes matching the curvatures range. This allows each Alexis to be delimit on the CBCT in the form of an elliptic ring. The future registration can then be carried out by superimposing the Alexis rings extracted from the videos and CBCT.

**Keywords:** Video-Assisted Thoracic Surgery, Color-based segmentation, curvature-based segmentation, ellipse fitting, augmented reality

## 1. INTRODUCTION

Lung nodules are small, abnormal growths typically identified by computed tomography (CT) of the lungs. In cases where a needle biopsy is not feasible due to the size or location of the lung nodule, a wedge resection is indicated to establish the diagnosis by anapathological examination. In the case of malignancy, the surgeon proceeds to enlarge the resection to the lobe containing the nodule. This wedge resection (and, if required, the enlarged resection) is performed during minimally invasive video-assisted thoracoscopic surgery (VATS). To achieve this wedge resection, first, the surgeon locates the nodule on the preoperative CT scan. Then, in the operating room, he makes 3 incisions between the ribs and creates so 3 passageways using retractors (Alexis O wound retractor/protector) to insert an endoscope and surgical instruments. However, these openings cause a lung collapse (pneumothorax), making the nodule difficult to localize beneath the surface of the lung that has undergone such a large deformation. In previous studies, we showed that it was possible to localize the nodule in the lung during surgery using a C-Arm generated cone-beam computed tomography (CBCT), either directly in the CBCT volume if the nodule is visible there,¹ or by simulating lung deformations using a finite element model.²,³

However, the link between the intra-pulmonary location of the nodule and the endoscopic view of the external lung surface remains a challenge. The idea is therefore to reproject the 3D position of the nodule localized in the CBCT onto the endoscopic view using augmented reality techniques. To this end, one of our partners has

<footnote><sup>a</sup>Univ Rennes, Inserm, LTSI - UMR 1099, F-35000 Rennes, France</footnote>

<footnote><sup>b</sup>Univ Rennes, CHU Rennes, Inserm, LTSI – UMR 1099, F-35000 Rennes, France</footnote>

<footnote>Further author information: (Send correspondence to Jean-louis Dillenseger.)</footnote>

<footnote>Jean-Louis Dillenseger: E-mail: jean-louis.dillenseger@univ-rennes.fr</footnote>
proposed a 2D/3D (endoscopic view/CBCT) registration using natural information (anatomical structures or
surgical ancillaries) common to both modalities.⁴ More specifically, these authors used as common information
visible in both modalities the ribs and Alexis retractors (Fig. 1).

<figure><img src="image_2.png" /><figcaption>Figure 1. Intraoperative imaging after pneumothorax of a patient during wedge resection: endoscopic view (left) and a slice of a intraoperative CBCT (right). Two of the Alexis retractors can be seen in blueish color in the endoscopic view and are pointed by arrows in the CBCT slices.</figcaption></figure>

Their method is based on the feature extraction in both modalities (Fig. 2-left and -middle), followed by
the use of a variant of ICP<sup>5</sup> adapted to 2D-3D registration<sup>6</sup> and modified to work within the specific context of
VATS.<sup>4</sup> The general idea is to find a virtual camera model that superimposes the 2D projection in the video image
of the features extracted in 3D from the CBCT onto the corresponding 2D features in the video image. Because
the segmentation has high uncertainty on the edges, each extracted feature has been sampled and converted to
a point cloud. The registration has been performed on this point clouds (Fig. 2-right).

<figure><img src="image_3.png" /><figcaption>Figure 2. Registration framework. Left: Ribs and Alexis segmented in CBCT. Middle: Ribs and Alexis segmented in the video images. Right: 2D/3D Registration result where most features are properly registered. Because the segmentation has high uncertainty on the edges, each extracted feature has been sampled and converted to a point cloud. The registration has been performed on this point clouds. The registration method and images of this Figure are extracted from Noblet et al.<sup>4</sup></figcaption></figure>

One of the key points is the segmentation of the features required for registration. As the Noblet *et al.* study was a feasibility study of augmented reality and tracking, these features (except ribs in CBCT) were segmented manually, view by view on video and by 3D picking in the CBCT. Our aim is to automate the ribs and Alexis features segmentation.

Initially, we focused on the delineation of the ribs. Ribs can be easily segmented automatically in the CBCT
with classical image processing methods and morphological operations. But ribs are difficult to segment in the
endoscopic video, even manually, because adipose tissue (yellow tissue in Fig. 1-left) can appear on the parietal
pleurae and the outline is not visible as ribs gradually disappear under intercostal muscle. We proposed a
deep-Learning-based approach to solve this problem.⁷

This paper is therefore focused on the automatic segmentation of the second class of feature, Alexis retractors,
both on endoscopic video in 2D and on intraoperative CBCT in 3D.
## 2. DATA AND METHODS

The images from this study were acquired as part of a clinical routine, following the VATS protocol established by Rouzé et al.¹ In this protocol, the three incisions are protected by two XXS and one XS Alexis O wound retractors/protectors from Applied Medical*.

Endoscopic images were acquired with a Stryker RGB HD monocular camera mounted on a Stryker 30° 10 mm endoscope†.

In accordance with the clinical protocol of Rouzé et al, we acquired intraoperative images using a hybrid GE Discovery CBCT‡ in a hybrid operating room. As the field of view (FOV) of the device is smaller than the thoracic cavity, the image volume was centered on the operated lobe as shown in Fig. 1-right.

As the use of CBCT for nodule localization after pneumothorax is not yet widespread, we had at our disposal only a few intraoperative CBCTs and the corresponding endoscopic videos. This number of data was too small to consider learning-based methods. We therefore applied classical image processing methods.

Concerning the degree of accuracy required by the segmentation results, as described in the introduction, the registration method proposed by Noblet et al. is a classical method of iteratively estimating the position of a camera model inside the CBCT by maximizing the overlap between the features seen on the video and the same features seen by the virtual camera in the CBCT.⁴ As the criterion to be maximized was an overlap rate, it was not essential to have a very precise segmentation, especially on the edges. Overall, on the video images, we were interested in the surface of the Alexis rolls, and in the CBCT, we were interested in the same surface approximated either by a torus or by a ring, both of elliptical cross-section.

### 2.1 Alexis segmentation in the video images

The inner ring of the Alexis retractor used during the surgical procedure is blue-green in color. This hue is quite distinct from that of the other anatomical structures seen by the endoscope (flesh or pink hue for lung and muscle, yellowish hue for fat, whitish hue for bone, etc.) (see Fig. 1-left).

The inner rings of Alexis can thus be estimated by simple narrow band thresholding in the hue plane (H) after converting the color coding of the video images from Red-Green-Blue (RGB) space to Hue-Saturation-Luminosity (HSL) or Hue-Saturation-Value (HSV) space. This conversion also makes it possible to deal partially with some of the luminous spots caused by the reflection of the endoscopic lighting on specular surfaces because hue can still predominate even in relatively strong lighting conditions. All you need are fairly wide Saturation and Value thresholds. These processes are very fast and, if necessary, allow images to be processed at video frame rates.

### 2.2 Alexis segmentation in the CBCT volume

Looking at the CBCT volume, we can see several characteristics of the data and of the Alexis in this volume (Fig. 1-right). Firstly, the Alexis have exactly the same Hounsfield unit (HU) range as the surrounding intercostal muscles or fat. We find it difficult to propose a segmentation strategy based on HU. On the other hand, the HU contrast is very high between the rib cage/Alexis and the space left by the pneumothorax. Alexis retractors appear as a more or less elliptical 3D ring on the inside of the rib cage. However, the CBCT field of view (FOV) is rather limited (e.g. the outside of the rib cage cannot be seen, or some Alexis may be cut by the FOV). We therefore decided to characterize the Alexis ring bulges by some geometric properties of their surface (curvatures) and then model their shapes as an elliptical 2D ring applied to the inner surface of the rib cage. More specifically, the segmentation strategy is as follows (Fig. 3):

1. Segment automatically or semi-automatically the inner surface of the rib cage. This can be done by roughly cropping the volume containing the internal surface of the chest wall using a volume manipulation software (e.g. Fiji§ or others) (Fig. 3-a); setting a Hounsfield unit threshold to separate air and soft tissues; and

<footnote>*https://www.appliedmedical.com/Products/Alexis/OWoundProtectorRetractor</footnote>

<footnote>†https://www.stryker.com</footnote>

<footnote>‡https://www.gehealthcare.com/products/image-guiding-solutions/discovery-igs-7-or</footnote>

<footnote>§https://fiji.sc/</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 3. Alexis retractors segmentation framework. a) Volume rendering of the inner rib cage with the 3 Alexis; b) Rib cage rough segmentation. Smoothed surface mesh of the inner rib cage; c) Mesh curvature estimation. The color is representative to the mean curvature value; d) Curvature thresholding. In red, the points corresponding on the thresholded mean curvature at the two edges of the retractors rings; e) Separating and labeling the several Alexis retractors. A Gaussian mixture model is fitted on the extracted curvature points. In blue, orange and turquoise the 3 Gaussian models fitted on each Alexis; f) Segmenting the Alexis ring. In green the surface between the 2 concentric ellipses fitted at each curvature point set at the inner and outer edge of the Alexis ring and projected on the inner rib cage surface.</figcaption></figure>

creating a surface mesh of the inner surface of the rib cage using some classical algorithms as marching cubes⁸ implemented in the volume manipulation software. On this surface, the Alexis ring bulges can easily be seen (Fig. 3-b). To minimize outliers we performed the 3 following post-processings on the mesh: 1) we smoothed the surfaces by mesh filtering (Laplacian or Taubin⁹ mesh filters); 2) we reduced the number of edges (50% decimation) using Quadratic Edge Collapse Decimation¹⁰ and 3) we applied a second round of surface smoothing by Laplacian mesh filtering.

2. Curvature estimation. We then calculated the curvatures at the various mesh points by mesh adapted differential geometry (Fig. 3-c). There are several types of curvature: min and max principals, mean and Gaussian. In order to best estimate the limit of the ring bulges, after some preliminary tests, we used the mean curvature to discriminate the region where the rings meets the surface of the rib cage (the Gaussian curvature, which estimates saddle-shaped curvatures, was also a good candidate). Some Mesh libraries as Open Mesh Lab¹¹ can be used for this purpose. A simple thresholding of the mean curvature allowed us to extract the vertices belonging to the rings limits on the rib cage surface, as well as vertices not belonging to the rings (e.g. some rib limits, etc.) (Fig. 3-d).

3. Separating the several Alexis. Each Alexis is represented as a cluster of 3D points. The Alexis (and therefore the points of curvature depicting them) are all on the same side of the rib cage in a fairly flat area. To simplify the problem of separating the Alexis, we switched from 3D to 2D points. To do this, we estimated the regression plane of the 3D points using SVD or Ransac and projected the 3D points perpendicularly onto this plane.

As an *a priori*, we know the number of Alexis (from 1 to 3). Several techniques can be used to separate the clusters (K-Means, etc.). To take into account the ellipsoidal shape of the Alexis, we decided to model the distribution of points by a mixture of Gaussians, with the center of each Gaussian as the center of an Alexis and the covariance matrix modeling the ellipse formed by the ring. After a K-Means initialization, the expectation-maximization (EM) algorithm allowed to fit the mixture-of-Gaussian models to the 2D points and so separating and labeling the clusters (Fig. 3-e).

4. Segmenting the several Alexis rings. Once the clusters have been separated, we can segment each Alexis separately. To do this, we fitted a 3D plane to the segmented vertices near the center of an Alexis estimated in the previous step. We chose to fit the 3D plane using SVD for greater robustness. The vertices were then projected onto this plane to reduce their dimensionality for future fitting. The projected vertices with a few exceptions were found to be distributed on this plane in the form of two concentric 2D ellipses. All
we had to do was to fit two ellipses to these clouds of vertices. The parameters estimated by the mixture of Gaussians step were used for initialization. First, we fit a single ellipse per least squares on the point cloud. The result is an ellipse positioned roughly between the two edges of the Alexis. This separates the search domain into two parts of space. The points inside this ellipse will be used to estimate the inner edge of the Alexis, while the points outside it will be used to estimate the outer edge. We then fitted sequentially the inner ellipse on the inner edge first. This can be done by least squares fitting¹² or Ransac.¹³ The next step was to fit the second ellipse to the outer edge points, but imposed the same center as the inner ellipse to ensure concentricity. Finally, the Alexis on the CBCT was formed by all the vertices projected between the two concentric ellipses (Fig. 3-f).

# 3. IMPLEMENTATION AND RESULTS

The data are those described in the section 2.

## 3.1 Alexis segmentation in the video images

Fig. 4 shows the segmentation of an Alexis in one frame of the video. The back part of the Alexis is not required for registration. The simplicity of the segmentation allows processing at video frame rates. The method proved robustness on the different videos, with perhaps the adjustment of color thresholds to be reviewed according to the video.

<figure><img src="image_5.png" /><figcaption>Figure 4. Segmentation of an Alexis on one frame of the endoscopic video. Left, original video frame; right segmented image.</figcaption></figure>

## 3.2 Alexis segmentation in the CBCT volume

The segmentation of the Alexis retractors was carried out in two stages:

1. The first involved extraction and coding in the form of a mesh of the internal rib cage surface from the patient's intraoperative CBCT volume. This was done using the Fiji free software¹, which enables DICOM data to be loaded, ROIs of various shapes (including elliptical) to be defined, voxel volume to be segmented by thresholding and surfaces to be described using Marching Cubes. The output is a mesh description of the rib cage including the Alexis included in the field of view of the CBCT.

Because of the good contrast between the rib cage and air, the definition of the threshold value is not sensitive. The main effort lies in the interactive placement of a good ROI that does not take lung information into account.

2. A program written in Python that loads the mesh from the previous step, and performs the processing sequence for the automatic segmentation of the Alexis. This program aggregates the various libraries required for segmentation: pymeshlab, the Python library that interfaces to MeshLab¹¹ used for mesh smoothing and curvature computation; scikit-learn¹ for Gaussian mixture used to separate the several Alexis; and scikit-image¹⁴** for the Python implementation of a Ransac-based ellipse fitting algorithm.

<footnote>¶https://fiji.sc/</footnote>

<footnote>||https://scikit-learn.org/stable/index.html</footnote>

<footnote>**https://scikit-image.org/docs/stable/api/skimage.html</footnote>
The user has only to indicate the number of Alexis seen on the surface. The various internal variables to be set were the number of mesh smoothing iterations, the choice of curvature type (mean curvature in our case) and curvature thresholds, and the choice of ellipse fitting method. Once these parameters were adjusted in an *ad hoc* manner, the results were stable due to the equivalent size of the Alexis between the different patients.

Fig. 5 shows the results of the segmentation of the 3 Alexis of patient A (the whole process on this patient is shown in Fig. 3). If we take a closer look at how ellipses are fitted to points of curvature, we can observe that: for the Alexis a and c the inner ellipses are well fitted; but that for Alexis c the adjustment of the outer ellipse is a bit disturbed by outliers. Alexis a and b are on the border of the field of view and are only partially described in the CBCT (especially Alexis a). The algorithm had more difficulty adjusting to missing data. A method of fitting ellipses to elliptic arcs¹⁵ should be considered in this case.

<figure><img src="image_6.png" /><figcaption>Figure 5. Segmentation of the 3 Alexis of patient A. a, b and c show in detail the selected points of curvature of the 3 Alexis and the concentric ellipses fitted to these points.</figcaption></figure>

Fig. 6 shows the results for patients B and C, for whom only one Alexis was in the CBCT field of view. The segmentation parameters (apart from the number of Alexis) were the same as for patient A. Patient B was well segmented despite outlier points of curvature along the ribs (probably the surface was not sufficiently smoothed). For patient C, the algorithm was unable to capture the entire inner part of the ring. The Alexis was probably overstretched. The inner ellipse was partially adjusted on the outer edge and the outer ellipse on the adjacent rib. In any case, the results are quite good, and the virtual camera pose estimation solution (Fig. 2) is not so sensitive to segmentation accuracy.⁴

In conclusion, the crucial points were the interactive choice of the initial Region of Interest for extraction of the rib-cage surface (the rather inhomogeneous lung could present curvatures of the same range as the Alexis) and the degree of mesh smoothing. From time to time, the Ransac ellipse fitting also gave poor results, as it is based on a random selection of solutions. However, as the process is fast in computation time, it can be restarted in the event of poor results.

## 4. CONCLUSION

We presented two algorithms for segmenting the Alexis retractors in 2D endoscopic video images and in a 3D CBCT volume. These algorithms, using conventional image processing techniques, have demonstrated their ability to perform fast processing and their robustness on different datasets. The segmented Alexis retractors will be used to estimate the cost function in a registration process between an endoscopic camera and a CBCT volume for augmented reality purposes.
<figure><img src="image_7.png" /><figcaption>Figure 6. Segmentation of the Alexis of patients B and C. In each case, only one Alexis was in the CBCT field of view. The details of the estimated curvature points and the fitted ellipses are shown next to the mesh surfaces.</figcaption></figure>

ACKNOWLEDGMENTS

The work presented in this article was supported by the French National Research Agency (ANR) as part of the VATSop project (ANR-20-CE19-0015).

REFERENCES

[1] Rouzé, S., de Latour, B., Flécher, E., Guihaire, J., Castro, M., Corre, R., Haigron, P., and Verhoye, J.-P., "Small pulmonary nodule localization with cone beam computed tomography during video-assisted thoracic surgery: a feasibility study," *Interactive CardioVascular and Thoracic Surgery* **22**, 705-711 (Feb. 2016).

[2] Alvarez, P., Narasimhan, S., Rouzé, S., Dillenseger, J.-L., Payan, Y., Miga, M. I., and Chabanas, M., "Biphasic model of lung deformations for video-assisted thoracoscopic surgery (VATS)," in [*2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019)*], 1367-1371, IEEE (Apr. 2019).

[3] Alvarez, P., Rouzé, S., Miga, M. I., Payan, Y., Dillenseger, J.-L., and Chabanas, M., "A hybrid, image-based and biomechanics-based registration approach to markerless intraoperative nodule localization during video-assisted thoracoscopic surgery," *Medical Image Analysis* **69**, 101983 (Jan. 2021).

[4] Noblet, B., Chabanas, M., and Rouzé, S., "Registration of 2D monocular endoscopy to 3D CBCT for video-assisted thoracoscopic surgery," in [*SPIE Medical Imaging 2023: Image-Guided Procedures, Robotic Interventions, and Modeling*], **12466**, SPIE, San Diego (apr 2023).

[5] Besl, P. J. and McKay, N. D., "A method for registration of 3-D shapes," *IEEE Transactions on Pattern Analysis and Machine Intelligence* **14**, 239-256 (Feb. 1992).

[6] Pujol-Miro, A., Ruiz-Hidalgo, J., and Casas, J. R., "Registration of images to unorganized 3D point clouds using contour cues," in [*2017 25th European Signal Processing Conference (EUSIPCO)*], 81-85, IEEE (Aug. 2017).

[7] Tam, W., Dillenseger, J.-L., Babyn, P., and Alirezaie, J., "Rib segmentation in surgical images for Video-Assisted Thoracoscopic Surgery," in [*2024 IEEE 37th International Symposium on Computer-Based Medical Systems (CBMS)*], 450-455, IEEE, Guadalajara, Mexico (June 2024).

[8] Lorensen, W. E. and Cline, H. E., "Marching cubes: A high resolution 3D surface construction algorithm," *ACM SIGGRAPH Computer Graphics* **21**, 163-169 (Aug. 1987).

[9] Vollmer, J., Mencl, R., and Müller, H., "Improved Laplacian smoothing of noisy surface meshes," *Computer Graphics Forum* **18**, 131-138 (Sept. 1999).

[10] Garland, M. and Heckbert, P. S., "Simplifying surfaces with color and texture using quadric error metrics," in [*Proceedings of the Conference on Visualization '98*], VIS '98, 263-269, IEEE Computer Society Press, Washington, DC, USA (1998).
[11] Cignoni, P., Callieri, M., Corsini, M., Dellepiane, M., Ganovelli, F., and Ranzuglia, G., "MeshLab: an open-source mesh processing tool," in [*Eurographics Italian Chapter Conference 2008, Salerno, Italy, 2008*], 129–136 (2008).

[12] Halir, R. and Flusser, J., "Numerically stable direct least squares fitting of ellipses," in [*Proc. 6th International Conference in Central Europe on Computer Graphics and Visualization. WSCG*], **98**, 125–132. (1998).

[13] Fischler, M. A. and Bolles, R. C., "Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography," *Communications of the ACM* **24**, 381–395 (June 1981).

[14] van der Walt, S., Schönberger, J. L., Nunez-Iglesias, J., Boulogne, F., Warner, J. D., Yager, N., Gouillart, E., and Yu, T., "scikit-image: image processing in Python," *PeerJ* **2**, e453 (June 2014).

[15] Hashim, F. and Ramli, A., "Least squares fitting on a segment of ellipse and its application on road curvature estimation," *Computer-Aided Design and Applications*, 476–486 (Oct. 2024).