Segment Anything by Meta as a foundation model for image segmentation: a new era for histopathological images

To the Editor, Deep learning for computer vision is revolutionising histopathological image analysis. One of its major applications is image segmentation, where images are classified at pixel-level (e.g., tumour or stroma, nuclei segmentation). Supervised learning is the simplest and most common approach to model training but can require hundreds to thousands of labelled examples to expect reasonable and generalisable performance for complex tasks. Fully labelled images consist of segmentation masks, where expected elements must be annotated as precisely as possible. Manual annotations are standard practice, but are highly time-consuming and often require histopathological expertise.1Montezuma D. Oliveira S.P. Neto P.C. et al.Annotating for artificial intelligence applications in digital pathology: a practical guide for pathologists and researchers.Mod Pathol. 2023; 36100086Abstract Full Text Full Text PDF PubMed Scopus (4) Google Scholar Several tools or trained models have been developed to ease the manual annotation process, but are often tissue-specific, task-specific (tumour type, microenvironment, nuclei segmentation, etc), with variable generalisation performance. Moreover, many publications share their process but only a few of them share the trained models, often without the annotated dataset, limiting their actual reusability.2Bankhead P. Developing image analysis methods for digital pathology.J Pathol. 2022; 257: 391-402Crossref PubMed Scopus (19) Google Scholar Our aim here is to raise awareness about Segment Anything, a foundation-like model recently released by Meta, and to explain how it can simplify the annotation process in segmentation tasks from histopathological images. Lastly, we examine its major strengths and limitations. In April 2023, Meta released Segment Anything, an attempt to build a foundation model for image segmentation.3Kirillov A. Mintun E. Ravi N. et al.Segment Anything.http://arxiv.org/abs/2304.02643Google Scholar Foundation models are models trained on a large dataset, making them adaptable to multiple downstream tasks,4Bommasani R. Hudson D.A. Adeli E. et al.On the opportunities and risks of foundation models.http://arxiv.org/abs/2108.07258Google Scholar as evidenced by OpenAI's GPT-4 in natural language processing.5OpenAI. GPT-4 Technical Report.http://arxiv.org/abs/2303.08774Google Scholar The training dataset for the Segment Anything project, named SA-1B, consisted of 11 million images from diverse domains and over 1 billion corresponding segmentation masks. Using a large dataset of annotated images could theoretically allow the model to learn to recognise important objects from a human perspective, even in image domains that were not primarily covered by the training dataset. This is known as the ‘zero-shot’ performance of a model. For example, to our knowledge, the dataset used for SAM training does not contain many microscopic images, if any. Yet, as stated below, it can perform very decently for some applications. Unlike popular segmentation workflows in biomedical images, which are often based on convolutional neural networks (CNN) with a U-Net-like architecture,6Deng S. Zhang X. Yan W. et al.Deep learning in digital pathology image analysis: a survey.Front Med. 2020; 14: 470-487Crossref PubMed Scopus (70) Google Scholar SAM uses a vision transformer-based encoder to process the image. Transformers are deep learning models primarily used in natural language processing, and have more recently emerged in computer vision, known as vision transformers, showing at least equivalent performance in most common computer vision tasks (classification, segmentation and object detection).7Ghaffari Laleh N. Truhn D. Veldhuizen G.P. et al.Adversarial attacks and adversarial robustness in computational pathology.Nat Commun. 2022; 13: 5711Crossref PubMed Scopus (17) Google Scholar SAM is intended to be promptable and interactive, i.e., allowing the model to respond to a direct user interaction. Indeed, objects of interest can be highlighted by the user, mainly using one or several points and/or bounding boxes (Fig. 1A). Then, SAM can produce a segmentation mask proposal of the object under consideration, or multiple masks to deal with object ambiguity. For example, object ambiguity can occur when a single point on a nucleus is selected by the user, which could either refer to the nucleus itself, the corresponding cell or the corresponding group of cells (gland, tumour nest, etc). The main purpose of SAM is to produce a valid mask, i.e., a relevant mask from a human perspective, that is class agnostic, so that proposed masks do not have a label (such as gland, tumour, etc). In this setting, one must avoid ambiguity in the selection of the region of interest prior to segmentation by SAM and assign the label of the produced mask afterwards. SAM is also able to produce masks automatically, without requiring any prompt by the user, and so displays mask proposals for each identified object of the provided image. SAM and the corresponding dataset used to train it were released under a permissive open license by Meta, making them readily available for researchers. Yet, its current implementation still requires users to be familiar with Python programming and also a graphics processing unit (GPU) to expect an interactive use. Image segmentation from microscopic images encompasses multiple applications, due to either image types (from simple photographs to a whole slide image), the required task (histological or pathological segmentation) and/or the corresponding image magnification. In this setting, SAM's zero-shot performance can facilitate the manual and laborious annotation process, where an object's annotation can be reduced to one or a few points or one bounding box. Fig. 1B shows representative images of SAM-aided annotations with several stains and from a histological or pathological basis. Moreover, we quantified and compared the time required to annotate elements from various tissues and stains, using SAM and using the traditional annotation tools of the open-source software QuPath.8Bankhead P. Loughrey M.B. Fernández J.A. et al.QuPath: open source software for digital pathology image analysis.Sci Rep. 2017; 716878Crossref PubMed Scopus (2732) Google Scholar As displayed in Table 1, using SAM allowed a significant time saving for annotating glomeruli and tubules from kidney biopsies and striated muscle cells (from 30% to 70% compared to traditional tools). Indeed, from our experience, SAM shows good zero-shot performance for sharply defined object-like elements (tumour nests, vascular lumen, nuclei, structures surrounded by basement membranes such as tubules or glomeruli in the kidney) from multiple stains. SAM performs better in instance segmentation, where the user highlights each object of interest (points or bounding boxes) rather than in semantic segmentation tasks, where highlighting a few objects of interest should be sufficient to segment all objects of that class in the image. By combining SAM with a slide-centred software like QuPath, segmentation results can be integrated at the level of a whole slide image (WSI). Implementing SAM into WSI-centred software is currently under active development. Of note, a recent SAM-dedicated extension was released for QuPath,9Sugawara K. Training deep learning models for cell image segmentation with sparse annotations.bioRxiv. 2023; 13 Junhttps://www.biorxiv.org/content/10.1101/2023.06.13.544786v1Google Scholar which allows direct implementation of SAM for interactive object segmentation from fluorescent or brightfield WSI. We have prepared a video which shows examples of interactive annotations using SAM within QuPath from various tissues: https://drive.google.com/file/d/1nnlA3qHcrNq4PhTLN5Ih-MAUs8vKv1qp/view?pli=1.Table 1SAM-aided annotations can ease the annotation process from histopathological images as compared to traditional toolsTissue, stainInput prompts for SAMTime required, in seconds, median (min–max)Median time saving using SAM (%)p value (Wilcoxon signed-rank test)QuPath annotator toolsSAM-aided annotations within QuPathKidney biopsy, Masson's trichrome Glomeruli, n=10Points42 (36–58)27 (20–38)37%0.03 Tubules, n=25Points75 (66–90)34 (25–47)55%0.04 Vessels, n=5Boxes25 (22–42)28 (21–37)–12%1.0Kidney biopsy, periodic acid–Schiff Glomeruli, n=10Points39 (31–47)26 (17–35)33%0.03 Tubules, n=25Points66 (63–72)31 (28–35)53%0.04 Vessels, n=5Boxes26 (23–33)24 (18–34)10%0.28Kidney biopsy, haematoxylin Glomeruli, n=10Points39 (33–51)24 (15–30)39%0.04Striated muscle tissue, frozen H&E Muscle cells, n=30Points65 (56–69)20 (18–23)70%0.04To demonstrate how SAM can facilitate the annotation of objects from histopathological images, we quantified the time required to annotate several objects in slides from various tissues and stains. We compared two methods: (1) using the traditional tools (wand or brush tools) of the QuPath software,8Bankhead P. Loughrey M.B. Fernández J.A. et al.QuPath: open source software for digital pathology image analysis.Sci Rep. 2017; 716878Crossref PubMed Scopus (2732) Google Scholar and (2) using the QuPath extension dedicated to SAM,9Sugawara K. Training deep learning models for cell image segmentation with sparse annotations.bioRxiv. 2023; 13 Junhttps://www.biorxiv.org/content/10.1101/2023.06.13.544786v1Google Scholar which allows a direct implementation of SAM while staying in the software interface. The time required is displayed, in seconds, as the median (minimum–maximum) of six replicates (six different slides for biopsies and six different regions of the same slide for striated muscle) and encompasses both the annotations and corrections if necessary. All samples are formalin-fixed and paraffin-embedded unless otherwise stated. Haematoxylin refers to the counterstaining of a C4d immunostaining. In this case, tubules and vessels were not assessed, as both SAM and the human annotator (BC) struggled to detect and define object boundaries. Examples with non-object elements (which cannot be enumerated, e.g., fibrosis) are not presented, as SAM is not suited for their annotations. H&E, haematoxylin and eosin. Open table in a new tab To demonstrate how SAM can facilitate the annotation of objects from histopathological images, we quantified the time required to annotate several objects in slides from various tissues and stains. We compared two methods: (1) using the traditional tools (wand or brush tools) of the QuPath software,8Bankhead P. Loughrey M.B. Fernández J.A. et al.QuPath: open source software for digital pathology image analysis.Sci Rep. 2017; 716878Crossref PubMed Scopus (2732) Google Scholar and (2) using the QuPath extension dedicated to SAM,9Sugawara K. Training deep learning models for cell image segmentation with sparse annotations.bioRxiv. 2023; 13 Junhttps://www.biorxiv.org/content/10.1101/2023.06.13.544786v1Google Scholar which allows a direct implementation of SAM while staying in the software interface. The time required is displayed, in seconds, as the median (minimum–maximum) of six replicates (six different slides for biopsies and six different regions of the same slide for striated muscle) and encompasses both the annotations and corrections if necessary. All samples are formalin-fixed and paraffin-embedded unless otherwise stated. Haematoxylin refers to the counterstaining of a C4d immunostaining. In this case, tubules and vessels were not assessed, as both SAM and the human annotator (BC) struggled to detect and define object boundaries. Examples with non-object elements (which cannot be enumerated, e.g., fibrosis) are not presented, as SAM is not suited for their annotations. H&E, haematoxylin and eosin. The use of SAM for annotating histopathological images has several limitations. Firstly, SAM struggles with objects that (1) are interconnected or have loose borders (e.g., vascular wall), (2) are subject to prompt ambiguity (e.g., an entire vessel or its lumen), and/or (3) have a low contrast with the background (e.g., sparse tumour cells within an inflammatory background). Apart from the lack of microscopic images in the training set, this is at least partially explained by technical aspects of histopathological images as compared to real life photographs: (1) image colours are stereotyped and not always diverse between objects (e.g., interstitium and the media of an arteriole are globally green in Masson's trichrome), and (2) tissue elements are in essence on the same plane in histopathology, while real world photographs are two-dimensional images of a three-dimensional environment, leading to sharper edges between objects. Moreover, it remains difficult to obtain correct masks for non-object elements (i.e., non-enumerable elements such as the stroma or the interstitial compartment), even when multiple points are given. Technical artifacts can also lower the performance of the segmentation (e.g., clarification at the edges of a biopsy, tearing artifacts). Secondly, the overall performance of SAM remains lower than the performance of state-of-art models dedicated to a specific task, especially for semantic segmentation tasks such as nuclei segmentation or kidney segmentation. Yet, the purpose of foundation models is to be easily accessible and usable for diverse tasks or projects that were not originally planned and that are currently not solved by accessible models. The interactive nature of SAM through prompts is the hallmark of its ability to rapidly build an annotated dataset. Fine-tuning, i.e., adjusting SAM to a specific task by retraining it, could be a way to specialise SAM in the histopathological field. In this setting, Ma et al. introduced Segment Anything in medical images (MedSAM), an attempt to build a SAM-like foundation model to segment objects from medical images.10Ma J. He Y. Li F. Han L. You C. Wang B. Segment Anything in medical images.http://arxiv.org/abs/2304.12306Google Scholar By training MedSAM with over one million medical images, they demonstrated an increase in overall performance for image segmentation from various modalities (notably radiology, endoscopy, pathology and dermoscopy). Still, pathological images accounted for less than 1% of the training dataset of MedSAM, and there are still many efforts to be made in this field. Thirdly, SAM must be coupled with a deep learning-dedicated GPU and workstation to expect an interactive use and best performance. So far, at least minimal programming skills are still required to use SAM. To conclude, Meta recently released Segment Anything model, a foundation-like model dedicated to image segmentation. It was released under a permissive open license, making it readily available for researchers familiar with Python programming. SAM performs decently from histopathological images, especially with sharply defined objects, and can already significantly ease the annotation process in segmentation tasks when dedicated deep learning models do not currently exist or are not easily accessible. In the future, one can expect that foundation models dedicated to histopathological images will emerge, with a significant increase in overall performance in this domain. Integration of such foundation models for computer vision with open-source software dedicated to WSI analysis is currently under active development. This will truly reveal the potential of such approaches and make them easily usable and accessible to researchers, in an interactive way, for many histopathological applications. The code used to create Fig. 1 is freely accessible, with user instructions, at https://github.com/bertrandchauveau/FromQuPathToSAM. The authors state that there are no conflicts of interest to disclose.