3D-Organoid-SwinNet: High-content profiling of 3D organoids through Transformer-Based Architecture

Authors: muhammad  sohaib
siyavash  shabani
sahar a mohammed
qingsu  cheng
garrett  winkelmaier
bahram  parvin
Venue: IEEE_org_EMBS_BHI_2024_Conference
Published: 2024-06-28 00:43:32

Abstract: Profiling of Patient-Derived organoids is necessary for drug screening and precision medicine. This step requires accurate segmentation of three-dimensional cellular structures followed by protein readouts. While fully Convolutional Neural Networks are widely used in medical image segmentation, they struggle to capture long-range dependencies necessary for accurate segmentation. On the other hand, transformer models have shown promise in capturing long-range information across domain boundaries. Motivated by this, we present 3D-Organoid-SwinNet, a unique segmentation model explicitly designed for organoid semantic segmentation. We evaluated the performance of our technique using an Organoid dataset from four breast cancer subtypes. We demonstrated consistent top-tier performance in both the validation and testing phases, achieving a Dice score of 94.91 while reducing the number of parameters to 21 million. Our findings indicate that the proposed model offers a foundation for transformer-based models designed for high-content profiling of organoid models.

Main paper: 
 i. introduction: 
 Organoid models have emerged as a powerful tool for cancer therapeutics and investigating biological processes [1] [2] [3] [4] . The scope of drug screening in cancer therapeutics is not limited to apoptotic assays but also differentiation to a phenotype with a better prognosis [5] . In 2D monocultures, cells are exposed only to neighboring cells, attached to a plastic substrate, and lack access to the extracellular matrix (ECM). However, cells in 3D organoid models are organized in a three-dimensional structure, there are more cell-cell contacts, and basal cells have access to the ECM. As a result, 3D cultures can better recapitulate in vivo models for investigating biological processes and drug responses. For example, studies have shown that 3D systems are closer to in vivo in terms of tumor characteristics such as hypoxia and drug resistance [6] and can potentially reduce animal studies and improve the predictive value of drug efficacy [7] . Organoid models are often imaged using a confocal or a microscope equipped with an apotome and deconvolution, resulting in a 3D volumetric image stack. Identifying each nucleus in a 3D colony is essential to compute colony organization and positional protein measurements on a cellby-cell basis. The rationale is that for tumor cells, 3D colony organization can predict tumor subtypes [6] whereas 2D cultures reveal nothing about the subtype for the same cell lines. In addition, therapeutics can differentiate tumor subtypes, which can only be recognized in 3D [5] . These observations are the motivating factors in characterizing colony organization in terms of its nuclei and necessitating nuclei segmentation in 3D images.The current study aims to develop a dictionary of known breast cancer subtypes based on well-characterized cell lines. The dictionary can be represented as a heatmap and a classifier in terms of breast cancer subtypes. The heatmap is a powerful visualization tool since it gives insights into the comparative analysis of Patient-Derived organoids with known cell lines. The representation for each colony/organoid is based on its morphometric properties (e.g., roundness, elongation), which are important in breast cancer subtypes. For example, normal colonies should form hollow spheres, whereas malignant colonies should form solid spheres, elongated, or non-convex colonies. However, the basis for a colony representation is 3D nuclear segmentation, which is the focus of this manuscript. Here, we introduce 3D-Organoid-SwinNet, as shown in Fig. 1 , and it is motivated by recent advances such Swin-UNET [8] and SegFormer [9] . Here, we are motivated by developing and testing a simple architecture with a reduced number of parameters. Several model architectures were investigated, but the system in Fig. 1 produced the best results. 3D-Organoid-SwinNet consists of an encoder and a decoder module. The encoder module consists of a Swin-transformer and multiscale convolutional layers. The decoder module consists of a multiscale MLP layer, upsampling, and a convolutional block. The latter convolutional block operates on a concatenation of scale-space upsampling blocks. The central insight is that a rich representation can bypass secondary convolutional layers and replace them with the MLP layers with much fewer parameters. One disadvantage of transformers is that more training data is typically required. However, taking advantage of global consistency outweighs the requirements for a more extensive training set. Finally, the efficacy of the proposed model is evaluated on four model systems of breast cancer subtypes.The organization of this paper is as follows. Section II reviews prior literature. Section III outlines the approach. Section IV summarizes the results. Section V concludes the paper.Computational approaches for profiling organoid models are either at low resolution using wide-field microscopy [10] , high resolution using a confocal microscope, or a combination of the two [11] . One of the main barriers to highresolution microscopy is the sample thickness that exceeds the working distance of the microscope objectives. Some protocols include cryo-sectioning each spheroid for 2D imaging and nuclear segmentation [12] . Still, this approach requires accurate registration of adjacent sections and stacking the results together later. Because of the complexities associated with the imaging and analysis of organoid models, many protocols quantify organoids in terms of their gross size, morphometric features, and various fluorescent markers [13] [14] [15] [16] ; however, in some cases, these systems are also complemented with organoids cultured in microchips to allow for high throughput microscopy. Concerning high-resolution microscopy and nuclear segmentation, the main challenge is the variation in fixation and staining that leads to clumps of cells in 3D. One group of researchers used classical image analytic methods such as identifying points of maximum curvature or marker-based watershed for partitioning overlapping nuclei [17, 18] . During the last ten years, deep learning (DL) methods have significantly improved over classical techniques because of their intrinsic generalization associated with machine learning methods. Examples of these techniques include using shallow CNN for the detection of nuclei in 3D [19] , UNET [20] , and UNET with a modified loss function [21] , among others. One of the rationales for incorporating a modified loss function is to enhance the partitioning of overlapping nuclei. DL provides a suitable generalization of nuclear signatures (e.g., vesicular phenotypes, a continuum landscape of apoptosis, or fluorescent signature as a cell cycle function). However, DL is sometimes challenged by a clump of nuclei with no apparent boundaries due to fixation and staining. However, this issue can be alleviated by modifying the loss function [21] [22] .In this section, we discuss the structure of our framework, illustrated in Fig. 1 , which consists of Swin-Transformer blocks, 3D convolutional blocks, and MLP blocks.As illustrated in Fig. 1 , the proposed framework comprises a Swin Transformer block, 3D convolutional blocks, and MLP (Multilayer Perceptron) blocks. The Swin Transformer block outputs feature at four different resolution scales, with further details provided in the subsequent section. Following the extraction of global features by the Swin Transformer block, the architecture utilizes five 3D convolutional blocks designed for local feature extraction. Subsequently, MLP blocks are employed to reconstruct the output shape, drawing inspiration from [9] . In the final stage, an additional 3D convolutional layer, equipped with a 1 1 kernel size, serves as the model's terminal layer.Fig. 2 illustrates the Swin Transformer block, which comprises a 3D patch partition block, linear embedding, patch merging, and base Swin Transformer blocks, as presented in the Swin UNETR. In the initial step, the extracted patch, denoted as and with dimensions represented by , is fed into the patch partition block, where H, W, and D denote the three dimensions of the 3D patch and C represents the channel. In greater detail, is divided into N non-overlap patches where and ∈ 1, . In the subsequent layer, through an embedding layer, is projected into a C-dimensional space. According to Fig. 2 , the base Swin Transformer block comprises two layers. The first layer houses the W-MSA (Window-based Multi-head Self-Attention) block, which partitions the inputs into nonoverlapping windows and calculates the local self-attentions for each window as follows [23] : Where , !, " denote query, key, and value, respectively, the widow size for the first layer is / -/ -/ . For the second layer, the window is shifted by Where 8 and /8< stand for Linear Normalization and Multi-Layer Perceptron. Whereas -/6 and 6 -/6 are multi-head self-attention modules for regular and shifted partitioned windows, respectively. , 4H , 384B. These outputs then serve as inputs for 3D convolutions, which extract spatial features in a high dimensional space. This model captures long range dependencies and similarities for 3D organoid segmentation.EQUATIONThe model was trained on a server with 8 Nvidia RTX 2080TI GPUs with a total of 80GB RAM using PyTorch and MONAI. The learning rate is set to 0.0008. Input images with non-zero voxels are modified to have a zero mean and unit standard deviation. Random patches of 96 × 96 × 96 are cropped from 3D image volumes for training purposes. The model trained for 800 epochs with a batch size 1 per GPU. A sliding window approach is used for inference, with a voxel overlap 0.8 between nearby voxels. We incorporate a weight decay of 1e-4, dropout of 0.5 and utilize the Adam optimizer for model training. We implement a five-fold crossvalidation strategy to evaluate the model's performance robustly. An 80:20 split ratio between the training and validation data is maintained during each cross-validation fold. In this study, we utilized a range of data augmentation techniques to enhance the diversity and robustness of our training dataset. Specifically, we implemented RandFlipd, RandRotate90d, and RandShiftIntensityd functions. RandFlipd introduced random flips along specified spatial axes, RandRotate90d performed random 90-degree rotations, and RandShiftIntensityd randomly adjusted the intensity values of images. These augmentation methods were applied to our data's 'image' and 'label' keys with carefully chosen probabilities.Our loss function combines the widely adopted soft dice loss [24] with cross-entropy loss, leveraging the complementary strengths of both. This combined loss function is calculated on a voxel-wise basis. The resulting loss function is defined as:EQUATIONThe first term represents the Dice loss, while the second for binary cross-entropy loss.[ denotes the number of classes, and J and < represent the ground truth and the output probabilities of the model, respectively.The final step is thresholding the prediction, another hyperparameter used during the training, followed by a 3D connected component. Each colony is then represented in terms of nuclei organization and shape. Each nucleus within the colony is also represented in morphometric properties, protein expression, and location (e.g., basal versus luminal).One of the attributes of our approach is that our validation data included organoid cultures that are fixed on different days, and cells cover a wide mutation landscape of breast cancer cell lines. These include MCF10A, MCF7, MDA-MB-231, and MDA-MB-468. MCF10A is non-malignant; MCF7 is estrogen-(ER) and progesterone-receptor (PR) positive and ERBB2 negative, and MDA-MB-231 MDA-MB-468 are triple-negative (TNBC), i.e., ER, PR, and ERBB2. In 3D cultures, these cells can form a hollow sphere, a solid sphere, a sheet-like, or a grape-like structure [25, 26] . Notice that although MDA-MB-231 and MDA-MB-468 are both TNBC, their phenotypes differ. This is an important facet of our future research since we aim to classify phenotypes of primary cells into one of the four categories: hollow sphere, solid sphere, sheet-like, or grape-like. Furthermore, morphometric properties of nuclei tend to be different. For example, nuclei in MCF7 are larger and Most of the dataset includes public data collected from a Zeiss confocal microscope. This dataset was extended with six new image stacks using Zeiss Axiovert equipped with an apotome and a 40X objective with a working distance of 650 microns. This new dataset is continuously being extended, and we plan to include them in our ever-growing datasets. Furthermore, one significant feature of the dataset is the heterogeneity in image quality and microscopy, which promises a model with improved generalization.The proposed model was compared to state-of-the-art models for 3D organoid segmentation on our data.Using our organoid dataset, we compared 3D-Organoid-SwinNet against the previous year's winning approach, Swin UNETR. Table I summarizes the evaluation results. Our proposed 3D-Organoid-SwinNet model outperforms Swin UNETR [8] , Large Language Model (e.g. SAM) [27] and 3D U-Net [28] . The 3D-Organoid-SwinNet achieves the highest performance, attaining a Dice score of 94.91 ± 0.50 and showcasing the trade-off between model complexity and segmentation accuracy of HD95 of 04.03 ± 0.05, with 21M parameters. This comparison highlights the advancements in segmentation techniques. Fig. 3 visually compares the results obtained from our approach and those from other techniques. It demonstrates that our model, 3D-Organoid-SwinNet, outperforms the alternatives, particularly in the context of morphogenesis analysis. In morphogenesis analysis, the precise delineation of individual cells is crucial for understanding how tissues develop and organize themselves. The merging of cells, as observed in the outputs of other methods, can obscure essential details and hinder accurate analysis of morphogenetic processes. Therefore, 3D-Organoid-SwinNet's superior performance in preserving cell boundaries is advantageous for conducting thorough morphogenesis studies and gaining deeper insights into tissue development.Ablation studies include the effect of different numbers of nodes in MLP layers and the different loss functions.Effect of the Number of Nodes in MLP Layers: We empirically analyzed our proposed framework by varying the number of MLP (Multilayer Perceptron) layers. The metrics evaluated include the Dice coefficient, HD95, and the number of parameters for each configuration. The number of nodes considered in this study were 128, 192, 384, and 768. Table II visually represents the effectiveness of different numbers of nodes of MLP layers within our framework. This analysis helps understanding and understanding the trade-offs between model performance and complexity, guiding the selection of an optimal number of nodes, which was 384 for the proposed method.In our study, we also conducted an empirical analysis of our proposed framework using various loss functions, as detailed in Table III C. An application of high content profiling in terms of morphogenesis In this study, we conducted morphogenesis analysis across multiple model cell lines, where the organization of colonies was profiled using multiparametric techniques, notably employing Delaunay triangulation for neighborhood analysis. The mean computed indices are then visualized with a heatmap, as shown in Fig. 4 . These four cell lines correspond to breast cancer subtypes with known molecular aberrations that lead to unique phenotypes. This is evident by MDA-MB-231, which grows in a planar geometry (cultured using the "ontop" method) and is more elongated than the non-malignant MCF10A. As a result, cell lines can serve as a dictionary or a model for predicting the molecular aberrations from Patient-Derived organoids with unknown characteristics. For example, indices computed from the Patient-Derived organoids at a specific date of fixation can correlate with the relevant columns of the heatmap to find the closest match. Although similar analyses can be performed with transcriptomic or other molecular profiling, they are not functional assays and do not capture relevant 3D structures.In this study, we proposed 3D-Organoid-SwinNet, a unique architecture designed for the semantic segmentation of organoids. Our model has a U-shaped network architecture, with a Swin transformer as the encoder and a decoder augmented with an MLP layer that is inspired by a SegFormer framework. We established the efficacy of our approach by validating it on our organoid dataset. The proposed method has only 21M parameters and achieves a new state-of-the-art in terms of the mean Dice and HD95 scores. Our model performs better than the Swin UNETR with fewer parameters. One of the attributes of our design is to validate organoid formation using multiple cell lines with diverse molecular aberrations. For example, MCF7 is ER-and PRpositive; MDA-MB-231 is ER-, PR-, and ERBB2-negative; and MDA-MB-468 is ERBB2-positive. As a result, the Fig. 4 . The multiparametric analysis is visualized for colony organization across various cell lines. This analysis incorporates multiple parameters and offers insights into cellular dynamics and organization across different cell lines. computational method is stress-tested under a diversity of molecular aberrations. Furthermore, the 3D phenotypes of these cell lines serve as a dictionary for correlative studies of Patient-Derived organoids with unknown molecular characteristics. Our future research focuses on profiling Patient-Derived organoids and predicting their molecular signatures based on the model systems constructed in our current efforts. 
 
 Appendix:  



Reviews: review: title: title: A efficacious transformer architecture for organoids semantic segmentation, out performed the state-of-the-art models over standard dataset.
paper_summary: paper_summary: The authors proposed a transformer-based model with unique structure for organoids semantic segmentation:
For each of the downsampled (2/4/8/16) input layer , apply "Swin" Transformer for global features extraction, followed by 3D conv layers for local feature extraction,  MLP for reconstruction; Then upsample to original dimension for concatenation and final 3D conv as output.
The model evaluation with breast cancer organoid dataset indicates it outperforms other state-of-the-art models, it achieved top-tier performance with reduced parameters.
strength_weakness: strength_weakness: Fig 4 - the provided example is suppose to showcase the superior performance in terms of segmentation quality - but  the GT model also doesn't have any merged adjacent nuclei ( at least from the provided view)?
questions: questions: Fig 4 - the provided example is suppose to showcase the superior performance in terms of indicates superior performance in terms of segmentation quality -
title: title: Transformer-based architecture for profiling of 3D organoids
paper_summary: paper_summary: This paper presents 3D-Organoid-SwinNet, a Transformer-based architecture for organoid semantic segmentation. The authors evaluated the performance of 3D-Organoid-SwinNet using a oganoid dataset from four breast cancer subtypes. Experiment results demonstrated that the proposed method can achieve good performance while reducing the number of parameters.
strength_weakness: strength_weakness: The novelty of this work is insufficient. It seems that the author just use the swin-Transformer moduel to design a neural network for organoid semantic segmentation and do some experiments on some open dataset. The writing of the article is also not very good. Specifically, the title of section IV and section V are both "results and discussion".
questions: questions: 1. Are there some particular design and difficulties when using Transformer-based architecture on this question?
2. Does the author constructed a new dataset based on the existing public dataset for this study?
3. What are the disadvantages of this method compared with the existing method?
title: title: Well-written and demonstrating significant improvement using Swin transformer network with multilayer perceptron and reduced parameter complexity applied to morphological characterization of four breast cancer cell lines.
paper_summary: paper_summary: The paper proposes a transformer model consisting of a Swin transformer outputting four scales of resolution to multilayer perceptrons, upscaler, and finally a convolutional block. Four breast cancer cell lines are used, each at two timepoints, and characterized by nuclear shape and organization. This framework, which the paper names 3D-Organoid-SwinNet, is compared to OTSU Thresholding, Segment Anything 2D, 3D U-NET, and Swin UNETR. Despite having significantly fewer parameters than the latter three comparison methods, the paper demonstrates that 3D-Organoid-SwinNet improved segmentation accuracy based on Dice score of 94.91 and a HD95 of 4.03.
strength_weakness: strength_weakness: None.
questions: questions: Are the two timepoints chosen as a stepping stone for further time resolution of the proposed dictionary in the future, or is there a specific reason that days five and seven were chosen for this paper?