Histopathobiome – integrating histopathology and microbiome data via multimodal deep learning

Authors: agata  polejowska
annemarie  boleij
francesco  ciompi
Venue: MICCAI_org_2024_Workshop_COMPAYL
Published: 2024-06-29 20:40:19

Abstract: We introduce Histopathobiome, a term representing the integration of histopathology and microbiome data to explore tissue-microbe interactions. Using a dataset of colon biopsy whole-slide images paired with microbiota composition samples, we assess the benefits of
combining these modalities to distinguish patients with inflammatory bowel disease (IBD) subtype – ulcerative colitis (UC) from non-IBD controls. Initially, we evaluate the unimodal performance of state-of-the-art algorithms using vectors representing bacterial species abundances or histopathology slide-level embeddings. We compare single-modality models with bimodal networks with various fusion strategies. Our results prove that histopathology and microbiome data are complementary in UC classification. By demonstrating improved
performance over single-modality approaches, we prove that bimodal deep learning models can be used to learn meaningful and interpretable cross-modal tissue-microbe patterns.

Main paper: 
 introduction: 
 We investigate tissue-microbe interactions in inflammatory bowel disease (IBD) through the application of multimodal learning techniques fusing histopathology and microbiota abundance representations. The rationale for combining these two modalities lies in their complementary nature. Histopathology provides detailed insights into tissue morphology and pathological changes, such as ulcers, inflammation, and crypt distortion, which are important in diagnosing and staging diseases like ulcerative colitis (UC) and Crohn's disease (CD), the primary IBD subtypes. However, some features might overlap between them making it more challenging to use only histopathology for definite diagnostics (Kellermann and Riis (2021) ). Also, histopathology may not capture the underlying microbial composition and dynamics that can influence these pathological changes. Conversely, microbiome data can reflect the community of microorganisms residing in the tissue and provide information on microbial composition and potential dysbiosis -an imbalance in microbial populations often associated with diseases (Nishida et al. (2018) ; Wei et al. (2021) ). For example, known microbes, such as Escherichia coli and Bacteroides fragilis, have been implicated in UC pathogenesis (Bruggeling et al. (2023) ). Yet, relying solely on microbiome data for diagnosis is challenging due to the dynamic nature of microbial communities, which can be influenced by various factors and subjected to the quality of preparation methods (Garrett (2019) ; Hajjo et al. (2022) ; Schlaberg (2019) ).Thanks to recent advancements in computational pathology and metagenomics sequencing technology, the horizons for artificial intelligence to help understand the complexities behind diseases have been broadened. Computational pathology models are now capable of generating efficient slide-level representations in the form of embeddings, enabling compression of large histology slides into one vector (Xu et al. (2024) ; Song et al. (2024) ). Meanwhile, metagenomics sequencing can provide interpretable microbial profiles (Shi et al. (2022) ). However, integrating heterogeneous data, such as histopathology and microbiome data, into a common, learnable representation is still a challenge. Over the years, several fusion techniques and multimodal architectures have been proposed. The basic categories of fusion include early fusion that combines raw data early in the pipeline using simple operations like concatenation, but this may remove structural modality-specifc information. Another popular method is late fusion that uses outputs from separate models trained on different modalities, preserving modality-specific features but potentially missing cross-modal interactions. Recent multimodal frameworks (Hemker et al. (2023) , Jaume et al. (2024) ) address common drawbacks encountered in basic fusion architectures and demonstrate the potential to efficiently capture cross-modal information while preserving modality-specific features and being able to outperform other baselines.To our knowledge, our work presents the first application of deep learning to integrate histopathology and microbiome data. The primary aim is to verify whether a deep learning model can depict their complementary nature and thus improve classification of UC patients in remission versus non-IBD patients. We leverage advancements in computational pathology and metagenomics to use a common representation format. We conduct experiments on each modality independently using well-established models. We design and adapt distinct bimodal models by testing different fusion techniques while exploring their abilities to capture cross-modal patterns. The results are concluded not only based on various performance metrics but also by including interpretability measures as the focus is also placed on discovering new tissue-microbe interactions to broaden the understanding of IBD. 
methods: 
 Dataset The dataset originates from an investigation of the association of the mucosal microbiome, bacterial oncotraits with neoplasia development in UC-patients (Bruggeling et al. (2023) ). Biopsies were collected from the ascending and descending colon of 80 UC patients and 35 non-IBD controls. Inclusion criteria for UC patients included: > 8 years of disease, left-sided or pancolitis and patients in clinical remission. The ground truth was provided by a gastrointestinal pathologist. The detailed bacterial species composition from clinical tissue samples was derived thanks to isolating the bacterial DNA using an optimized method and shotgun metagenomic sequencing (Bruggeling et al. (2021) ). Haematoxylin and Eosin (H&E) stained pathology slides were analyzed together with other techniques. For our experiments, from the original study we selected two modalities: H&E images and relative abundances of bacteria taxa at genus level. Initially, 234 tissue samples were collected, from which we excluded 32 samples due to being out of focus. For image processing, tissue samples were divided into non-overlapping patches of 224 × 224 pixels at a spacing of 0.5 micrometers per pixel, resulting in a total of 30644 patches for the entire dataset. Using the extracted patches and their coordinates, we leveraged a foundation model's ability to generate slide-level representations. Specifically, we used a recent Prov-GigaPath model pretrained on a large pathology dataset in which bowel tissue is the second most common tissue type (Xu et al. (2024) ). For the microbiome modality, we use raw bacterial composition values at genus level stored in tabular form. Figure 1 provides a high-level overview of the data. We use pairs of microbiota composition from the tissue sample and the corresponding tissue histopathology image from the left and right side of the colon. For each histopathology whole-slide image, we used 768 values in the feature vector as a slide-level representation. For microbiota data, we obtained a vector length of 432 representing relative abundance values for each of 432 bacterial taxa classified at genus level. 2022)) within PyTorch Tabular framework (Joseph, 2021) .For comparison, we also check the performance of an ensemble gradient-based machine learning approach, namely GRANDE (Marton et al. (2023) ). From the deep learning approaches, FT-Transformer was chosen for the experiments as it has proven to be a versatile model for a wide range of tabular data problems despite being defined as a simple adaptation of the Transformer architecture to the tabular data domain (Gorishniy et al. (2021) ). The AutoInt model is also included in the experiments due to its ability to learn feature interactions in an explicit manner via using an interacting layer which determines the fea-tures' relevance. GANDALF was selected due to its reported robust feature selection and learning capabilities, excelling in datasets with numerous features or complex feature-target relationships. Also, empirical results demonstrate GANDALF's better performance over a plain Multi-layer Perceptron (MLP) (Joseph and Raj (2022) ).Bimodal models For our use case, we define multimodal, specifically bimodal, models as networks capable of learning cross-modal information from two inputs coming from two different data domains. Despite having transformed the image data into a common format aligned with the microbiota modality, the meaning of the values is different and each modality requires distinct registration and derivation techniques. Firstly, we use the GANDALF architecture on combined histopathology embeddings and microbiota abundances together as parallel input fed into the network. This network was selected as it performed the best among the introduced tabular models showing that it can learn from a data structure with a high number of features. Next, we design and test vanilla fusion models. The early fusion model firstly combines histopathology features and microbiota features via a concatenation operation. The combined input undergoes layer normalization to stabilize the training process. Then, linear layers are used to project the values onto a hidden representation. The late fusion model is built with separate networks to process each input feature independently. The outputs of these two networks are concatenated and fed into a simple fully connected fusion network. Also, we employ more complex fusion techniques. We adapt the Hybrid Early-fusion Attention Learning Network (HEALNet) (Hemker et al. (2023) ) for learning from whole-slide image embeddings and microbiota abundance vectors, both defined in a tabular structure for the network input. The next network tested, inspired by SurvPath work (Jaume et al. (2024) ), is comprised of a self-normalizing neural network (SNN) block for encoding microbiota vectors and a simple linear projection layer for WSI embeddings. The tokens from those blocks are concatenated and fed into the Nystrom Attention (NA) module. The embeddings obtained after this self-attention approximation are used in the network learning process.Training The models are trained to classify whether a sample belongs to an individual with confirmed UC or a non-IBD control group. As this is a binary classification, each model network uses Binary Cross Entropy loss during the learning procedure. The length of the training is dependent on the convergence speed and criteria. For this, an early stopping callback is set to stop the learning when the validation loss function does not decrease further after 5 consecutive epochs (patience number equal to 5). The maximum number of epochs is set to 50 with a batch size of 16, a learning rate equal to 0.001 and an Adam optimizer. For training, 64% of the dataset is used.Validation and testing KFold (k=5) cross-validation is performed. During the validation and testing phase, group ratios are maintained in each split without patient overlap.The validation includes 16% of the dataset. The metrics include: the Area Under the Receiver Operating Characteristic Curve (AUC), Average Precision (AP), and the normalized Matthews Correlation Coefficient (MCC) as it has been reported to address the other metrics' drawbacks (Chicco and Jurman (2023, 2020) ). However, as there is also evidence that the MCC alone might not be a suitable measure for an imbalanced dataset (Zhu ( 2020)), we also compute AP and AUC. The metrics values reported for cross-validation are the mean and standard deviation measured across all folds. For a fixed test set (20% of the whole data), the best performing model based on the cross-validation mean MCC score is selected, and metrics on the test set for this model are reported.Interpretability To gain insights into tissue-microbe relations, we designed a pipeline to interpret the model's decisions. Based on the conducted experiments, and to be able to look into the unimodal model feature importance assignment process as well as the model decisions that use both modalities, we select the GANDALF model. We use interpretability methods, such as Feature Ablation and GradientShap, implemented in Captum framework (Kokhlikyan et al. (2020) ) available for the models via PyTorch Tabular interface (Joseph (2021) ). When whole-slide image embedding indices are used as features, we perform patch ablation, checking which patch contributes to the greatest change in the corresponding value in the feature vector. The explanations are run on the whole test set. 
results and discussion: 
 Based on Table 1 , the most relevant difference in metrics is seen on the test dataset, where the GANDALF model leads. Since it does not outperform other methods on validation data, this suggests GANDALF may benefit from delayed generalization, with its true predictive power emerging on unseen data. Figure 2 indicates that Absiella, Leuconostoc, Lactococcus and Sporanaerobacter have the highest attributions for the UC vs. non-IBD classification. Results in Table 2 show that the self-attentive neural network (AutoInt) outperforms other models in this classification task based on whole-slide image embeddings. For congruity, embedding index relevance is computed for the GANDALF model (Figure 3 ), so that the most important patches for a given embedding index can be verified. We provide visualizations of the most impactful patches (Figure 5 ) for the most relevant index 621 value for the unimodal model as well as index 399 value since it appears as the feature with one of the highest attribution values in both settings (see Figure 3 and Figure 4 ). According to the obtained results, particularly the AP score on test data, microbiomeonly data shows some informative value for classifying IBD patients. However, when considering performance across all models and metrics, histopathology slides embeddings tend to outperform microbiome data overall. This suggests that the superiority of one modality is not definitive, highlighting the challenge of using a single modality for optimal performance. Therefore, performance gains may be achievable by employing fusion techniques to merge both modalities and capture interconnected patterns.For multimodal performance (Table 3 ), the self-normalizing network together with selfattention approximation module gives the best results on the test dataset given AUC and MCC metrics. Simpler approaches, like Early Fusion and Late Fusion, show even better performance, according to AP values, compared to more complex ones. Also, the GAN-DALF model benefits from learning on merged representations achieving improved results on the test set based on AUC and MCC values in contrast to when used for single-modality experiments. In Figure 4 , it can be seen that microbiota abundances largely dominate the model's decision-making process. Index 399 and 717 of whole-slide image embeddings appear twice in both interpretability methods. Values in those indices are mostly impacted by the patches shown in Figure 5 . It can be observed that more focus is placed on colon crypts in the case of non-IBD tissue samples. This is reasonable as in ulcerative colitis, the crypts might be distorted. Additionally, some patches capture mainly cells which are an essential feature to be inspected as a high number of inflammatory cells is attributed to UC (Rubio and Schmidt ( 2018)). When compared to input based solely on microbiota (Figure 2 ), Leuconostoc bacteria continue to be ranked at the top (Figure 4 ). Interestingly, a recent study has reported that Leuconostoc species, which are found in kimchi (Moon et al. (2023) ), are capable of attenuating the inflammatory responses associated with colitis. In fact, the control group has, on average, higher levels of Leuconostoc than the colitis group in our dataset.Furthermore, when adding histopathology data, some bacteria like Intestinibacter, Lachnobacterium and Agathobaculum become more important, while others like Marvinbryantia and Kocuria no longer appear in the top rankings. The increased importance of Intestinibacter is promising as a new finding, congruent with the reported association of this genus with lower genetic risk for autoimmunity in the gut (Russell et al. ( 2019)). Overall, the observations presented prove that a multimodal approach can reveal scientifically plausible insights into how specific bacteria can impact tissue and thus host health. 
limitations: 
 Further potential enhancement of model performance might be achieved through the implementation of alternative architectures and experimental set-ups. As model performance improves, the reliability of interpretability method outputs is expected to increase. However, the selection of interpretability methods requires careful consideration. For instance, tests with KernelShap demonstrated a higher prevalence of histopathology features in the attribution plots. Further, the exploration of other pretrained models for generating wholeslide image embeddings warrants further investigation. Next, the relevance of patches for histopathology and microbial abundance needs to be confirmed with experts and larger validation cohorts to confirm the biology and performance of the models. 
conclusions: 
 This study contributes the foundation for future development of the new Histopathobiome learning and exploration application. The experimental results suggest that the results obtained might reflect the inherent complexity of the dataset or UC in general. Nevertheless, combining different features using bimodal models can lead to better results compared to single-modality-based algorithms, thus proving Histopathobiome potential. Furthermore, we demonstrated the exploration of tissue-microbe patterns, using interpretability techniques, and confronted the outcomes with the literature. This application of artificial intelligence shows promise in uncovering previously unknown associations and demonstrates the benefits of combining histopathology with microbiome data. This approach may extend to other diseases and conditions as understanding the biological environment and the coexistence with bacteria can lead to exploring their potential to treat or prevent various diseases as we deepen our understanding of their impact with the help of deep learning. 
 
 Appendix:  



Reviews: review: title: title: Exploratory work on multi-modal learning combining WSIs and microbiome data
main_review: main_review: **Overview**

In this paper, the authors explore combining imaging data from Whole Slide Images (WSIs) with microbiome data from the colon to predict inflammatory bowel disease. The simultaneous utilization of these data is the main novelty of the paper. The authors explore different approaches, such as late and early fusion. Results show the benefits of combining multi-modal data. Furthermore, the authors attempt to interpret the most relevant features of the task. The methodology is simple, serving well the purpose of this exploratory work in an unexplored setting.

**Pros**

1. The text is well-written and easy to follow.
2. The motivation for combining imaging data with microbiome data is sound.
3. Exploring this combination of data is novel.
4. Using the multi-modal approach significantly improves results, thus demonstrating its usefulness.

**Cons**

1. The dataset is relatively small, with 202 tissue samples (after excluding 32 due to out-of-focus). 
2. While combining modalities helps, the results of unimodal approaches are intriguing. Some of the entries in Tables 1 and 2 show AUROC just around or slightly above 0.5, suggesting the classifier performs similarly or just a little better than a random classifier.


**Further comments**

Regarding the results:

1. In Table 1, the results of the validation and test set are similar for most methods. The exception is GANDALF, where the performance of the test set significantly improves. This is different from the other methods, and somewhat unexpected since validation is used to tune the models. Could the authors elaborate on what may be the reasons?
2. In Figure 5 the authors show influential patches for the predictions. It would be interesting if the authors could comment on the content of the patches and if the patterns make sense for the task at hand.
3. As mentioned in “Cons” the AUROC is relatively low for unimodal approaches but increases significantly with the multi-modal approaches.

    a. In the unimodal methods (Table 1 and 2) we observe AUROC mostly in the range [0.5, 0.6] and some around 0.5, being close to a random classifier. It would be interesting if the authors could comment on this aspect.

    b. In Table 3, however, several methods exhibit AUROC > 0.7, and the best is 0.8. This implies that combining modalities is effective. 

    c. However, it was very challenging for unimodal approaches. Perhaps, the interaction of the data unlocks key aspects for better predictions. Shedding light on this could be interesting in future research.
title: title: Review of Histopathobiome
main_review: main_review: The authors explore the impact of combining tissue based (from histopathology slides) and microbiome data for Inflammatory Bowel Disease (IBD) prediction. They use a foundational model (FM) to extract slide-level representations, whereas for microbiome data they use a vector representation in which each dimension corresponds to a bacteria, and the values represent their relative abundance.

Pros:
- The work proposes a novel integration of two modalities: histopathology and microbiome. 
- The authors conduct a cross-validation analysis and explore different methods to prove their claim: the interaction between tissue and microbiome leads to better understanding of IBD. 
- They also conduct an interpretability analysis for each experiment.
- Their results match with existing literature for microbiota data.
- Experimental choices and decisions are well justified and explained across the text.
- The manuscript is well written and easy to follow.

Cons/Comments:
- How was the data partitioned? Train + test were divided in a stratified manner, and then train is used for 5-fold cross-validation? Given the very small size of the dataset, as well as its imbalance, it would have been interesting to see an averaged performance across different train-test splits, to see the variability of the results for different data partitions. For instance, it's surprising that test metrics in Table 1 and Table 2 are significantly larger than cv ones for some methods. 
- The multi-modality is addressed by modality independent encoders (actually only for tissue) and then combined. It would be interesting to explore how to combine these modalities at earlier stages.
title: title: Original and timely paper
main_review: main_review: This work is a nice example of bimodal data integration, using histopathology and microbiome data (“histopathobiome”) to tackle the issue of classifying patients with ulcerative colitis from non-IBC patients. One question arises with regards to the ground truth used for these images- where does it come from? Also, the current classification of UC depends on 3 features: DCA  (distribution, chronicity and activity), where a score of 0 for each implies a normal biopsy. How do the models compare to the DCA score? One of the features (D) is not actually related to histomorphology per se, but rather to the area of the biopsy which is affected. This feature could of course not turn up in the patch-wise approach.