A Novel and Efficient Digital Pathology Classifier for Predicting Cancer Biomarkers Using Sequencer Architecture

In digital pathology tasks, transformers have achieved state-of-the-art results, surpassing convolutional neural networks (CNNs). However, transformers are usually complex and resource intensive. This study developed a novel and efficient digital pathology classifier called DPSeq to predict cancer biomarkers through fine-tuning a sequencer architecture integrating horizontal and vertical bidirectional long short-term memory networks. Using hematoxylin and eosin–stained histopathologic images of colorectal cancer from two international data sets (The Cancer Genome Atlas and Molecular and Cellular Oncology), the predictive performance of DPSeq was evaluated in a series of experiments. DPSeq demonstrated exceptional performance for predicting key biomarkers in colorectal cancer (microsatellite instability status, hypermutation, CpG island methylator phenotype status, BRAF mutation, TP53 mutation, and chromosomal instability), outperforming most published state-of-the-art classifiers in a within-cohort internal validation and a cross-cohort external validation. In addition, under the same experimental conditions using the same set of training and testing data sets, DPSeq surpassed four CNNs (ResNet18, ResNet50, MobileNetV2, and EfficientNet) and two transformer (Vision Transformer and Swin Transformer) models, achieving the highest area under the receiver operating characteristic curve and area under the precision-recall curve values in predicting microsatellite instability status, BRAF mutation, and CpG island methylator phenotype status. Furthermore, DPSeq required less time for both training and prediction because of its simple architecture. Therefore, DPSeq appears to be the preferred choice over transformer and CNN models for predicting cancer biomarkers. In digital pathology tasks, transformers have achieved state-of-the-art results, surpassing convolutional neural networks (CNNs). However, transformers are usually complex and resource intensive. This study developed a novel and efficient digital pathology classifier called DPSeq to predict cancer biomarkers through fine-tuning a sequencer architecture integrating horizontal and vertical bidirectional long short-term memory networks. Using hematoxylin and eosin–stained histopathologic images of colorectal cancer from two international data sets (The Cancer Genome Atlas and Molecular and Cellular Oncology), the predictive performance of DPSeq was evaluated in a series of experiments. DPSeq demonstrated exceptional performance for predicting key biomarkers in colorectal cancer (microsatellite instability status, hypermutation, CpG island methylator phenotype status, BRAF mutation, TP53 mutation, and chromosomal instability), outperforming most published state-of-the-art classifiers in a within-cohort internal validation and a cross-cohort external validation. In addition, under the same experimental conditions using the same set of training and testing data sets, DPSeq surpassed four CNNs (ResNet18, ResNet50, MobileNetV2, and EfficientNet) and two transformer (Vision Transformer and Swin Transformer) models, achieving the highest area under the receiver operating characteristic curve and area under the precision-recall curve values in predicting microsatellite instability status, BRAF mutation, and CpG island methylator phenotype status. Furthermore, DPSeq required less time for both training and prediction because of its simple architecture. Therefore, DPSeq appears to be the preferred choice over transformer and CNN models for predicting cancer biomarkers. The advent of transformers1Vaswani A. Shazeer N. Parmar N. Uszkoreit J. Jones L. Gomez A.N. Kaiser Ł. Polosukhin I. Attention is all you need.Adv Neural Inf Process Syst. 2017; 30Google Scholar has ushered in a new era for natural language processing. Drawing inspiration from this success, Dosovitskiy et al2Dosovitskiy A. Beyer L. Kolesnikov A. Weissenborn D. Zhai X. Unterthiner T. Dehghani M. Minderer M. Heigold G. Gelly S. An image is worth 16x16 words: transformers for image recognition at scale.arXiv. 2020; ([Preprint])https://doi.org/10.48550/arXiv.2010.11929Crossref Google Scholar proposed the Vision Transformer (ViT) for image analysis. Since then, numerous variants of the ViT have been introduced and have achieved state-of-the-art results, surpassing convolutional neural networks (CNNs).3Zhai X. Kolesnikov A. Houlsby N. Beyer L. Scaling vision transformers.in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 12104-12113Crossref Scopus (123) Google Scholar, 4Mehta S. Rastegari M. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer.arXiv. 2021; ([Preprint])https://doi.org/10.48550/arXiv.2110.02178Crossref Google Scholar, 5Liu Z. Lin Y. Cao Y. Hu H. Wei Y. Zhang Z. Lin S. Guo B. Swin transformer: hierarchical vision transformer using shifted windows.in: Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 10012-10022Crossref Scopus (5851) Google Scholar In the context of digital pathology, Laleh et al6Laleh N.G. Muti H.S. Loeffler C.M.L. Echle A. Saldanha O.L. Mahmood F. Lu M.Y. Trautwein C. Langer R. Dislich B. Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology.Med Image Anal. 2022; 79102474Google Scholar demonstrated that a ViT model outperforms certain CNN architectures in various tasks. Swin Transformer (Swin-T) transformers deliver superior predictive performance in predicting microsatellite instability (MSI) and other biomarkers in colorectal cancer and are more resilient than CNN models when dealing with limited training data.7Guo B. Li X. Yang M. Jonnagaddala J. Zhang H. Xu X.S. Predicting microsatellite instability and key biomarkers in colorectal cancer from H&E-stained images: achieving state-of-the-art predictive performance with fewer data using Swin transformer.J Pathol Clin Res. 2023; 9: 223-235Crossref PubMed Scopus (1) Google Scholar Although transformers have shown impressive results in computer vision tasks, they require significant computational resources to train on large images.8He K. Gan C. Li Z. Rekik I. Yin Z. Ji W. Gao Y. Wang Q. Zhang J. Shen D. Transformers in medical image analysis: a review.Intelligent Med. 2022; 3: 59-78Crossref Scopus (37) Google Scholar Hence, there is a pressing need to devise digital pathology models that can match or surpass the performance of transformers while being less computationally demanding and resource intensive. The success of transformers is thought to stem from the self-attention mechanism's ability to capture long-range dependencies. Although long short-term memory networks can also model long-term dependencies and prevent gradient vanishing through successive time steps, their use in digital pathology is relatively limited and mainly employed for segmentation tasks.9BenTaieb A. Hamarneh G. Deep learning models for digital pathology.arXiv. 2019; ([Preprint])https://doi.org/10.48550/arXiv.1910.12329Crossref Google Scholar Recently, Tatsunami and Taki10Tatsunami Y. Taki M. Sequencer: deep LSTM for image classification.arXiv. 2022; ([Preprint])https://doi.org/10.48550/arXiv.2205.01972Crossref Scopus (0) Google Scholar proposed a sequencer that uses a two-dimensional Bi-directional Long Short-Term Memory (BiLSTM2D) network in ImageNet classification tasks for regular images. The objective of this research was to develop an efficient digital pathology classifier (DPSeq) by fine-tuning the BiLSTM2D network from sequencer to achieve state-of-the-art predictive performance for critical molecular pathways and gene mutations [ie, MSI, hypermutation, chromosomal instability (CING), BRAF mutation, and CpG island methylator phenotype (CIMP)] in colorectal cancer (CRC) using hematoxylin and eosin (H&E)–stained whole-slide images (WSIs). This study provides valuable insights into the comparative performance of DPSeq versus popular transformer (ViT2Dosovitskiy A. Beyer L. Kolesnikov A. Weissenborn D. Zhai X. Unterthiner T. Dehghani M. Minderer M. Heigold G. Gelly S. An image is worth 16x16 words: transformers for image recognition at scale.arXiv. 2020; ([Preprint])https://doi.org/10.48550/arXiv.2010.11929Crossref Google Scholar and Swin-T3Zhai X. Kolesnikov A. Houlsby N. Beyer L. Scaling vision transformers.in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 12104-12113Crossref Scopus (123) Google Scholar) and CNN models (ResNet18,11He K. Zhang X. Ren S. Sun J. Deep residual learning for image recognition.in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778Crossref Google Scholar ResNet50,11He K. Zhang X. Ren S. Sun J. Deep residual learning for image recognition.in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 770-778Crossref Google Scholar MobileNetV2,12Sandler M. Howard A. Zhu M. Zhmoginov A. Chen L.-C. Mobilenetv2: inverted residuals and linear bottlenecks.in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4510-4520Crossref Scopus (12727) Google Scholar and EfficientNet13Tan M. Le Q. Efficientnet: rethinking model scaling for convolutional neural networks.in: International Conference on Machine Learning. PMLR, 2019: 6105-6114Google Scholar) in predicting pathology tasks. The DPSeq was developed to predict cancer biomarkers using WSIs. As illustrated in Figure 1, the classifier development process involved four steps: i) image preprocessing and tile selection, ii) fine-tuning a pretrained sequencer model (trained on the ImageNet data set) using pathology images to build the DPSeq, iii) using the DPSeq to classify tile-level biomarkers, and iv) predicting patient-level biomarkers by aggregating the tile-level predictions. DPSeq performance was assessed by using it to predict MSI, BRAF mutation, CIMP, and other key biomarkers for CRC. Histopathologic WSIs stained with H&E were acquired from two data sets related to CRC. The first data set, named MCO-CRC,14Ward R.L. Hawkins N.J. Molecular and Cellular Oncology (MCO) Study Tumour Collection. UNSW Australia, Kensington, NSW, Australia2015Google Scholar,15Jonnagaddala J. Croucher J.L. Jue T.R. Meagher N.S. Caruso L. Ward R. Hawkins N.J. Integration and analysis of heterogeneous colorectal cancer data for translational research.Stud Health Technol Inform. 2016; 225: 387-391PubMed Google Scholar was obtained from the Molecular and Cellular Oncology (MCO) and consists of patients who underwent curative resection for CRC between 1994 and 2010 in New South Wales, Australia. This data set is accessible through the Secure Research Environment for Digital Health (SREDH) Consortium (https://www.sredhconsortium.org, last accessed April 26, 2023), and MSI, BRAF mutation, and CIMP ground truth labels were available in this data set. The second data set, TCGA-CRC-DX (https://portal.gdc.cancer.gov, last accessed September 2, 2023), contains formalin-fixed, paraffin-embedded WSIs from two studies of colon adenocarcinoma (COAD) and rectum adenocarcinoma (READ) conducted by The Cancer Genome Atlas (TCGA), namely TCGA-COAD and TCGA-READ. In addition to the MSI, CIMP, and BRAF mutation that were present in the MCO-CRC data set, the TCGA-CRC-DX data set also had hypermutation, CING, and TP53 mutation information available. The code for this study is available (https://github.com/mimic0127/DPSeq, last accessed May 21, 2023). Numbers of whole-slide images of patients with CRC in these two data sets are listed in Supplemental Table S1. Access to the TCGA and MCO data sets was provided through data usage and ethics agreements. All patients in the TCGA and MCO data sets provided written informed consent. To develop the tissue classifier and refine the DPSeq, two public data sets that had been annotated by pathologists (these two data sets can be downloaded from https://zenodo.org/record/1214456#.ZC6beOxBxRE, last accessed September 2, 2023) were retrieved. These data sets are referred to as NCT-CRC-HE-100K and CRC-VAL-HE-7K, and they comprise 100,000 and 7180 H&E-stained tiles, respectively. Each tile measures 224 × 224 pixels at 0.5 μm per pixel and has been color normalized via Macenko method.16Macenko M. Niethammer M. Marron J.S. Borland D. Woosley J.T. Guan X. Schmitt C. Thomas N.E. A method for normalizing histology slides for quantitative analysis.in: IEEE International Symposium on Biomedical Imaging: From Nano to Macro. 2009: 1107-1110Crossref Scopus (804) Google Scholar The tiles were annotated with nine different tissue types, namely adipose, background, debris, lymphocytes, mucus, smooth muscle, normal colon mucosa, cancer-associated stroma, and colorectal adenocarcinoma epithelium. The Laleh et al6Laleh N.G. Muti H.S. Loeffler C.M.L. Echle A. Saldanha O.L. Mahmood F. Lu M.Y. Trautwein C. Langer R. Dislich B. Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology.Med Image Anal. 2022; 79102474Google Scholar approach was adopted for processing whole-slide images. All slides from the MCO-CRC and TCGA-CRC-DX data sets were segmented into nonoverlapping tiles of 512 × 512 pixels at 0.5 μm per pixel. The tiles were then color normalized using Macenko method16Macenko M. Niethammer M. Marron J.S. Borland D. Woosley J.T. Guan X. Schmitt C. Thomas N.E. A method for normalizing histology slides for quantitative analysis.in: IEEE International Symposium on Biomedical Imaging: From Nano to Macro. 2009: 1107-1110Crossref Scopus (804) Google Scholar and resized to 224 × 224 pixels to match the input requirement of the deep learning models. A tissue classifier, which was trained and tested on the NCT-CRC-HE-100K and CRC-VAL-HE-7K data sets, respectively,7Guo B. Li X. Yang M. Jonnagaddala J. Zhang H. Xu X.S. Predicting microsatellite instability and key biomarkers in colorectal cancer from H&E-stained images: achieving state-of-the-art predictive performance with fewer data using Swin transformer.J Pathol Clin Res. 2023; 9: 223-235Crossref PubMed Scopus (1) Google Scholar was employed to identify tumor tiles. Up to 500 randomly chosen tumor tiles from each patient were used as inputs for the subsequent DPSeq. DPSeq's underlying structure was developed using the sequencer framework.10Tatsunami Y. Taki M. Sequencer: deep LSTM for image classification.arXiv. 2022; ([Preprint])https://doi.org/10.48550/arXiv.2205.01972Crossref Scopus (0) Google Scholar The primary element of this network is the BiLSTM2D layer, which can integrate both horizontal and vertical patch data within a tile. Defining the input of BiLSTM2D as I∈RH×W×C, the input was split into vertical patch sequences {I:,w,:∈RH×C}w=1W and horizontal patch sequences {Ih,:,:∈RW×C}h=1H. For a fixed w0∈[1,W]∩Z, a vertical patch sequence I:,w0,: was entered into a bidirectional long short-term memory (BiLSTM) (BiLSTMver) to extract the vertical hidden feature: H:,w0,:vertical=BiLSTMver(I:,w0,:). A series of vertical hidden features: {H:,w,:vertical∈RH×2D}w=1W were obtained where D is the dimension of hidden layer of BiLSTMver. The weights of BiLSTMver for all vertical patch sequences were shared. Similarly, a series of horizontal hidden features {Hh,:,:horizon∈RW×2D}h=1H were obtained by Hh,:,:horizon=BiLSTMhor(I:,h,:), for ∀h∈[1,H]∩Z. All vertical hidden features were concentrated to matrix Hallvertical∈RW×H×2D and all the horizontal hidden features to matrix Hallhorizon∈RW×H×2D. The hidden feature for I was obtained by concatenating Hallvertical and Hallhorizon to Hall∈RW×H×4D. The output of the BiLSTM2D was obtained by a channel fusion with point-wise full connection: Iˆ=PointWiseFC(Hall)=WallHall+ball∈RW×H×C. As shown in tile-label predictor part of Figure 1, the input tiles had dimensions of 224 × 224 pixels, which were segmented into a total of 32 × 32 smaller patches with a size of 7 × 7 pixels. These patches were processed through a sequence of four stages of sequencer blocks, with each stage containing a different number of blocks (four, three, eight, and four blocks, respectively). The sequencer blocks replaced multihead attention in transformer blocks with BiLSTM2D to improve the efficiency of the classifier. Following layer normalization and average pooling at the end of the sequencer blocks, a three-layer multilayer perceptron with rectified linear unit (ReLU) and dropout layers (384-256-32) was added to extract molecular information from histopathologic images. Finally, a classification layer was added to the top of the network for accurate classification. To predict biomarkers from histopathologic images, the DPSeq classifier was fine-tuned for multiclass tissue classification. DPSeq parameters were used before the average pooling layer with the parameters of a sequencer model pretrained on ImageNet. A fixed learning rate of 0.0001, Adam optimizer, and cross-entropy loss were used to train DPSeq on NCT-CRC-HE-100K. To learn histologic information without overfitting to the tissue classification problem, the model was fine-tuned for two epochs. After fine-tuning the DPSeq, the last classifier layer was replaced with new linear layers to enable binary classification of CRC biomarkers. The biomarker classifiers were trained for up to 50 epochs with early stopping and a patience of 8, using a cosine annealing learning rate initially set to 0.0001. Weighted cross-entropy loss was employed to handle class-imbalanced data. Finally, patient-level biomarker score was obtained by averaging the tile-level scores of all tiles in the corresponding whole-slide images. DPSeq's core elements consist of sequencer blocks, which replace the multihead attention in the transformer blocks of ViT. These sequencer blocks incorporate a BiLSTM2D, along with a LayerNorm layer, a BiLSTM2D module, another LayerNorm layer, and a channel-level multilayer perceptron layer, following a specific order. The BiLSTM2D module is composed of two BiLSTM modules: BiLSTMver and BiLSTMhor. Assuming the input of BiLSTM2D as I∈RH×W×C, where H, W, and C represent the height, width, and channel dimensions, respectively. BiLSTMver is computed W times, whereas BiLSTMhor is computed H times. The input dimension of BiLSTMver is H×C, and the hidden feature dimension D of BiLSTMver is αC, where α is a constant. As a result, the computational complexity of BiLSTMver is O(HC2), and that of BiLSTMhor is O(WC2). The combined computational complexity of these two BiLSTM modules is O(HC2)×O(W)+O(WC2)×O(H)=O(HWC2). The computational complexity of channel fusion at the last layer of BiLSTM2D and the channel multilayer perceptron at the end of the sequencer block is also O(HWC2). Consequently, the overall computational complexity of BiLSTM2D is O(HWC2). The transformer blocks used in ViT and Swin-T are multihead self-attention module and Windows multihead self-attention module, respectively. The computational complexity for multihead self-attention and Windows multihead self-attention is O(2HWC2+(HW)2C) and O(2HWC2+M2HWC), respectively, where M is the size of the small patches and exhibits resemblances to DPSeq's sequencer block. More important, as the values of H and W increase, the computational complexity of multihead self-attention grows more rapidly compared with that of BiLSTM2D in DPSeq. For CNN models, the computational complexity of the ResNet block in ResNet18 and ResNet50 is O(HWCDk2), where the kernel size is k × k, the input channel size is C, and the output channel size is D. However, MobileNetV2 and EfficientNet showcase superior computational efficiency compared with ResNet. MobileNetV2 achieves this by introducing inverted residuals in its structure, whereas EfficientNet employs the deep scalable composite scaling method. Generally, the CNN models are less computationally complex compared with DPSeq and transformer models. Regarding memory cost, for a single standard BiLSTM, there are cell states of size LC/2, where C is the dimension of input features, and L is the length of tokens fed into the BiLSTM. Therefore, a BiLSTM2D with one vertical and one horizontal BiLSTM used cell states of dimensions (WC+HC)/2, where W and H represent the width and height of the input image, respectively. In comparison, the multihead attention in ViT involves an attention map with dimensions h∗(HW)2, where h represents the number of attention heads. For higher resolution images, the BiLSTM2D module saved more memory compared with ViT's multihead attention. Experiments to compare model complexity and time efficiency were also performed. To assess the performance of the DPSeq classifier, three experiments were devised: i) comparison with published models using internal cross-validation based on TCGA-CRC-DX data set; ii) comparison with published models via external, cross-cohort validation using the same testing dataset (TCGA-CRC-DX); and iii) comparison of DPSeq with other backbone networks using the same training and testing data sets. Deep-learning models have been developed using internal validation to predict key CRC biomarkers (including MSI, CIMP, hypermutation, CING, BRAF mutation, and TP53 mutation) based on the whole-slide images from TCGA-CRC-DX.7Guo B. Li X. Yang M. Jonnagaddala J. Zhang H. Xu X.S. Predicting microsatellite instability and key biomarkers in colorectal cancer from H&E-stained images: achieving state-of-the-art predictive performance with fewer data using Swin transformer.J Pathol Clin Res. 2023; 9: 223-235Crossref PubMed Scopus (1) Google Scholar,17Bilal M. Raza S.E.A. Azam A. Graham S. Ilyas M. Cree I.A. Snead D. Minhas F. Rajpoot N.M. Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study.Lancet Digit Health. 2021; 3: e763-e772Abstract Full Text Full Text PDF PubMed Google Scholar,18Kather J.N. Heij L.R. Grabsch H.I. Loeffler C. Echle A. Muti H.S. Krause J. Niehues J.M. Sommer K.A. Bankhead P. Pan-cancer image-based detection of clinically actionable genetic alterations.Nat Cancer. 2020; 1: 789-799Crossref PubMed Scopus (250) Google Scholar The study adopted the same fourfold cross-validation used in the literature and compared the DPSeq's prediction performance with the published state-of-the-art results. The data set was split into four folds in the same way as Kather et al18Kather J.N. Heij L.R. Grabsch H.I. Loeffler C. Echle A. Muti H.S. Krause J. Niehues J.M. Sommer K.A. Bankhead P. Pan-cancer image-based detection of clinically actionable genetic alterations.Nat Cancer. 2020; 1: 789-799Crossref PubMed Scopus (250) Google Scholar and/or Bilal et al.17Bilal M. Raza S.E.A. Azam A. Graham S. Ilyas M. Cree I.A. Snead D. Minhas F. Rajpoot N.M. Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study.Lancet Digit Health. 2021; 3: e763-e772Abstract Full Text Full Text PDF PubMed Google Scholar For each training iteration of the fourfold cross-validation, three folds for training and validation data sets were randomly separated with a ratio of 0.85:0.15, and the remaining fold was used for testing. To assess the generalization and robustness of DPSeq, an external validation using cross-cohort analysis was conducted. DPSeq was trained using the MCO-CRC data set (n = 1138, 1026, and 364) to predict MSI, BRAF mutation, and CIMP, respectively, and then tested the model on the TCGA-CRC-DX data set (n = 425, 500, and 235) that was unseen during the training. The testing subset of TCGA-CRC-DX has been commonly used as external testing data set in literature6Laleh N.G. Muti H.S. Loeffler C.M.L. Echle A. Saldanha O.L. Mahmood F. Lu M.Y. Trautwein C. Langer R. Dislich B. Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology.Med Image Anal. 2022; 79102474Google Scholar,7Guo B. Li X. Yang M. Jonnagaddala J. Zhang H. Xu X.S. Predicting microsatellite instability and key biomarkers in colorectal cancer from H&E-stained images: achieving state-of-the-art predictive performance with fewer data using Swin transformer.J Pathol Clin Res. 2023; 9: 223-235Crossref PubMed Scopus (1) Google Scholar,19Echle A. Laleh N.G. Quirke P. Grabsch H. Muti H. Saldanha O. Brockmoeller S. van den Brandt P. Hutchins G. Richman S. Artificial intelligence for detection of microsatellite instability in colorectal cancer—a multicentric analysis of a pre-screening tool for clinical application.ESMO Open. 2022; 7100400Abstract Full Text Full Text PDF PubMed Scopus (36) Google Scholar,20Echle A. Grabsch H.I. Quirke P. van den Brandt P.A. West N.P. Hutchins G.G. Heij L.R. Tan X. Richman S.D. Krause J. Clinical-grade detection of microsatellite instability in colorectal tumors by deep learning.Gastroenterology. 2020; 159: 1406-1416.e11Abstract Full Text Full Text PDF PubMed Scopus (160) Google Scholar and can provide relatively fair comparison of performance among different published models through the external cross-cohort validation. DPSeq was compared with seven published models, including Swin-T [trained on MCO-CRC (n = 1065 for MSI status, and n = 1026 for BRAF mutation)], EfficientNet [trained on the Darmkrebs: Chancen der Verhütung durch Screening dataset (DACHS) (n = 2069 for MSI status and BRAF mutation)], ViT [trained on DACHS (n = 2069 for MSI status and BRAF mutation)], ResNet18 [trained on pooled international data sets (n = 7917 for MSI status)], and ShuffleNet [trained on the Quick and Simple and Reliable trial (QUASAR) (n = 1016 for MSI status), DACHS (n = 2013 for MSI status), and the Netherlands Cohort Study (NLCS) (n = 2197 for MSI status)]. To avoid potential bias in comparing models and networks due to different training sets used in published articles,6Laleh N.G. Muti H.S. Loeffler C.M.L. Echle A. Saldanha O.L. Mahmood F. Lu M.Y. Trautwein C. Langer R. Dislich B. Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology.Med Image Anal. 2022; 79102474Google Scholar,7Guo B. Li X. Yang M. Jonnagaddala J. Zhang H. Xu X.S. Predicting microsatellite instability and key biomarkers in colorectal cancer from H&E-stained images: achieving state-of-the-art predictive performance with fewer data using Swin transformer.J Pathol Clin Res. 2023; 9: 223-235Crossref PubMed Scopus (1) Google Scholar,19Echle A. Laleh N.G. Quirke P. Grabsch H. Muti H. Saldanha O. Brockmoeller S. van den Brandt P. Hutchins G. Richman S. Artificial intelligence for detection of microsatellite instability in colorectal cancer—a multicentric analysis of a pre-screening tool for clinical application.ESMO Open. 2022; 7100400Abstract Full Text Full Text PDF PubMed Scopus (36) Google Scholar,20Echle A. Grabsch H.I. Quirke P. van den Brandt P.A. West N.P. Hutchins G.G. Heij L.R. Tan X. Richman S.D. Krause J. Clinical-grade detection of microsatellite instability in colorectal tumors by deep learning.Gastroenterology. 2020; 159: 1406-1416.e11Abstract Full Text Full Text PDF PubMed Scopus (160) Google Scholar an external cross-cohort validation was conducted using the same training and testing data sets. Specifically, four popular CNN models (ResNet18, ResNet50, MobileNetV2, and EfficientNet) and two transformers (ViT and Swin-T) were trained for prediction of MSI, CIMP, and BRAF mutation using the MCO-CRC data set and evaluated their external predictive performance using the same TCGA-CRC-DX data set. In addition to comparing the model predictive performance, the model efficiency was analyzed by measuring training time per epoch and prediction time for MSI status across all patients in TCGA-CRC-DX. To assess the predictive performance of the models, the study computed the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). For the fourfold cross-validation experiment, the study obtained the average AUROC and AUPRC values across the four test folds and calculated their SD. For the external validation experiment, the study computed the AUROC and AUPRC values for the TCGA-CRC-DX data set and estimated their 95% CIs using the bootstrap method (1000 iterations). DPSeq was used to predict six clinically relevant biomarkers for CRC (namely, MSI, CIMP, hypermutation, BRAF mutation, TP53 mutation, and CING) through fourfold cross-validation on TCGA-CRC-DX data set. To facilitate the comparison of the performance of DPSeq with previous published models, the study adhered to the same TCGA-CRC-DX split as the earlier publications.7Guo B. Li X. Yang M. Jonnagaddala J. Zhang H. Xu X.S. Predicting microsatellite instability and key biomarkers in colorectal cancer from H&E-stained images: achieving state-of-the-art predictive performance with fewer data using Swin transformer.J Pathol Clin Res. 2023; 9: 223-235Crossref PubMed Scopus (1) Google Scholar,17Bilal M. Raza S.E.A. Azam A. Graham S. Ilyas M. Cree I.A. Snead D. Minhas F. Rajpoot N.M. Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study.Lancet Digit Health. 2021; 3: e763-e772Abstract Full Text Full Text PDF PubMed Google Scholar,18Kather J.N. Heij L.R. Grabsch H.I. Loeffler C. Echle A. Muti H.S. Krause J. Niehues J.M. Sommer K.A. Bankhead P. Pan-cancer image-based detection of clinically actionable genetic alterations.Nat Cancer. 2020; 1: 789-799Crossref PubMed Scopus (250) Google Scholar Table 1 presents the mean AUROC/AUPRC values from the fourfold cross-validation along with their corresponding SDs.Table 1Predict