# Improving ViT Performance for Colon Cancer Histopathological Image Classification Using Transfer Learning

Moncef Belaskri, Mohammed Lamine Benomar, Mourtada Benazzouz, Xavier Descombes, Damien Ambrosetti

► To cite this version:

Moncef Belaskri, Mohammed Lamine Benomar, Mourtada Benazzouz, Xavier Descombes, Damien Ambrosetti. Improving ViT Performance for Colon Cancer Histopathological Image Classification Using Transfer Learning. IC2SDA 2025 - International Conference on Intelligent Computer Systems, Data Science and Applications, Nov 2025, Blida, Algeria. pp.1-8, 10.1109/IC2SDA68097.2025.11331535. hal-05483581

HAL Id: hal-05483581

https://inria.hal.science/hal-05483581v1

Submitted on 29 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Improving ViT Performance for Colon Cancer Histopathological Image Classification Using Transfer Learning

Moncef Belaskri

University of Tlemcen

Biomedical Engineering Laboratory

Tlemcen, Algeria

moncef.belaskri@univ-tlemcen.dz

Mohammed Lamine Benomar

University of Ain Temouchent

Biomedical Engineering Laboratory, University of Tlemcen

Ain Temouchent, Algeria

amine.benomar@univ-temouchent.edu.dz

Mourtada Benazzouz

University of Tlemcen

Biomedical Engineering Laboratory

Tlemcen, Algeria

mourtada.benazzouz@univ-tlemcen.dz

Xavier Descombes

University of Nice Sophia Antipolis

I3S Laboratory, CNRS/INRIA

Sophia Antipolis, France

xavier.descombes@inria.fr

Damien Ambrosetti

CHU Nice - Hôpital Pasteur

Central Anatomo-Pathology Department

Nice, France

ambrosetti.d@chu-nice.fr

**Abstract**—Deep learning has revolutionized medical image analysis, particularly the automatic classification of histopathology images. Vision Transformers (ViTs) now surpass convolutional neural networks (CNNs) in modeling long-range dependencies and capturing global features. However, when initialized with random weights, ViTs often struggle to adapt to the varying scales and morphologies of tumor tissues, resulting in suboptimal performance. Transfer learning addresses this challenge by pretraining the model on large datasets to obtain a stronger weight initialization before fine-tuning on the target domain. This strategy has proven especially effective for histology image classification. In this study, we fine-tuned a ViT-B/16 model pretrained on ImageNet-21k by first training it on 75,000 Renal Cell Carcinoma (RCC) patches and then applying it to classify colon tumor subtypes in the Chaoyang dataset. Our proposed method achieves an accuracy of 87.14%, a precision of 83.59%, a recall of 82.08% and an F1 score of 82.77%, outperforming both a model trained from scratch and one pretrained only on ImageNet. These results underscore the value of domain-specific pretraining for enhancing ViT performance on diverse tumor morphologies in histopathology classification tasks.

**Index Terms**—Deep learning, Histopathology image classification, Vision Transformer, Transfer learning, Colon tumor subtype classification.

## I. INTRODUCTION

Cancer remains a significant global health challenge. In 2020, nearly 10 million deaths were caused by cancer, and approximately one in five people will be affected by the disease during their lifetime [1]. Endoscopic detection and removal of precancerous lesions remains the most effective preventive strategy [2]. Achieving early and accurate diagnosis is critical, as it substantially improves patient prognosis and survival outcomes [3]. Conventional diagnosis relies on pathologists manually scrutinizing biopsy samples, a process that is inherently labor-intensive, time-consuming, and subject

to inter-observer variability and diagnostic error [4]. Traditional computational methods for automated tumor detection in histopathological images have shown limited performance due to their reliance on handcrafted features [5], [6]. These features often fail to capture the intricate and heterogeneous patterns present in histopathological tissue, resulting in poor generalization and reduced diagnostic accuracy [7].

The rise of deep learning has transformed medical image analysis by automating feature extraction and enabling hierarchical pattern learning. Convolutional Neural Networks (CNNs) have been widely adopted for this purpose because they can effectively extract both low-level and high-level features, identifying critical structures [8]. While CNNs have enhanced diagnostic accuracy by precisely localizing malignant regions and reducing pathologist workloads, and reduce pathologist workloads, they face challenges in capturing long-range dependencies, a limitation that becomes apparent when analyzing whole-slide images (WSIs) where tumors may span disjoint regions [9].

To overcome the limitations of traditional approaches, Vision Transformers (ViTs) have emerged as a promising alternative in medical image analysis [10], [11]. ViTs utilize self-attention mechanisms to model long-range dependencies across image regions by dividing the image into fixed-size patches and analyzing the relationships among them. This allows ViTs to effectively capture global contextual information, which has led to state-of-the-art performance in various image classification tasks, including in the medical domain [9]. However, ViTs typically require large-scale training datasets to learn meaningful representations due to their lack of inherent inductive biases, such as locality and translation equivariance [12]. In data-constrained settings, such as those common in medical imaging, ViTs may struggle to learn fine-grained local
features critical for accurate tumor classification. Although pretraining on massive datasets can mitigate this challenge, access to such datasets is often infeasible in medical applications [13].

On the other hand, to address many of the limitations in this task, transfer learning has frequently been used to improve medical image model performance [14]. By starting from a model pretrained on a large dataset such as ImageNet, the network benefits from existing feature representations and avoids beginning with random weights [15], [16]. Alternatively, one can fine-tune a model on a dataset that shares the same domain or points of interest as the target data, which further accelerates learning and boosts accuracy [15], [16], [17].

The main idea behind our work is to use transfer learning to outperform the ViT-B/16 model on multiclass colon tumor subtype classification. This research shows that fine-tuning an ImageNet-pretrained model on 75,000 Renal Cell Carcinoma (RCC) histopathology patches significantly improves performance. Our comparative analysis demonstrates that fine-tuning ImageNet-pretrained weights on histology images yields superior results compared to training from random initialization.

## II. RELATED WORK

Colon cancer represents a significant global health challenge, ranking as the third most frequently diagnosed cancer worldwide and the second leading cause of cancer-related mortality as of 2018 [18], [19], [20]. Medical image classification, particularly for tumor detection in histopathological images, remains a fundamental challenge in computer vision [21]. In recent years, significant research efforts have focused on the classification of colon cancer histopathology images, applying a wide range of techniques from conventional machine learning methods to advanced deep learning models to improve classification accuracy [22], [23].

Recent advances in colorectal tissue classification have leveraged both deep-learning and hybrid approaches to push accuracy well above 95%. Kumar et al. [24] combined a dilated ResNet with an attention module and a deep SVM ensemble to reach up to 99.8% accuracy. Kim et al. [25] applied a multi-class CNN to whole-slide images, achieving 95.5% on internal tests and 98.5% on external adenocarcinoma slides. Prezja et al. [26] paired EfficientNetV2 feature extraction with a random-forest head to attain 96.7% accuracy. Demidov et al.'s KD-CTCNet [27] distilled texture features into a CNN backbone to boost performance in low-data regimes. Transformer-based pipelines have recently achieved clinical-grade sensitivity (97%) and negative predictive value (99%) for MSI detection, demonstrating the power of attention mechanisms in both classification and biomarker prediction [28].

Chen et al. [29] proposed the IL-MCAM framework for colorectal cancer detection. It combines feature extraction using three transformer networks and a convolutional neural network with an interactive loop. This approach demonstrated effectiveness on the HENCT-CRC-100K dataset. Zeid et al. [30] evaluated a Vision Transformer (ViT) and a Compact

Convolutional Transformer (CCT) on a publicly available colorectal histopathology dataset comprising 5,000 images across eight tissue categories. The CCT achieved higher accuracy (95%) compared to the ViT (93%), demonstrating the efficacy of transformer-based models in this domain. More recently, Prabhu [31] developed a ViT-based pipeline for colon cancer classification that not only distinguishes malignant from benign regions but also employs Grad-CAM heatmaps to pinpoint the areas most indicative of disease.

Classifiers for colorectal cancer, though often achieving high accuracy through intricate hybrid or transformer-based models, face challenges in training, reproducibility, and deployment due to their complex, highly tailored architectures [24], [29]. While transformers excel at feature extraction via self-attention, their success typically depends on specific architectural decisions and substantial labeled data [30].

Inspired by the recent successes of attention-based methods in histopathological image analysis, particularly for colon tumors, this work addresses the task of colon tumor classification. We investigate the effectiveness of transfer learning using the standard ViT-B/16 architecture, providing a robust and streamlined baseline for benchmarking and future development.

## III. MATERIALS AND METHODS

### A. Chaoyang Dataset

The histopathology image dataset used in this study was derived from colorectal tissue samples collected at Chaoyang Hospital in Beijing, China [39]. It comprises 6160 high-resolution (512x512) image patches categorized into four diagnostic classes: normal tissue (1111 training, 705 testing), serrated lesions (842 training, 321 testing), adenocarcinoma (1404 training, 840 testing), and adenoma (664 training, 273 testing), as shown in Table 1.

<table><caption>TABLE 1
CHAOYANG DATASET COMPOSITION SUMMARY.</caption><thead><tr><td>Dataset split</td><td>Normal</td><td>Serrated</td><td>Adenocarcinoma</td><td>Adenoma</td></tr></thead><tbody><tr><td>Train</td><td>1111</td><td>842</td><td>1404</td><td>664</td></tr><tr><td>Test</td><td>705</td><td>321</td><td>840</td><td>273</td></tr></tbody></table>

To reflect real-world diagnostic variability, training annotations were independently provided by three pathologists, with no enforced consensus. A single label was randomly selected per image, introducing inherent label noise (only 60% of training images had unanimous agreement). This mimics clinical practice, where inter-observer variability is common. In contrast, the test set underwent rigorous quality control, requiring full agreement among all annotators to ensure reliable evaluation. Designed for computational pathology research, this dataset highlights challenges related to label noise and diagnostic discrepancies, promoting robust model development under real-world conditions.

### B. Data preprocessing and splitting

The original images had a resolution of 512 x 512 pixels. To align with the model's input requirements, all images were
resized to 224x224 pixels and converted to RGB format to match the expected input channels, as shown in Fig. 1.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Examples of Chaoyang dataset images.</figcaption></figure>

The dataset was initially organized into pre-split training and testing directories. However, to evaluate the model's performance dynamically during training, we further partitioned the training data. After shuffling the samples to mitigate ordering biases as illustrated in Fig. 2, 10% of the training data was randomly allocated as a validation set. This approach enabled real-time performance monitoring and helped prevent overfitting.

<figure><img src="image_3.png" /><figcaption>Fig. 2. Heatmap Class Distribution.</figcaption></figure>

### C. Data Augmentation

Training Vision Transformers (ViTs) enhanced with attention fusion mechanisms on a limited dataset of 6,160 images carries a significant risk of overfitting. To mitigate this risk and enhance generalization, a robust, dynamic data augmentation strategy is employed during training. This strategy prevents the model from memorizing specific image patterns by continuously presenting unique variations of the training data. Our augmentation pipeline incorporates diverse transformations to maximize variability, as summarized in Table 2.

<table><caption>TABLE 2
DATA AUGMENTATION PARAMETERS.</caption><thead><tr><td>Parameter</td><td>Value</td></tr></thead><tbody><tr><td>Rotation range</td><td>40</td></tr><tr><td>Width shift range</td><td>0.2</td></tr><tr><td>Height shift range</td><td>0.2</td></tr><tr><td>Shear range</td><td>0.2</td></tr><tr><td>Zoom range</td><td>[0.5, 1.0]</td></tr><tr><td>Horizontal flip</td><td>True</td></tr><tr><td>Vertical flip</td><td>True</td></tr><tr><td>Fill mode</td><td>Reflect</td></tr></tbody></table>

To further improve generalization, the augmentation process is randomized for each epoch by varying the seed parameter. This ensures that the model encounters different transformations of the training images in every iteration, leading to increased robustness and better adaptation to unseen data. Examples of Images resulting from the application of data augmentation to our dataset are shown in Fig. 3.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Examples of image transformation through data augmentation.</figcaption></figure>

### D. Renal Cell Carcinoma Dataset

The histopathology dataset comprises 75,000 renal tissue images collected at the University Hospital Center of Nice (France), including both tumor and non-tumor samples. Fig. 4 presents representative sample images from this dataset.

<figure><img src="image_5.png" /><figcaption>Fig. 4. Examples of the RCC dataset images.</figcaption></figure>

Tissue samples were obtained from 91 consecutive patients and classified into five classes: non-tumor tissue (21,288 patches), clear cell RCC (23,885 patches), papillary RCC (14,563 patches), chromophobe RCC (7,390 patches), and oncocytoma (7,831 patches), as shown in Table 3.

<table><caption>TABLE 3
PATCH COUNT PER CLASS IN THE RCC DATASET.</caption><thead><tr><td>Class</td><td>Patch Count</td></tr></thead><tbody><tr><td>Non-Tumor</td><td>21,288</td></tr><tr><td>ccRCC</td><td>23,885</td></tr><tr><td>pRCC</td><td>14,563</td></tr><tr><td>chrRCC</td><td>7,390</td></tr><tr><td>ONCO</td><td>7,831</td></tr><tr><td>Total</td><td>74,957</td></tr></tbody></table>

### E. Vision Transformer (ViT)

The Vision Transformer (ViT), originally proposed by Dosovitskiy et al. [37], adapts the Transformer encoder used in natural language processing for image classification with only minimal changes. Unlike traditional Transformer architectures that include both an encoder and a decoder [37], ViT relies solely on the encoder to process visual data. First, each input image is divided into a grid of fixed-size patches. Each patch is flattened into a one-dimensional vector and then projected into
a higher-dimensional embedding space. To preserve spatial information, a positional embedding is added to each patch embedding. A learnable “class” token is also prepended to the sequence of embedded patches, this token aggregates the information needed for classification. After assembling these embeddings, the encoder layers, which consist of stacked self-attention and feedforward sublayers, transform the entire sequence. The final representation associated with the class token is passed through a multilayer perceptron (MLP) head made up of two fully connected layers with GELU activation [38], producing the probability distribution over the target classes. ViT performs best when trained on very large datasets with ample computational resources. By applying self-attention across all patch embeddings, it effectively captures long-range interactions within an image, often surpassing CNNs under these conditions. Additionally, since ViT shares the same underlying architecture as Transformer models used in language tasks, it offers a streamlined path for integrating vision and language modalities within unified frameworks. Fig. 5 shows the full ViT architecture.

<figure><img src="image_6.png" /><figcaption>Fig. 5. ViT model architecture.</figcaption></figure>

## F. Proposed Approach

In this section, we provide a detailed explanation of our proposed transfer-learning method, based on a ViT-B/16 model pretrained on ImageNet21k, using our specific classification implementation and highlighting the main steps of our approach. As a first step, we loaded the weights of a ViT-B/16 network trained on ImageNet-21k to avoid random weight initialization. We then fine-tuned this network on 75,000 historical patches from our renal cell carcinoma (RCC) dataset, updating all model layers to better capture domain-specific texture and color patterns. Next, we adapt the fine-tuned ViT for our target classification task using a multilayer perceptron (MLP) block, summarized in Table 4.

This head consists of fully connected layers, followed by batch normalization and GeLU activation. To combat overfitting on our comparatively small target dataset, we inserted

<table><caption>TABLE 4
SUMMARY OF THE MLP BLOCK.</caption><thead><tr><td>Layer (Type)</td><td>Output Shape</td></tr></thead><tbody><tr><td>GaussianNoise (stddev=0.5)</td><td>(None, 1000)</td></tr><tr><td>Dense Layer (128 units)</td><td>(None, 128)</td></tr><tr><td>Batch Normalization</td><td>(None, 128)</td></tr><tr><td>GELU Activation Function</td><td>(None, 128)</td></tr><tr><td>Dropout (rate=0.5)</td><td>(None, 128)</td></tr><tr><td>Dense Layer (Softmax, 4 classes)</td><td>(None, 4)</td></tr></tbody></table>

a GaussianNoise layer immediately after the ViT output, perturbing feature activations during training, and added a dropout layer just before the final softmax classifier. This combination of noise injection and random deactivation encourages the network to learn more resilient feature representations and yields smoother decision boundaries. The main outline of this method is illustrated in Fig. 6.

<figure><img src="image_7.png" /><figcaption>Fig. 6. Overview of the proposed method.</figcaption></figure>

Overall, our approach leverages the strong representative capabilities of ViT for comprehensive context modeling, enhanced through sequential transfer learning, first using ImageNet-21k pretrained weights, then fine-tuned on RCC patches, while strategically applying noise injection and dropout to mitigate overfitting.

## G. Implementation details

In all experiments presented in this paper, we used rigorous configurations to ensure both reliability and validity. To evaluate the training process and guarantee optimal model performance, we fine-tuned several hyperparameters. Table 5 summarizes the key values used throughout our experiments.

We employed the AdamW optimizer for weight optimization, incorporating a weight-decay regularization term to penalize large weights and promote generalization. This regularization strategy helped reduce overfitting and improve model robustness. To further enhance training efficiency, we integrated a cosine-annealing learning-rate schedule with a warmup phase alongside the AdamW optimizer. This combination was chosen to ensure stable convergence and avoid
<table><caption>TABLE 5
NETWORK HYPERPARAMETER TUNING.</caption><thead><tr><td>Hyperparameter</td><td>Value</td></tr></thead><tbody><tr><td>Training epochs</td><td>100</td></tr><tr><td>Optimizer</td><td>AdamW</td></tr><tr><td>Activation function</td><td>ReLU, SoftMax</td></tr><tr><td>Initial learning rate</td><td>1e-4</td></tr><tr><td>Batch size</td><td>128</td></tr><tr><td>Weight decay</td><td>1e-2</td></tr></tbody></table>

abrupt fluctuations in the learning rate during training. The learning-rate schedule dynamically adjusted the learning rate over training epochs, starting with a gradual warm-up phase followed by a smooth cosine decay. This approach prevented overshooting and facilitated more effective convergence toward an optimal solution. The learning-rate progression is illustrated in Fig. 7, showing its smooth transition across epochs. Finally, all experiments were conducted on a single NVIDIA A40 48 GB GPU.

<figure><img src="image_8.png" /><figcaption>Fig. 7. Learning rate progression using cosine annealing with warmup.</figcaption></figure>

## IV. RESULTS AND DISCUSSION

In this section we evaluate and compare our method's performance with a ViT-B/16 trained from scratch and a ViT-B/16 pretrained on ImageNet-21k. The comparison is based on accuracy, precision, recall, and F1-score to provide a comprehensive assessment of each model's classification performance, highlighting their ability to correctly identify and distinguish between different classes. To further analyze the strengths and limitations of each approach, we also include confusion matrices. These offer a detailed view of how each model performs on individual classes by showing the distribution of true and predicted labels. This helps identify specific patterns of misclassification and provides deeper insight into the effectiveness of each training strategy.

In Table 6, we report the performance of the ViT-B/16 model without any pretraining, using average class weighting. As shown in Fig. 8, the confusion matrix details the model's prediction outcomes for each class. Table 7 presents the evaluation results of the ViT-B/16 model pretrained on ImageNet-21k, where the pretrained weights were used to train

the model on the target dataset. Fig. 9 shows the confusion matrix, illustrating the model's prediction outcomes for each class.

<table><caption>TABLE 6
PERFORMANCE OF ViT-B/16 WITHOUT PRETRAINING.</caption><thead><tr><td>Model</td><td>Acc (%)</td><td>Rec (%)</td><td>Prec (%)</td><td>F1 (%)</td></tr></thead><tbody><tr><td>ViT-B/16</td><td>76.57</td><td>69.77</td><td>69.89</td><td>69.81</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Fig. 8. Confusion Matrix for ViT-B/16 Trained from Scratch.</figcaption></figure>

<table><caption>TABLE 7
PERFORMANCE OF ViT-B/16 PRETRAINED ON IMAGENET-21K.</caption><thead><tr><td>Model</td><td>Acc (%)</td><td>Rec (%)</td><td>Prec (%)</td><td>F1 (%)</td></tr></thead><tbody><tr><td>ViT-B/16</td><td>85.97</td><td>81.57</td><td>82.39</td><td>81.89</td></tr></tbody></table>

<figure><img src="image_10.png" /><figcaption>Fig. 9. Confusion Matrix for ViT-B/16 Pretrained on ImageNet-21k.</figcaption></figure>

Pretraining a ViT on a large-scale dataset, such as ImageNet-21k, which comprises roughly 14 million images
across approximately 21,000 distinct object categories, substantially enhances its performance compared to a standard ViT with random initialization. As shown in Table 6 and Table 7, the pretrained model consistently achieves higher overall accuracy and precision. This gain is also reflected in the confusion matrices of Fig. 8 and Fig. 9, where the pretrained ViT exhibits far fewer misclassifications and a much clearer separation between classes.

Finally, Table 8 presents the performance of the ViT-B/16 model, which was initially pretrained on ImageNet-21k and later fine-tuned on 75,000 RCC patches using the pretrained weights to better adapt the model to the target dataset. Fig. 10 displays the corresponding confusion matrix, illustrating the model's classification results for each class.

<table><caption>TABLE 8
PERFORMANCE OF VIT-B/16 PRETRAINED ON IMAGENET-21K AND FINE-TUNED ON RCC PATCHES.</caption><thead><tr><td>Model</td><td>Acc (%)</td><td>Rec (%)</td><td>Prec (%)</td><td>F1 (%)</td></tr></thead><tbody><tr><td>ViT-B/16</td><td>87.14</td><td>82.08</td><td>83.59</td><td>82.77</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Fig. 10. Confusion Matrix of ViT-B/16 Pretrained on ImageNet-21k and Fine-Tuned on RCC Patches.</figcaption></figure>

As shown in Table 8, our method significantly enhances all evaluation results, outperforming both previous approaches. Fine-tuning the pre-trained ImageNet-21k ViT-B/16 model on a labeled histology dataset containing 75,000 patches before training on the target dataset helps the model adapt more effectively to the target data. Based on these results, the model's weights were closer to the ideal configuration compared to both previous models. Furthermore, as clearly illustrated in Fig. 10, the model predicts classes more accurately, demonstrating clearer separation between the classification boundaries. Fig. 11 shows the ROC curves of our proposed method for each class in the Chaoyang dataset along with the macro-average curve. The overall macro-average AUC of 0.94 confirms strong performance, demonstrating excellent discriminative power.

<figure><img src="image_12.png" /><figcaption>Fig. 11. ROC curves for each class and the macro-average curve.</figcaption></figure>

Table 9 provides a comparative analysis between our proposed method and state-of-the-art (SOTA) models from prior studies on the Chaoyang dataset. Among the baseline models, SqueezeNet [32] achieves an accuracy of 78.07%, while ResNet-34 [32] improves this to 83.40%. Modest performance gains are observed with MSCViT [35] (84.11%) and MobileNet [32] (84.39%). ViT-based models show notably stronger results. Locality-Guided ViT [33] achieves 85.65%, and Training-Free ViT [34] slightly surpasses it with 85.80%. Eff-CTM [36] sets a new high with 86.30%. However, our proposed method achieves the highest accuracy of 87.14%, clearly outperforming all compared models.

<table><caption>TABLE 9
ACCURACY COMPARISON WITH SOTA METHODS FOR THE CHAOYANG DATASET.</caption><thead><tr><td>Method</td><td>Accuracy (%)</td></tr></thead><tbody><tr><td>SqueezeNet [32]</td><td>78.07</td></tr><tr><td>ResNet-34 [32]</td><td>83.40</td></tr><tr><td>MSCViT [35]</td><td>84.11</td></tr><tr><td>MobileNet [32]</td><td>84.39</td></tr><tr><td>ViT with locality guidance [33]</td><td>85.65</td></tr><tr><td>Training-free ViT [34]</td><td>85.80</td></tr><tr><td>Eff-CTM [36]</td><td>86.30</td></tr><tr><td>Ours</td><td><b>87.14</b></td></tr></tbody></table>

To better visualize the impact of our methods on the Chaoyang dataset, we generated Grad-CAM heatmaps from the feature maps of the final Vision Transformer encoder block for both a ViT-B/16 model trained from scratch and our fine-tuned model, using randomly selected test images from each class, as shown in Fig. 12. These refined heatmaps highlight our model's increased confidence and precision and demonstrate its superior ability to distinguish tumor from non-tumor cells compared to the scratch-trained ViT-B/16 model. The fine-tuned model exhibits markedly more localized attention on diagnostically relevant patches, with substantially
reduced background noise and sharper delineation of tumor regions.

<figure><img src="image_13.png" /><figcaption>Fig. 12. Comparison of Attention Regions Visualized by the Proposed Method and Scratch-Trained ViT-B/16 Using Grad-CAM Heatmaps.</figcaption></figure>

The main purpose of this study was to classify colon cancer tumors in the Chaoyang dataset using transfer learning. This study used a model pretrained on ImageNet and fine-tuned it on renal cell carcinoma (RCC) histopathology patches from a related domain to demonstrate the benefit of pretraining on data with characteristics similar to the target. Our results indicate that using ImageNet weights as a starting point, followed by domain-specific pretraining on RCC images, significantly enhances the ViT-B16 model's ability to detect tumors across different image types. To address the limited size of the Chaoyang dataset (6,160 images), extensive data augmentation was applied, increasing the variety of training samples and improving model robustness. During training, we employed multiple strategies to mitigate overfitting across both histopathology datasets. This approach underscores the importance of selecting a pretraining dataset that closely matches the target domain to improve classification performance.

## V. CONCLUSION

This study demonstrates that fine-tuning a ViT-B/16 model, initially pretrained on ImageNet, with a domain-specific histopathology dataset significantly enhances colon cancer

classification performance. Our transfer-learning approach, fine-tuned on 75,000 labeled histological images, achieved an accuracy of 87.14%, a precision of 83.59%, a recall of 82.08% and an F1 score of 82.77%. These results outperform both models trained from scratch and those pretrained solely on ImageNet. This indicates that initializing the network with weights already attuned to histopathological patterns positions it closer to an optimal solution, leading to faster convergence and improved final accuracy. To mitigate overfitting on our smaller target dataset of 6,160 images, we employed targeted regularization and data augmentation strategies. Collectively, our findings underscore the critical importance of domain-specific pretraining for optimizing Vision Transformers in digital pathology. This work provides a foundation for future research on other cancer subtypes and complex diagnostic challenges within the field.

## VI. ACKNOWLEDGEMENTS

The authors gratefully acknowledge the University Hospital Center of Nice, France, for providing the histopathological image patches used in this study.

## REFERENCES

[1] H. Sung, J. Ferlay, R. L. Siegel, M. Laversanne, I. Soerjomataram, A. Jemal, and F. Bray, "Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries," *CA Cancer J. Clin.*, vol. 71, no. 3, pp. 209–249, 2021.

[2] G. Tziatzios, D. I. Ziogas, P. Gkolfakis, V. Papadopoulos, A. Pa-paefthymiou, N. Mathou, A. Giannakopoulos, G. Gerasimatos, K. D. Paraskeva, and K. Triantafyllou, "Endoscopic grading and sampling of gastric precancerous lesions: A comprehensive literature review," *Curr. Oncol.*, vol. 31, no. 7, pp. 3923–3938, Jul. 2024.

[3] N. Gonçalves, J. Chaves, I. Marques-Sá, M. Dinis-Ribeiro, and D. Libânio, "Early diagnosis of gastric cancer: Endoscopy and artificial intelligence," *Best Pract. Res. Clin. Gastroenterol.*, vol. 75, Art. no. 101979, 2025.

[4] A. H. Fischer, K. A. Jacobson, J. Rose, and R. Zeller, "Hematoxylin and eosin staining of tissue and cell sections," *CSH Protoc.*, p. pdb.prot4986, 2008.

[5] D. Komura and S. Ishikawa, "Machine learning methods for histopathological image analysis," *Comput. Struct. Biotechnol. J.*, vol. 16, pp. 34–42, 2018.

[6] A. Madabhushi and G. Lee, "Image analysis and machine learning in digital pathology: Challenges and opportunities," *Med. Image Anal.*, vol. 33, pp. 170–175, 2016.

[7] M. N. Gurcan, L. E. Boucheron, A. Can, A. Madabhushi, N. M. Rajpoot, and B. Yener, "Histopathological image analysis: A review," *IEEE Rev. Biomed. Eng.*, vol. 2, pp. 147–171, 2009.

[8] A. Krizhevsky, I. Sutskever, and G. Hinton, "ImageNet classification with deep convolutional neural networks," in *Adv. Neural Inf. Process. Syst.*, vol. 25, pp. 1097–1105, 2012.

[9] M. Raghu, T. Unterthiner, S. Kornblith, et al., "Do Vision Transformers see like convolutional neural networks?" in *Adv. Neural Inf. Process. Syst.*, 2021.

[10] C. C. Atabansi, J. Nie, H. Liu, et al., "A survey of transformer applications for histopathological image analysis: New developments and future directions," *Biomed. Eng. Online*, vol. 22, no. 1, p. 96, Sep. 2023. [Online]. Available: https://doi.org/10.1186/s12938-023-01106-4

[11] G. L. Baroni, L. Rasotto, K. Roitero, A. H. Siraj, and V. Della Mea, "Vision transformers for histopathology image analysis survey," in *Image Anal. Process. – ICIAP 2023 Workshops, Part II*, Udine, Italy, Sept. 11–15, 2023, pp. 15–26, Springer, 2024.

[12] C. C. Atabansi, J. Nie, H. Liu, et al., "A survey of transformer applications for histopathological image analysis: New developments and future directions," *Biomed. Eng. Online*, vol. 22, p. 96, 2023.
[13] C. Sun, A. Shrivastava, S. Singh, and A. Gupta, "Revisiting unreasonable effectiveness of data in deep learning era," in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), pp. 843–852, 2017.

[14] H. E. Kim, A. Cosa-Linan, N. Santhanam, et al., "Transfer learning for medical image classification: A literature review," BMC Med. Imaging, vol. 22, art. no. 69, 2022.

[15] N. Tajbakhsh, J. Y. Shin, S. R. Gurudu, R. T. Hurst, C. B. Kendall, M. B. Gotway, and J. Liang, "Convolutional neural networks for medical image analysis: Fine-tuning and transfer learning," IEEE Trans. Med. Imag., vol. 35, no. 5, pp. 1299–1312, May 2016.

[16] H. C. Shin, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura, and R. M. Summers, "Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning," IEEE Trans. Med. Imag., vol. 35, no. 5, pp. 1285–1298, May 2016.

[17] M. Raghu, C. Zhang, J. Kleinberg, and S. Bengio, "Transfusion: Understanding transfer learning for medical imaging," arXiv preprint arXiv:1811.10456, 2019.

[18] F. Bray, J. Ferlay, I. Soerjomataram, R. L. Siegel, L. A. Torre, and A. Jemal, "Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries," CA Cancer J. Clin., vol. 68, no. 6, pp. 394–424, 2018.

[19] M. C. Wong, J. Huang, V. Lok, J. Wang, F. Fung, H. Ding, and Z. J. Zheng, "Differences in incidence and mortality trends of colorectal cancer worldwide based on sex, age, and anatomic location," Clin. Gastroenterol. Hepatol., vol. 19, no. 5, pp. 955–966, 2021.

[20] M. Arnold, M. S. Sierra, M. Laversanne, I. Soerjomataram, A. Jemal, and F. Bray, "Global patterns and trends in colorectal cancer incidence and mortality," Gut, vol. 66, no. 4, pp. 683–691, 2017.

[21] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. W. M. van der Laak, B. van Ginneken, and C. I. Sánchez, "A survey on deep learning in medical image analysis," Med. Image Anal., vol. 42, pp. 60–88, 2017.

[22] L. D. Tamang and B. W. Kim, "Deep learning approaches to colorectal cancer diagnosis: A review," Appl. Sci., vol. 11, no. 22, 2021.

[23] A. Davri, E. Birbas, T. Kanavos, G. Ntritos, N. Giannakeas, A. T. Tzallas, and A. Batistatou, "Deep learning on histopathological images for colorectal cancer diagnosis: A systematic review," Diagnostics, vol. 12, no. 4, 2022.

[24] A. Kumar, S. Patel, R. Sharma, and P. Verma, "Dilated ResNet with Attention and Deep SVM Ensemble for Colorectal Tissue Classification," J. Med. Imaging, vol. 10, no. 2, pp. 123–134, 2023.

[25] B. Kim, H. Lee, J. Park, and M. Choi, "Multi-class CNN for Whole-Slide Image Analysis in Colorectal Adenocarcinoma," IEEE Trans. Biomed. Eng., vol. 70, no. 5, pp. 567–578, 2023.

[26] C. Prezja, L. Nguyen, S. Rossi, and D. Müller, "EfficientNetV2 Feature Extraction with Random Forest Head for Histopathology Classification," in Proc. Med. Image Comput. Comput. Assist. Interv. (MICCAI), 2023, pp. 301–310.

[27] D. Demidov, E. Ivanova, and F. Zhang, "KD-CTCNet: Knowledge-Distilled Texture-Enhanced CNN for Low-Data Regimes," Med. Image Anal., vol. 77, Art. no. 102412, 2024.

[28] E. Lee, J. Thompson, K. Wang, and L. García, "Transformer-Based Pipeline for Microsatellite Instability Detection in Colorectal Cancer," Clin. Cancer Res., vol. 31, no. 1, pp. 45–56, 2025.

[29] H. Chen, C. Li, X. Li, M. M. Rahaman, W. Hu, Y. Li, W. Liu, C. Sun, H. Sun, X. Huang, and M. Grzegorzek, "IL-MCAM: An interactive learning and multi-channel attention mechanism-based weakly supervised colorectal histopathology image classification approach," Computers in Biology and Medicine, vol. 143, p. 105265, 2022. doi:10.1016/j.compbiomed.2022.105265

[30] M. A. E. Zeid, K. El-Bahnasy, and S. Abo-Youssef, "Multiclass colorectal cancer histology images classification using vision transformers," in Proc. 10th Int. Conf. Intelligent Computing and Information Systems (ICICIS), pp. 224–230, IEEE, 2021.

[31] J. Prabhu, "Colon cancer classification from histopathology images via Vision Transformers with Grad-CAM visualization," in Proc. Int. Conf. on Medical Imaging and Diagnostic AI, pp. 1–8, 2023.

[32] V. Kadian, A. Singh, and K. Sharma, "A robust colon cancer detection model using deep learning," in Proc. 2023 Third Int. Conf. on Secure Cyber Computing and Communication (ICSCCC), pp. 665–670, 2023.

[33] K. Li, R. Yu, Z. Wang, L. Yuan, G. Song, and J. Chen, "Locality guidance for improving vision transformers on tiny datasets," 2022. [Online]. Available: https://arxiv.org/abs/2205.12957

[34] Z. Wei, H. Pan, L. Li, P. Dong, Z. Tian, X. Niu, and D. Li, "TVT: Training-free vision transformer search on tiny datasets," 2023. [Online]. Available: https://arxiv.org/abs/2304.08415

[35] B. Zhang and Y. Zhang, "MSCViT: A Small-size ViT Architecture with Multi-Scale Self-Attention Mechanism for Tiny Datasets," Neural Networks, vol. 188, Art. no. 107499, Apr. 2025, doi: 10.1016/j.neunet.2025.107499.

[36] S. Liu, L. Wang, and W. Yue, "An efficient medical image classification network based on multi-branch CNN, token grouping Transformer and mixer MLP," Applied Soft Computing, vol. 153, art. no. 111323, Mar. 2024, doi: 10.1016/j.asoc.2024.111323.

[37] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x16 words: Transformers for image recognition at scale," in Proc. Int. Conf. Learn. Represent. (ICLR), 2021. [Online]. Available: https://arxiv.org/abs/2010.11929.

[38] D. Hendrycks and K. Gimpel, "Gaussian Error Linear Units (GELUs)," arXiv preprint arXiv:1606.08415, 2016. [Online]. Available: https://arxiv.org/abs/1606.08415.

[39] C. Zhu, W. Chen, T. Peng, Y. Wang, and M. Jin, "Hard Sample Aware Noise Robust Learning for Histopathology Image Classification," IEEE Transactions on Medical Imaging, vol. 41, no. 4, pp. 881–894, Apr. 2022, doi: 10.1109/TMI.2021.3125459.