# Ulcerative Severity Estimation Based on Advanced CNN-Transformer Hybrid Models
## 1 Introduction
* To enhance model interpretability and validate practical usability, we visualize the decision-making process using class activation maps (CAMs), offering intuitive insights into how predictions are made. Related Work  
**CNNs in the Field of Medical Imaging**: Convolutional neural networks have brought about transformative advancements in image analysis by emulating the visual processing mechanisms of the brain [7]. Researchers have extensively applied CNN-based models to extract distinctive and meaningful features from medical images, and they often surpass traditional machine learning methods and manually intensive volumetric analysis techniques [8]. For instance, [PERSON] et al. proposed a CNN-based regular pattern discovery model that effectively extracts disease-related knowledge from unstructured medical records [9]. [PERSON] et al. developed the multi-instance multiscale CNN (MIMS-CNN) model, which integrates MSConv layers and top-k pooling, and demonstrates superior performance across three medical-related tasks [10]. Moreover, in the area of endoscopic analysis,numerous CNN-based models have been developed for wireless capsule endoscopy (WCE) image classification [11]. For example, [PERSON] et al. introduced the BIR (Bleedy Image Recognizer) model, which combines MobileNet with a custom CNN to classify WCE images showing bleeding [12]. [PERSON] et al. employed a novel fusion approach combining deep CNN features (VGG16 and VGG19) with geometric descriptors using a Euclidean Fisher Vector [13]. While the mentioned demonstrate effectiveness in specific tasks, they exhibit notable limitations. For instance, MIMS-CNN relies on fixed convolutional kernel designs and specific pooling strategies, which restrict its adaptability to variable and multiscale lesion features in medical images. The CNN-based approach of [PERSON] et al. for text extraction lacks deep integration with standard medical terminology, knowledge graphs, and structured medical knowledge, thereby limiting clinical interpretability and generalizability. The BIR model, despite combining MobileNet with a custom CNN, heavily relies on local convolution operations, resulting in reduced robustness to changes in lesion locations within images. Additionally, [PERSON] et al. 's introduction of geometric descriptors and deep feature fusion does not include visualization or decision-making analysis illustrating how the model integrates such information for diagnostic purposes. **Transformers in the Field of Medical Imaging**: Recently, with transformer models extending to computer vision fields [14], transformer-based architectures have demonstrated significant potential across various medical imaging tasks [15]. For robust medical image classification, the self-ensembling vision transformer (SEViT) introduces a self-distillation framework that enhances the resilience of early layer feature representations in vision transformers against adversarial perturbations [16]. In the context of 3D medical image segmentation, the UNEt transformer (UNETR) architecture leverages a transformer-based encoder to effectively model global multiscale contextual information while maintaining the well-established U-shaped design to integrate both high-level semantic features and low-level spatial details [17]. Furthermore, in the domain of gastroscopy and colonoscopy, a vision transformer model incorporating hybrid shifted windows was proposed to capture both short-range and long-range dependencies for digestive tract image classification [18]. However, despite the strong performance of vision transformer-based models, several challenges remain. For example, although SEViT employs a self-distillation mechanism to address the issue of limited training data, it is still prone to overfitting, particularly in cases with extremely small sample sizes or imbalanced class distributions.