# Ulcerative Severity Estimation Based on Advanced CNN-Transformer Hybrid Models
## 2 Materials and Methods
### CoAtNet Models  
CoAtNet is a CNN-transformer hybrid model introduced by the Google team. It combines the strong generalization ability of convolutional networks--particularly effective under limited data--with the scalability and representation capacity of transformer models when ample data are available. It also exhibits faster convergence and enhanced efficiency.  
The model adopts a multi-stage architecture that mimics traditional ConvNets by gradually performing pooling operations across stages. Specifically, the network is composed of five stages (S0 to S4), where each stage reduces the spatial resolution by a factor of two while increasing the number of channels. In Stages S1 and S2, the network stacks MBConv blocks to capture spatial interactions, leveraging prior layers, given that convolution is more effective at extracting local features, particularly in early layers. In the later stages, transformer blocks are introduced to expand the global receptive field and model long-range dependencies and the model employs a pre-normalized relative attention variant in the transformer blocks, which serves as a core component of the architecture. Both the Feed-Forward Network (FFN) in the transformer and the MBConv block utilize the inverted bottleneck design to balance model capacity and generalization [6]. The overall structural pattern is C-C-T-T, where C denotes convolutional modules, and T denotes transformer modules. The main flowchart and structure are shown in Figure 1.  
It is worth noting that a key innovation of CoAtNet lies in its combination of a global static convolution kernel with an adaptive attention matrix, as defined in Equations (1) and (2):  
\[Y_{i}=\sum_{j\in\mathbb{G}}\frac{exp(x_{i}^{\text{T}}x_{j}+w_{i-j})}{A_{ij}}x_ {j}, \tag{1}\]  
\[A_{ij}=\sum_{k\in\mathbb{G}}exp(x_{i}^{\text{T}}x_{k}+w_{i-k}), \tag{2}\]  
where \(x_{i},Y_{i}\in\mathbb{R}^{D}\) denote the input and output at position \(i\), respectively. For example, \(x_{i}\) and \(x_{j}\) represent two patches from the same image. Each patch is compared with all other patches in the image using their dot product, i.e., \(x_{i}^{\text{T}}x_{j}\). \(G\) refers to the global spatial space, and \(A_{ij}\) represents a modified attention weight incorporating an additional local component. For any position pair \((i,j)\), the corresponding convolutional weight \(w_{i-j}\) depends solely on their relative displacement.  
The above formulas are based on the standard attention weight computation  
Figure 1: The main flowchart and structure of our experiment and model.  
\[Attention(x_{i},x_{j})=\sum_{j\in G}\frac{exp(x_{i}^{\rm T}x_{j})}{\sum_{k\in G}exp(x_ {i}^{\rm T}x_{k})}, \tag{3}\]  
which relies solely on the similarity between input features. An additional term, \(w_{i-j}\), is introduced to represent a convolution-style spatial bias. This term depends only on the relative positional difference \((i-j)\) and not on the input features themselves. This means that for each patch, while computing its similarity with other patches, the model also takes into account their relative spatial distance. This approach effectively integrates the translational invariance and local structural modeling capabilities of CNNs with the global contextual modeling strength of transformers.