# Combining the Transformer and Convolution for Effective Brain Tumor Classification Using MRI Images
## 1 Introduction
Furthermore, in the fully connected layer, each neuron of the previous layer is differently connected to the next layer neuron similar to mesh topology in the network [22]. Finally, the output layer is responsible for class prediction by assigning labels to each class. Figure 1 shows the overall structure of the CNN. Some CNN architecture for the categorization of tumor and normal MRI images was implemented in [23; 24]. Furthermore, CNNs were employed to partition the damaged tissues of the brain from the non-tumor regions in [25], in addition to being used in tumor categorization. To better categorize BT, a capsule network architecture was developed in [26]. BT classification is an example of many medical image analysis tasks where pre-trained CNN networks have been used to improve classification accuracy in recent studies [27]. To assist radiologists in visualizing [6; 23; 25] and classifying tumor types [7], multi-grade BT categorization systems, or CAD, have been developed. Deep neural networks are used to classify various grades using high-level data taken from MRI scans, which may aid radiologists in making choices regarding early diagnosis and potential treatment modalities. This classification, offered by many researchers, provides information regarding the percentage of malignant tumors. The main issues with current methods for classifying BT are binary categorization, a paucity of data samples for various grades, and poor accuracy. With only two categories (benign and malignant) for BT classification, the methods used now make it exceedingly difficult for radiologists to proceed with additional testing or treatment. The lack of publicly available data is another main obstacle in the classification of BT. Additionally, the current methods have not yet demonstrated satisfactory accuracy. For these reasons, we suggest a \"transformer-enhanced convolutional Neural network (TECNN)\" for tumor classification. The Transformer is first proposed by [PERSON] et al. [28]. for image analysis, which utilized a standard Transformer structure directly to the images. In the Transformer structure, an image is divided into various image patches, afterword these patches are subjected to a Transformer encoder as shown in Figure 2. The Transformer treats these patches as tokens similar to Naturel Language Processing (NLP). After the encoding layer, the classification layer is used for accurate categorization. In the proposed TECNN we use two primary parts, (1) features extraction using CNN, and (2) a transformer, as shown in Figure 3. We use features extraction to extract the local features using CNN that focuses on pixel-wise information by using various convolutional and pooling layers. The Transformer focuses on the patch-wise information instead of focusing on pixels. Initially, the convolution operation is applied to a given image before feeding them to our proposed model. The feature extractor and transformer pipeline are each individually fed with the processed image at this point. The Transformer patch embeddings (PE) and sets of CNN's feature maps have different dimensions, and as a result, there is a clear semantic gap between them. We use the feature fusion module (FFM) and intelligent merge module (IMM), as proposed by [PERSON] et al. [29] so that the CNN and transformer techniques could be used to convert between feature maps and patches while maintaining their benefits. Both the feature extraction using CNN and transformer provide inputs to the FFM and IMM. The CNN's feature maps are transformed into transformer space in the FFM and fused with transformer patches. Global pooling (GP) is used to choose an informative channel of the given input feature maps rather than performing a straight conversion. Channels with more class-specific functionality are emphasized in this section. The transition from feature maps to patches is thus considerably more seamless. CNN's local information guidance can be spread to the transformer patches.