# Efficient Knowledge Distillation for Brain Tumor Segmentation
## 1 Introduction  
In the field of medical image processing and analysis [1], medical image segmentation is a critical and complex step. Deep learning is used to extract the important regions of medical images to assist doctors in clinical treatment and pathology research automatically. The task of segmenting brain tumors, as a type of medical image segmentation, is also a focus of researchers' attentions. Tumors originating from the neuroepithelium are collectively called gliomas and are the most common intracranial primary tumors. Typically, gliomas are classified into four grades [2], where the higher the grade, the worse the prognosis is. Much research on brain tumor segmentation has been proposed; however, due to the uncertainty of a brain tumor's shape, location and size, it is impossible to accurately locate it during segmentation, making brain tumor segmentation a severe challenge [3, 4, 5]. Meanwhile, a large majority of the approaches improve the original model accuracy by modifying the model architecture, inevitably adding various expensive computational components and expanding the required storage space. For a long time, people have discussed how to solve this dilemma and find a better balance between efficiency and accuracy. [PERSON] [6] introduced knowledge distillation (KD) to the deep-learning domain, which has gained much attention for its simplicity and efficiency. The core idea of KD is to use soft labels to learn class distribution. Specifically, first train a teacher network to obtain its output, and then, in the process of student network training, let the output of the student network learn the output of the teacher network. The final effect of the knowledge distillation method is that the output of the student network is similar to that of the teacher network. Many methods have been proposed to improve the learning accuracy of students' networks following KD. For example, when using KD, [PERSON] et al. [7] found that compared to the ground truth class, the secondary class could effectively learn inter-class similarity to obtain knowledge better. [PERSON] et al. [8] focused the problem on a data issue; they first trained the teacher network to generate pseudodatasets, and then they combined it with the original datasets to form noise datasets to learn knowledge. [PERSON] et al. [9] stated that KD performed poorly when the difference between the student and the teacher capacity waslarge, and they proposed a new ensemble distribution KD for training. [PERSON] et al. [10] proposed learning knowledge through intermediate feature layers. [PERSON] et al. [11] proposed a new transformer distillation, which performed the transformer distillation in the pretraining and specific learning phases. [PERSON] et al. [12] just simply applied KD to Bert, making it 60% faster and guaranteeing 97% accuracy. [PERSON] et al. [13] distilled the final transformer layer from the teachers and introduced teaching assistants to aid distillation. Unfortunately, the above approach did not take into account the validity of distillation in brain tumor segmentation scenarios. Only a few researchers have applied knowledge distillation methods to brain tumor segmentation; however, most of them focused on solving the problems of small datasets and multimodality [14; 15; 16], and they did not focus on the efficiency of brain tumor segmentation. In this paper, considering brain tumor segmentation as a structured dense prediction task, we proposed a new knowledge distillation method. Different from channel attention, which just considers the importance of different channels, the proposed method of coordinate distillation (CD) considers spatial information. It encodes channels along two spatial directions, which is particularly important for spatial structure capture in segmentation tasks and helps the model to segment brain tumors accurately. Our extensive experiments on the public dataset BraTS2018 demonstrated the remarkable performance of the method. In general, our contributions are summarized as follows.