Network-Agnostic Knowledge Transfer for Medical Image Segmentation
Authors: Shuhang Wang
Eugene Cheah
Elham Yousef Kalafi
Mercy Asiedu
Alex Benjamin
Vivek Kumar Singh
Ge Zhang
Viksit Kumar
Anthony Edward Samir
Venue:  (2021)
Keywords: Knowledge Transfer
Deep Learning
Medical Image Segmentation
Pseudo Annotation

Abstract: Conventional transfer learning leverages weights of pre-trained networks, but mandates the need for similar neural architectures. Alternatively, knowledge distillation can transfer knowledge between heterogeneous networks but often requires access to the original training data or additional generative networks. Knowledge transfer between networks can be improved by being agnostic to the choice of network architecture and reducing the dependence on original training data. We propose a knowledge transfer approach from a teacher to a student network wherein we train the student on an independent transferal dataset, whose annotations are generated by the teacher. Experiments were conducted on five state-of-the-art networks for semantic segmentation and seven datasets across three imaging modalities. We studied knowledge transfer from a single teacher, combination of knowledge transfer and fine-tuning, and knowledge transfer from multiple teachers. The student model with a single teacher achieved similar performance as the teacher; and  the student model with multiple teachers achieved better performance than the teachers. The salient features of our algorithm include: 1) no need for original training data or generative networks, 2) knowledge transfer between different architectures, 3) ease of implementation for downstream tasks by using the downstream task dataset as the transferal dataset, 4) knowledge transfer of an ensemble of models, trained independently, into one student model. Extensive experiments demonstrate that the proposed algorithm is effective for knowledge transfer and easily tunable. 

Under review as a conference paper at ICLR 2021
NETWORK-AGNOSTE KNOWLEDGE TRANSFER FOR MEDICAL IMAGE SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Conventional transfer learning leverages weights of pre-trained networks, but mandates the need for similar neural architectures. Altematively, knowledge distillation can transfer knowledge between heterogeneous networks but often requires access to the original training data or additional generative networks. Knowledge transfer between networks can be improved by being agnostic to the choice of network architecture and reducing the dependence on original train- ing data. We propose knowledge transfer approach from teacher to stu- dent network wherein we train the student on an independent transferal dataset, whose annotations are generated by the teacher. Experiments were conducted on five state-of-the-art networks for semantic segmentation and seven datasets across three imaging modalities. We studied knowledge transfer from single teacher, combination of knowledge transfer and fine-tuning and knowledge transfer from multiple teachers. The student model with a single teacher achieved similar perfor- mance the teacher, and the student model with multiple teachers achieved better performance than the teachers. The salient features of our algorithm include: 1) no need for original training data generative networks, 2) knowledge trans- fer between different architectures, 3) ease of implementation for downstream tasks by using the downstream task dataset as the transferal dataset, 4) knowledge transfer of an ensemble of models, trained independently. into one student model. Extensive experiments demonstrate that the proposed algorithm is effective for knowledge transfer and easily tunable.
INTRODUCTION
Deep learning often requires sufficiently large training dataset, which is expensive to build and not easy to share between users. For example, big challenge with semantic segmentation of medical images is the limited availability of annotated data (Litjens et al. 2017). Due to ethical concerns and confidentiality constraints, medical datasets are not often released with the trained networks. This highlights the need for knowledge transfer between neural networks, wherein the original training dataset does not need be accessed. On the other hand, according the black-box metaphor in deep learning based methods, transferring knowledge is difficult between heterogeneous neural networks. To address these limitations, algorithms were proposed to reuse or share the knowledge of neural networks, such as network weight transfer (Tan et al. 2018), knowledge distillation (Hinton) al. 2015), federated learning YYang et al. 2019). and self-training (Xie et al.) 2020b).
Some conventional algorithms directly transfer the weights of standard large models that were trained on natural image datasets for different tasks (Kang Gwak 2019 Motamed et al. 2019 Jodeiri et al.] 2019 Raghu et al. 2019). For example, Igllovkko & Shvets (2018) adopted VGGIT pre-trained on ImageNet as the encoder of U-Net for 2D image segmentation. Similarly, the con- volutional 3D (Tan et al.) 2015), pre-trained on natural video datascts, was used the encoder of 3D U-Net for the 3D MR (Magnetic Resonance) medical image segmentation (Zeng et al. 2017). Transferring the network weights generally requires adjustments to be made to the architecture of the receiver model, this in turn, limits the flexibility of the receiver network.
Another technique that involves knowledge transfer is federated learning (Yang et al. 2019 it has received attention for its capability to train a large-scale model in a decentralized manner without re- quiring users' data. In general, federated learning approaches adopt the central model to capture the

---

Under review as a conference paper at ICLR 2021
shared knowledge of all users by aggregating their gradients. Due to the difficulties in transferring knowledge between heterogeneous networks, federated learning often requires all devices, including both the central servers and local users, to the same neural architecture (Xie et al. 2020a). To our best knowledge, there has been no federated learning system that uses heterogeneous networks.
Knowledge distillation is the process of transferring the knowledge of large neural network an ensemble of neural networks (teacher) to a smaller network (student) (Hinton et al., 2015). Given a set of trained teacher models, one feeds training data to them and uses their predictions instead of the true labels to train the student model. For effective transfer of knowledge, however, it is essential that a reasonable fraction of the training examples are observable by the student (Li al.) 2018) or the metadata at each layer is provided (Lopes et al., (017)) Yoo et al. (2019) used generative network to extract the knowledge of a teacher network, which generated labeled artificial images to train another network. As can be seen, Yoo et al.'s method had to train an additional generative network for each teacher network.
Different from knowledge distillation, self-training aims to transfer knowledge to a more capable model. The self-training framework (Scudder 1965) has three main steps: train a teacher model on labeled images; use the teacher to generate pseudo labels on unlabeled images; and train student model on the combination of labeled images and pseudo labeled images. Xie et al. (2020b) proposed self-training with noisy student for classification, which iterates this process few times by treating the student as a teacher to relabel the unlaheled data and training a new student. These studies required labeled images for training the student; moreover, they implicitly required that the pseudo- labeled images should be similar in content to the original labeled images.
Inspired by self-training, we propose a network-agnostic knowledge transfer algorithm for medical image segmentation. This algorithm transfers the knowledge of a teacher model to a student model by training the student on transferal dataset whose annotations are generated by the teacher. The algorithm has the following characteristics: the transferal dataset requires no manual annotation and is independent of the teacher-traiiigg dataset; the student does not need to inherit the weights of the teacher, and such, the knowledge transfer can be conducted between heterogeneous neural architectures; is straightforward to implement the algorithm with fine fine-tuning to solve downstream task, especially by using the downstream task dataset as the transferal dataset; the algorithm is able to transfer the knowledge of an ensemble of models that are trained independently into one model.
We conducted extensive experiments on semantic segmentation using five state-of-the -httt--rttttttt neural networks and seven datasets. The neural networks include DecpLabv3+ (Chen et al., 2018). U- Net (Ronneberger et al.] 2015), AttU-Net (Oktay et al., 2018), SDU-Net (Wang et al.) 2020), and Panoptic-FPN (Kirillov et al., 2019). Out of the seven datasets, four public datasets involve breast lesion, nerve structure, skin lesion, and natural image objects, and three internal/in-hout house datasets involve breast lesion (a single dataset with two splits) and thyroid nodule. Experiments showed that the proposed algorithm performed well for knowledge transfer on semantic image segmentation.
ALGORITHM
The main goal of the proposed algorithm is to transfer the knowledge of one or more neural networks to an independent one, without access to the original training datasets. This section presents the proposed knowledge transfer algorithm for semantic segmentation in Algorithm I and its application with fine-tuning for a downstream task in Algorithm A.4).
The knowledge transfer algorithm first employs one or more teacher models to generate pseudo masks for the transferal dataset and then trains the student model on the pseudo-annotated transferal dataset. For an image x the transferal dataset D, we get the pseudo mask by the weighted average of the teacher models' outputs: = Σr,er W; (x)), where w; is the weight for the model T:. The output of model (xx) is either soft mask (pixel value varies from 0 to 1) or binary mask (pixel value is 0 or 1). Since the ensemble of teacher models is not our primary focus, we simply set the weights equally for all teacher models.
We adopt two constraints to exclude images without salient targets, as shown in Figure | The first constraint is on the target pixel number in the pseudo mask, where target pixel indicates the pixel with a value above 0.5. We exclude the image (of size 384 x384 pixels) if the target pixel number is less than a threshold of 256. The second constraint is on the gray-level entropy of the pseudo
2

---

mask. high entropy value implies the target and the background have no obvious difference and vice versa. We exclude images with gray-level entropy higher than a threshold of 2.5. We employ the second constraint only in the case that the teacher model outputs a soft mask rather than a binary Under review as a conference paper at ICLR 2021
Algorithm Network-Agnostir Knowledge Transfer for Semantic Segmentation
Require: Teacher set I" with trained models, randomly initialized student model S
Require: Transferal dataset D)
Require: Target pixel number threshold a, gray-level entropy threshold
1: for each datapoint r in D do
2:
Obtain pseudo mask y/ - ET,TT w; Tx(x)
3: N + number of target pixels (value above 0.5) in
4:
E f gray-level entropy of
5:
if' N or E > 3 then Exclude x from D 7: else
6:
8:
Update x as (x, y)
9:
end if
10:
end for
Train student model on aseutt-annotated
12: return Trained S
mask.
Figure 1: Pseudo mask target breast lesions in images (converted grayscale) from PASCAL VOC2012. The top row is original images and the second row is pseudo masks. The first column presents an example with few target pixels. The second column is an example with high gray-level entropy of pseudo mask. The third column shows an example with salient target.
In this algorithm, the teacher models and student model can be independent of each other and do not rely on each other during the training or testing phase. The transferal dataset is independent of the teacher-training datasets, which are not observable for the student model. The student model gains the knowledge of the teacher by the transferal dataset. As result, the student model is trained to work well on datasets similar to the teacher-training dataset, while it is not aimed to predict the ground truth of the transferal dataset.
In the case that small dataset D, with ground truth available for a downstream segmentation task (target task), it is straightforward to further fine-tune the student model that has been trained on secudo-annottted transferal dataset, as Algorithmm2ia shows. The transferal dataset is independent of the target task, while can also be non-annotated dataset from the target task.

---

Under review as a conference paper at ICLR 2021
3 RELATED NEURAL NETWORKS
We posit that if the teacher model is ineffective, more capable student model can easily achieve similar performance; if the student model, however, is ineffective, the cause cannot be easily identi- fied and the reason can be attributed to the knowledge transfer algorithm and the inherent limitations of the student model. We experimented with five neural networks, all which have different archi- tectures and have been shown to provide the best outcome various segmentation tasks.
Deep 3ab33+:
abb33+ (Chen et al. 2018) one of Google's s latest and best performing seman- tic segmentation models. It combines the advantages of spatial pyramid pooling with encoder- decoder structure. Spatial pyramid pooling captures rich contextual information while the decoder path is able to gradually recover object boundaries.
U-Net: U-Net (Ronneberger et al. 2015) has been the most widely used network for medical image segmentation. U-Net consists of encoder-decode, structure with the encoder path extracting rich semantic information, and the decoder path recovering resolution and enabling contextual informa- tion.
AttU-Net (Oktay et al. 2018) is a modification of U-Net, where attention gates are used control the skip connections from the encoder the decoder. Attention gates make the model focus on target structures by highlighting important features and suppressing irrelevant features.
AttU-Net:
SDU-Net: SDU-Net (Wang al., 2020) utilizes parallel atrous convolutions with different dilation rates in each layer. which is effective in capturing features in both small and large receptive fields. SDU-Net has demonstrated better performance using merely 40 percent of the parameters U-Net.
Panoptic-FPM Panoptic-FPN (Krilllov et al. 2019) merges semantic segmentation and object de- tection, such that each pixel given a class as semantic segmentation and each object is given unique ID as in object detection. Panoptic-FPP Erootiic-FPN provides a more rich and complete segmentation.
DATASETS
Table I presents an overview of the seven image datasets that were used for the experiments. One example for each dataset is presented in Figure 2
Example for each datasets. From left to right, the top row presents the images from XXX Breast Lesion-1, XXX Breast Lesion-2, Baheya Breast Lesion, and Thyroid Nodule; the second row presents the images from Nerve Structure. Skin Lesion, and PASCAL VOC2012.
Figure 2:
XXX Breast Lesion: Patients evaluated and referred for Ultrasound (US)-guided biopsy (BI-RADS or 5 mass) at XXX Institute between September 2017 and October 2019 were recruited. Eligible subjects were adult patients with breast masses detected by conventional US imaging. A total of 1385 US images were adopted and manually annotated by two radiologists. These images were randomly assigned to XXX Breast Lesion-1 (n=1237) and XXX Breast Lesion-2 (n=148).

---

Under review as a conference paper at ICLR 2021
Table 1: Datasets. XXX is used for Anonym.
Dataset
Image Number
Modality
Data Source
XXX Breast Lesion-1
1,237
Ultrasound
XXX
XXX Breast Lesion-2
148
Ultrasound
XXX
Baheya Breast Lesion
697
Ultrasound
Baheya Hospital
Thyroid Nodule
3,830
Ultrasound
XXX
Nerve Structure
11,143
Ultrasound
Kaggle Challenge
Skin Lesion
25,331
Dermoscopy
ISIC Challenge
PASCAL VOC2012
17,125
Natural
VOC Challenge
Baheya Breast Lesion: The Baheya Breast Lesion dataset was released by Al-Dhabyani et al. (2020) and it contains 780 breast US images among women between 25 and 75 years old. Radi- ologists from Baheya Hospital, Egypt manually annotated the lesion boundaries. The images categorized into three classes, which are normal (n=133), benign (n=487), and malignant (n=210). In our experiment, we only used images with benign malignant lesions, totaling 697 images.
Thyroid Nodule: A total of 3,830 US images of thyroid nodules were retrospectively collected form the XXX Thyroid Nodule Dataset. Data from patients (more than 18 years old) who underwent US examination before their thyroid biopsy between April 2010 and April 2012 were included. Manual delineation of thyroid nodules by two expert radiologists
the reference standard.
Nerve Structure: Hosted on Kaggle, the US Nerve segmentation dataset formed the basis a competition in segmenting nerve structures US images of the neck. The dataset includes potential noise, artifacts, and mistakes when labeling the ground truth as well as near-identical images due to the inherent image acquisition technique. In our experiment, we utilized all 11,143 original images.
Skin Lesion:
The skin-lesion dataset was from a challenge hosted by the International Skin Imaging Collaboration (ISIC) in 2019 where the task was to perform multiclass classification to distinguish dermoscopic images among nine distinct diagnostic categories of skin cancers. We employed the original images of the training set with 25,331 images across classes
Pascal VOC2012 challenge dataset is popular for object detection and seg- mentation tasks. This dataset consists of natural images belonging to 20 different types of objects. We adopted total of 12, original images both for object detection and segmentation.
PASCAL VOC2012:
EXPERIMENT
Experimental evaluation was conducted the space of semantic image segmentation. The exper- iments included: knowledge transfer with single teacher, combination of knowledge transfer and fine-tuning, and knowledge transfer with multiple teachers. As mentioned in Algorithms T and 2 excluded images without salient targets in their pseudo masks. So we also tested and compared the algorithm's 's performance with and without image exclusion.
In addition, evaluated the knowledge transfer capability of each transferal dataset, which is defined as the ratio of the highest Dice score of all student models that are trained on the transferal dataset the Dice score of the teacher model that generates pseudo masks for the transferal dataset. It is obvious the student and the teacher Dice scores should be calculated on the same test dataset.
5.1 KNOWLEDGE TRANSFER WITH A SINGLE TEACHER
We adopted DeepLabv3+ as the teacher model with XXX Breast Lesion-1 as the teacher-training dataset, and employed U-Net, AttU-Net, SDU-Net, and Panoptic-FPN as the student models. We used Baheya Breast Lesion, Thyroid Nodule, Nerve Structure, Skin Lesion, and PASCAL VOC2012 as transferal datasets. The trained teacher processed the transferal datasets to generate pseudo masks, which were used to train the students. We tested the teacher and students XXX Breast Lesion-2 and Baheya Breast Lesion. The performance was evaluated by computing the Dice score between the model output and manual annotations, i.e., ground truth.
5

---

Under review as a conference paper at ICLR 2021
Table 2: Average Dice scores (-sstd, %) of segmentation XXX Breast Lesion-2 by student models
trained on aseudo-annotated datasets. indicates image exclusion was employed. KTC indicates knowledge transfer capability.
U-Net
Aut-NNet
SDU-Net
Panoptic
KTC
Baheya Breast Lesion
72.64::20.93
74.72:1.22.8
85.00=14.44
81.12=22.52
97.66
Baheya Breast Lesion*
71.74#227.65
67.56=132.94
82.33+20.89
79.95=22.62
Thyroid Nodule
66.28:29.223
71.17=25.10
86.12=14.11
82.77=18.70
98.94
Thyroid Nodule*
67.87++30.02
75.60++23.48
84.39+17.01
84.77=16.67
Nerve Structure
45.07++37.40
58.66433.61
78.53+22.224
80.06-23.41
93.19
Nerve Structure*
58.19:132.94
$7.54133.30
78.92422.27
81.11=21.75
Skin Lesion
38.92:122.12
34.93-122116
50.10131.64
55.16=33.85
81.16
Skin Lesion
56.81+28:26
53.33-122.18
69.17=29.91
70.64-28.50
PASCAL VOC2012
61.83+33.61
66.18431.39
82.44+18.35
79.68-20.73
94.72
PASCAL VOC2012*
70.94+27.40
76.22422.31
81.92+19.30
81.28=18.97
The teacher model had an average Dice score of 87.04% (1...) on XXX Breast Lesion-2 and an average Dice score of 65.93((13..55% on Baheya Breast Lesion. As shown in Tables and and all student model obtained impressive knowledge. For XXX Breast Lesion SDU-Net and Panoptic- FPN obtained Dice scores similar and even better than the teacher in most of the cases, even though the transferal datasets have different kinds of objects and even different image modalities, such as PASCAL. VOC2012. For Baheya Breast Lesion, all students trained on pseudo-annotated Baheya Breast Lesion resulted in higher Dice scores than the teacher model even though the size of Baheya Breast Lesion is much smaller than the teacher- teacher-training dataset. The superior performance of the students could be partly due to domain adaptation, which is one of the advantages of our algorithm when the tansferral dataset is from the target task.
The performance of student models improved by excluding images that had few target pixels or high gray-level entropy from the transferal dataset, especially when the transferal dataset modality was different from the teacher-training dataset. It is because the image of a different modality may result in more pseudo masks with few target pixels or high gray-level entropy, which could be seen from the entropy distribution in Figure (.2)). However, image exclusion did not work that well on Baheya Breast Lesion. It is probably because Baheya Breast Lesion is similar to the teacher-training dataset and of much smaller size; image exclusion could further harm its representation capability rather than improve its representation quality.
All the transferal datasets have strong knowledge transfer capabilities that are close to and even above 100%, except the Skin Lesion. Although Skin Lesion did not perform as well as the other transferal datasets, its knowledge transfer capability achieved 81.16% and 87.30% according to the two tests. In general it is not challenge to build a transferal dataset with strong knowledge transfer capability. At the same time, it necessary to point out that the four student networks have different learning abilities on the same transferal dataset. However, it is common for different networks to learn differently even on the dataset with manual annotation.
5.2 COMBINATION WITH FINE-TUNING
To evaluate the usefulness of knowledge transfer for downstream tasks, we conducted fine-tuning using 50 images with ground truth annotations (from Baheya Breast Lesion and Thyroid Nodule, respectively). Both the teacher model and student models that were trained in the previous experi- ments were further fine-tuned. For comparison, we also conducted the experiment by training the student models from scratch these images with ground truth. All the models were tested on the rest of the images of Baheya Breast Lesion and Thyroid Nodule, respectively
The fine-tuned teacher model, DeepLabv3+, resulted in average Dice score of 69.90%-131.0 on Baheya Breast Lesion and an average Dice score of 66.56%#47.788 on Thyroid Nodule. The results of the student models are presented in Tables and 5] All fine-tur tuned student models outper- formed the models trained from scratch by a large margin. With the same mini-tning dataset, some of the student models, such SDU-Net and Panoptic-FPN, performed similar or even better than

---

Under review as a conference paper at ICLR 2021
Table 3:
Average Dice scores (+stdd %) of segmentation Baheya Breast Lesion by student
models trained on pseudo-aunotated datasets. indicates image exclusion was employed. KTC indicates knowledge transfer capability.
U-Net
AttU-Net
SDU-Net
Panoptic
KTC
Baheya Breast Lesion
63.28:131.22
65.14:31.07
69.36110.82
67.89:131.93
106.66
Baheya Breast Lesion* *
66.73+29.86
64.98-31.83
69.66133.31
68.84:31.29
Thyroid Nodule
47.21=35.20
54.63=33.118
63.71=32.80
62.30=13.27
103.90
Thyroid =odule*
52.07+33584
57.92433.87
68.50+30.0
67.39++30.42
Nerve Structure
35.47++6.60
42.86
59.62
62.39
94.63
Nerve Structure*
41.83-134.91
44.26113.922
59,41132.98
62.10:32.22
Skin Lesion
28.16:22.57
27.641:22.77
30.17433.53
40.94137.19
87.30
Skin Lesion
37.15-129.99
37.03+26.36
54.54136.20
57.56135.78
PASCAL VOC2012
39.30+27.48
40.62430.04
58.68430.57
57.24131.82
99.38
PASCAL VOC2012*
52.76:+35.19
51.52437.13
65.52+31.87
63.20+34.35
Table 4: Average Dice scores (+stdd %) for student models fine-tuned or trained from scratch on Baheya Breast Lesion. indicates image exclusion was employed.
U-Net AttU-Net
SDU-Net
Panoptic
Trained from Scratch
37.20+24.665
37.15=224.7
46.88+28.87
52.88=30.31
Baheya Breast Lesion
49.294-29.00
65.31422968
70.03++-1.25
10.61-29..6
Baheya Breast Lesion*
61.45+-29.95
64.07++30.48
70.85+28.88
70.97-30.04
Thyroid Nodule
57.08+228.51
60.09+33.04
33.22
68.04-29.50
Thyroid *odule*
54.061199.41
59.38419.54
71.60128.95
68.60-29.94
Nerve Structure
56.58
63.21430.61
66.55131.83
71.19-28.29
Nerve Structure*
57.70+228.95
56.76.12940
69.33+-9.60
68.13=30.21
Skin Lesion
45.03+226.08
43.29+22.52
53.45+29.92
62.18=31.91
Skin Lesion *
52.41
51.71
70.62+29.39
67.73-29.00
PASCAL VOC2012
60.87
62.41130.57
69.93129.79
70.91 28.40
PASCAL *
57.11
61.63430.24
70.83429.92
71.67=29.32
the fine-tuned teacher model. This further verified that the effect of knowledge transfer between heterogeneous neural networks and demonstrate that the proposed algorithm can be well applied to downstream task.
5.3 KNOWLEDGE TRANSFER WITH MULTIPLE TEACHERS
We also tested if it was possible to transfer the knowledge of multiple weak independent teachers build a stronger student. We adopted U-Net, AtUUNeet, SDU-Net, and Panoptic-FPN the teacher models and employed DeepLabv3+ as student model. We train the teacher models on XXX Breast Lesion-1 for few epochs ensure under- fitting, thereby resulting in weak teachers. Baheya Breast Lesion was used as the transferal dataset. The trained teachers processed each image in the transferal dataset, with the average soft mask as pseudo mask. Then we trained the student model on the secudoaannotaeed transferal dataset.
We compared the performance of the teacher models and the student model on XXX Breast Lesion- 2 and Baheya Breast Lesion (Table 6). The student model outperformed all the teacher models on both test datasets. This is primarily due to fact that the student model actually learned from the ensemble of the teacher models, rather than any individual weak teacher model. Since the process of generating ensemble networks is not the primary focus this study, we simply average their outputs. With advanced ensemble, may be able train the student model better.

---

Under review as a conference paper at ICLR 2021
Table 5: Average Dice scores (+stdd, %) for student models fine-tuned trained from scratch
Thyroid Nodule. indicates image exclusion was employed.
U-Net
AttU-Net
SDU-Net
Panoptic
Trained from Scratch
34.02:1:3.89
35.85::24.31
46.00:127.08
48.96:126.92
Baheya Breast Lesion
50.93:127995
41.28147.74
57.99:130.00
64.92.128.22
Baheya Breast Lesion *
47.65++28.7
50.51+28.40
56.80=29.94
63.59=29.35
Thyroid Nodule
50.16:::99.0
47.91=30.37
59.26=32.555
64.73+29.9
Thyroid *odule*
55.41+-27.69
57.18+28.88
60.15+31.34
65.65++29.32
Nerve Structure
44.90++27.00
45.51+30.80
53.05-+33.07
65.37+28.23
Nerve Structure*
47.1626.66
45.650-128.98
53.92-28.577
63.26:+29.55
Skin Lesion
41.32::2.37
34.41124.68
49.97-29.69
54.30:130.29
Skin Lesion
43.21125.98
43.44128.61
55.91130.04
56.00:128.25
PASCAL VOC2012
41.93++26.00
48.59427.08
57.45-28.35
63.09427.31
PASCAL V
44.81
51.47299433
59.85-29.90
64.63++2..35
Table Average Dice scores (#std, %) for knowledge transfer with multiple teacher models.
U-Net
AttU-Net
SDU-Net
Panoptic
DeepLabv3+
XXX Br. Lesion-2 67.47++27.98
76.994221.96
71.49+ 73.63+ 49.22+34.25
78.71-18.6 48.95+34.01
Baheya Br. Lesion 49.30++34.33
48.75
58.40--31.05
6 DISCUSSION
The salient features of our algorithm include: no need for original training data extra generative networks; facilitating knowledge transfer between networks with different architectures; ease of implementation with fine-tuning to solve downstream task especially when using the downstream task dataset as the transferal dataset; capability of knowledge transfer from multiple models, trained independently. into student model. The algorithm also perform domain adaptation without extra computation in the case that the downstream task dataset used as the transferal dataset.
To the best of our knowledge, this is the first demonstration of image segmentation using pseudo annotations only. A transferal dataset with a large number of images of various contents has higher possibility to capture rich targets in the pseudo masks, but may also include many images without salient targets. Future work may include optimization of single transferal dataset or a combi- nation of multiple transferal datasets build better one. Different models may have different abilities to learn from the pseudo-and pseudo-annotated annotated transferal dataset. Understanding the differences in stu- dent model learning on weeudo-annotaan generate confidence in the proposed algorithm. Further study on knowledge transfer between neural networks with different input data structures different number of prediction classes also war- ranted. Another possible direction includes extending the ensemble of teacher models to produce pseudo masks.
transferal dataset and manually annotated dataset can help
Medical imaging applications require interpretable algorithms which generate confidence in the clin- ical user. Knowledge transfer employs pseudo annotation for training; these pseudo annotations have no physical meaning. It is imperative to examine and quantify the interpretability of student model before deploying models clinically.
CONCLUSION
Knowledge transfer be achieved between neural networks having different architectures and using an independent transferal dataset. Knowledge transfer from multiple networks improves the performance of the student network over the teacher networks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy, Dataset of breast ultra- sound images. Data in Brief, 28: 18:166 2020.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 801-818, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.02531, 2015.
Vladimir Iglovikov and Alexey Shvets. Ternausnet: U-Net with VGGI1 encoder pre-trained on imagenet for image segmentation. arXiv preprint (eti::8801,07700 2018.
Ata Jodeiri, Reza A Zoroofi, Yuta Hiasa, Masaki Takao, Nobuhiko Sugano, Yoshinobu Sato, and Yoshito Otake. Region- based convolution neural network approach for accurate segmentation of pelvic radiograph. In 2019 26th National and 4th International Iranian Conference Biomedical Engineering (ICBME), pp. 152-157. IEEE, 2019.
Jaeyong Kang and Jeonghwan Gwak. Ensemble of instance segmentation models for polyp segmen- tation in colonoscopy images. IEEE Access, 7:26440-26447, 2019.
Alexander Kirillov, Ross Girshick. Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6399-6648, 2019.
Tianhong Li, Jianguo Li, Zhuang Liu, and Changshui Zhang. Knowledge distillation from few samples. 2018.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. survey deep learning in medical image analysis. Medical Image Analysis, 42: 60-88, 2017.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535. 2017.
Saman Motamed, Isha Gujrathi, Dominik Deniffel, Anton Oentoro, Masoom A Haider, and Farzad Khalvati. A transfer learning approach for automated segmentation of prostate whole gland and transition zone in diffusion weighted mri. arXiv preprint arXiv:1909.0954/ 2019.
Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention U-Net: Learning where to look for the pancreas. arXiv preprint 1r84.v/804.03990 2018.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances Neural Information Processing Systems, pp. 3347-3357, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomed- ical image segmentation. In International Conference on Medical Image Computing and Computer-assisted Intervention, pp. 234 241. Springer, 2015.
Scudder. Probability of error of some adaptive pattern-rooognition machines. IEEE Transactions on Information Theory, 11(3):3633371, 1965.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pp. 270-279. Springer, 2018
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3D convolutional networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 4489-4499 2015.

---

Under review as a conference paper at ICLR 2021
Shuhang Wang, Szu- Yeu Hu, Eugene Cheah, Xiaohong Wang, Jingchao Wang. Lei Chen, Masoud Baikpour, Arine Ozturk, Qian Li, Shinn-Huey Chou, et al. U-Net using stacked dilated convolu- tions for medical image segmentation. arXiv preprint arXlv:2004.03466, 2020.
Ming Xie, Guodong Long, Tao Shen, Tianyi Zhou, Xianzhi Wang, and Jing Jiang. Multi-center federated learning. arXiv preprint arXiv:2005, 0/0266, 2020a.
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves ImageNet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020b.
Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-19, 2019.
Jaemin Yoo, Minyong Cho, Taebum Kim, and U Kang. Knowledge extraction with no observable data. Advances in Neural Information Processing Systems, pp. 2705-2714, 2019.
Guodong Zeng, Xin Yang. Jing Li, Lequan Yu, Pheng-Ann Heng, and Guoyan Zheng. 3d U-Net with multi-level deep supervision: Fully automatic segmentation of proximal femur in 3D MR images. International Workshop on Machine Learning Medical Imaging, pp. 274-282. Springer, 2017.
A APPENDIX
A.1 DATA PRE-PROCESSING AND AUGMENTATION
In addition pre-processing, employed five methods for data augmentation.
Pre-processing: All the images were resized 384 x 384 and the color images in Skin Lesion and PASCAL VOC2012 were converted into gray-scale images.
Random Cropping: A percentage consecutive rows and columns were cropped from the orig- inal image. The percentage
random value in the range [70%, 100%] and [80%, 100%] for segmentation and classification, respectively.
Horizontal Flipping: The image reversed horizontally, that is, from left right.
Random Rotation: The image was rotated around its center. The rotation angle was a random value in the range [-45 45%] and : for segmentation and classification, respectively.
Gamma Adjustment: Gamma correction was conducted on an image by the gamma value, which was randomly sampled in the range [0.7, 1.3].
Contrast Adjustment: The image contrast of an image was adjusted by the contrast factor, which was random value in the range [0.7, 1.3).
A.2 DATA DISTRIBUTION AND SELECTION
We used DeepLabv3+ that trained XXX Breast Lesion-1 to process the images in the five transferal datasets including Baheya Breast Lesion, Thyroid Nodule, Nerve Structure, Skin Lesion, and PASCAL VOC2012, and obtained the pseudo mask. Figure presents the geael entropy distribution of the pseudo mask for each transferal dataset.
By excluding images without obvious targets, where the target pixel number is less than 256 and the gray-level entropy is over 2.5, we could reduce the dataset to a smaller one. Table / presents the comparison of image numbers with and without image exclusion.
10

---

Under review as a conference paper at ICLR 2021
Baheys Breast Lesion
Thyroid Nedule
Nerve Structure
3000
(00
200
000
100
200
3
entrosy
entropy
Gray
Scn Lesion
1460
1000
number imagge
1200-
1000
FAUTOUT
603
Gray locel entropy
Gray lovel entropy
Figure 3: Image distribution with respect to the entropy. From left to right, the row presents the
result for Baheya Breast Lesion, Thyroid Nodule, and Nerve Structure; the second row presents examples for Skin Lesion and PASCAL VOC2012.
Table Image number of each transferal dataset before and after the exclusion of images without salient target in pseudo masks. Transferal Dataset
Images before Exclusion
Images after Exclusion
Baheya Breast Lesion
697
525
Thyroid Nodule 3,830
2,790
Nerve Structure
11,143
8,977
Skin Lesion 25,331
6,960
PASCAL VOC2012
17,125
5,503
A.3 TRAINING DETAILS
Two common loss functions were adopted with the same weight for the segmentation tasks, the Dice loss function and binary cross entropy loss (BCE). The Dice loss is defined
2 Ph.w ph, +
Luicc
(1)
Ph.w + Σ Ph.w +
where (h, w) represents the pixel coordinate, Ph.w {0, 1} is the mask ground truth, Ph, = 1 indicates the pixel belonging to the target, 0 S Ph. S the prediction probability for the pixel belonging to the target, is small real number.
The binary eross entropy loss is defined as:
1 1/2 Σ Ph,. log(Pn,w)) (1 Ph,w.) log(1 Ph, N
Lice
where (h, w) represents the pixel coordinate, Ph.o {0. 1} is the mask ground truth, Pha = 1 indicates the pixel belonging to the target, 0 S Ph. S the prediction probability for the pixel belonging to the target, N is the number of pixels.
Adam optimizer was used for training all the networks. The learning rate was set to 0.0001, and all the other parameters were set to the default values in Porchc. The batch size was set to 4. The epoch number was set to 500 for the training on the teacher teacher-training dataset. Inversely proportional to the 500, 175, 60, and 30 to for the training pseudo-
image number, the epoch numbers annotated Baheya Breast Lesion, Thyroid Nodule, Nerve Structure, and Skin Lesion. Moreover, the training epoch number for fine-tuning to 50.

---

Under review as a conference paper at ICLR 2021
A.4 ALGORITHM OF KNOWLEDGE TRANSFER COMBINED WITH FINE-TUNING
Iggorith@@combiinn knowledge transfer and fine-tuning for a downstream task of semantic image segmentation. In the algorithm, the transferal dataset is independent of the downstream task, but is from the downstream task, the algorithm can benefit from domain adaptation without extra computation.
Algorithm 2 Knowledge Transfer Combined with Fine-tuning for Semantic Segmentation
Require: Teacher set with trained models, randomly initialized student model S Require: Transferal dataset D, target task dataset D, with annotation
Require: Target pixel number threshold a, gray-level entropy threshold
I: for each datapoint r in D do
2: Obtain pseudo mask y = =ET, w; Tx(x)
3: N + number of target pixels (value above 0.5) in
4:
E f gray-level entropy of y
5:
if' N or E >> B then Exclude x from D 7: else
6:
8:
Update as (I, 3)
9: end if
10:
end for
Train student model on sneudo-annotated
12: Fine-tung mine-tunean student model S on target task dataset D,
return Trained student model
12


Reviews:
[{"review_id": "kyjYw8PPnAm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work the authors propose to transfer knowledge between teacher and student networks trained on separate datasets, and claim to overcome challenges in availability of data annotations for challenging semantic segmentation in medical imaging domain.\n\nStrengths\nThe proposed model is simple to follow and is targeted towards a significant problem in medical imaging analysis domain. \n\nComments\nThe authors have used five segmentation networks, it is suggested that the selection of these five algorithms is further justified. \nOne of the major concern in medical imaging domain is the black-box nature of the DL algorithms used, authors should comment on how relying on this black-box nature for knowledge transfer would effect the interpretability of these results. \nThe method relies on three datasets and three models to come up with the final target segmentation, what are the requirements on the size of these datasets, in general the authors should discuss the effect of this on the overall performance.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Lf6stxW5EUN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use student-teacher training as a way of knowledge transfer between neural networks with different architectures without access to the source data. Instead the authors propose to use a separate dataset to transfer the knowledge of the teacher network and a potential different dataset for fine-tuning. The paper evaluates their method with various segmentation architectures by pretraining a DeepLab v3+ on an internal breast lesion dataset and testing transfer and fine-tuning using different medical datasets. The authors find that knowledge transfer performs similar to regular transfer learning in most combinations of datasets.\n\nI believe that the paper tackles an interesting scenario of transferring knowledge from a fixed pre-trained network to a potentially different application without access to the original training data and sets up an extensive set of combinations of target tasks, student network architectures and transferral dataset (called dataset agent in the paper).\n\nHowever, I cannot recommend the paper for acceptance in its current form. Most importantly the paper seems to be lacking proper positioning within the space of knowledge distillation and student-teacher training which leads to an unclear message about the novelty of the paper. Student-teacher training for knowledge distillation is not novel and the message that it is possible to transfer knowledge using student-teacher training  is unsurprising given that it has been shown before that you can transfer knowledge without any observed data (e.g. KegNet: Knowledge Extraction with No Observable Data. Yoo et al. NeurIPS 2019).\nFurther, the experiments do not contribute any new insights about how to chose the best student network nor which transferral dataset to use even though the introduction refers to unsuitability of certain pretraining tasks for a different target task. The first example (Table 2) seems to be using an internal dataset that randomly has been split into train/val/test splits and therefore resembles no real transfer learning task and it seems unsuprising that the 'direct learning' approach yields the best results. The second example (Table 3) using the Baheya breast lesion dataset seems to tackle the problem of unsupervised domain adaptation rather than transfer learning: the teacher and target dataset both tackle breast lesion segmentation on ultrasound images. Here it could be that using the target dataset as transferral dataset might help to adjust batch statistics for potential normalisation layers to improve the performance. This leaves the last two examples as only real transfer learning experiments. Lastly, the whole setup assumes that the input and output space of the student and teacher network are always the same, while it is argued that this approach allows for flexibility in difference of network architectures between the student and teacher network. However, semantic segmentation tasks in medical imaging often appear with various numbers of classes and 'input channels' requiring more advanced knowledge distillation -- this could be an interesting problem to tackle in a later version of this work.\n\nFurther comments:\n- Data preprocessing: What preprocessing are you using for the training of the networks? How are you handling different shapes of the images? Are the segmentation algorithms trained on the full images or on patches?\n- Transferral datasets: Looking at the images in Fig. 2, it seems that even the same modality ultrasound images show different sorts of image artefacts - do you clean those at all? Do you think domain shift might be something that's interacting with your setup?\n- Do you have any theoretical or intuitive justification why you would want to perform knowledge transfer using unrelated data (skin lesion / different anatomy etc)? Why should this be better than using no data at all or regular computer vision datasets? Do you think the number of training examples for transfer matters?\n- What do example segmentations look like? Are there similar shapes for different datasets? Does the network also learn some sort of shape prior? (see Oktay, et al. Anatomically constrained neural networks (ACNNs): application to cardiac image enhancement and segmentation. 2017)\n- Which loss did you use: CE / Dice-loss or a combination of the two?\n- The term dataset agent is already used in the abstract and is not very clear - I'd personally find something like 'transfer[ral] dataset' easier to grasp.\n- Introduction, first paragraph: 'black-[space]box' -> 'black-box'\n- Introduction: 'network(teacher)' -> 'network (teacher)'\n- What's a latent dataset? I would rather simply refer to 'learned representations' or 'knowledge'\n- 'XXX' is already used in the introduction and not explained - I would simply refer to 'internal / in-house datasets'. Also, note the comment on the breast lesion dataset only being a single dataset with different splits.\n- I have not seen the term 'educated' in reference to neural networks before - it would be more common to say 'trained'.\n- Section 5.3.2): You mention that the networks trained from scratch have poor performance because of the small tuning dataset - I guess you are referring to the small training set?\n- Another potential reference for knowledge transfer for medical imaging could be Kuzina, et al. Bayesian Generative Models for Knowledge Transfer in MRI Semantic Segmentation Problems. 2019\n- As this work is relatively application-specific it might be better suited for one of the more medically inclined venues like MIDL, MICCAI, ...\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-6rkWfXRaw5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a  knowledge transfer technique based on  training a student network using annotation creating by  a teacher network. This is actually not a summary of the method but  the method itself. Most the the rest of the paper is devote do describe experiment details. \n\nThe idea is  well known in machine learning community see e.g. Distilling the Knowledge in a Neural Network by Hinton. where is used to transfer knowledge from a huge network to a small network. Hence, there is not much novelty in the paper.\n\nAlthough the method is very simple it is difficult the follow the experimental results. It is written in a very unclear way.\nDo you use step 3 in the experiments? \n\nwhat is your conclusion regarding parameter fine tuning vs. your approach?\n\nOver all the paper is more suitable for a medical imaging conference than  fro a general deep learning conference. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A majority of the reviewers find the paper lacks novelty and provides an insufficient discussion of the state-of-the-art in knowledge distillation and student teacher training to warrant publication.
The approach is quite narrow to the application domain and the paper does not provide novel insights on how to chose a good network.
A subset of the experiments performed on an internal data set with random train-test-splits do not evaluate a realistic transfer setting.


Decision: Reject