Cross-Level Distillation and Feature Denoising for Cross-Domain Few-Shot Classification
Authors: Hao ZHENG
Runqi Wang
Jianzhuang Liu
Asako Kanezaki
Venue: ICLR 2023 poster (2023)
Keywords: cross-domain few-shot classification
cross-level distillation
feature denoising

Abstract: The conventional few-shot classification aims at learning a model on a large labeled base dataset and rapidly adapting to a target dataset that is from the same distribution as the base dataset. However, in practice, the base and the target datasets of few-shot classification are usually from different domains, which is the problem of cross-domain few-shot classification. We tackle this problem by making a small proportion of unlabeled images in the target domain accessible in the training stage. In this setup, even though the base data are sufficient and labeled, the large domain shift still makes transferring the knowledge from the base dataset difficult. We meticulously design a cross-level knowledge distillation method, which can strengthen the ability of the model to extract more discriminative features in the target dataset by guiding the network's shallow layers to learn higher-level information. Furthermore, in order to alleviate the overfitting in the evaluation stage, we propose a feature denoising operation which can reduce the feature redundancy and mitigate overfitting. Our approach can surpass the previous state-of-the-art method, Dynamic-Distillation, by 5.44% on 1-shot and 1.37% on 5-shot classification tasks on average in the BSCD-FSL benchmark. The implementation code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/CLDFD.

Published as a conference paper at ICLR 2023
CROSS-LEVEL DISTILLATION AND FEATURE DENOIS- ING FOR CROSS-DOMAND FEW-SHOT CLASSIFICATION
Hao ZHENG*
Runqi Wang"
Tokyo Institute of Technology zheng. h, ad0m. titech.aate ac. jp
Huawei Noah's Ark Lab
com
Jianzhuang Liu
Asako Kanezaki!
Huawei Noah's Ark Lab
Tokyo Institute of Technology kanezaki@c. tiwechttthhh..e.... acc. ip
1iu. jianzhuang@huaweii. com
ABSTRACT
The conventional few-shot classification aims at learning model on a large la- beled base dataset and rapidly adapting to a target dataset that is from the same distribution as the base dataset. However, in practice. the base and the target datasets of few-shot classification are usually from different domains, which the problem of cross-domain few-shot classification. We tackle this problem by making a small proportion of unlabeled images in the target domain acces- sible in the training stage. In this setup, even though the base data are suffi- cient and labeled, the large domain shift still makes transferring the knowledge from the base dataset difficult. We meticulously design cross-level knowl- edge distillation method, which can strengthen the ability of the model to ex- tract more discriminative features in the target dataset by guiding the network's shallow layers to learn higher-level information. Furthermore, in order to al- leviate the overfitting in the evaluation stage, propose feature denois- ing operation which can reduce the feature redundancy and mitigate overfit- ting. Our approach can surpass the previous state-of-the-ar method, Dynamic- Distillation, by 5.44% on 1-shot and 1.37% on 5-shot classification tasks on aver- age in the BSCD-FSL benchmark. The implementation code will be available http://iceccommmmoecmmmmeeeeeeee
INTRODUCTION
Deep learning has achieved great success on image recognition tasks with the help of a large number of labeled images. However, it is the exact opposite of the human perception mechanism which can recognize new category by learning only few samples. Besides, a large amount of annotations costly and unavailable for some scenarios, It is more valuable to study few-shot classification which trains a classification model on base dataset and rapidly adapts to the target dataset. However, due to the constraint that the base data and the target data need to be consistent in their distributions, conventional few-shot classification may not cater to the demands some practical scenarios.
For example, it may fail in scenarios where the training domain is natural images, but the evaluation domain is satellite images. Considering this domain shift in practical applications, we focus on cross-domain few-shot classification (CD-FSC) in this paper. Previous methods, such (Mangla et al., 2020; Adler et al., 2020; Tseng et al., 2020), can handle this problem with small domain gaps. However, the CD-FSC problem with a large domain gap is still a challenge.
BSCD-FSC (Guo et al., 2020) is a suitable benchmark for studying this problem, where the base dataset has natural images and the target datasets contain satellite images, crop disease images, skin disease images and X-ray images of sundry lung diseases. On this benchmark, previous methods following the traditional CD-FSC protocol train their models on the base dataset and evaluate them
"Co-first author. Part of this work was done during an internship in Huawei Noah's 's Ark Lab. 'Correponding author.

---

Published as a conference paper at ICLR 2023
Draiet
Tirge Ditect
Terpe Ou line
Thryet Daaant
Urliiie
Fastory
City
Quary
Teacher
Student
Toacher
Student
Student
Cross-Fatiopy Loss
Cress- evel Distilationn
Clasifier
Festure Denoising
Chassifer
Pro-Training Phase
Training Phose
Fino-Tuning Phase
Evaluation Phase
Figure I: Our CD-FSC framework. The first phase is pre-training, which trains the teacher network on the labeled base dataset by optimizing the cross-entropy loss. The second phase trains the stu- dent network using our proposed cross-level distillation (CLD). The third phase fine-tunes linear classifier on few labeled images in the target domain, and feature denoising (FD) is conducted to remove the noise in the final feature vectors. The final phase classifies images in the target domain.
on the target dataset, but their performances are far from satisfactory. STARTUP (Phoo & Hariharan, 2021) and Synaamc--istillaann
(Islam et al., 2021a) introduce a more realistic setup that makes a small portion of the unlabeled target images accessible during the training phase. These target images bring a prior to the model and dramatically promote the model's performance on the target datasets. Inspired by that, follow their setup to explore the CD-FSC problem with large domain shift. In this work, we propose a cross-level distillation (CLD), which can effectively transfer the knowledge from the base dataset and improve the performance of the student network on the target domain. Besides, we propose feature denoising (FD) to remove the noise in the features during the fine-tuning stage. Our CD-FSC framework is given in Figure 1.
The detail of CLD is shown in Figure 2, which distills a teacher's deeper layers to a student's shal- lower layers, where the student and the teacher share the same structure. Unlike the distillation methods in STARTUP and Synamic-Distillation which only distill the teacher's last layer to the student's last layer, our CLD leads the shallow layers of the student to mimic the features generated from the deeper levels of the teacher so that the student can learn more deeper semantic information and extract more discriminative features on the target dataset. Additionally, since the teacher net- works in STARTUP and Dynamic-Distillation Distillation are pre-trained on the base dataset only, the teacher's s observation of the target data is biased. In order to calibrate the bias, we design an iterative process by building another network which shares the same structure and parameters with the historical stu- dent network, named old student network. In each training iteration, the features from the teacher and the old student in the same layers are dynamically fused to guide the corresponding layers of the student. The latter the training iteration, the fewer fusion features from the teacher network, and the more from the old student network. Due to the target data used in training is unlabeled, the self-supervised loss is introduced to excavate the target domain information further.
The sele-supe loss only supports the network in mining valuable information on the target domain, but also brings a phenomenon where the final feature vector for classification has a small number of dominant (strongly activated) elements with the others are close to zero (Hua al., 2021: Kalibhat et al., 2022). We find that during the fine-tuning phase in Figure 1, these small activated elements are redundant and considered as noise. Our FD operation keeps the top h largest elements and sets the others to zero. It is experimentally verified that FD can greatly improve the model's performance.
Above all, our main contributions are summarized below:
We propose a cross-level distillation (CLD) framework, which can well transfer the knowl- edge of the teacher trained the base dataset to the student. We also use old student network mechanism is also necessary to calibrate the teacher's bias learned from the base data.
Considering the noisy feature activations, we design feature denoising (FD) operation that can significantly improve the performance of our model.
Extensive experiments are conducted to verify that our proposed CLD and FD can achieve ofth--f-t results the BSCD-FSLL benchmark with large domain gaps.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Cross-domain few-shot classification. The cross-domain few-shot classification is firstly defined by (Chen et al., 2018), which trains a model on the base dataset and evaluates it on the target dataset in a different domain from the base dataset. LFT (Tseng et al., 2020) simulates the domain shift from the base dataset to the target dataset by meta-learning and inserts linear layers into the network to align the features from the different domains. Meta-FDMixup (Fu et al., 2021) uses several labeled images from the target dataset for domain-sharee feature disentanglement and feeds the domain- shared features to the classifier. FLUTE (Triantafillou et al., 2021) learns the universal templates of features across multi-source domains to improve the transferability of the model. However, all these methods concentrate on the CD-FSC problem with small domain shifts. Some methods handle CD- FSC with large domain gaps, in which the target datasets have obvious dissimilarity from the base dataset on perspective distortion, semantics, and/or color depth. For example, ConfeSS (Das et al., 2021) extracts useful feature components from labeled target images. ATA (Wang & Deng, 2021) does not require any prior of the target dataset and proposes plug-and and-play inductive bias-adaptive task augmentation module. CI (Luo et al., 2022) trains an encoder on the base dataset and converts the features of the target data with a transformation function in the evaluation stage. UniSiam (Lu et al., 2022) adopts a supersuee approach to address the CD-FSC problem. Among the methods dealing with large domain shifts, STARTUP (Phoo & Haribaran, 2021) is strong baseline, which uses a few unlabeled target images in the training stage. It firstly trains a teacher network on the base dataset in supervised fashion and transfers the teacher's s knowledge to the student by knowledge distillation (KD). It jointly optimizes the cross-entropy loss with the base dataset, contrastive loss of the unlabeled target images and the KD loss to upgrade the student network. Dynamic-Distillatii (Islam et al., 2021a) also uses a small number of unlabeled images and KD. The main difference between Dynamic-Distillaatee dynamically by exponential moving averages, while the latter fixes the teacher. In our work, we follow their data setup allowing a small proportion of the unlabeled target images be seen during the training phase. Different from the two methods that perform KD at the last layers of the teacher and the student, our KD is carried out at cross levels. Besides, our denoising operation further improves the performance.
and STARTUP is that the former upgrades the pre-trained teacher
Self-supervised learning. Self-supervised learning is widely used in the scenarios where labels are not available for training. It defines a "pretext task" to pre-train network. For example, (Gidaris et al., 2018) pre-trains the model by predicting the rotation angle of the image. One popular method is contrastive learning, such as SimCLR (Chen et al., 2020) which pulls different augmented versions of the same image closer and pushes the versions from different images away. Beyond contrastive learning, BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021) rely only positive pairs.
Knowledge distillation. (Hinton et al., 2015) firstly propose knowledge distillation by guiding a compact network (student) mimic the output of large network (teacher). Since features in the intermediate layers are informative, some previous methods distill the teacher's intermediate features (Romero et al., 2014) attention maps of the features (Zagoruyko & Komodakis, 2017). Besides, delf-distillation methods, such as BOT (Zhang et al., 2019), distill the last layer of the network to its own shallower layers. BYOT's teacher and student are the same network.
In our framework (Figure 2), in addition to the KD in the intermediate layers, we design an old student that shares the same structure as the student but with different parameters. The introduction of this old student not only alleviates the teacher's 's bias learned from the base dataset, but also has the effect of assembling multiple historic students during training.
METHODOL.OGY
3.1 PRELIMINARY
During the training period, we follow the setting in STARTUP (Phoo & Haribaran, 2021) and Dynamic-Distillatio (Islam et al., 2021a), where labeled base dataset DB and a few unlabeled target images sampled from the target dataset D1 are available. In the testing stage, the support set Dg comprises N classes, and K samples are randomly selected from each class in Dr, which is the so-called N- N-way K-shot task. The support set Ds is for fine-tuning a new classifier with the frozen encoder (the student network in this work). The images in the query set D2 are randomly picked
3

---

Published as a conference paper at ICLR 2023
X3
Stndent
Tencher
Old Student
8
Ais
0j
8:
dis
0²
8
0
Projectx
Supp
Figure 2: Framework of CLD for knowledge distillation (KD). The teacher network is pre-trained on the labeled base dataset with the cross-entropy loss and fixed during KD. When training the student network, the target image x is augmented into x1, X2 and X3- Then X1, is fed to both the student and the teacher, and X3 is fed to the old student. At the i-th iteration, the parameters of the old student copy of those of the student at (i 7)-th iteration. The feature sj of the student is firstly projected by Wi for dimensionality alignment, where the block index. Then we fuse the features t/41 and tf obtaining uf+1 which are from the (I 1)-th block of the teacher and the old student, respectively. The KD conducted by forcing (s() to mimic u/41- Additionally, the self-supervised loss C., is introduced on the student network.
from the selected classes for evaluating the classification accuracy. The support set D3 and the query set Do have overlap.
3.2 CROSS-LEVEL DISTILLATION
The proposed cross-level distillation (CLD) framework is shown Figure 2. The teacher network fi is pre-trained on DB with the cross-entropy loss. The student network fa is expected to inherit the knowledge of the teacher and extract discriminative features of Dr. However, the teacher's the base dataset only. In the i-th
observation of the target data is biased since it is pre-trained training iteration, if the features extracted by the student f: directly mimic the features of the teacher, the teacher's bias will be transferred the student. To reduce the bias, introduce an old student network for which is a copy of fi-", where the hyper- parameter denotes the training iteration interval between f fi-7 and fg.
To simplify the KD complexity, we divide each backbone of fa. for and ft into L residual blocks. Let s/. o/, and t/ be the features obtained by the student, the old student, and the teacher in the 1-th block the i-th iteration. The fusion between tj and of is defined as:
u a of + (1 a )t
(1)
where a" = T is dynamic weight with T being the total number of training iterations. loss L in the i-th iteration is defined as:
The KD
Σle [|w(())) 11 if
Cass
(2)
El= [|w((s)) uiu1l2 otherwise,
where (.) is projector of the 1-th student's block for feature dimensionality alignment. It is comprised of a convolutional layer, a batch normalization operator and ReLU activation. Note that the KD in Equation is from the (1 + 1)-th block of the teacher the (-th block of the student. In our experiments, we find that this style is better than others (see Section 4.3).
The total loss L for training the student network is:
L Loss XLdis:
(3)
4

---

Published as a conference paper at ICLR 2023
where Les is self- celffsupervised loss, and X is weight coefficient of loss function for balancing the two losses.
For Les, off-the-shee self- self-suue losses like SimCLR (Chen et al., 2020) BYOL (Grill al,, 2020) can used. The contrastive loss in SimCLR is:
exp (sim (Z., Znd /7)
Ssimer
log
(4)
B
E 2g=1.gmm exp (sim (Zm 2ny /7)
M,nEB
where Zm. Zy and zy are the projected embeddings of different augmentations, (m, n) is a positive pair from the same image, sim(-) is similarity function, B is mini-batch of unlabeled target images, and y is a temperature coefficient. The self-supervised loss in BYOL is:
sim(p( km) km)
byyol
2
(5)
[[p(zm)
mEB
where Zm and z'n are embeddings of the online network and the target network, respectively, and p() is a linear predictor.
Note that the last convolution block of the student network is not involved in KD and is trained by minimizing Las only. The reason that the last block mainly discovers semantic information that is highly domain-specifie. Therefore, we constrain it the target data rather than letting it learn from the teacher that is pre-trained on the base data.
3.3 FEATURE DENOISING
The self-supervised loss brings a phenomenon where the final feature vector for classification has small number of strongly activated elements while the others are close to zero (Hua et al., 2021; Kalibhat et al., 2022). These elements of small magnitudes are regarded as noise, which may cause overfitting, We propose feature denoising (FD) operation to remove their contribution during the fine-tuning phase (see Figure 1). FD keeps the largest elements the feature from the student network and zeros the other elements. The FD operation is only performed the feature sz in the last layer of the student network. Specifically, let S, = [$(L.1): S(L.2), (...))........ SLL.D.)) and $1. = be the features before and after the FD operation, respectively, where Dr is the feature's is dimensionality. Then the FD operation is defined as:
(B(L,)), f(,.2)....,
{ (sL if $L,d (op) d 1,2, Dr otherwise,
where 3 is dayper-parmmtte which makes the non-zero elements more distinguishable and topa (-) is the operator selecting the largest / elements of the feature. Finally, SM is fed to the classifier for fine-tuning.
EXPERIMENTS
4.1 DATASETS AND IMPLEMENTATION
Datasets. We evaluate the proposed CLD and FD for the CD-FSC problem on the BSCD-FSL benchmark (Guo al., 2020) with large domain gaps. The minilmageNet dataset (Vinyals al., 2016) serves as the base dataset D, which has sufficient labeled images. EuroSAT (Helber et al., 2019), CropDisease (Mohanty et al., 2016), ISIC (Codella et al., 2019) and ChestX (Wang et al., the unlabeled target datasets Dr. We follow the training protocol in STARTUP (Phoo & Hariharan, 2021) and Dynamic-Distie Distillation (Islam et al., 2021a), which al- lows the whole labeled training set of minilmageNet and small proportion (20%) of unlabeled target images available during the training period. The remaining 80% of target images are utilized for fine-tuning and evaluating by building 5-way K-shot tasks, K E (1,5).
2017) in BSCD-FSL
Implementation details. We implement our model using the MindSpore Lite tool (Mindsporo). For fair comparison, all the methods use ResNet-10 (Guo al., 2020) the backbone. Our
5

---

Published as a conference paper at ICLR 2023
Table 1: The averaged 5-way 1-shot and 5-shot averaged accuracy and 95% confidence interval among 600 episodes are given. The reported results of SimCLR (Basc) and previous state-of-the-art Dynamic-Distillatte
are from (Islam et al., 2021a). The results of CI are from (Luo et al., 2022). The results of ATA are from (Wang & Deng, 2021). The performance of ConFeSS refers to (Das al., 2021) and they do not give the confidence interval. The champion results are marked in bold.
ATTSAT
CropDisease
ISIC
Chest
1-shot
5-shot
1-shot
5-shot
1-sbot
5-shot
I-shot
5-shot
Transfor
58. 38.42:0.94
75.31=0.71
68.45=0.87
89. 19.1220.52
32.82:00:0
47.13=+0558
22.48+041
26.65=0043
SimCLR (Base)
58. 58,28:00.00
80.83++0.64
68 0.88
83.44+0.61
32.15:+0.59
45.90=0.58
22,37=0022
26.63=0046
58.8210.92
76.26.10.70
61.58_0.88
89.2510.51
32.43:0.56
44.0420.55
23.23.10411
27.20:10.44
TransferSSmmCRR
65.9220.88
81.83:10.59
81.39_0.80
94.8510388
33.86:10.61
47.25:0.59
22.9110.45
27.03:20:43
STARTUP
65.24+00.88
$1.60+0059
18-0.3
92.86+0.44
34.15-00.22
46.42+0.588
22.86+00.3
2698-+0.43
ATA
65.94:0.50
79.47=0.30
77.82=0.50
88. 15.0550.50
34.70:20:40
45.83=0.30
21.67:0.200
23.60:00200
BYOL
64.38:0.93
82.44:0.56
67.15=1.17
99.4020.63
29.21:0.511
41.4420.51
21.66+0.31
25:99=:0:3
ConTeSS
84.65
88.88
48.85
27.09
Dynami--Disillation
73:14:0.84
89.07=0.47
82, 0.700
95.5440.38
34.66::0.58
49.36:2.59
23.38+0043
28.31:0:46
SimCLR
75.10=0088
90.17:0443
88. 08.54=0.80
96.09+0.39
36.71=0066
48.84=0.61
22.00+0.42
24.84=0.41
BYOL&CI.DeFI (ours)
72,78+0.84
88.50+0.45
86.94-0.73
95.85+0,38
37.70+00.7
51.19+0.63
24.34+0.44
30.15-0.44
SinCIR&+C (ours)
82.52++0.76
92.89+0.34
90.48 0.72
96.58+0.39
39.70-0.69
02.29-0.62
22.39+0.44
25.98+00.3
Block-1
Block-2
Block-3
Block
(a)
(b)
Figure 3: t-SNE results of (a) STARTUP and (b) SimCLR+CLD+FD at different blocks on the EuroSAT dataset. It is obvious that method can extract more discriminative features.
model optimized by SGD with the momentum 0.9, weight decay le-4, and batch size for 600 epochs. The learning rate is 0.1 at the beginning and decays by 0.1 after the 300th epoch and 500th epoch. The nyper-parameters in Equation 3, and h and in Equation 6 are set to 2, 64, and 0.4, respectively. For fine-tuning and evaluating the student network, we randomly sample 600 episodes of 5-way K'shoo tasks. The performance is represented by the average classification accuracy over the 600 episodes within the 95% confidence interval. In each episode, the parameters of the student are frozen. An additional linear classifier is fine-tuned by minimizing the cross-entropy loss. The above mentioned hyperparameters are determined based the AT dataset only, and then they are used for evaluation on all the datasets, showing the generalization ability of our method.
4.2 MAIN RESULTS
We select several basic and competitive methods for comparison in Table 1. The basic model is Transfer that trains the encoder on the labeled base dataset with the cross-entropy loss. SimCLR (Base) (Chen et al., 2020) trains the model on the base dataset (minilmageNet) with the contrastive loss of SimCLR. CI (Luo et al., 2022) trains the encoder on the base dataset and converts the fea- tures of the target dataset with a transformation function in the evaluation stage. Transfer+SimCLL is proposed in (Islam et al., 2021b) and exhibits good transferability. which simultaneously opti- mizes the cross-entropy loss on the base dataset and the contrastive loss of SimCLR on the target dataset. STARTUP (Phoo & Hariharan, 2021) trains the model with three loss functions: cross-

---

Published as a conference paper at ICLR 2023
Student
United Teacher
Student
United Teacher
Student
United Teacher
(a) B2-to-B1
(b) B3-to-B2
(c) B4-to-B3
Figure 4: Different single-block distillation structures. B2-to-B1 means that the second block of the teacher is distilled to the first block of the student. B3-to-B2 and B4-to-B3 are similar.
Table 2:
Results of different single-block distillation stuctures on EuroSAT (5-way 1-shot). B2-to- B1, B3-to-B2, and B4-to-B3 are defined in Figure
SimCLR
B2-to-B1
B3-to-B2 B2
B4-to-B3
CLD
75.10::0.55
78.5410.80
78.78
77.9244..3
79.38:1082
entropy loss on the labeled base dataset, and contrastive loss and KD loss on the unlabeled target domain. ATA (Wang Deng, 2021) designs a plug-and-play inductive bias-adaptive task augmen- tation module. BYOL (Grill et al., 2020) is trained on the target dataset. ConfeSS (Das et al., 2021) uses labeled target images find out the useful components of the target image features. Dynamic-Distilaation Distillation (Islam et al., 2021a) designs distillation loss that draws support from the dynamically updated teacher network. SimCL. trains the encoder on unlabeled target data only and shows good generalization ability. For our method, we build two models B BYOL+CLD+FD and SimCLR+CLD+ED, which use Equations and for self-supervised losses, respectively.
Table gives the comparisons among our models and the baselines. All the best results on the four datasets are obtained by either of our two models. particular on AuroSAT, our SimCLR+CLD++D bring 9.38% and 7.42% gains over Dynamic- Distiiiiistiilattion and SimCLR on the 5-way 1-shot task, respectively. On average, on the four datasets, SimCLD+CLD+FD outperforms Dynamic- Distillation significantly (58.77% 53.33%) for 1-shot; 66.94% 65.57% for 5-shot. Be- sides, although SimCLR+CLD+#II and BYOL+CLD+FD are based on SimCLR and BYOL, re- spectively, the performances of SimCLR and BYOL improved greatly. Finally, we visualize the feature distributions of STARTUP and our SimCLR+CLD+FD in Figure 3. It can be seen that SimCLR+CLD+HE exhibits better clustering results at all the blocks, especially the last one.
4.3
ABLATION STUDY ON CROSS-LEVEL DISTILLATION
Student
United Teacher
Student
United Teacher
Student
United Teacher
(a) same-level distillation
(b) one-to-all distillation
(c) CLD
Figure 5: Different multi-blocks distillation structures. (a) and (b) are common KD methods. (c) is our cross-level KD.
For simplicity, the combination (according to Equation 1) of the teacher and the old student in Figure 2 is denoted as the united teacher Figure and Figure 5. In these experiments, Csimch is used as Les and FD is not employed. Table 2 gives the results of SimCLR, different single-block distillation structures, and CLD on EuroSAT (5-way 1-shot). We can notice that all the single-block distillation structures perform better than SimCLR, and CLD outperform all the single-block distillation

---

Published as a conference paper at ICLR 2023
Table 3: Results of different multi-blocks KD structures.
EuroSAT
CropDiscase
ISIC
ChestX
I-shot
5-shot
1-shot
5-sbot
I-shot
5-shot
1-shot
5-shot
SimCLR
75. 15.15
90.17:00.3
88.54+0.80
96.09=0.339
35.71=0.66
48.84:20.1
22.00=0.42
24.84:20.41
same-level distillation
75.48=0.85
90.71
14+0.80
95.76+0.44
36.56=0.65
50.51+0.58
22.12:40.42
25.20::0.42
one-to-all distillation
77.70=0.83
91.80+0.38
89. 39.11--00 11+0.79
95.83+4044
36.09=0.64
50.64+0.62
22.10+0.43
24.77:40.43
CLDD
79.38- 0..8
92.78 +0.35
0.78
96.08-00.43
37.82 0.68
52.28 +0.61
22.08-+0.44
25.08++0.42
0.75
sarte-los lovel diatiaaier
0.65
CCID
Rual bado
0.35
0:25
0.15
Blod-1
Block2
Blode3
Bloc-4
Figure 6: Rand Index values in the residual blocks of the methods in Table 3.
structures. Table 3 gives the results of SimCLR, different multi-blocks distillation structures and CLD. Compared with SimCLR without KD, the multi-blocks KD structures improve the model's overall performance. Second, the one-to-all KD outperforms the same-level KD in most cases. Finally, our CLD performs best on average among the three multi-blocks KD structures.
We further explore the reason that CLD can exceed other methods. Rand Index (Rand, 1971) is metric to reflect the quality feature clustering. The bigger Rand Index is, the better clustering the method provides. Figure 6 is comprised of the Rand Index values of the features in each residual block on the EuroSAT test dataset. Our CLD increases more than all the other methods in each block. It means that our CLD can pick up more discriminative information each level that the model gradually hunts the useful features as the network deepens.
Next, examine the usefulness of the old student and the effect of the training iteration interval (Section 3.2). Table shows the experimental results of different settings, First, CLD without the old student outperforms SimCLR. Second, using the old student is better than not using it. Considering the performance and the memory requirement, we choose = in all the other experiments on the four datasets.
Table 4: Effects of the old student and the training iteration interval r. All the methods in the table use Laimelr as Lys.
ET
CropDisease
ISIC
ChestX
1-shot
5-sbot
I-shot
5-sbot
1-shot
5-shot
1-shot
5-shot
SimCLR
75.1020.85
90.1720.44
88.5420.80
96.09:10.39
35.7110.66
48.84200.61
22.0020:42
24.8420.41
CLD (wlo old student)
76.85=0.83
01.58=0044
88.72#0.78
95 05.85:00.33
37.0420.65
$0.90:20.0
21 21.40=0.32
23.91=0.39
CLD (r = 0)
78.56:0082
92.32=0.35
89.11
05.69=20.45
37, 37.8240.69
51.17:0.62
22.03=0.43
25. 25:12:0.44
CLD (r 1)
79.38=0082
92.78-00.35
89.43-10.78
06.08-0.45
37.82+0.66 68
52.28=0.61
22.08=00.43
25.08:40.42
CLD (7 10)
78.1810081
02.17:10.6
89.24:10.00
95.83:20.44
38.02:10.68
51.77:0.62
22.02:0:44
25.04:0.43
CLD (r 100)
78.7920.80
02.44:20.37
89.56:10.81
96.06:20.42
36.8110065
50.60:2062
22.29:0:43
25.17:20.43
4.4 ABLATION STUDY ON FEATURE DENOISING
On EuroSAT, the model SimCLR+CLD+FD is used to find optimal h and 3 on the 5-way 1-shot tasks. As shown in Figure 7, h = 64 and 3 = 0.4 are chosen for the best accuracy.
With = 64 and 3 = 0.4 FD, we compare using FD or not in Table 5. We can see that the first five models improved with FD, while the last two are not. The different positions, which the
8

---

Published as a conference paper at ICLR 2023
Accumay
78
60
04 03
56
(A)
(
Figure 7: (a) Accuracy vs. h when / = 0.4. (b) Accuracy vs. 3 when h = 64.
Table 5: Comparison of using FD or on EuroSAT (5-way 1-shot).
SimCLR
BYOL
SinCLR+CLD
BYOL+CLD
one-to-all distillation
TransferSSmmCLR
STARTUP
w/o FD
75.10=0.85
64.3810.93
79.38:10.82
69.47:10884
77.70=1.83
65.92:10.88
65.2410.88
w/ FD
78.02-00.77
69.80+0.85
82.5240.766
72.78+0.84
80.62+0.77
63.97-40.85
60.87+0888
+5.42
+3.20
4.37
auxiliary loss (cross-entropy and/or KD) is applied in the two groups are shown Figures (a) and 8(b), respectively. The reason why FD does not help in Figure (b)) is that the loss Laux used in the final layer reduces the noisy elements of the final feature vector for classification.
Self Supervised Loss
83
Amcilory Loss
Accuncy (NS)
Natwork Layer
82
81.5
641
0.5
2.5
Figure 8: Las is the self-supervised loss and Laux includes other losses such as the cross- entropy and KD losses. (a) Laux is applied to intermediate layers. (b) Laux is applied to the final layer.
Figure 9: Results of SinCLR+CLD+FD with the different values on the EuroSAT dataset (5-way 1-shot).
4.5 ABLATION STUDY ON WEIGHT COEFFICIENT OF LOSS FUNCTION
We give the results of SimCLR+CLD+FD with the different À values in Equation 3 on the EuroSAT dataset (5-way J-shot), as shown Figure 9, We can that the best result obtained with À 2. So we set = in all the related experiments.
CONCLUSION
In this work, we handle the CD-FSC problems with large domain gaps between the base dataset and the target datasets. We propose the cross-level distillation KD for better transferring the knowledge of the base dataset to the student. We also present the feature denoising operation for mitigating the overfitting. Our method improves the performance of strong baseline Dhamii--Dstilllttonn by 5.44% on I-shot and 1.37% on 5-shot classification tasks on average in the BSCD-FSL benchmark, establishing new state-of-the-ar results.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We gratefully acknowledge the support of MindSpore (Mindspore), CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research. We would also like to express our sincere gratitude to Mr. Haichen ZHENG, Mrs. Jing LIN, and Ms. Xiao WANG for their unwavering support and confidence in our work.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Thomas Adler, Johannes Brandstetter, Michael Widrich, Andreas Mayr, David Kreil, Michael Kopp, Günter Klambauer, and Sepp Hochreiter. Cross-domain few-shot learning by representation fu- sion. wwXiv:20100064400 2020.
Ting Chen, Simon Kombllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look few-shot classification. In ICLR, 2018.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In CVPR, 2021.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). htti:::900033000 2019.
Debasmit Das, Sungrack Yun, and Fatih Porikli. Confess: A framework for single source cross- domain few-shot learning. In ICLR, 2021.
Yuqian Fu, Yanwei Fu, and Yu-Gang Jiang. Meta-fdmixup: Cross-domain few-shot learning guided by labeled target data. In ACM MM, 2021.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. In NeurlPS, 2020.
Yunhui Guo, Noel C Codella, Leonid Karlinsky, James V Codella, John R Smith, Kate Saenko, Tajana Rosing, and Rogerio Feris. A broader study of cross-domain few-shot learning. In ECCV, 2020.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, pp. 2217-2226. 2019.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. NeurlPS Workshop, 2015.
Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao. On feature decorrelation self-supervised learning. In ICCV, 2021.
Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, and Richard J Radke. Dynamic distillation network for cross-domain few-shot recognition with unla- beled data. In NewrIPS, 2021a.
Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, and Rogerio Feris. A broad study on the transferability of visual representations with contrastive learning. In ICCV, 2021b.
Neha Mukund Kalibhat, Kanika Narang, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Towards better understanding of self-supervised representations. In ICML Workshop. 2022.
Yuning Lu, Liangjian Wen, Jianzhuang Liu, Yajing Liu, and Xinmei Tian. Self-supervision can be good few-shot learner. In ECCV, 2022.
Xu Luo, Jing Xu, and Zenglin Xu. Channel importance matters in few-shot image classification. In ICMI, 2022.
11

---

Published as a conference paper at ICLR 2023
Puneet Mangla, Nupur Kumari, Abhishek Sinha, Mayank Singh, Balaji Krishnamurthy, and Vi- neeth N Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In WACV, 2020.
Mindspore. https//www..midssprreen.//
Sharada P Mohanty, David P Hughes, and Marcel Salathé. Using deep learning for image-based plant disease detection. Frontiers in plant science, pp. 1419-1432, 2016.
Cheng Perng Phoo and Bharath Hariharan. Self-training for few-shot transfer across extreme task differences. In ICLR, 2021.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the Ameri- can Statistical association, pp. 846-850, 1971.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitmets: Hints for thin deep nets. arXiv:1412.65500 2014.
Eleni Triantafillou, Hugo Larochelle, Richard Zemel, and Vincent Dumoulin. Learning universal template for few-shot dataset generalization. ICML, 2021.
Hung- Yu Tseng, Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. In ICLR, 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NeurlPS, 2016.
Haoqing Wang and Zhi-Hong Deng. Cross-domain few-shot classification via adversarial task aug- mentation. In IJCAI, 2021.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks weakly-supervise classification and localization of common thorax diseases. In CVPR, 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In ICLR, 2017.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In ICCV, 2019.
12

---

Published as a conference paper at ICLR 2023
A
APPENDIX
A.1 RESULTS OF THE DIFFERENT BATCH SIZES
We present the results of the SimCLR+CLDD+FD trained with different batch sizes on the EuroSAT dataset (5- (5-way 1-shot) in Figure 10. We can that the optimal batch size is 32, we utilize this value all the experiments of our method.
82.5
80.0
Accuracy (%)
77.5
75.0
70.0
16 32
64
128
256
Batch Size
Figure 10: Results of the SinCLR+CLD+FD with the different batch sizes on the EuroSAT dataset (5-way 1-shot).
A.2 RESULTS OF THE DIFFERENT PERCENTAGES OF UNLABELED IMAGES
Figure 11 shows that unlabeled target images become available, the accuracy of the model will gradually converge to plateau.
18
$4
76
72
10
50
Reserrages
Figure 1I: Results of the SimCLR+CLD+FD with the different percentages on the EuroSAT dataset (5-way I-shot).
A.3 RESULTS WITH SELF-SUPERVISED PRE-TRAINED TEACHER
Table 6: Results with self-supervised supervised pre-trained teacher.
EunSAT
CropDiscase
ISIC
ChesiX
I-shor
S-chot
L-shoe
5-chot
1-bbot
5-shot
1-shot
S-cbor
SimCLLR-C R-CLR4FD (eel---peerviso))
00.30-0.72
89 90.94-19.72
06.50---335
37.42-0.46
00:66-0.6
22.5K+0.45
25.96-0.43
SimCLR+CLR+FD (supervisod)
02.52-0076
92 02.8941.344
90.48.10.72
96.58.10.9
39.70.10.69
52.29-10.62
22.39.10.44
25.98-20.43
13

---

Published as a conference paper at ICLR 2023
On the EuroSAT dataset (5-way 1-shot), give the results of the SimCLR+FD+CLD where the teacher is pre-trained self-supervised scheme. The performances of the SimCLR+CLD+FD are shown in Table 6.
14


Reviews:
[{"review_id": "nZWQP-vz0k", "reviewer": "Reviewer_8cJ8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a cross-level distillation method for cross-domain few-shot learning. The are four stages in the pipeline. A pre-trained model is trained on the base classes data, then a student network is trained with cross-level distillation by mimicking the teacher network. A feature denoising step is further added to reduce overfitting of the features. The authors conduct extensive experiments on the BSCD-FSL benchmark to show the effectiveness of the proposed approach. ", "summary_of_the_review": "The paper proposes a novel method for cross-domain few-shot learning and extensive experiments show the effectiveness of the method. The main drawbacks of the method are the 1) introduction of excessive hyperparameters and 2) the computation complexity. \n\nPost-rebuttal:\n\nI have read the authors' rebuttal. It partially addresses my concerns. I tend to keep my rating but do not have an inclination to support the paper if other reviewers have concerns. ", "strength_and_weaknesses": "Strength: \n\n1. The writing of the paper is clear and the organization is also well structured.\n2. The idea of cross-level distillation is also interesting and novel. \n3. The authors also conducted extensive ablations to show the benefits of the proposed approach. \n\nWeaknesses:\n\n1. There are too many hyperparameters in the method. It may be difficult to tune the parameters for new applications.\n2. The introduction of the old student network may introduce additional overhead.\n\nThe authors may have some discussions on how to choose the hyperparameters and compare the computational complexity of the method with the baselines. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aGKR4mN8Di", "reviewer": "Reviewer_yRW4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of cross-domain few-shot learning, where the base dataset and the target domain are far away from each other. It builds upon past distillation-based frameworks, where unlabeled data from the target domain is used to distill a source teacher, pretrained on the base dataset. It introduces multiple modifications to this basic approach, including using distillation of multiple layers, a way of feature selection on the target problem, and an approach for avoiding inheriting the teacher's biases.\n", "summary_of_the_review": "Despite my short review, I find the contributions useful, the problem significant, and the result impressive. I am in favor of acceptance.", "strength_and_weaknesses": "Strengths:\n1. The problem of cross-domain few-shot learning is a practically significant problem.\n2. The results obtained by the proposed method are much more accurate than prior state-of-the-art.\n3. The paper has thorough ablations for its many contributions.\n\nWeaknesses:\n1. The contributions seem like engineering innovations rather than new insights. However, each contribution has well thought-out motivations, so I am less worried.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VIHbMWu922Y", "reviewer": "Reviewer_am4M", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to tackle the problem of cross-domain few-shot classification, where such problem scenario especially considers the few-shot learning under the case of having (large) domain-shift between base and target domains. In particular, with being motivated to address the large domain gap, this paper follows the prior works (i.e. STARTUP@ICLR2021 and Dynamic-Distillation@NeuraIPS2021) to include a specific assumption of having access to a small portion of unlabelled images in the target domain during the training stage. \n\nMoreover, the proposed method of this paper also follows the similar train of thoughts as the prior works: 1) having a teacher model trained on base dataset; 2) having self-supervised contrastive learning on the unlabeled target images; 3) the student for the target domain tries to learn from the teacher model by using the knowledge distillation (KD) technique, while the knowledge distillation strategy gradually changes from taking \"the teacher trained on base dataset\" as the main source of knowledge (similar to STARTUP) in the early iterations to taking \"the moving average of the updated teachers on previous iterations\" as the main source of knowledge (similar to Dynamic-Distillation) in the later iterations. \n\nIn addition to three aforementioned points which are similar to prior works, the proposed method in this paper has another two key designs: 4) the knowledge distillation implementation is cross-level (i.e. encouraging the shallow layers of student to learn from the deeper layers of teacher), in which the idea is similar to BYOT@ICCV2019 (the slight difference is that BYOT operates self-distillation); 5) due to the phenomenon stemmed from the self-supervised loss and discovered by previous works where the final feature vector for classification has a small number of strongly activated elements while the others are close to zero, the proposed method introduces the \"feature denoising\" technique to truncate the elements with small magnitudes to zero during the fine-tuning stage (i.e. updating the model to the target domain by using the given target-domain few-shots with labels). \n\nThe proposed method is experimentally evaluated on the BSCD-FSL benchmark and the comparison with respect to several baselines (including ATA and CI which tackle the cross-domain few-shot classification problem without access to target domain data during training, self-supervised methods, STARTUP, and Dynamic-Distillation) is made, where the proposed method is shown to provide superior performance. ", "summary_of_the_review": "Due to the limited/incremental novelty, potentially skeptical improvement with respect to baselines, as well as not-well-motivated and impractical problem setting (please refer to weaknesses for more details), I would suggest to reject this paper from ICLR2023 but is willing to change my rating if my concerns can be well addressed.  ", "strength_and_weaknesses": "Strength:\n+ The proposed method is able to outperform different types of baselines (including ATA and CI which tackle the cross-domain few-shot classification problem without access to target domain data during training, self-supervised methods, STARTUP, and Dynamic-Distillation, where the latter two are under the same problem scenario as the proposed method) in a challenging benchmark.\n+ The detailed investigations/ablation studies are performed to validate the model designs and hyper-parameter settings in the proposed method (e.g. the model structure for cross-level knowledge distillation, the hyperparameter $\\tau$ for building the old student, ablation study for feature denoising, the hyperparameter $\\lambda$ to balance self-supervised loss and knowledge distillation loss for training student model, and the hyperparameter $h$ for determining the number of truncation in feature denoising operation).\n+ The overall organization is clear and the writing is easy to follow.  \n\nWeaknesses:\n- Though I can understand that the large domain gap between base dataset and target domain is quite challenging to be tackled, but the assumption of having access to a portion of target domain data (even it is unlabeled) seems to be quite impractical and hard to envision its usage in real-world scenarios. Moreover, even there exists two prior works (i.e. STARTUP and Dynamic-Distillation) addressing the same problem setting as the proposed method, the authors should still provide sufficient motivations as well as the description on the potential usage of such problem setting in their paper to make the whole story self-contained and persuasive. \n(Besides, in this paper they set to have 20% of target domain data available during training, in which such amount is not \"a small portion\" in my perspective).\n- In addition to the teacher model trained on base dataset and kept fixed during knowledge distillation, the authors introduce another model, named as \"old student\" (which in implementation is the student model in the previous learning iteration), in order to keep the knowledge of historical students during the learning of student model. Though such design seems to provide benefit to the overall model performance, having such additional model actually increase the computational cost for the overall training procedure. Moreover, the proposed method adopts the linear combination of such old student model and the teacher model as the actual source of knowledge to execute the knowledge distillation, in which such old student model acts similarly to the moving average of historical students (i.e. an ensemble) if we expand the overall iterative procedure (thus being closely related to Dynamic-Distillation@NeuraIPS2021), hence the linear combination of the old student and the teacher becomes an \"ensemble of ensemble\" or analogously a straightforward integration over the distillation strategies of STARTUP and Dynamic-Distillation. Overall, such the distillation strategy of the proposed method seems to be incremental from the methodology perspective. \n- The cross-level distillation mechanism is closely related to the well-known BYOT@ICCV2019 work (where the shallow layers try to learn the knowledge from the deeper layers), in which the only difference exists on that BYOT is doing self-distillation while the proposed method applies the same idea across two networks. The cross-level distillation proposed in this paper thus seems to be incremental as well. \n- Though the experimental results show that the proposed method provide superior performance, there are some concerns as detailed below: 1) we see that the proposed method using BYOL as its self-supervised loss (denoted as BYOL+CLD+FD (ours) in Table.1) does not present significant performance improvement with respect to Dynamic-Distillation (especially the loss function in Dynamic-Distillation is closely related to BYOL); 2) the baseline SimCLR (self-supervised learning applied on unlabeled target domain data) already provides strong performance (e.g. better than BYOL+CLD+FD (ours) and even competitive with SimCLR+CLD+FD (ours) which also adopts SimCLR in its loss functions). From these two observations, the contribution of the proposed method becomes skeptical (in which the proposed method adopts many different designs into its framework but in results only achieves sightly better, comparable, or even worse performance with respect to the baselines). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster