Transfer Learning with Context-aware Feature Compensation
Authors: Long Bo
Su Yang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Transfer learning aims to reuse the learnt representations or subnetworks to a new domain with minimum effort for adaption. Here, the challenge lies in the mismatch between source domain and target domain, which is the major gap to be tackled by transfer learning. Hence, how to identify the mismatch between source and target domain becomes a critical problem. We propose an end-to-end framework to learn feature compensation for transfer learning with soft gating to decide whether and how much feature compensation is needed, accounting for the mismatch between source domain and target domain. To enable identifying the position of the input in reference to the overall data distribution of source domain, we perform clustering at first to figure out the data distribution in a compact form represented by cluster centers, and then use the similarities between the input and the cluster centers to describe the relative position of the input in reference to the cluster centers. This acts as the context to indicate whether and how much feature compensation is needed for the input to compensate for the mismatch between source domain and target domain. To approach that, we add only two subnetworks in the form of Multilayer Perceptron, one for computing the feature compensation and the other for soft gating the compensation, where both are computed based on the context. The experiments show that such minor change to backbone network can result in significant performance improvements compared with the baselines on some widely used benchmarks.

Under review as a conference paper at ICLR 2023
TRANSFER LEARNING VIA CONTEXT-AWARE FEA- TURE COMPENSATION
Anonymous authors Paper under double-blind review
ABSTRACT
Transfer learning aims to reuse the learnt representations or subnetworks a new domain with minimum effort for adaption. Here, the challenge lies in the mis- match between souree domain and target domain, which is the major gap be tackled by transfer learning. Hence, how to identify the mismatch between source and target domain becomes a critical problem. We propose an end-to-end frame- work to learn feature compensation for transfer learning with soft gating to de- cide whether and how much feature compensation is needed, accounting for the mismatch between source domain and target domain. To enable identifying the position the input in reference to the overall data distribution of source domain, we perform clustering first figure out the data distribution in compact form represented by cluster centers, and then use the similarities between the input and the cluster centers describe the relative position of the input in reference to the cluster centers. This acts as the context to indicate whether and how much fea- ture compensation is needed for the input to compensate for the mismatch between source domain and target domain. To approach that, we add only two subnetworks in the form of Multilayer Perceptron, one for computing the feature compensation and the other for soft gating the compensation, where both are computed based on the context. The experiments show that such minor change to backbone network can result in significant performance improvements compared with the baselines on some widely used benchmarks.
INTRODUCTION
Transfer learning aims to reuse the knowledge obtained from one domaín solve the problem in a new domain with little effort and minor change. A classical problem that needs intensive effort is pattern recognition, where training of a model consumes a lot of time and requires large number of data examples with annotations. If we can reuse the pre-trained model for another task, it leads to efficient and rapid development of a new solution. For example, a speech recognition model language can be transferred to recognize another language more easily by using transfer learning (Huang et all (2013). In addition, the knowledge learnt for visual navigation can be transferred new environment with ease through transfer learning (Al-Halah et al., 2022). So far, a lot of transfer learning methods have been proposed and it has been attracting much attention.
trained
In general, pattern recognition system is composed of feature extraction and classifier. In the con- text of deep neural networks with end-to-end learning, some portions of a network act as filters to perform feature extraction while the last layer is in general a fully-connected layer conduction classification. One solution for transfer learning is to share the feature extraction portion of the net- work and apply different classifiers for different tasks (Oquab et al_ 2014). When there is big gap between the source and target domain in terms of feature distribution, simply modifying classifier will fail to solve the mismatch between the two domains. Hence, another solution aims to identify reusable features or subnetworks performing feature extraction (Huang et al. 2013)). Yet, this relies on the assumption that there exist some coherent features that can be shared by different domains. In view of the distinction between source and target domain in terms of feature representation, an- other solution tries to make the features of two domains approach each other through representation learning, for example, applying certain regularization loss function (Zhong & Maki, 2020) domain classifier (Ajakan et all. 2014) to perform adversarial test on homogeneous or not. Even though, they still based such assumption that feature distributions of two domains can have

---

Under review as a conference paper at ICLR 2023
substantial overlap. However, this hypothesis might not always hold significantly in practice. Here- after, another solution to modify the network structure arises, for example, using an agent to search for reusable subnetworks to form new pipeline for target domain, where the agent could be im- plemented using neural networks (Guo et al., 2020 Liu et all 2021). However, these works change the backbone network greatly, which deviates from the origin of transfer learning, that is, solve new problem with off-the-shelf solution by paying minimum effort to make tiny change.
As aforementioned, the critical issue in terms of transfer learning to identify the reusable part of the existing learnt representations or subnetworks. In another word, due to the overlap between source domain and target domain, not every data example needs transferring, and how much trans- ferring is needed for a data example is subject to its position in the overall data distribution. If the data example of interest locates the overlap part between source domain and target domain, needs minor change. On the contrary, if the data example exists in feature space where the data distribution of source domain is significantly different that of target domain, needs significant change through transfer learning correspondingly. This gives rise to new problem, that is, the method to evaluate how much mismatch exists around the input between source domain and target well domain. This involves development of means to figure out the overall data distribution as a descriptor to locate the position of the input in reference to the overall data distribution, say, context. With sound context description in terms of figuring out the position of the input in regard to the overall data distribution, one can then proceed to evaluate whether and how much transfer- ring needed to compensate for the mismatch between source domain and target domain. For this sake, we propose a new method for context description: We apply clustering at first to represent the data distribution in a compact form of cluster centers, and then compute the similarities between the input and the cluster centers to define its location in the overall data distribution, which acts as a clue to lead subsequent computation on how much compensation needed to enhance the feature representation from backbone network. Based on such scheme for context description, we propose an end-to-end framework to learn feature compensation for transfer learning with soft gating to de- cide whether and how much feature compensation is needed, accounting for the mismatch between source domain and target domain. To approach that, we add only two subnetworks in the form of Multilayer Perceptron (MLP) backbone network, one for computing the feature compensation and the other for soft gating the compensation, where both are computed based on the context, that is, the feature compensation is context adaptive.
The contribution of this study is as follows:
1. A novel method is proposed for context-aware feature compensation, where similarities between the input and the cluster centers representing the overall data distribution is used the clue evaluate how much feature compensation is needed to solve the mismatch between source domain and target domain. The new architecture only incorporates two additional MLP modules, one for computing feature compensation and the other for soft gating the compensation. This meets almost all the requirements of transfer learning: Mi- nor network-leve change with two tiny components added, feature-level compensation with context awareness on whether and how much transferring is needed, and minimum effort to render a sound solution with big performance improvement;
2. The concept of context based on self- seff-positioning in reference to the anchors (cluster cen- ters) is proposed for figuring out the demanding degree in transferring the representation of each given example. Correspondingly, feature compensation is computed by incorporating the context into the existing representation while a soft gating is applied to highlight the degree that each data example needs transferring, where both the context-directed feature compensation and the soft gating are leant end to end;
3.
The experiments on data sets demonstrate the effectiveness as well as the advantages of our solution in comparison with the baselines.
2 RELATED WORK
Transfer learning aims to transfer the learnt knowledge from source domain to target domain (Thrun, (1955) to allow pre-trained models reused with minimum effort and good transfer- ability. The existing methods can be sorted into 4 categories (Tan et all. (2018): Instances- mapping-based addversarial-baset and network-based deep transfer learning.
2

---

Under review as a conference paper at ICLR 2023
Instances- based deep transfer learning aims to reuse the instances of source domain. For example, TrAdaBoost (Dai Wenyuan et all (007)) weights the instances of source domain to make the distri- bution similar to that of target domain, and perform model training based on the weighted instances from source domain along with those from target domain.
Mapping-based deep transfer learning maps the source domain and target domain to new space, in which they could be coherent to each other to some extent, and model is trained in the new space. A representative work in this direction is Transfer Component Analysis (Pan et al. 2010), and Tzeng et al. (Tzeng et al. 2014)) extents it to deep learning based solutions.
Adversarial-based deep transfer learning is based on Generative Adversarial Network (GAN) (Goodfellow et all 2014) to search the transferring solution from source to target domain. Domain Adversarial Neural Network (DANN) (Ajakan et al. 2014) introduces the idea of GAN transfer learning by applying a domain classifier to minimize the difference between source and target do- main in feature learning, that is, optimization on transferred feature representation is driven by not only classification but also removing the gap between source and target domain.
Network-based deep transfer learning reuses the model pre-trained on source domain, including architecture and parameters, to form part of the solution of target domain, since filters in convo- lutional layers act as feature extractors in fact and the feature embedding turned out from them are regarded generic to some extent. An early work for speech recognition (Huang et al. 2013) 3) parti- tions the neural networks of a solution into two parts: Language-irrelevan and language relevant, and reuse the language- irrelevant -nreeeeannreee part for other recognition tasks. Oquab et al. (Oquab et al 2014) reuses the Convolutional Neural Network (CNN) pre-trained on ImageNet with additional fully con- nected layers to adapt to target-domain classification. Another classical method, namely fine tune (Yosinski et all 20144), reuses the original network without any change on the structure and just modifies the parameter values with the data from target domain, where fine tune can be performed on the whole network, or a part of them after fixing the other part. There are two tracks to fur- ther improve (Yosinski et al. 2014):: Some methods (Xuhong et al., 2018: Li all, 2019: Zhu al. 2018: Zhong & Makil 2020) advance it by introducing new loss functions and the others (Guo et all 2020: Liu et al., 2021) achieve advance by optimizing the network structure. Fine tune is limited in that the difference between source domain and target domain is not taken into account. In view of that, LPPSP (Xuhong et al. 2018) and DELTA L et al. 201 99 apply regularization to loss function in order to compensate for such difference. GTN (Zhu et all 2018) holds in terms of feature selec- tion to weight features with gating neural network such that the common features shared by source domain and target domain can receive more attention. PtR (Zhong & Maki, 2020) aims to optimize feature learning by altering loss function to include regularization on the fly when the training is approaching a sound solution. Head2Toe (Evci et all (022) makes use of the representations from all layers and applies Lasso to obtain feature selection in their transfer learning framework. From another point of view, parameter modification on the original network (Xuhong et al 201 Li et al, 2019: Zhu et al., 2018: Zhong & Maki, (020) is not able to tackle all mismatches between source and target domain. Hereafter, AdaFilter (Guo et al. 2020) and TransTailor (Liu et al 2021) allevi- ate this by reorganizing the network architecture. As confirmed in (Iofinova et all (002). pruning the network into sparse ones in general leads to better discriminant for downstream tasks. AdaFilter applies Layer-wise Recurrent Gated Network to select reusable filters in original solution to form a new pipeline as target-domain solution. TransTailon introduces pruning to make network structure as concise as enough for target domain, which leads to optimal sub-model. However, they change the network substantially, which deviates a bit from the origin of transfer learning. that is. gain better discriminant with minor change and little effort. In contrast to &Guo et all 2020: Liu et all. 202 our method can approach this goal since we just add a tiny network to obtain enhanced fea- ture compensate for the feature from backbone network. Compared with the Pseudo label used in iZhong & Maki, (222)), we use the similarity between the data example of interest to all the cen- ters from clustering, which figures out the contextual situation regarding feature discriminant more comprchensively than a simple Pseudo label.
3

---

Under review as a conference paper at ICLR 2023
Backbone Network
Feature Extractor
Classifier
X
W.A
MLP
Compensation
W
Auto-Encoder (DEC)
Soft Gating
Context
Cosine Similarity
K Cluster Centers C=..,,Cc..... Gj)
Clustering (DEC)
MLP: Multilayer Perceptron; E: Concatenation; @: Scalar Product: Add; - Training Only
Figure 1: Context-directed Transfer Learning
METHOD
3.1 FRAMEWORK OF TRANSFER LEARNING
Our motivation is simple: For a given example, whether and how much the corresponding embed- ding should be changed to approach a new task is subject to its position in the data distribution. its position is closer to existing cluster center, in general, higher probability of correct classi- fication can be expected due to the explicit association with known data representatives. In such a case, the embedding should be changed little. On the contrary, the embedding should be remedied remarkably if its association to any cluster center is not so deterministic. Therefore, we first perform clustering training data to obtain the centers representing data distribution. Then, compute the similarity between the input data example and each of the centroids resulting from clustering figure out the position of the input in reference the cluster centers, where the cluster centers act as anchors to figure out data distribution as well as how far from the input to each an- chor. We denote the K cluster centroids as C (C1, C2...., Ck) and the input as X. Then, compute the cosine similarity between X and C1, C2..... Cx respectively, and the corresponding similarity is denoted as (S1, S Sk). Note that should undergo into-EEceodrr based representation learning prior to clustering and similarity computing, as required by the clustering module applied this study, namely, Deep Embedded Clustering (DEC) (Xie et all 2016).. These similarities are referred to as context since they figure out the relative position of X in regard to the cluster centers, based on which the compensation to the original feature embedding is computed. As shown in Figure [ the compensation to the original embedding is subject to two branches: The first branch appends the context vector to the original embedding X via concatenation, and then multilayer perceptron (MLP) is used transform X C.S into A MLP(X where MIP is designed to force has the same dimension as X, and ( denotes concatenation. The second branch computes the soft gating factor W M LL(SS so to obtain the weighted compensation subject to the context, that is, W.A, where W is a scalar. Following the aforementioned transfer network, the overall embedding fed classifirr becomes:
X W X + MLP(S)
The transfer network to produce W.A MLP(S).MMP(X(SS) is trained end to end. Once W = 0, the overall embedding in Eq. ① degrades to that of backbone network, and a higher W corresponds with a bigger transfer in terms of compensating the embedding. After applying transfer learning, the classification changes from classifier and y the class label resulting from the classifier.
o(X) to y = s(X + W.A), where & denotes
4

---

Under review as a conference paper at ICLR 2023
3.2 ANCHORS FOR CONTEXT DESCRIPTION
In Eq. ①. we concatenate X and S to produce the compensation (LP(((S)) for transferring feature embedding because S indicates how ambiguous X is in terms of being associate with de- terministic cluster, say, context. Moreover, the soft weighting M ML((S) aims to control the overall compensation adaptive to the context S. apparent that anchors resulting from clustering deter- mines the performance of the proposed solution since they are the base to figure out data distribution and aware of the contextual clue to produce compensation adaptively. Here, we make use of the al- gorithm referred to as Deep Embedded Clustering (DEC) (Xie et all, (016)) to obtain such anchor points, and the experiments confirm that DEC performs very well in the proposed framework.
3.3 WORKING PIPELINES
The proposed transfer learning framework work in 4 manners to form the following solutions:
(I) At first, fine tune the backbone network, and then fix the backbone and train the transfer network only.
(II) Train the backbone and transfer network as whole end to end.
(III) Fix the backbone network and just train the transfer network.
(I+) Except for the clustering, which is performed the whole training data without ran- dom sampling for Caltech 256 data, the other implementation is the same as (I)
EXPERIMENTS
4.1 DATA SETS
We use the following data sets for performance evaluation they widely used for evaluat- ing transfer learning algorithms, especially the baselines peered in this study sunch as TransTailor (Liu et al. 2021):
Caltech 256-30 & Caltech 256-60 (Griffin et all 2007) data sets contain 30,607 images of 256 objects. Due to the unbalanced data distribution for each category. the data are sampled at random to form balanced data for training with 30/60 data examples per class (Xuhong et al. 2018 Li et all 2019), referred to as Caltech 256-30/Caltech 256-60.
CUB-200 Wahh et al. 201 data set contains 11,788 images of 200 categories of birds.
MIT Indoor-67 (Quatooni Torralba, 2009) data set includes 67 indoor scenes with 80 training examples and 20 testing data examples per class.
Stanford Dogs (Khosla et al., 201 data set contains 20,580 images of 120 categories of dogs, where 12,000 are used for training and the rest 8,580 for testing.
4.2 BACKBONE
ResNet-50 (He et all, 2016). ResNet-101 (He all. 2016), and VGG-16 (Simonyan & Zisserman 2014) pre-trained on ImageNet- 1k (Deng et al. 2009) are used as backbone network, respectively.
4.3 BASELINES
TransTailor (Liu et al. 2021): Branch pruning performed iteratively to remove filters of less importance in order to approach optimal sub-model.
AdaFilter (Guo et all (020): Layer-wise Recurrent Gated Network is used to select the
pre-trained as well as fine-tuned filters reusable to form a new pipeline.
GTN İZhu all 2018): Gated Transfer Network used weight features from pre-
trained model, which works in the sense of feature selection.
PtR (Zhong & Maki, 2020): Pseudo-regression Task Loss is combined with Cross Entropy Loss regularization term once the training approaches sound solution, by which
5

---

Under review as a conference paper at ICLR 2023
Table 1: Classification precision (%)
compared with baselines 5 data sets
Backbone
Method
Caltech 256-30
Caltech 256-60
CUB-200
Stanford Dogs
MIT Indoor-67
TransTailor
85,3
87.3
80.7
91
78.2
PtR
84.5
87.2
79.2
I.2
81,5+0.2
85.3+0.22
81.4+10.2
79.6=t0.5*
S.P-PP
83.5+0.1
86.4+0.2
15.120.2
84.2#0.3*
ResNet101
DELTA
86.6
88.7
80.5
88.7
85.5+
Fine tune
82.65
85,91
83.7
91.17
80.82
Ours (I)
83.43
86.36
83.91
92.24
80.22
(II)
83.02
85.27
84.89
91.75
Ours (I+)
86.62
86.66
PtR
83.9
87.1
81.9
77.9
AdaFilter
80.62
84.31
82.44
77.53
GTN
79.95
77.77
ResNe150
78.44
86.72
DELTA BSS
78.63
86.01
78.85
87.18
StochNorm
79.58
Co-Tuning
81.24
Fine
79.94
83.04
83.83
89.79
79.55
Ours (I)
80.58
83.39
83,98
90.5
79.4
Ours (II)
81.09
83.73
83,93
90.92
79.62
Ours (I+)
83.32
83.96
TransTailor
76.4
81.8
79.2
84.2
76.5
PtR
75.1
73
GTN
72.14
71.19
Fine tune
54.06
59.14
76.57
75.5
72.91
VGG16
Ours (I)
52.61
58.47
76.51
75,81
72.23
Ours (II)
60.46
66.42
80.3
77.65
75.59
Ours (III)
66.46
70.66
64.84
83.39
71.34
Ours (I+)
63.37
66.28
feature learning takes into account not only the classification perspective but also the trans- ferability of features.
L2-SP (uhhong al. 2018 L2-SP penalty applied the parameters of pre-trained model in the form of regularization so as to enforce the weights hold sparsely.
DELTA (Li et al., 2019): DELTA characterizes the distance between source/target networks using their outer layer outputs, and incorporates such distance the regularization term of the loss function.
BSS (Chen al. 2019): Batch Spectral Shrinkage (BSS) inhibits negative transfer by sup- pressing the spectral components with small singular values that correspond to detrimental pre-trained knowledge.
StochNorm (Kou all 2020): Stochastic Normalization (StochNorm)
two-branch normalization architecture to avoid model over-depending on biased statistics.
Co-Tuning (You et al. 2020): The pre-trained model is fine-tuned using ground truth la- bels and probabilistic labels with learned relationship between source categories and target categories.
4.4 EXPERIMENTAL SETTING
All the experiments are conducted using Poroc,, running on NVIDIA GeForce RTXTM 3090 GPU for training. The pre-trained models of ResNet and VGG are provided by Timm (Wightman. 2019). Every image is resized to 224 x 224 resolution with every color channel normalized to zero mean. Training is based on SGD (stochastic gradient descent) with decaying rate being 0.005, momentum being 0.9, and learning rate being 0.0003. For our solution, the number of cluster centers is set to the true class number of the corresponding data set, which is 256, 200, 120, and 67 for Caltech 256, CUB-200, Stanford Dogs, and MIT Indoor-67, respectively.

---

Under review as a conference paper at ICLR 2023
4.5 PERFORMANCE COMPARISON
We list in Table D the performance comparison with baselines. Here, we compare these methods from two perspectives: (1) We compare different methods with a given backbone network: ResNet- 101. ResNet-50, and VGG-16, respectively, (2) We compare the overall performance across all backbone models.
ResNet-101: On CUB-200 and Stanford dog data, the proposed method outperforms the baselines. For CUB-200 data, the precision improvement is about 4% compared with TransTailor, which is significant.
For Caltech 256-30 data, only 30 data samples are randomly selected per class for training. This degrades our method much because we use clustering to select representatives but the random selec- tion prevents our method from finding the truly representative examples due the too sparse random sampling of 30 data examples per class in contrast to the 30,607 images in total. To verify this issue, we reestablish our solution (I) by training the clustering model with all the data samples other than the testing samples, which is referred to as (I+) in Table I Note that the backbone network is stilly trained using the randomly sample 30 data examples per class. Consequently, the performance goes up to 86.62%, which outperforms all the baselines.
For Caltech 256-60 data, DELTA performs best. For our solution (I+), however, its performance im- provement compared with that Caltech 256-30 is minor as the full training data based clustering has lead to the best extent to figure out the data distribution, and the minor improvement comes from the backbone trained with more training data.
For MIT Indoor-67 data, the target is scene classification, which differs much from the source do- main, say, object recognition. This might account for why the other methods fail outperform fine tune in this case. Note that we annotate 1,2, L2-SP and DELTA with in Table as they use Places 365 (Zhou et al., 2017) as source-domain data for pre-training which is also a scene classification data. As result, their performance seems more promising since the source domain and the target domain are focused on the same visual recognition task.
ResNet-50:
The result well as the conclusion is similar to that of ResNet-101 based test in regard our solution, and the exception is: PtR performs best on Caltech 256-30/60.
VGG-16: Our solution outperforms all baselines only on CUB-200 data, and is close to that of TransTailor on Stanford Dogs Data. For the other cases, the precision our model quite lower compared with the baselines. However, VGG-16 is not state-of-the-art model, not easy to train, and performance is remarkably inferior to that of ResNet- 101 according to Table I We see that all the best performances with ResNet-101.
Overall Comparison: Regarding all backbones, ResNet-101 leads to the best performance on each data set. The result listed in Table shows that model performs best and achieves remarkable performance improvement on Caltech 256-30, CUB-200 and Stanford Dogs Data. For Caltech 256-60 and MIT Indoor-67, the performance of our solution is acceptable. In overall sense, solution is technically sound and performs better in comparison with the state-of-the-ar methods.
Table 2: Classification precision (%) under different settings
W
Clustering
A
Caltech 256-30
Caltech 256-60
CUB-200
Stanford Dogs
MFT Indoor-67
MLP(S)
DEC
ML
83,02
85.27
84.89
91.75
80.52
DEC
MLP(X@8)
82.96
85.44
84.6
91.5
81.11
MLP(S)
Random
MLP(X(S
83.49
86.05
84.53
91.41
80.89
MLP(S)
K-Means
MLP(X@S)
84.04
85.93
85.07
91.39
80.29
MLP(S)
DEC
MLP(X)
83.24
85.78
84.58
91.63
80.82
4.6
ABLATION STUDY
We conduct ablation study as follows: According to Table the full model (solution II) outperforms the other settings CUB-200 and Stanford Dogs data. For MIT Indoor-67 data, it seems that the
7

---

Under review as a conference paper at ICLR 2023
1000
maximum coshee smilntt
0.
6.1
-02
0.50
0.91
0.92
0.53
0.94
0.55
0.96
0.57 0.98
soit gating factor
(b)
Figure 2: (a) Visualization of the data distribution of Stanford Dogs data via PCA. The heat value of each pixel corresponds with the value of the soft gating factor W in association with this pixel. (b) 2-dimensional histogram of maximum cosine similarity against soft gating factor (W).
embedding of every data sample should be transferred, where W=1 leads to the best performance. Since this data set is developed for scene classification with layout not fixed between objects, this causes the target domain totally different from the source domain. In such case, every data exam- ple undergoing transfer learning sound. For the Caltech 256 data, 30/60 data samples per class are randomly selected for training. Since so few data are too sparse to figure out fully the actual dis- tribution of the data, it is impossible for any clustering algorithm capture the truly representative centers. In such cases, randomly selected representatives even work hetter for Caltech 256-60 data as shown Table For Caltech 256-30 and CUB-200 data, clustering with K-Means leads the best performance while DEC performs best on the other two data sets. This means that clustering works our framework but which one is more suitable merits further studies. Yet, DEC fits the end-to-end learning well due to its neural network nature.
Table 3: Classification precision against number of cluster centers CUB-200 data
#Centers
200
300
400
500
600
700
%
84.89
83.98
84.69
84.48
84.74
84.08
Table Classification precision against feature dimensionality arising from DEC Auto-Encoder on CUB-200 data
Dim
100
200
300
400
500
600
700
84.89
84.32
84.27
84.19
84.24
84.41
84.79
4.7 SENSITIVITY
As shown in Table 3 the number of cluster centers does affect the performance much and the best one is 200. This means that the contextual description robust, not subject to the number of cluster centers. Further, Table shows that the performance change is minor with different feature dimensionality of DEC Auto-Encoder.
4.8 VISUALIZATION
We illustrated in Figure 2(a) the distribution of Stanford Dogs data, where the dimensionality of the feature space is reduced via Principal Component Analysis (PCA) and only the primary
8

---

Under review as a conference paper at ICLR 2023
components with biggest variations preserved. Here, the color of each pixel corresponds with the corresponding soft gating factor W. and a higher heat value W means bigger compensa- tion applied to transfer the feature embedding. We can see that bigger compensations applied to the data points locating in a self-contained region. This coincides with the principle that transfer learning is in general focused such data examples that incompatible with the target domain. Figure 2(b) shows the 2-dimensional histogram of maximum cosine similarity against soft gating factor W which shows such a trend that lower similarity corresponds with in general higher compensation level, say, higher soft gating. Since the maximum similarity to cluster centers indi- cates the membership degree the data example in association with deterministic cluster, and smaller value means higher ambiguous case in terms of membership association, it is thus natural to result in higher-level transferring for feature compensation. This fully exhibits the motivation of this study, that is, why we make use of clustering centers as the anchors to figure out the contextual clue in directing feature compensation.
mostly
4.9 NETWORK DETAILS
Either MLP in Figure 1 contains two-layer network. shows Encoder (DEC) in Figure [ Encoder/Decoder is composed of 4 linear layers, and the number of the neurons of each layer for the encoder is 1000,1000, 4000, and 100, respectively. Details of Auto-Encoder are listed in Table
shows the details. As for the Auto-
Network Details of MLP (K is the number of cluster centers and 2048 is the feature dimension of ResNet)
Table 5:
A= MLP(X(S)
W=MLP(S)
Layer
Layer
Layer
Layer 2
#Input
2048+K
2x(2048+K)
K
2xK
#Neurons
(20488KK)
2048
2xK
#Output
2x (2048KK
2048
2x
Activation
GELU
GELU
Sigmoid
Table 6: DEC Into-Encoder Details (2048 the feature dimension of ResNet)
Encoder
Decoder
#Input
#Neurons
#Output
#Input
#Neurons
#Output
Layer 2048
1000
1000
100
4000
4000
Layer
1000
1000
1000
4000
1000
1000
Layer 1000
4000
4000
1000
1000
1000
Layer
4000
100
100
1000
2048
2048
5 CONCLUSION
We propose a new transfer learning method with context-aware compensation to feature embedding, where a contextual descriptor figuring out how close from the input to each cluster center is applied, and soft gating is used to make the feature transfer focused on the data samples that truly need trans- ferring. The experiments demonstrate that the proposed feature transferring scheme is technically sound, and big performance improvement is achieved using a tiny network. The strong point pect as well as the weakness lies in the clustering bassed contextual description, which determines the overall performance. In the future, will investigate into contextual description schemes.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain-adversarial neural networks. arXiv preprint arXiv: 1412.444, 2014.
Ziad Al-Halah, Santhosh Kumar Ramakrishnan, and Kristen Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 17031-17041, 2022.
Xinyang Chen, Sinan Wang, Bo Fu, Mingsheng Long, and Jianmin Wang. Catastrophic forgetting meets negative transfer: Batch spectral shrinkage for safe transfer learning, Advances in Neural Information Processing Systems, 32, 2019.
Yang Qiang Dai Wenyuan, Xue Guirong, and Yu Yong. Boosting for transfer learning. In Proceed- ings of the 24th International Conference on Machine Learning, Corvallis, USA, pp. 193-200, 2007.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C Mozer. Head2toe:
Utilizing in- termediate representations for better transfer learning. In International Conference on Machine Learning. pp. 6009-6033, PMLR, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neurul information processing systems, 27, 2014.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Yunhui Guo, Yandong Li, Liqiang Wang, and Tajana Rosing. Adafilter: Adaptive filter fine-tuning for deep transfer learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4060-4066, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jui-Ting Huang, Jinyu Li, Dong Yu, Li Deng, and Yifan Gong, Cross-languaae knowledge transfer using multilingual deep neural network with shared hidden layers. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7304-7308 IEEE, 2013.
Eugenia Iofinova, Alexandra Peste, Mark Kurtz, and Dan Alistarh. How well do sparse imagenet models transfer? Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22266-12266 2022.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- grained image categorization: Stanford dogs. In Proc. CVPR workshop on fine- grained visual categorization (FGVC), volume 2. Citeseer, 2011.
Zhi Kou, Kaichao You, Mingsheng Long, and Jianmin Wang. Stochastic normalization. Advances in Neural Information Processing Systems, 33: 13:16304-16314, 2020.
Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, Zeyu Chen, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. arXiv preprint arXiv:1901.00 099229, 2019.
Bingyan Liu, Yifeng Cai, Yao Guo, and Xiangqun Chen. Transtailor: Pruning the pre-trained model for improved transfer learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8627-8634, 2021.
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid-level im- age representations using convolutional neural networks. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1717-1724, 2014.
10

---

Under review as a conference paper at ICLR 2023
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. IEEE transactions on neural networks, 22(2):199-22) -10, 2010.
2010.
Ariadna Quattoni and Antonio Torralba. Recognizing indoor scenes. In 2009 IEEE conference on computer vision and pattern recognition, pp. 413-420. IEEE, 2009.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409v 1556, 2014.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pp. 270-279. Springer, 2018.
Sebastian Thrun. Is learning the n-th thing any easier than learning the first? Advances in neural information processing systems, 8, 1995.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 1412.3474, 2014.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Ross Wightman. Pytorch image models. http:////t...t.t.....///////a//a 2019.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pp. 478-487, PMLR, 2016.
LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In International Conference Machine Learning, pp. 2825-2834, PMLR, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances neural information processing systems, 27, 2014.
Kaichao You, Zhi Kou, Mingsheng Long, and Jianmin Wang. Co-tuning for transfer learning. Ad- vances in Neural Information Processing Systems, 18:12366-12446 2020.
Yang Zhong and Atsuto Maki. Regularizing transfer learning with randomised regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13637-13646, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 4056)1445-11464 2017.
Yi Zhu, Jia Xue, and Shawn Newsam. Gated transfer network for transfer learning. In Asian Conference on Computer Vision, 354-369. Springer, 2018.


Reviews:
[{"review_id": "mnUZlaPca7m", "reviewer": "Reviewer_thJK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an end-to-end architecture from transfer learning that learns a feature compensation for the input data. For this purpose, a clustering of the data is performed on a pre-processing step and the similarities between an input example and each of the clusters are used as inputs to two MLPs that essentially decide i) the magnitude ($W$) of the transformation to apply to the given input example and ii) the direction ($\\Delta$) of that transformation. The method is validated on a set of benchmark datasets for transfer learning from ImageNet, using different backbone pre-trained networks. Different training strategies are compared.", "summary_of_the_review": "The paper lacks any theoretical guarantees. Moreover, as explained in 1. in the section \"Weaknesses\", I am not fully convinced by the appropriateness of the proposed method. The experimental results are not excellent either. All these issues justify my recommendation for rejection.", "strength_and_weaknesses": "Strengths:\n- The problem addressed by the paper is relevant.\n- The methodology is clear and well-described.\n- The experiments are fair and show that the method is competitive with SOTA in some cases.\n\nWeaknesses:\n1. In the introduction, the authors remark that \"due to the overlap between the source domain and the target domain, not every data example needs transferring, and how much transferring is needed for a data example is subject to its position in the overall data distribution\". Nonetheless, in their algorithm, clustering is performed using the target data. Thus, the computed similarities will not tell how far a given target example is from the source distribution. Therefore, I don't see how they are useful to assess \"how much transferring is needed\" for that example. When the source distribution is very different from the target, the method seems inappropriate. The poor results on MIT Indoor-67 reinforce this suspicion.\n2. Are the displacements $\\Delta$ normalized? Otherwise, the role of the predicted magnitude $W$ is redundant.\n3. The experimental results are not particularly convincing. In particular, it is unclear which of the training strategies proposed in section 3.3 is the best one. In addition, it would be interesting to visualize what kind of transformations the model is learning, i.e. if it's learning $\\Delta$s that pull representations towards the cluster centroids (or something else).\n4. A theoretical analysis of the proposed methodology is missing. It is presented as a mere heuristic without any theoretical guarantees.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bUfeTj2CyxF", "reviewer": "Reviewer_a21v", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an feature compensation method by two MLPs that computes the distance of data points to data centers. .The method is evaluated on a set of benchmark datasets for transfer learning from ImageNet, showing somewhat on par result using this technique.", "summary_of_the_review": "This paper is well written and it addresses a problem relevant to transfer learning. However, it is just an empirical work that lacks theory backing the idea, such that using distance to data clusters to do compensation as data transferring. Finally, the results are just on par with existing SOTA methods, not proving the effectiveness of this method.", "strength_and_weaknesses": "Strengths:\nthe paper is well written, methodology is well-described, and the problem addressed is relevant.\n\nWeaknesses.\nThe author is describing how to use data centers to do data compensation, however, how this is related to data transferring is poorly described. For examples, is data centers computed on target data set? how is its feature vector computed, using the pretrained network? using distance to get compensation factor seems only to be working under a strong assumption that this is used for classification, and it only works for image classification tasks? why using distance to centers work? all these questions are not answered properly.\n\nBased on the experimental setup, it is just finetuning the pretrained network using some downstream tasks. It just barely competes with SOTA methods, seems that with or without such 2 layers of compensation networks does not change the result at all.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "V_N2lQXhe6", "reviewer": "Reviewer_9GK7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for identifying the mismatch between a source domain and a target domain for transfer learning. The method relies on first identifying cluster centers using all the training data, and then comparing the current input data example to these center for determining how much compensation should be applied for the network to adapt to that particular example. This method requires to add two MLP network to the network architecture: one for compensation, and a second for soft gating this compensation. The experiments consider 5 benchmark datasets which are trained as downstream tasks of a backbone network trained on ImageNet-1k. The authors test several network architectures. The results show that the proposed method gains in terms of classification precision in some cases.", "summary_of_the_review": "As explained in the 'Strengths and Weaknesses' section, I think that the paper lacks a lot of technical details regarding the proposed method and key technical decisions. The method itself does not seem to related well with the original research question (i.e. the mismatch between a source and a target domain). Finally, the authors do not provide insights into the experimental results that explain in which scenarios would the method be applicable, which undermines the contribution of the paper. For these reasons, I recommend a reject.", "strength_and_weaknesses": "Strengths:\n- The paper tackles an important problem of identifying the mismatch between a source domain and a target domain before transfer.\n- The paper is, in general, well-aligned with literature in the area.\n\nWeaknesses:\n- The paper does not provide explanations / intuitions to most technical decisions. For instance, the proposed method is very quickly explained in roughly half a page (sections 3.1 - 3.2). For example, why is it reasonable to use DEC? Why is compensation necessary - what are the underlying factors that make it necessary once the closest cluster centers have been identified? Why is cosine similarity, and no other measure, applicable in this case? \n- It is very unclear to me how the proposed method is actually helping to identify the mismatch between a source domain and a target domain. In several parts of the paper, it is mentioned that the cluster centers are calculated on the full training set, and according to my understanding it is the whole training set of the target domain for which cluster centers are being calculated? I think it would greatly benefit the explanation of the method if it would clearly distinguish which parts of it come from the source domain and which from the target domain.\n- In the experiments, I do not see any reasoning as to what the proposed method works in some cases but not in others. In which cases doing this kind of clustering + compensation approach would work better? This kind of description is very important to really see the contribution of the proposed method to the problem of transfer learning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GXo8c7bjcs", "reviewer": "Reviewer_AuzT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how to effectively fine-tune a pre-trained network for a downstream task. The challenge here is to identify the mismatch between source and target distributions. To this end, this paper proposes context-aware feature compensation, which learns context information among the training data. They first perform clustering on the training data, which is used to learn cluster-to-data similarity matrix. Then, the similarity matrix is used to generate compensation embeddings using two MLPs. The compensation embeddings are then concatenated with the original training data. Experiments are performed on different fine-tuning datasets, with a broad comparison of recent fine-tuning methods.", "summary_of_the_review": "This paper aims to tackle transfer learning in context view. I find it a little novel, but the approach is not novel, with so many flaws, as listed in weakness section. Experimental results are not comparable, with missing references. Therefore, I tend to reject this paper.", "strength_and_weaknesses": "### Strength\n\n- The method is interesting. Studying the contextual relation between source and target domains is novel.\n- The related work section is sufficient. So much literature is surveyed.\n- Experimental datasets and comparison methods are representative.\n\n### Weakness\n\n- Although the idea of learning contextual relation is novel, this approach is not. It remains unclear whether clustering can model the context information, or if the results of clusters and original inputs can solve the mismatch between source and target domains in transfer learning.\n- The claim that \"this method only adds two MLPs to the backbone network\" is not correct. As can be seen from Figure 1, there are more than two MLPs: it still has an autoencoder (DEC), which is much more complicated than MLPs.\n- The method part is extremely confusing and not clear, which should be rewritten. I find figure 1 hard to understand, specifically how to compute $\\mathbf{X} \\textcopyright \\mathbf{S}$ and $\\mathbf{W} \\cdot \\Delta$. (I even think that should be $\\cdot$ instead of $.$, since you claim it's a product.)\n- The success of \"compensation\" heavily relies on DEC (deep embedding clustering). If DEC does not give the correct clustering results, the method will fail. How to justify this? Additionally, DEC brings extra computations, which should be highlighted in the paper.\n- This method will consume a lot of training time than comparison methods, according to Sec. 3.3. We see that there is more than one step of fine-tuning, which is more complicated than existing work. Authors should compare your training and inference time in the experiments.\n- In experiments, you should also list the number of training parameters of each method for a more fair study since this method introduces extra neural network modules.\n- More importantly, the performance on general fine-tuning tasks of this method is NOT good. According to table 1, This approach only succeeds using Resnet-101, while it does not generate competitive results in Resnet-50 and VGG16. Hence, it is difficult to say that this approach actually works in different backbones.\n- No experiments to explain why this method works. Lack of detailed analysis.\n- With regards to related work, I think this paper misses two very important references on the same topic:\n\n[1] Jang Y, Lee H, Hwang S J, et al. Learning what and where to transfer[C]//International Conference on Machine Learning. PMLR, 2019: 3030-3039.\n\n[2] Murugesan K, Sadashivaiah V, Luss R, et al. Auto-Transfer: Learning to Route Transferrable Representations[J]. arXiv preprint arXiv:2202.01011, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject