HiFE: Hierarchical Feature Ensemble Framework for Few-shot Hypotheses Adaptation
Authors: Yongfeng Zhong
Haoang Chi
Feng Liu
Xiao-Ming Wu
Bo Han
Venue: Accepted by TMLR ()

Abstract: The process of transferring knowledge from a source domain to a target domain in the absence of source data constitutes a formidable obstacle within the field of source-free domain adaptation, often termed hypothesis adaptation. Conventional methodologies have been dependent on a robustly trained (strong) source hypothesis to encapsulate the knowledge pertinent to the source domain. However, this strong hypothesis is prone to overfitting the source domain, resulting in diminished generalization performance when applied to the target domain. To mitigate this issue, we advocate for the augmentation of transferable source knowledge via the integration of multiple (weak) source models that are underfitting. Furthermore, we propose a novel architectural framework, designated as the Hierarchical Feature Ensemble (HiFE) framework for Few-Shot Hypotheses Adaptation, which amalgamates features from both the strong and intentionally underfit source models. Empirical evidence from our experiments indicates that these weaker models, while not optimal within the source domain context, contribute to an enhanced generalization capacity of the resultant model for the target domain. Moreover, the HiFE framework we introduce demonstrates superior performance, surpassing other leading baselines across a spectrum of few-shot hypothesis adaptation scenarios.

Published in Transactions on Machine Learning Research (09/2024)
HiFE: Hierarchical Feature Ensemble Framework for Few- Shot Hypotheses Adaptation
Yongfeng Zhong
byyhhong@coopp-poyyn.com
Deportment of Data Science and Artificial Intelligence The Hong Kong Polytechnic Universily
Haoang Chi
haoungchi6 & com
Intelligent Game and Decision Lab, Defense Innovation Institute National University of Defense Technology
Feng Liu
Computing and Information Systems University of Melbourne
Xiaoming Wu 1
siao-ming.an arring.arri ar Spolyu. edu.hk
Department Data Science and Artificial Intelligence The Hong Kong Polytechnic University
Bo Han
bhanml Siaaemm@@comp.hhkee hkbb.. edu.bk
Department of Computer Science Hong Kong Baptist University
Reviewed OpenReview: https: oppnnrvveee... net/ forum? id= B6RS6DNOG'
Abstract
Transferring knowledge from source domain to target domain in the absence of source data constitutes formidable obstacle within the field of sourco-free domain adaptation, often termed hypothesis adaptation. Conventional methodologies have depended on a ro- bustly trained (strong) source hypothesis to encapsulate the knowledge pertinent to the source domain. However, this strong hypothesis prone overfitting the source domain, resulting in diminished gencralization performance when applied to the target domain. To mitigate this issue, we advocate for the augmentation of transferable source knowledge via. the integration of multiple (weak) source models that underfitting, Furthermore, proposc novel architectural framework, designated as the Hicrarchical Feature Ensem- ble (HiFE) framework for Few-Shot Hypotheses Adaptation, which amalgamates features from both the strong and intentionally underfit source models. Empirical evidence from our experiments indicates that these weaker models, while not optimal within the source domain context, contribute an enhanced generalization capacity of the resultant model for the target domain. Moreover, the HiFE framework introduce demonstrates superior performance, surpassing other leading basclines across spectrum of few-shot hypothesis adaplation scenarios.
"Equal contribution. i "Corresponding author.

---

Published in Transactions on Machine Learning Research (09/2024)
(a)
Target samples
(b)
Target samples
Source
Data
Source
Data
N2
R2
(c)
(d)
Target samples
Target samples
Source
Ag
Data
Source
hg
Source Data
ng
Data
R2
h
Source
Data
Figure 1: Different approaches to improve the performance of Few-shot Hypothesis Adaptation (FHA). (a) Conventional FHA. (b) Enhancing target performance by increasing the number of target samples. (c) Typical ensemble approaches to improve target performance by increasing the number of source domains. (d) Improving target performance under resource constraints with only one source domain and a limited number of target samples by generating multiple source hypotheses from the given source domain.
1 Introduction
Domain adaptation (DA) Ben-David et al. 2010) refers to the study of leveraging labeled data in source domain (SD) to obtain a predicted model for a given lorgel domain (TD) where labels are insufficient or
unavailable. Conventional DA methods (Ahmed et al. 2021 Jiang et al. 2021 Kang et al. 2019 Sukhija ett al.) 2016 Wang et al., 2019) pose potential risk of exposing private information caused by accessing the source data. To mitigate this concern, recent studies have introduced source-free DA, also referred
hypothesis adoptation (HA) (Liang et al., 2020 T.i et al. 2020 Yang et al. 2021 Yi et al. 2023), which leverages source model to encode the knowledge from the SD rather than the source data. Recently, few- shot HA (FHA) (Chi et al. 2021: Yazdanpanah & Moradi 2022). which operates effectively in scenarios with limited labeled data from the TD, has emerged as an appealing approach to address data scarcity (Figure (a)). To improve the FHA performance, recent approaches have attompted to increase the size of target samples via data labeling or generation (Figure (b) or gather multiple source domains generate more source hypotheses (Figure (c) (Ahmed et al.) 2021 Shu et al. 2022 et al. 2024
However, the abovementioned approaches require additional effort to label the target data to collect hypotheses from different source domains. Moreover, these approaches rely best-performing strong source models from the SD, which may overfit the SD and subsequently perform worse the TD after adaptation. This overfitting issue has been demonstrated by pilot experiment in which models with varying accuracies from the digit dataset SVHN were adapted to the target task Mnist using a straightforward fine-tuning approach (details are provided in Appendix A). In this experiment, under-fitted weak source models may exhibit superior performance compared the strong model after adaptation (Figure 4.. This indicates that some weak hypotheses, although suboptimal for the SD, may contain underlying knowledge that could be beneficial for the TD. Inspired by this experiment, problem by generating multiple source hypotheses with varying source accuracies from single SD (Figure) (d)). Notably, the routine training of source models naturally produces a series of weak intermediate models that are often overlooked and discarded. These models are easily obtainable, as reaching out to the source provider to acquire additional weak models incurs minimal additional cost.
propose addressing the FHA
A new problem: FHAW. Thus, we propose to study new problem called few- shot hypotheses adoptation with weak models (FHAW). Unlike previous single-source FHA approaches (Chi et al. 2021 Liang et. al. 2020
2

---

Published in Transactions on Machine Learning Research (09/2024)
that depend a single strong hypothesis, FHAW aims to leverage multiple source hypotheses with varying degrees of accuracy from a single SD to enhance the diversity of source models. While some researchers have advocated for utilizing multi-source hypotheses for НА (Ahmed et al. 2021 Shu et al. 2022; Li et al. 2024), these ensemble methods typically rely on multiple models originating from different source domains, each contributing its best-performing hypothesis to the ensemble (Figure | (b)). This approach becomes ineffective when dealing with weak hypotheses that have low source accuracies. FHAW addresses a more challenging scenario where access to multiple source domains is not feasible, offering a new perspective on tackling FHA challenges.
A viable approach to solve FHAW. To mitigate the potential negative transfer arising from weak source models and extract valuable source knowledge for the target task, we have developed a new and efficient framework called the Hierarchical Feature Ensemble framework (HiFE) to address FHAW, as illustrated in Figure 2 The HiFE method employs hicrarchical ensemble techniques to enhance the representativeness of intermediate features. It utilizes weighted residual units (WRU) to aggregate features induced by the source hypotheses, as illustrated in Figure 2 (b). WRU merges similar features with skip connections to reduce the risk of forgetting the source knowledge, alleviating the overfitting problem when fine-tuning with few-shot samples. Furthermore, we incorporate feature decorrelation learning (DcCL) by integrating a correlation penalty term into the standard classification loss, thereby enhancing the diversity of intermediate features (refer to Figure 2 (c)). Comprehensive results indicate that HiFE delivers state-of-the-art (SOTA) performance across various domain adaptation tasks.
Main contributions. Our contributions are three-fold:
To the best of our knowledge, this is the initial investigation into the FHAW problem. FHAW holds practical relevance in numerous privale data-based scenarios, as source providers are inclined to offer 'redundant" weak models instead of disclosing sensitive datasets. Our work introduces a fresh perspective to encode the source knowledge in the absence of the source data.
We propose a new framework to aggregate all source hypotheses at the feature level to address the FHAW problem. We are the first to apply a hierarchical ensemble at the feature level in hypotheses adaptation. We effectively alleviate the over-fitting problem by the design of WRU and improve the generalization of the final hypothesis by incorporating feature DeCL loss under the few-shot sctting.
The comprehensive evaluation of the proposed HiFE methodology, conducted over an array of benchmark datasets including Mnist, SVHN, USPS, CIFAR-10, STL-10, Amazon, DSLR, Webcam, and VisDA-C- has established that our approach achieves performance on par with or exceeding current SOTA methods in various FHA tasks. Notably, as detailed in Table 2 the HiFE method surpasses the SOTA by an average accuracy of 4.3% in the digit dataset task USPS + Mnist. Similarly, in the task of adapting DSLR to Webcam datasets, as shown in Table 3 HiFE outperforms the SOTA by 3.6% in accuracy.
2
Related Work
This section presents brief overview of the literature about traditional domain adaptation, hypothesis adaptation, multi-hypotheses adaptation, and ensemble methods for hypothesis adaptation.
Domain adaptation (DA). Traditional DA is a subfield of machine learning that focuses on learning a hypothesis for TD when labeled data is insufficient or unavailable by leveraging labeled data from an SD. Numerous DA methods have been proposed for various tasks such as object classification (Liang et al. 2018). object detection (Hsu et al. 2020), and semantic segmentation (Zou et al.) 2018). Existing approaches for DA can mainly be categorized into two classes: feature- based DA and instance- based DA. The former aims to learn a domain-invarian representation by minimizing the domain discrepancy in shared space (Kang et al. 2019 Long et al. (017). For example, Gradually Vanishing Bridge (Cui et al. 2020) uses bi-directiona invariant representations. The latter minimizes the discrepancy by re-weighting the source samples for better training. Despite the success achieved by these mothods, they require access to source data during the learning process, which incurs significant costs in terms of data transfer and storage well as risks related to personal information leakage.
generation to learn
3

---

Published in Transactions on Machine Learning Research (09/2024)
Hypothesis adaptation (HA). Researchers have started exploring source-free domain adaptation (SFDA); namely HA, to mitigate the issues arising from accessing source data. Early works addressed the problem by fine-tuning the source hypothesis on the target data (Girshick et al. 2014). However, recent studies have delved into unsupervised DA to investigate the limitations of this straightforward strategy (Ding ct al. 2022
Liang et al. 2020 Yang el. al. 2022 Yi et al., 2023). Among these methods, SHOT (Liang et al. proposes a representation learning framework to update the feature extractor through information maximization and self-supervised pseudo-labeling loss. In this framework, pseudo-labels of the target data are refined using the nearest centroids. Similarly, eYi al. 2023 views SFDA as the problem of learning with label noise and suggests exploiting the early-time training phenomenon to tackle the issue of pseudo-labels. Notably, these methods rely on large amount of unlabeled data from the TD to purify the pseudo-labels. On the other hand, TOHAN (Chi et al., 2021) is the first study to explore the HA under a few-shol selting. It proposes generating an intermediate domain that is compatible with the TD to facilitate transfer learning. Many previous works rely on strong source hypothesis for adaptation, which may not always be the most suitable one for adapling a specific TD.
2020
Multi-hyyooteess adaptation (MHA). MHA extends the НA paradigm by integrating knowledge from source hypotheses from multiple domains. To tackle this, model selection methods (Nguyen et al. 2020 You et al. 2021 have been developed to estimate the transferability of each pre-trained hypothesis. However, the single selected hypothesis may not be able to carry the rich knowledge encapsulated in all of the source hypotheses. Thus, some researchers have turned to parameter ensemble methods (Ahmed et al. 2021 Rusu et al. 2016 Shu et al. 2022; Li et al., 2024). Yet, these approaches often require significant amounts of unlabeled target data be effective, making them less tenable in the FHA setting. Moreover, these approaches operate under the assumption that each source hypothesis is strong
from the corresponding SD, rendering them ineffective when presented with weak hypotheses. Besides, these approaches require accessing multiple source domains related to the TD, which is often not feasible. Our research focuses on a more practical and challenging scenario: only one SD is available.
Ensemble methods for HA. Ensemble methods are prominent research in machine learning (Dietterich 2000 Dong et al., 2020 Sagi & Rokach 2018 Eilers al.] (022). These methods have demonstrated that combining multiple hypotheses is advantagcous over a single hypothesis in classification and regression problems. However, traditional ensemble methods rely on weighted voting for the final decision and lack the capability of representation learning (Cao et al. 2012). Thus, some researchers have proposed feature-level ensembling. Studies have demonstrated the effectiveness of hierarchical feature representation in improving classification accuracy (Cai et al.) 2018 Su et al.) 2009). In the area of FHA, research on developing hierarchical feature-level ensemble methods to derive a comprehensive knowledge representation of all source hypothesos has been limited.
3
Few-Shot Hypotheses Adaptation with Weak Models
3.1 Problem Definition
We address the problem of few-shot hypotheses adaptation with weak models, where several pre-trained source hypotheses, including one strong and some weak hypotheses, are given. Let A' c Rd be an input space and y = (1...., CJ be the label space, where C' is the number of classes. To formalize the problem clearly, some definitions are presented as follows.
Definition 1. (Expected and empirical risk). Given a data distribution P over X X),, let H (h: X - }} be the hypothesis space and h E H with the parameter 0 E O, then the expected and empirical risks are defined as
L(0) (xxy)~ [((0,, v)],
L(, D) = Σ (E(0, x4, v)),
1
1=1
where l is proper loss function and D (1r yi) pn denotes the i.idd observations.
4

---

Published in Transactions on Machine Learning Research (09/2024)
Definition 2. (Strong Hypothesis). Given a set of hypotheses H {hT" J from domain S and validation
set Dval: where m is the hypothesis TD) and M is the number of hypothesis, a hypothesis h, E H is called a strong hypothesis if vh E H. L(On,, Dvai) S [(0,, Dvai).
Definition 3. (Weak Hypothesis). Given a set of hypotheses H = { from domain S and validation set Dval. where m is the hypothesis ID and M is the number of hypothesis, a hypothesis h E H is called a weak hypothesis if 3he E H, T(OO,, Drai) < [(O,,,, Paa))-
Problem 1. Shot Hypotheses Adaptation with Weak Models (FHAW)), Given a set of hypotheses amm
H with a strong hypothesis h2 and M weak hypothescs {hm Im=1 trained on the SD Ps(X,Y), nr target labeled data D, = ((1), ) that i.i.d. drawn from Pr(X, Y) with n4 & As and Ps(X, & Px Y), M FHAW is learn a target hypothesis hy X + y with hag (hw I==1 and Dr to minimize the expected risk on the TD.
Comparison with FHA. FHA involves the use of strong hypothesis derived from the SD. However, such hypothesis is prone to over-fitting on the SD, and their generalizability towards the TD can be limited. To address this challenge, we introduce FHAW by leveraging multiple weak hypotheses to facilitate more effective adaptation. These weak hypotheses can be easily obtained by saving model snapshots during the training of the strong source model with minimal additional cost.
3.2
Addressing FHAW in Principle
We will present a theoretical view based on the PAC-Bayesian framework (Germain et al. 2009 McAllester 1999 Masegosa, 2020) to demonstrate why we propose to incorporate multiple weak hypotheses for FHA and why our HiFE, Tramework works. Tn the PAC-Bayesian framework, each hypothesis hg has prior knowledge of the hypothesis space O, and this prior distribution π updated to a posterior distribution p after feeding samples D to ha- In FHAW, multiple models (ho,), are given with 0 E O, 0 {6, JMM and M p(0) = II.- (0.). For a given sample (a, y), we apply the cross-entropy loss E(0, xr y) log p(y| 0). A bound thcorem proposod by Deng et oor the model ensemble restated below, and some other previous related theorems are shown in Appendix C
Theorem 1. (Model ensemble ertor bound (Deng et al., 2023)). Given a data distribution P over X x y, a set of model parameters [ with associated prior ( where ne is defined over O₂ with n, (0,) N(0, I)1,, a 6E E (0, 1], a real number e >> O, and p,(O.) is a Dirac-delta distribution centered around 62 with pe(O.)) 56g (0:), then we have that the (pp0)(((0)) is upper bounded by
M (L
Σ
1 2eno
0 11?
log(2n0?)) (o10), D) enL' 2en
d:
M
where (p((), D) is the empirical version of a variance term V(p(O)), which is defined as
M
M
27
Ept(r B(
p(ylor,))) - M
1
Σ
plyle, 0k)
2Mmaxop 0)2
1=1
and e is defined
on Eii W
e
log
6
In Theorem the variance term (p((), D) measures the diversity of all models Masegosa 2020). If there exists an input sample such that he (x) + he, (æ), then we have (p((), D) > 0. Therefore, in the setting of FHAW, adding weak hypotheses increases the diversity of the source models and provides opportunities to decrease this error bound. Minimizing the first term of the error bound in Theorem D is equivalent to finding 0 = [M by ming ΣM,, (L,0,, D) + A₁ || 0g 1|2 +x2dk) /M, where de is the dimension of 0; and A1
X > are paper-parameters. Based on this formula, we propose a hicrarchical feature ensemble module reduce the dimensionality of features.
5

---

Published in Transactions on Machine Learning Research (09/2024)
Source Encoders
Hierarchical Feature Ensemble Module
Layer
Fe
Encoder
WRU
Layer L-1
WRU
WRU
Layer WRU
X
Classifier
CLs
No Sw
WRU
Encoder
Ct
Encoder
DDcL
(b)
Weighted Residual Unit (WRU)
Decrrelation Learning (DeCL)
Layer L
fi
WRU
27
Sout
Pairwise feature conrllatin minimization
C
WRU
Orthogonal
fa
WRU
Weighted
Concutenation
Fully connected layer
x
The architecture of HiFE framework. Each source hypothesis consists of feature encoder and classifier. We train model with hierarchical feature ensemble module to merge features from all source encoders. In this module, features are grouped according to the cosine similarity, and WRU merges the grouped features to generate new features for the next layer. Each WRU leverages skip connections to avoid over-fitting. Besides, we apply the decorrelation learning (DeCL) strategy by adding a correction penalty term to the loss function encourage feature diversity. The target classifier ct, initializing with the average of all the source classitiers, is fixed during the training. Only the parameters of the strong encoder gs and the hicrarchical feature ensemble module are updated.
Figure 2:
4 Few-shot Hypotheses Adaptation via Hierarchical Feature Ensemble
To aggregate knowledge from both strong and weak source hypotheses, we introduce the HiFE framework, depicted in Figure 2 HiFE hierarchically merges features induced by all the source hypotheses. We assume each source hepothesis has been embedded with specific discriminative knowledge about the SD. Hence, during the aggregation, independent possible at cach level to increase the reprosentative power of the intermediate features. We describe the design insights of HiFE in Section 4.1 and Section 4.2
use feature de-correction learning module, making the features as mutually
4.1 Hierarchical Feature Ensemble
Ensemble learning is widely recognized as an effective approach for combining multiple learning methods and improving overall performance (Beven & Binley] 1992 Kuczera & Parent 1998). While ensemble methods have been utilized for HA in past rescarch (Ahmed al. 2021). ensemble learning at the feature representation level has received relatively less attention. However, prior research has shown that hierarchical feature representations ensemble-base basdbl-b-bsedd approach for FHAW to leverage such benefits. Specifically, our method involves merging source features that contain knowledge of the SD using a hierurchical feature ensemble module before feeding them to the final classifier.
significantly enhance classification accuracy. We propose hierarchical feature
To simplify the feature extraction process with the source hypotheses, we follow Mootian et al. 2017 and
(Ahmed al. 2021 to decompose each hypothesis into two modules: a feature encoder g X E and
classifier Rd RC where d denotes the dimension of the output feature. Thus, have h(x) c(g(x)),

---

Published in Transactions on Machine Learning Research (09/2024)
Le., h = = cog. g. In our problem setting, source hypotheses are decomposed to C, o 9s and {c I J Features induced by 9s, [0") m==1 are fed into the feature ensemble module. Tet xt denote the i-th feature at layer 1 (1 E {0, (0.1,..,,)) and {=)) be the No imput features. The hierarchical feature ensemble module aims to aggregate all these features into one single feature 2t, through a hierarchical method so that =t, contains as much source knowledge as possible. To this end, we must tackle two questions: 1) which features to merge and 2) how to merge the chosen features.
m==
1) Which features to merge? According to the Gestalt principles of psychology (Koffka 2013), humans tend to group similar information during cognitive processing. Taking inspiration from this, we utilize feature similarity as metric to group similar input features together from the previous layer for the purpose of merging. Given set of Ni features, we first create a similarity matrix S E RNX,, where Suy cos(2f, 2f) is the cosine similarity of features of and zं. Next, we repcatedly choose two or more features with the highest similarity and merge them into a new feature for the next layer. The merging is done only bet features from the previous layer. Such a hierarchical merge process repeats layer by layer until only one output feature is left.
2) How to merge features? In the FHA setting, the small sample size problem limits the feasibility of maintaining adequate validation sets to assess performance before testing unknown samples. Without such validation sets, optimizing the model could cause over-fitting to the limited target data, leading to a local optimum and performance degradation (Goodfellow et al.) 2014 Kirkpatrick et al.) 2017). To address this issue, we propose the Weighted Residual Unit (WRU), adding the "shorteut connections" of the input features to the block output after feature merging (see Figure 2 (b))- The shorteut connections allow the upper layer's features to be directly sent to the next layer, maintaining the source knowledge during adaptation and alleviating the over-fitting problem. Within each WRU, the input features {f are concatenated and fed into a fully connected (FC) layer, where K is the number of input features. The output of the FC layer is denoted as F Wre) with Wre be the learnable parameters of this layer. Unlike the shortcut connections that perform identity mapping in ResNet (He et al., 2016), we perform a weighted element- addition of { and FC( WFF) to balance the influence of different input features induced by the source hypotheses. The function of WRU can be formalized as follows.
K Σ oz Si
WRU((A) = o0 FC(LA) WFC) +
(1)
where {a; are the learnable weights. We add the batch normalization and ReLu layers after the weighted sum for better performance. If the dimension of fo is not equal to that of the output of FC, we can make linear projection of Sa by extending og to square matrice W, to match the dimension. The application of WRU allows us to preserve some source knowledge and learn new information from the target samples simultaneously.
In our approach, we follow the aforementioned principle to determine which features to merge and then use the WRU to merge the features based on the first batch of input data. Once we pass the first batch of input data, the merging network is built. This merging network then remains fixed, and all subsequent samples share the same merging network. Such a setting ensures stable fine- fine-tuning on target samples.
4.2 Decorrelation Learning
It has been commonly agreed that diversity is a success factor of ensemble algorithms. Different opinions from multiple classifiers are expected to reduce the generalization error. Traditional decorrelation learning (DeCL) methods encourage diversity explicitly by adding correlation penalty term to the final error function (Liu & Yao 1999: Shi et al. 2018 Wang et al.) 2010). When it comes to feature ensemble, learning the features with good discriminative power is also essential for various high-level vision tasks (Wen el. al. 2016 Cheng et al. 2018). To promote the learning of features widely distributed across the feature space and embed various forms of source knowledge, we apply DeCL in the fcature space to encourage independence between features in each layer. In this regard, we introduce a cosine similarity penalty to decrease feature correlation and encourage feature diversity (see Figure (c)). Specifically, we calculate the pairwise square values of
7

---

Published in Transactions on Machine Learning Research (09/2024)
cosine similaritics for all features in the same layer and sum them up from all layers. The corresponding feature DeCL, loss is defined as
1. N₁ ---
N₂ cos cos(z) x/)²
DDCLL Σ
(2)
j=++1
where Ni is the number of features at layer and cos(z),x/) (of 2/)([[=[l| |=2/1). enable the adaptation of the ensemble network to the TD, we incorporate the knowledge of TD by fitting the network to the labeled target data. To accomplish this, we adopt the standard cross-entropy loss, which is defined as follows,
Furthermore, to
LCts = B( (CE(cr(A(zs))
(3)
where CE(() denotes the cross-entropy loss and A(x:) refers the output of the feature ensemble module when fed Te to the source encoders. To summarize, we train the ensemble network using joint supervision that combines the target supervised loss (Fquation (3)) and a feature DeCL, penalty term (Fquation (2) with a hyper-parameter 3 to trade off the two aspects (Equation ( The target supervised loss guides the network in learning the knowledge from the larget samples, while the feature DeCL loss promoles mutual independence amongst features in each layer, thereby increasing the feature diversity and preserving the distinct discriminative knowledge of each source hypothesis.
[(3) = (1- - B) LOLS + B- DDCLL
(4)
5 Experiments
5.1 Experimental Setup
Datasets. We conduct experiments various standard DA benchmarks to evaluate our approact
I
Digits. We choose three-digit datasets, i.e., Mnist (M), USPS (U), and SVHN (S) for our experiments.
Following Motiaan et al. 2017 Chi et al. 2021 we experiment with different numbers of target samples from 7 per class.
Office. We use three domains of the office datasets (Saenko et al. 2010 Amazon (A), DSLR (D), and Web- cam (W). Each domain contains 31 object classes in the office environment. We conduct several experiments with different numbers of target samples per class ranging from 1 to 5.
Image classification. We use two image classification benchmarks CIFAR-10 (CF) (Krizhevsky) 2009 and STL-10 (ST) (Coates et al. 2011). Each benchmark consists of 10 classes of objects, and nine classes are overlapped. We remove the non-overlapped classes ("monkey" and "frog") and reduce the tasks to a 9-class classification problem following the procedure in (Shu al., 2018). As the domains are more complex than digits, we increase the number of larget samples to 15 and 20 for each class.
VisDA-C. VisDA-C (Peng et al. 2017) is demanding large-scale benchmark designed primarily for the 12-class synthesis-to-rcal object recognition task. The source domain comprisos 152,000 synthetic images created by rendering 3D models from different angles and with different lighting conditions, whereas the target domain includes 72,000 real object images sourced from Microsoft COCO. We randomly choose 10% of the target data set (7200 images) as the testing set. Since the domain gap between the synthesis and real-object images is large, we experiment on a larger number (10, 30, and 50) of the target samples.
Baseline methods. In the context of the novel FHAW problem setting, establish our baseline compar- isons by adapting and refining several established approaches in the field. We conducted comprehensive evaluation against four existing methods for HA and their respective variations. Initially, SHOT (Liang et al. 2020), a hepothesis transfer learning framework tailored for unsupervised HA, served as a foundation. In our
The full code available https://githul: http:/tttw..ccl/////o/////te.e.

---

Published in Transactions on Machine Learning Research (09/2024)
study, we preserved its model adaptation module, tweaking it to leverage labeled target data to support su- pervised HA, aligning it seamlessly with our experimental setup. The performance outcomes from employing solely the strong hypothesis with SHOT are denoted as SHOTTssrong. The subsequent contender, TOHAN (Chi ct al. 2021 specifically tackles the FHA challenge. Both SHOT and TOHAN are engincered to adapt a singular hepot to the TD independently. To evaluate against our mottishy adaptation ap- proach, we extended these methods to SHOT-ens and TOHAN-en by employing a straightforward ensemble technique, following the methodology outlined in (Ahmed et al. 2021). Furthermore, we included the models DECISION (Ahmed et al., 2021) and Bi-ATEN (Li et al. 2024), specifically designed for multi-source-free insupervised HA, to contrast with our single-source multi-model HA strategy.
Network architecture. For digit recognition tasks, we employ the same architectures utilized in SHOT [Liang et al. 2020), namely using the LeNet-5 (LeCun et al. 1998) for Mnist, USPS, and a modified version of LeNet for the slightly more complex SVHN dataset. For the image classification tasks, we adopt ResNet- 18, ResNet-50, and ResNet-101 ((He et al., 2016)) as the backbones for the CIFAR-10/STI-10, office, and VisDA-C datasets, respectively.
Source hypotheses preparation. We train single optimal hypothesis strong source hypothesis for each SD and save seven intermediate snapshots as weak source hypotheses with varying accuracy levels. To acquire the heset with different source accuracies, we first set an accuracy range [acomin, accs], where acCmin is a preset value around at 40-60% and acc, is an estimation of the accuracy of the strong hypothesis. Then, we split this range into several uniform intervals and save one model snapshot at each interval to get weak hypotheses for each SD. The source data can be discarded after getting all the required source hypotheses. Section 5.1 shows the source models generated with the source dataset Mnist and their corresponding accuracy ranges, We generate 12 source models (1 According to our definition in Section 3 the first 11 models { are weak souree hypotheses, while the last one h12 is the strong hypothesis. Among these models, we used {/ili2s} (8 models) in the experiments shown in Table / and Table 8 while all models are prepared for ablation studics. The process of training the target ensemble hypothesis with HiFE is detailed in Appendix
MI
MI
AC
MI
AC
MI
hn
140,
45)
[55, 60)
hr
770, 75)
85, 90)
hz
45, 50)
b5
60, 65)
hs
75. 80)
h11
190, 95)
hg
[0, 55)
bg
665, 70)
hg
[80, 85)
[95, 100)
Table 1: An example demonstrating the generation of source hypotheses using the Mnist dataset. We first set an accuracy range of [40%, 100%). This range is then divided into 12 uniform intervals. During training on the Mnist training set, we save one model snapshot at each interval to obtain the corresponding hypotheses.
5.2 Result Analysis
Results of digit classification tasks. We evaluate the effectiveness of our approach on six closed-set adaptation tasks for digit classification. These tasks are by pairwise combinations of the three domains S. M, and U. We report the results of three tasks in Table 2 (more results can be found in Appendix D). Firstly, as shown in Table 2 there exist some weak heset that can perform better than the strong hypothesis after adaptation (see the comparison of SHOT-best and SHOT-strong), supporting our motivation of adopting the weak hypotheses. Moreover, incorporating the weak hypotheses allows our proposed HiFE to outperform SHOT-strong, For instance, compared with the average accuracy of SHOT-strong (77.3%), HiFE leads higher average accuracy (88.7%) in the task S + M. Additionally, despite some weak hypotheses with bad adaptation performance (see SHOT- HiFE can largely avoid the severe negative transfer and achieve the best performance than previous ensemble approaches. For example, HiFE outperforms the SOTA (TOHAN-ens) by 4.3% in the average accuracy of U + M task. Notably, we can observe that our approach not only maintains competitive accuracy but also decreases the standard deviation (std) of accuracy across different target samples. This reduction in std is paramount as indicates a more consistent and reliable performance of our HiFE.
Results of office object classification tasks.
We show the results of three closet-set adaptation tasks
9

---

Published in Transactions on Machine Learning Research (09/2024)
hesot Number
Number of Target Data per Class
Tasks
Method
Avg
1
2
3
4
SHOTT-oort
42.1#1.2 44.3:0.88
49.911.0
48.4=1.6
50.740.6
50.9+1.1
50.9:0..8
48.2
Single
SHOT-best
92.1#1.5
93.4+1.2
93.7+0.9
93.6=1.0
93.7+1.5
93.5+0.8
94.0+0.6
93.4
SHOT-strong
89.8#1.1
90.311.3
92.0+1.5
91.3=1.6
92.010.7
92.0=0.8
91.910.5
91.3
U M
SHOT-ens
86.841.6
88.521.8
90.042.1
89.5-2.3
90.5+1.9
90.6+1.0
90.811.3
89.5
TOHAN-ens
87.311.8 89.711.
90.1+1.6
90.5=14
91.211.5 92.510.9
98.5+0.7
90.7
Multiple
DECISION
88.712.3
88.811.8
89.612.4
89.822.1
90.311.7 90.2+1.3
90.5
89.7
Bi-ATEN
89.5#1.1 90.9+1.2
91.5+0.8
91.1=0.4
92.1 92.5+2.3
90.5
91.2
HiFE (ours)
92.7:0.8
94.910.2
95.0=0.4
95.240..95.440.. 95.440..
95.210.6
96.1
95.0
SHOT-worst
40.9#1.0 45.1#1.2
50.9+1.1
51.6=0.9
51.7#1.1
51.820.8
51.9
49.1
Single
SHOT-best
74.8#144
75.1#1.2
79.8+1.3
79.1=0.9
80.6+1.1
79.8+0.5
79.1
78.3
SHOT-strong
74.5#2.0
73.5#1.1
78.7+1.8
78.2=1..5
78.8+1.
78.6:0.99
78.7:0.88
77.3
S >>
SHOT-ens
75.6#2.2
74.911.2
81,212.6
81.5=1.4
82.0+1.
81.611.0
81.7
79.8
TOHAN-ens
79.0#1.9
85.942.1
87.5+1.6
89.5=1.1
90.1
90.6
91.1
87.7
Multiple
DECISION
71.9+1.3
72.1#2.1
72.5+2.0
73.4=1.5
75.0+1.2
76.7
79.2
74.4
Bi-ATEN
75.1#1.7
77.211.1
77.14223
79.7=2.5
80.142.9
82,5+1.9
83.111.6
79.3
HiFE (ours)
79.242.1
85.742.0
38.111.090..:0..00.
92.8+1.0
88.7
SoOT-worst.
15.842.1
14.841.9
14.740.9
14.3:1.5
14.3+1.7
14.0=0.9
14.4+0.5
14.6
Single
SHOT-best
32.6:1.1
32.411.6
34.51112
37.3+2.0
38.410.8
40.6-0.66
40.5.0.7
36.6
SHOT-strong
32.6:1.1
32.311.7
34.3:11.6
37.0=1.3
38.210.9
40.240.8
40.410.9
36.4
U + S
SHOT-ens
33.342.1
32.111.8
34.141.9
36.5=1.2
38.1
39.9+0.9
40.4
36.3
TOHAN-ens
31.711.8
31.011.4
35.8:1.3
36.920.9
40.510.6
42.6-0.88
43.1
37.4
Multiple
DECISION
30.3:12.1
30.5:2.0
31.21188 31.5-199
32.011.2
32.141.3
32.4:0.9
31.4
Bi-ATEN
31.5
30.9
33.5+1.7 33.1 35.3
35.310.8
1..1 33.8
HiFE (ours)
33.042.0
32.911.2 37.5 39.8.0.8 40.1 42.741.1 43.3 39.5
Table 2: Classification acturacysstandard deviation (%) on three adaptation tasks of digit datasets. M, U, and S refer to MNIST, USPS, and SVHN, respectively. The suffixes -best and -worst refer to the best and
worst results after adapting each single source hypothesis. The sulfixes -strong and -ens refer to the result
of adapting the strong hypothesis and the ensemble of all hypotheses, respectively. Results of SHOT [Liang
let al., 2020 TOHAN (Chi et al. (202)), DECISION (Ahmed et al. 2021), Bi-ATEN [Li et al. 2024 and our HiFE are presented. The highest accuracy is marked in bold.
with office datasets in Table 3 (more results can be found in Appendix E The proposed HiFE consistently improves the adaptation performance, boosting the average accuracy from 60.1% to 64.2% in task W + A. While HiFE designed to adapt source models from single SD to TD, our approach also works effectively when the source models come from multiple domains (see Appendix F). In addition, HiFE also outperforms the SOTA in the partial FHA scenario (as shown Appendix
Results of image classification tasks. For image classification, evaluate our approach on two adap- tation tasks, CF + ST and ST - CF. As shown in Table 4 we achieve 1.3% average improvement over the SOTA ensemble approaches in the CF + ST' task.
Results of VisDA-C object classification tasks. Table Section 5.2 presents the results of the synthesis- to-real adaptation task using the large-scale VisDA-C dataset. When the number of target samples limited to 10 per class, SHOT-best achieves significantly higher accuracy (74.8%) compared to SHOT-strong (70.8%). This observation supports our hypothesis that certain weak hypotheses hypotheses after adaptation. As the number of target samples increases to 50 per class, the performance gap between SHOT-strong and SHOT-best narrows, suggesting that the dependency on the source model diminishes with more target data. A detailed comparison of HiFE with previous methods reveals that HiFE excels, particularly when fewer samples are available. For example, when the number of target samples per class 10, HIFE (76.1%) outperforms the SOTA methods Bi-ATEN and TOHAN (72.5%) by 3.6%. This finding indicates that HiFE effectively leverages weak hypotheses to enhance adaptation accuracy under conditions of data scarcity.
outperform strong
10

---

Published in Transactions on Machine Learning Research (09/2024)
Hypothesis Number
Number of Target Dats Class
Tasks
Methods
Average
2
3
SHOT-ens
76.5.11.9
78.611.5
78.711.7
80.1
81.211.2
79.0
A + D
Multiple
TOHAN-ans
DECISION
79.2=1.5
7..5+1.6
80.2+2.2
1.3
80.8+12
83.210.9
85.1=1.1
81.5=200
17.1
83.5+1.3
82.7
81.0
Bi-TEN
79.1+12
81.611
81.5+09
82.1=15
81.1221
HiFE (ours)
79.2-1.0
84.310.4
85.7
86.210.9
89.210.8
85.0
SHOT-ens 56.842.0
58.0#1.9
69.2+11.7
61.8=0.5
62.6+0.9
59.7
D + A
Multiple
TOHAN-ens
58.1+113
60.8+1.2
63.1#199
63.8=0.8
61.99.99
62.0
DECISION
54.11:1.6
54.2112.5
56.1122.1
58.510.7
BI-ATEN
55.2
57.1
60.3+00.9
61.3=1.1
62.5+188
59.3
HiFE (ours)
61.8=1.0
64.7.00.7
67.2=0.6
66.84:1.0
67.5+0.9
65.6
SHOT--es 55.1411.
58.211.6
59.91114
60.8
61.21111
59.0
W - A Multiple
TeHAN-ens 56.5+1.0
60.1
00.1112
61.2=0.8
62.5+0.7
60.1
DECISION
54.142.11
54,9118
55.6416
56.5=1.2
58.14:1.2
55.8
BLATEN
56.21119
58.411.6
58.91222
61.3.1.3
61.71111
59.3
HiFE (ours)
62.5-2.5
65.111.5
6..421.2
64.310.9
64.820.8
64.2
Table 3: Classification accuracy-sstandard deviation (%) three adaptation tasks of office datascts. A, D, and W are abbreviations of Amazon, DSLR, and Webcam. The sullix -ens refers to the result of the ensemble of all adapted hypotheses. Results of SHOT (Liang et al., 20200,, TOHAN (Chi et al. (021), DECISION (Ahmed et al. 2021 Bi-ATEN [Li ct al. 2024), and our HiFE are presented. The bold value represents the highest accuracy.
Number of Target Data per Class
Tacks
Methods
Average
15
20
SHOT-ens
70.3:0.4
70.3
CF ST
TOHAN-cut
67.51.0.6
69.8100.5
68.7
DECISTON
70.4:10.4
70.610.6
70.5
BI-ATEN
70.7=0.3
70.9=0.5
70.8
HIFE (ours)
71.6#0.4
71.9#0.3
71.8
SHOT- 53.11:0.6
03.5
53.3
ST + OF
TOHAN-ens 52.540.6
52.6#0.8
52.6
DECISION 54.2100.5
54.510.6
54.4
BI-ATEN
53.710.4
54.62004
54.1
HIFE (ours)
55.020.3
55.3#0.3
55.2
Table 4: Classification Accuracyzstandard deviation (%) two tasks between CIFAR-10 (CF) and STL-10 (ST).
5.3 Ablation Studies
A Ablation study on the feature DeCL loss. We study the advantage our training loss by incorporating feature DeCL. loss LDcCI. Equation with different 8 values ranging from 1.0 with the digit datasets. In this context, corresponds to training the network using only the feature DeCL loss LDCCL As shown in Table G optimal results generally occur at B = 0.1, which yields an average improvement of 1.4% compared to the result obtained when no feature DeCL loss is used (3 = 0). Notably, even when the model is trained solely using the feature DeCL loss LDCCT. (3 = 1.0), it still achieves an average improvement of 10.9% compared to the accuracy before the adaptation (WA), demonstrating the effectiveness of the feature DeCL loss.
corresponds training the network using only the supervised loss LCLS, while 3 =
We also visualize the correlation matrices of the features after the merge the first layer of the task U M when B = 0.1. As depicted in Figure 3 as the training progresses, the feature DeCL loss guides the decrease of most of the correlation values between the four features, thereby increasing feature diversity.
Ablation study on the number of weak hypotheses. We conduct an ablation study to analyze the impact of the number of weak hypotheses on the final performance, providing insights into choosing a proper number of weak hypotheses to balance the cost and performance. We do this experiment in an adaptation task from domain Mnist to USPS. We select varying numbers (from to 6) of source hypotheses from the models provided in Section 5.1 and make sure the accuracy range of each group of source models is the same. For each experiment, the selected weak hypotheses started from hy and ended with h11, ensuring the
11

---

Published in Transactions on Machine Learning Research (09/2024)
Hypochesis
Number of Target Data Class
Tasks
Methods
Average
Number
10
50
SHOT-worst
70.81112
76.111.3
80.410.6
75.8
Single
SHOT-bost
74.810.8
78.811.1
81.34
78.3
SHOT-strong
70.8+1.3
71.7+1.7
81.1=04
76.5
Synthesis Real
SHOT-ens
72.1
77.14
80.9
76.7
Multiple
TOHAN-ens
72.3+1.5
7T.741.2
81.3=0.9
77.2
DECISION
71.2122.1
76.5.1.9
79.811.1
75.8
BIATTEN
72.542.0
77.741.9
81.320.8
77.2
HiFE (ours)
76.1#1.1
80.3+0.8
82.120.7
Table 5: Classification Accuracyzstandard deviation (%) on the adaptation task from VisDA-C synthesis data to real object data.
The Value of
Tasks
WA
0.1
0.2
0.3
0.5
0.7
0.9
1.0
5 - 93.2
94.0
93.9
93.2
92.3
90.0
86.4
77.0
67.
U
94.5
95.6
941.1
92.8
91.2
88.7
82.3
78.2
M + S
52.2
51.2
54.3
53.22
52.8
52.7
52.1
46.0
23.2
97.1
96.3
95.1
94.6
91.2
70.5
U 8
44.5
46.2
44.5
42.5
40.1
39.6
39.1
33.2
26.2
+ 94.9
95.9
95.3
95.1
94.6
92.9
91.3
89.9
88.5
Average
79.2
80.6
80.1
79.1
78.1
76.9
75.4
69.9
59
Table 6: Ablation study on the feature decrrelation learning loss balance parameter 3 in Equation 4 M, U, and S are abbreviations of MNIST, USPS, and SVHN. WA indicates the accuracy of the model without the adaptation. The bold value represents highest accuracy (%).
selected accuracy range was [60, 95). The results are presented in Exp1-1 Exp1-s of Table 7 As shown in Table 7 when the number of weak hepotheses is less than 3, the average accuracy (94.2% in Expı-1) is lower than that when the number of weak hypotheses is greater than 3. Moreover, the average accuracy remains consistent when the number of weak hypotheses exceeds (see the comparison of the result of Exp1-3 Expiss..
Ablation study on the accuracy range weak hypotheses. To investigate the impact of the accuracy range of weak hypotheses the final performance, we leverage the source models presented in Section 5.1 and do the digit adaptation task from Mnist to USPS using source models with varying accuracy ranges. We conducted experiments from Exp2-1 to Exp2-4 as outlined in Table 7 Our results indicate that the accuracy range increases, the performance after adaptation improves. It is important to note that weak source models with an accuracy lower than 55% may harm the final performance. Our comparison of Exp2-1 and Exp2-5 revealed that using weak hypotheses with such low accuracy resulted in worse performance than
0.060
0.066
0.060
0.06C
0.056
0.00
0.057
0.05
0.061
0.052
0.049
0.049
0.046
0.045
0.047
0.045
0.055
0.055
0.055
0.055
0.056
0:045
0.057
0.045
3.043
0.049
0.040
0.015
0.045
0.045
0.043
0.045
0.045
0.045
0.045
0.047
0.045
0.040
0.04C
0.040
0.04C
0.045
12
13
14
11
12
(b)
Emas
(c)
Emaa
(d)
Emax
Figure 3: The correlation matrixes of the features after the merge at the first layer on digit task U - M when the 3 value is set to 0.1. (a) ((d) shows the results over different training stages with e, Emax being the current and maximum number of epochs. As the training continues, we observe that most of the correlation values between the four features in this layer decrease (the darker the color, the lower the corresponding correlation value).
12

---

Published in Transactions on Machine Learning Research (09/2024)
Weak Hlypothesis Weak Hypothesis Indices FN LN
Number of Target Data per Class
Exp ID HN
UH
Average
Accuracy Range
3
5
7
Expi-1
[60, 95)
his. hi las, bs, his
2
3
h12
92.04112
94.610.77
94.5=0.5
95.64-0.5
94.2
Exp1-2 3
'0, 95)
2
3
h12
92.3+1.0
94.8+0.6
96.7=0.7
96.1+0.7
94.7
Exp1-3
60, 95)
has ler, has ley les, lir, he, by: hy1
2
3
h12
92.7+111
95.2+0.6
96.1=0.5
96.740.2
95.2
Exp1- -4
60, 95)
2
h12
92.611.
95.3+0.7
96.0=0.5
96.6+0.4
95.1
Exp1- 5
660, 95)
hy. ho. by, has heg h11
2
3
h12
92.8:0.9
95.110.5
96.210.7
96.810.5
95.2
Exp2-1
[40, 55)
hy hg ha ha, hs, ha
2
3
h12
91..140.7
93.840.6
94.5-0.8
95.210.8
93.7
Exps-2
[55, 70)
2
3
h12
91.5+0.77
94.1+0.6
95.3-0.6
96.0+0.8
94.2
Exp2-3
70, 85)
by, hgs hgg
2
3
h12
92.2+0.6 94.7+0.8
95.7=0.8
96.7+0.6
94.8
Exp2-4 3
88, 95)
hgg lgo; hes
2
3
92.4+0.8
94.8#0.4
96.1=0.8
96.7+0.6
95.0
Expg-5
1
h12
91.550066 94.210.6
94.2
95.5
93.9
Expa-1
660 95)
h5-11
2
4
h12
93.041.4 95.310.5
96.0=0.3
96.7100.3
95.3
Exps-2
660, 95)
h5-11
4
3
h12
93.241.1 94.910.6
96.2
96.64104
95.2
Exps-3
660 95)
h5
8
92.3+1.2
93.1+1.0
95.4-0.8
95.6+1.5
94.1
Exps-4
60, 95)
R5
h12
92.1+10 92.5#0.9
94.=006
95.1
93.6
Exp4-1
60, 95)
h5-11
2
8
h12
|93.0
95.3+0.5
96.0=0.3
96.7+0.3
95.3
Exp4-2 2
60, 95)
h5-11
2
K11,12
93.2
95.610.7
95.8-0.8
96.910.5
95.4
Exp4-3 3
(60, 95)
h5-11
2
8
h1o-12
92.5:10.6
95.110.7
95.420.5 96.110.4
94.8
Exp4-4
[60, 95)
hg-11
2
8
h5-12
91.040.8
93.110.6
93.8-0.6
94.1+00.3
93.0
Exp1-5
[60, 95)
h5-11
2
8
15-10
88.1
89.1#0.8 89.5# 1.1 89.7
89.1
Table 7: Classification Hssanday deviation (%) on digit, adaptation task Mnist + USPS with varying parameters including the number of weak hypotheses (HN), the accuracy ranges of the weak hy- potheses, the number of input features to each WRU (FN), the layer number (LN) in the hicrarchical feature ensemble module and the updated hypotheses during fine-tuning (UH). h22 is the strong hypothesis.
adaptation without weak hypotheses. To summarize, our approach, HiFF, performs well when incorporating larger number of weak hypotheses, provided that these hypotheses are not of very low accuracy. In our experiments, we selected 7 weak hypotheses as our base setting to balance performance and computational cost. In practical ical applications, the selection of weak hypotheses can be guided by the available computational resources and the performance of each weak hypothesis on the SD.
Ablation study on the hicrarchical layer number. To investigate the impact of the layer number in the ensemble module on the final performance, we have experimented with varying numbers of input features fed into WRU, which subsoquently alters the number of merge layers within the hicrarchical feature ensemble module. we conducted experiments from Exp3-1 to Exp3-4, modifying the nunber of input features for Notably, in Exp3-1, we utilized a simple weighted feature
each WRU. The results are presented in Table [ sum to merge all source features rather than using HEFM with WRU for feature aggregation. When we set the number of input features for cach WRU to 2 or 4, with the corresponding layer number in the ensemble module to be and 3, respectively, the adaptation average accuracy is similar. However, when the number of input features for each WRU increases to 8, we apply one WRU to merge the eight source encoders at once, and the adaptation average accuracy decreases to 94.1%.
Ablation study on the updated hypotheses. In our HiFE approach, we aggregate multiple hypot During the fine-tuning stage, we fix all the weak hypotheses and only update the parameters in the strong hypotheses. We conducted experiments from Exp4-1 to Exp4-4, modifying which hypotheses are updated during the fine-tuning stage. Exp1-1 is our default setting, where only the last strong hypothesis h12 is updated and it achieves good performance. When we update all the weak and strong hypotheses together (Exp4-3), or only update all the weak hypotheses (Exp4-4), the performance deteriorates. Notably, when update the last two models hn and h12 (Exp4-2), we achieve the best performance. This is because h12 tends to overfit to the SD while h1 is better suited for the TD (see the comparison of the result of adaptation of h1 (SHOT-best) and h12 (SHOT-strong) of task M + U in Table & in Appendix See Sc In practice, the few-shot larget samples are insufficient to guide the fine-tuning of both weak and strong hysot together. Additionally, it is unrealistic to determine which source hypothesis is best for the target domain beforehand. Therefore, the most suitable approach is to only update the final strong hypotheses while keeping all other weak hypotheses unchanged among the source encoders. By designing and updating the hierarchical feature ensemble module, we can merge all these models to obtain the best final model.
13

---

Published in Transactions on Machine Learning Research (09/2024)
6 Limitations and Future Work
HiFE leverages multiple source hypotheses with varying accuracy levels from the SD to improve the per- formance of models in the TD. By exploiting the diversity of source models, HiFE has the potential to enhance the generalization capabilities of the adapted models. However, the additional hypotheses result in increased model transfer and storage costs. Moreover, the increase the number of parameters of the target model lcads to higher computational costs. Nonetheless, we argue that the benefits of leveraging multiple source models with different strengths outweigh the costs associated with processing additional hypotheses, particularly when source data absent for transfer. With the growing need to address privacy concerns and mitigate data-sharing challenges in real-world applications, opting for weak models simplifies collaboration between source providers and users.
For future rescarch, it would be beneficial to investigate methods for generating weak hypotheses with higher diversity. Although the current experiments obtained weak hypotheses in the same run as generating the final strong hypotheses for simplicity, there is potential for improvement by obtaining weak hypotheses through different random seeds, hyperparameter choices, training different subsets of the source data. By increasing the diversity of weak hypotheses, we could obtain better performance after adaptation and further improve the elfectiveness of the proposed approach.
7 Conclusion
In this paper, investigate the potential of utilizing weak source hypotheses for domain adaptation and introduce new problem setting termed "few-shot hypotheses adaptation with weak models". To tackle this problem, we design a new framework called HiFE, which leverages an array of readily available weak hypotheses to improve the adaptation performance of a strong source hypothesis. As a result, HiFE signifi- cantly mitigates the occurrence of over-fitting under the few-shot setting and achieves the SOTA performance across various adaptation tasks. This research introduces an innovative perspective for addressing the FHA problem in scenarios where the source data is inaccessible and the target data is limited. Additionally, this research shed light the use of weak source model to boost the practical application of transfer learning in scenarios where data privacy concerns are on the risc.
8 Acknowledgments
We would like to express sincere gratitude to the anonymous reviewers for their valuable feedback and constructive suggestions. Their insightful comments greatly contributed to improving the quality and clarity of this paper.
14

---

Published in Transactions on Machine Learning Research (09/2024)
References
Sk Miraj Ahmed, Dripta S Raychaudhuri, Sujoy Paul, Samet Oymak, and Amit K Roy- Koy-Chowdhury. Unsu- pervised multi-source domain adaptation without access to source data. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2021.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):151 175, 2010.
Keith Beven and Andrew Binley. The future of distributed models: model calibration and uncertainty prediction. Hydrological processes, 6(3):279 298, 1992.
Yaoming Cai, Xiaobo Liu, Yongshan Zhang, and Zhihua Cai. Hierarchical ensemble of extreme learning
machine. Pattern Recognition Letters, 116:101-106, 2018.
Jiuwen Cao, Zhiping Lin, Guang-Bin Huang, and Nan Liu. Voling based extreme learning machine. Infor- mation Sciences, 185(1):66-77) 77, 2012.
Gong Cheng, Ceyuan Yang, Xiwen Yao, Lei Guo, and Junwei Han. When deep learning meets metric learning: Remote sensing image scene classification via learning discriminative enns. IEEE Transactions on Geoscience and Remole Sensing, 56(5):2811 2821, 2018.
Haoang Chi, Feng Liu, Wenjing Yang, Long Lan, Tongliang Liu, Bo Han, William Cheung, and James Kwok. Tohan: A one-step approach towards few-shot hypothesis adaptation. In Adoances in Neural Information Processing Systems, volume 34, pp. 20970 20982, 2021.
Adam Coates, Andrew Ng, and Honglak Lec. An analysis of single-layer networks in unsupervised feature learning. In Proccedings of the Fourteenth International Conference Artificial Intelligence and Statistics, pp. JMLR Workshop and Conference Proceedings, 2011.
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Chi Su, Qingming Huang, and Qi Tian. Gradually vanishing bridge for adversarial domain adaptation. In Proceedings of the IEFE/CVV Conference on Computer Vision and Pattern Recognition, pp. 12455 12464, 2020.
Danruo Deng, Guangyong Chen, Furui Liu, Jinpeng Li, Jianye Hao, and Heng Pheng-Ann. A simple and provable method to adapt pre-trained model across domains with few samples, 2023. URL https:
//openreview //openreriee.net//forrrr// wwt.netooooummddee220007 gm3
Thomas G Dietterich. Ensemble methods in machine learning, In Multiple Classifier Systems: First In- ternational Workshop, MCS 2000 Cagliari, Italy, June 21-23, 2000 Proceedings 1, pp- 1-15. Springer, 2000.
Ning Ding, Yixing Xu, Yehui Tang, Chao Xu, Yunhe Wang, and Dacheng Tao. Source-free domain adaptation via distribution estimation. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 7212-7222, 2022.
Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. A survey ensemble learning. Frontiers of Computer Science, 14:241-258, 2020.
Dirk Eilers, Felippe Schmoeller Roza, and Karsten Roscher. Ensemble-based uncertainty cstimation with overlapping alternative predictions. In Deep Reinforcemenl Learning Workshop NewrIPS 2022, 2022.
Pascal Germain, Alexandre Lacasse, François Laviolette, and Mario Marchand. Pac-bayesian learning of linear classifiers. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 353 360, 2009.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition, pp. 580-587, 2014.
15

---

Published in Transactions on Machine Learning Research (09/2024)
lan J Goodfellow, Mchdi Mirza, Aaron Courville, and Yoshua Bengio. An empirical investigation of catas- trophic forgetting in gradient-based neural networks. Stat, 1050:6, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Han-Kai Hsu, Chun-Han Yao, Yi-Hsuan Tsai, Wei-Chih Hung, Hung- Yu Tseng, Mancesh Singh, and Ming- Hsuan Yang. Progressive domain adaptation for object detection. In Proceedings of the IEEE/CVE winter conference on applications of computer vision, pp. 749-757, 2020.
Pin Jiang, Aming Wu, Yahong Han, Yunfeng Shao, Meiyu Qi, and Bingshuai Li. Bidirectional adversar- ial training for semi-supervised domain adaptation. In Proceedings of the Twenty-Nin Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 934 940, 2021.
Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0993-4902, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaune Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnicszka Grabska-Barwinssee ct al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-355) 2017.
Kurt. Koffka. Principles of Gestall psychology. routledge, 2013.
A Krizhevsky. Learning multiple layers of features from tiny images. Moster's thesis, University of Tront, 2009.
George Kuczera and Eric Parent. Monte carlo assessment, of parameter uncertainty in conceptual catchment models: the metropolis algorithm. Journal of Hydrology, 211((-4):69 85, 1998.
Yann LeCun, Lćon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied docu- ment recognition. Proceedings of the IEEE, 86(11))2278-2333 2324, 1998.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaplation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. -6500, 2020.
Xinyao Li, Jingjing Li, Fengling Li, Lei Zhu, and Ke Lu. Agile multi-source-free domain adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 13673 13681, 2024.
Jian Liang, Ran Hc, Zhenan Sun, and Tieniu Tan. Aggregating randomized proommnin--rmottngg invariant projections for domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41 (5):1027-1042, 2018.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 6028-6039. PMLR, 2020.
Yong Liu and Xin Yao. Ensemble learning via negative correlation. Neural Networks, 12(10):1399 1404, 1999.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan. Deep transfer learning with joint adapta- tion networks. In International Conference Machine Learning, pp. 2208-2217 PMLR, 2017.
Andres Masegosa. Learning under model misspecification: Applications to variational and ensemble methods. Advances in Newral Information Processing Systems, volume 33, pp. 5479-5499 2020.
David A McAllester. Pac-bayesian model averaging. In Proceedings of the Twelfth Annual Conference on Compulational Learning Theory, pp. 164-170, 1999.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Sacid Motiian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. Advances in Neural Information Processing Systems, 30, 2017.
Tristan Needham. visual explanation of jensen's inequality. The American Mathematical Monthly, 100(8): 768-771, 1993.
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Codric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference Machine Learning, pp. 7294 7305. PMLR, 2020.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint Xiv: 010.06924, 2017.
Sebastian Ruder, An overview of gradient descent optimization algorithms. arx preprint arXiu:1609.04747, 2016.
Andrei Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arviv preprinl arXiv:1606. 146.6 2016.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 213-226. Springer, 2010.
Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1249 2018
Zenglin Shi, Le Zhang, Yun Liu, Xiaofeng Cao, Yangdong Ye, Ming-Ming Cheng, and Guoyan Zheng. Crowd counting with deep negative correlation learning. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition, pp. 5382 5390, 2018.
Rui Shu, Hung H Bui, Hirokazu Narui, and Stefano Ermon. A dirt-t approach to unsupervised domain adaptation. arXiv preprint arXiv:1802.00 08735, 2018.
Yang Shu, Zhangjie Cao, Ziyang Zhang, Jianmin Wang, and Mingsheng Long, Hub-pathway: transfer learning from a hub of pre-trained models. Advances in Neural Information Processing Systems, 35: -29227, 2022.
Yu Su, Shiguang Shan, Xilin Chen, and Wen Gao. Hicrarchical ensemble of global and local classifiers for face recognition. IEEE Transactions on Image Processing, 18(8):1885 1896, 2009.
Sanatan Sukhija, Narayanan C Krishnan, and Gurkanwal Singh. Supervised heterogencous domain adapta- tion via random forests. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pp. 1039-20455 2016.
Boyu Wang, Jorge Mendez, Mingbo Cai, and Eric Eaton. Transfer learning via minimizing the performance gap between domains. Advances in neural information processing systems, 32, 2019.
Shuo Wang, Huanhuan Chen, and Xin Yao. Negative correlation learning for classification ensembles. In
The 2010 International Joint Conference on Newral Networks (IJCNN), pp. 1-8. IEEE, 2010.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 499 515. Springer, 2016.
Guanglei Yang, Hao Tang, Zhun Zhong, Mingli Ding, Ling Shao, Nicu Sebe, and Elisa Ricci. Transformer- based source-free domain adaptation. arXiv preprint arXiv:2105.14 2021.
Shiqi Yang, Shangling Jui, Joost van de Weijer, al. Attracting and dispersing: A simple approach for source-free domain adaptation. Advances in Neural Information Processing Systems, 35:5802 5815, 2022.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Moslem Yazdanpanah and Parham Moradi. Visual domain bridge: A source-free domain adaptation for cross-domain few-shot learning. In Proceedings of the IEFE/CVV Conference Computer Vision and Pattern Recognition, pp. 2868 2877, 2022.
Li Yi, Gezheng Xu, Pengcheng Xu, Jiaqi Li, Ruizhi Pu, Charles Ling, A Tan McLeod, and Boyu Wang. When source-free domain adaptation meets learning with noisy labels. arXiv preprint arXiv:2301.1388 2023.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical asscssment of pre-trained models for transfer learning. In International Conference Machine Learning, pp. 12133 12143.. PML.R, 2021.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balancod self-training. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 289 305, 2018.
18

---

Published in Transactions on Machine Learning Research (09/2024)
Supplementary Material for "HiFE: Hierarchical Feature Ensemble Framework for Few-shot Hypotheses Adaptation"
We organize the appendix as follows:
In provide pilot study on the FHA process from SVHN to Mnist data.
In we provide some implementation details of the experiments.
In C we provide a brief overview of the related theorems that support the proposed approach and highlight their relevance to the work presented the paper.
In D we present additional results of experiments on digit datasets to supplement and reinforce the findings discussed the main text.
In provide more results of the experiments on office datasets.
In we apply HiFE to multi-source multi-model adaptation scenario and analyze the performance.
In we extend HiFE to a partial few-shot hypothesis adaptation scenario, exposing the model's ver- satility and potential for future applications.
In wc provide some analyses on the model complexity.
A
Pilot Study on the FHA Process From SVHN to Mnist.
The strong source model may overfit the SD and perform worse the TD after the adaptation. To justify our assumption, we empirically design an experimental task that adapts source models with varying source accuracies from the digit dataset SVHN to the target dataset Mnist. Figure shows that the weak source hypotheses (e.g., Model-4 [source acc=76.0%]) could perform better than the strong source models (e.g.. Model-1 [source acc 92.2%)) on the TD, indicating that weak hypotheses can generalize better on the TD.
B Implementation Details
Training target hypotheses with HiFE. As mentioned in the manuscript Section 5 we generate 12 source models {h: Among these models, we used {h:1)2s} (8 models) in the experiments shown in Section 5.2 In our HiFE, we set the number of input features for each WRU to two, leading to four-layer feature ensemble structure.
The network uses the PyTorch framework on a PC with four NVIDIA 2080ti GPUs. We trained the source hypothesis using stochastic gradient descent. (SGD) optimizer with a momentum value of 0.5 with the learning rate initialized 1e-2 and decreased to 1e-5 step by step. During the adaptation, we adopt SGD with Nesterov momentum (Ruder 2016) with momentum value of 0.9. Following (Liang et al. 2020 insert batch normalization layer and weight normalization layer before the end of each encoder and classifier, respectively.
C Review of the Theorems
In this section, we present a brief overview of the related theorems of Theorem 1 which is derived under the PAC-Bayes framework (McAllester 1999). The PAC-Bayes theory provides data-dependent generalization bounds over the generalization error E,rey (I(0) of model with parameter under i.i.d. data. In recent years, PAC-Bayes bound (Germain al. 2009) has been widely adopted because it can apply general unbounded losses (e.g., log-loss). We restate here this PAC-Bayes bound follows:
19

---

Published in Transactions on Machine Learning Research (09/2024)
Model-1 (92.9%)
Model-2 (87.1%)
Model-3 (82.4%)
Modcl-4 (76.0%)
1st
Accuracy rank
2nd
4th
2
6
The number of samples per class the target domain
Figure 4: This figure depicts the FHA process from SVHN to Mnist. Four source models were generated using the training data from SVHN and fine-tuned with different quantities of samples from Mnist. The y-axis indicates the performance rank after adaptation, with the highest accuracy in the TD ranked first. Model-4, despite having the lowest source accuracy (76%), exhibits notably superior performance on the TD compared to Model-1, which has the highest source accuracy (92.9%).
Theorem 2. IPAC-Bayes bounds GGermain et al. 2009: Masegosa 2020 McAllester 1999
Given a data distribution P over X y, hypothesis space O prior distribution a over O, for any 8 E (0, and A O, with probability at least 8 over samples D pa, we have for all posterior p,
(o(o)( s (100, D)) [Px m) Ing Ve,.(,n)).
where Vr, n) = (n (0) P (e (21L10)-L1)
Based on the socond-order Jensen inequalities (Needham) 1993) ), the second-order oracle bound with tighter
PAC-Bayes bounds is derived. (Deng et al. 2023) further extends this theory to ensemble models as shown
in Theorem 3 DDng et al. 2023) also shows second-order PAC-Bayesian bound over the performance of
the posterior predictive distribution of the averaging ensemble model, which is shown in Theorem
Theorem 3. (Second-order oracle bound ((Deng et al. 2023 MMasegosa 2020 NNedham 1993 Given data distribution P, set of model parameters {0:).21 for ony distribution {p: that
AM over [
MM satisfies
M
(L()) <
(L(0)=
E (.) [L()) V(p(O)).
M
where 8, O, 0 == [0:) and p(0) M Pc(O:) and vo(()) is variance term defined as
2
V(p())) Eptay F(
Σ p(ul,, 0.)
rlul B&)
MMmaxoppylz.. 0)
Rm
Lemma 1. For any distribution M over (0,)M, the Second-order Jensen bound of Theorem can be
eapressed follows Deng et al. 2023
1
(ot0))((00)) -(((O)) = Eplo) (z))),
M
20

---

Published in Transactions on Machine Learning Research (09/2024)
where
1 (plvlz,0) 2maxop(ylz; 8)2
E p(y12, 0k.)
M
T2(0) (
log(y/>, 0:)
M
Theorem 4. (Model ensemble error bound ( (Deng et al. 2023 ) Given data distribution P over X x y,
LM set of model parameters {Or)-= and associated prior n. ii=1, where is defined over U₁ a 8 E (0, and a real number c. > O, with probability at least over samples ~ pm, we have for all posterior {mr over {0:
M
Ep.to.) (0 D)] + (xr, (.))
E P( (I(0) S
Σ M
V((((), D) + cnL'
en
where (p((0), D) is the empirical version variance term V(p(0) and is defined
e(,, (0) DM mm on (E (1.00))- M (100)) +ogg
1
Lemma 2. If there evists inpul sample E Px such that ha, (1) + (x), we then have V(p(0) > 0 (Deng et al. 2023)).
Our method proposes to add multiple weak hypotheses for training, thereby improving the diversity of the source model. From the above theorem, approach intends to look for tighter upper bound of the expected loss.
To further analyze the term (R (p: Tt) in Theorem 4 when providing n training samples from the target domain, we assume 0g E Rd, (8,) ~ ~N(0 22), and p.(),) is Dirac-delta distribution centered around 0, with (=(0,) = 56: (6.), vi E [M], then we have
Ep Ep, D)) Es (o)1 D) -f 60 L(0,, D)d0, L(0,, D),
and
DKL(P: || nz)) = 5 f log 50 (0.)
T(,)) d8, = -loga0; = di log tog(2 + 20 || T| 11
1
Hence, the equation in Theorem 4 can be updated to
(p (L(0) < M Σ [L06). D)
d, log (mo²)) (p(()), + onL'
0 112
2en
2en
This error bound is what we showed in Theorem in the manuscript.
D More Results on Digit Datasets
We evaluate our approach on six closod-sct adaptation tasks on digit datasots. We report the results of M + S, S + U, and M in Table g Our HiFE achieves 3.7%, 2.1%, and 2.3% average improvement over the SOTA these three tasks, respectively.
21

---

Published in Transactions on Machine Learning Research (09/2024)
Hypothesis
Number of Target Data per Class
Tasks
Methods
Average
Number
2
4
SHOT-worst
16.341..7
15.141.5
16.641.1
17.341.2
16.742.1
16.84199
17.540.9
16.6
Single
SHOT-best.
28.2:1..
33.5
35.3+2.1
36.912.3
43.0
47.242.1
50.1
39.2
SHOT-strong
28.011..
33.011.5
35.011.3
36.212.0
39.0
47.010.8
50.0+0.7
38.3
SHOT-ens
28.2:41.8
29.941.6
33.441.4
34.241.2
34.611.3
35.740.9
37.340.8
33.3
TOHAN-ens
30.2:11.2 41.011.5
41.111.
41.410.9
42.811.1
43.1
45.5
40.1
Multiple
DECISION
24.2:10.9
26.911.6
27.611.1
29.411.3
30.910.5
33.210.8
35.310.9
29.6
Bi-ATEN
28.5:10.9
31.111.2
32.110.8
35.511.7
38.111..
38.51199
39.11114
34.7
HiFE (ours)
30.411.4
40.811.1
43.511.2
45.710.5
46.911.9
48.8
50.3
43.8
SHOT-worst
44.4#1.4
45.5#1.1
51.240.9
5..3+1.5
50.8+19
50.8+0.7
50.640.6
49.1
Single
SHOT-best.
82.9+0.8
81.410.7
8..3+
86.911.8
87.5+0.8
87.1
88.1
86.0
SHOT-strong
81.111.6
80.811.4
85.210.9
85.311.1
85.2
85.011.7
85.1
84.0
S +
SHOT-ens
80.8:2..1
81.811.6
1.
86.1
86.540.9
86.21 0.5 87.540.4
84.8
TOHAN-ens
86.1+1.0
89.1
90.010.8
91.510.2
92.810.5
93.411.1
92.5
90.8
Multiple
DECISION
73.511.2 73.711.
76.210.8
78.510.8
79.740.5
80.1
82.1
77.7
Bi-ATEN
75.5#2.1 78.1
78.9+1.4
83.2#3.0
84.+2.3
88.2 88.9
82.4
HIFE (ours)
86.1+1.6
90.6+0.4
9.2
94.0
95.0
9.. 0.5
92.9
SHOT-worst
54.0+2.3
54.1+1.7
58.3+1.2
58.2+1.6
58.6+0.7
58.8+1.5
58.5+1.5
57.2
Single
SHOT-best.
92.2+2.3
94.2#2.3
94.6+1.8
94.7
94.7+
94
94.9+1.2
94.3
SHOT-strong
92.042.4
94.142.0
91.5199
94.31.66
94.311.5
94.641.6
94.240.9
94.0
M + U
SHOT-ens
86.3+1.1
86.8+1.5
88.811.8
88.1+1.6
88.240.4
88.0
88.0+0.6
87.7
TOHAN-ens
88.4+1.8
92.8+1.5
93.5+1.6
92.8+0.9
93.540.6
94.8+0.8
95.1
93.0
Multiple
DECISION
82.942.3
82.941.0
83.341.5
8424188
85.541.3
84.6 0.9 85.1
84.1
Bi-ATEN
81.5+1.6
84.142.3
85.64188
88.1+1.5
87.940.9
89.7+0.7 91.+1.6
86.9
HiFE (ours)
93.0+0.9
94.0#1.2 95.3+0.5 95.5 96.0=0.3 96. 96.7=0.3
95.3
Table 8: Classification astanaay deviation (%) on two adaptation tasks of digit datasets. M, U, and S refer to Mnist, USPS, and SVHN, respectively. The suffixes of -best and -worst refer to the best and worst results after adapting each single source hypothesis. The sulfixes -strong and -ens refer to the result after adapting the strong hypothesis and the ensemble of all adapted hypotheses, respectively. Results of SHOT (Liang et al. 2020), TOHAN (Chi al. 2021), DECISION (Ahmed et al., (022),, Bi-ATEN (Li al. 2024), and our HiFE are presented. The highest accuracy is marked in bold.
E
More Results on Office Datasets
The complete results of the closest-set adaptation tasks with office datasets are presented in Table g Our analysis reveals that HiFE outperforms SHOT-strong, indicating that incorporating weak hypothescs can lead to improved performance. Regarding multi hilti--ypotheses adaptation results, HiFE achieved average accuracy improvements of 4.1% and 3.6% over the SOTA on W A and D - A tasks, respectively. These results demonstrate that HiFE can circunvent the potentially severe negative transfer induced by weak source hypotheses (as observed in the SHOT-worst result) and outperform previous model ensemble approaches.
F Experiments on Multi-domain Multi-hypotheses Adaptation Scenario
In our manuscript, HiFE is designed for adapting source models from one single source domain target domain. Individuals may wonder whether HiFE still works if the source models come from multiple domains. To answer this question, we do experiments office datasets (Amazon (A), DSLR (D), Webcam (W)). We train one strong and seven weak hypotheses for each domain in our singlc-source multi-model adaptation experiment. To adopt models from multiple domains, we randomly choose one strong and seven weak hy- potheses among the 16 hypotheses from domains and W and adapt them to A. As shown in Table [0] HiFE outperforms the SOTA with 4.4% average improvement in this multi-domain and multi-model adap- tation task. Moreover, compared with the single-source domain adaptation tasks D + and W A, HiFE achieves 1.0% and 2.4% improvement the multi-source domain adaptation task D, W A, respectively.
22

---

Published in Transactions on Machine Learning Research (09/2024)
Hypothesis
Number of Target Data per Class
Tasks
Methods
Average
Number
4
wort-
42.6:2.5
45.6
45.3+2.3
46.4
46.820.9
45.3
Single
SHOT-beat
78.1 78.6
79.0#2.1
80.0#1.8
81.210.8
79.4
SHOT-strong
74.0+2.1
76.6#1.5
77.411.9
76.210.9
81.041.2
77.1
A A
SHOT-ens
76.5#1.9 78.621.5
78.7#1.7
80.120.9
81.2
79.0
TOHAN-ens
78.511.6 79.5#1.3
83.2#0.9
85.1
87.1
82.7
Multiple
DECISION
79.2:1.5 80.212.2
80.8+1.2
81.512.0
83.5
81.0
Bi-ATEN
79.1 81.6#1.4
81.520.9
82.1
84.122.1
81.7
HiFE (ours)
79.211.0 84.3
85.711.0
86.210.9
89.2
85.0
wort-worst
17.4 18.8#2.1
20.3#1.0
20.1#1.5
22.320.7
19.8
Single
SHOT-best
60.2
62.812.0
63.3
66.812.0 66.3
63.9
SHOT-strong
60.110.9
62.511.5
63.111.2
63.212.1
61.142.0
62.6
D A
SHOT-ens
56.812.0
58.0
59.2
61.820.5
62.510.9
59.7
TOHAN-ens
58.111.3 60.811.2
63.1#1.9
63.810.8
64.1
62.0
Multiple
DECISION
54.1#1.6 54.2$2.5
56.122.1
57.420.9 58.520.7
56.1
Bi-ATEN
55.211.3
57.1#2.1
60.3#0.9
61.521.1
62.511.8
59.3
HiFE (ours)
61.8+1.0
64.7:0.7
67.2:0.6
66.811.0
67.
65.6
SHOT-worst
48.112.1 48.5#1.9
48.8#1.5
49.1
49.640.5
48.8
Single
SHOT-best
77.3
77.9#1.2 77.9#1.5
78.9 79.5
78.3
SHOT-strong
76.2$0.9
76.2:0.5
75.7#1.9
75.2
75.2
75.7
SHOT-ens
77.6 78.310.9
78.5+1.3
79.5+1.7
81.1
79.0
TOHAN-ens
82.5+1.2 88.312.1
88.911.0 87.1 88.910.6
87.1
Multiple
DECISION
82.211.1 82.8#2.1
83.6#2.0 84.1#0.9 85.640.4
83.7
Bi-ATEN
81.511.7
83.5#1.9
85.2#1.1
87.1#0.9
88.511.3
85.2
HiFE (ours)
84.211.3
88.020.6
88.9
90.210.5
90.310.7
88.4
woOt-worst
31.9 32.1
31.8
32.1
33.0
32.2
Single
SHOT-best
61.910.8
6..011.2
61.7+1.1 61.910.7
63.1
62.1
SHOT-strong
60.911.5
60.9
61.6#1.
61.810.9
62.1
61.5
W A
SHOT-ons
55.111.2
58.211.6
59.9#1.4
60.811.1
61.211.1
59.0
T-HAN-ens
56.5+1.0 60.1
60.411.2
61.210.8
62.510.7
60.1
Multiple
DECISION
54.1+2.1 54.9#1.8 55.611.6
56.5+1.2
58.1+1.2
55.8
Bi-ATEN
56.2$1.9 58.411.6
58.9+2.2
61.511.3
61.711.1
59.3
HiFE (ours)
62.512.5
65.111.5
64.411.2
64.310.9
64.820.8
64.2
woOs-
48.811.7 48.5
48.811.2
49.3
51.210.8
49.3
Single
SHOT-best
78.4+1.6 81.1 81.811.1 82.311.2
83.211.0
81.4
SHOT-strong
76.2#1.9
76.2
75.7
75.2
80.1
76.7
D W
SHOT-ens
79.3+1.5
80.1
80.211.0
82.111.1
83.810.7
81.1
TOHAN-ens
93.1 96.510.9
97.1
97.511.2
97.7
96.4
Multiple
DECISION
95.611.2 95.311.0 95.320.8
95.311.0
95.611.2
95.4
Bi-ATEN
95.241.9 96.111.7
96.5
97.1
97.741.3
96.5
HiFE (ours)
96.610.6 97.3 97.
97.510.4
98.020.6
97.4
wHOs-worst
60.911.2 57.8
55.9#1.1
55.510.9
53.910.8
56.8
Single
SHOT-best
98.611.3
98.6 98.4
99.610.8 99.611..8
99.0
SHOT-strong
98.640.5
98.610.4
98.210.5
98.810.3
99.640.9
98.8
W -
SHOT-ens
99.240.4
99.210.3
99.310.4
99.510.3
99.210.3
99.3
TOHAN-ens
99.2#0.2
99.4
99.3#0.2
99.410.4
99.550.2
99.4
Multiple
DECISION
99.510.6
98.940.6
99.320.3
99.210.4
99.240.1
99.2
Bi-ATEN 99.410.6
99.610.4
99.420.6
99.310.3
99.61:..4
99.5
HiFE (ours)
99.5 99.610.11 99.6.0.11 99.610.3 99.910.2
99.6
Table 9: Classification accuracydstandard deviation(%) on six adaptation tasks of office datasets. A, D, and are abbreviations of Amazon, DSLR, and Webcam. The suffixes -bst and worst refer to the best and worst results after adapting each single source hypothesis. The sulfixes -strong and -ens refer to the result after adapting the strong hypothesis and the ensemble of all adapted hypotheses, respectively. Results of SHOT (Liang al.] 2220)), TOHAN (Chi et al. 2021). DECISION Ahmne al. 2021 and HiFE are presented. The highest accuracy is marked in bold.
23

---

Published in Transactions on Machine Learning Research (09/2024)
Number of Target Data per Class
Taske
Methods
Average
SHOT-ens
1.2
1.
56.742.1
59.1
60.94:1.2
56.4
D, W
TOHAN-eDs
67.1+1.4
61.7+1.2
03.2+1.1
64.1
05.1
62.2
DECISION
53,3+1.7
54.1#22
55.2#2.1
55.9+166
57.1+1.2
55.1
BLATEN
55.41.1.3
55.312.1
58.711.6
60.111.7
62.51:.5
58.4
HiFE (ours)
63.2:0.55
65.520.66
67.8=0.8
68.1+0.7
68.3#0.4
66.6
D - A W - A
HIFE (ours)
61.8+1.0
61.7200.7
67.240.6
66.8+1.0
07.5+0.9
65.6
HIFE (ours)
62.5+2.5
65.1
64.411.2
64.3+0.9
64.8+0.8
64.2
Table 10: Classification accuracy-ssanndar deviation (%) on multi-domain multi-hypotheses adaptation task from the office datasct DSLR (D), Wcbcam(A) to Amazon (A). SHOT-ens (Liang et al. 2020), TOHAN-
ens (Chi et al.) 2021) refers to the result of the ensemble of all corresponding adapted source hypotheses.
The highest accuracy (%) is marked in bold. We also list the results of single-source adaptation tasks D - A and W + A of HiFE for further comparison.
G HiFE for Partial Few-Shot Hypothesis Adaptation
We extend HiFE to partial few-shot hypothesis adaptation scenario. The office dataset contains three domains, including Amazon (A), DSLR (D), and Webcam (W), with each domain containing images of 31 object categories. We use images from the first 17 classes as the target domain and images from all 31 classes as the source domain. In this way, we obtain six partial domain adaptation tasks. HiFE is also better than the SOTA in the partial fow-shot hypothesis adaptation scenario (sec details in Table 11
Number of Target Data per Class
Tasks
Methods
Average
SHOT-ens
82.5+22.2
82.8+1.9
83.0+1.1
84.6+1.6
8..0+1.0
84.0
TOHAN-ens
83.612..1
83.622.0
89.2
91.1=1.3
90.7+1.6
87.6
A
DECISION
83.0+:2.5
83.312.1
84.7116
86.3119
87.0+1.2
84.9
HiFE (ours)
89.0#22.1
89.6+1.3
89.2=2.2
19.7
91.8+1.1
89.9
SHOT-ens 61.0
63.4+2.1
62.811.6
6..7
68.1
64.7
TOHAN-ens 64.24:2.11
66.5+1.5
71.0114
70.6:0..
69.641.1
68.4
D
DECISION
58.7#1.4
58.4+1.7
59.7+2.0
61.1+11.
61.0+1.9
59.8
HiFE (ours)
65.3+22.0
77.0+1.5
76.9=1.1
76.2=0.8
77.4+2.1
74.6
SHOT-ens 55.9112.0
59:11.8
60.011.6
64.2:10.9
65:1..2
60.8
TOHAN-ens 59.4.10.7
60.3.10.8
62.111.9
64.2.L1.
70.6.11.4
63.3
-A
DECISION
54.5+1.2
55.9+1.9
58.0+3.0
60.0+22.2
63.6+2.0
58.4
HIFE (ours)
67.1+1.8
72.0+1.0
71.3=2.1
71.9=2.1
72.2+0.9
70.9
Table 11: Classification accuracies (%) on six partial few-shot hypotheses adaptation tasks office datasets. A, D, and W are abbreviations of Amazon, DSLR, and Webcam. Results of SHOT (Liang et al. 22020), TOHAN (Chi et al. 2021), DECISION (Ahmed et al. 2021 and our HiFE presented. The bold value represents the highest accuracy (%).
H Complexity Analysis
We compare the time complexity and parameter size of our framework HiFE, single model, and the ensemble of all models using DECISION in Table 12 The experiments are conducted using the digit dataset for the SVHN to Mnist task with a LeNet model backbone. The batch size, the number of target samples per class, and the number of pre-trained models are set to 128, 7, and 8, respectively. HiFE, being a multi-hyypotheses adaptation approach, incurs additional computational costs compared to the single model bascline. Specifically, the cost is approxinately 8 times higher, corresponding to the number of pre- trained models. While HiFE has only slightly more parameters than the ensemble method DECISION, it is significantly more efficient in its usage. Overall, HiFE strikes better balance between performance and efficiency for multi-hypothees adaptation.
24

---

Published in Transactions on Machine Learning Research (09/2024)
Model
Acc (%)
Params (M)
Memory (M)
Speod (samples/s)
LeNet (Single)
46.2 82.2
2.1
16
142201
DECISION (Multiple)
16.1
128
17851
HiFE (our) (Multiple)
89.2
16.6
131
17758
Table 12: Performance and complexity of HiFE, ensemble method DECISION and the LeNet models.
25


Reviews:
[{"review_id": "QmmtAwekQn", "reviewer": "Reviewer_1C1M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- idea on exploiting the weak models along with strong for domain adaptation\n- reasonable problem formulation for domain adaptation\n- new block on how to combine features from different models\n\n**Weaknesses**\n- empirical analysis is weak (see requested changes) as model are very outdated and datasets are very simple (having that there is no theoretical analysis)\n- definitions on the weak and strong are unrelated to the empirical analysis and how weak models are defined. There is no connection to overfitting, which make it questionable how to select the weak models (see requested changes)\n- no analysis / ablation how to select the weak models (see requested changes)\n- the method seems not to be batched for the inference and input dependent (for the merge operation) - so in practice it will be slow for the data as e.g. images (see requested changes)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper is about domain adaptation: authors formulate a new problem on how to use set of weak models (hypothesis) in addition to strong model trained on one source domain to perform finetuning (domain adaptation) on the target domain with limited labeled data. Authors advocate that this formulation of the problem is realistic and practical - we can store several snapshots of the same model and providers can give them w/o leaking the source private data. \n\nAuthors show that 1) usage of weak models is helpful and ensembling of them and strong model improves domain adaptation to the target domain with limited labeled data 2) authors propose a new block how to ensemble the weak and strong models via residual hierarchical ensembling in the feature space. Evaluation is purely empirical with set of the small datasets like MNIST, SVHN, CIFAR-10, STL-10 and USPS and Resnet / LetNet models showing that proposed framework outperforms other SOTA models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper itself is well written and reads smoothly and clearly. I have several suggestions for couple of the places for the readability and formatting:\n- In the introduction and contribution it is worth maybe mention ensembling in general and compare with it, otherwise while reading I am in question what you did exactly as this is known in the boosting / ensembling that weak models together improve generalization.\n- page 3 \"to encoder the source knowledge\" -> \"to encode ...\"\n- page 3 \"amounts of unlabeled\" -> \"amount of ...\"\n- MNIST is used in many places as \"mnist\" or \"Mnist\"\n- page 4 \"In FHA, the research ...\" there is missing verb - so sentence in unclear from English language.\n- notation $h_\\theta$ and $\\theta_h$ - a bit misleading as the use is mixed. Maybe use only  $\\theta_h$?\n- \"Which feature to merge\" - is the merging done only between features from the previous layers? I think a bit clear description is needed here. Do we also recompute grouping after every layer? \n- Is merging order depends on the input itself? I mean that for every sample the merging order will be different? It means it is not parallelizable at all, which makes this **really impractical** as it is not batched then. Any comments on this?\n- One point authors need to highlight is that their method definitely reduce std\n- \"matrixes\" -> \"matrices\"\n- Appendix \"we shown\" -> \"we showed\"\n- Appendix theorems and lemmas - not clear if this is contribution from authors or it is prior work\n- \"In our HiFE, We\" -> \"In our HiFE, we\"\n- What is speed for the framework? - mainly for the inference - what is additional overhead?\n\nRegarding the empirical analysis I have the following thoughts:\n- Fig 1 motivation plot - I wanna to see the y-axis to be exact accuracy numbers with std, as otherwise this could be misleading and maybe models are different in a very little performance in ranking within std even. Also this plot I found controversial as you are showing that model 2/4 are top, while model 1/3 are not and there is no really correlation between model performance on the source domain and the domain adaptation performance\n- Definitions of weak and strong models is strange as it is based only on the training loss and not validation loss. During the empirical analysis authors in contrary used the accuracy range (I presume computed on the validation) which then contradicts the given definition. I think some work here should be done on clear understanding what weak models should be taken and maybe it should be level of overfitting (diff between train and test loss) and also the validation accuracy itself. In the main text authors speak a lot about overfitting, but this metric never used / analysed in the empirical part of the paper.\n- As there is no new theoretical proofs / results in the paper and it is purely empirical (which is totally fine) I believe empirical analysis should be done with more realistic data too, not only small outdated datasets like MNIST and CIFAR and small old models like Lenet. I agree that a lot of classic models are used in many applications like MRI where the data are very limited and we don't need to have fancy models. However, either you consider more practical datasets and close to current models usage in the applications or switch to e.g. iWildCam setting or at least Imagenet + CIFAR. Also very simple setting is considered like 10 classes only. \n- There is no understanding what happens in more realistic scenario of more classes and moreover different distribution of classes in the source domain and target domain (e.g. heavily unbalanced classes in the source domain). \n- There is no ablation why only strong encoder is updated, and why all weak are kept?\n- What is performance if we use simple ensemble of models w/o finetuning on the target domain?\n- Deeper analysis / results discussion on how we depend on the number of samples in the target domain - it seems e.g. it doesn't matter what weak models to take if there are few samples, while for more samples it is better to have stronger weak models\n- There is no discussion on how to select the weak classifiers at all, and from Tables it seems we depend on it a lot.", "broader_impact_concerns": "I don't see ethical concerns as the paper about general ML method which targets improvements for the generalization on out of domain data.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZllJjCA7LO", "reviewer": "Reviewer_s9f7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The author has made sufficient analysis and deductions of the theory in the paper.\n2. The author has conducted sufficient ablation study on proposed framework to prove method efficiency.\n3. The paper has cited enough relevant paper to prove the reliability of the viewpoint.\n4. The author provides the complete code for verification\n5. The writing of whole paper is excellent.\n\nWeaknesses:\nSome details need more explanation. e.g. 'two or more features with the highest similarity and merge them into a new feature for the next layer'", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. Points out that in the few-shot hypothesis adaptation task, using only one single strong model will lead to model overfit SD and poor performance in TD.\n2. Proposed a new problem: FHAW to enhance the diversity of source models.\n3. Proposed a new framework to solve FHAW problem and achieve excellent performance on multiple benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The feature that needs to be merged needs more discussion. Why select highest similarity rather than similarity more than a threshold?", "broader_impact_concerns": "Nothing.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9SLv13JbvW", "reviewer": "Reviewer_Qbkv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n* Leveraging data from a variety of sources to improve transfer learning has obvious tangible benefits to the community\n*Performance is evaluated on several different datasets across digits, office, and broad image classification.  Comparison to several baselines are made- these are reasonably chosen.\n* Results are shown with uncertainty values\n* Ablation studies capture the impact of the DeCL loss and number of hypotheses.  \n\nWEAKNESS:\n* While the writing is clear grammatically, there could be improvements made to the overall flow and some of the sentences to improve understanding.  Specifics are given in the Requested Changes.  \n*The main contribution is around proposing multiple weak hypotheses as opposed to multiple strong or a single weak hypothesis.  I'm not sure I would consider this a novel \"problem\" as the authors present it.  Ultimately the goal is to perform source to target transfer.  Whether that is done with a single or multiple (weak or strong) hypothesis, doesn't make it a new task, but simply a different approach.  \n* The importance of the problem is introduced in the paper as having privacy importance for fields like medical.  However, none of the experiments address this application.  I would like to see this method tested on a \"hard\" or \"real world\" dataset, instead of just academic datasets.\n*From the ablation studies it's not clear if the recommendation around the number of weak hypotheses is specific to these task/data or whether a different task/dataset/amount of data would impact this choice.  Similarly is the limit on the performance of the weak hypothesis relative to the overall (potential) strength of that model, or is that unique to this data?\n* All of the experiments revolve around image classification.  Is this the only task where this is valid?  What about other image tasks such as segmentation?  What about non-image tasks?  I think addressing this limitation is key for it delivering significant value to the community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a hierarchical feature ensemble framework which is useful for source-free domain adaptation.  This approach combines features from both strong and weak leaners which enables the use of incorporation of features learned from weak, intentionally underfit source models.  These models enable greater generalization for the target domain.  This framework shows enhanced performance on few-shot hypothesis adaptation scenarios.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While overall the writing is relatively clear, some of the sentences are a bit awkward.  For example, page 1 opening sentence of 2nd paragraph \"as indicated by the results of an experiment\".  Is this referring to a specific experiment (if so cite), or is this referring to a subsequent abstract test the person performing the adaptation makes?  Sentences like this could be clearer.  The following sentence \"we adapt models...of SVHN to the target task of MNIST\" makes that sound as if it is the ultimate goal of the approach, which it is not.  The method may be demonstrated on this task (\"we demonstrate our approach on the toy example of ....\"), but just a change in the language would help frame this better.  The authors reference a meaningful example within the medical space, but it's not sufficiently explored or explained, IMO.\n\nSimilarly Figure 1 is discussed specifically in terms of the SVHN to MNIST transfer, so it's unclear whether the details are related to this specific task, or more broadly.  In general, I would suggest Figure 1 be broadly capture the approach so that the readers know immediately what the problem, solution, and impact is.  Focusing on such a narrow result leaves the reader with a lot of questions after the introduction.  Futhermore, Figure 1 is not particularly clear in illuminating what problem is being solver or how this particular problem solves it.  One alternative could be to pick a tangible relevant examples from these different domains and show how they are integrated. \n\nDefinition 2: \"M is the hypothesis number\".  \"m\" is the hypothesis number/ID and \"M\" is the number of hypotheses.  \n\nTable 1 is very unclear\n\nTo better speak to the impact in areas with privacy concerns which the authors raise, I think it is important to show these results on a real world or difficult dataset and not just toy datasets.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes HiFE which studies how to conduct effective domain adaptation by combining multiple weak checkpoints with a strong one. The reviewers generally find the paper to be well-written and the idea to be interesting and reasonable. The authors have done a good job answering the reviewers' questions and incorporating their very detailed feedback. The reviewers are generally satisfied with the paper after revision.
Decision: Accept (poster)