What Has Been Overlooked in Contrastive Source-Free Domain Adaptation: Leveraging Source-Informed Latent Augmentation within Neighborhood Context
Authors: Jing Wang
Wonho Bae
Jiahong Chen
Kuangen Zhang
Leonid Sigal
Clarence W. de Silva
Venue: Accepted by TMLR ()

Abstract: Source-free domain adaptation (SFDA) involves adapting a model originally trained using a labeled dataset (source domain) to perform effectively on an unlabeled dataset (target domain) without relying on any source data during adaptation. This adaptation is especially crucial when significant disparities in data distributions exist between the two domains and when there are privacy concerns regarding the source model's training data. The absence of access to source data during adaptation makes it challenging to analytically estimate the domain gap. To tackle this issue, various techniques have been proposed, such as unsupervised clustering, contrastive learning, and continual learning. In this paper, we first conduct an extensive theoretical analysis of SFDA based on contrastive learning, primarily because it has demonstrated superior performance compared to other techniques. Motivated by the obtained insights, we then introduce a straightforward yet highly effective latent augmentation method tailored for contrastive SFDA. This augmentation method leverages the dispersion of latent features within the neighborhood of the query sample, guided by the source pre-trained model, to enhance the informativeness of positive keys. Our approach, based on a single InfoNCE-based contrastive loss, outperforms state-of-the-art SFDA methods on widely recognized benchmark datasets.

Published in Transactions on Machine Learning Research (04/2024)
What Has Been Overlooked in Contrastive Source-Free Do- main Adaptation: Leveraging Source-Informed Latent Aug- mentation within Neighborhood Context
Jing Wang
jing. wong Bmnghueeh.ube.... ubc.cc
Department of Mechanical Engineering University of British Columbia
Wonho Bae
whbae ubc..a. ca
Department of Computer Science Universily of Brilish Colambia
Jiahong Chen
jiahong Sjahong.chen@kkeer.oyy
Department of Mechanical Engineering University of British Columbia
Kuangen Zhang
kuangon.ahangg@akannd ubc.
Department of Mechanical Engineering University of British, Columbia
Leonid Sigal
Isigol@cs. ubc.ca
Department of Computer Science Universily of Brilish Columbia
Clarence W. de Silva
desilva @mech, ubc.
Department of Mechanical Engineering University of British Columbia
Reviewed on OpenReview: https: // ope.review. net/ forum? id- dulMde3dP1
Abstract
Source-free free domain adaptation (SFDA) involves adapting a model originally trained using a labeled dataset (source domain) to perform effectively on an unlabeled dataset (larget do- main) without relying on any source data during adaptation. This adaptation is especially crucial when significant disparitics in data distributions exist between the two domains and when there are privacy concerns regarding the source model's training data. The absence of access to source data during adaptation makes it challenging to analytically estimate the domain gap. To tackle this issue, various techniques have been proposed, such as un- supervised clustering, contrastive learning, and continual learning. In this paper, we first conduct an extensive theoretical analysis of SFDA based on contrastive learning, primarily because it has demonstrated superior performance compared to other techniques. Moti- vated by the obtained insights, we then introduce a straightforward yet. highly effective latent augmentation method tailored for contrastive SFDA. This augmentation method leverages the dispersion of latent fcatures within the neighborhood of the query sample, guided by the source pre-trained model, to enhance the informativenesss of positive keys. single InfoNCF-based contrastive loss, outperforms state-of-the- art SFDA methods on widely recognized benchmark datasets. The code for our implemen- tation: https: http://gtbbcmmm//gvvaaaoto
Our approach, based

---

Published in Transactions on Machine Learning Research (04/2024)
1 Introduction
Supervised learning has proven successful in mimicking human behaviors in situations such as manipu- lation Muih et al. (2015), recognition Russakovsky et al. (2015), and understanding Fawzi et al.] (2022) primarily due to its accessibility to vast amounts of labeled data. However, this success hinges upon the assumption that both the training and the test data originate from the same underlying probability distri- bution, which often does not hold in various real-world scenarios. Consequently, model performance tends to deteriorate when applied to novel (largel) data domains such real-world images whose underlying data distribution markedly deviates from that of the training (sowrce) domain (e.g., computer gentreeed images). This divergence in data distribution is commonly referred to as domain shift Ben-David et al. (2010). Domain adaptation (DA) tackles the performance degradation that stems from the domain shift, by acquiring representations that remain invariant to such shifts Ganin & Lempitsky (0015).
Lately, a significant and practical challenge has emerged, known as source-free domain adaptation (SFDA). This challenge revolves around the goal of developing the domain-invariant representation without using any labeled source data during the adaptation. This shift in focus is motivated by real concerns related to training data privacy and intellectual property Liang et al. (2020); Yang et al. (2021b) : Huang et al. 2021a
Zhang et al. (002)); Yang et al. (2022). The inability to access the source domain during adaptation adds complexity to the task of estimating the degree of domain shift, making it challenging to learn a shared roprosentation that bridges divergent domains. In response to this, the concopt of contrastive lcarning has attracted significant attention. In this approach, discriminative clustering plays a crucial role in defining the decision boundaries that guide the classification efforts, by utilizing these clustered groups. However, in scenarios where labels are unavailable, the decision boundarics inferred from these clusters may not consistently align with the actual classification boundaries based the target labels.
Existing contrastive SFDA methods can be categorized into two types of approaches, based on how they generate positive keys for contrastive learning, whether neighborhood- based or augmentation-basel The neighborhood- based approaches rely on transferring neighborhood information from the source to the target by initializing the model for target adaptation with a model pre-trained on the source domain Yang et al., (2022). However, as the adaptation progresses and the model undergoes unsupervised clustering, the sig- nificance of this source domain neighborhood information diminishes. Conversely, the augmentation- based approach establishes source-like set with distribution aligned with the source domain during pre- vre-training for generating pscudo-labels. Subscquently, it employs contrastive learning with data augmentation to facili- tate discriminative clustering during larget adaptation Zhang et al. 2022]. However, this method encounters two significant problems. Firstly, like aseudo-labeling based SFDA, its performance can be negatively af- fected by the presence of noisy labels Liang et al. (2021a). Secondly, achieving an optimal tradeoff between distribution alignment and contrastive clustering is challenging, especially when the source data is inacces- sible during larget adaptation.
The primary goal of our study is to conduct thorough analysis of how the design of positive keys could impact target domain classification performance. Moreover, we seek to explore how insights obtained from this analysis can be utilized to enhance contrastive SFDA frameworks. This understanding is grounded in empirical observations within the latent feature space, as illustrated in Figure In this figure, we can observe that target features extracted from the source pre-trained encoder exhibit siguificant dispersion because of domain shift, leading to increased target classification errors. Despite this feature dispersion, it worth noting that nearby target features often share the same ground truth labels. These two observations are elaborated upon as follows:
Observations on Target Feature Dispersion: A domain shift induces a significant dispersion of target features, as extracted from the source pre-trained encoder (as clear when comparing the t-SNE plots for source data features and target data features). This dispersion reduces the discriminative quality of the associated features, consequently making it difficult to effectively classify samples within the target domain.
Observations on Neighborhood Informativeness: Despite the reduced discriminability, which leads to the lack of well-defined target feature clusters extracted from the source pre-trained encoder, neigh-
2

---

Published in Transactions on Machine Learning Research (04/2024)
Source Pretrain
Output Vectors
motorcycle
matorcycle
Maximize
Alignment
Source
Source Classifier
Encoder
motorcycle
Minimize Alignment
bike
Target Images
Source Feature L-SNE
Target Feature I-SNE
Figure 1: t-SNE visualization of target features extracted by the source pre-trained encoder, revealing significant. feature dispersion due to domain shift. However, nearby larget samples still tend to share similar ground truths, motivating our in-depth exploration of contrastive clustering in SFDA.
boring target features still tend to belong to the same ground truth class. This suggests that valuable information, w.r.l larget ground truth, likely exists within the local vicinity of target features.
Motivated by these observations, conduct comprehensive analysis of the underlying principles of con- trastive SFDA and explore aspects that have been overlooked in the existing contrastive SFDA frameworks. To be specific, we identify three overlooked factors: 1) standard data augmentation techniques might not effectively reduce the likelihood of the model misclassifying positive transformations; 2) increasing the num- ber of nearest neighbors, i.e., larger results in smoother predictions but also leads to greater overlap of logit clusters; and 3) effective utilization of the source pre-trained model to leverage neighborhood la- bel consistency for enhancing the informativeness of positive key generation has not yet been explored. Subsequently, we introduce our Source-informed Latent Augmented Neighborhood (SiLAN) method, built upon the acquired theoretical insights. This involves applying Gaussian noise to the latent features of the neighborhood centroid of a target query sample, mirroring the source model's standard deviation latent features from the neighboring target samples, to enhance positive key generation. This latent augmentation is consistently applied during the positive key generation process for cach target query sample, combining the benefits of both neighborhood exploration and data. augmentation. Aligning the standard deviation of the random noise with that determined by the source pre-trained model allows contrastive clustering to ef- fectively leverage the dispersion of the neighbors' features, which is often regarded potentially detrimental to the model's billii Our empirical findings demonstrate that optimizing an InfoNCE- -bseed con- trastive loss Oord et al. (2018); Chen et al. (2020) alone, combined with our SiLAN augmentation method, yields state-of-the-art performance across a range of benchmark SFDA datasets.
In summary, our paper's contributions are outlined follows:
We hypothesize that domain shift causes significant dispersion in target features yet nearby points still tend to share similar labels, explaining the success of contrastive clustering in SFDA.
Our thcoretical analysis of contrastive SFDA reveals that three often-overlooked factors, associated with the aforementioned hypotheses, have significant implications for target classification performance.
To address these three issues, introduce SiLAN, a simple yet effective latent augmentation technique explicitly designed to improve contrastive SFDA.
3

---

Published in Transactions on Machine Learning Research (04/2024)
Experimental results support theoretical findings, demonstrating that InfoNCE, when augmented with SILAN, achieves state-of-the-art performance in SFDA.
2 Related Work
2.1 Source-Fres Free Domain Adaptation
The existing methods for SFDA can be grouped into two categories: contrastive and non-contrastive. Non- contrastive methods rely on extra guidance, such pseudo labels Liang al. (2020) 2021b) or samples generated through adversarial learning Li et al (2020), from the source pre-trained model during adap- tation. However, this additional guidance can potentially have detrimental effects on target classification performance Li et al. (2020). In this paper, we focus on contrastive SFDA methods, which have shown better performance. Attracting and Dispersing (AaD) Yang et al. (2022) ident nearby samples to a target query using neighborhood searching and directly uses their latent features as positive keys. Historical Contrastive Learning (HCL) Huang et al. (2021a) introduces an instance-wise loss to enhance fata-augmentation basod contrastive learning for larget adaptation. Divide and Contrast (DaC) [Zhang el. al. (2022) investigates con- trastive clustering using augmented inputs, effectively applying it to SFDA. However, existing contrastive SFDA methods overlook the alignment of different domains during target adaptation. In contrast, our approach incorporates domain alignment into the contrastive clustering process, which is achieved while optimizing single contrastive loss.
2.2 Latent Augmentation
Data augmentation can change the semantics of input samples by causing major shifts in the latent space Up- church et al. (2017)- These techniques also vary based data modality. Conversely, augmenting in the latent space Cheung & Yeung (2020) offers diverse transformations with lower risk of altering the seman- tics of queries and is not dependent on the data modality. Common latent augmentation techniques include interpolation, extrapolation, random translation, and adding Gaussian noise DeVries & Taylor (2017). which have been shown effectiveness in various fields such as computer vision Liu et al. (2018); Stutz et al. (2019) natural language processing Kumar et al. (2019), and graph representation learning Cheng et al. 20022)
this paper, we investigate using latent augmentation tailor the variance of the random noise used in latent augmentation according to the feature cluster variances from the source pro-trained model. This key technical distinction differentiates our method from the existing latent augmentation approaches.
address challenges contrastive SFDA. Specifically,
2.3 Contrastive Learning
Contrastive learning is an effective framework for training models to differentiate between samples, which is versatile, and suitable for both supervised learning Khosla al. (2020) and self-supervised learning (SSL). Contrastive SSL methods utilize the InfoNCE-bart based loss draw samples (positive keys) closer to a query sample if they are similar, and to push away samples (negative keys) that dissimilar, all within designated embedding space. These methods often require large batch sizes for an effective contrasting Chen et al. (2020), additional memory banks for momentum updates He et al. (2020), or additional negative sampling strategies Hu et al. (2021). In our study, we use an InfoNCE-based contrastive loss, applied to the output logit space, to address SFDA challenges.
3 Source-Free Domain Adaptation
In this scction, we outline the problem sctup for SFDA and explain the implementation of contrastive lcarning solve it. We also define the concept of neighborhood in our study, a key element for our theoretical analysis and proposed method.
4

---

Published in Transactions on Machine Learning Research (04/2024)
3.1 Problem Statement
In customized applications, users typically possess their own data and model pre-trained on large dataset such as ImageNet [uusskkosky et al. (2015), but do not have access to the model's original training data. SFDA methods aim to address this by enabling the model to adapt to the new data domain (target domain) without needing access to the original training data domain (source domain).
Pro-training on sourcc domain We define a dataset from a source domain as D3 == [(x], (os..y)) ) where x denotes the i-th data sample from the source dataset, and y is its corresponding class label. The objective of the pre-training stage is to derive model fs F3 G. (referred to the source model) that minimizes the classification error on the source data eDs (f.) ;- Σ P(Ja(x.) o y'l. where G is a feature extractor that encodes imput samples into latent features, and is a task-specific classifier.
Adaptation on target domain Consider a dataset sampled from target domain, denoted Dr {x) M where x refers to the i-th data sample from the target dataset. During adaptation, it is important to note that class labels for all target samples are not accessible. In this stage, the goal is to have a target model ft that minimizes the generalization error on the target data (Di = > zi-1 P(fa(x)) + vil, possibly guided by the source pre-trained model fa without having access to the source data. One example of leveraging the source pre-trained model fa is to initialize the parameters the target model ft with those of fs.
3.2 Contrastive SFDA
Contrastive learning is a widely recognized algorithm for SSL. It operates on the principle that data samples can be effectively clustered within a specific embedding space, space,, the resulting model has the potential to exhibit robust performance across various downstream tasks, provided that fine-tuning conducted appropriately.
In the context of SFDA, we choose to formulate the contrastive objective within the output logit space instead of the feature or embedding space used in SSL. This decision is motivated by the use of equal number of classes for classification in both the source and target domains. Note that in our empirical findings, there is no noticcable difference between performing distribution alignment on the output logit space and the output probability space. To simplify the mathematical notations, we conduct all theoretical analysos focusing on the alignment of output logits. For clarity, the output probability vector the softmax output of logit vector. For any input x, the sum of all elements in the output probability vector equals one, where Z denotes the number of classes
i.e. s=1 (Soffmaz(f(()))
In comparison to SSL., formulating in either the logit or probability space simplifies the direct clustering of data samples according to the number of ground truth classes.
3.3 Definition of Neighborhood
To establish neighborhood, we first define feature bank := = (G(x)) x E D where G is feature extractor initialized with the parameters of G, and updated throughout the adaptation. Similar to the neighborhood discovery used in unsupervised representation learning Huang et al. (019); Van Gansbeke et al. [2020), given & query sample, we define a neighborhood of the sample as its K-Nearest Neighbors (K-NNs), with proximity determined by the cosine similarity between the features of the query sample and those of another sample drawn from Br:
G(x)) ByJ |[G()) ||BTIll
d( d(x), x/)
(1)
where BTy denotes the j-th element of BT corresponding to the features of data sample x/. Thus, the neighborhood Ng (x)) of x, comprises the top K similar samples with respect to feature similarity, defined as:
Ng (x)) = arg max marscRr.
Σ d(x;,x).
(2)
XES
5

---

Published in Transactions on Machine Learning Research (04/2024)
The centroid of the neighborhood, denoted px(x'), is the mean of the feature vectors of each sample within Nx (x+).
4 Theoretical Analysis of Contrastive SFDA
In this section, we engage in theoretical analysis concerning alignment errors when performing contrastive learning in the output logit space. In the meantime, we forge links between these error terms and the ground truth labels of target samples in the context of SFDA. This theoretical analysis not only sheds light on the crucial aspects of SFDA that nocessitate attention from the rescarch community, but it also clarifies the existing research gaps that our work endeavors to fill. All the prools can be found in the appendix.
4.1 The Behavior of Contrastive Loss SFDA
We start by closely examining how TnfoNCF-based contrastive loss can be applied to address SFDA problems. For brevity, we use the term trunsformation to encompass the process generating positive samples x* from the query x, whether through neighborhood searching or data augmentation. The translormations of other samples within the same mini-batch are used as negative samples x
Cosine similarity is commonly used in SSL to formulate contrastive loss within high-dimensional embedding space. However, when applying contrastive loss in the output logit space for SFDA, further clarification is necessary. To provide a clearer understanding of this modification, we begin our discussion by redefining an upper bound for the contrastive loss formulated the output logit space. This refinement allows us to focus on analyzing the alignment of output predictions through contrastive loss, using Euclidean distance as our measure.
When formulated in the output logit space, the InfoNCE-based contrastive loss, denoted as Lcont, serves as an upper bound for the misalignment associated with two distinct alignment errors in predlcctions.
Proposition 1.
m2
log(m- 1) f((x)) - f((xt)
[1f((x;)- - fe((x))1BB
Lcont.
2T
m
jti
Here, т represents the lemperature parameter, m is the mini-batch size, X5 denotes a negalive sample x given the query X,, and Ccont defined as follows:
el (x))117))/
Leont Σ log Ejjt ell (x)fdx))/7*
The proof for the proposition is provided in Appendix B.1 According to Proposition minimizing the InfoNCE loss involves reducing the alignment error between the predictions of the query sample and its positive key xt, while introducing misalignment with the predictions of transformations applied to other samples within the same mini-batch. Therefore, optimizing the InfoNCE loss formulated in the logit space leads the formation of discriminative clusters, however, it does not guarantee the alignment of these clusters with the ground truth.
4.2 What Has Been Missing in Contrastive SFDA
Now, let us begin cstablishing a connection between the prediction alignments introduced by contrastive loss and the larget classification error in a lask involving Z classes. With = denoting an index, which corresponds to specific ground truth class, for a group of output logits. C2 represents a set of target samples belonging class z and having prodictions that are close to each other in the output logit space. Within each C,, there exists subset o that exclusively comprises samples predicted to be class and exhibits no overlapping reasonably assume that there exists Fuclidean space in
with any other Ct for I + Therefore,

---

Published in Transactions on Machine Learning Research (04/2024)
which the logits of this subset of samples can be enclosed within a ball C of diameter 8. The set of positive keys, whose logits can confidently be considered to lie within this ball, can be represented as:
s = {x* E C2 : Vx E C2 [1 - fx(x - d},
and a set of negative keys, whose logits are confirmed to exist outside the ball, is denoted as:
S = {x E C. : Vx E co [xx f+(x) > 63.
To establish a coherent connection between the target classification error and the prediction alignment errors, we introduce dcfinition for the target classification crror concerning the logit groups and their associated ground truth:
2 CDr Σ (P(f(())) # Vx, E C.] + P[2 + yı, VX, E C2l),
(3)
The first term represents the probability of misclassification by the classifier ft for a given group class x within the logit group C.. The second term indicates the probability that the ground truth for a sample x within the logit group C, does not align with the group class 2. This notation for classification error is inspired by the theoretical framework developed within the context of contrastive SSL Huang et al. (2021b).
Assuming that near- cear-perfect prediction alignments, as introduced in Proposition can be achieved by mini-
mizing Leont, we can derive an upper bound for the error defined in Equation 3 as follows:
Lemma 2. If C. c = @ holds for any # 2, then the error ED, defined on the groups of logits is upper bounded by:
DD S Rs + Σ (F))(x*) # yo, VX* +S = yv, Vx ES;1):
CC, c²) and (C= - CC may overlap with (C, - ct) for any + 2.
where Rg-
0 C.
2 suggests that the target classification error, as defined in Equation 3 is affected by the overlap between C2 and C, for any & z, as well as the misclassification of the transformations x* and x Given the connection between the prediction alignments introduced by contrastive loss and the target classification error, we will now explore how the two types of transformations, namely, neighborhood searching and augmentation, contribute to reducing the upper bound provided in Lemma 2
The proof for the lemma is provided in Appendix B.2 Lemma
Data augmentation. Augmentation-basee based contrastive SFDA methods augment data in the imput space and use the logits of the augmented view as the positive key for contrastive clustoring Zhang et al. (2022).
However, solely relying on the augmented view transformed in the input space has limitations in mitigating the overlap of clusters clustersHHunng et al. (202Tb). Standard data augmentation without distribution alignment in a contrastive SFDA framework may not effectively address misclassifications of positive keys due insufficient information about the target ground truth the first oversight.
Neighborhood searching. It involves using the logits of the query's K nearest neighbors, located in the latent feature space w.r.t the model parameters, as the positive key for contrastive clustering. However, mitigating the misclassification of transformations through neighborhood searching, akin to K-NN classifiers, requires an intensive search to find the most suitable K for the given task and data. Increasing the number of ncarest neighbors, i.e., larger K, results in smoother predictions but also leads to a greater overlap of logit groups, i.e., higher R& Wu al. (2002) the second oversight. Existing methods Yang et al. (2022) assume that the informativeness of query's neighborhood can be achieved by initializing the model f with the parameters of the source model f.. However, as the model is updated for contrastive clustering, the initially informative neighborhood information from fa diminishes. This underscores the necessity for a more effective utilization of fs during adaptation to enhance the informativeness of query neighborhoods and to further reduce the risk of misclassifying transformations the third oversight.
In summary, existing contrastive SFDA methods can mitigate either Rs (with data augmentation) or the misclassification of transformations (with neighborhood searching), but they cannot handle both of them.
7

---

Published in Transactions on Machine Learning Research (04/2024)
5 The Proposed Framework
In this section, we present our source-informed latent augmented neighborhood (SILAN), which leverages the advantages of both neighborhood searching and augmentation to address the previously mentioned oversights.
5.1 Source-informed Latent Augmentation
e During adaptation, Gs remains unchanged and is used to search the source-informed neighborhood (xi) for a given target query x4. For a given x we independently and identically (iidd) sample random noise & E R from (00 o 2 ((x)), where of (x{) the variance N (xf), and the dimension of a feature vector. We then add & to the latent features of the query's neighborhood (w.r. the current target model fi) centroid, denoted n (x4), to generate positive keys:
To clarify the proposed work, we introduce an additional feature bank denoted as Bs ==
Dr
h G:=G((uk(x))) +
(4)
Note that fi follows a Gaussian distribution, i.e., h
5.2 InfoNCE-based Contrastive Framework
To illustrate the integration of SiLAN into existing contrastive SFDA frameworks, we specifically employ the InfoNCE contrastive framework Oord et al. (2018: Chen et al. (2020): ; He et al. (2020) as an example. We
adhere to the settings of existing contrastive SFDA methods, formulating it within the output logit space for each mini-batch of size m.
CSILAN =
Σ log 1
(5)
Σ Ej4t (((((())) P/(h)//T
where G. and F; represent the feature extractor and task-specific classifier of ft, respectively. hy denotes the augmentation of other samples the same mini-batch, and t is the temperature parameter. Note that the focus of our work to examine the informativeness of the dispersion of Ni in the latent space. Consequently, refrain from incorporating additional techniques, such momentum update He al. (2020), to enhance contrastive clustering, despite their potential to improve the performance.
From an intuitive perspective, optimizing a contrastive objective with positive keys defined by ((; can be likened to applying forces that aat the predictions of h; which are informed by (x(),, towards the logit group where the query sample x belongs to. In the meantime, this optimization exerts a repelling effect on those hy that are located outside the logit group, pushing them away from
5.3 Integrating SiLAN into InfoNCE during Adaptation
To clarify the integration of our SiLAN into the InfoNCE contrastive framework, we provide a detailed description of the integration process in Algorithm which outlines the target adaptation phase when incorporating SiLAN.
5.4 Unveiling Rationale for Latent Augmented Neighborhood
Diversifying data augmentation tochniques in the input space enhances the concentration of augmented data Wang & Liu 22021), thereby reducing Ra. Yet, it introduces notable variations in the latent space due to minor changes in input, posing risk of altering the semantics of queries. To address this, we propose performing latent augmentation on the query's neighborhood centroid for positive key generation.
To understand the rationale behind our latent augmentation on the query's neighborhood centroid, let us first examine a general case where random noise to ~ N/ o2) is added to the latent features of μk (x). Such augmentation enriches the diversity of the neighborhood by allowing the transformation to traverse around a Gaussian profile whose radius is determined by its standard deviation o. A larger promotes extensive exploration, capturing diverse but potentially irrelevant data variations. Conversely, smaller a
8

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm 1:
InfoNCE for Target Adaptation
Input: mini-batch largel data Xt. Dr; source model F.(G,())); neighbor number K; temperature and maximum epoch N.
Initialization: Fr(G.(-)) + (.)G(())) and 0
while n S N do
3 Find K-NNs for Xt using G. to form their target neighborhoods (xt);
4 Find K-NNs for Xt using G, to form their source-informed neighborhoods No (xt):
Compute centroids for target neighborhoods: nk (x+) * Ei= N (x+);
Compute variances for source-informed neighborhoods:
(x+) R Σド= (G. (x))) K C.,
Sample noises from Gaussian with variance o "x2((xx (x+): & NO ok 2
Augment the latent features for the positive keys to guide contrastive clustering:
h= b (xx)) +
Optimize the model parameters for F+) to minimize the InfoNCE loss LSILAN:
Ea loy o m/bh//
Ejer (iGGe)""Raa))"" F.yy7
10
= n n +1;
end
restricts exploration local and fine-grained variations within a specific logit group. This flexibility enables the model to adapt to varying degrees of data complexity and distribution shifts, making it well-suited for reducing misclassification of the transformation in contrastive SFDA.
Moreover, traversing angmentations around Gaussian profiles enhances contrastive clustering by pushing non-overlapping logit. group regions apart, ensuring well-separated features. This improved separability aligns with the desired behavior as establishod by the thcoretical findings in the subsequent lemma. To be specific, for a query sample within a logit group C,, traversing its augmentations around the Gaussian profile (while aligning their predictions with that of the query through a contrastive objective) will stretch the non-overlapping region CØ and extend its boundary. Conversely, traversing the augmentations of its negative samples, which likely do not belong to the same category (diverging their predictions from that of the query), will push C farther away from other logit groups to which these negative samples belong, illustrated in Figure 2 This adaptive process encourages the model fr, optimized by contrastive loss, to dynamically reduce R₂ during adaptation.
To clarify the subsequent. lemma, consider Ge as a target feature extractor that converges on a contrastive objective with augmentation traversing a Gaussian profile with a. To quantify the gap between clusters post- convergence, employ the concept of calculating the effective region of Gaussian beam as introduced in Hogg & Lang (2013) Following their approach, we denote the noise variance per augmentation in the extrancous noise a Pext from the overlapping regions.
Proposition 3. If C. nej = holds for any z # I, and the assumption that the representation of query sample in the feature space remains close to those of its posilive augmentations and far away from those of its negative samples holds, then for all z + and + j (where x, E C and x, E Ct), their distance in the latent feature space has lower bound as we take the limåt o2 - O, as follows:
GG (x.) Go (x,) 3.1704a.
The proof for the proposition is provided in Appendix B.3 Here, the optimal aperture radius for a Gaussian profile approximates 1.5852o. In the optimal scenario, the Gaussian profile maximizes its transformation- to-noise ratio under the assumption that. o2 is close to 0.
Extrancous noise is term derived from physics that refers transicnt portion wave, which does not belong to the ambient waves, nor does originate from the source being studied. is important to distinguish this concept from the notion of random noise used for augmentation.
9

---

Published in Transactions on Machine Learning Research (04/2024)
source neighbors
query samples
Cluster I
Cluster
Cluster
Cluster
The transformation of query sample traversos Gaussian profile defined by sourco-informed neighbors in the latent space, initialing pull-and-push effect to reduce feature overlaps among logit groups.
Figure 2:
Similarly, let Fere denote the target classification head, which is linear in our work. With Proposition in considcration, we can derive a lower bound for the gap between any two logit groups in the output space when a Gaussian profile is used to augment the query's neighborhood features:
Lemma 4. Vz # and i i H if the linear classifier Fopt is T--bi Lipschits continuous, C2 ncf = holds for any z # l where x; E C and X; E C2,, and dezt + 0:
(x,= - ((t0(x 3.1704g
where popt F. (G (-) and the Lipschitz constont depends on the number of parameters of Ft..).
The proof for the lemma provided in Appendix fi()) converge on the contrastive loss with SiLAN augmentation for positive keys, the augmented views of the query's neighborhood centroid within Gaussian profile ensure lower bound, dependent the standard deviation of the Gaussian and the number of parameters of the linear classifier, for the minimum gap among the non-overlapping regions of the logit groups.
Lemma suggests that when the parameters of
Therefore, we can conclude that better separation among logit groups (larger non-overlapping regions) can be achieved by using a Gaussian profile with a higher standard deviation for SiLAN, as stated in Lemma However, if the Gaussian profile extends beyond the boundarics of the qucry's logit group the latent feature space (when is too large), the augmentation might traverse outside the logit group. This introduces ambiguity into contrastive clustering, leading to more errors in mislabeling transformations. To be specific, large for the augmentation might cause the predictions of positive keys to fall outside the query's logit group, leading to an increase in 22 ((F(((((( * ye. Vx* E S) Meanwhile, it could result in the predictions of negative keys falling within the query's logit group, leading to an increase in P[f((( = Yt. Vx E (.])).
5.5 Informativeness of
Recalling the phenomenon highlighted in Observations on Neighhorhood Informativeness, which suggests that the dispersed neighbors derived from (x,) carry valuable insights into the ground truth of x, (as nearby points share the same labels), the incorporation of this information becomes significant in the pro- oess of discriminative clustoring. Therefore, is crucial to perform SILAN a manner that covers these neighbors to reduce the likelihood of mislabeled angmentations. To achieve this, expanding the scope of the Gaussian profile's influence based the query's neighborhood determined by the source model is
10

---

Published in Transactions on Machine Learning Research (04/2024)
necessary. Specifically, the radius of the Gaussian profile should be determined by N (x,), thereby setting to ok.
In summary, the optimal value for should be large enough to create a substantial gap among logit groups, making them more distinguishable, while avoiding the generation of an excessive number of ambiguous augmentations. Interestingly, the dispersion of N (xx) that adversely impacts discriminability on Dr actually proves beneficial in determining the optimal value of for the proposed SiLAN.
6 Experiments
6.1 Experiments on Toy Dataset
In this section, we demonstrate the effectiveness of SiLAN on a toy dataset, the moon dataset, which simulates domain shift by rotating data sample orientations. The source domain features interleaving half cirele with 1000 data points, and the target domain replicates this structure with 30-degree rotation around the mean. Both domains incorporate Gaussian noise with standard deviation of 0.1 and are generated using distinct random seeds. The experimental setup includes 5-layer fully connected network. As depicted in Figure E the source-only model encounters challenges in accurately classifying numerous points, while our SiLAN method excels achieving accurate classification.
(a) Source Only
(b) SiLAN
Figure 3: (Best viewed in color.) Decision boundaries derived from moon datasct w/wo SILAN.
Table Comparison of SFDA methods using ResNet-50 on Office- 31.
The best results are highlighted.
Method
A-D
D-W
D
W-D
W-A
Avg.
ResNet-50 He et al.
2016
68.9
68.4
96.7
62.5
99.3
60.7
76.1
SHOT
Liang et
2020
94.0
90.1
98.4
74.7
74.3
88.6
NRC
Yang et al.
2021a
96.0
90.8
99.0
75.3
100.0
75.0
89.4
3C-GAN La al. 2020
92.7
93.7
98.5
75.3
99.8
77.8
89.6
HCL Huang et al. 2021a.
94.7
92.5
98.2
75.9
100.0
77.7
89.8
AaD Yang et al.
2022)
96.4
92.1
99.1
75.0
100.0
76.5
89.9
SF(DA) [Iwang al. 2024
95.8
92.1
99.0
75.7
99.8
76.8
89.9
SiLAN (Ours)
97.1
95.8
98.9
76.4
100.0
76.9
90.7
Table Comparison of the SFDA methods on Office-
Office-Home Home (ResNet-50).
Method
Ar-CI Ar-Pr Ar-Rw Cl-Ar Cl-Pr Cl-Rw Pr-Ar Pr- Cl Pr- Rw Rw-Ar Rw-Cl Rw-Pr Avg.
ResNet-50 He et al.
(0066)
34.9
50.0
58.0
37.4
41.9
46.2
38.5
31.2
60.4
53.9
41.2
59.9
46.1
G-SFDA
Yang et al
b021b
57.9
78.6
81.0
66.7
77.2
77.2
65.6
56.0
82.2
72.0
57.8
83.4
71.3
SHOT
Liang
2020
78.1
81.5
68.0
78.2
54.9
73.3
58.8
84.3
NRC
Y Yang et al.
2021a
57.7
80.3
82.0
68.1
79.8
78.6
65.3
56.4
83.0
71.0
58.6
85.6
72.2
AaD
Yang et al.
2022
79.3
82.1
68.9
79.8
79.5
67.2
57.4
72.1
85.4
72.7
DaC
Zhang
2022
59.5
79.5
81.2
69.3
78.9
79.2
67.4
56.4
82.4
74.0
61.4
84.4
72.8
SF(DA)
Hwang et al. (2024)
57.8
80.2
81.5
69.5
79.2
79.4
66.5
57.2
82.1
73.3
60.2
83.8
72.6
SiLAN (Ours)
58.2
81.2
82.5
69.8
78.6
80.3
68.4
58.6
82.5
75.6
60.8
86.1
73.6
11

---

Published in Transactions on Machine Learning Research (04/2024)
Table 3: Comparison the SFDA methods on 22i1 (ResNet--01))
Method
plane
beyel
bus
car
horse
knife meyel person plant sktbrd train truck
Avg.
Res 101
He
(2016)
55.1
53.3
61.9
59.1
80.6
17.9
79.7
31.2
81.0
26.5
73.5
8.5
52.4
SHOT
Liang et al.
2020
94.3
88.5
80.1
57.3
93.1
94.9
80.7
80.3
91.5
89.1
86.3
58.2
82.9
HCL
Iuang et al.
2021a
93.3
85.4
80.7
68.5
91.0
88.1
86.0
78.6
86.6
88.8
80.0
74.7
83.5
G-SFDA
Yang ct. al
2021bb
96.1
88.3
85.5
74.1
97.1
95.4
89.5
79.4
95.4
92.9
89.1
42.6
85.4
NRC
Yang et al.
2021a
96.8
91.3
82.4
62.4
96.2
95.9
86.1
80.6
94.8
94.1
90.4
59.7
85.9
AaD
Yang al.
20222
95.2
90.5
85.5
79.2
96.4
96.2
88.8
80.4
93.9
91.8
91.1
55.9
87.1
DaC
Zhang et al.
(2022
96.6
86.8
86.4
78.4
96.4
96.2
93.6
83.8
96.8
95.1
89.6
50.0
87.3
SF(DA)* 1 Hwaang et
(2024
96.8
89.3
82.9
81.4
96.8
95.7
90.4
81.3
95.5
93.7
88.5
64.7
88.1
SiLAN (Ours)
97.5
90,1
85.8
80.4
97.6
95.5
92.0
82.9
96.5
95.3
92.6
53.4
88.3
6.2
Experiments on Benchmark Datasets
Datasets. We have evaluated our SiL.AN on three benchmark datasets for SFDA:
Office-31 Saenko et al. 20100) which consists of 4,652 images of 31 object classes captured from three domains: Amazon (A), Webcam (W), and DSLR (D).
Office-Home Venkateswara et. al.
(0117) which has 15,500 images of 65 classes from four domains:
Artistic (Ar), Clipart (CI), Product (Pr), and Real-World (Rw).
V-sDA-C 2017 Peng et al. (2017), large-scale dataset used for the 2017 ICCV visual DA challenge, with 280K images of 12 object categories. The source domain contains synthetic images generated via 3D model rendering, while the target domain consists of real images.
Experiment Setup. For fair comparisons with other SFDA methods, we maintain consistency in net- work architecture, training techniques, and hyperparameters across all experiments. Specifically, we utilize ResNet-50 for Office-31 and Office- and ResNet-101 for VisDA-C. The classifier F consists of two linear layers. We adopt the SraaCC-bbsse loss from SimCLR Chen et al. (2020) for contrastive learning, The optimization uses SGD optimizer with momentum of 0.9, and the batch size is set to 32 for Office datasets and 64 for VisDA-C. The learning rates for all experiments are fixed at le-3, Results for Office-31 and Office-Home are reported after 100-epoch training, while VisDA results are presented after 15-epoch training. In Office-Home experiments, we apply regularization to the diagonal matrix of predictions in a mini-batch, achieved through singular value decomposition Cui et al. (2020). The SiLAN hyperparameters, including K, and Ks, represent the number of neighbors determined by the larget and source models, re- spectively. They are to 3 for most Office experiments and 15 for VisDA-C, The temperature parameter T for InfoNCE loss is set to 0.11 for most experiments.
Results. Results for Office-31, Office-Home, and VisDA-C 2017 are shown in Tables 1 and 3 respectively. The results for ResNet-X represent applying the source pre-trained model directly to target domain without adaptation. Our framework achieves state-of-the-art performance across all three benchmarks compared to other SFDA methods.
7 Conclusions
The present paper provided a thorough analysis of SFDA methods based on contrastive learning, revealing that their effectiveness depends the degree of overlap among logit groups and the informativeness of augmentations for positive key generation. Building these insights, we introduced a novel latent augmen- tation method to address the limitations of existing contrastive SFDA methods. This method utilized the dispersion of latent features around query samples, guided by the source pre-trained model, to reduce logit group overlaps and improve positive key generation. The proposed method, relying on a single InfoNCE loss, demonstrated superior performance compared to existing SFDA methods across various benchmark datasets. Our research contributed to the understanding of applying contrastive learning in the context of SFDA and provides practical solution to enhance model generalization in the absence of source data.
12

---

Published in Transactions on Machine Learning Research (04/2024)
References
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Percira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79(1):151 175, 2010.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive lcarning of visual representations. In International Conference on Machine Learning (ICML), pp. 1597- 1607. PMLR, 2020.
Jiashun Cheng, Man Li, Jia Li, and Fugee Tsung. Latent augmentation for better graph self-supervised learning. arXiv preprint arXiv:2206. 129933, 2022.
Tsz-Him Cheung and Dit-Yan Yeung. Modals: Modality-agnostic automated data augmentation in the latent space. In International Conference on Learning Regresentations (ICLR), 2020.
Shuhao Cui, Shuhui Wang, Junbao Zhuo, Liang Li, Qingming Huang, and Qi Tian. Towards discriminability and diversity: Batch nuclear-norm maximization under label insufficient situations. In Proceedings of the
IEEE/CVF Conference on Computer Vision and pattern Recognation (CVPR), pp. 3941-3950, 2020.
Terrance DeVries and Graham W Taylor.
Dataset augmentation in feature space.
arXiv preprint
OrXiv:1702.0055 2017.
Alhussein Fawzi, Matej Balog, Aja Huang, Thomas Hubert, Bernardino Romera-Paredes, Mohammadamin Barckatain, Alexander Novikov, Francisco J R Ruiz, Julian Schrittwicser, Grzegorz Swirszez, al. Dis- covering faster matrix multiplication algorithms with reinforcement learning. Nalure, 610((930):47 53, 2022.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Interna- tional Conference on Machine Learning (ICML), pp- 1180-1189. PMLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin WI, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738) 2020.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02555 2(7), 2015.
David W Hogg and Dustin Lang. Replacing standard galaxy profiles with mixtures of gaussians. Publications of the Astronomical Society of the Pacific, 125(928):719, 2013.
Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1074 10-3, 2021.
Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu. Unsupervised deep learning by neighbourhood discovery. In International Conference on Machine Learning (ICML), pp. 2849- 289-.85.. PMLR, 2019.
Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Model adaptation: Historical contrastive learning for unsupervised domain adaptation without source data. Advances in Neural Information Processing Systems (NeurIPS), 34:3635-3666 3649, 2021a.
Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of contrastive self-supervised learning. Xr preprint arXiv:2111000743, 2021b.
Uiwon Hwang, Jonghyun Lec, Juhycon Shin, and Sungroh Yoon. Sf(da)2: Sourco-free domain adaptation through the lens of data augmentation. In The Twelfth International Conference on Learning Represen- lations (ICLR), 2024.
13

---

Published in Transactions on Machine Learning Research (04/2024)
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Newral Information Processing Systems (NeurIPS), 33:18661- 18673, 2020.
Varun Kumar, Hadrien Glaude, Cyprien de Lichy, and William Campbell. closer look at feature space data augmentation for few-shot intent classification. ar preprinl 1iv::11900044 2019.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9641-9650, 2020.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning (ICML), pp. 6028 6039. PMIRR 2020.
Jian Liang, Dapeng Hu, Ran He, and Jiashi Feng. Distill and fine-tune: Effective adaptation from a black-box source model. arXiv preprint arXiv:2104.0155 1(3), 2021a.
Jian Liang, Dapeng Hu, Yunbo Wang, Ran Hc, and Jiashi Feng. Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Trunsactions on Pallern Analysis and Machine Intelligence, 2021b.
Xiaofeng Liu, Yang Zou, Lingsheng Kong, Zhihui Diao, Junliang Yan, Jun Wang, Site Li, Ping Jia, and Jane You. Data augmentation via latent space interpolation for image classification. In 2018 24th International Conference Pattern Recognition (ICPR), pp. 728-733. IEEE, 2018.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through doep reinforcement lcarning. Nature, 518(7540):529--55 2015.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint [xXi::807.087/53 2018.
X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko. Visda: The visual domain adaptation challenge. arXiv preprint, pp. arXiv:1710.0690 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211 252, 2015.
K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual calegory models to new domains. In European Conference on Computer Vision (ECCV), pp. 213-226, 2010.
David Stutz, Matthias Hein, and Bernt Schicle. Discntangling adversarial robustness and generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6976-690 6987, 2019.
Paul Upchurch, Jacob Gardner, Geoff Pleiss, Robert Pless, Noah Snavely, Kavita Bala, and Kilian Wein- berger. Deep feature interpolation for image content changes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7064-7073, 2017.
Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Scan: Learning to classify images without labels. In European Conference on Computer Vision (ECCV), pp. Springer, 2020.
H. Venkateswara, J. Euscbio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervisod domain adaptation. In the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5394, 2017.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEFF/CVF Conference on Compuler Vision and Pollern Recognition (CVPR), pp- 2495-2500 2021.
Yingquan Wu, Krassimir lanakiev, and Venu Govindaraju. Improved k-nearest neighbor classification. Pat- tern recognition, (5(10):2311225) 2002.
Haifeng Xia, Handong Zhao, and Zhengming Ding. Adaptive adversarial network for source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1010-9019 2021.
Tongkun Xu, Weihua Chen, Pichao Wang, Fan Wang, Hao Li, and Rong Jin. Cdtrans: Cross-domain transformer for unsupervised domain adaptation. ar preprinl 2Xxx:2109.06166 2021.
Jinyu Yang, Jingjing Liu, Ning Xu, and Junzhou Huang. Tvt: Transferable vision transformer for unsu- pervised domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (CVPR), pp. 520-58 520-530, 2023.
Shiqi Yang, Joost de Weijer, Luis Herranz, Shangling Jui, et al. Exploiting the intrinsic neighborhood structure for source-free domain adaptation. Advances in Neural Information Processing Systems, 34: 29393 29405, 2021a.
Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Generalized source- free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8978-899 8887, 2021b.
Shiqi Yang, Yaxing Wang, Kai Wang, Shangling Jui, et al. Attracting and dispersing: A simple approach for source-free free domain adaptation. In Conference on Neural Information Processing Systems (NewrIPS), 2022.
Ziyi Zhang, Weikai Chen, Hui Cheng, Zhen Li, Siyuan Li, Liang Lin, and Guanbin Li. Divide and contrast: Source-free domain adaptation via adaptive contrastive learning. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
A Ablation Studies
A.1 Number of Nearest Neighbors for N and Nj-
During target adaptation, we distinguish the number of nearest neighbors for No determined by the target model as K, and for N% determined by the source model as K,. As discussed earlier, selecting a small value for K2 can enhance clustering but may introduce less smooth predictions, susceptible noise from inconsistent neighbors. Conversely, choosing a large value of K, yields smoother predictions but increases the overlap among logit groups. The value of K2 should be determined empirically to find balance. It should ensure sufficiently large standard deviation for traversal around N while avoiding excessive values that introduce augmentations whose predictions lie in other logit groups. We conduct this ablation study on Office-31 to adapt the model pre-trained on Amazon to Webcam. As shown in Table our framework is robust to the choices both K, and K, within a reasonable range.
A.2 Standard Deviation N
The impact K, on the standard deviation of latent features within N₂ is evaluated through experiments on the Office-31 dataset. In this ablation study, model trained on Amazon adapted Webcam. Table [ presents the average standard deviations of the features extracted from all samples in Webcom using the converged model, with varying values of K,, under the column Noise Std.
15

---

Published in Transactions on Machine Learning Research (04/2024)
Table 4: Ablation study for the number of nearest neighbors, K₁ and Ks; on the adaptation performance on Office-31 using ResNet-50.
Office-31 Amazon Webcam
K₁
K
Result
Noise Std.
Ke
K₂
Result
Noise Std.
92.7
0.049
3
93.6
0.078
2
3
93.2
0.068
3
3
94.6
0.105
93.0
0.059
94.0
0.085
2
5
92.8
0.052
3
5
93.2
0.071
4
2
92.8
0.058
5 2
92.2
0.032
93.4
0.062
5
93.2
0.068
4
4
93.2
0.056
5
4
93.4
0.070
4
93.9
0.078
5
93.2
0.072
A.3 Temperature for Contrastive Loss.
The strength of penalties on negative keys in the contrastive loss is governed by the temperature parameter 7. A small temperature increases the ponalization of negative samples, pushing their latent features farther away from those of the query, as highlighted in Wang & Liu (202T). Conversely, a large temperature results in more compact latent features within each logit group, but reduces sensitivity to negative samples clustering. To empirically evaluate the effect of 7, experiments are conducted on Office-31 with K, and K2 set to 3, respectively. The results, illustrated in Figure demonstrate the impact of t on the target classification performance.
A W
D D W
WA
A
W
100
95-
Accuracy (%))
80-
75
70-
0.05
0.07
0.09
0.10
0.11
0.13
0.15
0.17
0.5
1.9
Temperature
Figure 4: Ablation experiments Office-31 using ResNet-50 to evaluate how classification performance varies with temperature parameter 7.
A.4 Ablation on SiLAN's Impact on InfoNCE Loss.
In this ablation study, we evaluate the effectiveness of SiLAN in improving the InfoNCE baseline using the Office-Home dataset. We compare the classification accuracies in the target domain for models trained with and without the use of SiLAN for positive key generation to guide InfoNCE contrastive clustering. The results, as shown in Table 5 demonstrate that SiLAN significantly improves the performance of the InfoNCE baseline, highlighting its robustness.
A.5 General Guidance for Hyperparameter Selection
Our InfoNCE- SiLAN introduces three additional hyperparameters: the number of the courco-informer kNNs K,, the number of target kNNs Kr, and the logit. temperature for contrastive loss In this section,
16

---

Published in Transactions on Machine Learning Research (04/2024)
Table 5: Comparing the InfoNCE baseline performance with and without SiLAN on Office- Home using ResNet-50.
Ar +
Cl
Method
C1
Pr
Rw
Ar
Pr
Rw
InfoNCE+ K-NNs
55.6
76.4
80.6
66.4
75.2
76.4
InfoNCE SILAN (Ous)
58.2
81.2
82.5
69.8
78.6
80.3
Pr +
Rw ++
Method
Ar
CI
Rw
Ar
CI
Pr
InfNCC++K-NN:
66.2
53.8
80.5
72.8
56.8
83.5
InfoNCE- SILAN (Ours)
68.4
58.6
82.5
75.6
60.8
86.1
will offer general guidance for setting values for these hyperparameters based on the results of the sensitivity analysis obtained from Appendices A.1, A.2, and A.3.
To identify the optimal set of hyperparameters, we recommend the following systematic approach: begin by determining the number of target &NNs, denoted as Kg- This parameter determines the mean of the Gaussian for our latent augmentation, directly influencing the effectiveness of contrastive clustering. In general, K2 can be roughly estimated based the number of samples per class in the target dataset; smaller target dataset should use smaller K,, For instance, for Office datasets, cach containing about 200 images per class. However, for VisDA-2017, which comprises around 23K images per class, the optimal K, is 15. Therefore, to determine the optimal K,, we suggest a search range from 2 to 8 for a target dataset with 100 to 1K samples per class, and a search range from 10 to 20 for a target dataset with more than 20K images per class.
experiments show that the optimal K₁ ranges from 3 to 5
Subsequently, the number of the nource-informed kNNs, denoted K,, which determines the standard de- viation of the Gaussian for SiLAN, could be decided based on K,- Our analysis in Section 5 indicates that K, should be large enough to contain the farthest source-informed neighbor that may share the same ground truth as the target query. However, it should not be excessively large to avoid including features with inconsistent ground truth in the positive key generation. We found that the standard deviation of these source-informed kNNs' features is a reliable indicator for selecting K, to define the optimal latent augmen- lation region. As detailed in Appendix A.2, a larger standard deviation of the latent feature vectors of the source-informed kNNs generally correlates with better performance on target classification tasks. Typically, the largest standard deviation occurs when K2 is approximately equal to K, Therefore, preliminary strategy could be setting K, equal to K,, with the optimal K, lying within the range of K2 2
Finally, the temperature for contrastive logits should be determined similarly to self-supervised learning frameworks, as regardless of the mathematical space in which clustering occurs, this parameter influences the degree of penalization applied to hard negative samples Wang & Liu (2021). Unlike unsupervised representation learning, which aims for universal representation across tasks, our contrastive objective operates in the output logit space, favoring a small value for 7 (similar to identifying the temperature in knowledge distillation frameworks Hinton al. (2015)). In our experiments, the optimal value for falls between 0.07 and 0.15. Therefore, search range between 0.05 and 0.2 is recommended for practitioners.
A.6 Computational Analysis
In this section, we conduct a detailed runtime analysis on our SiLAN and compare it with other advanced SFDA methods. To be specific, we conducted a runtime analysis comparing the one-epoch training time and overall convergence runtime of AaD Yang et al. (2022) of our SiLAN with other SFDA methods. To demonstrate scalability with respect to dataset size and model complexity, we performed this computational analysis on both the small-scale dataset Offico-31 using ResNet-50 and the large-scale dataset VisDA-2017 using Res All experiments were conducted on a machine equipped with an Nvidia V100 GPU.
17

---

Published in Transactions on Machine Learning Research (04/2024)
Table 6: Time Analysis of One-Epoch Target Adaptation on Office-31 AD (ResNet-50).
Method
One- Spoch Time (sec)
One- Epoch Performance (%)
NRC
Yang al.
2021a
21.4
85.3
DaC
Zhang
(202
40.5
80.6
AaD
Yang et al.
(2022)
19.8
84.4
SiLAN (ours)
20.6
84.9
A SiLAN (ours)
20.2 (+0.4)
85.6
(+1.2)
Table 7: Convergence Time Analysis for Target Adaptation on Office-31 AD (ResNet-50).
Method
Convergence Time (sec)
Best Performance (%)
NRC
Yang et al.
(2021a)
856.7
96.0
DaC
Zhang et al.
(2022
1417.5
94.2
AaD
Yang et al.
(0222)
714.9
96.4
SiLAN (ours)
618.9
97.1
A (ours)
584.5
(-130.4)
97.5 (+1.1)
Table 8: Time Analysis of One-Epock Target Adaptation on VisDA-2017 Dataset (ResNet-101).
Method
One-Epoch Time (sec) One- Spoch Performance (%)
NRC
Yang et al.
(2021a)
469.2
79.2
DaC
Zhang et al.
(2022
632.8
82.4
AaD
Yang et al.
2022)
453.6
82.6
SiLAN (ours)
465.9
84.5
AiDAN (ours)
462.3 (+8.7)
86.8 (+4.2)
Moreover, as our SiLAN can also serve as a general latent augmentation method to enhance other SFDA methods, we have included a runtime analysis for the scenario where our SiLAN latent augmentation is applied to improve the informativeness of the latent features of the neighbors for AaD (referred to as AiDAS) In this analysis, we present the performance gain and additional runtime compared to AaD alone, providing practitioners with the information needed to balance performance gains against, runtime considerations.
illustrate that our SiLAN, NRC, and AaD exhibit similar target adaptation times per epoch, whereas D requires substantially more time due to its adaptive process and self-training steps, such as pseudo label generation and re-training. This observation aligns with the fact that AaD, NRC, and our SiLAN are all rooted in neighborhood scarching and involve aligning predictions between query predictions and those of the neighbors. Meanwhile, our SiLAN, along with AaD and NRC, typically converge within about 10 epochs for the VisDA-2017 dataset, whereas DaC requires around 20 epochs. Notably, the small- scale Office-31 dataset, our SiLAN achieves faster convergence compared to other methods. We hypothesize that this is because SiLAN, serving as augmentation method, significantly enhances model convergence, particularly in situations where data is scarce. The total convergence time for target adaptation is also provided for comparison.
Tables
Therefore, we conclude that despite our SiLAN having a similar one-epoch runtime compared to NRC and AaD, its use as latent augmentation leads to faster convergence compared to other SFDA methods, resulting in overall runtime benefits.
A.7 SiLAN as General Latent Augmentation Method
18

---

Published in Transactions on Machine Learning Research (04/2024)
Table 9: Convergence Time Analysis for Target Adaptation on VisDA-2017 Dataset (ResNet-101)
Method
Convergence Time (sec)
Best Performance (%)
NRC
Yang et
2021a
4692.8
85.9
DaC
Zhang et al.
2022
12656.3
87.3
AaD
Yang al.
(002))
4536.2
87.1
SiLAN (ours)
3727.2
88.3
A (ours)
3236.1
(236.1(11300) (-1300.1)
89.7 (+2.6)
In this ablation study, we demonstrate the versatility of our proposed SiLAN as general latent augmentation method for source-free domain adaptation (SFDA) frameworks. To validate this, we incorporate SiLAN inlo advanced SFDA methods, namely HCL Huang et al. (2021a), Net, Xia et al. 2021 AaD Yang et al. (2022) and NRC Yang et al. (2021a). While both AaD and NRC utilize the predictions of query neighbors to optimize the model, their implementation details vary.
Among them, HCL Huang al. (021a)) is contrastive SFDA framework built upon pscudo-labeling, and A Net Xia et al. (2021) is an adversarial SFDA framework. Additionally, AaD Yang et al. (2022) features more advanced contrastive learning objective tailored for solving SFDA problems, while NRC Yang et al.
(021aa) utilizes: a hierarchical neighborhood searching strategy.
The results, presented in Table I0 clearly indicate that our proposed SiLAN augmentation significantly im- proves the performance of both AaD and NRC. To be specific, our proposed SiL.AN enhances the performance of HCL, Net NRC, and AaD by 0.6%, 2.0%, 1.2%, and 2.6%, respectively. These results demonstrate that SiLAN is an effective latent augmentation method for addressing SFDA problems across various SFDA frameworks.
Table 10: Ablation studies of integrating SILAN into various SFDA frameworks for enhanced performance on VisDA2017 (ResNet-101).
Method
plane beyel bus
car
horse knife meyel person plant sktbrd train truck
Avg.
ResNett-10 101
He et (2016)
55.1
53.3
61.9
59.1
80.6
17.9
79.7
31.2
81.0
26.5
73.5
8.5
52.4
HCL
Huang al.
2021a
93.3
85.4
80.7
68.5
91.0
88.1
86.0
78.6
86.6
88.8
80.0
74.7
83.5
A Not
Xia et al.
2021
94.0
87.8
85.6
66.8
93.7
95.1
85.8
81.2
91.6
88.2
86.5
56.0
84.3
NRC
Yang et al.
2021a
96.1
90.8
83.9
61.5
95.7
95.7
84.4
80.7
94.0
91.9
89.0
59.5
85.3
AaD
Yang et al.
[2022]
95.2
90.5
85.5
79.2
96.4
96.2
88.8
80.4
93.9
91.8
91.1
53.9
87.1
SiLAN (Ours)
97.5
90.1
85.8
80.4
97.6
95.5
92.0
82.9
96.5
95.3
92.6
53.4
88.3
HCL+SSLLAN (Ours)
94.6
85.4
83.2
67.3
94.2
86.5
86.3
80.8
88.2
85.2
83.4
74.6
84.1
A SiLANN (Ours)
97.2
91.2
87.4
66.8
96.9
96.9
88.0
80.9
93.5
93.3
91.1
53.2
86.3
NRC+SILAN (Ours)
97.2
90.5
84.3
63.8
96.7
95.4
86.2
85.1
95.6
93.2
90.2
59.8
86.5
AaD+SILAN (Ours)
98.4
91.8
86.2
80.6
96.3
95.7
94.4
87.5
95.8
94.2
93.4
62.1
89.7
A.8 Sensitivity Analysis on The Effect of The Source Pre-Traine Training
In this sensitivity analysis, we evaluate how the quality of the source pre-trained model impacts target classification performance. We select source pre-trained models based on their performance on the source 'ataset's test sets, consistent with common practice in other SFDA methods.
To validate this approach, conduct sensitivity analysis on the mid-scale Office-Home dataset using ResNet- 50. We maintain consistency by using the same target dataset (Artistic) while varying the source domain datasets (Cliparl, Producl, and Real- World). Figure 5 illustrates the experimental results of the sensilivily analysis conducted on the effect of the source model pretraining on the target-domain classification perfor- mance. The left subfigure of Figure illustrates the relationship between test performance on the source test set and the number of epochs used to pre-train the model on the source dataset. On the other hand, the right, subfigure demonstrates how the quality of the source pre-trained model, measured by the number
19

---

Published in Transactions on Machine Learning Research (04/2024)
of epochs used for pre-training the source dataset, influences the classification performance in the target domain after adaptation convergence. The results indicate that, in the context of our SiLAN approach, selecting models that exhibit superior performance on the source test set as the source pre-trained model generally leads to optimal target adaptation outcomes.
The diversity the source dataset also influences target adaptation performance. For instance, the synthetic (source) domain in the VisDA-2017 dataset includes 152,397 images generated from 3D models across 12 object categories. These images feature diverse shapes, colors, textures, and sizes. In contrast, real (target) domain images vary in backgrounds, lighting, occlusions, and object poses. This diversity poses challenges for domain adaptation algorithms, requiring effective generalization across disparities for optimal target domain performance.
Sensitivity Anelysis on The Number of Epoch for Source Training
Sensitivity Analysis on The Number of Epach for Source Training
Torve: Demain Acurasyy
Accuaccy in
60
85
70
00
Clppatt Artalicc
Artictic Product titic
Product Atc
1.0
15
20
25
30
35
40
45
50
20
35
40
50
Number of Epoch for Source Traiiin
Number Epoch Scurce Training
(a) Performance on Source Test Set.
(b) Performance on Artistic After Adaptation.
Figure 5: Sensitivity analysis on the Office-Home dataset using ResNet-50 to evaluate how the quality of the source pre-trained model impacts target-domain classification performance.
A.9 Additional Experiments on Vision Transformers
To demonstrate the effectiveness of our SiLAN across different backbone architectures, performed exper- iments on all benchmark datasets using the ViT-16-B vision transformer as our backbone, as suggested in prior work Xu et al. 22021). The results, presented in Tables 11
12 and 13 highlight the efficacy of our SiLAN addressing SFDA problems with a vision transformer backbonc, outperforming existing ViT-basod SFDA methods.
Table 11: Comparison of SFDA methods using ViT-B-16 on Office-91.
Method
A-D
A-W
D-W
D-A
W-D
W-A
Avg.
ViT-B-16
90.8
90.4
76.8
98.2
76.4
100.0
88.8
TVT Yang
2023)
96.4
96.4
84.9
99.4
86.1
100.0
93.8
CDTrans Xu et al 2021
97.0
96.7
81.1
99.0
81.9
100.0
92.6
SiLAN (Ours)
95.3
97.2
88.1
99.6
89.2
100.0
94.6
CDTrans (Ours)
97.4
98.1
83.4
99.4
83.3
100.0
93.6
20

---

Published in Transactions on Machine Learning Research (04/2024)
Table 12: Comparison of the SFDA methods on Office-HHome (ViT-B-16).
Method
Ar-CI Ar-Pr Ar-Rw Cl-Ar Cl-Pr Cl-Rw Pr-Ar Pr- CI Pr-Rw Rw-Ar Rw- Cl Rw-Pr
Avg.
ViT-B-16
61.8
79.5
84.3
75.4
78.8
81.2
72.8
55.7
84.4
78.3
59.3
86.0
74.8
TVT Yang et al.
2023
74.9
86,8
89.5
82.8
88.0
88.3
79.8
71.9
90.1
85.5
74.6
90.6
83.6
CDTrnn Xu all
[2021]
85.0
86.9
81.5
87.1
63.3
82.0
66.0
90.6
80.5
SILAN (Ours)
70.4
90.5
88.6
85.3
83.1
86.5
85.2
73.9
88.6
86.1
80.0
92.5
84.2
CDTTrans+SSLLAN (Ours)
72.1
86.2
85.4
81.7
88.5
86.9
82.4
81.1
87.9
84.5
78.2
90.4
83.8
Table 13:
Comparison of the SFDA methods on VisDA2017 (ViT-B-16).
Method
plane beyel
bus
car
horse
knife meyel person plant sktbrd train truck Avg.
ViT-B-16
97.7
48.1
86.6
61.6
78.1
63.4
94.7
10.3
87.7
47.7
94.4
35.5
67.1
TVT Yang et al. (2023
92.9
85.6
77.5
60.5
93.6
98.2
89.4
76.4
93.6
92.0
91.7
55.7
83.9
CDTrans Xu
(2021
97.1
90.5
82.4
77.5
96.6
96.1
93.6
88.6
97.9
86.9
90.3
62.8
88.4
SiLAN (Ours)
92.5
90.1
92.4
70.6
92.1
98.5
95.8
89.2
94.5
93.3
90.6
64.4
88.7
C (Ours)
92.5
86.2
75.2
98.5
95.5
95.8
90.6
98.2
92.1
87.5
63.2
B Proof
B.1 Proof of Proposition
Proposition 1. The InfoNCE-based contrastive loss, denoted as Lcont, serves as an upper bound for achieving two distinct alignments in the output logit space. Formally,
log(m- 1) [lfe(x)) - f((x
1
[1f((x)) - fe(x)))11
Coonts
2T
m
jAL
where with being a temperature and being the size of a mini-batch, Lcont defined as,
m
ef (x))xx0)//
Leont
log Ejye ell (xi16x3))/""
Proof. the InfoNCE. loss defined the output logit space is formulated as:
e.t (x)))1 1/T
Leont =
Σ
log
==1
(6)
Σ
log
Σ
m
Then, we apply Jensen's incquality to Equation G
Σ log (m 1) et ef (60)51tt)/))))
Lcont
m
1t1
- -T(()))(xt)/T m
Σ [xx)) (x+
Σ
log(m Tog(m-1))
(7)
m
log(m 1) -f. (xx))
Σ
m
21

---

Published in Transactions on Machine Learning Research (04/2024)
Then, express cosine similarity between two functions in terms of Euclidean distance:
2
(8)
2
Plugging Equ S into Inequality 7
log(m - 1) [15e(x.) - Sa (x
[1f((x,) f((x5))l 0112
Σ
1 Leant:
(9)
2T
m
2T
End of the proof.
B.2 Proof of Lemma 2
Lemma 2. If Co n CØ = 0 holds for any 1 + z, then the error eDy defined on the groups of logits is upper
bounded by:
Dr I Ra Σ (P(f((xt) yo, Vxt E eSt ye,
+ P[f(((x yo, Vx E ET))),
_ (C, and (C2 -o) may overlap with (G: - CJ) for any I + 2.
where Rs
U6 C.
Proof. Based on Huang et al. 2021b), our study refines the theoretical problem of target classification error within the context of contrastive clustering. decomposes the classification error, focusing on the groups of output logits C2 where E [1, Z], as illustrated in Equation 3
The first error term only occur within the overlapping regions (i.e., interscctions) between the groups of logits generated through contrastive learning Huang et al. (2021b). Therefore, we have:
Σ +2, C.J S P
(10)
The informativeness the group assignment depends entirely on the informativeness of the transformations of querics when optimizing the contrastive loss. Thus, the upper bound of the second error term can be established as:
z Σ Pl= ecJ S Σ 4 yxt E 8 P[f((x vx,
Vx- E
(11)
2=1
where the first term represents the error of assigning the logits of positive samples to group that does not correspond to the ground truth of the query; and the second term denotes the error that occurs when assigning the logits of negative samples to the group to which the query should belong based on its ground truth.
Incorporating Tnequalities
10 and
11 into Equation we have:
Z
CDr
P Co
2=
+
P[f((xt) # Yr, Yx, Vx+ E
Vx E ES71)
==1
+
Σ6
(Plf((xt) vx, Vx* s P(f((x) Vx &s,).
2
C=
==1
End of the proof.
22

---

Published in Transactions on Machine Learning Research (04/2024)
B.3 Proof of Proposition 3
Proposition 3. If C. nct = 0 holds for any = + I, and the assumption that the representation of query samples in the feature space will stay close to their positive augmentations and be far away from their negative samples holds, then Vz + I and i + (where x; E C and xj E cf), their distance in the latent space lower bounded when we take the limit dext + 0 as follows:
(xtx*((x)
G. (x,)1 3.1704a,
Proof. Inspired by the methodology introduced in Hogg & Lang 2013) for calculating the effective region of Gaussian beam, we introduce the concept of the Gaussian beam to estimate the effective region of our SiLAN augmentation, whose effect is within a Gaussian profile. Here, all the possible augmented views, originating from query sample x, center at the mean fux (x) of the neighborhood of the query.
Hence, a radial profile with radius R and centroid μx (x) can be employed to represent the effective region of the potential augmented views from SiLAN:
AR(R) = 1/" h(r)2ardr,
n
where h(r) is radial profile function well-sampled and determined by SiLAN and the feature extractor parameters.
Assuming perfect alignment achieved after G converges on the contrastive objective, meaning positive augmentations (keys) lie in the same cluster as the query while negative keys do not lie in the query's cluster, then, the gap between any two distinct clusters will be determined by the augmentation that belongs one cluster and is ncarest to the other cluster. In summary, under perfect alignment, there is no overlap between the Gaussian profiles of samples from different clusters. Thus, we have:
GG (x.) - - (x)) 2R.
Then, considering the extraneous noise with variance o the total variance in the informative radius, with respect to (x), out to R can be wrillen as:
Lotal (R) AR(R) n
Peats 2
where T TR² is the aperture area given R2 containing possible augmented views centered at yu (x).
The transformation-ton to-nose ratio T/N, which is similar to the definition signal-to-nois ratio to quantify Gaussian bcam Hogg& & Lang 20033,, as a function of radius R can be written as:
AR(R)
T/N = V VAR(R R) + RPo
So" hi(r) (n(r)2xrdr
R fi(r)2mrdr + 02
After adding random noise ç, we have a 2D Gaussian for the source profile centered at pxx (x):
hi(r)
2T02 e 207
where a is the standard deviation of the profile. Then,
.22
T/N =
R2
1
3
πI 02 c
23

---

Published in Transactions on Machine Learning Research (04/2024)
Taking ((T/N) 0 and the limit as Pext O, we have the optimum radius R = 1.5852a.
By incorporating R =: 1.5852a into the inequality, we have:
1G (x:) G. (x,) 3.1704a,
End of the proof.
B.4 Proof of Lemma
Lemma 4. V2 & l and * J. if the linear classifier Popt I-bi-Tipschitz continuous, Cº n nc = 0 holds for any + where x, Cá and Xj E C2, and oe +
3.1704g
(xx)
(xx) f0 (x,)1 L
where fopt F.opt ((pP (-)).
Proof. As the classifier F is linear and L-bi-Lipee L-bi-Lipschitz continuous, we have:
1 (x,) f2 (x,)
1i (Go (xx)) Eoop (x,))
2 L
G (x,)) G. (x,))-
From Proposition we have G.tt - G.pd (x,)) 2 3.14407, thus
(xp))(x;)
-pept (x9((x,)] - 3.14076
End of the proof.
C Further Intuitive Insights into The Theoretical Work
In this section, we aim to provide practitioners with clearer understanding of our proposed SiLAN. To achiove this, we offer more intuitive explanations and visual illustrations to clarify the purpose and outcomes of our theoretical work.
C.1 Intuition of Proposition 3
Proposition 3 establishes a theoretical lower bound for the L1 distance between the latent features of data samples belonging to different logit clusters. This bound is derived under the assumption that the classi- fication model optimized by a contrastive loss to align the query predictions with those of our SiLAN augmentations, while intentionally misaligning the query predictions with those of the data samples within the same mini-batch.
This analysis presupposes that perfect alignment occurs after GoDD converges such contrastive objective. In other words, positive augmentations (kcys) are positioned within the same cluster as the query, while negative keys do not lie within the query's 's cluster. As illustrated in Figure D the gap between any two distinct clusters (i.e., (x=) - G (x;)l where X; and x, belong to different logit clusters) is determined by the augmentation nearest to one cluster but belonging to the other.
It important to recall that our SiLAN augmentation introduces Gaussian noise, incorporating guidance from the &NNs found by the source pre-trained model to the centroid of the query neighborhood found by
24

---

Published in Transactions on Machine Learning Research (04/2024)
gaussian Profile
query sample
R
Non-overlapping region different
Figure 6: The minimum gap between the non-overlapping regions of two clusters has lower bound deter- mined by the radius of the Gaussian profile.
the current larget model. Mathematically, the augmented features used to generate the positive keys for contrastive clustering can represented as:
h : (()k(((()))) +
(12)
Here, x denotes the target query sample, while p (x() denotes the centroid of its ANN neighborhood. The noise for the augmentation (extrancous noise introduced in Proposition 3), drawn from Gaussian distribution NNO,, of (x()), is determined by the variance of the kNNs as determined by the source pre- trained model.
Thus, employing our SiLAN-angmented noighbors as positive keys for contrastive clustering enables the establishmeet. of a lower bound on the distance between logit clusters, as our SiLAN induces a Gaussian profile for the positive augmentation. Proposition 3 utilizes geometric approach Hogg & Lang (2013) caleulate the effective region of Gaussian beam (in our case, the centroid of the query's 's target neighbors serves the center of this Gaussian beam) in computational optics determine this lower bound.
C.2 Intuition of Lemma 4
Proposition 3 establishes the theoretical lower bound of the L1 distance between latent. features belonging to different logit clusters. However, primary interest lies not in the cluster distance within the latent feature space, but rather in the distance lower bound between the output logits of different clusters. A larger distance lower bound in the output logit space facilitates the derivation of decision boundaries for target classification.
This distance lower bound in the latent feature space can be straightforwardly extended the output logit space the classifier is linear. Hence, we employ a linear fully-connected classifier all our experiments ensure consistency between theoretical framework and empirical implementations.
25

---

Published in Transactions on Machine Learning Research (04/2024)
C.3 Derivation Details of Equation 3
In this section, we provide additional details regarding the derivation of the error term represented by Equation 3 which forms the cornerstone of our theoretical analysis.
To initiate our analysis, we first reinterpret the classification error on the target domain CDr = number of misclassifier samples by considering it as the probability of the model's predictions failing to align total number of sammles with the true labels corresponding samples, given the current target classification model ft. The classifica- tion error quantifies the ratio of the misclassified instances to the total number of instances. Mathematically, this is represented as:
number of misclassi fied samples total number of semples
(OC P(f((X)) + Yr). N
CDr =
Here, 1 is the binary indictor that equals one when f((x)++))
Instead of evaluating misalignment based solely on individual sample indices, we refine this analysis to focus on samples within each cluster (where the number of clusters corresponds to the number of neurons in the last fully connected layer of the classifier). Thus, we express the classification error as:
EDr = EE # Y7))).
Here, X and Y indicate the target samples and their labels with cluster 2. Therefore, after model conver- gence and cluster formation, errors or misalignments can oceur due to two main factors:
Incorrect cluster assignment, where the model assigns a sample to the wrong cluster (P(z # vil, Vr, E C2).
Overlapping between clusters leading to ambiguous model predictions for samples within cluster (P[fi(z; # 2)), Vx, E Cz).
By re-examining the classification error in the context of possible errors during the cluster formation process Equation 3 we can then proceed with the subsequent analysis to investigate how contrastive clustering influences the performance of target classification.
C.4
Interconnections Among Three Theoretical Insights on Contrastive SFDA
In Section 4.2 we identified three overlooked factors in existing contrastive SFDA methods based on our theoretical analysis. Here, we clarify the interconnections among these insights and elucidate how our exploration of them informs our proposed solution to enhance the contrastive SFDA framework.
In summaryy, examining Insights and 2 provides the foundational understanding necessary to address Insight effectively. Insight highlights the inadequacy of relying solely on data augmentation for generating positive keys in contrastive SFDA. This realization motivates to explore latent augmentation an alternative way for positive key generation. Expanding on Insight 1, our empirical observations from latent t-SNE analysis shown in Figure 1 indicate that generating target features through the source pre-trained model, which also initializes the target adaptation model, provides valuable information for target-domain classification beyond mere initialization. This information, as highlighted in the two observations in the introduction, indicates that neighboring target data points, determined in the latent feature space by the source pre-trained model, still share the same labels. As a result, we explore Insight to examine whether we can capture this crucial information by controlling the range of neighborhood searching. To be specific, we investigate how the number of &NN influences contrastive SFDA.
Insight 3 highlights an overlooked problem, which can be addressed through a straightforward solution derived from the analyses of Insights 1 and 2: incorporating guidance from the ANNs identified in the source-pre- trained model-determined latent feature space directly into the latent augmentation process to generate the positive key for contrastive clustering given a query sample. Our subsequent studies Sections 5.4 and 5.5 demonstrate the significance of the standard deviation of the Gaussian noise for such latent augmentation, it determines the range of neighborhood searching. Thus, by utilizing the standard deviation derived
26

---

Published in Transactions on Machine Learning Research (04/2024)
from the kNNs of the source pre-trained model to control the range of latent Gaussian augmentation, we enable the model update to identify neighborhoods in the target domain that exhibit. consistent ground truth alignment with the current target query sample. In simpler terms, employing positive keys generated in this manner steers the clustering process toward samples sharing the same ground truth. In essence, our derivations of Insights 1, 2, and 3 are interrelated and complementary, leading us to our final solution.
To sum up, addressing insight 3 involves setting the standard deviation of the Gaussian used generate positive keys in the latent space (Insight f) for contrastive clustering to match the standard deviation of the neighborhood determined by the source pre-trained model (Insight 2). This neighborhood comprises samples sharing the same ground truth as the query target sample. Conducting latent augmentation based on this profile is crucial, as it offers additional guidance regarding the target ground truth to contrastive clustering,
27


Reviews:
[{"review_id": "6tRlSWiazj", "reviewer": "Reviewer_KBch", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The paper provides a rigorous theoretical analysis of contrastive SFDA, offering valuable insights into the key factors influencing its performance. The analysis is well-motivated by empirical observations of target feature dispersion and neighborhood informativeness, and clearly presented.\n\n2. The proposed SiLAN method is a simple yet effective approach that addresses several limitations of existing contrastive SFDA methods identified in the theoretical analysis. It is grounded in the theoretical findings and leverages both neighborhood exploration and source model knowledge.\n\n3. SiLAN is evaluated on multiple benchmark datasets, with results consistently showing it outperforming state-of-the-art SFDA methods. Ablation studies are also provided to justify the design choices.\n\n4. The paper advances both the theoretical understanding and practical methodology of contrastive learning for the challenging problem of source-free domain adaptation. It is well-structured and clearly written, making it easy to follow the main ideas and contributions.\n\n## Weaknesses\n\n1. The hyperparameter selection for SiLAN (e.g., number of neighbors K, temperature τ) is not extensively discussed. More guidance or sensitivity analysis on these choices would be helpful for practitioners. How sensitive is SiLAN's performance to these hyperparameters, and can the authors provide general guidelines for setting their values?\n\n2. The computational cost of SiLAN compared to other SFDA methods is not reported. An analysis of the tradeoff between performance gain and additional computation would further strengthen the practical value of the method. What is the computational overhead of SiLAN, and how does it scale with the size of the dataset or model?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a theoretical analysis of contrastive learning for source-free domain adaptation (SFDA), identifying key factors that influence its effectiveness. Motivated by the findings, the authors propose a novel latent augmentation method called Source-informed Latent Augmented Neighborhood (SiLAN), which leverages the dispersion of latent features guided by the source pre-trained model to reduce logit group overlaps and enhance positive key generation. SiLAN is integrated into the InfoNCE contrastive learning framework and achieves state-of-the-art performance on multiple SFDA benchmark datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "While the theoretical analysis provides valuable insights, some parts (e.g., Proposition 3 and Lemma 4) are quite technical and may be difficult for some readers to fully grasp. It would be helpful if the authors could provide more intuition or visual illustrations to make these aspects more accessible.\n\nI also have some questions:\n\n1. How sensitive is SiLAN's performance to the choice of hyperparameters (K, τ, etc.)? Can the authors provide some general guidelines for setting these values?\n\n2. The theoretical analysis assumes convergence of the contrastive loss. How well does this assumption hold in practice, and how might non-convergence impact the performance of SiLAN?\n\n3. Could the proposed latent augmentation be combined with other advanced techniques (e.g., adversarial learning, pseudo-labeling) to further boost SFDA performance?\n\n4. The method is evaluated on image classification tasks. How might it generalize to other data modalities or tasks (e.g., semantic segmentation, object detection)?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1BZ3qwjb6G", "reviewer": "Reviewer_Bpp9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nStrong Theoretical Basis: In Section 4.2, The analysis of contrastive learning in SFDA provides a solid foundation for understanding how and why the proposed method works. The observation in Figure 1 about feature dispersion and neighborhood informativeness offers unique insights.\n\nNovel and Effective Augmentation: The SiLAN method is a creative approach to address the issues identified in the analysis. It leverages domain knowledge without requiring direct access to the source domain data. Fig 2 and 3. well illustrated the idea.\n\nSOTA Performance: Achieving state-of-the-art results on standard benchmarks demonstrates the effectiveness and practicality of the proposed method. The fact that good performance is achieved with a simple InfoNCE loss (Eq. 5) highlights the elegance of the approach.\n\n\nWeaknesses\n\nDependency on Source Model: While the method doesn't use source data directly, it still relies on a pre-trained source model. Analyzing its sensitivity to the quality of this source model would be valuable.\n\nLack of Ablation Study: A thorough ablation study could help isolate the specific impact of the SiLAN augmentation and how it interacts with other components of the framework. Table 1, 2, 3 are based on ResNet, does it generalize well in ViTs?\n\nComputational Overhead: It's worth investigating the computational cost of the neighborhood-based feature dispersion calculations and the potential trade-offs between performance gains and efficiency. I cannot find it anywhere in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors analyzed contrastive learning in SFDA and found that domain shift causes dispersion of target features, but neighboring features often still share the same class. This explains why contrastive clustering approaches can still be successful for SFDA.\nThey proposed a new latent augmentation method called SiLAN. This method uses the dispersion of latent features in the neighborhood of a query sample (as informed by the source model) to improve how positive keys are generated for contrastive learning.\nThe proposed method uses a simple InfoNCE contrastive loss, and yet outperforms prior SFDA approaches on standard benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Report results in Table 1,2,3 using ViTs.\n\n2. Add a table benchmarks trade-offs between performance gains and efficiency.\n\n3. add more discussion on pretrained source model. How diverse the data it should be trained on?", "broader_impact_concerns": "Domain shift itself can introduce new biases or change how existing biases manifest. SFDA methods need careful analysis to understand how they interact with potentially biased datasets, especially in sensitive applications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sALp7sO2CR", "reviewer": "Reviewer_rFpU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The author combines augmentation with a neighbor-based approach and continues introducing information from the source model during training. This is quite innovative.\n2. The performance is impressive, surpassing existing methods.\n\nWeaknesses:\n1. The cornerstone of this paper lies in formula (3), which raises the question of how the errors in the target domain can be adequately represented by these two terms. A comprehensive and detailed explanation is essential to elucidate this critical aspect.\n\n2. The methodology section raises several concerns:\n(a) The first insight posits that achieving domain adaptation without addressing distribution divergence is challenging. It remains ambiguous how this insight is derived from the theory proposed in this paper. Furthermore, the proposed method lacks a clear elucidation of how it mitigates domain divergence.\n(b) The methodology section extensively delves into explaining the workings of contrastive learning and the theoretical errors related to contrastive learning in the target domain. This inclination towards unsupervised learning does not harmonize well with the theme of source-free domain adaptation.\n(c) The first and second insights, being intuitive, do not necessarily require extensive theoretical analysis. The third insight, emphasizing the necessity of further leveraging source information, should be the primary focus of this paper. However, it remains unclear how the author addresses this crucial point.\n\n3.  Several descriptions are ambiguous:\n(a) In the fourth paragraph of the introduction, the author claims that various design choices for positive keys in constrive SFDA can mitigate performance degradation due to domain shift. However, it is unclear whether the author provides evidence for this statement in the paper. Existing SFDA works utilizing contrastive learning methods in SFDA do not seem to lead to performance degradation.\n(b) In the introduction's summary, the author mentions \"three often-overlooked factors,\" but these factors are not explicitly stated in the introduction, leading to confusion.\n\n4.  Regarding the experiments:\n(a) The paper lacks a comparison with the latest data augmentation-based method, SF(DA)^2, as presented by U. Hwang, et al. in their work \"SF(DA)^2: Source-free Domain Adaptation Through the Lens of Data Augmentation,\" ICLR 2024.\n(b) Considering the paper's approach, combining SiLAN with InfoNCE raises the question of whether similar analysis could be done with advanced SFDA methods like AaD or NRC.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the topic of source-free domain adaptation and discusses a classic contrastive learning approach. \n\n(1) The author first provides a theoretical analysis of SFDA based on contrastive learning and then derives several insights from the theoretical analysis.\n\n(2) Inspired by these insights, the author proposes a method (SiLAN) that combines augmentation and neighbor information methods, generating better positive key pairs by introducing information from the source model. \n\n(3) Rigorous experimentation on various benchmarks substantiates that the proposed method attains state-of-the-art performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)