NOTELA: A Generalizable Method for Source Free Domain Adaptation
Authors: Malik Boudiaf
tom denton
Bart van Merrienboer
Vincent Dumoulin
Eleni Triantafillou
Venue: Submitted to ICLR 2023 (2023)
Keywords: source-free domain adaptation
robustness to distribution shifts
bioacoustics

Abstract: Source-free domain adaptation (SFDA) is a compelling problem as it allows to leverage any off-the-shelf model without requiring access to its original training set and adapts it using only unlabelled data. While several SFDA approaches have recently been proposed, their evaluation focuses on a narrow set of distribution shifts for vision tasks, and their generalizability outside of that scope has not yet been investigated. We put those recent approaches to the test by evaluating them on a new set of challenging---due to extreme covariate and label shift---and naturally-occurring distribution shifts in the audio domain. We study the task of adapting a bird species classifier trained on focalized recordings of bird songs to datasets of passive recordings for various geographical locations. Interestingly, we find that some recent SFDA methods underperform doing no adaptation at all. Drawing inspiration from those findings and insights, we propose a new method that improves on noisy student approaches by adjusting the teacher's pseudo-labels through Laplacian regularization. Our approach enjoys increased stability and significantly better performance on several of our proposed distribution shifts. We then look back at SFDA benchmarks in the vision domain and find that our approach is competitive with the state-of-the-art there as well. 

Under review as a conference paper at ICLR 2023
NOTELA: A GENERALIZABLE METHOD FOR SOURCE FREE DOMAIN ADAPTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Source-free domain adaptation (SFDA) is a compelling problem as it allows to leverage any off-the-shelff model without requiring access its original training set and adapts it using only unlabelled data. While several SFDA approaches have recently been proposed, their evaluation focuses on a narrow set of distribu- tion shifts for vision tasks, and their generalizability outside of that scope has not yet been investigated. We put those recent approaches to the test by evaluating set of challenging due extreme covariate and label shift- and aaturall-oocurring distribution shifts in the audio domain. We study the task of adapting a bird species classifier trained focalized recordings of bird songs to datasets of passive recordings for various geographical locations. Inter- estingly, we find that some recent SFDA methods underperform doing no adap- tation at all. Drawing inspiration from those findings and insights, we propose a new method that improves on noisy student approaches by adjusting the teacher's pseudo-labels through Laplacian regularization. Our approach enjoys increased stability and significantly better performance on several of our proposed distribu- tion shifts. We then look back at SFDA benchmarks in the vision domain and find that our approach is competitive with the state-of-the-art there as well.
them
INTRODUCTION
Deep learning has made significant progress on a wide range of application areas. An important contributing factor has been the availability of increasingly larger datasets and models (Kaplan et al., 2020 Song al. 2022). downside of this trend is that training state-of-the-art models has also become increasingly expensive. This is not only wasteful from an environmental perspective, but also makes the training of such models inaccessible to some practitioners due to the prohibitive resources required, or potential difficulties with data access. On the other hand, directly reusing already-trained models is often not desirable, as their performance can degrade significantly in the presence of distribution shifts during deployment (Geirhos et al., (2220). Therefore, a fruitful avenue is designing adaptation methods for pre-trained models to succeed on a new target domain, without requiring access the original (source) training data, i.e., "source-free". Preferably this adaptation can be performed unsupervised. This is the problem of source-free domain adaptation (SFDA) that we target in this work.
Several models have been proposed recently to tackle SFDA. However, we argue that evaluation in this area is significant challenge in and of itself: We desire SFDA methods that are general, in that they can be used for different applications to adapt an cope with a wide range of distribution shifts. Unfortunately, the standard evaluation protocol only considers narrow set of shifts in computer vision tasks, leaving us with limited view of the relative merits among different SFDA methods, as well as their generalizability. In this work, address this limitation by studying evaluation methods, in order to gain as much new information as possible about SFDA methods. We also argue that we should target distribution shifts that naturally-occura This maximizes the chances of the resulting research advances being directly translated into progress in solving real-world problems.
approorrraa pre-trained model
set of distribution shifts. We expand on the existing
To that end, we propose to study a new set of distribution shifts in the audio domain. Specifically, use bird species classifier that trained on a large dataset of bird song recordings as our pre-

---

Under review as a conference paper at ICLR 2023
Table 1: Relationship of problem settings, x and y denote inputs and labels, and s and "source" and "target", respectively (note that in some cases, as in DG, might be union of source domains / environments). For TTA and SFDA, the * in their training data and loss reflects that they are entirely agnostic to how source training is performed, allowing the use of any off-the-shelf model.
DA
DG
TTT
TTA
SFDA
Data used for training Data used for adaptation
x², y. xf v', x
v²
x² x²,""
x", V x
x²'
x²
x²
Training loss Adaptation loss
[(*", v²) + f(x",x') x²)
[(x",y") L(x", .v)) 3')+ L(x')
*
((x')
[(x²)
L(x')
trained model. This dataset consists of focalized recordings, where the song of the identified bird is at the foreground of the recording. Our goal is to adapt this model to a set of passive recordings (soundscapes). The shift from focalized to soundscape recordings is substantial, in the latter often feature much lower signal-to-noise ratio, several birds vocalizing at once, as well as significant distractors and environmental noise like rain or wind. In addition, the soundscapes we consider originate from different geographical locations, inducing extreme label shifts.
the recordings
Our rationale for choosing study these shifts is threefold. Firstly, they are challenging, as evi- denced by the poor performance on soundscape datasets compared to focalized recordings observed by Goëau et al. (2018); Kahl al. (20TT) Secondly, they are naturally occurring and any progress in addressing them can support ecological monitoring and biodiversity conservation efforts and re- search. Finally, our resulting evaluation framework is "just different enough" from the established one: It differs in terms of i) the modality (vision vs. audio), ii) the problem setup (single-label vs multi-label classification), and iii) the degree and complexity of shifts (we study extreme covariate shifts that co-occur with extreme label-space shifts). Existing SFDA methods designed for vision tasks can also be evaluated using this new framework, since audio inputs are often represented as spectrograms and thus can be treated as images.
We perform a thorough empirical investigation of established SFDA methods on our new shifts. Interestingly, not only do some of the methods not improve the performance of the pre-trained model, they often degrade it. Studying this striking finding generates insights that lead us to make substantial modelling improvements. Notably, in the presence of extreme shifts, we observe that the confidence of the pre-trained model drops significantly and its calibration is poor as result. This in turn poses a challenge for entropy minimization. While Noisy Student (Xie al.. 2020) copes best with this shift, it exhibits poor stability, necessitating careful carly-stopping. This violates our assumption that labelled target data is unavailable. Our insight is that we can leverage the model's 's feature space as another "source of truth", as this space carries rich information about the relationship between examples. We propose Noisy Student Teacher with Laplacian Adjustment (NOTELA), new model that enhances the Noisy Student model with a Laplacian regularizer. This model enjoys significantly increased performance on our new shifts as well as increased stability and resilience to model miscalibration. Closing the loop, we also evaluate our model on established vision benchmarks and observe that it is competitive with the state-of -ffte-of-faa -f---of-thhheee ahe there; in fact largely surpassing it on two datasets.
RELATED WORK
Our discussion of related work is summarized in Table
Domain adaptation (DA). DA assumes a setting in which labelled data is available for a source domain, and unlabelled data for a target domain. The goal is to maximize performance on the target domain. DA methods can be roughly divided into three types (Sagawa et al.) 2022): domain- invariant training (also called feature alignment) aims to ensure that the features generated by the model for the source and target domain are indistinguishable by some metric (Sun et al. 2016
Sun & Saenko 2016 Tzeng et al. 2014 Long et al. 2015 Ganin al. 2016 Long et al. 2018
2

---

Under review as a conference paper at ICLR 2023
Tzeng al. 2017
Sankaranarayanan al. 2018);
8); self-training involves generating apeudo-labell
for the unlabelled data (Xie et 2020); and relfssupevviiion involves training an unsupervised/self- supervised model, later finetuned jointly trained with supervised objective (Shen et al. 2022).
Source-Free Domain Adaptation (SFDA) and Test-time Adaptation (TTA), These methods additionally assume that the source data itself is not available. e.g., because of resource, privacy, or intellectual property concerns. The distinction between SFDA and TTA is subtle: the latter is transductive, motivated by an online setup where adaptation happens on (unlabelled) target examples as they appear and evaluation is subsequently performed on the same examples. SFDA considers an offline adaptation phase and the adapted model then evaluated on different set of held-out examples. In practice, though, the methods developed for either are similar enough to be applicable to both. Related problems also include black-box (Zhang et al.) (021), online (Yang et al. 2020). continual WWag et al. 2022b), and universal (Kundu et al. 2020 source- free domain adaptation.
Of the three types of DA methods discussed above, self-training most easily transfers to the SFDA and TTA settings (Liang et al. 2020: Kim et al.. [021], and in this work we focus this category since it's also the most generaliza to new modalities. Other methods use output prediction uncer- tainty for adaptation (Yang al. 2020; Wang al. 2022 Roy et al., 2022) or generative training to transform target domain examples or synthesize new ones (Li et al. 2020 Hou & Zheng. 2020
Kurmi et al., 2021 Morerio al., 2020; [Shhoo et al., 2020). Interestingly, Boudiaf al. (2022) show that previous methods suffer from large hyperparameter sensitivity, and may degrade the per- formance of the source model if not tuned a scenario spprciii manner; an unrealistic setting that violates the assumption of unavailability of labelled target data.
Test-time Training (TTT). TTT (Sun et al., 2020) is a related problem where, like in TTA, a pre-trained model is adapted the target test examples using self-supervised loss, before making a prediction on those examples. Unlike SFDA and TTA, though, TTT modifies the source training phase by incorporating similar sule-supe loss there too.
Domain generalization (DG). In DG (Wang et al. 2022a), like in SFDA, the target domain is unknown. However, unlike SFDA, no adaptation set available. Instead the aim is to train robust source model which works directly on new target distributions. Another important distinction that DG assumes that information about the source domain available during deployment on the target domain. A popular strategy for DG is to increase the source model's 's generalizablity by exposing to diverse "conditions" at training time via domain randomization (Tobin et al. 2017) or adversarial data augmentation (Volpi et al. 2018: Zhou et al. 2020), or to learn domain-invarenn representations by training to match all available training "environments"" (Arjovsky et al. 2019 Creager et al.) (221)) minimizing the worst-case loss
set of such environments (Sagawa et al.
2020). or decomposing the latent space or model weights into domain-speente sppecffc and domain-geners
components (Ilse al. 2020 Khosla et al. 2012).
BACKGROUND AND PRELIMINARY INVESTIGATION
3.1 PROBLEM FORMULATION
Notation. In SFDA for classification, we assume access to a pre-trained model fe X - RC, where X denotes the input space and C' represents the number of classes. This model was trained on a source dataset D. sampled from a source distribution p, (x)), and needs to be adapted to a shifted target distribution (x( # (x). We assume only have access unlabelled data DDdddape sampled from Pt. The goal is formulate an adaptation procedure (0g, Deddapt) + O, that produces an adapted version of the original model using the unlabelled dataset. The adapted model's 's performance then evaluated on held-out data Dojest sampled from Pt.
Single- vs. multi-label classification. SFDA methods have traditionally addressed single-label classification, in which exactly one category of interest is present in a given sample. Multi-label classification relaxes this assumption by considering that any number of categories (or none) may be present in a given sample, which is common in real-world data. In the single-label case the output probability for sample x; is noted p.. softmax(ff(xx)) [0, [0. In the multi-label
3

---

Under review as a conference paper at ICLR 2023
case, the predictions for each class treated separate binary classification problems, resulting in e (o((f(x;))) (ff(()))) E [0, 1]2xc, where is the logistic function.
3.2 BIOACOUSTICS TASK
We use Xeno-Canto (XC; Vellinga & Planqué 2015) as the source dataset D, for bird species clas- sification in the audio domain. XC is a large collection of user-contributee recordings of wild birds from across the world. Recordings are focal (targeted recordings of an individual, captured in natu- ral conditions, as opposed to the passive capture of all ambient sounds). Each recording is labeled with the species of the targeted individual; other birds may appear in the recording.
To evaluate adaptation to distribution shifts, we use multiple collections of passive (also called soundscape) recordings from various geographical locations datasets exhibit major covariate and label distribution shift from the source dataset. By virtue of being passively recorded, the annotated species tractors. Additionally, the geographical concentration of the datasets means that only a subset of XC's large number of species is present in each dataset, and the label distribution of those species does not necessarily follow that of XC. As result, models trained on focal recordings have trouble generalizing soundscapes recordings (Gočau al. 2018 Kahl et al. 2021
target datasets. The soundscape
occluded by environmental noise and dis-
3.3 CATEGORIZATION APPROACHES
We will evaluate several methods which have been proposed both for (source- domain adapta- tion on our new shifts. We choose to investigate methods that fit two criteria: generality across tasks and modalities, and high performance. In this section we present our categorization of methods we consider, which we then build upon in Section
Entropy Minimization (EM). These methods enforce the cluster assumption, i.c., that the bound- aries described by the model's head should not cross any high-density region of samples in the feature space. Geometrically, this enforces large margins between the classifier's boundaries and the provided samples by encouraging the model to output increasingly confident predictions for the unlabelled samples (Grandvalet & Bengio 2004).
We evaluate TENT (Wang et al., 2021 as a representative example of the EM approach. TENT adapts the source model by minimizing the entropy of its predictions though tuning the normal- ization layers' channel-wise scaling and shifting parameters, and updating the layers' population statistics estimates accordingly.
Teacher-Student (TS). This is self- training paradigm where teacher provides pseudo-labels for the unlabelled examples, and a student then trained predict them. More formally, TS minimizes
Tr Ex. yT log (D..T10)) s.t 1yy = 20
min
Yt...
where y; and (o.-(0) represent the pseudo-label and the model's soft predictions for the sample x;- The pseudo-label is dependent on the model's weight's, O. which are transformed using a weight transformation, т, which is typically set to the identity in TS methods.
Optimization happens in an alternating fashion between student and teacher updates. Specifically, the teacher-step minimizes equation [] w.r. the pseudo-labels (y...... .N while the student-step minimizes equation] [ w.r.t 0 using gradient descent. Different TS methods utilize soft (iie et al. 2020) or hard (Lee et al. 2013) pseudo-labele
indirect way of minimizing entropy: By training to predict each example's pseudo-label (i.c., its most likely label, based on its confidence), the model reinforces its own predictions, thereby increasing its confidence. At the same time, though, it has consistency maximization flavour: Because of the alternation between teacher and student updates, predicting the correct lsbell-label requires consistency throughout time.
Intuitively, TS can be seen
We consider two methods from this category in our investigation: First, pseudo-labelling (PL; Lee et al. 2013) assigns pseudo-labels to unlabelled examples by picking the maximum-probabilng
4

---

Under review as a conference paper at ICLR 2023
NOTELA -
Psecudo-Laablll
-+- Dropout Stadent
SHOT
TENT
--- AdoBN
Sierra Nevada
Powdermill
Hawail
0.6
0.6
0.5
0.5
0.5
0.4
0.4
0.3
0.3
0.3
I0P
Caples Creek
SapSucker Woods
Calombia
0.5
0.50
0.4
0.45
0.3
0.4
0.2
0.3
0.1
2
8 10
8
10
4
8
10
Adaptation epech
Figure 1: All but one SFDA method fail to consistently improve the source model in terms of mAP on Iitsst across distribution shift?" Dropout Student succeeds, but only through early stopping, which is infeasible in the SFDA setting. NOTELA achieves consistently stable convergence while improving upon Dropout Student's performance.
class according to the trained model. In the context of SFDA, this translates into assigning pseudo- labels to unlabelled target domain examples using the model trained on the source dataset. Second, SHOT (Liang et al. 2020) adapts the feature extractor to the target domain while freezing the classifier head. It performs adaptation through a combination of nearest-centroid pseudo-labelling and an information maximization loss.
Denoising Teacher-Student (DTS). This method builds upon the TS framework by adding some form of "noise" to the student, while keeping the teacher clean. Intuitively, predicting clean pseudo- labels from noisy student predictions leads maximizing another type of consistency: between different views of the same inputs. Mathematicallyy DTS differs from TS by setting t * Id during the student's s forward pass, while keeping set to the identity during the teacher's forward pass.
Ideas related to DTS have been explored in semi- superviid (Xie et al., 2020 Miyato et al. 2018) and supe-supe learning (Grill et al., 2020)) Chen et al.) 2020). Notably, Noisy Student (Xie al., 2020) is popular representative that we build upon. However, to keep the approach light, both in terms of computation and hyper-parameter load, we consider simplified model in our investigation, where the same network is used for both the teacher and the student, and dropout is the sole source of noise. We refer to this variant as dropout student (DS). Our proposed model which will be presented later also falls in this category.
3.4 PREVIOUS APPROACHES STRUGGLE ON OUR CHALLENGING SHIFTS
While EM has shown strong performance on various tasks WWang et al. 2021 Vu et al. 2019). it performed badly on the new shifts we studied. We hypothesize that this due a combination of the extremity of distribution shift, as well as the multi-label nature of our classification problem.
Specifically, the single-label scenario minimizing entropy leads to foreing the model to choose a single class for each example, in an increasingly confident manner. On the other hand, in our multi-label scenario there is no constraint that a class should be chosen. This fact, combined with the potential poor calibration and underconfidence caused by very large distribution shifts (Figure 2 (1,2)) can drive the model to a collapsed state where all class probabilities are zero (Figure (3)).
Note that this analysis's 's purpose is to illustrate the failure modes of SFDA methods; the above plots cannot he used for hyperparameter selection because looking at the test. sets.
5

---

Under review as a conference paper at ICLR 2023
In fact, we find in subsectionn 5.1 that TENT underperforms the baseline of no adaptation several datasets and hyperparameter settings due to these issues. Later, we will show that both SHOT and pseudo-labelling perform better, but neither systematically prevents the model from degenerating (see Figure 1) In contrast, we find that DTS works significantly better on our challenging shifts, but suffers from stability issues and necessitates precise early-stopping procedure. This is an important drawback for SFDA, where obtaining a dataset-specife (labelled) validation set may not be possible. Motivated by these findings, we next present our approach SFDA that increases both the performance and stability of DTS methods.
4 LAPLACIAN ADJUSTMENT
(1)
(2)
(3)
(4)
In this section, introduce Noisy TEacher student with Laplacian Ad- justment (NOTELA), new method we designed for SFDA, inspired by
Source data
Target data
After EM
After NOTELA
10:44
Model probsbiitss
10-4
the desiderata of stability and ro- bustness to very large distribution shifts and different problem settings (single- and multi-label, different hy- perparameter configurations).
10-14
10-1
amwerr
imagli
amorad
aneero
umagil
amorod
amerro
amegii
amered
amweyo
ameafi
amared
We begin by exploiting the intuitions obtained from our preliminary inves- tigation. Firstly, EM may not be ap- propriate in the face of severe dis- tribution shifts which result in very poorly calibrated models. On the other hand, the fact that TS consis- tently improves upon EM signals that consistency maximization is a useful "auxiliary task", perhaps due to be- ing more robust miscalibration. In- deed, the fact that DTS further im- proves, with the additional consis-
Figure 2: (1) and (2): When shifted to a domain the source model exhibits poorer calibration than on the origi- nal source model, seen from the lower per-class probabil- ity distributions assigned by the model. (3): Applying en- tropy minimization only worsens the problem, and leads to general collapse. (4); Our method largely boosts the source model without affecting model calibration. Species names are represented using their corresponding eBird (Sullivan et al. 2009) 2014) species code.
tency maximization "task" between clean and noisy views, further strengthens this hypothesis. Tak- ing DTS as our starting place then, we seek to further improve it by utilizing another source of information that hypothesize is also robust to miscalibration. Specifically, while the head of the model may be poorly calibrated, its feature space may still carry useful infor- mation about the relationship between examples. In fact, find that adapting only the linear head using labels on the validation domain described in sscciion 5 boosts the source model by 30% in mAP.
insight is that,
Stemming from this observation, we advocate probing the feature space directly as an auxiliary source of truth. Drawing inspiration from classic ideas on manifold regularization (Belkin et al., 2006). NOTELA instantiates this idea by encouraging that nearby points in the feature space be assigned similar pseudo-lahels. We hypothesize that this may also help with stability, since the targets that the student is asked to predict will vary less over time, due to the slower-changing similarity in feature space.
Formalization. We augment the denoising teacher-student formulation in equation I with Lapla- cian regularization:
N a N y, log(y.)
N
N
A
min V
Tr
N y. log (pi.r()) +
Σ
Wsj y. y,
(2)
N
There are two changes in |qquation 2 compared to Equation First, we introduce scalar weight R that controls the softness of pseudo-labels, which treat as a hyperparameter. Second, have added the third term that represents Laplacian regularizer. The value wiy denotes the affinity between samples and j and is obtained by the penultimate layer of the network.

---

Under review as a conference paper at ICLR 2023
Optimization. Disregarding the pairwise Laplacian term allows to directly obtain a closed-fom solution to Equation 2, namely y; x 1/a However, adding the pairwise affinities makes optimiza- tion more challenging. We simplify the problem by linearizing the Laplacian term, which allows us to recover a closed-form solution:
1/a Y1 P
exp
P
a
The full proof of [Equation can be found in the Appendix. Furthermore, show in Appendix A.6 that under the assumption of positive semi-definite affinity matrix (wyy). Equation becomes an instance of a concave-convex procedure (CCP: Yuille & Rangarajan 2003).
Complexity. Theoretically, the added Laplacian regularization scales quadratically in the number of samples N. In practice, set Wty W/e & where 0 d < k is the number of mutual k-nearest neighbours (Brito et al. 1997) of samples i and j. and Wiy = Wyj 0 if these samples are not in each other's k-nearest neighbours lists. Finding the k-nearest neighbours can be done with O( N log N) average time complexity. [Equation 3 scales as (NCC)) but can be fully vectorized across samples.
EXPERIMENTS
Data processing and source model. XC large dataset containing total of 10,932 species. We process XC recordings by resampling the audio to 32 kHz and extracting 6-second slices of relevant audio (see Appendix A.2 for details). We process soundscape recordings by extracting 5-second slices using the provided bounding-box labels (see Appendix A.3 for details).
During training, we take a random 5-second crop and the audio signal's gain normalized a random value between 0.15 and 0.25. A technique similar to LEAF (Zeghidour et al. 2021 is used to convert the waveform into a spectrogram. Unlike LEAF, we do not learn Gaussian lowpass filter and instead apply the Gabor filters with a stride of 320. The resulting output is a power-spectrogram with time resolution of 100 Hz. For each soundscape target dataset, we use 75-25% split to obtain Dadapt used to adapt the model, and Diest, used to evaluate the adapted model. These datasets are described in more detail in the appendix.
The most comprehensive bird species classifier we are aware of is BirdNet (Kahl 2019: Kahl et al, 2021], however the publicly available checkpoints scape recordings (including many of the test sets considered in this paper) which incompatible with our SFDA methodology. We instead use an EfficientNet-B 1 (Tan & Le 2019) model we trained ourselves. The output of this model is flattened and projected into a 1280-dimensional embedding space. In addition to species prediction, the model is trained with three auxiliary losses for the bird's order, family, and genus (each having 0.25 weight).
trained with combination of focal and sound-
Metrics. We use saiple-wise mean average precision (mAP) and class- mean average preci- sion (cmAP) for evaluation. Both of these metrics are threshold- free and appropriate for multi-label scenarios. Each can be interpreted as a multi-label generalization of mean reciprocal rank, where ranking of model logits is performed either per-sample (for mAP) or per-class (for cmAP) See subsection A.4 in the appendix for formal definitions.
The mAP metric measures the ability of the model to assign higher logits to any species present in an example. By contrast, cmAP is the mean of the model's per-species classification quality (similar to the average of per-species AUC scores). Note that class-averaging in cmAP corrects for class imbalance, while mAP reflects the natural data distribution. To avoid noisy measurements we only consider species with least five vocalizations in the dataset when computing cmAP.
Baselines. In addition to TENT, pseudo-labelling (PL), SHOT, and dropout student (DS), which described in section 3.3
we also consider AdaBN (Li et al.] 2018). This method re- computes the population statistics of batch normalization (Ioff & Szegedy 2015) layers in the pre-trained model using the unlabelled target dataset.
Personal correspondence with the authors.

---

Under review as a conference paper at ICLR 2023
Table Test results on the 6 test target domains (averaged over random seeds) using the optimal hyperparameter configuration found on the validation domain. See Figure for convergence plots.
S. Nevada
Powdermill
Hawai'i
Caples
SSW
Colombia
mAP
cmAP
mAP
cmAP
mAP
cmAP
mAP
cmAP
mAP
cmAP
mAP
emAP
Source
56.2
36.4
51.1
32.2
51.8
33.2
48.5
41.2
42.2
24.7
48.9
37.7
AdaBN
46.6
34.6
37,5
29.8
28.5
27.7
40.3
38.3
37.7
25,4
47.1
36.1
SHOT
$7.8
37.8
53.8
32.5
57.6
29.0
52.1
43.0
36.8
20.0
49.2
38.1
TENT
26.5
25.2
23.9
16.4
30.2
8.1
5.8
29.3
PL.
57.9
38.2
54.8
32.5
25.7
22.6
49.3
42.0
51.6
29.0
49.3
38.0
DS
$9.8
37.6
52.1
32.2
62.7
36.4
56.3
43.0
47.5
26.2
51.1
40.6
NOTELA
65.4
40.0
61.6
33.7
65.8
35.4
58.2
44.2
54.1
31.2
Table 3: Validation and ablation results from the High Sierras bioacoustics dataset.
Method
mAP
cmAP
Dropout
Softness
Laplace Reg.
mAP
cmAP
Source
60.7
46.1
46.0
AdaBN
35.8
/
55.8
44.8
SHOT
65.4
48.7
70.2
48.6
TENT
50.7
40.7
72.7
47.9
PL
65.2
47.9
75.0
50.9
DS
72.7
47.9
NOTELA
75.0
50.9
(b) Dropout Noise, Softness and Laplacian regularization ingredients act symbiotically to provide the best performances. Removing any ingredi- ent leads to significant drops in performances. The first row reflects the source model's performance (no adaptation).
(a) Best validation results for each method on High Sierras.
Hyperparameter selection. SFDA fully unsupervised and hence there is annotated valida- tion set for each target domain. Therefore, we approach hyperparameter selection in a non-standard way: We assume have one annotated domain which can be used for validation and finding the optimal syper-parameter configuration. Those hyyer-parameee each method's generalization ability on held-out target domains. We employ the High Sierras dataset for validation and use the remaining domains described in insubbection 3.2 for testing only, For every method, we search over hyperparameters such the learning rate, the subset of parameters to search
are then fixed, and used to evalu-
adapt, and whether parameters such as the weight in SHOT, the confidence threshold PL, or the weights {a, A} in NOTELA. The overview of tuned hyperparameters be found in the appendix.
dropout during adaptation. Additionally,
specific hyper-
5.1 TAKEAWAYS
Existing SFDA methods degrade the baseline. We found neither AdaBN TENT able to improve the source model, regardless of the hyperparameter configuration or the domain. As for SHOT and PL, we were able to find hyper-parameters that improved over the source model's performance on the validation set, as observed in Table 3a We observed however that the gains did consistently translate to the test domains. For example, the PL method was able to boost the source model by 5% mAP on the validation domain, while degrading it by more than 25% mAP on Hawai'i test domain.
NOTELA improves stability and performance. We found that all existing methods displayed plateau followed by degradation of the model's performance. This a serious drawback given the absence of domain-specifie labelled validation set for tuning the training schedule. This is true even for Dropout Student, which was the only previous approach able to consistency outperform find that NOTELA not only addresses these stability
the baseline of no adaptation. In contrast, issues, as demonstrated by Figure 1] but also outperforms considered baselines, setting the state of the art
challenging shifts.
Laplacian regularization, softness, and noise are symbiotic. Following our previous observa- tions, try gain insights into the relative importance of each ingredient in NOTELA. We find in
8

---

Under review as a conference paper at ICLR 2023
Table Top-1 accuracy (averaged over random seeds) on vision test benchmarks. NOTELA ap- proaches best methods on CIFAR- 10-C, and surpasses them on ImageNet variants. t: Aggregat- ing confidence intervals across the 15 corruptions is not trivial. Instead, we report per-corruption confidence-intervals in Appendix subsection A.7
CIFAR-101 10-C1
Method
ImageNet-R
ImageNet-Sketch
(Average across severity comruptions)
Source
56.66
23.16 0.0
21.67
AdaBN (i et al. 2018)
80.10
24.54 + 0.08
22.55 :.14
SHOT
Liang
2020
82.62
25.76 0.15
24.9
TENT
Wang al.
2021
83.41
28.55 + 0.09
25.04
PL (Lee ct al. 2013
83.85
26.16
Dropout Student
82.17
29.15 + 0.05
27.68 + 0.08
NOTELA (ours)
83.67
35.23 + 0.14
32.50 + 0.09
Table 3b) that the presence of the noise is crucial, and that removing it leads to worse performances than the non-adapted model. Furthermore, removing softness in labels (a = 0) or Laplacian Regu- larization (A 0) significantly under-performs the full method, thereby highlighting the symbiosis of all three components.
5.2 NOTELA PERFORMS STRONGLY ON VISION TASKS
Since our objective to develop a general approach, we turn back to vision to see how NOTELA fares in that modality.
Data. We evaluate on several robustness benchmarks, most of which are used by prior SFDA ap- proaches. First, we use CIFAR- 10-R and ImageNet-C (Hendrycks & Dietterich 2019), a collection of corruptions applied to the CIFAR-IO and ImageNet test sets, spanning 15 corruption types and levels of severity. Second. we use ImageNet-R (Hendrycks al., 2021). which consists of 30,000 images of 200 of ImageNet's 's classes obtained by querying for renditions such as "art", "cartoon", "graffiti", etc. Finally, we experiment on ImageNet-Sketch (Wang et al., 2019) consisting of 50,000 images constructed by querying Google Images for "sketch of (class) 1 for all ImageNet classes.
Models. We adopt model architectures from previous works, namely a ResNet-50 (He et al. 2016) for ImageNet benchmarks and Wide ResNet 28-10 (Zagoruyko & Komodakis, (016)) for CIFAR- 10 benchmarks. We use the same CIFAR-10 Wide ResNet model checkpoint (provided by Croce al. 2021) Wang et al. I for fair comparison. We reimplemented all SFDA methods to ensure they could be tuned and evaluated in the exact same conditions. For each vision dataset we again use a 75-25% split to obtain madapt and Deest, We use the most challenging corruption from ImageNet-C (contrast) as the validation domain for vision tasks. We report top-1 accuracy for all vision SFDA benchmarks.
Results. From Table 4. we find that NOTELA is very competititve vision benchmarks, despite the change in modality and the single-label setup. It performs on-par with the top performer on CIFAR-10- 10-C. but sets the new state-of-the-art on ImageNet-R and ImageNet-Sketce domains.
5.3 CONCLUSION
In this work, we investigate the generalization of recent source-free domain adaptation methods in a new modality to a new and challenging set of distribution shifts: going from focal to passive record- ings of wild bird vocalizations for multi-label audio classification. While trying to adapt it, we made the surprising observation that recent approaches sometimes degrade the source model. Building on our observations, we proposed an approach called NOTELA, which builds upon Dropout Student by incorporating a Laplacian regularizer when computing the Teacher's pseudo-labels and offer- ing lightweight optimization procedure. Our proposed approach greatly outperformed existing SFDA approaches both the studied bioacoustic domains, well the established vision SFDA benchmarks. We believe that NOTELA represents promising first step towards making SFDA approaches applicable to wide range of modalities and distribution shifts.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint urXiv:/9077 907.02893, 2019.
Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric frame- work for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 2006.
Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test- time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
M.R. Brito, E.L. Chávez, A.J. Quiroz, and J.E. Yukich. Connectivity of the mutual k-nearest- neighbor graph in clustering and outlier detection. Statistics & Probability Letters, 35(1))33 -2, 1997. ISSN 0167-7152. doi: https///bbooogio...oooooooo11sss
URL https:
/www htt.://ww...................eeaa
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning, 2020.
Lauren M Chronister, Tessa A Rhinehart, Aidan Place, and Justin Kitzes. An annotated set of audio recordings of eastern north american birds containing frequency, time, and species information, 2021.
Elliot Creager, Jörn-Henrik Jacobsen, and Richard 7emel Environment inference for invariant learning. In Proceedings of the International Conference on Machine Learning, 2021.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adver- sarial robustness benchmark. In NeurlPS Track on Datasets and Benchmarks, 2021.
Madhav Desai and Vasant Rao. A characterization of the smallest eigenvalue of graph. Journal of Graph Theory, 182))181-194, 1994.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. Journal of Machine Learning Research, ((1):2096-20334 2016.
Robert Geirhos, Jörn Heen-k Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):666-673, 2020.
Hervé Goëau, Stefan Kahl, Hervé Glotin, Robert Planqué, Willem-Pier Vellinga, and Alexis Joly. Overview of BirdCLEF 2018: monospecies soundscape bird identification. In Working Notes of CLEF 2018-Conferee and Labs of the Evaluation Forum, 2018.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances Neural Information Processing Systems, 2004.
dean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In Proceedings of the International Conference on Learning Repre- sentations, 2019.
10

---

Under review as a conference paper at ICLR 2023
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Pro- ceedings of the International Conference on Computer Vision, 2021.
Yunzhong Hou and Liang Zheng. Source free domain adaptation with image translation. arXiv preprint arXiv:2008.077 075 14, 2020.
Maximilian llse, Jakub M Tomezak, Christos Louizos, and Max Welling. Diva: Domain invari- ant variational autoencoders. In Proceedings of the Conference on Medical Imaging with Deep Learning, 2020.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference Machine Learning, 2015.
Stefan Kahl, Identifying Birds by Sound: Large-scale Acoustic Event Recognition for Avian Activity Monitoring. PhD thesis, Chemnitz University of Technology. 2019.
Stefan Kahl, Connor M Wood, Maximilian Eibl, and Holger Klinck. BirdNET: deep learning solution for avian diversity monitoring. Ecological Informatics, 61:101236, 2021.
Stefan Kahl, Russell Charif, and Holger Klinck. A collection of fully-annotated soundsccpe record- ings from the Northeastern United States, August 2022a. URL https://dbi.czg//1
2281/zenodo. 7079380
Stefan Kahl, Connor M. Wood, Philip Chaon, M. Zachariah Peery, and Holger Klinck. A collection of fully-annotated soundscape recordings from the Western United States, September 2022b. URL
https://doi. org/10 5281/ /eoooo. 7050014
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A Efros, and Antonio Torralba. Undoing the damage of dataset bias. In Proceedings of the European Conference on Computer Vision, 2012.
Youngeun Kim, Donghyeon Cho, Kyeongtak Han, Priyadarshini Panda, and Sungeun Hong. Do- main adaptation without source data. IEEE Transactions on Artificial Intelligence, 2021.
Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source-free domain adap- tation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Vinod K Kurmi, Venkatesh K Subramanian, and Vinay P Namboodiri. Domain impression: A source data free domain adaptation method. Proceedings of the IEEE/CVE Winter Conference Applications of Computer Vision, 2021.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop Challenges in Representation Learning. ICML, 2013.
Rui Li, Qianfen Jiao, Wenming Cao, Hau-San Wong, and Wu. Model adaptation: Unsuper- vised domain adaptation without source data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
Yanghao Li. Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80: 80: 109-117,7 2018.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do really need to access the source data? Source hypothesis transfer for unsupervised domain adaptation. In Proceedings of the International Con- ference Machine Learning, 2020.
11

---

Under review as a conference paper at ICLR 2023
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In Proceedings of the International Conference on Machine Learning, 2015.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems, 2018.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
Pietro Morerio, Riccardo Volpi, Ruggero Ragonesi, and Vittorio Murino. Generative pseudo-label refinement for unsupervised domain adaptation. Proceedings of the IEEE/CVF Winter Confer- ence on Applications of Computer Vision, 2020.
Amanda Navine, Stefan Kahl, Ann Tanimoto-Johnson, Holger Klinck, and Patrick Hart. A collection of fully-annotated soundscape recordings from the Island of Hawai'i, September 2022. URL. https://doi... org
7078499
Subhankar Roy, Martin Trapp, Andrea Pilzer, Juho Kannala, Nicu Sebe, Elisa Ricci, and Arno Solin. Aincertainty-guider source- free domain adaptation. In Proceedings of the European Conference on Computer Vision, 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In Proceedings of the International Conference on Learning Representations, 2020.
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, et al. Extending the WILDS bench- mark for unsupervised adaptation. In Proceedings of the International Conference on Learning Representations, 2022.
Roshni Sahoo, Divya Shanmugam, and John Guttag. Unsupervised domain adaptation in the absence of source data. In ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning, 2020.
Swami Sankaranarayanan, Yogesh Balaji, Carlos Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018
Kendrick Shen, Robbie Matthew Jones, Ananya Kumar, Sang Michael Xie, and Percy Liang.
How does contrastive pre-training connect disparate domains?, 2022. URL https://
openreview http://ww...eeooormmm./////////a
Li Dong, Wei-Nan Zhang, Ting Liu, and Furu Wei. CLIP models are few-shot learners: Empirical studies VQA and visual entailment. arXiv preprint arXiv:2203.07/90. 2022.
Brian L Sullivan, Christopher L Wood, Marshall J Iliff, Rick E Bonney, Daniel Fink, and Steve Kelling, ebird: A citizen-based bird observation network in the biological sciences. Biological conservation, 142(10):222--2222 2009.
Brian L Sullivan, Jocelyn Aycrigg, Jessie H Barry, Rick E Bonney, Nicholas Bruns, Caren B Cooper, Theo Damoulas, André A Dhondt, Tom Dietterich, Andrew Farnsworth, et al. The ebird enterprise: An integrated approach development and application of citizen science. Biological conservation, 169:31 2014.
Baochen Sun and Kate Saenko. Deep CORAL: Correlation alignment for deep domain adaptation.
In Proceedings of the European conference on computer vision, 2016.
Baochen Sun, Jiashi Feng, and Kate Saenko. Return of frustratingly easy domain adaptation. In Proceedings of the AAAI Conference Artificial Intelligence, 2016.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the International Conference Machine Learning, 2020.
12

---

Under review as a conference paper at ICLR 2023
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105 6114. PMLR, 2019.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In Proceedings of the JEEE/RSJ International Conference on Intelligent Robots and Systems, 2017.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 141..344 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2017.
Willem-Pier Vellinga and Robert Planqué. The xeno-canto collection and its relation to sound recog- nition and classification. CLEF (Working Notes), 2015.
Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing unseen domains via adversarial data augmentation. Advances in Neural Information Processing Systems, 2018.
Mtan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adver- sarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2517-2526, 2019.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In Proceedings of the International Conference on Learning Representations, 2021.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems, 2019.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022a.
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022b.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves ImageNet classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. Casting BAIT for offline and online source-free domain adaptation. arXiv preprint (tti:20000.142222 2020.
Alan Yuille and Anand Rangarajan. The concave-convex procedure. Neural Computation, 2003.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference, 2016.
Neil Zeghidour, Olivier Teboul, Félix de Chaumont learnalle frontend for audio classification. In Proceedings of the International Conference on Learning Representations, 2021.
Quitry, and Marco Tagliasacchi. LEAF: A
Haojian Zhang, Yabin Zhang, Kui Jia, and Lei Zhang. Unsupervised domain adaptation of black-box source models. In Proceedings of the British Machine Vision Conference, 2021.
Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang. Deep aaveriaraa im- age generation for domain generalisation. In Proceedings of the AAAI Conference Artificial Intelligence, 2020.
Imtiaz Ziko, Eric Granger, and Ismail Ben Ayed. Scalable laplacian k-modes. Advances in Neural Information Processing Systems, 2018.
13


Reviews:
[{"review_id": "fCcA3LtDGrl", "reviewer": "Reviewer_qADF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors try to propose a novel method to the task of source-free domain adaptation. In these settings, the method cannot access the instances from the source domain while adapting to the target domain. The authors find gaps in current computer vision literature by proposing a new dataset in audio domain, where the source domain contains audio recordings with each sample having a recording of a single bird and the target domain contains audio with each sample containing recording of multiple birds in the sample. These recordings are converted into an audio spectogram which is an \"image\". Therefore, the authors draw comparisons with several computer vision baselines.", "summary_of_the_review": "This paper makes new contributions to the audio community by proposing a challenging benchmark on bird songs with significant shifts. Further a simple regularizer is proposed which demonstrates improvements over the methods developed for a different computer vision task. Based on the strengths and weaknesses, this paper needs to be revised.", "strength_and_weaknesses": "**Strengths**\n\n* The method identifies potential gaps with existing source free domain adaptation methods.\n* The paper proposes a new audio dataset for improving generalizability of existing computer vision methods. The target domains contain significant domain shift in comparison with source domain.\n* The proposed method is simple and works on the proposed dataset.\n\n**Weaknesses**\n\n* The paper does not compare with works that have been published in speech domain. For a problem like this, I believe that speech domain is more closely related than vision domain. [ref 1]\n\n* The baselines were developed on the task of computer vision classification. It is not surprising that the methods underperform on this task. \n\n* The creation of domain gap does not seem to be systematic - atleast from the perspective of vision methods. In case of target domain, a sample recording seems to contain multiple birds as opposed to a single bird; while the source domain only has a single bird in the dataset. This does not translate well to methods proposed in vision domain since many of these baseline methods were tested on single class. \n\n* The paper does not seem to compare with more advanced source-free methods that were developed for noise from different classes. There has been some recent work in computer vision literature that tries to induce artificial noise by mixing information from different classes. [ref 2] This would be a better source-free computer vision baseline to compare with since the features are mixed from different classes.\n\n\nReferences\n\n[ref 1] Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training, ICASSP'21\n\n[ref 2] Towards Inheritable Models for Open-Set Domain Adaptation, CVPR'20\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OVADXKe4SW", "reviewer": "Reviewer_mxPp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores the feasibility of source-free domain adaptation in the problem of bird classification using focalized recordings of bird songs as the source domain and the datasets of passive recordings in a target domain. The idea is based on training a teacher-student network paid to predict pseudo-labels for the target domain. Laplacian adjustment then is used to improve the prediction accuracy. Experiments on the bird dataset are provided to demonstrate that the method is effective and outperforms existing methods.", "summary_of_the_review": "The area of domain adaptation and its instantiations such as source-free formulation is an overcrowded field with many existing works. Hence, simply proposing a new algorithm is not enough. It should be demonstrated that the proposed method is addressing a challenge that existing methods do not address and outperforms existing methods significantly.  Or, a deeper theoretical understanding must be offered. However:\n\n1. The claim that \"the standard evaluation protocol only considers a narrow set of shifts in computer vision tasks\" is not necessarily true. DomainNet dataset is proposed for exactly the same reason and it has many domains, some with large gaps.\n\n2. The paper has performed experiments on a new dataset. This is great but it makes evaluation difficult. The proposed algorithm should be run on 5-6 comment benchmarks, e.g., Office-Home and DomainNet, that most works in this area use to enable comparison. Currently, the comparison is done with a small number of algorithms. It is not clear how well the proposed method will perform compared to other works not used for comparison. It is also not clear how well the proposed method will work on common benchmarks. For these reasons, I cannot judge how competitive the proposed method is.\n\n3. Experiments should include supervised training on the target domain to enable seeing how challenging the tasks are.\n\n4. The ideas of using Laplacian regularization or teacher-student networks for domain adaptation are not new and have been used in several prior works.\n\nIn conclusion, I think this work is not ready for publication and only is a good start.", "strength_and_weaknesses": "Strengths\n\n1. The paper is written well and easy to follow.\n\n2. The paper recognizes a good research question.\n\nWeaknesses:\n\n1. The claims of the paper are unfounded.\n\n2. The idea is incremental.\n\n2. Experiments do not help the evaluation of the proposed method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "LeAv0Plf7L", "reviewer": "Reviewer_kHC4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This submission studies the source-free domain adaptation (SFDA) problem under both covariate shift and label shift in the audio domain. Specifically, the authors study the task of adapting from focal to passive recordings of wild bird vocalizations for multi-label audio classification. The authors find some SFDA baselines underperform the non-adapted model in the considered task, which motivates them to propose the method NOTELA. NOTELA is based on the noisy student framework and involves two extra terms: encouraging soft prediction and Laplacian regularization. Experiments on the audio task demonstrate NOTELA outperforms other SFDA methods. Experiments on several vision benchmarks also show the efficacy of NOTELA.", "summary_of_the_review": "This paper targets solving a meaningful and realistic source-free domain adaptation problem for audio tasks with complex distribution shifts. Experiments on the audio benchmarks and vision benchmarks prove the efficacy of the proposed method NOTELA. However, I have two main concerns. The first concern is that NOTELA lacks novelty due to combining existing techniques. The second is that the empirical evaluation is not comprehensive for the problem of source-free domain adaptation.", "strength_and_weaknesses": "### <Strengths>\n1. It is meaningful to study the domain adaptation problem in modalities besides vision and under complex distribution shifts.\n2. The self-training framework and the neighborhood consistency term are reasonable and effective.\n3. Implementation details are provided for good reproducibility.\n\n### <Weaknesses>\n1. The proposed method lacks novelty and seems a combination of techniques in previous works. The dropout-based noisy student is the same as [A] as agreed by the authors. The other part in NOTELA, i.e., the Laplacian regularization term, is very similar to the neighborhood regularization of target data in [B, C]. \n2. The proposed method requires carefully tuning many hyper-parameters including $\\lambda$ and $\\alpha$ in Eq. (2). Although the authors use another labeled domain for validation, the condition is not always available and it is common that different target domains prefer different hyper-parameters in realistic scenarios.\n3. The problem setting and considered baselines are not clear and even confusing. The authors claim that they consider SFDA under complex distribution shifts with both covariate shifts and label shifts. However, the paper did not consider any popular vision benchmarks to verify the proposed method, including CIFAR10 with curated label shift for domain adaptation under label shift and CIFAR10 with curated label shift, Office-Home, VisDA, DomainNet, and WILDS for SFDA. In addition, the considered baselines are not competitive, even the most up-to-date SHOT and TENT are SFDA works of two years ago. Recent SFDA such as [B, C, D, E] works should be considered for comparison. \n\nReferences\n[A] Self-training with Noisy Student Improves ImageNet Classification, CVPR 2020\n[B] Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation, NeurIPS 2021\n[C] Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptation, arxiv 2205.04183\n[D] Adaptive Adversarial Network for Source-Free Domain Adaptation, ICCV 2021\n[E] Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data, NeurIPS 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Cdxfpf7OWu", "reviewer": "Reviewer_2cf7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of source-free domain adaptation (SFDA), where a pre-trained source model is adapted using unlabeled target domain data without accessing any source domain data. While previous SFDA algorithms only considered a narrow set of domain shifts in computer vision tasks, the authors put them on a new set of more challenging naturally-occurring scenarios with extreme covariate and label shifts. They propose a new benchmark dataset based on focalized and passive recordings of bird songs. Interestingly, they show that previous SFDA methods underperform in the new dataset. To propose a more generalizable SFDA method, the authors propose NOTELA which enhances the Noisy Student Teacher with a Laplacian Adjustment. ", "summary_of_the_review": "While the paper has its merits, unfortunately, it also has several issues which need to be addressed: (1) unclear motivations for the proposed method, (2) questionable novelty due to the similarities between NOTELA and NRC, and (3) more widely used vision datasets for evaluating SFDA methods.", "strength_and_weaknesses": "Strengths\n- I like the idea of evaluating the previous SFDA methods on a new set of more challenging naturally-occurring scenarios. I do agree that they should be also working well on other datasets unless they leverage techniques specifically applicable to the computer vision domain. \n- I’m not familiar with the audio datasets. However, it seems the proposed audio dataset contains naturally occurring extreme covariates and label shifts. It will provide a good testbed for developing more generalizable SFDA methods.\n- I also like how the authors deal with the hyperparameter selection in SFDA. Previous works often neglect the difficulty of the hyperparameter selection and violate the assumption of the unavailability of labeled target data. In contrast, the authors use one dataset for validation and fix the hyperparameters for the other datasets.\n\nWeaknesses\n- The authors stated that their insight for the proposed NOTELA is that they can leverage the model’s feature space as another “source of truth”. However, in my view, the motivation does not seem new and it has been the common motivation for numerous previous SFDA methods (Ex. Yang et al., Exploiting the intrinsic neighborhood structure for source-free domain adaptation, NeurIPS 2021.). They assume that the feature space forms meaningful clusters and rely on the clusters to generate pseudo-labels for unlabeled target data. It would be nice if the authors can provide more specific motivations and insights for the proposed model.\n- The proposed NOTELA encourages the nearby points in the feature space to be assigned similar pseudo-labels. Specifically, they define an affinity score between samples using the number of mutual k-nearest neighbors. I think NOTELA holds great similarities with NRC proposed in the previous work (Yang et al., Exploiting the intrinsic neighborhood structure for source-free domain adaptation, NeurIPS 2021.). In essence, NRC also defined local affinity with reciprocal nearest neighbors within the feature space and encourage label consistency among data with high local affinity. Please look into the work and explain how the NOTELA is different. Considering the similarities, I think NRC should be also included in all the experiments as a baseline.\n- The authors claim that NOTELA is a more generalizable method for SFDA methods. Even though they evaluated the proposed method in some computer vision datasets, they are not widely used datasets for evaluating SFDA methods. To claim that NOTELA is also better in computer vision datasets, I recommend using more widely used vision datasets for evaluating SFDA methods (e.g. OfficeHome and VISDA-C).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject