Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors try to propose a novel method to the task of source-free domain adaptation. In these settings, the method cannot access the instances from the source domain while adapting to the target domain. The authors find gaps in current computer vision literature by proposing a new dataset in audio domain, where the source domain contains audio recordings with each sample having a recording of a single bird and the target domain contains audio with each sample containing recording of multiple birds in the sample. These recordings are converted into an audio spectogram which is an "image". Therefore, the authors draw comparisons with several computer vision baselines.
strength_and_weaknesses: **Strengths**

* The method identifies potential gaps with existing source free domain adaptation methods.
* The paper proposes a new audio dataset for improving generalizability of existing computer vision methods. The target domains contain significant domain shift in comparison with source domain.
* The proposed method is simple and works on the proposed dataset.

**Weaknesses**

* The paper does not compare with works that have been published in speech domain. For a problem like this, I believe that speech domain is more closely related than vision domain. [ref 1]

* The baselines were developed on the task of computer vision classification. It is not surprising that the methods underperform on this task. 

* The creation of domain gap does not seem to be systematic - atleast from the perspective of vision methods. In case of target domain, a sample recording seems to contain multiple birds as opposed to a single bird; while the source domain only has a single bird in the dataset. This does not translate well to methods proposed in vision domain since many of these baseline methods were tested on single class. 

* The paper does not seem to compare with more advanced source-free methods that were developed for noise from different classes. There has been some recent work in computer vision literature that tries to induce artificial noise by mixing information from different classes. [ref 2] This would be a better source-free computer vision baseline to compare with since the features are mixed from different classes.


References

[ref 1] Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training, ICASSP'21

[ref 2] Towards Inheritable Models for Open-Set Domain Adaptation, CVPR'20

clarity,_quality,_novelty_and_reproducibility: The paper is well written but the motivation is not placed appropriately. The paper seems to indicate that audio datasets would be able to benchmark computer vision methods which is incorrect in my opinion.

The paper contributes a new dataset and a simple method that is able to address the domain gap and perform well.

It should be possible to reproduce the method based on the details from the paper.
summary_of_the_review: This paper makes new contributions to the audio community by proposing a challenging benchmark on bird songs with significant shifts. Further a simple regularizer is proposed which demonstrates improvements over the methods developed for a different computer vision task. Based on the strengths and weaknesses, this paper needs to be revised.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This submission studies the source-free domain adaptation (SFDA) problem under both covariate shift and label shift in the audio domain. Specifically, the authors study the task of adapting from focal to passive recordings of wild bird vocalizations for multi-label audio classification. The authors find some SFDA baselines underperform the non-adapted model in the considered task, which motivates them to propose the method NOTELA. NOTELA is based on the noisy student framework and involves two extra terms: encouraging soft prediction and Laplacian regularization. Experiments on the audio task demonstrate NOTELA outperforms other SFDA methods. Experiments on several vision benchmarks also show the efficacy of NOTELA.
strength_and_weaknesses: ### <Strengths>
1. It is meaningful to study the domain adaptation problem in modalities besides vision and under complex distribution shifts.
2. The self-training framework and the neighborhood consistency term are reasonable and effective.
3. Implementation details are provided for good reproducibility.

### <Weaknesses>
1. The proposed method lacks novelty and seems a combination of techniques in previous works. The dropout-based noisy student is the same as [A] as agreed by the authors. The other part in NOTELA, i.e., the Laplacian regularization term, is very similar to the neighborhood regularization of target data in [B, C]. 
2. The proposed method requires carefully tuning many hyper-parameters including $\lambda$ and $\alpha$ in Eq. (2). Although the authors use another labeled domain for validation, the condition is not always available and it is common that different target domains prefer different hyper-parameters in realistic scenarios.
3. The problem setting and considered baselines are not clear and even confusing. The authors claim that they consider SFDA under complex distribution shifts with both covariate shifts and label shifts. However, the paper did not consider any popular vision benchmarks to verify the proposed method, including CIFAR10 with curated label shift for domain adaptation under label shift and CIFAR10 with curated label shift, Office-Home, VisDA, DomainNet, and WILDS for SFDA. In addition, the considered baselines are not competitive, even the most up-to-date SHOT and TENT are SFDA works of two years ago. Recent SFDA such as [B, C, D, E] works should be considered for comparison. 

References
[A] Self-training with Noisy Student Improves ImageNet Classification, CVPR 2020
[B] Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation, NeurIPS 2021
[C] Attracting and Dispersing: A Simple Approach for Source-free Domain Adaptation, arxiv 2205.04183
[D] Adaptive Adversarial Network for Source-Free Domain Adaptation, ICCV 2021
[E] Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data, NeurIPS 2021
clarity,_quality,_novelty_and_reproducibility: 1. Clarity: The presentation of the method is well-organized and easy to understand. The presentation of the studied problem is confusing.
2. Quality: There is no theoretical analysis of the proposed method under complex distribution shifts. The empirical evaluation is not comprehensive.
3. Novelty: The novelty is weak due to combining lots of existing techniques.
4. Reproducibility: The reproducibility is good due to the provided details of the hyper-parameter setting.
summary_of_the_review: This paper targets solving a meaningful and realistic source-free domain adaptation problem for audio tasks with complex distribution shifts. Experiments on the audio benchmarks and vision benchmarks prove the efficacy of the proposed method NOTELA. However, I have two main concerns. The first concern is that NOTELA lacks novelty due to combining existing techniques. The second is that the empirical evaluation is not comprehensive for the problem of source-free domain adaptation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper tackles the problem of source-free domain adaptation (SFDA), where a pre-trained source model is adapted using unlabeled target domain data without accessing any source domain data. While previous SFDA algorithms only considered a narrow set of domain shifts in computer vision tasks, the authors put them on a new set of more challenging naturally-occurring scenarios with extreme covariate and label shifts. They propose a new benchmark dataset based on focalized and passive recordings of bird songs. Interestingly, they show that previous SFDA methods underperform in the new dataset. To propose a more generalizable SFDA method, the authors propose NOTELA which enhances the Noisy Student Teacher with a Laplacian Adjustment. 
strength_and_weaknesses: Strengths
- I like the idea of evaluating the previous SFDA methods on a new set of more challenging naturally-occurring scenarios. I do agree that they should be also working well on other datasets unless they leverage techniques specifically applicable to the computer vision domain. 
- I’m not familiar with the audio datasets. However, it seems the proposed audio dataset contains naturally occurring extreme covariates and label shifts. It will provide a good testbed for developing more generalizable SFDA methods.
- I also like how the authors deal with the hyperparameter selection in SFDA. Previous works often neglect the difficulty of the hyperparameter selection and violate the assumption of the unavailability of labeled target data. In contrast, the authors use one dataset for validation and fix the hyperparameters for the other datasets.

Weaknesses
- The authors stated that their insight for the proposed NOTELA is that they can leverage the model’s feature space as another “source of truth”. However, in my view, the motivation does not seem new and it has been the common motivation for numerous previous SFDA methods (Ex. Yang et al., Exploiting the intrinsic neighborhood structure for source-free domain adaptation, NeurIPS 2021.). They assume that the feature space forms meaningful clusters and rely on the clusters to generate pseudo-labels for unlabeled target data. It would be nice if the authors can provide more specific motivations and insights for the proposed model.
- The proposed NOTELA encourages the nearby points in the feature space to be assigned similar pseudo-labels. Specifically, they define an affinity score between samples using the number of mutual k-nearest neighbors. I think NOTELA holds great similarities with NRC proposed in the previous work (Yang et al., Exploiting the intrinsic neighborhood structure for source-free domain adaptation, NeurIPS 2021.). In essence, NRC also defined local affinity with reciprocal nearest neighbors within the feature space and encourage label consistency among data with high local affinity. Please look into the work and explain how the NOTELA is different. Considering the similarities, I think NRC should be also included in all the experiments as a baseline.
- The authors claim that NOTELA is a more generalizable method for SFDA methods. Even though they evaluated the proposed method in some computer vision datasets, they are not widely used datasets for evaluating SFDA methods. To claim that NOTELA is also better in computer vision datasets, I recommend using more widely used vision datasets for evaluating SFDA methods (e.g. OfficeHome and VISDA-C).

clarity,_quality,_novelty_and_reproducibility: Clarity
The paper was generally clear and easy to follow.

Quality
I think the paper holds good quality. It presents new datasets for evaluating SFDA methods and demonstrates empirical justifications for their motivation and proposed method.

Novelty
The originality of the work is questionable. I think NOTELA holds great similarities with NRC proposed in the previous work (Yang et al., NeurIPS 2021)

Reproducibility
The paper seems to provide enough details to reproduce their experiments. I also expect the authors to open-source their codes in the future for better reproducibility.

summary_of_the_review: While the paper has its merits, unfortunately, it also has several issues which need to be addressed: (1) unclear motivations for the proposed method, (2) questionable novelty due to the similarities between NOTELA and NRC, and (3) more widely used vision datasets for evaluating SFDA methods.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper explores the feasibility of source-free domain adaptation in the problem of bird classification using focalized recordings of bird songs as the source domain and the datasets of passive recordings in a target domain. The idea is based on training a teacher-student network paid to predict pseudo-labels for the target domain. Laplacian adjustment then is used to improve the prediction accuracy. Experiments on the bird dataset are provided to demonstrate that the method is effective and outperforms existing methods.
strength_and_weaknesses: Strengths

1. The paper is written well and easy to follow.

2. The paper recognizes a good research question.

Weaknesses:

1. The claims of the paper are unfounded.

2. The idea is incremental.

2. Experiments do not help the evaluation of the proposed method.
clarity,_quality,_novelty_and_reproducibility: The paper is written well and easy to follow. The quality of the presentation is good but some claims are unfounded (see below). In terms of novelty, the idea is incremental because Laplacian regularization and teach-student networks both have been used for domain adaptation before. In this sense, the paper is not very novel. The code is not included and it is not easy to judge whether the results are reproducible, particularly because the dataset in not common in the area of domain adaptation.
summary_of_the_review: The area of domain adaptation and its instantiations such as source-free formulation is an overcrowded field with many existing works. Hence, simply proposing a new algorithm is not enough. It should be demonstrated that the proposed method is addressing a challenge that existing methods do not address and outperforms existing methods significantly.  Or, a deeper theoretical understanding must be offered. However:

1. The claim that "the standard evaluation protocol only considers a narrow set of shifts in computer vision tasks" is not necessarily true. DomainNet dataset is proposed for exactly the same reason and it has many domains, some with large gaps.

2. The paper has performed experiments on a new dataset. This is great but it makes evaluation difficult. The proposed algorithm should be run on 5-6 comment benchmarks, e.g., Office-Home and DomainNet, that most works in this area use to enable comparison. Currently, the comparison is done with a small number of algorithms. It is not clear how well the proposed method will perform compared to other works not used for comparison. It is also not clear how well the proposed method will work on common benchmarks. For these reasons, I cannot judge how competitive the proposed method is.

3. Experiments should include supervised training on the target domain to enable seeing how challenging the tasks are.

4. The ideas of using Laplacian regularization or teacher-student networks for domain adaptation are not new and have been used in several prior works.

In conclusion, I think this work is not ready for publication and only is a good start.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the source-free domain adaptation problem - it proposes a new algorithm (NOTELA), a new benchmark dataset for evaluation, and a hyperparameter tuning protocol for evaluating such methods. 

Reviewers appreciated that the new audio dataset reflects complex real-world domain shifts and that the proposed NOTELA method performs well in the experiments. However, they had major unresolved concerns regarding novelty and comprehensiveness of the empirical evaluations; authors provided a response but did not provide a revision of the paper or additional evaluations. On novelty, reviewers thought that the method is a combination of techniques that have been previously proposed for domain adaptation. On the experiments, the concern was that recent baselines and standard datasets used in the SFDA literature were not included in the comparisons; reviewers thought that claims of generality of the method were not supported due to this lack of evaluation on widely used datasets.

Overall, the paper needs a more comprehensive evaluation to support claims of generality, and a clear discussion of the specific conceptual advances provided by the proposed method before it is ready for publication.
justification_for_why_not_higher_score: - Lack of conceptual novelty of the method; it is a straightforward combination of existing techniques
- Lack of comprehensive evaluation on common benchmarks to support the claim that the method is generalizable
Decision: Reject