DA-Bench: Benchmarking Unsupervised Domain Adaptation Methods with Realistic Validation On Diverse Modalities

Authors: Yanis Lalou, Theo Gnassounou, Antoine Collas, Antoine de Mathelin, Oleksii Kachaiev, Ambroise Odonnat, Alexandre Gramfort, Thomas Moreau, Rémi Flamary
Primary area: datasets and benchmarks
Keywords: Benchmark, Unsupervised Domain Adaptation, Model selection

Abstract:
Unsupervised Domain Adaptation (DA) consists of adapting a model trained on a labeled source domain to perform well on an unlabeled target domain with some data distribution shift.
While many methods have been proposed in the literature, fair and realistic evaluation remains an open question, particularly due to methodological difficulties in selecting hyperparameters in the unsupervised setting.
With DA-Bench, we propose a framework to evaluate DA methods on diverse modalities, beyond computer vision task that have been largely explored in the literature. We present a complete and fair evaluation of existing shallow algorithms, including reweighting, mapping, and subspace alignment.
Realistic hyperparameter selection is performed with nested cross-validation and various unsupervised model selection scores, on both simulated datasets with controlled shifts and real-world datasets across diverse modalities, such as images, text, biomedical, and tabular data.
Our benchmark highlights the importance of realistic validation and provides practical guidance for real-life applications, with key insights into the choice and impact of model selection approaches.
DA-Bench is open-source, reproducible, and can be easily extended with novel DA methods, datasets, and model selection criteria without requiring re-evaluating competitors.

Decision: reject
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730125018858

thread_type:
official_review

<thread_object>
creator: reviewer_rwx7

<content>
summary:
This paper introduces DA-Bench, a framework for evaluating unsupervised domain adaptation (DA) methods across diverse modalities, such as images, text, biomedical, and tabular data. It emphasizes fair and realistic hyperparameter selection using nested cross-validation and unsupervised model selection metrics.

strengths:
DA-Bench provides a relatively comprehensive evaluation of existing shallow algorithms and offers practical insights for real-life applications. It is open-source, reproducible, and easily extendable with new DA methods, datasets, and model selection criteria.

weaknesses:
The paper has the following shortcomings. First, the methods tested are outdated, lacking work from the past three years. Second, the correlation between different modalities is weak, casting doubt on whether transferability can be achieved. Lastly, the paper does not propose any new theories.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642585620

thread_type:
official_review

<thread_object>
creator: reviewer_qzy2

<content>
summary:
This paper introduces DA-Bench, a framework for evaluating Unsupervised Domain Adaptation methods across diverse modalities beyond computer vision. Through realistic assessments and nested cross-validation, it comprehensively evaluates shallow algorithms like reweighting and mapping. The study emphasizes the importance of realistic validation, offering insights into model selection approaches.

strengths:
1.	The paper is well-written and easily comprehensible.
2.	The proposed benchmark is comprehensive and straightforward.
3.	The motivation for validating the parameters of the methods, such as unsupervised scorers that do not rely on target labels, is commendable.
4.	Extensive experiments across multiple datasets are adequate.

weaknesses:
1. Figure 1 is far from self-contained. Could you explain the detailed relationships and distinctions among these four shifts according to the Figure? Also, in this figure, it appears to represent a binary classification task. Do these four shifts generally exist in other tasks as well?
2. This paper evaluated three outdated deep DA methods. Could you provide the latest deep DA methods for evaluation, such as [1] and [2]?
3. In Table 3, accuracy scores for deep methods are only presented for selected real-life datasets. Could you provide the scores for simulated datasets and conduct the relevant comparisons?
4. In line 217，this paper claims that the supervised scorer cannot be used in practice.  Does this imply that the scorer is meaningless in practice? In other words, how can we evaluate DA methods in practice when target labels are completely absent?

[1] Quantitatively measuring and contrastively exploring heterogeneity for domain generalization." In SIGKDD, pp. 2189-2200. 2023.
[2] Domain adaptation for time series under feature and label shifts. In ICML, pp. 12746-12774. 2023.

questions:
Please address the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693695865

thread_type:
official_review

<thread_object>
creator: reviewer_pvs6

<content>
summary:
This paper proposes DA-Bench, a comprehensive benchmark for evaluating unsupervised domain adaptation (DA) methods across diverse modalities such as computer vision, natural language processing, tabular data, and biosignals. The benchmark includes a set of simulated and real-world datasets with different types of distribution shifts, a wide range of shallow DA methods designed to handle these shifts, and an evaluation of deep DA methods. A nested cross-validation procedure with multiple unsupervised scorers is used for realistic hyperparameter selection. The results show that few shallow DA methods consistently perform well, and model selection scorers significantly influence their effectiveness. Deep DA methods, while promising, require extensive hyperparameter tuning and often fail to outperform shallow methods in low-data regim

strengths:
1. Comprehensive benchmark: The benchmark covers a wide range of DA methods and datasets with diverse modalities and distribution shifts, providing a more realistic evaluation of DA techniques.
2. Realistic model selection: The use of nested cross-validation with unsupervised scorers for hyperparameter selection addresses the challenge of lacking target labels in practical DA scenarios.

weaknesses:
1. The paper mainly focuses on empirical results and benchmarking, without providing deep theoretical insights into why certain methods perform better or worse under different domain adaptation scenarios. Incorporating more theoretical analysis could help in understanding the underlying principles of domain adaptation and guide the development of new methods.

2. While the paper includes a variety of domain adaptation methods in its benchmarks, some of the baseline methods are relatively outdated or not state-of-the-art. Including more recent and advanced baseline methods would provide a more accurate assessment of the performance of new domain adaptation techniques.

3. The paper uses accuracy as the primary evaluation metric for the benchmarks. While accuracy is a common metric for classification tasks, it may not be sufficient to fully capture the performance of domain adaptation methods in all scenarios. Considering additional evaluation metrics such as ECE, Brier Score, F1-score, or even domain-specific metrics could provide a more comprehensive evaluation.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721116294

thread_type:
official_review

<thread_object>
creator: reviewer_fk3w

<content>
summary:
This paper introduces DA-Bench, a benchmark for evaluating unsupervised domain adaptation (DA) methods across diverse data modalities with realistic validation techniques. Unlike existing benchmarks, DA-Bench uses unsupervised model selection scores and nested cross-validation to better reflect real-world settings where labeled target data is unavailable. Key contributions include:
- A comprehensive benchmark framework spanning multiple data types (e.g., computer vision, NLP, tabular, biosignals) with 20 shallow and 3 deep DA methods.
- Realistic evaluation protocols that avoid overestimating performance by selecting hyperparameters without target labels, using methods like Importance Weighted (IW) and Circular Validation (CircV) scores.
- Practical guidance and open-source code for DA model selection, offering insights on method and scorer effectiveness under different shift types.

DA-Bench represents a valuable resource for researchers and practitioners by setting a realistic standard for evaluating DA methods and by supporting the field’s advancement through rigorous, reproducible benchmarking.

strengths:
- The paper is well-organized, with clear explanations of DA methods, evaluation metrics, and data shifts. Effective use of tables and visuals enhances clarity and enables easy interpretation of results.
- DA-Bench is a benchmark that addresses a gap in UDA across multiple data modalities, such as NLP, tabular, and biosignals; multiple DA methods such as 20 shallow and 3 deep DA methods; and multiple model selection strategies. 
- The paper is open-sourced and provides a useful resource for researchers and practitioners to conduct rigorous, reproducible benchmarking.

weaknesses:
1.  Lack of Clarity in Figure 1
  - Interpretability Issues: Figure 1 is difficult to interpret, particularly in terms of what each shift type represents. For instance, it’s unclear what each color indicates; if colors represent different classes, why are there multiple clusters for each class?
  - Subspace Shift Classification: Subspace Shift is more accurately described as a methodology for handling other types of shifts—primarily Covariate or Conditional Shifts—by finding a common subspace where distributions align. It may be misleading to categorize Subspace Shift as a standalone shift type.
  - Suggestion: A clearer, more detailed illustration is needed to make Figure 1 more intuitive and to accurately represent the nature of each shift type.
2. Limited Analysis of Dataset and Shift Impact
- Shift Types in Existing Benchmarks: While the benchmark covers method performance across various datasets, it lacks an analysis of the types of shifts present in commonly used benchmarks. Understanding the specific types of shifts (e.g., Covariate, Target, Conditional) in datasets like Office-31, Office-Home, and MNIST/USPS would provide valuable insights.
- Impact on Method Performance: It would also be helpful to examine how specific shift types influence the performance of different DA methods. This analysis would allow researchers and practitioners to make more informed choices when selecting methods for datasets with specific types of shifts.
3. Limited Evaluation of Deep DA Methods
- Missing Methods: The benchmark currently omits certain popular deep domain adaptation techniques, such as pseudo-labeling methods, which are widely used and impactful. For example, pseudo-labeling approaches like MIC (Masked Image Consistency). 
4. Lack of Clarity on Scorer Variability within the Same Method
- Table 2 Confusion: In Table 2, different DA methods with varying model selection techniques are compared, which makes interpretation challenging. A clearer comparison would involve analyzing the same DA method across different model selection approaches to understand the variability introduced by model selection. This would make it easier to interpret the effectiveness of each scorer within a consistent method framework.

Overall, the paper ambitiously aims to create a comprehensive benchmark that covers various domain adaptation (DA) methods and model selection strategies across multiple data modalities—a valuable and useful contribution. However, since the majority of recent unsupervised domain adaptation (UDA) research and datasets are predominantly in the computer vision (CV) domain and deep learning-based (to the best of my knowledge), it’s challenging to derive insights from the current conclusions regarding the performance of recent advanced deep methods and classic DA techniques specifically on popular CV benchmarks (e.g., Office-31, Office-Home, DomainNet, VisDA). While the paper successfully establishes a versatile and valuable toolbox, it lacks constructive insights for practitioners due to its focus on cross-modality comparisons using only the best-performing model selection strategy for each method.

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732215458268

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
# General comment

We thank all the reviewers for thoroughly and carefully reading our paper. Their remarks helped improve its quality. We are deeply grateful to them for acknowledging the **clarity and quality** of our **comprehensive** study that **adresses a gap** in the UDA literature (Reviewers fk3w, PVS6, qzy2, rwX7), noting that we provide an **open-source, reproducible, and easily extendable** benchmark with **extensive experiments** (Reviewers fk3w, qzy2, rwX7).

We first provide some clarifications on the scope of our paper and the recent deep methods bellow, then list our revisions in the paper and provide detailed answers to each reviewer.

## Scope of the paper and recent deep methods:

Recent deep DA papers mostly focus on multi-source DA or test-time DA, which do not fit into our scope and evaluation process designed for pairwise DA. Moreover, the DA community is centered mostly around multi-domain Computer Vision problems, which made recent classical DA methods, as studied in our benchmark, less common. With our benchmark, we offer a larger view of DA, its potential use cases and its current limitations, by broadening the application spectrum. In our opinion, our benchmark is timely as it gives an overview of the field as it stablizes and highlight open research directions to ofer UDA methods that work outside CV.


With this goal in mind, we propose a reproducible benchmark over different modalities and datasets, and included pariwise DA methods that can work on all modalities. The methods suggested by the reviewers are often task-specific and cannot be extended easily to divers modalities. However, we agree with the reviewers that our deep benchmark should be extended with more recent methods. This is the reason why we added **4 recent deep methods** to the benchmark [1, 2, 3, 4]. Besides, it should be noted that our benchmark is designed to easily add new methods and new datasets, as described in appendix. Note that this benchmark will also be maintained and updated in the future and will evolve with contributions from the research community, ensuring its continued relevance and utility. We believe this flexibility and long-term vision enhance the value of our work for the broader community.


[1] Xiao et. al., SPA: A Graph Spectral Alignment Perspective for Domain Adaptation. In Neurips, 2023.

[2] Ying Jin et al., Minimum Class Confusion for Versatile Domain Adaptation. In ECCV, 2020.

[3] Zhang et al., Bridging Theory and Algorithm for Domain Adaptation. In ICML 2019

[4] Mingsheng Long et al., Learning Transferable Features with Deep Adaptation Networks. In ICML, 2015.


## Updates on the PDF in blue:
We list below the updates we made to the revised version of the paper (in blue).

- Add 4 new Deep DA methods. (fk3w, PVS6, qzy2, rwX7)
- Add discussion about the shift impact on real dataset in Section 4.1 and 4.2 (fk3w, PVS6)
- Clarify caption of Figure 1 and show the invariant subspace. (fk3w, qzy2)
- Clarified the "subspace shift". (fk3w)
- Add F1-score table in appendix D.4. (PVS6)
- Add new visualization for scorer comparison in appendix D5. (fk3w) 

We hope to have adequately answered the reviewers' concerns and we remain open to continuing this constructive discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732646086839

thread_type:
official_comment

<thread_object>
creator: area_chair_vnfm

<content>
comment:
Dear Reviewers,

The authors have responded - please see if they have addressed your concerns and engage in further discussion to clarify any remaining issues.

Thanks!
AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a benchmarking study for unsupervised domain adaptation (UDA) algorithms that evaluates methods on a range of datasets spanning multiple modalities. Specific care is taken to ensure that model selection/hyperparameter tuning is done in a realistic and unsupervised fashion in accordance to the UDA setting. An extensive evaluation is done across a range of 20 shallow and 7 deep learning based UDA methods, and the source code will be released. 

Key strengths of the study include a standardized benchmarking pipeline that will be open-sourced, especially the focus on ensuring unsupervised model selection, as well as the evaluation on datasets from multiple modalities. Key weaknesses include a lack of recent UDA methods and insight resulting from the benchmarking. 

Overall, the paper is borderline and the AC tends to agree with the sentiment expressed by fk3w in the concluding paragraph - while this is an ambitious attempt at a benchmark which is a useful contribution, it is unclear what insight it delivers for practitioners or in driving the field forward, given the lack of comparison to state-of-the-art methods for the respective domains or substantive theoretical insight. The AC does not agree with the characterization in the authors' rebuttal "General Comment" that "the DA community is centered mostly around multi-domain Computer Vision problems" - there are many domain-specific DA methods being developed in various communities (e.g. NLP and time-series). In some sense it is because of the difficulty of developing a universal DA method that many domain-specific approaches have been introduced; from this perspective, without including such methods in the comparison, the practical value of the benchmark is diminished. On the other hand, from the perspective of advancing DA methodology there is a lack of theoretical insight (e.g. analysis of shift types and impact on methods) as identified by multiple reviewers (fk3w, PVS6, rxw7). Thus, while the AC acknowledges the value of the benchmarking framework, the paper lacks compelling insights and falls short of the bar for ICLR.