Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a method that extends weighted least squares to deep neural networks for hyperparameter selection in unsupervised domain adaptation. In theory, the authors show that the target error of the proposed algorithm is asymptotically not worse than twice the error of the unknown optimal aggregation. Numerical validate the superiority of the proposed method over the DEV and the IWV method. 
strength_and_weaknesses: Strengths:

1: This paper proposes a simple but effective method that extends weighted least squares to deep neural networks for hyperparameter selection in unsupervised domain adaptation.

2: Experiments on different tasks are conducted to verify the superiority of the proposed algorithm.

3: This paper is well-written and easy to follow. The background knowledge is presented well. The readers can easily understand the paper.


Weaknesses:

1. The baselines are few. Do the baselines include important and all the SOTA UDA methods?

2. In theory, the authors show that the target error of the proposed algorithm is asymptotically not worse than twice the error of the unknown optimal aggregation. Such a bound cannot provide guarantees of better performance of the proposed method over others.



clarity,_quality,_novelty_and_reproducibility: Novelty: This proposed method is somewhat novel.

Quality: The experimental validation is extensively evaluated for different tasks.

Clarity: This paper is well written, and the theoretical analysis is easy to follow from the outline provided.

Reproducibility: The code needed to reproduce the experimental results is provided.
summary_of_the_review: This paper proposes the first method that extends weighted least squares to deep neural networks for hyperparameter selection in unsupervised domain adaptation. The method is technically sound. 

------Update------

I have read the author's response and most of my concerns are addressed. I will keep the score for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work proposes a theoretical approach for choosing hyper-parameters in unsupervised domain adaptation. The main strategy is to compute an aggregation of models with target error bound, which theoretically relies on the extension of importance weighted least squares to linear aggregation of vector-valued functions. The paper conducts large scale comparative experiments on language, images, time-series classification tasks.The proposed method outperforms IWV and DEV and sets a new state-of-the-art performance.
strength_and_weaknesses: Strenth:
1. The proposed aggregation approch is general without strong assumptions.
2. The paper gives detailed theoretical analysis.

Weaknesses:
1. For experiments, though the paper’s benchmarks are large and diverse, but still some popular UDA benchmarks, such as VisDA-2017, DomainNet etc are missing. Besides, the paper mainly compares with conventional methods, while certain more recent works are missing.
2. The novelty of the paper is limited besides the combination of IWV with linear aggregation. 

clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow.
summary_of_the_review: The paper proposes a new aggregation models based method in unsupervised domain adaptation. Experiments on language, images, time-series classification tasks show effectiveness of the work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies how to address the issue of hyper-parameter selection in unsupervised domain adaptation. Specifically, the authors propose a method that subsequently computes a linear aggregation of the models. In addition, theories for bounding the target error are added to make the method convincing. Experiments show the effectiveness of the proposed method.
strength_and_weaknesses: Strengths:
1. The targeted problem is important and valuable for domain adaptation (without labeled data).
2. This paper is well-written, it is enjoyable to read.
3. Experiments on language, image, text and time-series data show the effectiveness of the proposed method.

Weaknesses:
1. My main concern is the theories for bounding the target error. The conclusion is the target error of the computed aggregation is asymptotically at most twice the target error of the optimal aggregation. However, this bound is loose and cannot give a guarantee of superior performance.
2. Another concern is the selection of datasets. For example, the performance gains achieved by DA methods on Amazon Reviews and WISDM are marginal (The performance of So is close to TB). 
3. The authors should conduct experiments on the datasets in the original paper of baselines. For example, experiments on digit, office and visda should be considered.
4. More insightful analyses should be provided. For example, the visualization of the importance weight of each model should be shown. It is interesting if the importance weight can tell us what is the best model in the sequence. Now the paper only tell us what the method is and if the proposed method can achieve better results without further discussions.
5. The scope of the selection model should also be discussed. The ensemble method works well given many effective models. However, if the candidate set of the model contains a bad model, the ensemble method may not work well.


clarity,_quality,_novelty_and_reproducibility: Clarity: Good. This paper is well-written.

Quality: Needs improvements. More insightful discussions, more experiments on existing benchmarks should be provided.

Novelty: Good. The proposed method is somewhat novel.
summary_of_the_review: Overall, this paper is well-written and contributions are not very marginal. However, more improvements should be added to the paper.

Strengths:
1. The targeted problem is important and valuable for domain adaptation (without labeled data).
2. This paper is well-written, it is enjoyable to read.
3. Experiments on language, image, text and time-series data show the effectiveness of the proposed method.

Weaknesses:
1. My main concern is the theories for bounding the target error. The conclusion is the target error of the computed aggregation is asymptotically at most twice the target error of the optimal aggregation. However, this bound is loose and cannot give a guarantee of superior performance.
2. Another concern is the selection of datasets. For example, the performance gains achieved by DA methods on Amazon Reviews and WISDM are marginal (The performance of So is close to TB). 
3. The authors should conduct experiments on the datasets in the original paper of baselines. For example, experiments on digit, office and visda should be considered.
4. More insightful analyses should be provided. For example, the visualization of the importance weight of each model should be shown. It is interesting if the importance weight can tell us what is the best model in the sequence. Now the paper only tell us what the method is and if the proposed method can achieve better results without further discussions.
5. The scope of the selection model should also be discussed. The ensemble method works well given many effective models. However, if the candidate set of the model contains a bad model, the ensemble method may not work well.


------Update------

I carefully read the author's response and most of my concerns are addressed. I'm happy to improve my rating.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper handles the model selection problem in the context of unsupervised domain adaptation. In classical domain adaptation literature, model selection is a hard problem because of the lack of labeled target data. In recent years, several linear aggregation methods have been proposed yet they lack theoretical guarantee on the target error. Under the covariate shift assumption, the authors propose a simple yet effective linear aggregation approach based on importance weighting and provide a target error guarantee with respect to the target error incurred by the best linear aggregation. The proposed adaptation algorithm outperforms not only heuristic aggregation methods but also existing single-model selection approaches with theoretical guarantees.
strength_and_weaknesses: ### Strengths

- **Simple method**: The proposed method is based on importance weighting, which is a well-understood yet pretty simple method. In spite of its popularity, importance weighting has not been well studied in the context of linear aggregation for unsupervised domain adaptation. Hence, it does not undermine the novelty or contribution of this paper.
- **Effective empirical performance**: The empirical performances through massive experiments demonstrate the practical effectiveness of the proposed method. It is remarkable that the proposed aggregation approach outperforms the existing methods to select a single best model.

### Weaknesses

- **Theory does not elucidate the effect of the model size.** Although the experiments demonstrate that the aggregation of multiple model candidates outperforms a single best model, it is not evident from the claim of Theorem 1. I tried to figure out from its proof but failed because the model size $\\ell$ is hidden in absolute constants of concentration bounds. Is it possible to comment on what goes on with the target error when we increase $\\ell$?
clarity,_quality,_novelty_and_reproducibility: The paper is written very clearly and easy to follow. Below I give a few suggestions though they are minor.

- In references, Sugiyamai et al. (2007) should be corrected to Sugiyama et al. (2007).
- When showing the main theoretical claim Theorem 1, I would emphasize that this error analysis is independent of the estimation error of density ratio.
- In Section 5.1, the meaning of Figure 2 is a little bit ambiguous. For example, "the approach proposed in You et al. (2019) and a heuristics baseline" (12nd line of Section 5.1) are compared in Figure 2, but I could not find which parts in Figure 2 indicate them. It is not clear which part in Figure 2 corresponds to "the given sequence of models" (14th line). In addition, I do not understand why three models (IWA, SOR, and DEV) are aggregated as shown at the bottom of Figure 2, given my understanding that IWA is already an aggregation of multiple models.
summary_of_the_review: The authors provide a transparent algorithm with a clear presentation, hence I would suggest accepting this paper. Minor issues should be easily fixed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Authors have developed a general and theoretical approach to choose good hyper-parameters in unsupervised domain adaptation. Its key idea is to compute an aggregation of models with target error bound. In addition, it performed extensive experiments on language, images, and time-series classification tasks, demonstrating it performs better than existing state-of-the-art methods. While authors can use more benchmark datasets to do more comparison, overall it has done quite a number of experiments. It is a well written paper and easy to follow.
justification_for_why_not_lower_score: A good paper with good theoretical foundation and extensive experimental evaluation.
Decision: Accept: notable-top-5%