Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This paper studies how to perform HPO for domain adaptation algorithms where the labeled validation set cannot be accessed and provides an empirical study of HPO and model selection for domain adaptation problems.
overall_review: 1. This paper provides an empirical study of HPO and model selection for domain adaptation problems and shows some useful findings for domain adaptation. But this paper does not have much potential impact on the AutoML field.
2. It seems the HPO method used in this paper does not been introduced. Is it Eq. (4)? Why not use Bayesian HPO methods?
3. A typo in Line 563.
4. Different validators in Tab. 2 have different time complexity. It is better to consider the time complexity when performing HPO for domain adaptation problems.
review_summary: I think this paper provides some useful empirical findings for domain adaptation but limited impact on the AutoML field. I lean to accept this paper but I don't object to reject it either.
actions_required_to_increase_overall_recommendation: If the authors can address my concerns in the "Overall Review" part, I am willing to raise my score.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: The paper introduces a benchmark for domain adaptation techniques across unsupervised domain adaptation, source-free domain adaptation, and test-time adaptation that focuses on ensuring that the evaluation methods are aligned with practical application by avoiding using test data labels for tuning / model selection when in practice these labels would not be available. Within this context, the paper evaluates a variety of validators and adaptation algorithms across various datasets and source-model hyperparameters to identify the combinations that perform well in practice, showing that while strong results can potentially be obtained via UDA, that results for SFDA and TTA are mixed, with high volatility and many algorithms underperforming the no-adaptation baseline. The paper additionally highlights that using a separate target validation set for selection / tuning rather than purely the target train set results in better performance in the UDA setting.
overall_review: The paper provides a useful empirical analysis of DA techniques and reports important negative results that should be taken into careful consideration when thinking about integrating DA into AutoML systems. The paper additionally highlights that using a separate target validation set for selection / tuning rather than purely the target train set results in better performance in the UDA setting.

A lack of multiple random seeds does hinder the analysis, as well as there being little analysis on any "auto" component to assist AutoML in applying DA practically. In particular, I am still confused on the train/val target set details, which I hope is clarified during the rebuttal. Overall, I am borderline leaning accept, and will consider updating my score once my key questions to the authors are addressed.
review_summary: Overall, I am borderline leaning accept, and will consider updating my score once my key questions to the authors are addressed.
actions_required_to_increase_overall_recommendation: 1. Most importantly I'd like to understand the train/val target set reasoning (Is it reasonable to assume these would exist separately from the test target set? What if they don't?). What are the downsides of splitting the target set into train/val?

2. Additional metrics or indicators of stability of the approaches would be useful when evaluating if a DA technique is worth the risk of a worst-case outcome. I listed some example ideas earlier in my review, but I am curious what the authors think would be meaningful metrics in this regard, given the apparent volatility of DA algorithm performance.

3. What, if anything, is "auto" in this work? Does AutoML have any application in this work beyond referencing the results tables and choosing a validator/algorithm pair to apply for DA after AutoML training?
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: In this paper, the authors analyze the general field of Domain Adaptation by benchmarking a suite of candidate validation criteria and then use these to compare popular adaptation algorithms for the purpose of hyper-parameter optimization and model selection. They consider different branches of domain adaption, Unsupervised Domain Adaptation (UDA), Source free domain adaption (SFDA), and Test time augmentation (TTA) because there seems to be a lack of consistency amongst the validation criteria used to compare different models. In the analysis of methods, covering 15 validators each with 10 hyper-parameter choices, they find that previously unexplored validators (they bring up RankMe and V-Measure) consistently perform well but generally, the best validator is setting/algorithm dependent.  
overall_review: Pros
+ The author choose a very broad and thorough sweep of methods within UDA, SFDA, and TTA as well as a large sweep of validators.
+ The questions both asked and the results answered in the analysis section do a good job of answering the high-level question in the paper regarding the evaluation of the UDA methods.
+ The final investigation of TTA seems very strong in particular. From the analysis surrounding the statement "we suggest that the strong success of TTA methods on synthetic benchmarks may not be representative of real-world adaptation problems", this would be a major contribution that changes how future TTA methods are developed.

Cons
- How to read the correlations is confusing. It seems to me like the correlation of even the final methods (RankMe and V-Measure) the authors propose doesn't seem to be that high. I was wondering where the authors drew the conclusions that these were good validators from.
- Not a con necessarily but something that I felt might be incomplete was the analysis of tuning sensitivity. Tuning sensitivity was mentioned in the results section 4.1.2 but then was not addressed in 4.2.1 or 4.3.1. How does tuning sensitivity compare to the TTA or SFDA regimes?
review_summary: Based on my points above, I think that this is a very solid piece of work that merits acceptance to the conference. Their effort to make a common ground for evaluation and thorough investigation of different methods and validators is definitely accepted paper quality work. I believe as the authors point out that they could potentially sample more hyper-parameter settings for each algorithm because, with the minor improvements incurred, might choose better checkpoints.
actions_required_to_increase_overall_recommendation: I am fairly confident that Accept is the right evaluation for this piece of work. I have a few questions that I listed in the cons but overall think that they would not cause me to raise my score unless the authors were able to perform an even larger hyper-parameter suite to make their results even stronger and maybe tighten up the writing to make the paper less dense to read.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper identifies the problem that many unsupervised domain adaptation (UDA) works use unrealistic validation methods especially for AutoML, so they propose methods for and perform benchmarks using only validation data that is available in each UDA scenario of increasing constraints. 
overall_review: **Positive aspects:**

* The problem is identified and motivated very clearly.

* A good breadth of datasets (including both simulated and natural domain shifts), data availability constraints, and backbones are studied.

**Negative aspects:**

* While some HPO is performed within each experiment, there is no analysis of how the training recipe (such as the number of source-only checkpoints, number of target hyperparameter sets, or the optimizer and it's fixed hyperparameters) affects the variance of results.

* Could these studies be used to automatically select the best validators (and subsequently the best algorithms) for a given UDA scenario? This could just be proposed within this paper to help improve the impact.

* The title is a bit too broad. Perhaps something like "Better Practices for Validation of Unsupervised Domain Adaptation" would be better.

* Can you further analyze the "+SO" results? Why did you decide to include the source-only model, and when does it affect the results? How realistic is this setting? Like Table 6, can you add results for the algorithms without the source-only model?

* Why are certain rows/columns mostly blank in Tables 4-6? 

* What happens outside of image classification? Are there other relevant tasks and data modalities that could be benchmarked to see if your claims extend beyond image classification? In Section 6, you mention a "preliminary study of regression adaption" but don't seem to actually include these results.

**Minor comments:**

* Within-paragraph numbering is inconsistent: sometimes numbers (1),(2),(3) are used, other times numerals (i), (ii), (iii) are used.

* In line 24, you state "a different conclusion", but some of your conclusions are similar. 

* Explicitly mention how to find your code in the main paper, in case the reader does not see it in the checklist.

review_summary: This paper is a clear and impactful study on valid validation of UDA techniques. It could be further supported by additional analysis detailed above. I think it is worth accepting, especially if the recommended actions are taken.
actions_required_to_increase_overall_recommendation: * Clarify and further analyze your tables of results
* Explicitly propose how to practically apply the findings of this paper
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: The paper examines the current state of domain adaptation using approapriate evaluation methods by testing various validation criteria and applying them to assess popular adaptation algorithms. It shows that all three branches of domain adaptation consistently face significant challenges. Although attainable performance are generally lower than expected, using appropriate validation splits and metrics is beneficial. 

overall_review: Pros:

The authors clearly identify bad practices for HPO in conjunction with DA and motivate the paper in an appropriate way accordingly.
The experimental setup covers a large spectrum for DA scenarios and considers a large variety of validators.
The outcomes of different setups show some general trend despite some inconsistencies which makes them valuable take-home messages. 

Cons:

The paper barely fits into the scope of an AutoML conference since hyperparameters are fixed in their respective setups. The promise that the introduced data split combined with an appropriate validator could help reseach on HPO under DA conditions was not sufficiently supported (although it sounds plausible) 


review_summary: In general, this is a good paper.  The fact that it is a bit off topic and has little to do with AutoML makes it potentially not suitable for this venue. I am willing to upgrade my voting if the author manage to show a stronger connection to the conference topic.
actions_required_to_increase_overall_recommendation: Given the empirical aspect of the paper, it is hard to substantially improve the quality of the paper.
review_title: null

Decision: Accept(Main Track)