Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper raises an important question about reproducibility of results for partial domain adaptation. The authors show that the stopping criterion is key to the success of partial domain adaptation algorithms in the literature, and there can be massive variations for various stopping criterions commonly used in the literature. Some algorithms even produce wide variations in performance for different random seeds. 
strength_and_weaknesses: Strength
- The paper have done considerable amount of experiments along various dimensions
- The paper is well-written and easy to follow

Weakness
- While the paper does a lot of experiments, the paper does not establish a strong baseline. The main takeaway from the paper is that partial domain adaptation works in literature produce high variations in performance to stopping criterions and random seeds. 
The expected takeaway from such papers should be rather a simple strong baseline which can be concluded from the experiments under a certain fair experimental protocol, which works well across stopping criterions and random seeds. This would then encourage researchers to follow that experimental protocol and report their results and compare against that baseline. 

The paper is overall in a good direction and I highly encourage the authors to consider this to refine their work. 
clarity,_quality,_novelty_and_reproducibility: - In Table 1 it is not clear how the authors got the "best" and "worst" performances. Please clarify.

- How are the seeds chosen? Randomly? 

- What effect can the optimizer and learning rate strategy have on the algorithms? I suppose they would also have some dependency on it.

- 
summary_of_the_review: The paper raises an interesting question, presents various experiments to showcase the problems with the current methods. However, a simple strong baseline and a suggestion for a robust experimental protocol is missing, which should be the take-away from such works.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a benchmark for partial domain adaptation (PDA) to evaluate the model selection strategies for different PDA approaches on two different real-world datasets. The experimental results show that 1) Target labels are critical for model selection strategies; 2) only one method and model selection pair perform well on both datasets; 3) The results of PDA methods are influenced by the rand seeds. The authors also give some recommendations for future works.
strength_and_weaknesses: Strength:
1.	This paper studies an important domain adaptation problem, particularly for partial domain adaptation. Using the target label to select models violates the domain adaptation assumption. But there is no suitable and effective metric for model selection. This is why researchers still use target labels for model selection.
2.	This paper evaluates 7 representative PDA algorithms on 2 different real-world datasets using 7 different model selection strategies. This benchmark will foster the research community.
3.	Based on the extensive experiments, the authors give some conclusions and suggestions for future works.

Weakness:
1.	This work raises an essential issue in partial domain adaptation and evaluates many PDA algorithms and model selection strategies. However, it does not present any solution to this problem.
2.	The findings of the experiments are a bit trivial. No target label for model selection strategies will hurt the performance, and the random seed would influence the performance. They are commonsense in domain adaptation, even in deep learning.
3.	The writing needs to improve. The Tables are referenced but always placed on different pages. For example, Table 2 is referred in page 4 but placed on page 3, making it hard to read. The paper also has many typos, e.g., ‘that’ instead of ‘than’ in section 3.
4.	Many abbreviations lack definition and cause confusion. ‘AR’ in Table 5 stands for domain adaptation tasks and algorithms.
5.	In section 4.2, heuristic strategies for Hyper-parameter turning is not clearly described. And the author said, “we only consider the model at the end of training”, but should we use the model selection strategies?
6.	In section 5.2, part of Model Selection Strategies, the authors give a conclusion that seems to be wrong “only the JUMBOT and SND pair performed reasonably well with respect to the JUMBOT and ORACLE pair on both datasets.” In Table 6, the JUMBOT and SND pair performs worse than the JUMBOT and ORACLE pair by a large margin. For instance, on OFFICE-HOME, the JUMBOT and SND pair reaches 72.29 accuracies, while the JUMBOT and ORACLE pair achieves 77.15 accuracies.
clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and novelty are only marginally significant. For reproducibility, the authors provide the key resources to reproduce the results.
summary_of_the_review: The authors study an important issue in partial domain adaptation and conduct extensive experiments to benchmark different PDA methods and model selection strategies. Based on these experimental results, this paper shows some findings. However, this paper does not present solutions for this issue, and the findings are also obvious.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Existing algorithms for the problem of PDA(Partial Domain Adaptation) are considered and evaluated. While most previous successes have relied on target labels during training(model selection strategy), the availability of target domain labels is unrealistic. 

With a more realistic evaluation of the PDA methods, the authors claim that performance drops are significant without target labels. The finding urges the community to rethink the evaluation protocol for practical PDA methods. 
strength_and_weaknesses: As the study of PDA methods is relatively new to the community, the benchmark and evaluation protocol for the field still needs a careful study/investigation. The submitted paper has dealt with the problem of interest comprehensively and stated the PDA model selection checkpoints. This is an interesting and significant technical challenge that matters to the community. 

The benchmark and relevant code are open-sourced for reproducibility. 

While the paper is well-written overall, some sentences are difficult to follow. Some examples are as follows: 
p.2 protocal -> protocol, methods pairs -> method pairs, 100 random samples is -> 100 random samples are 
p.3 that distinguish -> that distinguishes, a high confident model -> a highly confident model
p.4  transform ... in a vanilla -> transform ... into a vanilla, found than target -> found that target.



clarity,_quality,_novelty_and_reproducibility: While the tackled problem of interest, a realistic evaluation of PDA methods, is original, the novelty of the work overall is still in question. The assessment and the reproduction of the existing methods are relatively straightforward, and the most significant claim that the absence of target labels can hurt the performance of the finally chosen PDA model is not surprising. 

The paper is challenging to follow and understand. More detailed analyses, explanations, and visualizations of the major claim and the pitfall of the existing algorithms would help the readers understand more clearly. 

The code is open-sourced for reproducibility. 

summary_of_the_review: While the paper is somewhat interesting, it does not suggest any interesting algorithm or an exciting finding. Most experiments are straightforward and there are not many insights the readers can learn from. 
The paper is not clear and is difficult to comprehend. 
For the aforementioned reasons(novelty, technical contribution, and clarity), I recommend rejecting it.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Authors have designed a benchmark comparison for partial domain adaptation to evaluate the model selection strategies across two different real-world datasets. It also conducted comprehensive experiments to draw some conclusions.

While this is an important research area, it does not provide any solution to this problem. It is also recommended to use more datasets for more solid evaluation. In addition, the insights obtained is also not so significant. Authors also need to put more effort in the writing as it has typos and sometimes hard to follow.


justification_for_why_not_higher_score: While this is an important research area, it does not provide any solution to this problem. It is also recommended to use more datasets for more solid evaluation. In addition, the insights obtained is also not so significant. Authors also need to put more effort in the writing as it has typos and sometimes hard to follow.
Decision: Reject