Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduced a new setting of few-shot supervised mulit-source domain adaptation with a few labeled target samples and labeled multiple source domain data. Under this new setting, they proposed a progressive mix-up method for few-shot supervised multi-source domain transfer. Specifically, it creates an intermediate mix-up domain and gradually updates the mix-up ratio $\lambda$ to mitigate the domain between the target domain and the source domain. They followed MAML(Finn 2017) to train the model. Their experiments show that the proposed method can outperform the previous methods on the two benchmarks of "Office-home" and "DomainNet". 
strength_and_weaknesses: Strength:

The work introduced a new setting of domain adaptation with a few labeled target samples and labeled multiple source domain data which hadn't been proposed before. The proposed setting is useful in practice. They proposed a  simple progressive mix-up scheme that outperforms the current SOTA on the two benchmarks. The paper is basically well-written and easy to follow.  

Weaknesses:
- I have some concerns about the method. And, some places of the method are not very clear. 
   - The progressive mix-up formulation of $\lambda$ in EQ8: there is no grantee that $\lambda$ is in the range of [0 1]. According to EQ8, the final value of $\lambda_N = \lambda_{N-1}$ with $q=0$, where the q ranges from $\frac{1}{e}$. 
   - In EQ6 and 4, the method uses mixed-up multi-class regression labels. It's not clear in the paper whether they linearly combine the labels across the classes or they only mix the labels across the domains with the same class. Few-shot samples from more classes should provide a more diversified and complete target domain distribution. However, the experiments show the method doesn't get better results with more few-shot samples. What is the reason? 

- The experiments are not very strong with a few concerns.
   - The paper only provided their results on two benchmarks. Usually, more results are required.
   - The few-shot results with all the class samples are required. It only shows the results with the samples from 10~25 classes.  It is not clear whether the method can deal with a big class number (i.e. will the performance deteriorate with a big class number ?)
   - More ablation results and discussions about the meta-training process are required, e.g. the split selection of 60%source domains only for meta- train + 40% all domains for meta-test.  
       

clarity,_quality,_novelty_and_reproducibility: The work introduced a new setting of FSMDA which hadn't been proposed before.  Under this new setting, they proposed a simple progressive mix-up method that outperforms the SOTA on two benchmarks, which has enough novelty. 
summary_of_the_review: In summary, they proposed a progressive mix-up method with a new setting of FSMDA. However, there are some concerns about the proposed method and the experimental results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper targets a research problem of how to transfer knowledge from multiple source domains to a single target domain with very limited target data. To solve the problem, a new method based on mix-up is proposed. It progressively pushes both the source domains and the few-shot target domain aligned to the mix-up domain.
strength_and_weaknesses: Strength:

+ The research problem is important and may have many practical applications. In real-world machine learning applications, it is possible that the data from the target domain is limited. Improving the generalization of the pre-trained model on the source domain could be important and is a practical setting.

+ Extensive experiments are conducted.  The proposed method demonstrates strong performance and surpasses baselines by a large margin on two benchmark datasets. 

+ This paper generally is well-written and easy to follow.

Weakness:

+ The technical insight may not be enough, the proposed method is mainly based on mix-up. Most of the motivation for using techniques is based on intuitive explanations. Transferring the knowledge to the target domain with very limited target data seems to be an ambitious task.  For me, it is not clear whether this method works in general or not. I think it is better to add some insightful analysis. For example, are there any assumptions to make the method useful? When will the proposed method fail to transfer the knowledge?
+ The related work under the proposed problem settings may not be fruitful. I am wondering that is it possible to compare one or two more baselines with minor tweaks in the related problem settings, (e.g., few-shot learning). 

clarity,_quality,_novelty_and_reproducibility: This paper leverages mix-up for domain transfer, which can be viewed as an application extended from previous techniques. The written quality overall is high.
summary_of_the_review: This paper focuses on a practical and challenging domain transfer problem. Using mix-up for reducing transferring knowledge from the source domain to the target domain is an interesting method. My largest concern is that the technical insight may not that strong.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on multi-source domain adaptation with limited target data.  Since in this setting there are few target data, current methods fail to learn efficient model to model it. In this paper, authors provide a progressive  mix-up method to learn it. The experiment results support the effectiveness of the proposed method. 
strength_and_weaknesses: Pons:
This paper explores new setting which follows practical  application.

The paper gives clear description about this setting which is convincing. 

This article is well-written and easy to follow.

The experiment results are sufficient  to support the proposed method.

I have a few concerns which are as following: 

1) The proposed method is not vary suitable for multi-source domain adaptation. Why the mix-up technique works well for few-shot learning.  Since the number target dataset is small, the goal should locate the local information which is good for target domain. I just feel weird about the mix-up technique for few-shot learning.

2) Reproducibility is hard, since it refers more detail precessing about the proposed method. Also the code is not provided, thus I am not sure about reproducibility.

3) More datasets should be considered, since two datasets seem to be less. 

4) It still works well if the network is transformer, do authors try it?


clarity,_quality,_novelty_and_reproducibility: see the above section.
summary_of_the_review: I am not sure why the motivation why the mix-up is suitable for few-shot. Also the reproducibility should be challenge.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper propose a new setting named Few-shot Supervised Multi-source Domain Transfer (FSMDT),  which try to generalize from multiple source domains to a single target domain with few labeled samples. And  authors propose a novel method based on the mix-up trick to create an intermediate mix-up domain.  The experimental results show that the proposed framework outperforms the baselines in two datasets.

strength_and_weaknesses: **Strengths :**

1. The paper is the easy to read in most cases.
2. The proposed method seems to make sense and it is easy to reproduce.
3. The experimental results look good and outperforms the baselines  with large margin.

**Weakness or question**
1. Sec 3.3 why choose the Wasserstein distance as the measurement ?  As far as I know, the the Wasserstein distance is hard to compute and how to compute it in your detail implement?

2. To the best of my knowledge, the proposed Few-shot Supervised Multi-source Domain Transfer (FSMDT) is similarity with the multi-domain few-shot learning \[1]\[2][3]   ,  I am very interested that the methods  \[1]\[2][3] designed for the multi-domain few-shot learning will work well in setting of FSMDT or not. 

3. The performance improvement is significant. But could you design some more useful baselines? From my view, you used many weak baselines. So it is not a fair comparison.



[1] Liu L, Hamilton W L, Long G, et al. A Universal Representation Transformer Layer for Few-Shot Image Classification[C]//International Conference on Learning Representations. 2021.

[2] Dvornik N, Schmid C, Mairal J. Selecting relevant features from a multi-domain representation for few-shot classification[C]//European Conference on Computer Vision. Springer, Cham, 2020: 769-786.

[3] Triantafillou E, Larochelle H, Zemel R, et al. Learning a universal template for few-shot dataset generalization[C]//International Conference on Machine Learning. PMLR, 2021: 10424-10433.

clarity,_quality,_novelty_and_reproducibility: 1. The paper is the easy to read in most cases.
2. The proposed method seems to make sense and it is easy to reproduce.
3. The experimental results look good and outperforms the baselines  with large margin.

summary_of_the_review: In this paper, the authors propose a new setting and a new method.  But I think there are still many details are unclear, or something importance reference and baselines is missing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents work on multi-source domain adaptation with limited target data. Considering that there are a few target data, prior works fail to learn effective models for domain adaptation. To handle the problem, a mixup-based method is proposed, where a mix-up domain is produced. Experiments on benchmarks confirm the effectiveness of the proposed method. 

The strengths of this paper include (a) The introduced problem setting of domain adaptation is practical and has potential. In real-world scenes, the data of the target domain may be limited. (b) The experiments in this paper look great. The proposed method outperforms the baselines by a large margin. (c) The writing of this paper is overall great. The weaknesses of this paper mainly lie in the technical insight may be insufficient. The proposed method highly relies on the Mixup method. Besides, the rationality of exploited techniques is based on intuitive explanations. 

Before rebuttal, four reviewers post several concerns on this paper, including the discussions about related work, the characterizations of the proposed method, and questions about partial experiments. The authors provide detailed responses and actively communicate with reviewers. Reviewers are satisfied with their responses and the revised version of this paper. They hence tend to accept it. AC checks all activities in the review process and this paper, and agrees with the reviewers. Therefore, the recommendation is "accept". 

justification_for_why_not_higher_score: What didn't get this work a higher score was insufficient technical insights as mentioned in the meta review. 
justification_for_why_not_lower_score: This paper provides a new problem setting of domain adaption. Although the proposed method is mainly based upon the prior work (i.e., Mixup), the method works well under this new setting, which clearly outperforms baselines. As this paper can contribute to the research community and inspire follow-up research, AC chooses "Accept with poster".
Decision: Accept: poster