Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces prompt learning to multi-source UDA. A simple two-stage framework is proposed. In the first stage, individual prompts for each source and target pair are learned. In the second stage, Multi-Prompt Alignment (MPA) is proposed to align the learned prompts. The experimental results show the effectiveness of the proposed framework.
strength_and_weaknesses: Strengths:
1.	This paper is first to apply prompt learning to multi-source UDA problem.
2.	The proposed framework is simple and achieves state-of-the-art results on multi-source UDA benchmark.
3.	The paper is well written, easy to understand.

clarity,_quality,_novelty_and_reproducibility: - What is the difference between stage one and DAPL? Stage one (learning individual prompts) seems to be similar to DAPL. 

- The effectiveness of L_CLS loss in stage 2. Table.5 (a) shows the effectiveness of objective function Equation 9. However, the results of L_CLS loss are missed. 

- What does “Zero” mean in Table.5 (b)? “Zero auto-encoders means we completely discarded the auto-encoder structure” is not clear to me. If the autoencoder structure is discarded, what structure is adopted?

- The similarity between the reconstructed prompts. In Table.4, the reconstructed prompts of different domains achieve almost the same results on the target domain. Does this mean that these reconstructed prompts are the same?
summary_of_the_review: Overall, this is a good submission with strong results. I hope the author could provide more detailed explanations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-prompt alignment method for multi-source unsupervised domain adaptation. The main idea is to first learn an individual prompt for each source-target domain pair and then mine the relationships among learned prompts through deriving a shared embedding space. The resulting embedding is expected to be domain-invariant and can be generalize to unseen domains. 
strength_and_weaknesses: - Strength
    1. The introduction of prompt learning into multi-source domain adaptation is interesting. 
    2. The performance of the proposed method on MSDA is good compared to previous methods.
- Weaknesses
    1. The idea of prompt learning for domain adaptation is not new, and the prompt design in this paper is highly based on the method proposed by Ge et al., 2022. So the main contribution is to extend the prompt learning for domain adaptation from the single source domain to multiple source domains. The novelty is limited.
    2. How to ensure the two-stage training will achieve the desired solutions? Since the individual prompts are learned to reduce domain shift between each individual source and the target domain while the multi-prompt alignment stage aims to align multiple source domains. The learning objective is changing in the two stages. Is it possible to combine the two stages into a single one to achieve the alignment of all the domains simultaneously?
    3. Why is the auto-encoder necessary? If the two stages of prompt learning and alignment strategy can be combined into a single stage with multiple objectives for alignment and consistent prediction, is it still require the auto-encoder for reconstruction?
clarity,_quality,_novelty_and_reproducibility: The paper is well written in general. However, the originality of the work is limited.
summary_of_the_review: In summary, the overall novelty is limited. Moreover, the design of the model is not very well justified, which requires further clarification.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method based on prompt learning for multi-source domain adaptation. By adopting a pre-trained CLIP based text and image encoders, the authors design prompts that are learnable to adapt them for source and target domains. Specifically, there are two stages. First, prompts are designed with class-specific and domain-specific ones, and are learned for each source and target pairs individually via a contrastive loss. Then, to further align between target prompts, the authors apply the autoencoder-based method for prompt reconstruction, with a loss function to ensure output predictions are similar from different prompts. In addition, the authors also show a finetuning scheme that can adapt to new domains with small finetuning parameters. Experiments are conducted on three benchmark settings, including ImageCLEF, Office-Home, and DomainNet.
strength_and_weaknesses: **Strength**

* The idea of using prompt learning for domain adaptation is interesting, as prompt learning has the potential for transfer learning in the new domain under a cheaper cost
* Experimental results show improvements over the CLIP baseline
* Ablation study shows some gains for the designed modules

**Weakness**

Technical motivation and novelty
* Although applying prompt learning for domain adaptation is interesting, the motivation is not very intuitive as prompt learning is based on a highly generalized CLIP model, which already suffers less in the domain adaptation setting. This may raise two questions: 1) whether the experimental comparisons are fair (see comments below), and 2) whether the alignment really happens as CLIP already performs very well in target domains (see comments below).

* The proposed techniques in this paper are not new, e.g., extending the prompt design to multi-domains is straightforward and the loss functions (e.g., contrastive loss) are also not new. More importantly, some designs are not well motivated and lacks experimental validations (discussed below).

* For multi-prompt alignment, it's not clear how the autoencoder is needed to achieve alignment between prompts. In experimental results (Table 5 (a)), performance gain is also marginal.

* It seems the performance of using prompt learning would highly depends on the CLIP model (e.g., Qdr results in Table 2). This should be discussed in the paper.

Technical clarity
* For eq(8), it's not clear which prompts are aligned, e.g., between P_i and P_j
* d_I appears many times in the paper but the definitions are not clear, e.g., eq(6), dimensions in v_tune and d_tune.

Experimental results
* Since CLIP already has a strong generalization ability, e.g., many results in Table 1 and 2 (Zero-Shot) are already better than other existing DA methods, it's not clear whether the comparisons are fair, as CLIP has been trained with millions of images. While I understand it's not easy to perform experiments using the CLIP backbone with existing methods, the proposed training scheme also cannot validate it's DA ability but only show the transfer learning ability.
* Prompts are designed with class-specific and domain-specific ones. However, there are no experiments to validate whether it is necessary.
* Lots of sensitivity experiments are not provided, e.g., length of prompts, thresholds for pseudo-labels, /alpha in eq(9)
* The explanation of using LST for experiments is not clear (paragraph above Table 3)

clarity,_quality,_novelty_and_reproducibility: - Implementations are not provided
- Some technical details are not very clear for reproducibility (see above comments)
summary_of_the_review: Overall, using prompt learning is an interesting direction for domain adaptation. However, there can be issues by using pre-trained CLIP that already has strong generalization ability for DA. In addition, some design choices in the proposed method are not well motivated nor well validated. The authors should consider the above comments and address them carefully in the rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper explores how CLIP can be used effectively for multi-source domain adaptation. In this problem setting there is a single labeled source domain dataset and multiple target domain datasets that are not labeled. Several methods have been proposed in past to address this problem, but they all have limited accuracy owing the limited abilities of ImageNet trained model (in comparison to CLIP dataset). Authors propose a simple idea to learn prompts using pseudo-labels from CLIP and propose an auto-encoder network to learn a latent space that generalizes to new domains. While not surprising, the proposed CLIP based method outperforms the existing methods. It is interesting to note that ResNet models initialized from the CLIP's image encoder do not perform as well.
strength_and_weaknesses: **Strengths**

* The paper demonstrates how prompt engineering/learning can be performed in context of multi-source domain adaptation.
* The paper shows that the the text encoder plays an important role in obtaining higher accuracy.
* The results on the challenging domain-net dataset is impressive. The proposed method achieves state-of-the-art results with fewer trainable parameters.

**Weaknesses**

* While the paper includes ablation studies on other methods with image encoder, it is not clear (unfair comparison) if this enough to benchmark the proposed methods against the previous method. 
* It is not clear how to go about deciding the hyperparams $M_1$ and $M_2$. 16 seems a little too big for the proposed method.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and is novel in context of multi-source domain adaptation. The paper contains enough details for reproduction.
summary_of_the_review: Based on the strengths and weaknesses of the paper, I vote to accept the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper addresses the multi-source UDA problems by proposing the method Multi-Prompt Alignment (MPA). MPA is built on the pre-trained CLIP which can effectively encode images and texts. Compared to other existing approaches to mutli-source UDA, MPA only needs to train a small number of parameters by prompt learning.

strength_and_weaknesses: 
++ The proposed method based on CLIP can address the multi-source UDA problems by prompt learning which is more efficient during model training.
++ The proposed method can generalise to unseen domains and the parameters to learn can be further reduced.
++ The experimental results on three benchmark datasets show competitive performance compared with others. The ablation study also validates the effectiveness of different components of the proposed approach.

-- It seems the threshold for pseudo-label selection is a very important hyper-parameter; the author should discuss how the value affects the performance.
-- The proposed method uses pre-trained CLIP as the backbone. Since CLIP is pre-trained on a different dataset from other comparative models pre-trained on ImageNet. How to justify the performance gain is not from a more powerful pre-trained model when compared with other multi-source UDA approaches? A simple baseline can be "source combine" + "simple prompt learning".
-- The authors claim the proposed approach can generalise to unseen domains by the introduced Latent Subspace Tuning (LST) strategy. Does it mean it can solve the domain generalisation problem? It seems pseudo labels are required in Eq.(10) which means the "unseen" domain is actually seen during training, please clarify this.
-- Although the proposed method aims for multi-source UDA, it seems that it can also solve single-source UDA problems. How it performs when compared with SOTA UDA approaches?

clarity,_quality,_novelty_and_reproducibility: The paper is written well; the presentation is clear and easy to understand; the novelty is a little bit weak; the experimental settings are presented and the results may be reproduced based on the details given in the manuscript.
summary_of_the_review: The proposed approach is based on the pre-trained CLIP and utilises the prompt learning strategy to learn useful prompts for mutli-source UDA. However, the main concern is that it is not justified where the performance gains mainly come from, the proposed approach MPA or the pre-trained CLIP?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces prompt learning to multi-source unsupervised domain adaptation (UDA). A two-stage framework is proposed based on the pre-trained CLIP model. In the first stage, individual prompts for each source and target pair are learned via a contrastive loss. In the second stage, Multi-Prompt Alignment (MPA) is proposed to align the learned prompts, which applies the autoencoder-based method for prompt reconstruction, with a loss function to ensure that output predictions are similar from different prompts. In addition, the authors also introduce a Latent Subspace Tuning (LST) scheme that can further decrease the number of tunable parameters for new domains. Experiments are conducted on three benchmark settings, including ImageCLEF, Office-Home, and DomainNet.

Even after the discussion phase, there exists disagreement among the five reviewers --- three positive votes, and two negative votes. AC read the paper, and considered all reviews, author responses, and the discussions. The paper has some merits, such as the interesting idea of using prompt learning in domain adaptation and state-of-the-art results on multi-source UDA benchmark. However, there are still several reservations held both by the negative reviewers and further by the AC, which prevent from direct acceptance:

1. Some reviewers still have some concerns about the technical novelty and contribution. The idea of prompt learning for domain adaptation is not new, and the prompt design in this paper in particular the prompt learning of the first stage are highly based on the method proposed by (Ge et al., 2022). So, the main contribution is to extend the prompt learning for domain adaptation from the single source domain to multiple source domains, which may be limited and less novel. Besides, the insights and motivations of the method design are not described or validated clearly enough, as concerned by the reviewers.

2. The authors claim the proposed approach can generalize to unseen domains by the introduced LST strategy. However, data and pseudo labels are still needed to further tune some parameters. It is unclear whether such prompt tuning provides good insight, compared to other clearly practical usage, e.g., test-time adaptation with prompting.

3. Some reviewers had concerns that the success of the method may depend on the performance of the pre-trained model and that using a more powerful CLIP model may be unfair to the compared methods. The authors have made some efforts to clarify this, but the reviewers are not fully convinced finally.

Given the above reservations, AC could not accept the paper for now but encourage the authors to fully revise the paper and strengthen their work.
justification_for_why_not_higher_score: Out of the five review reports, two are negative. AC read the paper and considered all reviewing material, and agreed that the current version is relatively lacking in technical novelty and evaluation rigor. In general, its flaws are somewhat outweighing its merits.
Decision: Reject