Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an efficient model translation network (MTN) for text-image generation (T2I) by leveraging off-the-shelf pretrained language models with a pre-trained T2I diffusion model, such as T5 and XLM-R. The experiments demonstrate the superiority of this method.
strength_and_weaknesses: Strength:

1.Good performance.

2.This paper is well written and organized. Motivation is simple and straightforward.

3.As mentioned in the paper, this paper is able to train a model of T2I in an economical way without retraining whole model, and it can even be a multilingual as input as prompt by using XLM-R model.

Weaknesses:
Lack of ablation of text encoder. is it possible to try different text encoder, such as bert, roberta, clip's text encoder, etc.

clarity,_quality,_novelty_and_reproducibility: Lack of novelty, and please revise other mistakes carefully.
summary_of_the_review: I prefer to accept this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed to address the problem of replacing a text encoder in a text-to-image model without re-training the whole pipeline from scratch. In doing so, it is proposed that MTN (Model Translation Network) projects the output of the new text encoder to the old one's output space, enabling the replacement. Experiments show both qualitatively and quantitatively the proposed baseline is capable of doing a replacement.

strength_and_weaknesses: # Strength

The problem this paper addresses is a novel and very interesting one per se, and is well-motivated. If fully fulfilled, this would lead to a large number of applications.

The authors make a reasonable effort to show the performance of this method. The effort in qualitative experiments is great in that lots of examples are shown for comparison. The method would be welcomed by the community.

# Weakness

With the strength said, I'm not fully convinced of the proposed method, as detailed below:

Proposed algorithm for alignment: The proposed method projects the new text-encoder's output space to the old encoder's space, thus can serving as a wrapper over new text encoder. However, several issues are presenting:

1. The training loss, which consists of three losses (eq 3,4,5), are complex and the dynamic is unclear. No ablation study is presented to provide the justification of such a design.
2. It is unclear if the architecture of MTN could handle fixed length representation or the variant length ones.
3. The technical contribution of the proposed MTN is unclear. Related works in bridging the latent space of different language models are not discussed. Just to name a few, in machine translation:
    - Kulshreshtha et al. Cross-lingual Alignment Methods for Multilingual BERT: A Comparative Study
    - Cao et al. Multilingual Alignment of Contextual Word Representations
4. The performance contribution, or how good the MTN works, is unclear. MSE loss seems to be okay as shown in the experiment, however there is not a grounding of exactly how well MTN works.  We have no idea if the qualitative results come from the MTN being just okay but the generative model for images is powerful enough to compensate for MTN. Ideally, a good ablation is to have two runs of the same language models with different initial parameters and using the proposed technique to replace one with another.
5. One argument for this work is that a new language model's ability could be leveraged. However if it's demonstrated through an language model in another language, a justification should be provided through comparing it with simply translating the prompt.

# Questions for Authors

1. How much computation is required for the proposed MTN?

clarity,_quality,_novelty_and_reproducibility: The paper is a clear one. However it suffers issues in quality and novelty as mentioned above.

summary_of_the_review: While this paper addresses an interesting problem and is well-motivated, the issues in the proposed method and the experiment's design makes it hard to be presented in the conference in its current form. I would thus suggest a rejection of this manuscript based on the current understanding, but I'm happy to change my rating if my concerns and questions are addressed in the discussion.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper aims to efficiently replace the text encoder of the text-to-image generation model to improve the generation quality (or to make a multilingual text-to-image generative model). To this end, the authors proposed Model Translation Network (MTN) in which training loss is motivated by domain adaptation and cross-domain alignment. The proposed method shows that the performance gain is consistent over the recent latent diffusion model (LDM) and can make LDM a multilingual text-to-image generative model. 
strength_and_weaknesses: **Strength**

(1) The research direction itself is quite interesting, i.e., replacing the text encoder for a better (or multilingual) generative model.

(2) The presentation and organization are clear.

(3) The author provides comprehensive generated results (in the Supplementary).

----
**Weakness**

(1) Ablation study is missing. Only the quantitative results are given in Appendix (A.3). To claim the effectiveness of each component, the qualitative results should be given.

(2) The choice of using domain adaptation is somewhat unclear. Since authors have to run an additional discriminator, it may increase the computation, which is slightly against the motivation. Does this bring a significant improvement?

(3) The performance gain is somewhat marginal (in Table 1). Since the LDM is not properly reproduced, the gain seems to be about -0.5 FID or -1.5 FID, which is slightly marginal. Can the author try a bigger language model to achieve more gain? E.g., providing encoder size and FID results like in Figure 4 of [1].

(4) Comparison with a naive approach for a multilingual generation. One can simply use a state-of-the-art machine translation model (e.g., CONT [2]) on top of LDM (or SDM) and fine-tune it. Does the proposed approach outperform such a naive approach?

(5) (minor) Some analysis of the hyper-parameter will be great (e.g., hyper-parameter sensitivity)

[1] Saharia et al., Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding, NeurIPS 2022\
[2] An et al., CONT: Contrastive Neural Text Generation, NeurIPS 2022
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The writing and the presentation are clear. If the author can provide more clear mathematical notation, in Section 4.1, it will be much better, e.g., some problem setups in a mathematical formula.

**Quality**

The overall quality is fine.

**Novelty**

The method itself is a combination of existing works, but the research direction is novel.

**Reproducibility**

Since the authors did not provide the code and the hyper-parameter, I believe it is hard to be reproduced.
summary_of_the_review: I recommend weak acceptance. I believe the target problem is quite interesting and has the strength to be accepted. However, some concerns (mentioned in the weakness part) make me choose the weak acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method that allows replacing the text encoder in a text-to-image model without retraining the entire model. It uses a Model Translation Network (MTN) that is trained with text corpus data (or parallel text data for multilingual text encoders) to align the representation spaces of two text encoders. The paper demonstrates two application use cases: to update the text encoder of an existing T2I model to a more powerful one, and to update the text encoder of a monolingual T2I model to a multilingual one.
strength_and_weaknesses: My intuition divides the paper’s contribution into three major components.

1. An algorithm to translate between any two text encoder systems. The method is relatively simple and affordable, and uses three types of losses (adversarial loss, alignment loss and reconstruction loss) which are all very intuitive. The method also seems generic enough and I can imagine future applications to other domains. The writing is also very clear.

2. Applying the MTN to upgrade the BERT encoder of the LDM to the T5-3B encoder. This is an example of upgrading the monolingual text encoder of a T2I model. The result is a bit disappointing in this use case because
  - It was not until this part of the paper that I realized that end-to-end finetuning of the T2I is still needed in order to recover or improve its performance. Without the finetuning step, performance actually drops. Therefore it is not strictly “plug and play” of the new text encoder and the MTN. This is not mentioned in the paper until the experiments section, and therefore comes as a surprise and disappointment.
  - Because of the need for finetuning, it becomes a bit more complicated to argue the benefit of using the MTN. Firstly, in terms of performance, the paper compares baseline LDM, T5+FT, T5+MTN, and T5+MTN+FT, and argues that T5+MTN+FT performs better than both the baseline and T5+FT. It would be informative to also compare with LDM+T5 trained from scratch, and would be very convincing if T5+MTN+FT gets very close to that. Secondly, in terms of computational cost, it would be useful to point out the cost of training LDM from scratch, and how much cost is saved by using T5+MTN+FT vs training from scratch. (If I understand correctly, one of LDM’s advantages is training affordability, which means MTN may not be saving much. It would be nice to show this is not the case.)

3. Applying the MTN to upgrade the CLIP text encoder of SDM to the multilingual XLM-Roberta-L. This is a successful application. The hybrid text inputs are interesting as well, but not that surprising. I think it is more contributed by the capability of XLM-Roberta-L than the MTN, since XLM-Roberta-L aligns representations across languages very well, and therefore is able to handle hybrid text encoding as well. One disadvantage is that it requires parallel corpus to train the MTN, whereas retraining SDM with XLM-Roberta-L does not require extra data and should also automatically work with multilingual prompts. 


**Questions:**
- In the user study using Amazon Turk (Fig 4), were images from the three models displayed side by side, and the Turk picks the best out of three? If that is the case, I am not sure the results (31.7%, 32.7%, 35.7%) significantly deviate from chance statistically.
- What is the dataset used for finetuning T5+FT and T5+MTN+FT?
- Is the text encoder frozen during finetuning T5+FT and T5+MTN+FT?
- What’s the performance for LDM retrained with T5? How does it compare with T5+MTN+FT?
- What's the compute comparison between LDM retrained with T5 and T5+MTN+FT?

clarity,_quality,_novelty_and_reproducibility: The paper is written clearly. The proposal of switching the text encoder component of T2I without retraining is interesting and as far as I am aware, novel. The algorithm to translate between any two text encoder systems is also novel as far as I am aware.
summary_of_the_review: Overall I like the idea proposed in the paper. I think it is important to think about how to reuse trained models and components, as in many cases it is unaffordable or wasteful to retrain end-to-end models. The proposed algorithm to align two text encoders’ representations is simple and clear. The main drawback for me is that I am not convinced of its benefit in upgrading the text encoder in a T2I model, as shown by the results in Section 5.2. We may need better results, or better-organized results, to justify the claim that it “boosts the performance of existing T2I models”. Therefore I think it is marginally below the acceptance threshold. I am willing to change my score if either the claims or the results are adjusted to align better with each other.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a model translation network (MTN) to address the problem of replacing a text encoder in a text-to-image model without re-training the whole pipeline from scratch. Initially, the paper received scores of 3566. After author rebuttal, the scores have increased to 6666. The AC also held a virtual discussion meeting, where 2 of the reviewers have joined the discussion.  

On one hand, this paper is well written, motivation is simple and straightforward, and the authors have made a reasonable effort to show the performance of this method. During rebuttal, the authors provided additional results on comparison to a translation model baseline, analyzed the computational cost, and added additional ablation study. 

On the other hand, though all the reviewers gave scores of 6, during the virtual discussion meeting, several concerns still remain. (1) The novelty of the proposed method MTN is limited. It mostly uses standard losses from the domain adaptation literature. (2) It remains unclear how good the MTN works and how much benefit this method brings. For example, a) the image decoder still needs to be fully finetuned in order to have optimal performance. This makes the whole training pipeline complicated, for a somewhat small improvement in FID. b) The human evaluation results in Figure 4 are not convincing enough. It looks like all the 3 methods being tested have roughly the same level of quality, as the percentage difference is minor (though the authors claimed that it can be indeed statistically significant, but may be not significant enough). c) The interesting results of updating a CLIP text encoder of SDM to the multilingual XLM-RoBERTa-L are more contributed to the use of XLM-Roberta-L than the MTN. 

On balance, this paper is a borderline case. After virtual discussion meeting,  the AC decides to recommend rejection by the end, and encourages the authors to incorporate all the comments to submit to a future conference. 
justification_for_why_not_higher_score: The reviewers appreciated the reviewers for adding additional results to answer the reviewers' concerns. However, several concerns still remain as summarized in the meta review. Thus, on balance, the AC recommends rejection of the paper.
Decision: Reject