Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper investigates domain adaptation for an autoregressive language model.
A latent topic distribution is calculated based on the input words, which is then used to modify the token probabilities during generation.
Evaluation is performed on two review datasets (Amazon and Yelp).
strength_and_weaknesses: Strengths:
* The of estimating the topic of the text first, then generating based on that is interesting and possibly promising.
* The empirical results are good compared to the chosen baselines.

Weaknesses:
* The clarify of the paper is low, prohibiting understanding of how the method actually works.
* Experiments are performed only on two datasets, both from the same domain. For convincing results regarding the domain adaptation method, more than one domain should be investigated during evaluation.
clarity,_quality,_novelty_and_reproducibility: 
The clarity of the paper is low. The mathematical notation seems to be used in an inconsistent way, leading to impossible equations. There are also many grammatical errors, making understanding even more difficult.
For example, Section 3.5 specifies H_L as a vector of vectors (a matrix) of variable length (|x|), yet its length is somehow given as d_h, a simple 1-dimensional vector.
Eq 4 seems to specify that the probability distribution over topics is multiplied together with a probability distribution over words, which should not be possible unless the topics somehow match the vocabulary.

The task in this paper is domain adaptation for generative (auto-regressive) language models. This is a very well researched are with numerous papers over many years. Yet only a couple of very recent papers are cited and compared against. In addition, some of the chosen baselines are not really designed for this task.
Neither prefix-tuning nor DAPT+TAPT were designed or evaluated for language modelling itself, but they are methods for improving downstream performance on a supervised learning task (e.g. text classification).

In Section 3.4 it is mentioned that a bidirectional mask is used, implying that the model is allowed to look into the future. This should not be allowed in a language modelling task.
Section 4.1 is also specifying that the topic vector is averaged over the whole text. It is not clear if this is also done during testing, which would mean that the model gets to see the inputs it needs to generate, again something that would not be allowed.

The first sentence of the abstract claims that the proposed method is making it possible to use language models for unconditional generation. That is what language models already do, without any adaptation. They are conditioned only on the previous context, but so is the proposed model, so the claim is confusing.

Please look into using citet vs citep. The current use of citation formatting is making it difficult to understand the sentence structure.

summary_of_the_review: Possibly interesting and useful ideas.
The clarity of the paper needs improvement and the evaluation should be against more relevant baselines and on more datasets.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes to further fine-tune an unconditional language model on a domain-specific corpus via modeling the topics as a discrete latent variable. 
strength_and_weaknesses: Strengths:
1. The general idea of modeling topics as latent variables for domain adaptive pre-training does make sense. 
2. The results look good.

Weaknesses:
1. The format of this paper has been totally changed. I think it is not allowed to change the template. The font looks weird. And those table layouts look very condense and hard to read.
2. The notations of equations look very confusing. And the  descriptions of methods are also hard to follow. I will pose all my questions on the method section in the following section.
3. Since I cannot fully follow the method, I have concerns on the effectiveness of the method.
4. In the results section, those baselines have not been well described so it is impossible for me to judge whether the baselines are enough. Even the metric used for evaluating the topic coherence is not described in enough details so I do not know whether this metric is suitable for the task or not.
clarity,_quality,_novelty_and_reproducibility: Here are my questions on the method section:
1. In equation 2, it seems t represents the index of tokens, which aligns with the normal notation for token position. However, in equation 5, it seems that t represent the index of topics. Why do you use the same symbol for two concepts?
2. In equation 5, you have three kinds of F function for z>0, then which one do you use finally? 
3. Why do you need to describe the attention mechanism in section 3.4 in such details? 
4. In section 4.1, what do you mean by "while the other text representation, TID is taken as a negative embedding"?
5. In equation 5, TID is the average of h, which are the representation vectors after L layers of transformer blocks, while z is the latent variable of topics. Then why do we want to minimize the distance between these two vectors?

Here are my questions on the results section:
1. In Table 1, the number of Z is set to 100 and 200 for the two datasets. How do you decide it?
2. What are the details of those baselines used in section 5.2 and 5.3? Could you elaborate them instead of just giving references or links?
3. What is the topic coherence measure?
4. In table 3, whar is |z|?

There are too many abbreviations that look very similar in the paper, which makes it very hard for readers to understand and follow. 
summary_of_the_review: The method proposed may be good but the bad writing makes it hard for readers to discover its significance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new topic-aware transformer that alleviates the gap between the pretrained language model and the target domain. The paper introduces an additional topic latent variable $z$ as an additional condition when generating new tokens. The new topic latent variables come from the topic steering layer, which maps hidden representations into the topic vector. The model is optimized with both topic distribution modeling and maximum likelihood. The backbone of the paper is GPT2. The model is tested on two datasets: Amazon and Yelp. The paper checks the topic coherence and its text-generation ability  The paper did both automatic and human evaluation. The paper also did additional ablation analysis and case study. 
strength_and_weaknesses: Strength:
1. The paper proposes a new topic-aware transformer to resolve the domain shift for unconditional text generation. The motivation of the paper is clearly stated. The TAT achieves the best performance for topic coherence and text generation. The whole idea is interesting.

2. The paper conduct both automatic and human evaluation. The paper also did some ablation analysis to show the contribution of each component. The paper also presents some generation results as examples.


Weaknesses:
1. Some parts of the paper are not clearly written which will be discussed in the clarity section.

2. Some topic-related papers are not covered in the related work such as Zhu et al., (2021), and Chang et al., 2021.

3. The citation style is not correct. Authors need to use citep{} The paper uses large space to squeeze tables and captions, making readers hard to read. Authors should not abuse space.

4. The evaluation metrics used in the paper are a little bit old and only concentrate on the token overlaps. The paper needs to include some newer metrics such as BERTscore (Zhang et al., 2019), and BARTScore (Yuan et al., 2021) which can check semantic similarity. 


Zhu, L., Pergola, G., Gui, L., Zhou, D., & He, Y. (2021). Topic-driven and knowledge-aware transformer for dialogue emotion detection. arXiv preprint arXiv:2106.01071.

Chang, H. S., Yuan, J., Iyyer, M., & McCallum, A. (2021). Changing the mind of transformers for topically-controllable language generation. arXiv preprint arXiv:2103.15335.

Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Yuan, W., Neubig, G., & Liu, P. (2021). Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34, 27263-27277.
clarity,_quality,_novelty_and_reproducibility: 1. Some parts of the equation are unclear. For example, in equation 5, where does $z$ come from? Although readers can later learn that z actually comes from $LayerNorm(h_{L,t})W_Z$, it would be better to split equation 4 into two formulas. Authors also need to rewrite equation 5, because the definition of $\mathcal{F}$ is unclear. Which transformation does the paper finally used in the model? The human evaluation details are not clear. For example, the number of annotators in the human evaluation is unknown. The annotation guidelines should be attached to the appendix.

2. The effectiveness of the TAT is verified both in the topic coherence and text generation.

3. The paper provides some implementation details. However, the paper doesn't provide any code or dataset for reproduction.
summary_of_the_review: Overall, the proposed new model: the topic-aware transformer is quite interesting and follow s motivation. However, some parts of the paper are not very clear. The experiment section and paper format need to be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a topic aware transformer (TAT) to adapt pretrained language models to target domains through topic modeling. The motivation is, to alleviate forgetting, TAT explicitly quantify the domain shift as topic shifts. By introducing a topic steering layer (TSL) on top of transformers, TAT decompose the task of predicting next tokens into two tasks: (i) Firstly, predict the distributions of topics given previous tokens. (ii) Secondly, shift the word distribution according to predicted topics and then predict the next token. Combining the TSL with a newly introduced topic distribution modeling (TDM) loss, which aims at aligning the topic prediction, the proposed TAT achieves better performance of unconditional generation on two datasets, comparing to DAPT+TAPT and prefix tuning.
strength_and_weaknesses: Strength:
1. This proposed idea is novel, interesting and well-motivated.
2. The resulted models can bring us more interpretability of the process of language modeling, by predicting topics before predicting next tokens.
3. The evaluation on Yelp and Amazon datasets are comprehensive.

Weakness:
1. It seems that the proposed model can and should go beyond unconditional generation tasks. After introducing an extra TSL layer and learning meaningful topics, it seems unconditional generation tasks cannot fully leverage the potential of learned models. If we treat the learned topic distributions as learned interpretable latent variables, it should be easy and interesting to inject more topic control into the generation process.
2. While "alleviating catastrophic forgetting" is one of the authors' motivations, discussion and comparison of tuning-free adaption is missing. For example, Plug and Play Language Models: A Simple Approach to Controlled Text Generation (https://arxiv.org/pdf/1912.02164.pdf), and FUDGE: Controlled Text Generation With Future Discriminators (https://aclanthology.org/2021.naacl-main.276.pdf). By such tuning-free controlled generation approach, we can also easily adapt PLM to target domains by changing the predicted word distributions.
3. The presentation can be improved, such as the form of citation makes section 2 really hard to follow. The left part and the right part of figure 1 are also hard to understand.
clarity,_quality,_novelty_and_reproducibility: This paper proposes a novel idea, with detailed introduction and good evaluation.
The selection of datasets and the future code release make it easy to reproduce.
There are some points can be further clarified:
1. What is the catastrophic forgetting while adapting PLMs? The authors cite a lot of prior work without giving clear enough explanations.
2. How to define "global word distributions" and "target-specific word distributions" in Sec 3.1? As mentioned in sec 3.2, there might not be very clean definitions, but showing examples can be better.
3. Figure 1 left: what is "large" and "small" refer to?
4. "The topic vector is the average of zt over each i-th text with Eq (4)" in sec 4.1. What does "each i-th text" mean? Each token or each input text? Writing an equation can help with such ambiguous wording.
5. What is the motivation for the TDM loss? The authors say "align topics with each text", but what is the meaning of aligning the final predicted topics (TID) with previously predicted topics, since TID itself is not related to token generation.
6. What is the dynamic of topic prediction during inference? Are there any shifting of topics? More discussion on this can be interesting.

summary_of_the_review: I think this paper proposed an interesting and novel idea, with good experiments.
My biggest concern is the missing of comparisons to previous work like PPLM and FUDGE, which makes it harder to judge the real contribution of this work.
I also suggest there should be more tasks to play with the proposed model.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to adapt pre-trained language models to support unconditional text generation tasks. Reviewers all agreed that this idea of predicting topics first and then generating text is interesting and novel, which can bring good insights and interpretability into the process of language modeling. On the other side, the key concerns in the original reviews after the discussion remain, towards the clarity of this paper, the limited focus of using two datasets from the same domain for evaluation of this domain adaption method, as well as the lack of related work comparison/discussion to other adaption methods.  Reviewers have shared some concrete suggestions that the authors could use to better improve this work.


justification_for_why_not_higher_score: Please see the meta-review and discussion. 
justification_for_why_not_lower_score: Please see the meta-review and discussion.
Decision: Reject