Topic Aware Transformer: Domain Shift for Unconditional Text Generation Model
Authors: Noriaki Kawamae
Venue: Submitted to ICLR 2023 (2023)
Keywords: Text generation
Domain  adaptation
Domain  shift
Transformers

Abstract: Our goal is to adapt pre-trained language models (PLMs) to support unconditional text generation tasks.
Because Transformer-based models are pre-trained on more massive and heterogeneous corpora than specific target corpus,
the gap between these corpora and the target corpus raises the question of whether these PLMs will actually benefit this task even after fine-tuning.
As the domain adaptation of PLMs needs to bridge this gap,
we propose a framework, Topic Aware Transformer (TAT), that adapts PLMs for target-aware text generation while alleviating catastrophic forgetting.
The motivation of TAT to distill the target-specific knowledge as topics,
and steer PLMs toward these topics.
This requirement and motivation lead us to introduce a topic steering layer (TSL) as an additional layer,
and Topic Distribution Modeling (TDM) as a training task.
Experiments show that these components resolve the gap as the domain shift,
and can tailor PLMs to generate text to better reflect a given small fine-tuning corpus.

Under review as a conference paper at ICLR 2023
TOPIC AWARE TRANSFORMER: DOMAIN SHIFT FOR UNCONDITIONAL TEXT GENERATION MODELS
A Anonymous authors
Paper under double-blind review
A ABSTRACT
Our goal is to guide pre-trained language models (PLMs) towards uncon- ditional text generation tasks while resolving the domain gap and avoiding the catastrophic forgetting. Because Transformer-baat based models are pre- trained on more massive and heterogeneous corpora than specific target corpus, the gap between these corpora and the target corpus raises the question of whether these PLMs will actually benefit this task even after fine-tuning. As the domain adaptation of PLMs needs to bridge this gap, we propose framework, Topic Aware Transformer (TAT), that adapts PLMs for Largel-aware lext generation while alleviating catastrophic lorgetting. The motivation of TAT to distill the target-specific knowledge as topics, and steer PLMs toward these topics. This requirement and motivation lead to introduce topic steering layer (TSL. as additional layer, and Topic Distribution Modeling (TDM) as a training task. Experiments show that these components resolve the gap as the domain shift, and can tailor PLMs to generate text to better reflect a given small fine-tuning corpus.
1 INTRODUCTION
Our goal is to adapt pre-trained language models (PLMs) to achieve unconditional text generation toward target domain. The success of Transformer-baser PLMs motivates us to explore how to fine-tune them so as to well reflect a given target corpus thereby generating more personalized texts with very few specializations. The size of the target corpus is generally much smaller than that of existing pre-training corpora, which may lead to catastrophic forgetting (Ramasesh et al. 2021). For example, the popular pre-training data sets of Giga5en (Parker et al.] 2011), and Clu 2012-B occupy 16G, and 25TB, respectively. PLMs can become biased toward the patterns language used the training data (Keskar et al. 2019). Given the rapid diversification of applications, pre-training approach is needed to effectively achieve domain shift without catastrophic forgetting.
Toward this domain shift, we propose framework, Topic Aware Transformer (TAT), that adapts PLMs as unconditional gencrative tasks while alleviating catastrophic forgetting. As the domain knowledge consists of global (c.g., linguistic) and specific (e.g., semantic) knowledge, our intuition is that knowledge can be represented as a distribution of words, and the gap between the source and the target domain can be taken to be differences between distributions. These intuitions motivate TAT to detect these distributions via topics, and steer PLMs toward these topics to highlight the target-specifee knowledge. That is, the motivation of TAT is to introduce topic steering layer (TST.) as an additional layer that detects topics and helps training PLMS theoretically, and Topic Distribution Modeling (TDM) as a training task to align text on the topic representation on the target domain. To prevent catastrophic forgetting, TAT can fine-tune PLMs while bridging the domain gap without updating PLM parameters.
Experiments confirm that TAT supports PLMs and verify its advantages as follows; Theoretical contributions: TSL allows topics to act unsupervised labels that represent global and target-specific word distributions as domain knowledge, and adapts PLMs to
/

---

Under review as a conference paper at ICLR 2023
resolve the gap and perform domain shift over topics.
Practical contributions: As TAT updates only the target-specific word distributions, and does not need to update the parameters of PLMs, it generates more target-specific texts at lower computational cost than possible when using previous PLMs alone, while preserving PLM's advantages.
PREVIOUS WORK
Recently, pre-trained neural language models (NLMs), such as BERT (Dvvlin et al. GPT2 (Radford et al.) 2019 XLNet (Yang et al. 2019). RoBERTa LLu et al. 2019
2019
and ALBERT (Lan et al. 2020 use Transformer Vaswani et al. 2017 for learning con- textualized text representations, and have yiclded great advances in NLP tasks. Though achieving appealing performance, these Transformer-based models are better at exploring the relationships among local tokens than global semantics (e.g., word collocation over given corpus) (Wang el al. 2020). As no Transformer-basse model considers these explicit semantics, Wang al. (Wang et al. 2020) rearranged and explored the semantics of topic models and developed topic-triendly assistant for Transformer based abstractive summa- rization models. UNTfied pre-trained Language Model (Dong et al. 2019) supports NL. and natural language generation (NLG) tasks by employing shared Transformer network and utilizing specific self-attention masks to control which context the prediction is conditioned
While He et al. improve (He et al., 2018) existing NMT models through layer-wise coordination of the encoder and decoder, and use modified attention masks in train both the encoder and the decoder simultaneously, their framework cannot be applied directly to sppaif-specific text generation; our framework differs from pre- pra-nennn in terms of task ob- jectives, topic roductio,, and fine-tuning. BertSUM (Wang et al. 2020) notes that topic models are better at learning explicit document semantics than Transformer. Different from their work, TAT aims adapt NLMs to text generation tasks by performing domain shift.
As existing PLMs used large raw text data that do not necessarily contain sufficient knowl- edge or patterns that are directly related to the target-specific task, they still suffer from several potential limitations. More preciscly, texts of specific task, e.g., movie review, can differ from PLMs training data (Chen et al.) 2022). To address the question of whether pre-training on a corpus more directly tied to the task can further improve performance, continual pretraining (Gururangan & et all (020)) has shown benefit of optimizing a PLM to target domain before further linetuning. UDALM (Karouzos et al.) 2021) first trains PLMs by masked language modeling (MLM) on the target domain and then trains a target classifier with source domain labeled data, while keeping the MLM objective on unlabeled target domain data. AdaPrompt (Chen et al. 2022) is a framework that can adapt a PLM for the end task considering both the prompts and the verbalizer, and adaptively contin- ual pretraining on the retvidedd data, which can benefit prompt-based methods on NLP rowmstream lasks.
While these training based approaches show the improvement in solving the domain gap, these practice of adapting and controlling pre-trained generative models poses the catas- trophic forgetting: most approaches to enforcing control objective result in a dramatic loss of capabilities of the original model beyond the scope of the control objective. As way to take advantage of this achievement, we noted that some knowledge is universal across domains and some is not. Therefore, our approach aims avoid this problem by incorpo- rating mechanism to recognize these relative differences, and intensively update only the target-specific knowledge.
As with global semantic information, topic models (Blei et al. 2003 Kawamac 2018 Wang let al. 2020), and their extensions, take a global statistical view and look at the word distributions of topics across a given corpus; they represent each document as a bag-of-wort (BOW) vector. Although these models organize a given corpus into small sets of prominent topics and have been proven to be powerful tools for uncovering latent structure, they and their application (Chang et al. 2021 sequence models.
Wang et al. 2018 2020 are not, in the strict sense,
2

---

Under review as a conference paper at ICLR 2023
source domain
Larget domain
domain gap
am looking forward to meet you ....
TID
Xr
Transformer decoder
lingui stic knowledge
semantic knowledge
opic Steering Layer
IDM
Wy
Wy
Wer by
topic cistribution D
high
F(bea
bLA
h az
Waz
ba
L-th Transformer
transformer layer
alobal word distribution
target specittc
word distributton
1-th Transformer laver
2 Layer Norm(b.)
<>> am looking forward to meet you ...
Figure 1: (left) The gap between the source and the target domain, domain gap, topic distribution, and word distributions, (center) The architecture overview of TAT that can adopt both NLMs and PLMs (e-g-, GPT2), where their parameters are duplicated to form the TAT architecture. Newly introduced elements, Topic steering layer (TSL. and TDM, are highlighted in red, and TID denotes the representation of each text. (right) the detail of TSL with the affine in Eq (5). TAT learns Wv, Wz, Was and b, and trains itself using TID and TDM on each text.
That is, we focus on bridging the domain gap as the domain shift, and aim to distill target- specific knowledge as topics, and stcers PLMs toward these topics. This approach designed to be compatible with training strategies while alleviating catastrophic forgetting.
3 METHODOLOGY
3.1 PROBLEM FORMULATION
Neural language models (NLMs) are trained as conditional language models for those specific tasks that require text gencration (Bengio et al., 2003), PLMs. Given text sequence X = {.a,1.' ", (na) and dataset D X11 Xpi, NLMs are pre-trained by maximizing the following likelihood under forward autoregressive factorization:
[D]
[D] Iaal
(0) = -- log Po(xa) = ΣΣ
log Po(zat
(1)
d=
d=1
where 0 represents model parameters.
As PLMs are trained on heterogeneous corpora, we can observe that the source and the target domain have both common (e.g., linguistic) and different (e.g, semantic) knowledge, as shown in Figure Since the gap intuitively in the difference between distributions over words (i.e., global and target-specific word distributions), and their distribution (i.e., topic distribution) over domains, we must recognize these distributions and train PLMs update them. For example, given "My favorite artist is", PLM might predict "Michelangelo" as the next word, whereas the fino-tuned PLM yields "Botticell".. This leads us to introduce the latent variable of topic, 2, into the NLMs and then modify Eq (1 to:
[D] Iza
2
(TLM( = - log Σ d=
Polza,lze x1. Po(alxxaaa-1
(2)
2==1 word distribution topic distribution
where 2, that indicates which global/target distribution is used, and Z the number of topics. Different from previous NLMs, topic language model (TLM) explicitly introduces topics into the generative process to utilize richer contextual information for improving NLMs/PLMs performance. Note that Po Po(2)(xa.1a-1) is multinomial distribution over discrete variables, not the Gaussian distribution used in variational autoencoders (Kingma
& Welling 2014) and its extensions such as (Wang & Wan 2019 Zhu et al. 2021 Cai &
Cal 2022 This paper explores how to distill the domain-specific knowledge as topics, and
steer PLMs toward topics.
3.2 MOTIVATION AND ARCHITECTURE DESIGN
As our challenge is to resolve the gap between the source and the target domain for the domain shift, it adapts PLMS to generate unconditional texts that reflect the target-domain
3

---

Under review as a conference paper at ICLR 2023
more than the source. The motivation of TAT is that domain adaptation should detect topics and update related distributions in the fine-tuning stage while preserving the semantic meaning and language structural information that pre-trained NLMs have, discussed in 3.1, Following this motivation, (=(=)(xa,1a1 in Eq (2) can be interpreted as "the distribution over topics". 21, in Eq 2 is "the distribution over next words". They may be global or specific between the source and the target domain. The global common over both domains and can be distribution based on linguistic knowledge, while the specific is a distribution based on domain-specific knowledge. Although these differences are relative, vary with a given corpus, and cannot be clearly defined, the ratio of global word distribution in linguistic knowledge is intuitively considered much higher than that in semantic knowledge as shown in Figure [ lopics are designed to identify these different distributions and their weights.
These topics literally inherit the characteristics of latent topic models. While Transformer encodes context as local information, it requires large corpora to learn the higher-order and non-linear interactions between words, which demands more parameters, computation re- sources, and time. It is often observed that the learned attentive patterns of many heads are not as reasonable as we expect (Michel et al.) 2019), and we might obtain this global information from the upper blocks by increasing the number of transformer blocks (Doso- vitskiy et al. (02T); unfortunately, as the transformer architecture requires a large number of parameters, its computational cost is very high. Ramasesh et al. (Ramasesh et al., 2021 pointed that catastrophic forgetting occurs mainly in the higher layers. These insights lead us to place a topic steering layer (TSL) on the top Transformer layer and update only its related paramcters to avoid this forgetting. This does not break any PLM structure, and al- lows reuse of PLMs and their parameters. As 1) a topic describes a co-occurrence pattern of tokens with similar semantics, and 2) the differences between pre-training and fine-tuning data sets exist not only in the topic itself, but also in the ratio of topics, we develop training task, Topic Distribution Modeling (TDM), to align topics with each text.
Since global distributions do not require additional learning, our architecture is designed to find target-specific distributions through topics, and update them, P=((x((Xa,1a=1) and Po 2t, Xd,1:-1),, in fine-tuning. This design enables PLMs to emphasize knowledge that might otherwise have been buried, and prevent catastrophic forgetting.
3.3 INPUT
Given a larget domain corpus, TAT feeds the lext as input to the decoder, as shown in Fig Its layers convert the inputs into token (linguistic) embedding, and add special tokens [CLS], [SEP], [EOS], and 
. Following the text preprocessing of other Transformer- based NLMs, TAT tokenizes each input text to create the linguistic input of token embedding, where each sub-word is embedded with Word Piece (Wu et al. 2016) or another model-specific tokenizer (e.g., Byte-Pair Encoding (BPE) vocabulary (Radford et al. 2019 whose length equals input length. [CLS] token is only inserted prior to the token, and denotes the class of each source text. [SEP] token is assigned to the end of each sentence in each input sequence, and indicates a sentence break. [EOS] token is assigned only after the last token in each input sequence. 
 token is only inserted prior to the token in each target text. Following other models, a learnable sequence position embedding is added to every input element indicating its order in the input sequence.
3.4 ATTENTION MECHANISM
Inside each Transformer layer, the previous layer's output H E Rla|xda is aggregated using multi-head self-attention, where |x] is input sequence length. Thus the block core is multi-head attention with heads that use a causal mask to preclude attending to future tokens via the scaled dot-product attention:
Q Q= W K H WK. V - H WY,
QKT
--x if < j.
(3)
Attention Q K, V) = =softmaxt
+ M) Mi =
Vd
0
else
4

---

Under review as a conference paper at ICLR 2023
where W W wP E dd are learnable weights for computing the queries, keys, and values, Q K. V Rizlxde, respectively, dg is the shared dimensionality of the queries and keys. The self-attention mask, M E | determines whether a position can attend to other positions, where, My allows the i-th position to attend to the j-th position and Ma = -0 prevents attending.
3.5 TOPIC STEERING LAYER (TSL)
As shown in Figure I. TAT places these topies and their layer, Topic steering layer (TSL.) on the top of Transformer layers to steer the text decoder. Then, TSL, maps hidden repre- sentation vector Hr = "..,,,,.,,,,,, topic-specific distribution over words by defining lopic matrix, Wz E Rdd and word generation function, F(hr.t), where V is the size of the vocabulary. We apply these matrices to hL E Rd in the text decoder, gain Xr and use it to sample the next token, Fi, according to the probability:
R =|xx into topic vector z E R² and selects the given topic. This transformation yields Eq (2)
x F(hL,,,)), Po(2) x Layer.oor(bb..)) W2, X7 = Nayer ) W2
(o(ra,)xa,1011 21) x F(hL,t,))), Po(2aclx41111)
Σ
(2) (4)
cxp(Po((al n. x,110 2//Te) Iy ~ p(z; E XT)
P(z, E XT) xx)== E, Exp((P((Ed = 2 XX,
where Wz learnable weights, Te 0 is temperature and x; is the score of the i-th word in the vocabulary. As Xr is normalized into p(I; E XT) to yield the probability over words, the next token is chosen by sampling multinomial distribution with probabilities clipped to the top-k tokens. cemperature-controlli for generating text from trained NLMs or PLMs. While Te + 0 approximates a greedy distribution, which magnifies the peaks in the probability distribution, Te + oo flattens the distribution and makes it more uniform.
stochastic sampling methods are used
As with F(hL,1,2) 24), we propose three transformations (addition, multiplication, and affine) to generate Fd,t that accords with the given 2 and Xd,1:t-1, h
h
residual if 2
(1 - w--)) + wg. addition if 2 and > 0 multiplication if 21 and
TID, Exhi, F(hL,, = Wy x
(5)
h g:
affine if 21 = and x > 0, where Wy E XV ga E Rdh, Was Rdi and b. E dR are the topic z specific learnable weights. We prepare the residual to select the input if a = O, take h, as the global word distribution shown in Figure [ which preserves PLM functionality, propose an alternative (i.e., addition, multiplication, and affine) for = > 0, and confirm by a pre- ablation analysis that affine is the best. Since both the ratio of the global words distribution and type of target-specific word distribution depend on the given target corpus, they are determined relative via fine-tuning using Eq (4) TID; E Rdd denotes the average of h over each input, i-th text, without using W..
h1, Was + bN
H1
Note that just as Eq (I] is transformed into Eq (2) through the introduction of topies, the top layer of previous Transformer based NLMs/PLMs is decomposed into the product of Wz and F(hL,)) in Eq (4). While both Wz and Was (b..g:) are newly introduced parameters, Wr is the existing parameter and updated in fine-tuning. Different from other Transformer-based PLMs, TAT 1) aligns the i + 1-th topic of target text, Po(z (x,,==1),, and weights Pa 11.11 Pa 2)) according to the distribution over topics, and 2) samples each token according to p(r; E Xr).
The top hidden state, Hr, reflects the contextualized representation of the whole sequence in the decoder. As TAT applics the concept of topic to distill the target-spocific knowledge as topics, the average of the token-level hidden states over each i-th text corresponds to topic distribution topic models, and N-gram topics by incorporating both the preceding topics and the topic specific distributions over words.
5

---

Under review as a conference paper at ICLR 2023
4
MODEL TRAINING
Our training uses our newly introduced Topic Distribution Modeling (TDM)
4.1 TOPIC DISTRIBUTION MODELING (TDM)
The objective of TDM is to minimize the difference between document-specife topic distribution and a text specific representation, TID. The topic vector the average of 2 each d-th text with Eq 4 Rd, and is gained by multiplying the transpose matrix of Wz used in Eg (4) to match the number of dimensions z, and TID, on da While we can make each text-specific topic distribution, by computing the mean of max-over-time of all output topics, Z1:6: we examine the best, the mean of distribution, in our comparison experiment. Since Zd learns representation directly through topics as shown in Figure 1 texts with similar content, as discussed in topic models, are considered to have similar lopic distributions. As shown Figure I TID is the final output of the last loken of each input text sequence, and denotes the representation of each text.
H2
or
For the metrics of L2 regression, cross-entropy, Ki--Divegence, and triplet objective, our experiments confirm that TDM with triplet objective, CTDM((0), attains better performance than the alternative combinations, and so employ it in our framework. Since there no label that is the ground truth in unconditional text generation, we z the learning objective. Given Zd as an anchor representation representation, TIDa, is taken TID; is taken as a negative embedding. Triplet loss lunes the model such that the distance between 2a and TIDa is smaller than the distance between za and TID Mathematically, this objective minimizes the following loss function:
the i-th text, its corresponding text positive embedding, while the other text representation,
( (R. (II.TID..IIg)-
max (l|za TIDall 1|zl TID +c,0),
(6)
where B each batch, || - is a distance metric, and e is the margin that ensures that Za is at least closer to TIDa than TID;; the sampling targets are batch units.
4.2 TRAINING OBJECTIVE OF TAT
We employ a unified multi-task learning framework that updates the decoder. As TAT can adapt PLM-based NLMs, their parameters, 0, of Eq (2) used to initialize the TAT decoder, and a fine-tuning process is employed to adapt 0 to the finc-tuning data. To optimize these parameters and bridge the gap between the data used in the pre-training and the fine-tuning process, we optimize the model loss in this luning process. Using Eq G we can define the loss function, L(0), as the sum of these objective functions that optimized in the the-tuning stage:
LTAT( = -LTLAS(()) + ATDM
(7)
where A is the parameter set of TAT, ATDM are hyper parameters to balance the importance of TLM and TDM. We use Adaptive Moment Estimation (Adam) {Kingma & Ba 2015 mini-batches to update parameters, and adopt the dropout strategy (Srivastava al.
2014 to optimize networks.
5 EXPERIMENTS
DATASETS AND EXPERIMENT DESIGN
Datasets We conducted evaluations using Amazon review and Yelp they are large publicly available datasets and are manually evaluable for screened colleagues. We used these datasets because the resulting data size is computationally feasible general- purpose server, includes a variety of topics that are different from pre-training corpus, meets

---

Under review as a conference paper at ICLR 2023
Table 1:
Basic statistics of the datasets used in this paper: In the attributes, #Z denotes the number of topics for evaluating topic coherence.
Dataset
#reviews
#vocabulary
#7
Amazon
210,000
246,534
100
Yelp 6,685,900
365,762
200
the public reproducibility requirement, and validate insight that small corpus can provide significant benefits [Gururangan & et al, 2020). More prociscly their domain similarity is far from PLMs and high enough that they can be used additional training data for each other in domain-adaptive pretraining (DAPT) (Gururangan & et al 2020 Each record the dataset contains a review text, review title, star rating, anonymized ID. and coarse-gre coarse-grained grained product category, we use only review texts. All reviews were truncated after 2,000 characters, and all reviews were at least 50 characters long. Among the languages present, we used only English for ease of interpreting the results. We applied the same pre- treatment to the Yelp data set and statistics of the resulting data set are shown Table We used 90%, 5%, and 5% of each data set as training (e.g., text generation), validation and test sets, respectively. The final performance comparison results are derived from the test set, which corresponds to task-adaptive pretraining (TAPT) (Gururangan & al 2020).
Experiment Setup We implemented TAT by using Pytorch 1.7.1 and will release this code soon. We set in Eq (6) to 0.2, and ATDM in Eq (7) to 0.5. As the average length of each text used in fine-tuning data set is around 60, we set the maximum input sequence length to 64. Note that the ground truth texts were excluded from training, validatioo data to prevent information leakage. TAT uses GPT2 as the PLM. Following the training setting, we used Adam with 31 0.9, and B2 = 0.999 was used for optimization, over mini-batches to update parameters; the dropout strategy (Srivastava et al. 2014) was adopted for network optimization. The learning rate was 3c-5, with linear warmup over the first 500 steps and linear decay, where we set the dropout rate, the weight decay, and the batch size to 0.1, 0.01, and 256, respectively. We fine-trained all models on 8 Nvidia Tesla V100 GPUs, each with 32G memory.
5.2 TOPIC COHERENCE
This experiment aims to evaluate how well TAT discovers topics, and compare TAT with existing topic-based embedding models. In order to quantitatively assess topic quality, we use the topic coherence measure (Mimno et al.) 2010) to examine the relatedness of the top-ranked words. This score shows high consistency with human judgements in terms of topic quality (Mimno al.] 2010). where higher scores indicate greater topic coherency. As the baselines, we used NMF LLe & Seung 1999), and LDA+TWE (TWE-1 Topic2 with CLM' We set the iterations of the Gibbs sampler, parameter update or epochs to 200 for all models except TAT, where the first 50 itcrations were used to burn in the Gibbs sampler; CLM used matrix factorization in learning word embedding representation. We varied the number of top ranked words, and measured the resulting performance by using the coherence model [unction of gensin with "u mass".
Table leftt) shows that the top words of the learned topics are semantically coherent, which coincides with the finding that using word embedding improves the quality of topic models (Liu et al., 2015 Xun al.) 2017 Nguyen al. 2015). TAT learns topics using the order of both word and topic in each document and updates them iteratively, groups semantically--reltted words more efficiently than the alternative approaches, and yields more distinct topics.
'https://pytorrcheen
http:://higgiiiilcccccccamm/meep html
"https://githubb.oo

---

Under review as a conference paper at ICLR 2023
Table (left) Compparion of topic coherence, (right)Fxampik topies discovered by TAT: N is #ranked words. In this table, T2V denotes Topic2Vec. The size of the embedding space for Vee, CLM and TAT was set to (00(Amazon)/20) (aopazooon20((()l))))))) the skip length and #negative sampling were set to 5 and 5, respectivcly. The values in bold show best performance, and denotes the statistical significance for p < 0.01, compared to the best baseline. These words were selected from their distance from the topic embedding.
Amazon
Yelp
topic topic 2
topic topic 4 action sci-fi dark
N
5,10,20
5,10,20
iphone
small
excellent
TWE
-2.11,-2.26-33333
-..93,33.08,-3.76
os clear
NMF
-1.92.-2.54-33.23
-2.67.-2.91.-33822
touch
better
visual
cool pleasing
T2V
-1.75,-2.13-2.61 2.37
CLM
-1.71,-11.2.-2226 -1.52,-1.82,-2.22
camera
long
staring
battery
fantastic
effect classic
TAT
-0888-11.23-11.77
-0.92,1.455-11.01
Table 3: Comparison and Ablation analysis of various adaptation methods: In this table, F, P, T, D-N, B-N, M and RL denote Fluency, Perplexity, Topic, Dist-N, BLEU-N, METEOR, and Rouge-L respectively. In each row, upper/lower value is Amazon Yelp. GPT-2+I, +pt, and +DT is GPT-2 after fine-Luning, with prelix-tuning (Li & Liang 2021 , and with DAPT-TAPT (Gururangan GPT-2 to evaluate the effect of TSSL and TDM, TAT+DT freezes them after DT. The bold values have equivalent meaning to its usage in Table 2
al 2020), respectively. TAT freezes the parameters of
Model
F
P
T
D-2
D-3
B-2
B-3
M
RL
GPT-2
4
t
1
1
1
3.11
27.32
0.73
5.12
10.02
14.48
7.82
10.23
17.36
+f
2.65
32.32
0.68
4.55
9.03
14.21
7.22
9.67
16.11
3.08
27.38
0.73
5.02
10.02
14.93
7.88
10.83
17.8
+pt
2.63
32.34
0.68
4.57
9.06
14.33
7.25
10.45
17.21
3.26
26.12
7075
3.23
10.31
15.63
88.22
11.23
18.78
+DT
2.63
32.34
0.68
4.57
9.06
14.33
7.25
10.95
17.78
TAT
Z
TDM
10
3.45
25.16
0.79
5.64
11.18
17.61
8.88
12.21
21.14
10
W
2.85
29.88
0.75
4.67
9.52
15.38
7.71
11.78
20.05
20
3.56
23.21
0.81
5.91
12.12
18.13
9.12
13.14
22.15
20
2.97
27.23
0.78
5.06
10.25
16.42
8.38
12.16
20.98
20
W/O
3.41
25.21
0..78
5.62
II.I2
17.58
"888
12.12
20.85
20
W/O
2.81
29.91
0.74
4.62
9.47
15.35
7.69
11.72
19.92
20
3.45
25.16
0.79
3.64
11.18
17.61
8.88
12.21
21.14
+DT
20
2.85
29.88
0.75
4.67
9.32
15.38
7.71
11.78
20.05
We show examples discovered topics Table 2 where the ID of the topic arbitrary: topic2/4 co-occurs with topic1/3. This result shows that topic 1 and topic 3 have many words associated with products and contents, while topic 2 and topic 4 have many subjective words corresponding to features. This indicates that TAT extracts interpretable topics.
5.3 TEXT GENERATION
Baselines: To evaluate the quality of generated texts, we add the latest adaptation ap- proach, GPT2 with prefix-tuning [Li Liang 2021). DAPT+TAPT (Grurangan & all 2020
Automated evaluation: We used test-set perplexity, Dist [Li et al.
2016). and BLEU-
N (Papineni et al. 2002) metrics to mcasure performance, METEOR Lavic & Agarwal
2007), and ROUGE Lin 2004) metrics to measure performance (Sai et al.
2023). Perplexity
an automated measure of Huency, and while its effectiveness has been questioned in open- domain text generation [Liu et al. 2016), we use the well-known test-sct perplexity using different pre-trained NLMs. n-gram based metrics (Dist, BLEU, METEOR, ROUGE) count the overlap between the generated text, and its corresponding reference text in the test data after removing seed words.

---

Under review as a conference paper at ICLR 2023
Table 4: Case study for Amazon: (top) Ground Truth, (center) GPT2+TAPT+DAPT,
and (TAT), We set seed words of "I am disappointed in this purchase", and show the text generated by each model.
I am disappointed in this purchase. T bought one of these in another color and in size XL. The color is not as vibrant as would like. It does however still look great. will use I ordered an XL size in black which arrived with large hole. There's no way anyone
Human evaluation: We employ fluency lesting on altribute relevance as the human an- notation DDthathri et al. 2020). Annotators were asked to evaluate the fluency of each individual sample on scale of 1-5, with 1 being, not fluent at all, and 5 being, very fluent, as done in (Lample et al.] 2019). Topic reports the fraction of samples matching the target domain evaluated by the manual annotators. To consistently evaluate generated texts, we recruited and screened 10 colleagues who were familiar with movics, music (Amazon), and restaurants (Yelp) and who could interpret reviews.
H3
As shown in Table 3. TAT outperformed the baselines and achieved better performance over both data sets. These results support our hypothesis that TSL allows TAT to distill knowledge in the form of topics, and update only the target-specific word distributions to prevent catastrophic forgetting. The decline due the introduction of DT into TAT implics catastrophic forgetting.
Comparisons:
Ablation analysis: To investigate the respective contributions of TAT components, (i.c., TSL and TDM), we conducted an ablation analysis. We removed different components and the resulting text generation quality shown in Table This table shows that the complete setting of TAT achieves better performance across both datasels. A within table comparison shows that TSL is most effective. By comparing the effects of TSL. and TDM for the same pre-training data, these newly introduced tasks improve text generation performance, while TL was more effective than TDM and DAPT+TAPT.
Error analysis: A manual error analysis showed that some instances marked errors were in fact assessed correctly as allowed by partial matching of words in a text. When the ground truth text is personalized, human judgement is difficult even if the generated text is different from the ground truth, see Table 4 where the generated lexts included more abstract higher frequency words than the relerence sentences.
6 DISCUSSION
As TAT adapts PLMs toward the target domain as domain shift, it focuses on the difference between the source and the target domain as word distributions, uses topics as memory and an indicator to select these distributions through the use of TSL and TDM. Topics are based on word co-occurrence the texts, which varies with the dataset, and allows TAT to capture the global information. Previous NLMs use local context information to capture semantic meaning. Formally, TAT performs domain shift by recalculating Po [2, X1,1=-1) and aligning the (2 xd,1:t1)) with the target domain, see Eq (2) and Figurc]
TAT mitigates the gap between the source and the target by highlighting the target-specific word distributions through topics and updating only these distributions, even if Z is small.
The number of parameters in W E R44 x2 Z x W... E dRd, xdd, and x by E Rdd newly introduced by TSL, is a much smaller number of parameters than PLMs, and fixing PLMs's parameters in the fine-tuning stage lowers the overall computational cost; its effectiveness shown in Table 3 Just as topic models have enjoyed success in arcas other than text processing, fine-tuned PLMs with TAT could be applied to other tasks.
H4
7 CONCLUSION
The proposal of this paper, TAT, can adapt PLMs to the unconditional text generation task while using the target domain a constraint. Experiments showed that the components of TSL and TDM enable TAT to discover suget-domainnspeciffo topics, fine-tune PLMs over these topics, and generate valid texts reflecting given small fine-tuning data set.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural proba- bilistic language model. J. Mach. Learn. Res., 3:1137-1155, Mar 2003.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent dirichlet allocation. JMLR, 3:993 1022, 2003.
Zefeng Cai and Zerui Cai. PCVAE: generating prior context for dialogue response generation. In IJCAT, pp. 4063-4071, 2022.
Haw-Shiumg Chang, Jiaming Yuan, Mohit Tyyer, and Andrew McCallum. Changing the mind of transformers for topically-oomtrollalee language generation. In EACL, pp. 2601 2611,
2021.
Yulong Chen, Yang Liu, Li Dong, Shuohang Wang. Chenguang Zhu, Michael Zeng, and Yue Zhang. Adaprompt: Adaptive model training for prompt-based ulp. CoRR, abs/2202.04824 2022.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In ICLR, 2020.
Jacob Devlin, Ming-We Weig Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACLM, pp. 4171 1171-4186, 2019.
Li Dong, Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and understanding and generation. In NewrIPS, pp. 13042-13054 2019.
Hon. Unified language model pre-training for natural language
Alexey Dosovitskiy, Lucas Beyer, and etc. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Suchin Gururangan and et al. Don't stop pretraining: adapt language models to domains and tasks. pp.
2020.
Tianyu He, Xu Tan, Yingee Xia, Di He, Tao Qin, Zhibo Chen, and Tic-Yan Liu. Layer-wise coordination between encoder and decoder for neural machine translation. In NIPS, pp. 7955 7965, 2018.
Constantinos Karouzos, Georgios Paraskevopoulos, and Alexandros Potamianos. UDALM: Unsupervised domain adaptation through language modeling. In ACL, pp. 2579-2590, 2021.
Noriaki Kawamae. Topic chronicle forest for topic discovery and tracking, In WSDM, pp. 315-323, 2018.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05855, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Mare' Aurelio Ran- zato, and Y-Lan Boureau. Multiple-attribute text rewriting. In ICLR, 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, and etc. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR, 2020.
10

---

Under review as a conference paper at ICLR 2023
Alon Lavie and Abhaya Agarwal. METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop Statistical Machine Tronslation, WMTOACL 2007, pp. 228 231, 2007.
Daniel D. Lee and H. Scbastian Soung. Learning the parts of objects by nonnegative matrix factorization. Nature, 401(6755):788-79)) 1999.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-rromoting objective function for neural conversation models. In NAACL, pp. 110-119, 2016.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for genera- tion. In ACL/IJCNLP, pp. 4582 4597, 2021.
Chin- Yew Lin. Rouge: a package for automatic evaluation of summaries. In ACL-workshop, pp. 25 26, 2004.
Chia-Wer Liu, Ryan Lowe, Tulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In EMNLP, pp. 2122 2132, 2016.
Yang Liu, Zhiyuan Liu, Tal-Seng Chua, and Maosong Sun. Topical word embeddings. In AAAI, pp. 8418-2422 2015.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized 11692, 2019.
BERT pretraining approach. CoRR,
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In NewrIPS, pp. 14014-14024, 2019.
David Mimno, Hanna M. Wallach, and etc. Optimizing semantic coherence topic models. In EMNLP, pp. 262 272, 2010.
Dat Quoe Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. Improving topic models with latent feature word representations. TACL, 3(4):299 313, 2015.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing 7hu. Bleu: a method for automatic evaluation of machine translation. In ACL, pp. 311 318, 2002.
Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword fifth edition, June 2011. L.C2011T07.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Nya Sutskever. Language models are unsupervised multitask learners. 2019.
Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In ICLR, 2021.
Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. A survey of evaluation metrics used for NLG systems. ACM Comput. Surv., 55(2):26:1-26:39 2023.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Tlya Sutskever, and Ruslan Salakhut- dinov. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1999-1958, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, and etc. Attention is all you need. In NIPS, pp. 5998 00086008, 2017.
Tianming Wang and Xiaojun Wan. T-CVAE: transformer based conditioned variational autoencoder for story completion. In IJCAT, pp. 1233-5233 2019.
Wenlin Wang, Zhe Gan, Wenqi Wang, Dinghan Shen, Jiaji Huang, Wei Ping, Sanjeev Satheesh, and Lawrence Carin. Topic compositional neural language model. In AISTATS, pp. 356 366,,,652226 2018.
11

---

Under review as a conference paper at ICLR 2023
Zhengjue Wang. Zhibin Duan, Hao Zhang, Chaojie Wang, and etc. Friendly topic assistant for transformer based abstractive summarization. In EMNLP, pp. 485-497, 2020.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, and et. al. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144, 2016.
Guangxu Xun, Yaliang Li, Jing Gao, and Aidong Zhang. Collaboratively improving topic discovery and word embeddings by coordinating global and local contexts. In KDD, pp. 535 543, 2017.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. Xlnot: Generalized autoregressive pretraining for language understanding. In NIPS, pp. 5754-5764, 2019.
Lixing Zhu, Gabriele Pergola, Lin Gui, Deyu Zhou, and Yulan He. Topie-driven and knowledge-aware transformer for dialogue emotion detection. In ACL, pp. 1571-1582, 2021.
A APPENDIX
12


Reviews:
[{"review_id": "xDf0fyFD26V", "reviewer": "Reviewer_6wEU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates domain adaptation for an autoregressive language model.\nA latent topic distribution is calculated based on the input words, which is then used to modify the token probabilities during generation.\nEvaluation is performed on two review datasets (Amazon and Yelp).", "summary_of_the_review": "Possibly interesting and useful ideas.\nThe clarity of the paper needs improvement and the evaluation should be against more relevant baselines and on more datasets.", "strength_and_weaknesses": "Strengths:\n* The of estimating the topic of the text first, then generating based on that is interesting and possibly promising.\n* The empirical results are good compared to the chosen baselines.\n\nWeaknesses:\n* The clarify of the paper is low, prohibiting understanding of how the method actually works.\n* Experiments are performed only on two datasets, both from the same domain. For convincing results regarding the domain adaptation method, more than one domain should be investigated during evaluation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rLJRxIF_PB", "reviewer": "Reviewer_DzP1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new topic-aware transformer that alleviates the gap between the pretrained language model and the target domain. The paper introduces an additional topic latent variable $z$ as an additional condition when generating new tokens. The new topic latent variables come from the topic steering layer, which maps hidden representations into the topic vector. The model is optimized with both topic distribution modeling and maximum likelihood. The backbone of the paper is GPT2. The model is tested on two datasets: Amazon and Yelp. The paper checks the topic coherence and its text-generation ability  The paper did both automatic and human evaluation. The paper also did additional ablation analysis and case study. ", "summary_of_the_review": "Overall, the proposed new model: the topic-aware transformer is quite interesting and follow s motivation. However, some parts of the paper are not very clear. The experiment section and paper format need to be improved. ", "strength_and_weaknesses": "Strength:\n1. The paper proposes a new topic-aware transformer to resolve the domain shift for unconditional text generation. The motivation of the paper is clearly stated. The TAT achieves the best performance for topic coherence and text generation. The whole idea is interesting.\n\n2. The paper conduct both automatic and human evaluation. The paper also did some ablation analysis to show the contribution of each component. The paper also presents some generation results as examples.\n\n\nWeaknesses:\n1. Some parts of the paper are not clearly written which will be discussed in the clarity section.\n\n2. Some topic-related papers are not covered in the related work such as Zhu et al., (2021), and Chang et al., 2021.\n\n3. The citation style is not correct. Authors need to use citep{} The paper uses large space to squeeze tables and captions, making readers hard to read. Authors should not abuse space.\n\n4. The evaluation metrics used in the paper are a little bit old and only concentrate on the token overlaps. The paper needs to include some newer metrics such as BERTscore (Zhang et al., 2019), and BARTScore (Yuan et al., 2021) which can check semantic similarity. \n\n\nZhu, L., Pergola, G., Gui, L., Zhou, D., & He, Y. (2021). Topic-driven and knowledge-aware transformer for dialogue emotion detection. arXiv preprint arXiv:2106.01071.\n\nChang, H. S., Yuan, J., Iyyer, M., & McCallum, A. (2021). Changing the mind of transformers for topically-controllable language generation. arXiv preprint arXiv:2103.15335.\n\nZhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.\n\nYuan, W., Neubig, G., & Liu, P. (2021). Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34, 27263-27277.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kV66hHFAj4", "reviewer": "Reviewer_cPS3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes to further fine-tune an unconditional language model on a domain-specific corpus via modeling the topics as a discrete latent variable. ", "summary_of_the_review": "The method proposed may be good but the bad writing makes it hard for readers to discover its significance. ", "strength_and_weaknesses": "Strengths:\n1. The general idea of modeling topics as latent variables for domain adaptive pre-training does make sense. \n2. The results look good.\n\nWeaknesses:\n1. The format of this paper has been totally changed. I think it is not allowed to change the template. The font looks weird. And those table layouts look very condense and hard to read.\n2. The notations of equations look very confusing. And the  descriptions of methods are also hard to follow. I will pose all my questions on the method section in the following section.\n3. Since I cannot fully follow the method, I have concerns on the effectiveness of the method.\n4. In the results section, those baselines have not been well described so it is impossible for me to judge whether the baselines are enough. Even the metric used for evaluating the topic coherence is not described in enough details so I do not know whether this metric is suitable for the task or not.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cdqERELvCR", "reviewer": "Reviewer_qrSx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a topic aware transformer (TAT) to adapt pretrained language models to target domains through topic modeling. The motivation is, to alleviate forgetting, TAT explicitly quantify the domain shift as topic shifts. By introducing a topic steering layer (TSL) on top of transformers, TAT decompose the task of predicting next tokens into two tasks: (i) Firstly, predict the distributions of topics given previous tokens. (ii) Secondly, shift the word distribution according to predicted topics and then predict the next token. Combining the TSL with a newly introduced topic distribution modeling (TDM) loss, which aims at aligning the topic prediction, the proposed TAT achieves better performance of unconditional generation on two datasets, comparing to DAPT+TAPT and prefix tuning.", "summary_of_the_review": "I think this paper proposed an interesting and novel idea, with good experiments.\nMy biggest concern is the missing of comparisons to previous work like PPLM and FUDGE, which makes it harder to judge the real contribution of this work.\nI also suggest there should be more tasks to play with the proposed model.", "strength_and_weaknesses": "Strength:\n1. This proposed idea is novel, interesting and well-motivated.\n2. The resulted models can bring us more interpretability of the process of language modeling, by predicting topics before predicting next tokens.\n3. The evaluation on Yelp and Amazon datasets are comprehensive.\n\nWeakness:\n1. It seems that the proposed model can and should go beyond unconditional generation tasks. After introducing an extra TSL layer and learning meaningful topics, it seems unconditional generation tasks cannot fully leverage the potential of learned models. If we treat the learned topic distributions as learned interpretable latent variables, it should be easy and interesting to inject more topic control into the generation process.\n2. While \"alleviating catastrophic forgetting\" is one of the authors' motivations, discussion and comparison of tuning-free adaption is missing. For example, Plug and Play Language Models: A Simple Approach to Controlled Text Generation (https://arxiv.org/pdf/1912.02164.pdf), and FUDGE: Controlled Text Generation With Future Discriminators (https://aclanthology.org/2021.naacl-main.276.pdf). By such tuning-free controlled generation approach, we can also easily adapt PLM to target domains by changing the predicted word distributions.\n3. The presentation can be improved, such as the form of citation makes section 2 really hard to follow. The left part and the right part of figure 1 are also hard to understand.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject