Topic Aware Neural Language Model: Domain Adaptation of Unconditional Text Generation Models

1 INTRODUCTION . Our goal is to adapt pre-trained neural language models ( NLMs ) to the unconditional text generation task for the target domain . As with global semantic information , topic models ( Blei et al. , 2003 ; Kawamae , 2018 ; Wang et al. , 2020 ) and their extensions take a global statistical view and look at the word distributions of topics across a given corpus ; they represent each document as a bag-of-word ( BOW ) vector . Topics are global variables , describes the distributions over all tokens in the vocabulary , and form interpretable representations over texts . Although these models organize a given corpus into small sets of prominent topics and have been proven to be powerful tools for uncovering latent structure , they are not , in the strict sense , sequence models . Recently , Transformer has been applied as pre-trained NLMs ( Vaswani et al. , 2017 ; Yang et al. , 2019 ) , and are shown to be highly successful in the domain of natural language processing . Their success motivate us to explore how to adapt Transformer based NLMs for reflecting a given fine-tuning data and generating more private text with a few modifications than ever before . The corpus size of this fine-tuning is generally smaller than that of pre-training corpora , and may be unbalanced for training NLMs . For example , the size of the popular pre-training data , Giga5en ( Parker et al. , 2011 ) , and ClueWeb 2012-B1 is 16G , and 25TB , respectively . The fine-tuning needs a different approach to training NLMs than the pretraining approach , as pre-trained NLMs should be biased towards the patterns of language used in the training data ( Keskar et al. , 2019 ) , and may not lead to domain adaptation . To tackle this imbalance problem , we explore how to adapt pre-trained Transformer based NLMs to the target domain in the fine-tuning , while preserving their advantages . Since topics exist over these datasets and represents their semantic structures such as higher-order and non-linear interaction between words , our model , Topic Aware Neural Language Model ( TAN ) , focuses on topics to resolve this imbalance between these corpora and bridge their semantic gap . As topics reflect a set of co-occurring words , semantic information , and their 1https : //www.lemurproject.org/clueweb09.php/ syntactic structure , as global statistical information , TAN introduces a new manipulation , “ topic-alignment ( TA ) ” , and new training tasks such as Topic Distribution Modeling ( TDM ) and Topic Embedding Modeling ( TEM ) While previous Transformer based NLMs are better at learning from the predefined segment length such as the context , local information using Multi-head attention ( Vaswani et al. , 2017 ) , TAN injects topics into Transformer to train NLMs by fine-tuning with emphasizing these dependencies over segments in the target domain via topics . This model not only captures global semantic coherence of all segments and word concurrence patterns , but also enriches the representation of each token by adapting it to its local context , which goes beyond the segment it resides in and can be flexibly defined according to the target task . Experiments confirm that TAN and its augmentations supports existing state-of-the-art NLMs and verify its following advantages ; •Theoretical contributions : TAN adapts Transformer based pre-trained NLMs to the unconditional text generation task via topics while preserving their architectuures . •Practical contributions : As TAN is a flexible plug-and-play model , and does not need to update the parameters of pre-trained Transformer NLMs , it generates more target-specific text at a lower computational cost than using previous NLMs alone . 2 PREVIOUS WORK . Recently , pre-trained neural language models ( NLMs ) , such as BERT ( Devlin et al. , 2019 ) , GPT2 ( Radford et al. , 2019 ) , XLNet ( Yang et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) , and ALBERT ( Lan et al. , 2020 ) use of Transformer ( Vaswani et al. , 2017 ) for learning contextualized text representations , and have yielded great advances for NLP tasks . These NLMs can be fine-tuned on many natural language understanding ( NLU ) tasks such as named entity recognition , question answering and text classification . Though achieving appealing performances , these Transformer-based models are better at exploring the relationships among local tokens than the document global semantics ( Wang et al. , 2020 ) . As no Transformer-based model considers these explicit semantics , Wang et al ( Wang et al. , 2020 ) rearrange and further explore the semantics of the topic model and develop a friendly topic assistant for Transformer-based abstractive summarization models . UNIfied pre-trained Language Model ( UNILM ) ( Dong et al. , 2019 ) is applied to NLU and natural language generation ( NLG ) tasks by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction is conditioned on . However , Transformers are usually trained on disjoint fixed-length segments , without any information flow across segments ( Dai et al. , 2019 ) , limiting the contextualization within the current segment . Therefore , they often fail to take full advantage of many other rich contextual information , such as longer-range word dependencies beyond the segment length and semantic relationships between neighboring segments . While BertSUM ( Wang et al. , 2020 ) notes that topic models are better at learning explicit document semantics than Transformers , our model is applied to the domain adaptation of text generation task , and has the different architecture and training tasks to perform topic transfer between domains . 3 Methodology . 3.1 Problem formulation . Our approach aims to adapt pre-trained NLMs so that they generate unconditional texts reflecting the target-domain corpus more than corpora used in the pre-training . In NLP , language models are trained as conditional language models for specific tasks that require text generation Bengio et al . ( 2003 ) . Given text sequence x = { x1 , · · · , xT } and a dataset D = { x1 , · · · , xD } , NLMs are pre-trained by minimizing the negative likelihood under forward autoregressive factorization : minθLLM ( θ ) = minθ − |D|∑ d=1 T∑ t=1 logPθ ( xd , t|xd,1 : t−1 ) , ( 1 ) where θ represents model parameters . Since our approach focuses on the fact that the gap between the corpora for pre-trained models and the corpus of target domain is different in this distribution , it aims to train these models by bringing these distributions to the distributions observed in the target domain . For example , given “ My favorite artist is ” , the pre-trained model predicts “ Michelangelo ” as the next word , while the fine-tuned model predicts “ Botticelli ” . This leads us to introduce topics , z , into the NLMs and then modify Eq ( 1 ) to : minθLTLN ( θ ) = minθ − |D|∑ d=1 T∑ t=1 log Z∑ zt=1 Pθ ( xd , t|zt ) Pθ ( zt|xd,1 : t−1 ) , ( 2 ) where zt denotes the topic of t-th token , and Z is the number of topics . Different from the previous NLMs , TAN explicitly introduces topics into the generative process to provide richer contextual information for improving NLMs . This paper explores how to discover these topics , and fit this obtained these topics to existing pre-trained NLMs . 3.2 Our motivation and Architecture design . Our motivation for adaptation is to discover topics from a target domain and teach them to pre-trained NLMs while preserving the semantic meaning and language structural information�that these NLMs have . While Transformer encodes context , as local information , it requires the large size of source target data set to learn the higher-order and non-linear interaction between words , and will need more parameters , computation resource and time . Further , due to the limited position index during pre-training , most Transformer-based models have a maximum capacity of input tokens . Thus , they often truncate the length of a document to satisfy the length limitation of the encoder , which may lose some important semantics , especially for long documents . It is often observed that the learned attentive patterns of many heads are not as reasonable as we expect ( Michel et al. , 2019 ) , and we might obtain this global information in the upper blocks by increasing the number of blocks in transformers ( Dosovitskiy et al. , 2021 ) , where the transformer architecture requires a large number of parameters and its computational cost is very high . This motivates us to discover explicitly topics from a target domain share them with pre-trained NLMs . As 1 ) a topic describes a co-occurrence pattern of tokens with similar semantics , and 2 ) the differences between pre-training and fine-tuning data sets are not only in the topic itself , but also in the ratio of topics , our model needs to be trained separately from pre-trained NLMs so that topics , z , are not buried . This leads our model to have both the encoder and the decoder , as shown in Figure 1 . The encoder discovers topics from a given target domain and fuses topics to the decoder with their distributions . The difference between this encoder and the other encoders is in finding topic distributions from a given target domain , aligning topics between the encoder and the decoder on the top level encoder/decoder block , and replacing the word generation network in the decoder with these topic distributions . The decoder can reuse pre-trained NLMs ( e.g. , GPT2 ) , jointly learns and shares topics with the encoder . Therefore , this model does not break any structure of the original Transformer network , and can preserve the power of pre-trained NLMs . 3.3 Input . Given a target domain corpus , TAN feeds the same text as input to the encoder ( the source text ) , and the decoder each ( the target text ) , from each of this target , as shown in Fig 1 It layers convert the inputs into token ( linguistic ) embedding , and adds special tokens [ CLS ] , [ SEP ] , [ EOS ] , and < s > . Following the text preprocessing of other Transformer based NLMs , TAN tokenizes each input text as the linguistic input of token embedding , where each subword is embedded with Word Piece ( Wu et al. , 2016 ) or other model-specific tokenizer ( e.g. , Byte-Pair Encoding ( BPE ) vocabulary ( Radford et al. , 2019 ) ) whose length equals the length of its input . [ CLS ] token is only inserted prior to the token , and denotes the class of each source text . [ SEP ] token is assigned to the end of each sentence in each input sequence , and indicates a sentence break . [ EOS ] token is assigned only after the last token in each input sequence . < s > token is only inserted prior to the token in each target text . A learnable sequence position embedding is added to every input element indicating its order in the input sequence , the same as other models . Because the Transformer encoder has no recurrence , it adds some information about positions into the input embeddings . As the positions of [ CLS ] , [ SEP ] and [ EOS ] , are fixed in every input sequence , their position embeddings are also fixed as constant values for every input sequence . The final representation for each token is obtained by summing these embedding .

Summary: This work presents topic aware language model for unconditional text generation. Basic idea is to introduce a topic latent variable $z_t$ and prediction of the next symbol $x_t $is conditioned on $z_t$ together with the topic prediction from the history of previously generated symbols $x_{<t}$.  The distribution is marginalized by summing over $z-t$. The proposed model is based on GPT-2, but formulated as an encoder-decoder by concatenating the encoder and decoder in a single network, and BERT like encoding is used to predict topic distribution.  Experiments are carried out on text classification data set, e.g., Amazon and Yelp, with topic annotations, and show gains in terms of perplexity and BLEU.  The  contribution of this work is:  -  A topic aware model for language model with an encoder and decoder framework implemented within GPT-2 model structure.