Topic Aware Neural Language Model: Domain Adaptation of Unconditional Text Generation Models

1 INTRODUCTION . Our goal is to adapt pre-trained neural language models ( NLMs ) to the unconditional text generation task for the target domain . As with global semantic information , topic models ( Blei et al. , 2003 ; Kawamae , 2018 ; Wang et al. , 2020 ) and their extensions take a global statistical view and look at the word distributions of topics across a given corpus ; they represent each document as a bag-of-word ( BOW ) vector . Topics are global variables , describes the distributions over all tokens in the vocabulary , and form interpretable representations over texts . Although these models organize a given corpus into small sets of prominent topics and have been proven to be powerful tools for uncovering latent structure , they are not , in the strict sense , sequence models . Recently , Transformer has been applied as pre-trained NLMs ( Vaswani et al. , 2017 ; Yang et al. , 2019 ) , and are shown to be highly successful in the domain of natural language processing . Their success motivate us to explore how to adapt Transformer based NLMs for reflecting a given fine-tuning data and generating more private text with a few modifications than ever before . The corpus size of this fine-tuning is generally smaller than that of pre-training corpora , and may be unbalanced for training NLMs . For example , the size of the popular pre-training data , Giga5en ( Parker et al. , 2011 ) , and ClueWeb 2012-B1 is 16G , and 25TB , respectively . The fine-tuning needs a different approach to training NLMs than the pretraining approach , as pre-trained NLMs should be biased towards the patterns of language used in the training data ( Keskar et al. , 2019 ) , and may not lead to domain adaptation . To tackle this imbalance problem , we explore how to adapt pre-trained Transformer based NLMs to the target domain in the fine-tuning , while preserving their advantages . Since topics exist over these datasets and represents their semantic structures such as higher-order and non-linear interaction between words , our model , Topic Aware Neural Language Model ( TAN ) , focuses on topics to resolve this imbalance between these corpora and bridge their semantic gap . As topics reflect a set of co-occurring words , semantic information , and their 1https : //www.lemurproject.org/clueweb09.php/ syntactic structure , as global statistical information , TAN introduces a new manipulation , “ topic-alignment ( TA ) ” , and new training tasks such as Topic Distribution Modeling ( TDM ) and Topic Embedding Modeling ( TEM ) While previous Transformer based NLMs are better at learning from the predefined segment length such as the context , local information using Multi-head attention ( Vaswani et al. , 2017 ) , TAN injects topics into Transformer to train NLMs by fine-tuning with emphasizing these dependencies over segments in the target domain via topics . This model not only captures global semantic coherence of all segments and word concurrence patterns , but also enriches the representation of each token by adapting it to its local context , which goes beyond the segment it resides in and can be flexibly defined according to the target task . Experiments confirm that TAN and its augmentations supports existing state-of-the-art NLMs and verify its following advantages ; •Theoretical contributions : TAN adapts Transformer based pre-trained NLMs to the unconditional text generation task via topics while preserving their architectuures . •Practical contributions : As TAN is a flexible plug-and-play model , and does not need to update the parameters of pre-trained Transformer NLMs , it generates more target-specific text at a lower computational cost than using previous NLMs alone . 2 PREVIOUS WORK . Recently , pre-trained neural language models ( NLMs ) , such as BERT ( Devlin et al. , 2019 ) , GPT2 ( Radford et al. , 2019 ) , XLNet ( Yang et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) , and ALBERT ( Lan et al. , 2020 ) use of Transformer ( Vaswani et al. , 2017 ) for learning contextualized text representations , and have yielded great advances for NLP tasks . These NLMs can be fine-tuned on many natural language understanding ( NLU ) tasks such as named entity recognition , question answering and text classification . Though achieving appealing performances , these Transformer-based models are better at exploring the relationships among local tokens than the document global semantics ( Wang et al. , 2020 ) . As no Transformer-based model considers these explicit semantics , Wang et al ( Wang et al. , 2020 ) rearrange and further explore the semantics of the topic model and develop a friendly topic assistant for Transformer-based abstractive summarization models . UNIfied pre-trained Language Model ( UNILM ) ( Dong et al. , 2019 ) is applied to NLU and natural language generation ( NLG ) tasks by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction is conditioned on . However , Transformers are usually trained on disjoint fixed-length segments , without any information flow across segments ( Dai et al. , 2019 ) , limiting the contextualization within the current segment . Therefore , they often fail to take full advantage of many other rich contextual information , such as longer-range word dependencies beyond the segment length and semantic relationships between neighboring segments . While BertSUM ( Wang et al. , 2020 ) notes that topic models are better at learning explicit document semantics than Transformers , our model is applied to the domain adaptation of text generation task , and has the different architecture and training tasks to perform topic transfer between domains . 3 Methodology . 3.1 Problem formulation . Our approach aims to adapt pre-trained NLMs so that they generate unconditional texts reflecting the target-domain corpus more than corpora used in the pre-training . In NLP , language models are trained as conditional language models for specific tasks that require text generation Bengio et al . ( 2003 ) . Given text sequence x = { x1 , · · · , xT } and a dataset D = { x1 , · · · , xD } , NLMs are pre-trained by minimizing the negative likelihood under forward autoregressive factorization : minθLLM ( θ ) = minθ − |D|∑ d=1 T∑ t=1 logPθ ( xd , t|xd,1 : t−1 ) , ( 1 ) where θ represents model parameters . Since our approach focuses on the fact that the gap between the corpora for pre-trained models and the corpus of target domain is different in this distribution , it aims to train these models by bringing these distributions to the distributions observed in the target domain . For example , given “ My favorite artist is ” , the pre-trained model predicts “ Michelangelo ” as the next word , while the fine-tuned model predicts “ Botticelli ” . This leads us to introduce topics , z , into the NLMs and then modify Eq ( 1 ) to : minθLTLN ( θ ) = minθ − |D|∑ d=1 T∑ t=1 log Z∑ zt=1 Pθ ( xd , t|zt ) Pθ ( zt|xd,1 : t−1 ) , ( 2 ) where zt denotes the topic of t-th token , and Z is the number of topics . Different from the previous NLMs , TAN explicitly introduces topics into the generative process to provide richer contextual information for improving NLMs . This paper explores how to discover these topics , and fit this obtained these topics to existing pre-trained NLMs . 3.2 Our motivation and Architecture design . Our motivation for adaptation is to discover topics from a target domain and teach them to pre-trained NLMs while preserving the semantic meaning and language structural information�that these NLMs have . While Transformer encodes context , as local information , it requires the large size of source target data set to learn the higher-order and non-linear interaction between words , and will need more parameters , computation resource and time . Further , due to the limited position index during pre-training , most Transformer-based models have a maximum capacity of input tokens . Thus , they often truncate the length of a document to satisfy the length limitation of the encoder , which may lose some important semantics , especially for long documents . It is often observed that the learned attentive patterns of many heads are not as reasonable as we expect ( Michel et al. , 2019 ) , and we might obtain this global information in the upper blocks by increasing the number of blocks in transformers ( Dosovitskiy et al. , 2021 ) , where the transformer architecture requires a large number of parameters and its computational cost is very high . This motivates us to discover explicitly topics from a target domain share them with pre-trained NLMs . As 1 ) a topic describes a co-occurrence pattern of tokens with similar semantics , and 2 ) the differences between pre-training and fine-tuning data sets are not only in the topic itself , but also in the ratio of topics , our model needs to be trained separately from pre-trained NLMs so that topics , z , are not buried . This leads our model to have both the encoder and the decoder , as shown in Figure 1 . The encoder discovers topics from a given target domain and fuses topics to the decoder with their distributions . The difference between this encoder and the other encoders is in finding topic distributions from a given target domain , aligning topics between the encoder and the decoder on the top level encoder/decoder block , and replacing the word generation network in the decoder with these topic distributions . The decoder can reuse pre-trained NLMs ( e.g. , GPT2 ) , jointly learns and shares topics with the encoder . Therefore , this model does not break any structure of the original Transformer network , and can preserve the power of pre-trained NLMs . 3.3 Input . Given a target domain corpus , TAN feeds the same text as input to the encoder ( the source text ) , and the decoder each ( the target text ) , from each of this target , as shown in Fig 1 It layers convert the inputs into token ( linguistic ) embedding , and adds special tokens [ CLS ] , [ SEP ] , [ EOS ] , and < s > . Following the text preprocessing of other Transformer based NLMs , TAN tokenizes each input text as the linguistic input of token embedding , where each subword is embedded with Word Piece ( Wu et al. , 2016 ) or other model-specific tokenizer ( e.g. , Byte-Pair Encoding ( BPE ) vocabulary ( Radford et al. , 2019 ) ) whose length equals the length of its input . [ CLS ] token is only inserted prior to the token , and denotes the class of each source text . [ SEP ] token is assigned to the end of each sentence in each input sequence , and indicates a sentence break . [ EOS ] token is assigned only after the last token in each input sequence . < s > token is only inserted prior to the token in each target text . A learnable sequence position embedding is added to every input element indicating its order in the input sequence , the same as other models . Because the Transformer encoder has no recurrence , it adds some information about positions into the input embeddings . As the positions of [ CLS ] , [ SEP ] and [ EOS ] , are fixed in every input sequence , their position embeddings are also fixed as constant values for every input sequence . The final representation for each token is obtained by summing these embedding .

Summary: The paper describes an approach to adapting pre-trained Transformer-based NLMs to learn a topic distribution along with the language model. Motivated by the inability of Transformers to capture global semantic context from documents, they propose TAN, to fine-tune using the topic distribution on a target domain dataset. Experimental results on topic coherence and text generation show the strength of their model to ourperform benchmarks.