Topic Aware Neural Language Model: Domain Adaptation of Unconditional Text Generation Models

Abstract: Our goal is to adapt pre-trained neural language models (NLMs) to the unconditional text generation task within the target domain.
Because many Transformer based NLMs are trained on more massive and heterogeneous corpora than this target domain,
the difference between these corpora and the target domain raises the question of whether these NLMs can provide their benefits to this task even after the fine-tuning.
To tackle these problems, our approach focuses on topics to bridge the semantic gap between these corpora and the target domain corpus,
and relates them at a topic level.
That is, this approach injects topics into these NLMs and trains them via topics behind these dependencies over segments,
introducing both topic alignment (TA) and training tasks (TDM and TEM),
while previous Transformer based NLMs are better at learning from the predefined segment length such as the context.
Experiments show that this approach contributes to resolve the imbalance between these corpora,
and can tailor previous pre-trained NLMs to generate coherent and semantically valid text reflecting a given small fine-tuning corpus.

Reviews and author responses:

The paper proposes a new topic-aware transformer-based language model with better domain adaptation ability. The paper introduces topic alignment to the NLMs. The paper treats the topic as an additional latent variable in the NLMs. The authors test their models on two datasets. The contribution is to introduce topics into the pre-trained language model. Strength
1. The motivation of this paper is clearly given. The paper introduces a new topic alignment to align the target domain to the pre-trained NLM. The paper tried to treat non-linguistic tokens as topic words. The idea to combine topics implicitly in the transformer architecture is pretty interesting. The paper conduct an ablation study to analyze each component ()TA, TEM, TDM of the proposed model.

Weakness
1.  The paper fails to include enough qualitative analysis in the experiment section. The paper needs to show more insight analysis for the TAN. For example, The paper can conduct human evaluation to find the potential pros and cons of the proposed model. The paper can also visualize some top topics retrieved from each model. For automatic evaluation, the paper can also include more metrics (e.g. METEOR, BERTscore) to evaluate the semantic aspects of the proposed model.

2.  The font of the paper is not consistent with the template. The inline citation style is not consistent with guidelines. Figure 1 needs to be redrawn. It is quite confusing to understand Figure 1.  Overall, this paper tries to enhance the domain-adaptation ability of the NLMs.  I recommend rejection for this paper.

The paper describes an approach to adapting pre-trained Transformer-based NLMs to learn a topic distribution along with the language model. Motivated by the inability of Transformers to capture global semantic context from documents, they propose TAN, to fine-tune using the topic distribution on a target domain dataset. Experimental results on topic coherence and text generation show the strength of their model to ourperform benchmarks.  Pros:
- The empirical results in Table 2 and 3 show impressive gains over compared benchmarks. 
- There is a obvious depth of technical work and understanding in the development of the model.
- The combination of these two schools of thought, on topic model and embeddings is an interesting intersection for study. 

Cons:

- In section 4.2, it is not clear to me where the topic assignments come from, and how Eq 2) is implemented such that an existing transformer, like BERT/GPT2, can learn it without modification to the transformer. 
-If a stated goal is to overcome the need for Transformer models to truncate long documents, why do that in preprocessing here?
-Related work does not seem to capture similar previous compositions. While it is unreasonable to expect full coverage, papers such as "Topic Compositional Neural Language Model" and "Topically Driven Neural Language Model" seem relevant for true comparison.
- There are many grammatical mistakes that make it difficult to follow. It's also clear that space was very aggressively removed resulting in overlapping text and tables/figures. 

Overall, I have a hard time recommending this paper for acceptance in it's current form. While the technical depth and empirical results are impressive, it is difficult to comprehend core elements. 
 Promising approach and results, showing depth of technical knowledge, but lack of clarity in exposition. 

This work presents topic aware language model for unconditional text generation. Basic idea is to introduce a topic latent variable $z_t$ and prediction of the next symbol $x_t $is conditioned on $z_t$ together with the topic prediction from the history of previously generated symbols $x_{<t}$.  The distribution is marginalized by summing over $z-t$. The proposed model is based on GPT-2, but formulated as an encoder-decoder by concatenating the encoder and decoder in a single network, and BERT like encoding is used to predict topic distribution.  Experiments are carried out on text classification data set, e.g., Amazon and Yelp, with topic annotations, and show gains in terms of perplexity and BLEU.

The  contribution of this work is:

-  A topic aware model for language model with an encoder and decoder framework implemented within GPT-2 model structure. # Strengths

It is a potentially new approach for topic aware language model.

# Weakness

One of the major concerns to this work is the clarity and presentation. The model description is totally unclear in that symbols are not consistent and many terminologies are introduced, e.g., encoder and decoder, thought the model is based on GPT-2. Also, it is unclear why encoder-decoder like model structure is used in the proposed model, given that it is an unconditional generation model. I'd suggest author to revise the writing with proof reading.

* It is unclear what information is encoded as an encoder and how that would be used in the decoder. In Equation 1 and 2, there is no notion of encoder part in the model and it is one of the source of confusions. Note that the text generation can access only previously generated text, and thus, it is not obvious what is represented by the encoder. If it is previously generated texts, does it mean that the proposed model recomputes the whole network every time a new word is predicted?

* It is also unclear why BERT like encoding is employed for the encoder part, given that that the proposed model is unconditional generation.

* Other comment:

  * The  network structure sounds very similar to https://proceedings.neurips.cc/paper/2018/file/4fb8a7a22a82c80f2c26fe6c1e0dcbb3-Paper.pdf

  * There's confusion of symbols elsewhere, e.g., $Z_t$ and $z_t$, $x_{k, t}$ and $x^k_t$. Please fix all the math notations.

  * It is not clear what is linguistic input of token embedding in section 4.1

  * Citation style is not appropriate. Please fix according to the guideline of ICLR. I'd recommend rejection of this submission given that it is totally unclear what is described in this paper.

Meta-review: This paper proposes topic-aware NLMs, an approach to adapt pre-trained transformer based language models to a specific domain/topic. The work seems novel and interesting, and the paper presents good results. However, it bypasses many technical details that would enable a good understanding and appreciation of the work. In the responses, the authors clarify some of these issues, however, it would be beneficial to re-review the paper given the suggested changes. In addition to clarity, lack of detailed experimental analysis is an issue, and it would be useful to integrate the analysis suggested in the reviews.

Decision: Reject